Skip to content
This repository has been archived by the owner on May 5, 2023. It is now read-only.

Latest commit

 

History

History
275 lines (220 loc) · 11 KB

README.md

File metadata and controls

275 lines (220 loc) · 11 KB
Raw

testframework

The test script is a framework for the control of test case execution.

To learn more start with execution of help and man:

./bin/runTTF --help

and

./bin/runTTF --man

In directory samples you can see the Test Collection ANewTest, which is a sample for a basic Test Collection

Installation

The release package is a self extracting script. Execute it and follow the instructions. You can install the tool into an arbitrary palace. If you want to install the tool into a system directory execute the installation script as root. It is recommended qto enter the location of the test framework into your PATH like:

export PATH="/bin:$PATH"

Whats new:

Version 1.0.2

  • correction in streams initialization

Version 1.0.1

  • corrections in selftest

Version 1.0.0:

  • re-name TestProperties to TestProperties.sh
  • Removed propertie section in preamble, all properties moved to script section
  • Preamble now supports only variantList, variantCount and timeout
  • Preamble requires now quoting (for lists)
  • Check whether variantCount and timeout are digits
  • New test steps / pres / fins invocation local space separated list STEPS/PREPS/FINS
  • New test steps / pres / fins invocation local array STEPS/PREPS/FINS possible
  • New test steps / pres / fins invocation local functions testStep, testPreparation, testFinalization
  • Global space separated lists steps / pres / fins invocation re-named: TTRO_prepsCase TTRO_prepsSuite TTRO_preps TTRO_stepsCase TTRO_finsCase TTRO_finsSuite TTRO_fins
  • Improved handling and display in case of erroros in suite/collection initialization and preparation
  • All function calls for collection/suite/case preparation and steps and finalization are now unchecked called to improve failure propagation
  • Collections variant support
  • TstCollection.sh and TestSuite.sh are now only once sourced in collection and suite context
  • TTRO_extraTime renamed to TTP_additionalTime
  • New utis functions isArray, isFunction, arrayHasKey, isExistingAndTrue, isTrue, isFalse
  • Re-named properties TTRO_noStart -> TTPN_..
  • Handle debug and verbose as propertie
  • Re-names TTRO_caseVariant -> TTRO_variantCase TTRO_suiteVariant -> TTRO_variantSuite

Version 0.2.0:

  • Dummy Suite support: Test cases can be defined without suite context.
  • Test collection now requires a TestCollection.sh file and the TestProperties file is now optional
  • Installer
  • New global initialization in streamsutils
  • Use variables for sab file main composite ..
  • New test cases
  • Streams sample

Version 2.0.0:

  • Remove concept of TestCollection.sh
  • Introductio of nested Test Suites

Version 2.1.0:

  • New feature: Browser is started after test collection execution and execution index is displayed in browser

Version 2.2.0:

  • Simple initialization of test tools. Test tools are now imported one at the beginning of tests. Test tools function must be exported.

Version 2.3.0:

  • Changed reference display handling: The parameter --ref requires now the name of the module to display.
  • Changed prefixes TTP_ -> TTPR and TTPN_ -> TTPRN_
  • new function setFailure ti signal failure reason in test case
  • new function skip to skip case and suite

Version 2.4.0

  • New Feature: Category Control

Version 2.5.0

  • New functions: executeAndLog, executeLogAndSuccess, executeLogAndError
  • New functions: linewisePatternMatchAndIntercept, linewisePatternMatchInterceptAndSuccess, linewisePatternMatchInterceptAndError

Version 2.6.0:

  • New function names splCompile, splCompileAndLog, splCompileAndIntercept, splCompileInterceptAndSuccess and splCompileInterceptAndError
  • New variable TT_dataDir
  • New functions waitForFileToAppear, waitForFin
  • New functions checkJobNo
  • New variables TT_waitForFileName, TT_waitForFileInterval, TTTT-jobno
  • New spl utility toolkit streamsx.testframe with FileSink1
  • New directory structure for installation target now we have /bin /samples and /streamsx.testframe

Version 2.6.1:

  • New function
  • com.ibm.streamsx.testframe::FileSinke 1 : has new parameter singleTupleFiles with default true

Version 2.6.2:

  • New function getLineCount
  • Error corrections
  • Improvementes in builReleasePackage.sh

Version 2.7.0:

  • New functions to get streams job health: jobHealthyVariable, jobHealthy, jobHealthyAndIntercept,
  • New function to wait for final file and check health: waitForFinAndHealth
  • New utils function: trim, getSystemLoad, getSystemLoad100
  • Print the elapsed time

Version 2.7.2:

  • Some fixes in category handling

Version 2.7.3:

  • Simplified preambl handling: Now quote like a reqular assignement

Version 2.9.0:

  • New function copyAndMorph now works with expressions like: ^[[:space:]]//varid1:varid2.. are effective if the argument $3 equal one of the varid1, or varid2.. ^[[:space:]]//<!varid1:varid2> are not effective if the argument $3 equal one of the varid1, or varid2.. and is able to work with lists of version ids in one line
  • Better logging of job and pid specification
  • Introduced special report for suites for automated tests
  • Check variant id for valid characters [a-zA-Z0-0_]
  • New option setVar xtraprint
  • Make domain now with checkpointRepository and fileStoragePath

Version 3.0.0:

  • new command line option --summary
  • Corrections in streams utils: better jobno handling; submitjob may have optional submission time params
  • New command line parameter -s|--sequential - Sequential test execution and option -j with new semantics
  • Per default a suite has no category
  • Function setFailure with better state check
  • New functions: setSkip, isSkip. Print reason for skipped cases and failures
  • correction in props TTPRN and added property test
  • Print skip reason and failure reason to index
  • Suites with error cases are now in read in html
  • Allow preambl lienes to be continued
  • Add command line option --clean to enable clean start of streams instances
  • Write protect used utils and other functions
  • correct identifier character class of variants to 0-9a-zA-Z-_
  • Produce global SUMMARY.txt file
  • Allow preambl lines to be continued
  • Better jobid control. Now re-used jobids should not cause errors
  • Use other formatting in web index
  • Add elapsed time and add result to special report
  • New replacement in New function morphFile: patterns like <#$varname#> are replaced with the expansion of $varname

Version 3.0.1:

  • Add version number to final summary
  • Correction for more then 9 arguments

Version 3.1.0:

  • Use environment variable TTRO_propertyFiles to control Property files
  • Adapt ..._summary file to junit output format
  • Correct doc for return values
  • Print result summary in order: executed, failed, errors, skipped

Version 3.1.2:

  • Correct TTTT_state variable

Version 3.1.3:

  • Workaround for lang setting in jobs command for bash 4.1.2

Version 3.2.0:

  • New command line option --load-reduce
  • Use variables TTPR_noStartxxx, TTPR_noStopxxx instead of TTPRN_ variables
  • New functions cleanUpInstAndDomainAtStop and cleanUpInstAndDomainAtStart
  • Evaluate skip attributes if a cases list is given as command line parameter
  • Add skipped cases in special summary file

Version 3.3.0:

  • Print no warning if cancelJob is used in finalization phase and empty job variable
  • Correction spl compile when TT_toolkitPath is empty or not existent
  • Better function parameter check in utils function setFailure: must not be called with an empty argument
  • New utils functions : checkAllFilesExist, checkLineCount and checkAllFilesEqual
  • Enhancements in toolkit FileSink1
  • Add spldocs of spl toolkit
  • New streamsutils functions cancelJobAndLogVariable and cancelJobAndLogVariable

Version 3.3.1:

  • Allow compile time parameters in function splCompile

Version 3.4.0:

  • Function copyAndMorph: vartiant identifieres can now be a pattern

Version 3.4.1:

  • Remove unnecessary echos

Version 3.5.0:

  • Timeout in test case preambl is not used if the value is lesser than the timeout in property TTPR_timeout

Version 3.5.1:

  • Correct timer properties handling for nested suites

Version 3.5.2:

  • Preamble continuation lines must not remove leading whitespaces
  • Allow tool imports in property file
  • Remove deprecated variables in Streams1 sample

Version 3.6.0:

  • Improved error handling in case of duplicate case variant or suite variant
  • New utils functions findTokenInFiles, findTokenInDirs, checkTokenIsInFiles, checkTokenIsNotInFiles, checkTokenIsInDirs, checkTokenIsNotInDirs
  • New streamsutil functions: getHostList, getHostListVariable, getJobLogDirs, getJobLogDirsVariable, checkLogsNoError
  • Correction streamsutil: all TT_ variables set with function setVar to enable export

Version 3.6.1:

  • New function checkLogsNoError2, Check if pe log files and pe stdouterr have no error Token ERROR

Version 3.7.0:

  • Add command option --shell
  • breakdown in browser pages

Version 3.8.0:

  • Clearer output of function linewisePatternMatch
  • Introduce search path for modules
  • Removed command line option --tools
  • New command line option --threads-per-job
  • Better varaible protection and environment print before test scripts is sourced
  • Rename functions of module coreutils to special namespace TTTF_
  • Better handling of preambl errors
  • Move all used variables which may interfere user variables to namespace TTTT_ or TTTI_
  • Better info output for suite start and end
  • Breakdown in cases loop in suite
  • Display elapsed time in suite index even in case of test case timeout

Version 3.9.0:

  • Introduce new utils function arrayAppend, arrayInsert and arrayDelete
  • Allow --no-prompt and --noprompt command line option
  • Utils function linewisePatternMatchArray exits if input array is empty
  • New toolkit operator Echo
  • Utils function printErrorAndExit param #2 is now optional and runntime error is default
  • New streamsutils function waitForJobHealth
  • New streamsutils function waitForFinAndCheckHealth
  • Category Control is disables if a test case wildcard list was entered
  • New preambl feature exclusive execution property

Version 3.9.1:

  • Corrections in sigint handling in test suite
  • Better printout format for job start-and job-end
  • Cleanup in job control logic
  • Default timeout changed to 240 sec; defauld additional time changed to 60 sec.

Version 3.9.2:

  • Reset errexit- and nounset option is disabled in test cases finalization phase
  • Test case now kills child jobs when case is finished
  • Correction in jobcontrol when jobs are fast finished (checkDuplicateJobspec)

Version 3.9.3:

  • errexit and nounset now also disabled in suite finalization
  • Sort output for job START and END
  • Code cleanups

Version 3.9.4:

  • Test Case tear down: make killtree more verbose

Version 3.9.5:

  • Fix: Errors during Case preparation are not counted as an error
  • Avoid unnecessary error traps during Case/Suite finalization
  • Update SimpleCases sample to demonstrate failure propagation

Version 3.9.6:

  • The final result printout is now more verbose
  • In case of test suite exits, the finalization steps are now executed unconditional

Version 3.9.7:

  • Remove duplicate printout of error suites in final summary

Version 3.9.8:

  • Function waitForFinAndHealth returns immediately if a failure was set during waitForJobHealth