In order to write test suites you need to understand the output protocol, which is TAP, the Test Anything Protocol.
The protocol is trivially to produce, you can do it with simple Shell echos or you can use TAP emitting toolchains, like practically all Test::* modules from the Perl world.
This chapter explains the protocol and the Tapper specific extensions, which are usually headers that can be transported inside TAP comments.
Example:
1..3
ok
ok
not okRemarks:
3 single tests planned
the two first went good
the last went wrong
Example:
1..3
ok 1
ok 2
not ok 3Remarks:
Missing test lines (eg. due to internal bummers) can be detected.
Example with missing test:
1..3
ok 1
not ok 3Remarks:
Parsing will later say `` test 2 expected but got 3''
Example:
1..3
ok 1 - input file opened
ok 2 - file content
not ok 3 - last lineRemarks:
Readability.
Example:
1..3
ok 1 - input file opened
ok 2 - file content
not ok 3 - last line # TODORemarks:
mark not yet working tests as "TODO"
allows test-first development
"ok" TODOs are recognized later
("unexpectedly succeeded") =item * We also use it to ignore known issues with still being able to findthem later.
Example:
1..3
ok 1 - input file opened
ok 2 - file content
not ok 3 - last line # TODO just speccedRemarks:
comment the TODO reason
Example:
1..3
ok 1 - input file opened
ok 2 - file content
ok 3 - last line # SKIP missing prerequisitesRemarks:
mark tests when not really run (note it's set to ``ok'' anyway)
keeps succession numbers in sync
Example:
1..3
ok 1 - input file opened
ok 2 - file content
not ok 3 - last line # TODO just specced
# Failed test 'last line'
# at t/data_dpath.t line 410.
# got: 'foo'
# expected: 'bar'Remarks:
Unstructured details
Example:
1..3
ok 1 - input file opened
ok 2 - file content
not ok 3 - last line # TODO just specced
---
message: Failed test 'last line' at t/data_dpath.t line 410.
severity: fail
data:
got: 'foo'
expect: 'bar'
...Remarks:
Structured details
allows parsable diagnostics
we use that to track values inside TAP
have a leading test line with number+description
track complete data structures according to it
e.g., benchmark results
TAP allows comment lines, starting with #. We allow meta information transported inside those comment lines when declared with Tapper specific headers.
Example:
1..3
# Tapper-Suite-Name: Foo-Bar
# Tapper-Suite-Version: 2.010013
ok 1 - input file opened
ok 2 - file content
not ok 3 - last line # TODO just speccedRemarks:
we use diagnostics lines (``hot comments'')
semantics only to our TAP applications
These are the headers that apply to the whole report:
# Tapper-suite-name: -- suite name
# Tapper-suite-version: -- suite version
# Tapper-machine-name: -- machine/host name
# Tapper-machine-description: -- more details to machine
# Tapper-reportername: -- user name of the reporter
# Tapper-starttime-test-program: -- start time for complete test
(including guests)
# Tapper-endtime-test-program: -- end time for complete test
(including guests)
# Tapper-reportgroup-testrun: -- associate this report with other
reports of same testrun_id
# Tapper-reportgroup-arbitrary: -- associate this report with other
reports of same arbitrary id
(can be any string, but should be
unique between all groups of the db,
eg., an md5-hash of common
characteristics of all test of one
group)There are more headers that apply to single sections of a report.
Standard TAP contains of exactly one block with one plan (eg., 1..5) and some TAP lines. In Tapper you can concatenate several such blocks at once. They are interpreted like different files, and are named sections in Tapper jargon.
The delimiter between such sections is the plan line. This requires the plan to come first for each section. See chapters ``Explicit section markers with lazy plans'' and ``TAP archives'' below for explicitely providing other TAP section delimiters.
Please remember: Concatenating several sections into one big block of TAP is a Tapper extension. To interact with other TAP toolchains you should try to use ``TAP archives'' when submitting sections into Tapper.
Example:
1..2
# Tapper-section: arithmetics
ok 1 add
ok 2 multiply
1..1
# Tapper-section: string handling
ok 1 concat
1..3
# Tapper-section: benchmarks
ok 1
ok 2
ok 3Remarks:
we recognize ``sections'', each with its own plan
allows structuring of results,
better readability later in web interface
These are the headers that apply to single sections:
# Tapper-explicit-section-start: -- explicitely start a section now
instead of autorecognition
# Tapper-ram: -- memory
# Tapper-cpuinfo: -- what CPU
# Tapper-uname: -- kernel information
# Tapper-osname: -- OS information
# Tapper-bios: -- BIOS information
# Tapper-flags: -- flags, usually linux kernel
# Tapper-changeset: -- exact changeset of the currently
tested software or kernel
# Tapper-description: -- more description of the currently
tested software or kernel,
e.g., if changeset is not enough
# Tapper-uptime: -- uptime, maybe the test run time
# Tapper-language-description: -- for Software tests,
like "Perl 5.10", "Python 2.5"
# Tapper-reportcomment: -- Freestyle comment
# Tapper-xen-version: -- Xen version
# Tapper-xen-changeset: -- particular Xen changeset
# Tapper-xen-dom0-kernel: -- the kernel version of the dom0
# Tapper-xen-base-os-description: -- more verbose OS information
# Tapper-xen-guest-description: -- description of a guest
# Tapper-xen-guest-test: -- the started test program
# Tapper-xen-guest-start: -- start time of test
# Tapper-xen-guest-flags: -- flags used for starting the guest
# Tapper-kvm-module-version: -- version of KVM kernel module
# Tapper-kvm-userspace-version: -- version of KVM userland tools
# Tapper-kvm-kernel: -- version of kernel
# Tapper-kvm-base-os-description: -- more verbose OS information
# Tapper-kvm-guest-description: -- description of a guest
# Tapper-kvm-guest-test: -- the started test program
# Tapper-kvm-guest-start: -- start time of test
# Tapper-kvm-guest-flags: -- flags used for starting the guest
# Tapper-simnow-version: -- version of simnow
# Tapper-simnow-svn-version: -- svn commit id of simnow
# Tapper-simnow-svn-repository: -- used svn repository
# Tapper-simnow-device-interface-version: -- internal simnow device
interface version
# Tapper-simnow-bsd-file: -- used BSD file (machine model)
# Tapper-simnow-image-file: -- used OS image botted in simnow
(usually similar to
Tapper-osname or
Tapper-xen-base-os-description or
Tapper-kvm-base-os-description)There are groups of reports (e.g. for virtualization scenarios), optionally identified by a testrun ID or by an arbitrary ID. Every report has an ID and a set of meta information. A report consists of sections, which can each have section specific set of meta information.
The resulting meta information hierarchy looks like this.
Reportgroup
- testrun reportgroup ID - arbitrary reportgroup IDReport
- report ID - Tapper-suite-name - Tapper-suite-version - Tapper-machine-name - Tapper-machine-description - Tapper-reportername - Tapper-starttime-test-program - Tapper-endtime-test-program - Tapper-reportgroup-testrun - Tapper-reportgroup-arbitrarySection
- Tapper-explicit-section-start - Tapper-ram - Tapper-cpuinfo - Tapper-uname - Tapper-osname - Tapper-bios - Tapper-flags - Tapper-changeset - Tapper-description - Tapper-uptime - Tapper-language-description - Tapper-reportcomment - Tapper-xen-version - Tapper-xen-changeset - Tapper-xen-dom0-kernel - Tapper-xen-base-os-description - Tapper-xen-guest-description - Tapper-xen-guest-test - Tapper-xen-guest-start - Tapper-xen-guest-flags - Tapper-kvm-module-version - Tapper-kvm-userspace-version - Tapper-kvm-kernel - Tapper-kvm-base-os-description - Tapper-kvm-guest-description - Tapper-kvm-guest-test - Tapper-kvm-guest-start - Tapper-kvm-guest-flags - Tapper-simnow-version - Tapper-simnow-svn-version - Tapper-simnow-svn-repository - Tapper-simnow-device-interface-version - Tapper-simnow-bsd-file - Tapper-simnow-image-file
In TAP it is allowed to print the plan (1..n) after the test lines (a ``lazy plan''). In our Tapper environment with concatenated sections this would break the default section splitting which uses the plan to recognize a section start.
If you want to use such a ``lazy plan'' in your report you can print an Tapper header Tapper-explicit-section-start to explictely start a section. Everything until the next header Tapper-explicit-section-start is building one section. This also means that if you used this header once in a report you need to use it for all sections in this report.
The Tapper-explicit-section-start typically ignores its value but it is designed anyway to allow any garbage after the value that can help you visually structure your reports because explicit sections with ``lazy plans'' make a report hard to read.
Example:
# Tapper-explicit-section-start: 1 ------ arithmetics -------
# Tapper-section: arithmetics
ok 1 add
ok 2 multiply
1..2
# Tapper-explicit-section-start: 1 ------ string handling -------
# Tapper-section: string handling
ok 1 concat
1..1
# Tapper-explicit-section-start: 1 ------ benchmarks -------
# Tapper-section: benchmarks
ok 1
ok 2
ok 3
1..3Please note again: The sectioning in general and this auxiliary header for marking sections is a Tapper extension, not standard TAP. An alternative way better than fiddling with this sectioning is to produce TAP archives and submit them instead. See chapter ``TAP Archives''.
TAP consuming is provided via the Test::Harness aka. TAP::Parser Perl toolchain. The frontend utility to execute TAP emitting tests and evaluate statistics is prove.
$ prove t/*.t
t/00-load.........ok
t/boilerplate.....ok
t/pod-coverage....ok
All tests successful.
Files=4, Tests=6, 0 wallclock secs
( 0.05 usr 0.00 sys + 0.28 cusr 0.05 csys = 0.38 CPU)
Result: PASSRemarks:
TAP::Parser
provetooloverall success and statistics
allows
formattersused to produce web reports
It helps to not rely on Tapper extensions (like report sections) when using the prove command.
TAP is easy to produce but using it usefully can be a challenge.
Use invariable test descriptions.
Put meta information in diagnostics lines, not into test descriptions.
Use the description after
# TODO/SKIP.Cheat visible (or: don't cheat invisible).
Really use
# TODO/SKIP.
These tips keep later TAP evaluation consistent.
If we have a Xen environment then there are many guests each running some test suites but they don't know of each other.
The only thing that combines them is a common testrun-id. If each suite just reports this testrun-id as the group id, then the receiving side can combine all those autonomously reporting suites back together by that id.
So simply each suite should output
# Tapper-reportgroup-testrun: 1234with 1234 being a testrun ID that is available via the environment variable $TAPPER_TESTRUN. This variable is provided by the automation layer.
If the grouping id is not a testrun id, e.g., because you have set up a Xen environment without the Tapper automation layer, then generate one random value once in dom0 by yourself and use that same value inside all guests with the following header:
get the value:
TAPPER_REPORT_GROUP=`date|md5sum|awk '{print $1}'`use the value:
# Tapper-reportgroup-arbitrary: $TAPPER_REPORT_GROUP
How that value gets from dom0 into the guests is left as an exercise, e.g. via preparing the init scripts in the guest images before starting them. That's not the problem of the test suite wrappers, they should only evaluate the environment variable TAPPER_REPORT_GROUP.
Some TAP emitting toolchains allow the generation of .tar.gz files containing TAP, so called TAP archives. E.g., via prove:
$ prove -a /tmp/myresults.tgz t/You can later submit such TAP archive files to the Tapper reports receiver tha same way as you report raw TAP.
The Tapper reports receiver is a daemon that listens on a port and slurps in everything between the open and close of a connection to it. Therefore you can use netcat to report TAP.
Remember that using netcat in turn can be a mess, the are several flavours with different options which are also changing their behaviour over time. So to be sure, you better do your own socket communication with Perl or Python: open socket, print to socket, close socket, done. We just keep with netcat for illustrating the examples.
Simply submit all TAP directly into the socket of the reports receiver:
$ ./my_tap_emitting_test_suite | netcat tapper_server 7357You submit the content of a .tar.gz file in the same way you submit raw TAP, via the same API. The receiver recognizes the .tar.gz contenttype by itself.
$ prove -a /tmp/myresults.tgz t/
$ cat /tmp/myresults.tgz | netcat tapper_server 7357