-
Notifications
You must be signed in to change notification settings - Fork 213
Testing
The testing harness is based on using User Mode Linux . This is partially historic because back in the FreeS/WAN days its was the only virtualisation method. While there are some advantages using a modern virtualisation method, UML still provides key features that other methods do not:
- Using the hostfs file system allowing us to build images without root access
- hostfs further allows us to mount without requiring (nfs) network mounts (important because tests might cause network failure!). Test results are available directly on the host as well
- Being able to run as a userland application that could be launched in the cloud on many instances in parallel.
- Download our disk image LennyUML
- Gunzip it
- Boot it as a virtual machine (eg add via virt-mananger)
- login as user “build”, password “openswan”. The root user has the same password if you need it.
- Issue the following commands
cd openswan.git
git checkout master
git pull
make programs
make check
That’s it! You’re running the tests!
When using virt-manager, do not select “debian” or “rhel”, but a recent “fedora”, so the disk image is added using virtual IO, as the root image inside expects /dev/vda and not /dev/hda. To avoid having to hack the silly udev persistent network rules, you might want to add the ethernet with this specific mac address: 52:54:00:63:7f:eb
You can see the progress of the tests in the results/html directory defined in the umlsetup.sh file in the root of the openswan tree.
If you want, you can also test other git branches, or release tags. Examples:
git checkout rhel6-2.6.32-9 git checkout v2.6.38dr1
TODO: make the LennyUML host run an httpd with test results as DocumentRoot at http://testing.openswan.org/
The current status of the testing harness is that everything runs on modern systems, except for the uml_switch bug causing our tcpdump to fail to log the packets, resulting in false positives in all test cases. Despite the false positives, the tests are still very useful to run.
The output of various openswan core components has changed a lot, and many test cases do not yet reflect this, causing more false positives.
modernising the tests is a work in process, please join us at Freenode’s #openswan-dev
- Tests are run on a regular (currently debian/ubuntu) host OS
- Tests are run by a non-root user (“build” on the images provided by the openswan team)
- All test cases ship with openswan, you can find them in the “testing” subdirectory of each release. Or you can look at them in the git viewer
- The most important test cases reside in the pluto/ subdirectory
The easiest method at this point is to grab a bootable disk image from ftp://ftp.openswan.org/openswan/uml-testing/ (Paul: uploading my uml soon)
You can also create your own, but might encounter some undocumented secret sauce.
To build your own, you need a few components:
- A 32bit host (lenny is most tested at this moment) (64bit work in process, requires some changes to initrd building)
- An openswan source tree
- A root filesystem used by uml to create all the testing instances ( examples: ftp://ftp.openswan.org/openswan/uml-testing/umlrootfs/)
- A Linux kernel source tree (as most tests and their output are based on KLIPS, it needs to build a uml linux kernel and KLIPS module)
- The test configuration file umlsetup.sh (check our example or create your own)
- a LOT of free disk space on 1 filesystem (about 32GB to be safe). The uml guests use a lot of softlinks to reduce disk use, but that only works if you remain on the same filesystem
If all goes well, running “make check” will build all the uml guests and starts running all tests. You can also run individual tests using
Most tests consists of two uml’s. The test will boot the umls, and connect them to the predefined network (FIXME add link to netjig.jpg) using the uml utilities (uml_switch, uml_netjig). Then each uml is booted in single user mode, configured according to its role. It will mount the testing directory as /testing using hostfs. This is where the test results will be written to, and that is how the output end up outside the uml onto the host itself. Shell scripts are then ran which go through “expect” and are controlled within “tcl” for flow (this is sadly full of magic and secret sauce). This will cause the umls to execute a command at a time, and wait for proper output before the next command is run. This is used in the controlling scripts.
See the file “testparams.sh” for the test specific parameters. These include calling filter scripts, but is also responsible for firing up the right umls, as some tests need the KLIPS IPsec stack, and other tests need the NETKEY Psec stack. The script also directs the test which IKE daemon to start, as the interop tests can start either openswan, racoon2 or strongwan.
For each host, their test specific “init script” will then run. This will usually launch openswan, load any connections required, enable the various debug needed using “ipsec whack” commands. These can control the debug levels, but can also trigger test specific “impairments”, where openswan does something bad or wrong on purpose.
Then the test is ready to start, triggered by one end running its “run script”. All traffic between the hosts is logged with tcpdump to a file. At the end, information about state is collected using the “final.sh” script and the umls are then shut down.
tcpdump works quite different on KLIPS and NETKEY. With NETKEY, tcpdump cannot see outgoing encrypted packets on the machine, but since tcpdump runs on the host, and not on the uml guest, this is not an issue for the tests
After the test is run, the output of each uml host is stored in the tests’ OUTPUT/ directory, for example 26east-console.txt. These files are then processed by a bunch of filter scripts (as defined in testparams.sh) to filter out any instance specific output such as timestamps or randomness as result of keying material. The filtered output is then written, eg 26east-fixed-console.txt. These files are then compared with the known good output stored in the test directory, eg east-console.txt. If the output is different, the test is flagged as “failed”.
Often, code changes lead to different output. There are two places where this can be fixed. First, if this involves all test output, one could update or create a filter rule. If the code change only affects one or a few tests, the “fixed-console” files from the OUTPUT/ directory can be copied over the console.txt files to update the new reality. Extreme care should be taken with updating the console output files, because a mistake here will cause us to think the test passed while in fact it failed!
The most basic openswan to openswan test is basic-pluto-01 which can be found in openswan.git/testing/pluto/basic-pluto-01/ . You can look at those files to see the basic files required. Most tests require running only two uml’s for a test. For those type of tests, the “east” and “west” hosts are usually used.
fun fact: In the old days there were also hosts called ‘up’ and ‘down’, but Red Hat Linux 3.x did horrible things to ‘ifup’ if you named your host that way)
If you changed the source code of openswan, you should first recompile it using
make programs
If you setup the host yourself, using your umlsetup.sh, you must have run “make check” to create all the host’s filesystems and kernels and to compile and install the latest openswan in all the uml root filesystems. This will also fire off all test cases. Then you are ready to run an individual test case. The below example shows how this looks. The test case basic-pluto-01 is a good test case to test the testing harness. It is the most basic test case that runs two machines to establish and test a single IPsec connection.
This is how a failed test looks:
build@LennyUML:~/openswan.git$ cd testing/pluto/basic-pluto-01 build@LennyUML:~/openswan.git/testing/pluto/basic-pluto-01$ ../../utils/runme.sh ***** UML PLUTO RUNNING basic-pluto-01 ******* functions.sh:prerunsetup: klips: EAST_START="/vol/umlbuild/east/start.sh" EASTHOST="east" functions.sh:prerunsetup: klips: WEST_START=="/vol/umlbuild/west/start.sh" WESTHOST="west" EOF in killuml EOF in killuml reading from file OUTPUT/spi1-cleartext.pcap, link-type EN10MB (Ethernet) west side output differed 26eastConsole output differed 26westConsole output differed Recording "false: 1" to /vol/results/basic-pluto-01/status ******* FAILED basic-pluto-01 ********
Sometimes you want to run things manually, for instance when you want to attach gdb to pluto to get more information. Ensure you login to the host machine using “ssh -X” so that you can receive xterms. Then ssh in twice to get the console windows for “east” and “west”. The location of the uml roots depends on what you set in umlsetup.sh. The example below assumes the paths used in our provided LennyUML image.
ssh -X build@lennyumlhost build@LennyUML:$ cd /vol/umlbuild build@LennyUML:/vol/umlbuild$ ls beet east initrd.cpio Makefile netkey26 north pole sec sunset west carrot initrd japan module26 nic plain26 road sunrise swan26 build@LennyUML:/vol/umlbuild$ cd west build@LennyUML:/vol/umlbuild/west$ ls root startmodule.sh start-netkey.sh start.sh build@LennyUML:/vol/umlbuild/west$ ./start.sh
You will now get two xterms popping up for west. You should login as “root”, it should log you in without a password.
Do the same for east. Note that you should not run multiple copies of umls (eg two east instances) at once. That is, on the host, you can only run 1 test at a time. So give each developer their own virtual machine from which to fire of test cases. Using “start.sh” gives you a KLIPS kernel. Using “start-netkey.sh” gives you a NETKEY IPsec stack. You should use what the test case is using, or else the output will be massively different. To prepare east for a manual run of basic-pluto-01, you basically can run its scripts, or do part of its scripts manually, depending on your needs:
east:~# . /testing/pluto/basic-pluto-01/testparams.sh east:~# cat /testing/pluto/basic-pluto-01/eastinit.sh : ==== start ==== TESTNAME=basic-pluto-01 source /testing/pluto/bin/eastlocal.sh ipsec setup start ipsec auto --add westnet-eastnet ipsec whack --debug-control --debug-controlmore --debug-crypt /testing/pluto/bin/wait-until-pluto-started east:~# TESTNAME=basic-pluto-01 east:~# source /testing/pluto/bin/eastlocal.sh east:~# ipsec setup start [...]
For example, if you know the init is fine, you can just run . /testing/pluto/basic-pluto-01/eastinit.sh . One common mistake is to forget to set TESTNAME. After the inits have completed, you can step through the run script (eg
/testing/pluto/basic-pluto-01/westrun.sh) and you can run it and make it crash, and start gdb on it.
It is important that the host machine and the umlroot are based off the same compiler and development tools if you wish to run gdb on the core outside the umls on the host itself.