Warning
For very long, we recommended switching your CI jobs to Zuulv3 native jobs. In the Xena cycle, we are officially deprecating it. Devstack Gate will support only stable branches until stable/wallaby. From the Xena release onwards, we no longer guarantee it to work. If it fails, we strongly recommend to switch your CI jobs to Zuulv3 native or fork this repo to fix it for your CI.
Devstack-gate is a collection of scripts used by the OpenStack CI team to test every change to core OpenStack projects by deploying OpenStack via devstack on a cloud server.
All changes to core OpenStack projects are "gated" on a set of tests so that it will not be merged into the main repository unless it passes all of the configured tests. Most projects require unit tests with pep8 and several versions of Python. Those tests are all run only on the project in question. The devstack gate test, however, is an integration test and ensures that a proposed change still enables several of the projects to work together.
Obviously we test integrated OpenStack components and their clients because they all work closely together to form an OpenStack system. Changes to devstack itself are also required to pass this test so that we can be assured that devstack is always able to produce a system capable of testing the next change to nova. The devstack gate scripts themselves are included for the same reason.
The devstack test starts with an essentially bare virtual machine, installs devstack on it, and runs tests of the resulting OpenStack installation. In order to ensure that each test run is independent, the virtual machine is discarded at the end of the run, and a new machine is used for the next run. In order to keep the actual test run as short and reliable as possible, the virtual machines are prepared ahead of time and kept in a pool ready for immediate use. The process of preparing the machines ahead of time reduces network traffic and external dependencies during the run.
The Nodepool project is used to maintain this pool of machines.
When Jenkins runs gate tests for a change, it leaves comments on the change in Gerrit with a link to the resulting logs, including the console log. If a change fails in a devstack-gate test, you can follow these links to find out what went wrong. Start at the bottom of the log file with the failure, scroll up to look for errors related to failed tests.
You might need some information about the specific run of the test. In the devstack-gate-setup-workspace log, you can see all the git commands used to set up the repositories, and they will output the (short) sha1 and commit subjects of the head of each repository.
It's possible that a failure could be a false negative related to a specific provider, especially if there is a pattern of failures from tests that run on nodes from that provider. In order to find out which provider supplied the node the test ran on, look at the name of the jenkins slave in the devstack-gate-setup-host log, the name of the provider is included.
Below that, you'll find the output from devstack as it installs all of the debian and pip packages required for the test, and then configures and runs the services. Most of what it needs should already be cached on the test host, but if the change to be tested includes a dependency change, or there has been such a change since the snapshot image was created, the updated dependency will be downloaded from the Internet, which could cause a false negative if that fails. Assuming that there are no visible failures in the console log, you may need to examine the log output from the OpenStack services, located in the logs/ directory. All of the OpenStack services are configured to syslog, so you may find helpful log messages by clicking on the "syslog.txt[.gz]" file. Some error messages are so basic they don't make it to syslog, such as if a service fails to start. Devstack starts all of the services in screen, and you can see the output captured by screen in files named "screen-*.txt". You may find a traceback there that isn't in syslog.
After examining the output from the test, if you believe the result was a false negative, you can retrigger the test by running a recheck, this is done by leaving a review comment with simply the text: recheck
If a test failure is a result of a race condition in the OpenStack code, you also have the opportunity to try to identify it, and file a bug report, help fix the problem or leverage elastic-recheck to help track the problem. If it seems to be related to a specific devstack gate node provider, we'd love it if you could help identify what the variable might be (whether in the devstack-gate scripts, devstack itself, Nodepool, OpenStack, or even the provider's service).
Developers often have a need to recreate gating integration tests manually, and this provides a walkthrough of making a DG-slave-like throwaway server without the overhead of building other CI infrastructure to manage a pool of them. This can be useful to reproduce and troubleshoot failures or tease out nondeterministic bugs.
First, you can build an image identical to the images running in the gate using diskimage-builder. The specific operating systems built and DIB elements for each image type are defined in nodepool.yaml. There is a handy script available in the project-config repo to build this for you:
git clone https://opendev.org/openstack/project-config cd project-config ./tools/build-image.sh
Take a look at the documentation within the build-image.sh script for specific build options.
These days Tempest testing is requiring in excess of 2GiB RAM (4 should be enough but we typically use 8) and completes within an hour on a 4-CPU virtual machine.
If you're using an OpenStack provider, it's usually helpful to set up a clouds.yaml file. More information on clouds.yaml files can be found in the `os-client-config documentation <https://docs.openstack.org/developer/os-client-config/#config-files`_. A clouds.yaml file for Rackspace would look something like:
clouds:
rackspace:
auth:
profile: rackspace
username: '<provider_username>'
password: '<provider_password>'
project_name: '<provider_project_name>'
Where provider_username and provider_password are the user / password for a valid user in your account, and provider_project_name is the project_name you want to use (sometimes called 'tenant name' on older clouds)
You can then use the openstack command line client (found in the python package python-openstackclient) to create a VM on the cloud.
You can tell openstack to use the DFW region of the rackspace cloud you defined either by setting environment variables:
export OS_CLOUD=rackspace export OS_REGION_NAME=DFW openstack servers list
or command line options:
openstack --os-cloud=rackspace --os-region-name=DFW servers list
It will be assumed in remaining examples that environment varialbes have been set.
If you haven't already, create an SSH keypair "my-keypair" (name it whatever you like):
openstack keypair create --public-key=$HOME/.ssh/id_rsa.pub my-keypair
Upload your image, boot a server named "testserver" (chosen arbitrarily for this example) with your SSH key allowed, and log into it:
FLAVOR='8GB Standard Instance' openstack image create --file devstack-gate.qcow2 devstack-gate openstack server create --wait --flavor "$FLAVOR" --image "devstack-gate" \ --key-name=my-keypair testserver openstack server ssh testserver
If you get a cryptic error like ERROR: 'public' then you may need to
manually look up the IP address with openstack server show testserver and
connect by running ssh root@<ip_address> instead. Once logged in, switch to
the jenkins user and set up parts of the environment expected by devstack-gate
testing:
su - jenkins export REPO_URL=https://git.openstack.org export ZUUL_URL=/home/jenkins/workspace-cache export ZUUL_REF=HEAD export WORKSPACE=/home/jenkins/workspace/testing mkdir -p $WORKSPACE
Specify the project and branch you want to test for integration:
export ZUUL_PROJECT=openstack/nova export ZUUL_BRANCH=master
Get a copy of the tested project. After these steps, apply relevant
patches on the target branch (via cherry-pick, rebase, et cetera) and
make sure HEAD is at the ref you want tested:
git clone $REPO_URL/$ZUUL_PROJECT $ZUUL_URL/$ZUUL_PROJECT \ && cd $ZUUL_URL/$ZUUL_PROJECT \ && git checkout remotes/origin/$ZUUL_BRANCH
Switch to the workspace and get a copy of devstack-gate:
cd $WORKSPACE \ && git clone --depth 1 $REPO_URL/openstack/devstack-gate
At this point you're ready to set the same environment variables and run the same commands/scripts as used in the desired job. The definitions for these are found in the openstack/project-config project under the jenkins/jobs directory in a file named devstack-gate.yaml. It will probably look something like:
export PYTHONUNBUFFERED=true export DEVSTACK_GATE_TEMPEST=1 export DEVSTACK_GATE_TEMPEST_FULL=1 cp devstack-gate/devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh ./safe-devstack-vm-gate-wrap.sh
If you're trying to figure out which devstack gate jobs run for a given project+branch combination, this is encoded in the openstack/project-config project under the zuul/ directory in a file named layout.yaml. You'll want to look in the "projects" section for a list of jobs run on a given project in the "gate" pipeline, and then consult the "jobs" section of the file to see if there are any overrides indicating which branches qualify for the job and whether or not its voting is disabled.
After the script completes, investigate any failures. Then log out and
openstack server delete testserver or similar to get rid of it once no
longer needed. It's possible to re-run certain jobs or specific tests on a used
VM (sometimes with a bit of manual clean-up in between runs), but for
proper testing you'll want to validate your fixes on a completely fresh
one.
Refer to the Jenkins Job Builder and Zuul documentation for more information on their configuration file formats.
All of the OpenStack developer infrastructure is freely available and managed in source code repositories just like the code of OpenStack itself. If you'd like to contribute, just clone and propose a patch to the relevant repository:
https://opendev.org/openstack/devstack-gate https://opendev.org/zuul/nodepool https://opendev.org/opendev/system-config https://opendev.org/openstack/project-config
You can file bugs on the storyboard devstack-gate project:
https://storyboard.openstack.org/#!/project/712
And you can chat with us on OFTC in #openstack-qa or #openstack-infra.
It's worth noting that, while devstack-gate is generally licensed under the Apache license, playbooks/plugins/callback/devstack.py is GPLv3 due to having derived from the Ansible source code.