This repo contains a set of integration tests for NetworkManager and CentOS 8 Stream based VM test instructions

Code Branch	Build Status
main
1.48.x
1.46.x

• Prerequisites

  • CentOS qcow2 image, running libvirtd

    dnf -y install virt-install /usr/bin/virt-sysprep virt-viewer libvirt
    systemctl start libvirtd
    
    SSH_KEY=/home/vbenes/.ssh/id_rsa.pub
    IMG=CentOS-Stream-GenericCloud-8-20210603.0.x86_64.qcow2
    wget https://cloud.centos.org/centos/8-stream/x86_64/images/$IMG
    
    virt-sysprep -root-password password:centos -a $IMG
    
    # We need sudo to access default bridged networking
    sudo virt-install --name CentOS_8_Stream --memory 4096 --vcpus 4 --disk $IMG,bus=sata --import --os-variant centos-stream8 --network default
    
    # Virsh doesn't sort machines so you may need to use different VM
    # than the first. Alternatives are name resolution or port forwarding.
    IP=$(sudo virsh net-dhcp-leases default | grep ipv4 | awk '{print $5}' |head -1 | awk -F '/' '{print $1}')
    ssh-copy-id root@$IP

    Name resolution of or port forwarding to a given VM are described in Networking tips for local VMs section below.

• Running Tests

  • with NM compilation

    # NMCI test code. NMCI main should work everywhere
    TEST_BRANCH='main'
    # REFSPEC of your NM code change, work with repo below
    REFSPEC='main'
    # Change to whatever repo you want to compile NM from
    NM_REPO='https://gitlab.freedesktop.org/vbenes/NetworkManager'
    # Choose a list of features you want to test, you can have 'all' to test everything
    FEATURES='adsl, bond'
    
    INSTALL1="dnf install -y git python3 wget"
    INSTALL2="python3 -m pip install python-gitlab pyyaml"
    NMCI_URL="https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci.git"
    CLONE="git clone $NMCI_URL; cd NetworkManager-ci; git checkout  $TEST_BRANCH"
    TEST="cd NetworkManager-ci; python3 run/centos-ci/node_runner.py -t $TEST_BRANCH -c $REFSPEC -f \"$FEATURES\" -r $NM_REPO"
    
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $INSTALL1 && \
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $INSTALL2 && \
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $CLONE && \
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $TEST

  • you can avoid compilation and use already installed packages

    # NMCI test code. NMCI main should work everywhere
    TEST_BRANCH='vb/nmtest'
    # Choose a list of features you want to test, you can have 'all' to test everything
    FEATURES='all'
    
    INSTALL1="dnf install -y git python3 wget"
    INSTALL2="python3 -m pip install python-gitlab pyyaml"
    NMCI_URL="https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci.git"
    CLONE="git clone $NMCI_URL;cd NetworkManager-ci; git checkout  $TEST_BRANCH"
    TEST="cd NetworkManager-ci; python3 run/centos-ci/node_runner.py -t $TEST_BRANCH -f \"$FEATURES\" -D"
    
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $INSTALL1 && \
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $INSTALL2 && \
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $CLONE && \
    ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@$IP $TEST

  • or you can just ssh into the machine and run

    cd /root/NetworkManager-ci
    # Test by test as defined in mapper.txt
    run/runtest.sh your_desired_test
    # Feature by feature as listed too in mapper.txt
    run/runfeature.sh your_desired_feature

• Results check

  • you will see execution progress as it goes ( tests do have 10m timeout to prevent lockup )
  • there is a summary at the end

• We use slightly modified python-behave framework to execute tests

  • https://behave.readthedocs.io/en/stable/

• It's quite readable and easy to learn

• Let's describe directory structure and files of NMCI first

• /

  • mapper.yaml file
    • use for driving tests
    • all tests names are written there (together with features)
    • all dependencies and basically all metadata is there
  • nmci dir
    • various scripts used for driving tests
    • the most interesting are tags that are used for preparing and cleaning environment
    • we have unit tests for version_control here, libs for tags, run for running commands, etc
  • nmci/helpers/version_control.py script
    • we have just one NMCI branch for all NM versions, RHELs, Fedora, CentOSes
    • this is the control mechanism if we need to skip test here and there

• features dir

  • scenarios dir
    • sets of features in .feature files
    • test itself will be described deeper later on
  • environment.py script
    • script driving setup and teardown of each test
    • includes nmci/tags.py to be more readable
    • collects all logs and creates html log
  • steps dir
    • all steps (aka test lines) are defined here
    • strictly python, with paramterized decorators used in features itself
    • categorized into various functional areas like bond/bridge/team/connection/commands, etc

• prepare dir

  • various scripts for preparing environment
  • vethsetup.sh
    • this one creates whole 11 device wide test bed
    • executed at first test execution if needed
    • some CI machines have equivalent environment set up by physical switches or virt/cloud VM settings. If you need to tweak the environment for your scenario, use Prepare simulated test... steps instead, please.
  • envsetup.sh
    • installs various packages and tunes environment
    • executed at first test execution

• run

  • various executors for various envs
  • runtest.sh
    • the main driver of tests
    • execution of test looks like: run/runtest.sh test_name
    • to embed everything to HTML use: NMCI_DEBUG=yes run/runtest.sh test_name
  • runfeature.sh
    • doing the same as runtest.sh but for whole features
    • run/runfeature.sh bond for example
    • run/runfeature.sh bond --dry just lists tests

• contrib

  • various files and reproducers needed for tests

• Let's use an example test to describe how we do that *

  # Reference to bugzilla, doing nothing
  @rhbz1915457
  # Version control, test will run on F32+ or RHEL8.4+
  # with NM of 1.30+ only. Test will be skipped in CentOS
  @ver+=1.30 @rhelver+=8.4 @fedver+=32 @skip_in_centos
  # Bond and slaves residuals will be deleted after test
  # Test name as stated in mapper.txt. Must be last tag and sole tag on the line
  @bond_8023ad_with_vlan_srcmac
  # Human readable test name as stated in HTML report
  Scenario: nmcli - bond - options - mode set to 802.3ad with vlan+srcmax
  # Step for creation a NM profile with options
  * Add "bond" connection named "bond0" for device "nm-bond" and options
                                  """
                                  bond.options 'mode=802.3ad,
                                  miimon=100,xmit_hash_policy=vlan+srcmac'
                                  """
  # Step for creation slave connection, similar to above can be used too, we have two
  * Add slave connection for master "nm-bond" on device "eth1" named "bond0.0"
  # Bring up the connection, you can use down too
  * Bring "up" connection "bond0.0"
  # You can execute various commands too
  * Execute "echo 'Hello world'"
  # Check various value via period of time (once a second). Then keyword is useles, just marking results more visible
  Then "Bonding Mode: IEEE 802.3ad Dynamic link aggregation" is visible with command "cat /proc/net/bonding/nm-bond" in "5" seconds
  # Check various value twice (default)
  Then "Transmit Hash Policy:\s+vlan\+srcmac" is visible with command "cat /proc/net/bonding/nm-bond"
  # Check various value immediately
  Then "Transmit Hash Policy:\s+vlan\+srcmac" is visible with command "cat /proc/net/bonding/nm-bond" in "1" seconds
  # Very bond specific step checking if bond device is up.
  Then Check bond "nm-bond" link state is "up"

• Reports

  • You can see stdout output when running from command line
  • You can see nice HTML reports too stored in /tmp
    • PASS report has just after cleanup info in it
      • https://vbenes.fedorapeople.org/NM/PASS_bond_8023ad_with_vlan_srcmac.html
    • FAIL report has all commands listed and logs added from before, after, after cleanup
      • https://vbenes.fedorapeople.org/NM/FAIL_ipv6_survive_external_link_restart.html

1. Run your new or updated scenario by hand:

   run/runtest.sh my_new_scenario
   

   as already mentioned. The run/runtest.sh script comes with some handy features:
   • bash completion is available so you can type just run/runtest.sh my_new<tab> and it will complete the command to run/runtest.sh my_new_scenario
   • also, the run/runtest.sh script is forgiving of @-sign at the start of test_tag for mouse-friendly tag pasting
   • when you need to pretend that NM version is different from one actually installed, nmci will honor version set in environment variable such as NM_VERSION=1.39.1
   • run/runtest.sh produces terse report for PASSing scenarios by default to save space required by CI. To get full output always, set (as already mentioned) NMCI_DEBUG environment variable to yes
2. Unit tests
   • you can run them yourself before submitting or updating a MR:

     NetworkManager-ci$ python3 -m pytest nmci/test_nmci.py
     

3. or you can create a Merge Request in Freedesktop Gitlab right away and CI tooling will run both of these for you (changed scenarios only if unit tests pass successfully).

Tests have tags (with an @). The @ver tag checks the version of NetworkManager to run or skip the test. See also nmci/helpers/version_control.py script. Use following best practice for choosing a version tag.

• prefer @ver+=NUMBER over @ver+NUMBER because it is nicer to state the version when the feature starts working, and not the version before.

• similarly, prefer @ver-NUMBER over @ver-=NUMBER to mirror a @ver+=NUMBER. We can write two variants of a test with different version tags, and the version selection must only match for one variants of the test. Hence there should always be separate ranges. The use of @ver-NUMBER to mirror @ver+=NUMBER makes that clear.

• in NetworkManager, stable releases have an even second version number and an even third number (e.g. 1.42.2). A patch/feature/bugfix can only be added in a development version that leads up to a stable release. In the example, between the tag 1.42.1 and 1.42.2. The right way for a @ver+= tag is thus always the development version, like @ver+=1.42.1, because the patch is already upstream in nm-1-42 branch, which is currently between 1.42.1 and 1.42.2. The test should start working with 1.42.1+ already. Yes, early versions of 1.42.1+ don't have the patch yet, but it's more important to test the later versions of the development branch and run the test. The right @ver+=NUMBER is what git describe gives for the commit with the patch.

  • on main branch, the second number is odd and the third number can be odd or even. For example, 1.43.6 is a development version leading towards the next major release 1.44.0. Also there, a patch is always introduced between two development snapshots, like between 1.43.5 and 1.43.6. Here too, we shall use @ver+=1.43.5, so that testing current main branch (when 1.43.6 is not yet tagged) also covers the test. Note that we may take a devel snapshot (1.43.5) and package in RHEL. But that version doesn't have the patch yet, so for that RHEL package the test @ver+=1.43.5 will run but fail. There are are three possibilities. First, don't care. These are all just development versions and the situation resolves itself in a few days. Second, choose @ver+=1.43.5.2000, which would match for upstream builds but not for RHEL (note the package version from upstream builds adds a large 4th number). Third, add a RHEL specific override like @ver+=1.43.5 @ver/rhel+=1.43.6.

The tested NetworkManager version is parsed with a "stream" along the version number. The stream is a variant of the NetworkManager package or a specific build configuration. For example, we can have upstream release 1.44.2, which we can build in copr, as a Fedora package or as a CentOS Stream 9 package. The build configurations may slightly differ for the same tarball. For example, a RHEL 8.7 package will be detected to have "rhel/8/7" stream. For most cases, downstream RHEL/Fedora is very close to upstream so there is no difference between streams. However, when having a stream like rhel/8/7, then the runner will first search for version tags @ver/rhel/8/7, then @ver/rhel/8, then @ver/rhel and finally @ver. If any stream specific version tag is found, it is evaluated and more general tags (like @ver) are ignored. For example, @ver+=1.43.5 @ver/rhel/9+=1.43.6 means that RHEL 9 packages require a version 1.43.6 or newer, but all other packages require at least version 1.43.5.

Another possibility how to test the changes is to open a merge request in Gitlab. Pipeline first executes UnitTests (checks that the tests are consistent, well defined). Independently, remote jenkins triggers executes the tests, when a maintaner reviews and approves the code. The following apply for RHEL and CentOS trigger:

1. The tests can be retriggered by sending rebuild message to merge request discussion.

2. Only tests in changed .feature files are executed (whole features), if not overridden.

   • To override, use @RunTests:test1,test2,... in rebuild message, or in commit message or in merge request description.
   • @RunTests:* forces to execute all tests.
   • @RunFeatures:feature1,feature2,... override, which executes only specified features. It can be in either in the latest commit message, in the merge request description or in the rebuild comment.

3. Latest NetworkManager main copr build (CentOS) or stock NetworkManager RPM (internal) are tested.

   • To test on specific NetworkManager build, use @Build:main or @Build:0e5a4638807dc34c517988432120e3a5, in rebuild message, or in commit message or in merge request description.
   • In CentOS, if specified @Build branch is found in COPR, COPR build will be used instead of build.

4. By default, builds on CentOS 9 stream and RHEL8.X (latest release) are tested. These OSs can be overriden:

   • by rebuild OS_STRING GitLab comment. A build is going to be started just for the OS specified by the OS_STRING
   • by @os:OS_STRING lines in the latest commit message or in the MR description. Push events or rebuild comment in gitlab will trigger builds on these OSs only.

   The format of OS_STRING for respective OSs is:

   • for Centos X Stream: centosX-stream or short: cXs. So for 9 stream, it's centos9-stream or c9s
   • for RHEL: rhelX.Y, so for RHEL 9.3, the string is rhel9.3
   • for Fedora: Fedora-release, thus Fedora-39 or Fedora-Rawhide. Just rawhide is also recognized.

   All the strings mentioned are case insensitive so @oS:fEDoRa-rAWhIdE will still get recognized.

5. The priority of overrides is rebuild message, commit message, merge request description.

6. The tests can be skipped either by pushing with git push -o ci.skip, or "Rebase without pipeline button" in WebUI.

7. If you interlink merge requests (mention counterpart merge request in description), corresponding branch will be used for testing/build:

   • In a NetworkManager merge request description mention NetworkManager-ci!ABC or https://gitlab.freedesktop.org/NetworkManager/NetworkManager-ci/-/merge_requests/ABC and it will use test from merge request numbered ABC

   • In a NetworkManager-ci merge request description mention NetworkManager!XYZ or https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/XYZ and it will build NetworkManager from merge request numbered XYZ

   • Example: NetworkManager!1536 and NetworkManager-ci!1317

8. New build can be triggered by:

   • commenting to open merge request:
     • rebuild message will re-trigger build with no additional overrides,
     • message containing overrides starting with @ will automatically execute new builds,
   • pushing to open merge request - overrides defined in the commit message and merge-request description will be honored.

In CentOS, older builds running on the same CentOS release from the same merge request are stopped to save resources. So, if you push to the merge request before the tests are finished, you may get "Aborted" message in the merge request discussion.

1. When creating merge request, add overrides to description, one override per line:

   This is example Merge Request
   ...
   
   @Build:main
   @RunTests:my_new_regression_test
   @OS:rhel8.5
   

2. Edit merge request description and add rebuild comment to the discussion, which will force the new run of the tests with overrides. The overrides will apply for further pushes to the merge request.

3. For one-time overrides, you can either do rebuild comment in merge request discussion:

   @Build:nm-1-38
   @RunTests:my_test1,my_test2
   
   rebuild c8s  # this line can be ommited, works only in CentOS, rhel will be executed too.
   

   or specify overrides in the last commit before pushing to the merge request:

   This adds feature XYZ to the testsuite
   
   # this feature is not yet merged in NetworkManager
   @Build:my_supporting_branch_in_NM_repository
   # We want to execute all tests, not only changed files
   @RunTests:*
   

When testing at different machine than that running your browser, it is handy to have HTTP server on the test machine and some means to auto-archive your test report from /tmp somewhere else. NetworkManager-ci come with script that will configure httpd on your machine to serve the copied reports on port 8080 and a little system service that watches /tmp. When new report is created, the service copies it with a timestamp to httpd's DocumentRoot so you can compare multiple runs of the same scenario.

Service can be set up by script run/publish_behave_logs/setup_pbl.py or manually by copying service and its file to appropriate location and configuring web server as you need. CI machines install these by default so you can access their test reports this way when the test run is still underway and CI didn't collect them yet.

Default libvirt networking works just OK but leaves a bit of convenience or features for regular use or when one has multiple VMs. Following sections feature tips to make local VM network more convenient or more accessible from outside.

For local libvirt VMs with floating bridges or NAT'd networks, there are several good ways how to access VMs by name:

• libvirt-nss which when set up according to their docs allows you to access VMs by their ‘libvirt domain’ name (libvirt_guest nsswitch module) or by guest's hostname (libvirt nsswitch module). Prerequisite for that is however that libvirt gets IP information from within the guest system via agent, you can verify if this is the case that you can see the IP when you run:

  virsh domifaddr DOMAIN_NAME
  

  names exposed this way are always without a domain
• by hooking up libvirt-run dnsmasq that works as DHCP and DNS server for type="nat" libvirt networks to the system resolver. This is achiveved by:

  1. setting <domain name="DOMAIN"/> and <forward mode='nat'/> for the network

  2. pointing system DNS resolver to libvirt-run dnsmasq for this network. This is pretty easy to configure statically for dnsmasq or unbound however if you run systemd-resolved, it's only possible to configure specific DNS server for a given domain by changing runtime configuration.

     It's possible to automate this using libvirt hooks mechanism and systemd's commands resolvectl or systemd-resolve but neither project ships necessary glue script itself. Therefore I (@djasa) created this script that is fired by libvirt network events and when relevant, updates systemd-resolved about it. This configuration allows me to use gnome-boxes VMs with bridge/NAT network managed by system-level libvirt (or better say virtnetworkd these days) and access VMs by their hostname suffixed by custom domain, e.g. http://rhel9-nightly.vm:8080/ or ssh [root@]rhel9-nightly.vm

NAT or isolated network however don't allow easy access from outside network to VM's ssh or http ports.

There is also option to forward given port on host system to another port in the VM when using qemu's user mode networking and it's hostdev option. This is not integrated to libvirt, so in order to use this option, you need to add qemu namespace URI to libvit domain xml:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>nmci-vm</name>
  …

that allows specifying custom arguments (env vars and others, see libvirt docs for more information). Options that create user-mode network interface for the guest with port forwarding from the host can be:

  …
  <qemu:commandline>
    <qemu:arg value='-netdev'/>
    <qemu:arg value='user,id=mynet.9,net=192.168.120.0/24,hostfwd=tcp::4922-:22,hostfwd=::4980-:8080'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='virtio-net-pci,netdev=mynet.9'/>
  </qemu:commandline>
</domain>

Notes:

• net=192.168.120.0/24 specifies network range used by qemu. You may want to choose range that won't collide with private networks you need to reach from the guest. Guest by default gets .15 address of the range (can be changed by dhcpstart= according to Qemu docs), to this addres (192.168.120.15 for this configuration, 10.0.2.15 with no network= specification) qemu redirects host ports configured by hostfwd option unles overriden there
• hostfwd=::4922-:22 specifies host forwarding itself:
  • empty source address means v4 wildcard (0.0.0.0). If you wonder why qemu doesn't support dual-stack wildcard :: in 2022, you're not alone.
  • empty guest address (between - and :) means .15 address of range specified by net= or address specified by dhcpstart=
• Other arguments are well described in qemu manual page or online docs.

Please note that NetworkManager-ci expects just one interface with outside connectivity so this is either-or situation, there is either bridged networking or qemu-provided port forwarding. Having port forwarding with bridge networking would need port forwarding configured on system level (doable using already mentioned libvirt hooks mechanism but not used by anyone in the team right now. Feel free to contribute a working solution if you have one).

nmci is primarily developed and maintained on RHEL/CentOS/Fedora systems so there are distro-specific parts of the code. They are primarily in

• envsetup. Distro-specific parts are already largely or completely taken out to specific files in prepare/envsetup directory. Adding more of them for other distros should be straightforward
• other scripts in prepare and some tag implementations in tags.py (and few cases in prepare.py) use hardwired rpm/yum/dnf invocations. These would need to be generalized or where suitable, package instalation moved to envsetup
• version detection code in nmci/misc.py. Analogous may be needed for finer-grained decisions of what scenarios should run on given system
• CI runners in in run
• setup_pbl.py. You may try ansible version setup_pbl.yml which at least abstracts away package installation.