Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Various test improvement, especially for testing minimal templates #639

Open
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

marmarek
Copy link
Member

No description provided.

This allows more effective skipping tests for specific templates -
without doing any setUp/tearDown.
... using another template for management dvm. That other template can
be selected with QUBES_TEST_MGMT_TPL variable.

For this to fully work, test updating a template instead of AppVM, as it
will use updates proxy which should work with minimal template (contrary
to direct network access).
This makes sense for testing if in-vm kernel works out of the box in
selected templates.

QubesOS/qubes-issues#5212
Plain minimial template wont work, but this allows testing if minimal +
just networking packages installed. Specifically, this tests if
installing qubes-core-agent-networking is enough to get networking
working with minimal template.
Copy link

codecov bot commented Nov 30, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 69.44%. Comparing base (039b7d3) to head (469e147).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #639      +/-   ##
==========================================
- Coverage   69.49%   69.44%   -0.06%     
==========================================
  Files          58       58              
  Lines       12465    12434      -31     
==========================================
- Hits         8663     8635      -28     
+ Misses       3802     3799       -3     
Flag Coverage Δ
unittests 69.44% <ø> (-0.06%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@qubesos-bot
Copy link

qubesos-bot commented Nov 30, 2024

OpenQA test summary

Complete test suite and dependencies: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024113016-4.3&flavor=templates

Test run included the following:

New failures, excluding unstable

Compared to: https://openqa.qubes-os.org/tests/overview?distri=qubesos&version=4.3&build=2024111705-4.3&flavor=update

  • system_tests_pvgrub_salt_storage

    • TC_41_HVMGrub_fedora-41-minimal: test_000_standalone_vm (failure)
      AssertionError: Failed command: dnf clean expire-cache && dnf insta...

    • TC_41_HVMGrub_fedora-41-minimal: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_42_PVHGrub_fedora-41-minimal: test_000_standalone_vm (failure)
      AssertionError: Failed command: dnf clean expire-cache && dnf insta...

  • system_tests_network_ipv6

    • VmIPv6Networking_fedora-41-minimal: test_000_simple_networking (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_010_simple_proxyvm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_020_simple_proxyvm_nm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_030_firewallvm_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_031_firewall_dynamic_block (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_040_inter_vm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_050_spoof_ip (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_100_late_xldevd_startup (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_110_dynamic_attach (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_111_dynamic_detach_attach (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_112_reattach_after_provider_shutdown (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_113_reattach_after_provider_kill (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_114_reattach_after_provider_crash (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_200_fake_ip_simple (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_201_fake_ip_without_gw (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_202_fake_ip_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_203_fake_ip_inter_vm_allow (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_204_fake_ip_proxy (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_210_custom_ip_simple (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_211_custom_ip_proxy (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_212_custom_ip_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_500_ipv6_simple_networking (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_510_ipv6_simple_proxyvm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_520_ipv6_simple_proxyvm_nm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_530_ipv6_firewallvm_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_540_ipv6_inter_vm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_550_ipv6_spoof_ip (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_710_ipv6_custom_ip_simple (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_711_ipv6_custom_ip_proxy (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_712_ipv6_custom_ip_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

  • system_tests_network_updates

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_000_update (failure)
      AssertionError: qubes-dom0-update failed: Using test-inst-updatevm ...

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_001_update_check (failure)
      ^... AssertionError: '' is not true

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_005_update_flag_clear (failure)
      AssertionError: qubes-dom0-update failed: Using test-inst-updatevm ...

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_006_update_flag_clear (failure)
      AssertionError: qubes-dom0-update failed: Using test-inst-updatevm ...

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_010_instal (failure)
      AssertionError: qubes-dom0-update failed: Using test-inst-updatevm ...

    • TC_10_QvmTemplate_fedora-41-minimal: test_000_template_list (failure)
      AssertionError: qvm-template failed: [Qrexec] 2024-11-30 13:30:08.9...

    • TC_10_QvmTemplate_fedora-41-minimal: test_010_template_install (failure)
      AssertionError: qvm-template failed: [Qrexec]

    • TC_11_QvmTemplateMgmtVM_fedora-41-minimal: test_000_template_list (failure)
      AssertionError: qvm-template failed: ERROR: [Errno 2] No such file ...

    • TC_11_QvmTemplateMgmtVM_fedora-41-minimal: test_010_template_install (failure)
      AssertionError: qvm-template failed: ERROR: [Errno 2] No such file ...

    • VmUpdates_fedora-41-minimal: test_000_simple_update (failure)
      AssertionError: 1 not found in [0, 100] : dnf clean all; dnf check-...

  • system_tests_extra

    • system_tests_prepare_minimal: Failed (test died)
      # Test died: Can't "continue" outside a when block at qubesos/tests...
  • system_tests_network

    • system_tests_prepare_minimal: Failed (test died)
      # Test died: Can't "continue" outside a when block at qubesos/tests...

Failed tests

45 failures
  • system_tests_pvgrub_salt_storage

    • TC_41_HVMGrub_fedora-41-minimal: test_000_standalone_vm (failure)
      AssertionError: Failed command: dnf clean expire-cache && dnf insta...

    • TC_41_HVMGrub_fedora-41-minimal: test_010_template_based_vm (error)
      qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...

    • TC_42_PVHGrub_fedora-41-minimal: test_000_standalone_vm (failure)
      AssertionError: Failed command: dnf clean expire-cache && dnf insta...

  • system_tests_network_ipv6

    • VmIPv6Networking_fedora-41-minimal: test_000_simple_networking (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_010_simple_proxyvm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_020_simple_proxyvm_nm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_030_firewallvm_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_031_firewall_dynamic_block (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_040_inter_vm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_050_spoof_ip (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_100_late_xldevd_startup (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_110_dynamic_attach (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_111_dynamic_detach_attach (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_112_reattach_after_provider_shutdown (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_113_reattach_after_provider_kill (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_114_reattach_after_provider_crash (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_200_fake_ip_simple (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_201_fake_ip_without_gw (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_202_fake_ip_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_203_fake_ip_inter_vm_allow (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_204_fake_ip_proxy (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_210_custom_ip_simple (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_211_custom_ip_proxy (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_212_custom_ip_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_500_ipv6_simple_networking (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_510_ipv6_simple_proxyvm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_520_ipv6_simple_proxyvm_nm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_530_ipv6_firewallvm_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_540_ipv6_inter_vm (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_550_ipv6_spoof_ip (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_710_ipv6_custom_ip_simple (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_711_ipv6_custom_ip_proxy (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

    • VmIPv6Networking_fedora-41-minimal: test_712_ipv6_custom_ip_firewall (failure)
      AssertionError: Command 'ip link add test0 type dummy' failed: -bas...

  • system_tests_network_updates

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_000_update (failure)
      AssertionError: qubes-dom0-update failed: Using test-inst-updatevm ...

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_001_update_check (failure)
      ^... AssertionError: '' is not true

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_005_update_flag_clear (failure)
      AssertionError: qubes-dom0-update failed: Using test-inst-updatevm ...

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_006_update_flag_clear (failure)
      AssertionError: qubes-dom0-update failed: Using test-inst-updatevm ...

    • TC_00_Dom0Upgrade_fedora-41-minimal: test_010_instal (failure)
      AssertionError: qubes-dom0-update failed: Using test-inst-updatevm ...

    • TC_10_QvmTemplate_fedora-41-minimal: test_000_template_list (failure)
      AssertionError: qvm-template failed: [Qrexec] 2024-11-30 13:30:08.9...

    • TC_10_QvmTemplate_fedora-41-minimal: test_010_template_install (failure)
      AssertionError: qvm-template failed: [Qrexec]

    • TC_11_QvmTemplateMgmtVM_fedora-41-minimal: test_000_template_list (failure)
      AssertionError: qvm-template failed: ERROR: [Errno 2] No such file ...

    • TC_11_QvmTemplateMgmtVM_fedora-41-minimal: test_010_template_install (failure)
      AssertionError: qvm-template failed: ERROR: [Errno 2] No such file ...

    • VmUpdates_fedora-41-minimal: test_000_simple_update (failure)
      AssertionError: 1 not found in [0, 100] : dnf clean all; dnf check-...

  • system_tests_extra

    • system_tests_prepare_minimal: Failed (test died)
      # Test died: Can't "continue" outside a when block at qubesos/tests...
  • system_tests_network

    • system_tests_prepare_minimal: Failed (test died)
      # Test died: Can't "continue" outside a when block at qubesos/tests...

Fixed failures

Compared to: https://openqa.qubes-os.org/tests/119126#dependencies

1 fixed
  • system_tests_extra
    • TC_00_QVCTest_whonix-gateway-17: test_010_screenshare (failure)
      ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^... AssertionError: 0 == 0

Unstable tests

  • system_tests_pvgrub_salt_storage

    TC_41_HVMGrub_debian-12-xfce/test_000_standalone_vm (1/5 times with errors)
    • job 115648 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_41_HVMGrub_fedora-40-xfce/test_000_standalone_vm (1/5 times with errors)
    • job 115648 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_41_HVMGrub_debian-12-xfce/test_010_template_based_vm (1/5 times with errors)
    • job 115648 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
    TC_41_HVMGrub_fedora-40-xfce/test_010_template_based_vm (1/5 times with errors)
    • job 115648 qubes.exc.QubesVMError: Cannot connect to qrexec agent for 120 seco...
  • system_tests_network_updates

    VmUpdates_debian-12-xfce/test_020_updates_available_notification (1/5 times with errors)
    • job 117610 subprocess.CalledProcessError: Command '/usr/lib/qubes/upgrades-sta...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants