[Comp Eval] Updating fixture to SCA and Fixing failing tests #14167

ColeHiggins2 · 2024-02-23T16:03:42Z

Problem Statement

Many tests were failing in the UI due to several issues including outdated code, old non-sca fixtures, and more.

The associated issues were addressed and now passing
RHSAT6-45606: test_positive_generate_all_installed_packages_report
RHSAT6-44209: test_positive_gen_entitlements_reports_multiple_formats
RHSAT6-40833: test_positive_generate_registered_hosts_report

Solution

Update setup_content fixture to use SCA and enabled the custom repo on the Activation Key
docstring changes and manual org.disable for subscription-entitlements report
Rewrite outdated test using new rhel_content fixture

tests/foreman/ui/test_reporttemplates.py

lhellebr · 2024-02-26T15:06:32Z

tests/foreman/ui/test_reporttemplates.py

                      and it contains created host with correct data

    :CaseImportance: High
    """
-    # generate Host Status report
-    os_name = 'comma,' + gen_string('alpha')


The comma was there to test correct handling in CSV.

lhellebr · 2024-02-26T15:09:10Z

tests/foreman/ui/test_reporttemplates.py

-    with target_sat.ui_session() as session:
-        session.organization.select(module_org.name)
-        session.location.select(module_location.name)
-        # create multiple hosts to test filtering


So you removed testing filtering. It may be ok. Do you test it elsewhere? Is the change intended? You haven't mentioned it. This weakens the test.

It was failing so I just decided to use the new way of registering a host. This test is for report templates not host filtering, unless the report templates test filter, I dont think its necessary to keep? what do you think?

I also see some ui test for testing positive user access with host filtering, is that the same thing here?

omkarkhatavkar · 2024-02-29T07:35:25Z

trigger: test-robottelo
pytest: tests/foreman/ui/ -m e2e

Satellite-QE · 2024-02-29T11:37:07Z

PRT Result

Build Number: 5896
Build Status: UNSTABLE
PRT Comment: pytest tests/foreman/ui/ -m e2e --external-logging
Test Result : = 10 failed, 26 passed, 800 deselected, 1978 warnings, 5 errors in 13727.37s (3:48:47) =

vsedmik

A few proposals and questions posed

vsedmik · 2024-04-08T15:53:30Z

tests/foreman/ui/test_reporttemplates.py

-    )[0]
-    ak.add_subscriptions(data={'quantity': 1, 'subscription_id': subscription.id})
+    all_content = ak.product_content(data={'content_access_mode_all': '1'})['results']
+    content_label = content_label = [


Suggested change

content_label = content_label = [

content_label = [

vsedmik · 2024-04-08T15:55:29Z

tests/foreman/ui/test_reporttemplates.py

+    client.install_katello_ca(target_sat)
+    org, ak, _, _ = setup_content
+    client.register_contenthost(org.label, ak.name)


Use global registration

Suggested change

client.install_katello_ca(target_sat)

org, ak, _, _ = setup_content

client.register_contenthost(org.label, ak.name)

org, ak, _, _ = setup_content

client.register(org, None, ak.name, target_sat)

@ColeHiggins2 Based on the latest discussion in Automation Weekly BS call, this became rather a blocker than suggestion. Can you update the registering method?

vsedmik · 2024-04-08T16:10:10Z

tests/foreman/ui/test_reporttemplates.py

@@ -490,10 +481,12 @@ def test_negative_nonauthor_of_report_cant_download_it(session):


 @pytest.mark.tier3
+@pytest.mark.no_containers
 def test_positive_gen_entitlements_reports_multiple_formats(


Looking at org.sca_disable() bellow, is this test case really applicable to 6.16? Shouldn't it be raised against 6.15.z branch instead?

This change fixes the test in 6.15 and will be removed in 6.16. however, a lot of the other work in this PR still applies to 6.16. so I will just create a separate PR to remove this later in 6.16 as part of the RFE for SCA ONLY

vijaysawant

Ack
Can you please check this small change?

tests/foreman/ui/test_reporttemplates.py

sambible

1 change necessary, looks good otherwise. Can we get PRT as well

sambible · 2024-05-09T18:19:34Z

tests/foreman/ui/test_reporttemplates.py

+        query={'search': f'name="{DEFAULT_SUBSCRIPTION_NAME}"'}
+    )[0]
+    ak.add_subscriptions(data={'quantity': 1, 'subscription_id': subscription.id})
+    client.register_contenthost(org.label, None, ak.name, target_sat)


Need to change this to use global registration as well

vsedmik

One more suggestion from me. Sam's comment regarding GR and PRT run (at least for the first test since the second one would fail in stream) pending.

vsedmik · 2024-05-10T07:01:12Z

tests/foreman/ui/test_reporttemplates.py

+def test_positive_generate_registered_hosts_report(
+    session, target_sat, setup_content, rhel7_contenthost
+):


Let's get rid of rhelX_contenthost in favor of rhel_contenthost and parametrize what version we want to test with. Here I would say we could test with all supported versions (as suggested bellow), or pick one at least.

Suggested change

def test_positive_generate_registered_hosts_report(

session, target_sat, setup_content, rhel7_contenthost

):

@pytest.mark.rhel_ver_match(r'^(?!6$)\d+$')

def test_positive_generate_registered_hosts_report(

session, target_sat, module_setup_content, rhel_contenthost

):

I can see the setup_content fixture is module-scoped - good, we need it module-scoped so we don't create new and new content for every parametrized run, just consider renaming it to what it is - module_setup_content.

vijaysawant

Ack.. lgtm
Penidng few comments of other reviewers and PRT status, please take a look.

sambible · 2024-05-10T15:09:07Z

tests/foreman/ui/test_reporttemplates.py

 @pytest.mark.upgrade
 @pytest.mark.tier2
-def test_positive_generate_registered_hosts_report(target_sat, module_org, module_location):
+def test_positive_generate_registered_hosts_report(


lets run PRT on this at least, if that looks good I'm good with ACK

ColeHiggins2 · 2024-05-10T15:11:55Z

trigger: test-robottelo
pytest: tests/foreman/ui/test_reporttemplates.py -k test_positive_generate_registered_hosts_report

Satellite-QE · 2024-05-10T15:34:03Z

PRT Result

Build Number: 6882
Build Status: SUCCESS
PRT Comment: pytest tests/foreman/ui/test_reporttemplates.py -k test_positive_generate_registered_hosts_report --external-logging
Test Result : ========== 3 passed, 15 deselected, 61 warnings in 1014.84s (0:16:54) ==========

vsedmik

LGTM

* Fixing tests for reporting comp eval * updating to list comp * addressing comments * Addressing comments * addressing comments and rename setup content (cherry picked from commit 06e5529)

…teQE#14167) * Fixing tests for reporting comp eval * updating to list comp * addressing comments * Addressing comments * addressing comments and rename setup content

ColeHiggins2 added CherryPick PR needs CherryPick to previous branches AutoMerge_Cherry_Picked The cherrypicked PRs of master PR would be automerged if all checks passing 6.15.z Introduced in or relating directly to Satellite 6.15 labels Feb 23, 2024

ColeHiggins2 self-assigned this Feb 23, 2024

ColeHiggins2 requested a review from a team as a code owner February 23, 2024 16:03

Gauravtalreja1 reviewed Feb 25, 2024

View reviewed changes

tests/foreman/ui/test_reporttemplates.py Outdated Show resolved Hide resolved

lhellebr requested changes Feb 26, 2024

View reviewed changes

lhellebr reviewed Feb 26, 2024

View reviewed changes

Satellite-QE added the PRT-Failed Indicates that latest PRT run is failed for the PR label Feb 29, 2024

Satellite-QE removed the PRT-Failed Indicates that latest PRT run is failed for the PR label Apr 1, 2024

ColeHiggins2 requested a review from lhellebr April 5, 2024 13:10

vsedmik requested changes Apr 8, 2024

View reviewed changes

ColeHiggins2 force-pushed the reporting-comp-eval-test-fixes branch from 5443215 to 8624785 Compare April 30, 2024 17:14

ColeHiggins2 requested review from vsedmik and Gauravtalreja1 April 30, 2024 17:17

vijaysawant requested changes May 8, 2024

View reviewed changes

tests/foreman/ui/test_reporttemplates.py Outdated Show resolved Hide resolved

ColeHiggins2 force-pushed the reporting-comp-eval-test-fixes branch from 8624785 to dd3591e Compare May 8, 2024 18:11

ColeHiggins2 requested a review from vijaysawant May 8, 2024 18:12

sambible requested changes May 9, 2024

View reviewed changes

vsedmik requested changes May 10, 2024

View reviewed changes

vijaysawant approved these changes May 10, 2024

View reviewed changes

ColeHiggins2 added 5 commits May 10, 2024 11:04

Fixing tests for reporting comp eval

b6084f6

updating to list comp

4e22ccf

addressing comments

5d41d27

Addressing comments

9e54221

addressing comments and rename setup content

7aac374

ColeHiggins2 force-pushed the reporting-comp-eval-test-fixes branch from dd3591e to 7aac374 Compare May 10, 2024 15:04

ColeHiggins2 requested a review from sambible May 10, 2024 15:04

ColeHiggins2 requested a review from vsedmik May 10, 2024 15:05

sambible approved these changes May 10, 2024

View reviewed changes

Satellite-QE added the PRT-Passed Indicates that latest PRT run is passed for the PR label May 10, 2024

vsedmik approved these changes May 10, 2024

View reviewed changes

vsedmik merged commit 06e5529 into SatelliteQE:master May 10, 2024
10 checks passed

Satellite-QE mentioned this pull request May 10, 2024

[6.15.z] [Comp Eval] Updating fixture to SCA and Fixing failing tests #15034

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Comp Eval] Updating fixture to SCA and Fixing failing tests #14167

[Comp Eval] Updating fixture to SCA and Fixing failing tests #14167

ColeHiggins2 commented Feb 23, 2024

lhellebr Feb 26, 2024

lhellebr Feb 26, 2024

ColeHiggins2 Apr 1, 2024

ColeHiggins2 Apr 1, 2024

omkarkhatavkar commented Feb 29, 2024

Satellite-QE commented Feb 29, 2024

vsedmik left a comment

vsedmik Apr 8, 2024

vsedmik Apr 8, 2024

vsedmik Apr 25, 2024

ColeHiggins2 Apr 30, 2024

vsedmik Apr 8, 2024

ColeHiggins2 Apr 30, 2024

vijaysawant left a comment

sambible left a comment

sambible May 9, 2024

vsedmik left a comment

vsedmik May 10, 2024

vijaysawant left a comment

sambible May 10, 2024

ColeHiggins2 commented May 10, 2024

Satellite-QE commented May 10, 2024

vsedmik left a comment

[Comp Eval] Updating fixture to SCA and Fixing failing tests #14167

[Comp Eval] Updating fixture to SCA and Fixing failing tests #14167

Conversation

ColeHiggins2 commented Feb 23, 2024

Problem Statement

Solution

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

omkarkhatavkar commented Feb 29, 2024

Satellite-QE commented Feb 29, 2024

vsedmik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vijaysawant left a comment

Choose a reason for hiding this comment

sambible left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vsedmik left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vijaysawant left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ColeHiggins2 commented May 10, 2024

Satellite-QE commented May 10, 2024

vsedmik left a comment

Choose a reason for hiding this comment