Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kuadrantctl tests #463

Merged
merged 1 commit into from
Jul 3, 2024
Merged

Add Kuadrantctl tests #463

merged 1 commit into from
Jul 3, 2024

Conversation

pehala
Copy link
Contributor

@pehala pehala commented Jun 25, 2024

  • Kickoff of Kuadrantctl tests
    • Due to significant rework, works with only 0.2.4-alpha and newer releases
    • New top level test folder tests/kuadrantctl and target make kuadrantctl
    • Minimal abstraction, 90% things are done directly and can be reproduced easily
  • OAS manipulation
    • OAS is loaded as Pure OAS, that is further customized runtime by adding security or x-kuadrant fields
    • Customization done in two parts base_oas and oas_kuadrant fixtures
      • base_oas is for editing pure OAS specs, i.e adding security constrains
      • oas_kuadrant is for adding x-kuadrant fields

Reviewable but not mergable due to a few outstanding bugs and enhancements:
Gonna be merged even with temporary fixes due to outstandiong issues with kuadrantctl itself
Kuadrant/kuadrantctl#94, Kuadrant/kuadrantctl#91, Kuadrant/kuadrantctl@d198cc8

Fixes #434 #365

@pehala pehala requested review from jsmolar and azgabur June 25, 2024 12:44
@pehala pehala force-pushed the kuadrantctl branch 2 times, most recently from 18dd914 to 72c2aec Compare June 25, 2024 13:35
testsuite/kuadrantctl.py Outdated Show resolved Hide resolved
testsuite/oas.py Outdated Show resolved Hide resolved
@trepel
Copy link
Contributor

trepel commented Jun 27, 2024

Sorry for long comment, I was lazy/unable to identify pieces of code that these bulletpoints relate to

  1. running make kuadrantctl causes three gateway CRs being created instantly at once. Not sure if this can cause any potential issues or not, I just find it strange.

  2. RLPs and AuthPolicies name is route-* (it is actually the same as the target HTTPRoute). Again, not sure if this can cause any issues but less confusing names would be better

  3. Intermittent error on RLP test if executed via make kuadrantctl:

testsuite/tests/kuadrantctl/cli/test_simple_limit.py:49: in test_generate_limit
    responses.assert_all(status_code=200)
testsuite/httpx/__init__.py:93: in assert_all
    assert request.status_code == status_code, (
E   AssertionError: Status code assertion failed for request 3 out of 3 requests: Result[status_code=429] != 200
---------------------------------------------------------------------------------------- Captured log call ----------------------------------------------------------------------------------------
10:08:13 +0200 INFO:httpx:HTTP Request: GET http://hostname-trepel--qmv8-kuadrant.apps.kua-stage-single.osp.api-qe.eng.rdu2.redhat.com/anything "HTTP/1.1 200 OK"
10:08:13 +0200 INFO:httpx:HTTP Request: GET http://hostname-trepel--qmv8-kuadrant.apps.kua-stage-single.osp.api-qe.eng.rdu2.redhat.com/anything "HTTP/1.1 200 OK"
10:08:13 +0200 INFO:httpx:HTTP Request: GET http://hostname-trepel--qmv8-kuadrant.apps.kua-stage-single.osp.api-qe.eng.rdu2.redhat.com/anything "HTTP/1.1 429 Too Many Requests"

I suspect it is due to the fact that the RLPs are identical and HTTPRoute is re-used across the tests so there might be a clash time to time. Potential solution might be to make the duration shorter (10s or even less) or maybe re-create the HTTPRoute as well or maybe do not have the RLPs identical (might be enough to have some random string as part of the name or smt like that)

  1. It does not work with kuadrantctl binary built from main branch. It fails generating HTTPRoute with an error: resource name may not be empty. Probably bug in kuadrantctl itself rather that this PR so just fyi

@pehala
Copy link
Contributor Author

pehala commented Jun 27, 2024

  1. running make kuadrantctl causes three gateway CRs being created instantly at once. Not sure if this can cause any potential issues or not, I just find it strange.

This is plausible because tests are running in parallel

2. RLPs and AuthPolicies name is route-* (it is actually the same as the target HTTPRoute). Again, not sure if this can cause any issues but less confusing names would be better

That is out of my control, I apply exactly what I get from the kuadrantctl and as far as I know there is not option to specify name

3. Intermittent error on RLP test if executed via make kuadrantctl:

I will investigate that one, it could be because that route

4. It does not work with kuadrantctl binary built from main branch. It fails generating HTTPRoute with an error: resource name may not be empty. Probably bug in kuadrantctl itself rather that this PR so just fyi

Will try it out, so far I only tested released version

@jsmolar
Copy link
Contributor

jsmolar commented Jun 27, 2024

tests/conftest.py should not contain kuadrant fixture

@pehala
Copy link
Contributor Author

pehala commented Jun 28, 2024

tests/conftest.py should not contain kuadrant fixture

I do not touch that file at all, not sure if the change needs to be done here

@pehala pehala force-pushed the kuadrantctl branch 2 times, most recently from 6590439 to 778c3a2 Compare June 28, 2024 10:20
@pehala pehala marked this pull request as ready for review July 1, 2024 10:20
@jsmolar jsmolar requested a review from trepel July 1, 2024 14:09
Copy link
Contributor

@trepel trepel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This time all the tests passed using kuadrantctl built from current main branch but all the tests failed using kuadrantctl v0.2.3 (current GA). Exactly the opposite as it was before the force push (same error resource name may not be empty). I would hesitate merging this if not working on both (once 0.2.4 is GA'd we can merge imo, not sure about the timeline though).

I tried several times, I did not encounter the RLP issue I mentioned (unexpected 429s). It seems that re-creating the HTTPRoute indeed fixed the issue.

testsuite/oas.py Outdated Show resolved Hide resolved
testsuite/tests/kuadrantctl/cli/test_simple_auth.py Outdated Show resolved Hide resolved
@pehala
Copy link
Contributor Author

pehala commented Jul 1, 2024

I would hesitate merging this if not working on both (once 0.2.4 is GA'd we can merge imo, not sure about the timeline though).

Sadly, that is not easily achievable. There is a breaking change (Kuadrant/kuadrantctl#78) that changes a lot (where exactly the extension is specified) and I do not think it warrants the time investment and complexity needed to make it interchangable. However, we now have https://github.com/Kuadrant/kuadrantctl/releases/tag/v0.3.0, which is now GA

@trepel
Copy link
Contributor

trepel commented Jul 2, 2024

However, we now have https://github.com/Kuadrant/kuadrantctl/releases/tag/v0.3.0, which is now GA

Cool, I am happy to approve once you address the two comments then!

@pehala pehala requested a review from trepel July 2, 2024 09:18
@jsmolar
Copy link
Contributor

jsmolar commented Jul 2, 2024

@azgabur please review. I want you to take a look at this PR.

trepel
trepel previously approved these changes Jul 2, 2024
Copy link
Contributor

@trepel trepel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am far from being and expert but this is good to go afaict. It works with kuadrantctl v0.3.0 (current GA).

azgabur
azgabur previously approved these changes Jul 2, 2024
Copy link
Contributor

@azgabur azgabur left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, in this state it can be merged.

@pehala pehala dismissed stale reviews from azgabur and trepel via ba13072 July 2, 2024 12:54
@pehala pehala requested review from trepel and azgabur July 2, 2024 12:54
trepel
trepel previously approved these changes Jul 3, 2024
Copy link
Contributor

@trepel trepel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have no idea what you force pushed, but LGTM still, I tried again and it worked (all tests passed).

@pehala
Copy link
Contributor Author

pehala commented Jul 3, 2024

Removed committed

@pehala pehala merged commit d5e4d88 into Kuadrant:main Jul 3, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automation for KuadrantCTL
4 participants