feat(anta): Limit concurrency #680

carl-baillargeon · 2024-05-16T05:04:18Z

Description

This PR improves the test runner by introducing a generator-based approach for managing test coroutines and setting a configurable limit on the number of concurrent tests.

Instead of loading all test coroutines into a list, the runner now uses a generator to yield tests. This approach prevents memory overload and improves performance when dealing with a large number of tests.
A limit on the number of concurrent tests is introduced to avoid overwhelming the runner. This limit is configurable with an environement variable (hidden).

Implementation:
The generator yields test coroutines, ensuring that only a limited number of tests are scheduled and run concurrently.
Upon reaching the concurrency limit, the runner waits for some tests to complete before scheduling new ones from the generator.

Fixes: #713

Checklist:

Update FAQ to reference Scaling ANTA documentation
Update Scaling ANTA documentation - Add numbers @dlobato
Add section on JSON catalogs vs YAML (faster)
Need to retest on digital twin @dlobato
Support None timeouts
Confirm if PoolTimeout default should be set to None
Update benchmark + unit tests
ci: add codspeed to benchmark ANTA #826
Confirm that the number of open file descriptors never exceed device * max_connections (100 by default)

sonarcloud · 2024-06-12T16:43:30Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-07-04T14:03:49Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

anta/runner.py

github-actions · 2024-08-28T18:50:12Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

sonarcloud · 2024-08-28T19:07:52Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
84.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud

github-actions · 2024-09-11T13:47:41Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2024-10-02T15:05:25Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

codspeed-hq · 2024-10-02T15:15:13Z

CodSpeed Performance Report

Merging #680 will not alter performance

_{Comparing carl-baillargeon:refactor/runner_limit (ad22107) with main (8ac4477)}

Summary

✅ 6 untouched benchmarks

⁉️ 2 dropped benchmarks

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Benchmark	`main`	`carl-baillargeon:refactor/runner_limit`	Change
⁉️	`test_get_coroutines[1-device]`	48.1 ms	N/A	N/A
⁉️	`test_get_coroutines[2-devices]`	94.7 ms	N/A	N/A

github-actions · 2024-10-10T18:49:29Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2024-11-04T21:15:42Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

for more information, see https://pre-commit.ci

anta/runner.py

gmuloc · 2024-11-14T12:41:39Z

anta/device.py

+    The limits are set using the following environment variables:
+        - ANTA_MAX_CONNECTIONS: Maximum number of allowable connections.
+        - ANTA_MAX_KEEPALIVE_CONNECTIONS: Number of allowable keep-alive connections.
+        - ANTA_KEEPALIVE_EXPIRY: Time limit on idle keep-alive connections in seconds.


Link to httpx doc

gmuloc

Lots on comment on the guide but a very big Thank You for this amazing guide :)

🎉 🎈

gmuloc · 2024-11-14T12:55:09Z

anta/runner.py

 DEFAULT_NOFILE = 16384
+"""Default number of open file descriptors for the ANTA process."""
+DEFAULT_MAX_CONCURRENCY = 10000
+"""Default maximum number of tests to run concurrently."""


We may need to move all these to constants for simpler way to know where stuff is

gmuloc · 2024-11-14T12:59:00Z

anta/runner.py



 def adjust_rlimit_nofile() -> tuple[int, int]:
    """Adjust the maximum number of open file descriptors for the ANTA process.

    The limit is set to the lower of the current hard limit and the value of the ANTA_NOFILE environment variable.

-    If the `ANTA_NOFILE` environment variable is not set or is invalid, `DEFAULT_NOFILE` is used.
+    If the `ANTA_NOFILE` environment variable is not set or is invalid, `DEFAULT_NOFILE` is used (16384).


proabbly best to not put the value here - we then forget to update it :p

gmuloc · 2024-11-14T12:59:28Z

anta/runner.py

+        The result of each completed test.
+    """
+    # NOTE: The `aiter` built-in function is not available in Python 3.9
+    aws = tests_generator.__aiter__()  # pylint: disable=unnecessary-dunder-call


what does aws stands for here? :)

gmuloc · 2024-11-14T13:15:47Z

docs/advanced_usages/scaling.md

+
+# 🚀 Scaling ANTA: A Comprehensive Guide
+
+**Table of Contents:**


do we need this? this is redundant with

gmuloc · 2024-11-14T13:49:09Z

mkdocs.yml

@@ -179,6 +179,7 @@ nav:
    - Debug commands: cli/debug.md
    - Tag Management: cli/tag-management.md
  - Advanced Usages:
+    - Scaling ANTA: advanced_usages/scaling.md


not first - should be last or let's use alphabetical order

gmuloc · 2024-11-14T14:17:52Z

docs/advanced_usages/scaling.md

+
+### Results Management
+
+ANTA can output results in various formats:


do you need to repeat these? I would just point to the NRFU page documentatin and say you use JSON (less maintenance)

gmuloc · 2024-11-14T14:18:28Z

docs/advanced_usages/scaling.md

+3. Optional: You can merge the JSON results and generate a JUnit report using the following Python script (requires `junitparser`):
+
+    ```python
+    from pathlib import Path


please use snippets for this and we will try to test it :)

Should we provide this by default in ANTA as a feature?

gmuloc · 2024-11-14T14:20:17Z

docs/advanced_usages/scaling.md

+    - 📞 Reach out to your Arista SE for guidance
+    - 📝 Document your specific use case on [GitHub](https://github.com/aristanetworks/anta)
+    - 🔍 Share your findings with the community


repeat from the beginning - probably should be the same (or just a button that goes back to the beginning?:)

gmuloc · 2024-11-14T14:21:02Z

docs/advanced_usages/scaling.md

+
+## 📚 References
+
+- **Python AsyncIO**


need 4 spaces it does not render properly:

gmuloc · 2024-11-14T14:22:42Z

tests/benchmark/test_runner.py

@@ -7,8 +7,7 @@

 from typing import TYPE_CHECKING

-from anta.result_manager import ResultManager
-from anta.runner import get_coroutines, prepare_tests
+from anta.runner import prepare_tests


we can add back get_coroutines for now

sonarcloud · 2024-11-14T18:31:57Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
83.8% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

github-actions bot assigned carl-baillargeon May 16, 2024

carl-baillargeon force-pushed the refactor/runner_limit branch from bf88158 to d7de654 Compare June 12, 2024 16:43

github-actions bot added the state: conflict label Jul 4, 2024

gmuloc reviewed Aug 1, 2024

View reviewed changes

anta/runner.py Outdated Show resolved Hide resolved

gmuloc reviewed Aug 1, 2024

View reviewed changes

anta/runner.py Outdated Show resolved Hide resolved

carl-baillargeon force-pushed the refactor/runner_limit branch from b623e90 to 4879120 Compare August 28, 2024 18:49

github-actions bot removed the state: conflict label Aug 28, 2024

github-actions bot added the state: conflict label Sep 11, 2024

mtache changed the title ~~refactor(anta): Refactor runner to use a generator with a limit~~ feat(anta): Limit concurrency Oct 2, 2024

feat(anta): limit the number of tests run concurrently

83206e8

mtache force-pushed the refactor/runner_limit branch from 68e259e to 83206e8 Compare October 2, 2024 15:05

github-actions bot removed the state: conflict label Oct 2, 2024

gmuloc and others added 2 commits October 7, 2024 09:53

Merge branch 'main' into refactor/runner_limit

d86f85c

Merge branch 'main' into refactor/runner_limit

6d59aa9

github-actions bot added the state: conflict label Oct 10, 2024

Merge branch 'main' into refactor/runner_limit

d2aa7c7

github-actions bot removed the state: conflict label Nov 4, 2024

gmuloc and others added 2 commits November 4, 2024 23:30

Refactor: Fix github merge conflict resolution

7ac0358

[pre-commit.ci] auto fixes from pre-commit.com hooks

96da6ef

for more information, see https://pre-commit.ci

gmuloc reviewed Nov 4, 2024

View reviewed changes

anta/runner.py Show resolved Hide resolved

test: Temporarily disable one benchmark

f2f3c6a

gmuloc and others added 11 commits November 6, 2024 10:16

Merge branch 'main' into refactor/runner_limit

0919b20

Added HTTPX timeouts and limits

b91715e

Fix docstrings

bfd56d1

Add doc

d95cb52

Update doc

3dede8b

Single quote for Python

d3abe57

Merge branch 'main' into refactor/runner_limit

632610b

Merge branch 'main' into refactor/runner_limit

ee2ece2

Fix: Let's keep the artices

2ccc080

Merge branch 'main' into refactor/runner_limit

c6bd59a

Merge branch 'main' into refactor/runner_limit

ad22107

gmuloc reviewed Nov 14, 2024

View reviewed changes

Merge branch 'main' into refactor/runner_limit

b8bdc36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(anta): Limit concurrency #680

feat(anta): Limit concurrency #680

carl-baillargeon commented May 16, 2024 •

edited

Loading

sonarcloud bot commented Jun 12, 2024

github-actions bot commented Jul 4, 2024

github-actions bot commented Aug 28, 2024

sonarcloud bot commented Aug 28, 2024

github-actions bot commented Sep 11, 2024

github-actions bot commented Oct 2, 2024

codspeed-hq bot commented Oct 2, 2024 •

edited

Loading

github-actions bot commented Oct 10, 2024

github-actions bot commented Nov 4, 2024

gmuloc Nov 14, 2024

gmuloc left a comment

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

gmuloc Nov 14, 2024

sonarcloud bot commented Nov 14, 2024


		# 🚀 Scaling ANTA: A Comprehensive Guide

		Table of Contents:


		### Results Management

		ANTA can output results in various formats:

feat(anta): Limit concurrency #680

Are you sure you want to change the base?

feat(anta): Limit concurrency #680

Conversation

carl-baillargeon commented May 16, 2024 • edited Loading

Description

Checklist:

sonarcloud bot commented Jun 12, 2024

Quality Gate passed

github-actions bot commented Jul 4, 2024

github-actions bot commented Aug 28, 2024

sonarcloud bot commented Aug 28, 2024

Quality Gate passed

github-actions bot commented Sep 11, 2024

github-actions bot commented Oct 2, 2024

codspeed-hq bot commented Oct 2, 2024 • edited Loading

Merging #680 will not alter performance

Summary

Benchmarks breakdown

github-actions bot commented Oct 10, 2024

github-actions bot commented Nov 4, 2024

Choose a reason for hiding this comment

gmuloc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarcloud bot commented Nov 14, 2024

Quality Gate passed

carl-baillargeon commented May 16, 2024 •

edited

Loading

codspeed-hq bot commented Oct 2, 2024 •

edited

Loading