Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Release sockets without caching #131

Merged
merged 3 commits into from
Mar 21, 2024
Merged

Conversation

kevingrismore
Copy link
Contributor

The previous solution to the socket accumulation issue was overcomplicated and introduced caching when it wasn't necessary.

Instead, we can clear the pool_manager the same way we do for the other k8s client types.

The same test as the previous fix was conducted, and the same results were observed, with the occupied socket count on the worker pod returning to 4 after all flow runs completed.

root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
7
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
15
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
17
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
17
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
17
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
17
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
18
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
17
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
16
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
18
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
19
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
20
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
17
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
18
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
10
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
10
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
9
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
9
root@prefect-worker-7cc77cd984-88pvp:~# cat /proc/net/tcp | wc -l
4

Example

Screenshots

Checklist

  • References any related issue by including "Closes #" or "Closes ".
    • If no issue exists and your change is not a small fix, please create an issue first.
  • Includes tests or only affects documentation.
  • Passes pre-commit checks.
    • Run pre-commit install && pre-commit run --all locally for formatting and linting.
  • Includes screenshots of documentation updates.
    • Run mkdocs serve view documentation locally.

@kevingrismore kevingrismore requested a review from a team as a code owner March 21, 2024 19:31
else:
raise

@contextmanager
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This makes _get_configured_kubernetes_client work like all the other k8s client getters, where we client.rest_client.pool_manager.clear() on exit, which releases sockets.

enable_socket_keep_alive(client)

yield client
finally:
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above comment

Copy link
Contributor

@chrisguidry chrisguidry left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I LOVE it

@kevingrismore kevingrismore merged commit 0a16944 into main Mar 21, 2024
6 checks passed
@kevingrismore kevingrismore deleted the release-sockets-without-caching branch March 21, 2024 19:53
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants