New container tests fail intermittently on Aarch64 #835
Comments
troglobit
added a commit
that referenced
this issue
Nov 28, 2024
This is a complete redesign of how the system creates/deletes containers from the running-config. Containers are now removed synchronously from confd before any interfaces they may be using are removed, and created in parallel, using a Finit task, well after confd has finished setting up interfaces. The logic previously provided by execd to retry container create on any route/address changes, or periodically every 60 seconds, is now handled by a new 'setup' command in the container wrapper script. Additionally, container create is now split in wget/curl/podman pull of the image and 'podman create'. This to both consolidate image fetching and improve user feedback since most of the retry logic (above) revolves around the image download. Fixes #835 Signed-off-by: Joachim Wiberg <[email protected]>
troglobit
added a commit
that referenced
this issue
Nov 28, 2024
This is a complete redesign of how the system creates/deletes containers from the running-config. Containers are now removed synchronously from confd before any interfaces they may be using are removed, and created in parallel, using a Finit task, well after confd has finished setting up interfaces. The logic previously provided by execd to retry container create on any route/address changes, or periodically every 60 seconds, is now handled by a new 'setup' command in the container wrapper script. Additionally, container create is now split in wget/curl/podman pull of the image and 'podman create'. This to both consolidate image fetching and improve user feedback since most of the retry logic (above) revolves around the image download. Fixes #835 Signed-off-by: Joachim Wiberg <[email protected]>
17 tasks
17 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
The new use-case tests
ospf-containerandcontainer-firewall-basicfail intermittently on Aarch64 test systems. The failures result in "inconsistent container state" errors from Podman and manifest themselves as failure to create/start at least one of the containers.Further investigation show that the speed at which configuration is done at, using any of the M2M interfaces, unearths flaws in the asynchronous way that container creation/deletion is handled in the system. When a user adds/deletes a container the actual job is enqueued to
execd, which may lead to new configurations being applied beforeexecdhas completed a previous one.This requires a redesign/simplification to ensure the M2M interfaces wait for, at least, the deletion and cleanup of containers before applying a new configuration.
The text was updated successfully, but these errors were encountered: