Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Race condition mounting static build files to web/worker containers #15046

Open
1 task done
KevinMind opened this issue Sep 25, 2024 · 2 comments · May be fixed by mozilla/addons-server#22780
Open
1 task done
Assignees

Comments

@KevinMind
Copy link
Contributor

KevinMind commented Sep 25, 2024

What happened?

Sometimes when starting web/worker containers we experience a race condition from the static files volume being simultaneously mounted in both containers.

The error occurs in two instances

  1. during the docker build while parallel building stages depend on the same build cache mount (pip_dev + pip_prod)
  2. during docker compose up, while parallel starting containers depend on the same bind mount (web + worker both depend on data_olympia, data_static_assets, etc)
 Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/docker/volumes/4ea2bf3bbeca2c0fc0556d30eef0f6a0658efe9640c65066176ca80b3881b3e6/_data" to rootfs at "/data/olympia/static-build": create mount destination for /data/olympia/static-build mount: mkdirat /var/lib/docker/overlay2/17a38c9d8c0bde0b144f811d4425cd748b258d2e39ce6d43722b066f93b8f685/merged/data/olympia/static-build: file exists: unknown

We can avoid this by not mounting to the worker container as it is not needing these files.

What did you expect to happen?

It should not happen

Is there an existing issue for this?

  • I have searched the existing issues

┆Issue is synchronized with this Jira Task

@diox
Copy link
Member

diox commented Oct 3, 2024

This is happening too often to ignore. This is already annoying on PRs, but it also affects deploys sometimes, which creates friction and wastes our time.

@diox
Copy link
Member

diox commented Oct 9, 2024

@KevinMind is this a dupe of #15036 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants