-
Notifications
You must be signed in to change notification settings - Fork 8
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cleanup job for running containers #409
Comments
Took a look into this. Currently, when pressing CTRL+C the process gets terminated with a SIGINT signal by Linux. The nicest way to clean up containers would probably be implementing drop on the structure which holds the job with the container id. I already tried this and ran into issues with how tokio currently handles panics. When receiving the CTRL+C signal and trying to unwinding the program with std::process::ExitCode or calling down panic, just the current thread panics and other tokio threads continue to run. This is wanted by tokio but there are open issues which try to implement custom behaviur for how to handle panics in threads. tokio-rs/tokio#2002 The feature can already be used in tokio unstable. But the unstable version brings many other changes/issues and I would recommend waiting for it to become stable. https://docs.rs/tokio/1.40.0/tokio/runtime/struct.Builder.html#method.unhandled_panic |
Nice catch. The current code use a bit of a fire-and-forget approach where the containers get started but then run to completion with no way to stop them (IIRC) - butido only streams the logs, updates the current status, saves artifacts, etc. My idea with @ammernico was to use the RAII pattern to ensure that all containers that a butido-build command starts do also get stopped. Unfortunately it wasn't quite that easy as Ctrl+c aborts the program so we need to implement a graceful shutdown. Ideally we can shutdown the Tokio runtime but if that doesn't work we probably have to implement that part ourselves (i.e., we should at least check if Ctrl-c was pressed during the butido-build part where the containers are running). Some "random" notes:
|
- Add the `signal` feature to `tokio` to interrupt and handle the Control-C signal in Butido. - Add Control-C signal handling into the `Orchestrator`. - Implement `Drop` on the `JobHandle` to ensure container cleanup. Fixes science-computing#409 Signed-off-by: Nico Steinle <[email protected]>
- Add the `signal` feature to `tokio` to interrupt and handle the Control-C signal in Butido. - Add Control-C signal handling into the `Orchestrator`. - Implement `Drop` on the `JobHandle` to ensure container cleanup. Fixes #409 Signed-off-by: Nico Steinle <[email protected]>
- Add the `signal` feature to `tokio` to interrupt and handle the Control-C signal in Butido. - Add Control-C signal handling into the `Orchestrator`. - Implement `Drop` on the `JobHandle` to ensure container cleanup. Fixes #409 Signed-off-by: Nico Steinle <[email protected]>
- Add the `signal` feature to `tokio` to interrupt and handle the Control-C signal in Butido. - Add Control-C signal handling into the `Orchestrator`. - Implement `Drop` on the `JobHandle` to ensure container cleanup. Fixes science-computing#409 Signed-off-by: Nico Steinle <[email protected]>
- Add the `signal` feature to `tokio` to interrupt and handle the Control-C signal in Butido. - Add Control-C signal handling into the `Orchestrator`. - Implement `Drop` on the `JobHandle` to ensure container cleanup. Fixes science-computing#409 Signed-off-by: Nico Steinle <[email protected]>
atm if a longer job is running there is no possibility to stop the running containers
CTRL-C does not stop the currently running containers/jobs but just gives you back the command prompt but the jobs keep running
maybe introduce some kind of cleanup job?
regular use case here: you want to add a minor change and rerun
The text was updated successfully, but these errors were encountered: