Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Add worker pool to prefect #49

Open
phwissmann opened this issue May 2, 2024 · 0 comments
Open

[Feature] Add worker pool to prefect #49

phwissmann opened this issue May 2, 2024 · 0 comments
Assignees
Labels
enhancement New feature or request
Milestone

Comments

@phwissmann
Copy link
Collaborator

phwissmann commented May 2, 2024

Description
In order to scale up the number of parallel jobs and ease the deployment of flows, a worker pool needs to be added. This requires parameterize some of the jobs.

Update: Main goal is to be able to deploy updated flows to a remote prefect instance, i.e from the laptop to scopem-openem.ethz.ch

Solution proposals
Add a docker worker

Definition of Done

  • Implementation is deployed on Scopem-openem.ethz.ch
  • workers can be added to prefect on a different physical or the same machine
  • workers run in a container and span flows in a container
  • flows can be deployed from developer machine
@phwissmann phwissmann added the enhancement New feature or request label May 2, 2024
@phwissmann phwissmann added this to the v1.0-beta milestone May 2, 2024
@phwissmann phwissmann self-assigned this May 2, 2024
@phwissmann phwissmann modified the milestones: v0.9, v0.8 May 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant