How to update run config of currently running job #25060
-
|
TLDR: Is it possible to update the run config of an already running job? I have a prerequisite job A that runs in Dagster, that triggers either job B or C somewhere else (not in Dagster) once it completes. Users can either trigger A+B or A+C, and which downstream job to trigger is passed in the Dagster run config: {should_run_B: Boolean, should_run_C: Boolean}. When a user rapidly triggers both pipelines, then A is run twice which is wasted work. The most ideal solution would be: if A+B is already running, and a request to trigger A+C is received, then modify the run config of the currently running job A and set should_run_C to True. I'm very new to Dagster and can't find any mention of how to do this. Is this possible and how? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment
-
|
It's not possible to modify the run config of a running job. A way I can think to handle this is if the job that the users are initiating is really a stub job to act as a precursor. Then a sensor can detect when this precursor job has been run and logic within its evaluation code can handle whether to trigger a run for the real Job A. This way you could include a check for if Job A is already running, such as in this example |
Beta Was this translation helpful? Give feedback.
It's not possible to modify the run config of a running job.
A way I can think to handle this is if the job that the users are initiating is really a stub job to act as a precursor. Then a sensor can detect when this precursor job has been run and logic within its evaluation code can handle whether to trigger a run for the real Job A. This way you could include a check for if Job A is already running, such as in this example