-
I'm currently evaluating dagster for workflows orchestration. I saw the map reduce example in the documentation . Example pseudo code:
Expected results :
where all the A -> B -C chains are executed in parrallel and listTestCases returns a dynamic results at runtime. I can do that with a static range of test cases known at compile time but i did'nt find a way to do this with a number of test cases only known at runtime. Is this possible with dagster ? Thanks in advance for your time and help ! |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
Hi @snoopfab, the following should work:
|
Beta Was this translation helpful? Give feedback.
-
Thanks for your proposition. I forgot to mention that i need the whole process to be asynchronous and parallel. What i want is the B1 task to start as soon as the A1 task is completed and the C1 task as soon as the B1 is completed. In your proposed solution , All the A_i tasks are executed in parallel, then the results are collected, and after this All the B_i tasks are executed etc ... I can do what i want (asyncronicity and parallelism) with a static number of testcase known at job definition time but in my case it can only be known at runtime. Moreover, i cannot aggregate the A, B,C tasks in only one task as the B_i tasks needs a specific celery worker with unique ressources (ex: A GPU). |
Beta Was this translation helpful? Give feedback.
Hi @snoopfab, the following should work: