Packaging code for Dask execution

[ahoneycutt]

It seems to me that since we ship a @solid to the executor, we would need to have our @solid code available on the Dask side of things as well. This would also mean that we would need dagster and dagster-dask installed on the target Dask worker.

• Is my above understanding correct?
• Is there a way to package the code without the need of packaging the solid itself?
• Are there any examples of how to transmit code to Dask workers that doesn't include creating a package?

[alangenfeld]

Our current Dask integration is pretty basic - and operates under the assumption that the dask workers have access to the same dagster definitions as the process that is executing the pipeline run.

Is my above understanding correct?
This would also mean that we would need dagster and dagster-dask installed on the target Dask worker.

Yes, also the pipeline and solid code need to be available.

Is there a way to package the code without the need of packaging the solid itself?
Are there any examples of how to transmit code to Dask workers that doesn't include creating a package?

The model we use in our kubernetes deployment is to use the same docker image for both the "workers" and the box running dagit (or user code deployments dagit is set up to talk to). If not using docker you could have a scheme where you checkout the same git repo to the same commit in both places or something similar to that.

[ahoneycutt]

Hi @alangenfeld thank you for the response. I think the proposition of pulling the repo at the same revision is a perfectly suitable way to do business. I had initially thought that the code >inside< the solid needed to be callable from the worker context, but thanks for clearing this up!

Much appreciated,
Andy