Executing Distributed Compute over Data tasks from Flyte (using Bacalhau)

[aronchick]

Hi!

I'm David Aronchick, co-founder of Bacalhau. For those that don't know it, Bacalhau "is a platform for fast, cost efficient, and secure computation by running jobs where the data is generated and stored. With Bacalhau you can streamline your existing workflows without the need of extensive rewriting by running arbitrary Docker containers and WebAssembly (wasm) images as tasks."

It is open source and one of our core goals is to NOT build solutions better handled by other functions - specifically relevant here, is a we do not want to build a workflow engine :)

We already have an integration for Airflow and we are proud to announce this week a new integration for Flyte!

My thoughts in raising this RFC:

• Is this useful? The user story is "I have a workflow in Flyte. I need to execute this computation using WASM/container/arbitrary binary on data X but I don't know where it is/it is very far away."
• Did we do it "right"? E.g. we would like to idomatically build our integration so that it feels "Flyte native".
• Is there some form of a standard we want to work on here for "use this external thing to execute a step in Flyte"? Maybe we already have it, but there appears in this discussion similar needs for K8s, Spark, etc. Maybe it's worth an interface?
• Anything else?

Thank you so much!

[davidmirror-ops]

Did we do it "right"? E.g. we would like to idomatically build our integration so that it feels "Flyte native".

I'd add that Flyte recently introduced the Agents framework to build integrations that appeal to both plugin developers (writing plugins in Python) and also reduces the context switch and additional resource consumption that comes from instantiating a Pod for every Task execution that invokes an external system: https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/development_lifecycle/agent_service.html#extend-agent-service. That'd be an interesting way of leveraging recent improvements in Flyte to keep improving this integration.