Prefect vs. Dagster (from 2/2022)

[bendnorman]

Note: This discussion contains information about our comparison between prefect and dagster that originally lived in a google doc. This comparison was created in February 2022 so some information, stats, links might be out of date.

Prefect

Prefect’s main goal is to solve “negative engineering” which is when engineers write defensive code to make sure the positive code runs. Positive code is code to achieve an objective, for example, write an ETL to clean energy data. Examples of negative engineering from Prefect :

What happens if data arrives malformed? What if the database goes down? What if the computer running the code fails? What if the code succeeds but the computer fails before it can report the success? Negative engineering is characterized by needing to anticipate this infinity of possible failures.

Prefect will be releasing Prefect Orion soon, their 2.0 release. While their guiding philosophy is to tackle negative engineering, they add a new hypothesis to the mix: DAGs create complexity.

You don’t need to rewrite your entire workflow as a directed acyclic graph (DAG) to take advantage of Orion. DAGs represent a rigid, constrained, outdated framework for creating modern, dynamic data and computation workflows.” - Orion Docs

It’s popular to describe “workflows as code” or “orchestration as code,” but Orion represents “code as code”: rather than ask users to change how they work to meet the requirements of the orchestrator, we’ve defined an orchestrator that adapts to how our users work.” - Orion FAQ

It seems like Prefect wants to be as flexible as possible and make it easy for users to add Prefect features to pieces of existing “workflow” code. Do we want a super flexible orchestrator that doesn’t require DAGs? I can see how this philosophy would be helpful for Airflow users with large existing workflows who want a more flexible tool.

Dagster

Dagster is focused on solving the issue of “ Big Complexity ”. They don’t actually define the term, but I think this is the sentiment:

The process of transforming, managing, and making sense of data crosses personas and skillsets; involves many different types of tools and technologies; is riddled with organizational, operational, and technical complexity; and is critical to the success of every modern organization.

They believe orchestration is the layer in the data stack to tie data tools together:

Historically, orchestrators and workflow engines focused exclusively on operational concerns: the reliable scheduling and ordering of deployed computations. They weren’t focused on practitioner productivity, weren’t designed as multi-tenant platforms, and weren’t specifically geared towards data asset development and production.

Dagster is a lot more than a workflow engine: it’s an orchestration platform for the development, production, and observation of data assets. We built Dagster to bring support for the full engineering lifecycle to the orchestrator and explicitly target the data management use case.

What sets Dagster apart from other orchestration tools is its focus on developing tools to process and document data assets. It seems like delivering high-quality data is more important to Dagster than solving software infrastructure problems.

It’s interesting how Prefect and Dagster recently changed their philosophy on DAGs. Prefect thinks DAGs are too restrictive for complex data workflows. Dagster believes data asset DAGs are more important than computation DAGs.

Community

Both of these tools are supported by open-core startups, with most of the contributions being made by employees of the VC backed companies. We run the risk of the companies folding or close-sourcing the tools. Worst case scenario, there isn't enough community support to keep the project going, and we'd need to refactor our code base again. I gathered some statistics about each project (I included Airflow for comparison).

Features

I created a matrix of how Prefect and Dagster address Catalyst’s orchestration needs.