RFC: @pipeline and @composite_solid become @graph

[schrockn]

Context: At Dagster’s inception, there were only pipelines and solids. Pipelines were the entry point and unit of execution, and solid was a unit of computation/business logic. We decided to add composite solids as a separate unit of composition within a pipeline. In retrospect, both of these decisions appear to have been mistaken. Solids are not executable in the playground but pipelines are. Pipelines support neither inputs, outputs, nor config mapping. We have different UI displays for pipelines and composites. We have unnecessarily divergent python APIs for executing solids and pipelines. And our internal implementation is needlessly complicated.

Our current hierarchy looks like this:

The proposal to merge @pipeline and @composite_solid into a single concept, @graph. And then also to make @solids executable by allowing modes and presets on solids. Therefore graphs and solids are both executable units using a unified API, and support inputs, outputs, and config mapping.

Also nicely, using @graph lines up with our new messaging around the “orchestration graph”.

We believe this will provide value on a number of dimensions:

• One less concept for new users to learn
• Less duplicative UI
• Ability to execute solids in isolation
• More consistent solid subselection APIs and UIs
• Unified python APIs
• All graphs support inputs, outputs, and config mapping, which allows authors to design easier-to-operate top-level graphs.

This is quite an invasive change so we expect to have a longer period of time to migrate @pipeline and @composite_solid to @graph. Likely one major release without a deprecation warning, then another full cycle with deprecation, then deletion.

[kinghuang]

I generally agree with the direction of this proposal. I think it would help on a couple of difficulties I've run into.

1. If both solids and graphs have inputs, outputs, and configs, and are equally runnable, it should make composing graphs of graphs more natural. I have multiple pipelines that I would like to just hook into each other.
2. If graph configs are passed down to solids within, it would simplify configuring multiple solids within the same graph.

My main question is if both @solid and @graph have equal support for inputs, outputs, configs, modes, and presets, and are also executable, what differentiates them? They sound like the same thing to me.

[schrockn]

The difference is that the @solid would define a function that runs at execution time whereas as @graph defines a dependency graph of graphs and solids.

[kinghuang]

Ah, of course. I had a brain fart there. I was thinking why wouldn't I just label everything a @solid if they have the same capabilities as a @graph. But, then, that doesn't build up a graph. Oops!

[lee335]

Hi dagster team!

Always wondered why there was the distinction between pipeline and composite solid. This feels much more natural! We are looking forward for this change in the API.

[AdamDorwart]

Definitely a fan. Always thought composite solids were a bit confusing for new users.
#2374

Prefer the term pipeline or dag instead of graph (unless cycles in compute graphs will be allowed) but either way this is a good simplification.

[xinbinhuang]

Upvote on this to help move this proposal forward!

Also, I just want to mention some experiences, which you probably already know, from Apache Airflow - it has a similar concept as SubDag, and it has been causing a lot of issues because of its complexity. So yeah, strongly support for this unified and simplified concept and APIs.

[schrockn]

It has begun: https://dagster.phacility.com/D4847

[schrockn]

Update: We have decided to bump this out to the next major release (not 0.10.0) but we anticipate 0.11.0 will be a shorter timeline than 0.10.0.

[kinghuang]

I take it this didn't make it into 0.11? There's no mention of it in the release notes.

[schrockn]

It did not, unfortunately. We need to make some other changes in order to get this to land in a place that we are happy with.

[k0dit]

I'm really looking forward to this change. So I wanted to check in on when it will be available for use?

[alangenfeld]

We are finally on a trajectory we are happy with and I believe @sryza is working on writing up something to share soon. We are aiming to have a version people can try ready for 0.12.0 expected early July.

[alangenfeld]

A preview of the new APIs to try out and give feedback on are now available
https://github.com/dagster-io/dagster/releases/tag/0.12.0