Understanding the semantics of the @job, @graph, and @graph_asset composition functions

[sryza]

Dagster's @job, @graph, @graph_asset, and @graph_multi_asset decorators decorate what Dagster calls "composition functions". Even though they look like regular functions, they are actually a miniature DSL for constructing a graph of ops.

At the time that the function definition is evaluated, Dagster also evaluates the body of the function to build up the graph of dependencies. Dagster doesn't invoke the ops themselves - it only does that when executing the job.

I.e. this:

from dagster import graph

@graph
def graph1():
    op2(op1())

is essentially shorthand for this:

from dagster import DependencyDefinition, GraphDefinition

graph1 = GraphDefinition(
    name="graph1",
    node_defs=[op1, op2],
    dependencies={"op2": {"in1": DependencyDefinition("result")}},
)

Considered harmful: non-dep code inside composition functions

It's inadvisable to put code inside a @job- or @graph- decorated function that does anything other than define dependencies between ops. Doing so can lead to unexpected behavior. For example, in the following code, this "hello from job1" will be printed when you import the module, not when you execute the job:

from dagster import job, op

@op 
def op1():
    print("hello from inside op1")

@job 
def job1():
    print("hello from inside job1")
    op1()

"hello from inside op1" will only be printed when you execute the job, as expected.

Why evaluate the graph at definition time?

It allows Dagster to understand the topology of your graph before you run it. This is helpful for a few reasons:

• It makes it possible to render the graph in the UI before it runs
• If there's a failure, it allows Dagster to report what ops didn't run but were supposed to run
• It allows Dagster to catch problems, like cycles in the graph or invalid invocations, earlier