Dagster-spark submit with REST API

[andgineer]

I want to execute dagster solids on Spark Standalone cluster.

Spark Standalone cluster does not support cluster mode.
And I cannot use client mode because in that mode my laptop will be Spark Driver for the task but it has no access to the Spark workers running in the cloud, only to Spark Driver.

Possible solution - submit jobs with Spark REST API on 6066

[sryza]

Hi @andgineer - I am not an expert on Spark Standalone clusters, but my understanding is that, for any Spark client mode execution, the driver process needs to be started somewhere. If that place can't be your laptop, then you need some way of starting the driver inside the cluster. If you have a way of doing that, then you can invoke it from a Dagster solid, but Dagster doesn't have any specialized way of launching processes on Spark Standalone clusters.