[Question] - Plans to define a DAG to capture the ETL process

[goutamvenkatCatalan]

Preferably I'm looking for a tool similar to Dagster, Prefect, or Airflow, which are popular tools in the python ecosystem that allow users to express data pipelines in the form of DAGs and have them be recurring, containerizable etc. Are there plans to expose that sort of functionality in this project? Thanks.

[paillave]

I am afraid I'm not really sure what you are talking about as I'm not familiar with tool mentioned.

For now, you can get the estiamted execution plan (the directed acyclic graph in shape of a sankey graph) like described here: https://paillave.github.io/Etl.Net/docs/tutorials/trackAndCheck#get-the-estimated-execution-plan

If you want to build an ETL using a visual tool, you will find an answer here: https://paillave.github.io/Etl.Net/docs/intro#visual-designer

There is also the possibility to transmit a script to... something, but this "something" is not implemented yet.
In a company, we built a system that permits to have a CSX script as a parameter that is handled by roslyn.
It is not impossible that such a thing might be opensourced some day; but it is not on the road map at the very moment.