Measure adoption

[pankajkoti]

Description co-authored by @tatiana @pankajastro

The goal with this Epic is to allow us to track how is the DAG Factory adoption by Astro customers, if possible at a DAG run level.

@cmarteepants has been analysing how we can measure DAG factory adoption. Apparently , as of v0.19.0, DAG factory does not have a way of identifying in the created DAGs that they were created by DAG factory.

What Constance did was to align with multiple teams within Astronomer so we programatically query to find which dependencies are installed in Astro deployments. With this information, we'd know how many Astro customers have DAG factory installed.

As part of this epic, we review this approach and check if / how we should give visibility of which DAGs were created with DAG factory. The solution may only work with future versions of DAG Factory.

Some approaches we could look into:

• Generate some events during the conversion of DAG Factory using Add Scarf based telemetry apache/airflow#39510 (would only work for newer versions of Airflow, or we'd need to have DAG Factory using this tool to emit them)
• Check the possibility of annotating the DAGs created by DAG Factory in other ways - making sure we can query those in the Astro SF database

Tasks associated:

• PoC evaluate alternatives of identifying, in Airflow, a DAG was created with DAG Factory #210

More will likely be created after this PoC

[tatiana]

@cmarteepants will detail this epic further, but we can track the progress on Astro cloud side:
https://github.com/astronomer/astro/pull/22777
https://astronomer.slack.com/archives/C07DCAVK7TR/p1722950587384539

[cmarteepants]

Issue for the Scarf-based telemetry #214

[tatiana]

Closed by #250