Add support for loading dbt project from cloud store using Airflow Object Store

[CorsettiS]

Description

This PR is based on #1109 and intends to allow users to load entire dbt projects that are stored at a cloud provider. That way, it is totally possible to decouple dbt and airflow with cosmos.

As of now, the company I currently work at does not have dbt & airflow in the same repo, but by allowing cosmos to fetch entire dbt projects from a cloud provider we can work around that easily. dbt projects are not usually very heavy, so performance should not be drastically impacted (it was not in the local tests I have done).

Would be good if this could be included in the release 1.6.0 #1080

Breaking Change?

No

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 canceled.

Name	Link
🔨 Latest commit	8cde3a8
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66b66745ac04080008962bb5

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Name	Link
🔨 Latest commit	897090e
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/66bc95a4fa686b0008ad33fe
😎 Deploy Preview	https://deploy-preview-1148--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[CorsettiS]

honestly speaking I could not understand this even after reading the comment. both the self.dbt_project_path and self.models_path are already Path as specified in the init. I just reverted to its previous state

[CorsettiS]

This test is actually wrong since AIRFLOW_IO_AVAILABLE just means we are using airflow >=2.8.0, and not that we are using object storage.

[tatiana]

@corsettigyg wasn't object storage introduced in Airflow 2.8?
https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/objectstorage.html

[CorsettiS]

@tatiana indeed, although using airflow 2.8 does not exactly mean we are using object storage, which was the assumption of this test, so I just fixed it.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.53%. Comparing base (4886823) to head (eb3ec85).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1148   +/-   ##
=======================================
  Coverage   96.53%   96.53%           
=======================================
  Files          64       64           
  Lines        3374     3376    +2     
=======================================
+ Hits         3257     3259    +2     
  Misses        117      117           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tatiana]

@CorsettiS This is a very promising change, thank you!

Some questions:

1. How big are the dbt projects you've tested this feature with?
2. I'm slightly concerned about performance, since we'd be running this during DAG processing/parsing time very regularly usually in the scheduler - depending on the customer deployment. This may not scale for bigger dbt projects or for situations where users have many Cosmos DbtDags. Did you run any performance tests? Could you share some numbers / how you validated this feature?
3. Following (2), by syncing the whole dbt project from object store, we would potentially not benefit from the caching introduced in Cosmos 1.5 in PR Speed up LoadMode.DBT_LS by caching dbt ls output in Airflow Variable #1014. Do you have any proposals of how we could avoid performing this action every time?
4. Would using LoadMethod.MANIFEST + ExecutionMode.KUBERNETES be an option for your company? This should allow the decoupling you are aiming for. Or is there any particular reason why you'd need LoadMethod.DBT_LS and LoadMethod.LOCAL?

It seems unlikely we'll be able to release this in 1.6, given the number of things we're still trying to address before the release - but let's see!

[CorsettiS]

@CorsettiS This is a very promising change, thank you!

Some questions:

1. How big are the dbt projects you've tested this feature with?
2. I'm slightly concerned about performance, since we'd be running this during DAG processing/parsing time very regularly usually in the scheduler - depending on the customer deployment. This may not scale for bigger dbt projects or for situations where users have many Cosmos DbtDags. Did you run any performance tests? Could you share some numbers / how you validated this feature?
3. Following (2), by syncing the whole dbt project from object store, we would potentially not benefit from the caching introduced in Cosmos 1.5 in PR Speed up LoadMode.DBT_LS by caching dbt ls output in Airflow Variable #1014. Do you have any proposals of how we could avoid performing this action every time?
4. Would using LoadMethod.MANIFEST + ExecutionMode.KUBERNETES be an option for your company? This should allow the decoupling you are aiming for. Or is there any particular reason why you'd need LoadMethod.DBT_LS and LoadMethod.LOCAL?

It seems unlikely we'll be able to release this in 1.6, given the number of things we're still trying to address before the release - but let's see!

1. I have been dealing with a project with roughly 200 models. in our repo we do have heavy files that would impact performance since they would increase the time needed for airflow to fetch them, so we just upload the dbt "main" files to s3 to avoid under-performance (CI/CD scripts, JPEG files, etc)

2. About performance, I did some tests using my private dbt repo in s3 (size: 7 MB). For reference, my internet speed is 18 MB/s. I checked for the runtime of the command airflow dags reserialize (not the DAG build time provided by cosmos since I was confused if the fetching process happens inside or outside the DAG construction process itself, so I decided to be on the safe side). I ran the command 10 times and got the avg and std deviation.

in the first scenario, I loaded both the project and the manifest using a local path and rendered the DbtDag with LoadMode.DBT_MANIFEST

avg time: 2605 miliseconds
std dev: 98

In the second scenario, both the project and the manifest are fetched from S3 and the project parsed with LoadMode.DBT_MANIFEST

avg time: 3975 miliseconds
std dev: 209

So it took roughly 52% longer to parse a dbt dag that is fully stored in a cloud provider. Of course that performance would degrade for bigger repos or show better results in case of a better internet connection. Personally speaking, I do not find it a huge time increase considering we can benefit from all features developed for a local dbt project.

3. I confess I have not tried this feature in this setup, but when the dbt project is loaded with ObjectStorage airflow treats it as a local file path, so I would assume that nothing would break since the cloud path would be essentially a "local path"

4. It is possible for sure, but personally speaking I find more appealing to have the possibility to fetch the project from a cloud provider. Of course, the user should be aware that the scheduler will have to work extra hard for it to happen, but I believe it is beneficial to have as many alternatives as possible.

[CorsettiS]

I have decided to close the PR for now because I found a severe flaw when I was extending the functionality of this PR to the task execution, which is that each singular task will need to list and fetch all artifacts from the repo in s3, which is going to balloon costs in an unexpected way. I will work with it on the side and once I have a good alternative for the problem I will re-create the PR

[tatiana]

Thank you very much, @CorsettiS , for the additional information and context! We're looking forward to seeing your follow up PRs