Support to cache profiles created via ProfileMapping

[pankajastro]

Description

Add DBT profile caching mechanism.

1. Introduced env enable_cache_profile to enable or disable profile caching. This will be enabled only if global enable_cache is enabled.
2. Users can set the env profile_cache_dir_name. This will be the name of a sub-dir inside cache_dir where cached profiles will be stored. This is optional, and the default name is profile
3. Example Path for versioned profile: {cache_dir}/{profile_cache_dir}/592906f650558ce1dadb75fcce84a2ec09e444441e6af6069f19204d59fe428b/profiles.yml
4. Implemented profile mapping hashing: first, the profile is serialized using pickle. Then, the profile_name and target_name are appended before hashing the data using the SHA-256 algorithm

Perf test result:
In local dev env with command

AIRFLOW_HOME=`pwd` AIRFLOW_CONN_EXAMPLE_CONN="postgres://postgres:[email protected]:5432/postgres"  AIRFLOW_HOME=`pwd` AIRFLOW__CORE__DAGBAG_IMPORT_TIMEOUT=20000 AIRFLOW__CORE__DAG_FILE_PROCESSOR_TIMEOUT=20000 hatch run tests.py3.10-2.8:test-performance

NUM_MODELS=100

• TIME=167.45248413085938 (with profile cache enabled)
• TIME=173.94845390319824 (with profile cache disabled)

NUM_MODELS=200

• TIME=376.2585120201111 (with profile cache enabled)
• TIME=418.14210200309753 (with profile cache disabled)

Related Issue(s)

Closes: #925
Closes: #647

Breaking Change?

No

Checklist

• I have made corresponding changes to the documentation (if required)
• I have added tests that prove my fix is effective or that my feature works

[netlify]

✅ Deploy Preview for sunny-pastelito-5ecb04 ready!

Name	Link
🔨 Latest commit	865b760
🔍 Latest deploy log	https://app.netlify.com/sites/sunny-pastelito-5ecb04/deploys/667d69a74ecd530008ced7ec
😎 Deploy Preview	https://deploy-preview-1046--sunny-pastelito-5ecb04.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[tatiana]

Hi @pankajastro , thanks a lot for advancing on this!

Some questions/comments on the overall feature:

1. So we are consistent with other enabling cache mechanism (e.g. Speed up LoadMode.DBT_LS by caching dbt ls output in Airflow Variable #1014), please, rename the config to enable_cache_profile
2. Why are we forcing users to defined the cache path, differently from Improve performance by 22-35% or more by caching partial parse artefact #904, where we dynamically define a path - and we give the possibility for users to override it with the cache_dir config, if they want to. We could perhaps allow users to force a specific profile path if they want to, by having cache_dir_profile - but it would be great if this was optional.
3. What does the metadata.json relate to? A dbt profile, an Airflow connection, a DbtDag, or to a DbtTaskGroup, to the Airflow deployment..? The name is very generic and the scope is unclear. How are we handling scenarios where concurrent tasks/DAGs would be writing at the same time to this file?
4. How sensitive is the identifier of the cache (sha256) to changes in the Airflow connection being used?

[pankajastro]

How sensitive is the identifier of the cache (sha256) to changes in the Airflow connection being used?

hashing connection is still a TODO item and we need to pass the actual value of a connection in the hashing input

[pankajastro]

What does the metadata.json relate to? A dbt profile, an Airflow connection, a DbtDag, or to a DbtTaskGroup, to the Airflow deployment..?

A JSON file that stores key-value pairs, where each key is a hash for a Profile Mapping object and the value is the corresponding path to the DBT profile YAML file. I'm using this file to check if we already have profile YAML for this profile object hash

The name is very generic and the scope is unclear.

I'm. ok to change the name.

How are we handling scenarios where concurrent tasks/DAGs would be writing at the same time to this file?

I haven't considered the concurrent case, do we need locking here?

[pankajastro]

Why are we forcing users to defined the cache path, differently from #904, where we dynamically define a path - and we give the possibility for users to override it with the cache_dir config, if they want to. We could perhaps allow users to force a specific profile path if they want to, by having cache_dir_profile - but it would be great if this was optional.

We can make it optional, I wanted to keep the profile caching-related files in one place. Shall we keep the profile related files in cache_dir / profile ?

[pankajastro]

Improved the below items after our first discussion and review

• Added environment variable enable_profile_cache to toggle profile caching on or off.
• Introduced environment variable profile_cache_dir, which will be created within cache_dir. The default name is 'profile', and it will store the versioned DBT profile.
• Example Path for versioned provifile: {cache_dir}/{profile_cache_dir}/version/profiles.yml
• Implemented profile mapping hashing: first, the profile is serialized using pickle. Then, the profile_name and target_name are appended before hashing the data using the SHA-256 algorithm
• Removed metadata.json because now I'm not dealing with desired_profile_path variable
• Added a check to enable_profile_cache only if global caching is enabled

[tatiana]

HI @pankajastro thanks for addressing all the feedback!

Once you are able to uncomment that assert in the test and rename the config, I believe we're good to go! 🎉

All the comments/feedback were addressed. Following up on:

I haven't considered the concurrent case, do we need locking here?

I believe once we removed the metadata.json file, we're in a much safer position, since we reduce significantly concurrent writes to the same file. I suggest we go with the current solution and we review if users report any issues.

[tatiana]

Looks great, thanks for addressing all the feedback, @pankajastro !