-
Notifications
You must be signed in to change notification settings - Fork 161
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Refactor dbt ls to run from tempdir with symbolic links (#488)
Since Cosmos 1.0, `load_method.DBT_LS` is the default dbt project parsing method, unless the user gives a manifest. Using the original dbt project path has been a source of issues when that path is Read-Only. This issue was faced when running commands that generate `{project-dir}/target/` and `{project-dir}/logs/`, which was solved as part of #414. This issue is particularly problematic if we want to run `dbt deps` from the original project directory since dbt 1.6 saves adaptors to `{project_dir}/dbt_packages` unless specified in the user's `dbt_project.yml`. To our knowledge, dbt currently does not allow users to define this directory via flags or environment variables, as discussed in #481. This change aims to solve these issues, by creating a temporary directory and creating symbolic links to the original directory. Finally, during the development of this task, it was observed that when running dbt ls in a project with `packages.yml`, dbt raises a 'Compilation Error'. Since dbt may raise other errors in stdout, this PR captures "Errors" more generically - making it more evident potential issues to the end-users.
- Loading branch information
Showing
4 changed files
with
42 additions
and
18 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters