Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid installing deps in Local mode - behaviour change in 1.2.4 #715

Closed
sphenlee opened this issue Nov 26, 2023 · 2 comments · Fixed by #730
Closed

Avoid installing deps in Local mode - behaviour change in 1.2.4 #715

sphenlee opened this issue Nov 26, 2023 · 2 comments · Fixed by #730
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration area:dependencies Related to dependencies, like Python packages, library versions, etc area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc dbt:deps Primarily related to dbt deps command or functionality execution:local Related to Local execution environment priority:high High priority issues are blocking or critical issues without a workaround and large impact

Comments

@sphenlee
Copy link

We're using Cosmos in a regulated environment (FedRAMP) so we aren't allowed to let dbt install dependencies at runtime. Instead we package up the project with all the dependencies in our build pipeline and then unpack the whole thing into Airflow. We run Cosmos with the Local executor mode.

The recent release 1.2.4 has broken our workflow since the dependencies are no longer being copied, and the suggested fix (to use install_deps - see #713 ) isn't allowed due to regulations.

2b91285#diff-cdc7f3cedeca298d5c3bdd8d9be8fa15791b915fbe1b2810e9c86c271eb86446L200

This diff ⬆️ changed copytree into create_symlinks which skips over the dbt_packages directory.

(I'm happy to contribute a fix to make this behavior configurable but would appreciate any guidance about how to best achieve this!)

Copy link

dosubot bot commented Nov 26, 2023

🤖

Thank you for bringing this issue to our attention. We appreciate your interest in contributing to Cosmos to address this behavior change.

To ensure that the dependencies are copied correctly in the Local executor mode, we understand that you would like to make the behavior configurable. This is a great idea!

To get started with contributing a fix, we recommend following these steps:

  1. Fork the Cosmos repository on GitHub.
  2. Clone your forked repository to your local machine.
  3. Create a new branch for your changes.
  4. Make the necessary modifications to make the behavior configurable.
  5. Test your changes locally to ensure they work as expected.
  6. Commit your changes and push them to your forked repository.
  7. Open a pull request (PR) from your branch to the main Cosmos repository.

Once you have opened the PR, the maintainers will review your proposed changes and provide feedback. They will decide whether to merge your changes into the main repository.

If you have any questions or need further guidance during the process, feel free to ask. We appreciate your willingness to contribute and look forward to seeing your proposed solution.

@dosubot dosubot bot added area:config Related to configuration, like YAML files, environment variables, or executer configuration area:dependencies Related to dependencies, like Python packages, library versions, etc area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc dbt:deps Primarily related to dbt deps command or functionality execution:local Related to Local execution environment priority:high High priority issues are blocking or critical issues without a workaround and large impact labels Nov 26, 2023
@jbandoro
Copy link
Collaborator

jbandoro commented Dec 5, 2023

@sphenlee this issue was fixed in #730, and there will be a new alpha release for v1.3 that will have the fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:config Related to configuration, like YAML files, environment variables, or executer configuration area:dependencies Related to dependencies, like Python packages, library versions, etc area:execution Related to the execution environment/mode, like Docker, Kubernetes, Local, VirtualEnv, etc dbt:deps Primarily related to dbt deps command or functionality execution:local Related to Local execution environment priority:high High priority issues are blocking or critical issues without a workaround and large impact
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants