Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support Snowflake warehouse override without model configs #1280

Open
3 tasks done
christopher-turnbull-77 opened this issue Jan 3, 2025 · 5 comments · May be fixed by #1281
Open
3 tasks done

[Feature] Support Snowflake warehouse override without model configs #1280

christopher-turnbull-77 opened this issue Jan 3, 2025 · 5 comments · May be fixed by #1281
Labels
enhancement New feature or request

Comments

@christopher-turnbull-77
Copy link

Is this your first time submitting a feature request?

  • I have read the expectations for open source contributors
  • I have searched the existing issues, and I could not find an existing issue for this feature
  • I am requesting a straightforward extension of existing dbt-snowflake functionality, rather than a Big Idea better suited to a discussion

Describe the feature

Currently, users can specify Snowflake warehouses at the model level (using config blocks) or project level (in dbt_project.yml). However, there's no way to override the warehouse at runtime without modifying these files. This feature adds support for overriding the Snowflake warehouse using dbt vars, which works for both dbt run and dbt test commands.

Example usage:

dbt run --vars '{snowflake_warehouse: LARGE_WAREHOUSE}'
dbt test --vars '{snowflake_warehouse: LARGE_WAREHOUSE}'

This provides more flexibility for users who need to dynamically change warehouses based on different scenarios (e.g., using a larger warehouse for development/testing, or a smaller one for CI/CD pipelines).

Describe alternatives you've considered

Currently, users can specify Snowflake warehouses at the model level (using config blocks) or project level (in dbt_project.yml). However, there's no way to override the warehouse at runtime without modifying these files.

Who will this benefit?

This benefits dbt-snowflake users who need to:

  • Dynamically change warehouses without modifying model or project files
  • Use different warehouses for different environments or contexts
  • Override warehouse settings in CI/CD pipelines
  • Run tests with specific warehouse configurations

Are you interested in contributing this feature?

Yes

Anything else?

This extends the existing warehouse configuration functionality in a way that's consistent with dbt's var override patterns.

@christopher-turnbull-77 christopher-turnbull-77 added enhancement New feature or request triage labels Jan 3, 2025
@christopher-turnbull christopher-turnbull linked a pull request Jan 3, 2025 that will close this issue
4 tasks
@amychen1776
Copy link

amychen1776 commented Jan 6, 2025

@christopher-turnbull-77 Curious if there's a reason why you can't use env_vars for this? Since you could need to override the warehouse at the connection level, to me environment variables feel more appropriate for this use case. Is it simply because you want to supply the input via the command line?

@amychen1776 amychen1776 removed the triage label Jan 6, 2025
@christopher-turnbull-77
Copy link
Author

Thanks for the question! The key difference is in the use cases:

env_vars are great for connection-level settings that are environment-specific (dev/prod) and shouldn't change frequently.

vars are better suited for runtime configurations that might need to change between different dbt commands or job runs.

The implementation doesn't remove the ability to use env_vars - it simply adds flexibility by providing another option. Users can choose whichever method best fits their workflow, whether that's env_vars for stable connection settings or vars for dynamic runtime configuration.

@amychen1776
Copy link

amychen1776 commented Jan 7, 2025

@christopher-turnbull-77 Just to clarify, you want to be able to apply snowflake_warehouse = var("warehouse_name") to multiple models rather than updating the model configs for each individual model file? Because you can use vars for snowflake warehouses today in the model configurations.

@christopher-turnbull-77
Copy link
Author

Actually, this PR serves a different purpose. While you can indeed specify warehouses in model configs today, this implementation allows for warehouse override via command-line vars without touching any model configurations.

This is particularly useful during development when you might want to:

  1. Quickly test a heavy query on a larger warehouse without changing any configs
  2. Switch to a smaller warehouse when iterating on model development to save costs
  3. Experiment with different warehouse sizes to optimize performance

The existing model-level configuration still works - this just adds another layer of flexibility that's especially valuable during the development process.

@amychen1776
Copy link

Makes sense - appreciate the clarification. Currently this isn't a high priority for our roadmap and I don't see us picking this up in the near future. I will leave this issue open for now to get more community feedback but wanted to set expectations.

@amychen1776 amychen1776 changed the title [Feature] Support Snowflake warehouse override via dbt vars [Feature] Support Snowflake warehouse override without model configs Jan 8, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants