Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stage_external_sources Comparing source_name of the node instead of the name of the node #309

Closed
1 of 3 tasks
ward-resa opened this issue Aug 1, 2024 · 2 comments · Fixed by #312
Closed
1 of 3 tasks
Labels
bug Something isn't working triage

Comments

@ward-resa
Copy link
Contributor

Describe the bug

When calling the stage_external_sources macro and passing the source name as a list of values, when the name does not contain a '.' it is comparing the node.source_name instead of the node.name for the selected source.

Steps to reproduce

In my pre-hook I am can call the macro with the select parameter with the schema.source_name and it works. If I just call it with the source_name it does not.

Expected results

Calling the macro with the schema or without the schema should still find the source by name.

Actual results

Ignores the source if only the name is used in the select parameter.

Which database are you using dbt with?

  • redshift
  • snowflake
  • other (specify: BigQuery)

The output of dbt --version:

Core:
  - installed: 1.8.4
  - latest:    1.8.4 - Up to date!

Plugins:
  - bigquery: 1.8.2 - Up to date!

The operating system you're using:

macOS 14.5

The output of python --version:

Python 3.11.9

Additional context

I believe that Line 20 in the macro stage_external_sources is the culprit. I think that the following would fix the issue. Unless I am misinterpreting the intention of that line of code.

existing code

                        {% if src == node.source_name %}

proposed change

                        {% if src == node.name %}
@laurianneol
Copy link

Hi,

I've encountered an issue with the stage_external_sources macro in the latest release. The problem arises from this change. Because of this change, I'm unable to run the command dbt run-operation stage_external_sources --args '{"select": "source_name"}'. With this command line, I was previously able to drop and create all the tables from my source (using ext_full_refresh = true).

Could you advise on how I can run all tables using the current setup?

Locally, I added an argument to the macro to use tags that I had assigned to my sources, and it works perfectly. Could you please look into this issue and consider incorporating a similar solution?

I'm new to GitHub, so any guidance or help would be greatly appreciated!

Thanks!

@bullamiders
Copy link

Hi,
I observed the same issue as laurianneol.
This change broke the selection of all tables within a source like described in the syntax section of the readme: https://github.com/dbt-labs/dbt-external-tables?tab=readme-ov-file#syntax

# stage all Snowplow and Logs external sources:
$ dbt run-operation stage_external_sources --args "select: snowplow logs"

If I find time, I'll open a bug.

Best regards

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
3 participants