Skip to content

Commit

Permalink
Handle BigQuery non-string option 'max_staleness' (#237)
Browse files Browse the repository at this point in the history
* Handle BigQuery non-string option 'max_staleness'

* Integration test for max_staleness BigQuery option
  • Loading branch information
marcbllv authored Mar 11, 2024
1 parent 5f79eea commit 383ee3c
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 1 deletion.
10 changes: 10 additions & 0 deletions integration_tests/models/plugins/bigquery/bigquery_external.yml
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,16 @@ sources:
columns: *cols-of-the-people
tests: *equal-to-the-people

- name: people_csv_with_max_staleness
external:
location: 'gs://dbt-external-tables-testing/csv/*'
options:
format: csv
skip_leading_rows: 1
max_staleness: INTERVAL 1 HOUR
columns: *cols-of-the-people
tests: *equal-to-the-people

# - name: people_json_unpartitioned
# external: &json-people
# location: 'gs://dbt-external-tables-testing/json/*'
Expand Down
3 changes: 2 additions & 1 deletion macros/plugins/bigquery/create_external_table.sql
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
{%- set external = source_node.external -%}
{%- set partitions = external.partitions -%}
{%- set options = external.options -%}
{%- set non_string_options = ['max_staleness'] %}

{% if options is mapping and options.get('connection_name', none) %}
{% set connection_name = options.pop('connection_name') %}
Expand Down Expand Up @@ -38,7 +39,7 @@
uris = [{%- for uri in uris -%} '{{uri}}' {{- "," if not loop.last}} {%- endfor -%}]
{%- if options is mapping -%}
{%- for key, value in options.items() if key != 'uris' %}
{%- if value is string -%}
{%- if value is string and key not in non_string_options -%}
, {{key}} = '{{value}}'
{%- else -%}
, {{key}} = {{value}}
Expand Down

0 comments on commit 383ee3c

Please sign in to comment.