From b2dd470d3d75fd9c91a931d447bbc67ab68b9aae Mon Sep 17 00:00:00 2001 From: Edward Gao Date: Tue, 26 Jul 2022 16:40:14 -0700 Subject: [PATCH] Handle ints and longs in normalization (#14362) * generate airbyte_type:integer * normalization accepts `airbyte_type: integer` * handles ints+longs * update avro for consistency * delete long type for now, treat all ints as longs * update avro type mappings {type:number, airbyte_type:integer} -> long {type:number, airbyte_type:big_integer} -> string (i.e. "unbounded integer") * fix test * remove long handling * Revert "remove long handling" This reverts commit 33ade8d2831e675c3545ac6019d200ec312e54d9. * Revert "update avro type mappings" This reverts commit 5b0349badad7545efe8e1191291a628445fe1c84. * Revert "delete long type for now, treat all ints as longs" This reverts commit 018efd4a5d0c59f392fd8e3b0d0967c666b72947. * Revert "update avro for consistency" This reverts commit bcf47c6799b5906deb4f219d7f6e64ea73b41b74. * newline@eof * update test * slightly better local tests * fix test * missed a few cases * postgres tests use correct hostnames * fix normalization * fix int macro * add test case * normalization test output * handle int/long correctly * fix types for other DBs * uint32 -> bigint; tests * add type value assertions * more test updates * regenerate output * reconcile big_integer to match docs * update comment * fix type * fix mysql constructor call * bigint only has 38 digits * fix s3 ints, fix DAT test case * big_integer should be string * reduce to 28 digit big_ints * fix test setup, mysql * kill big_integer tests * regenerate output * version bumps * auto-bump connector version [ci skip] * auto-bump connector version [ci skip] * auto-bump connector version [ci skip] * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III --- .../resources/seed/source_definitions.yaml | 8 +- .../src/main/resources/seed/source_specs.yaml | 8 +- .../db/bigquery/BigQuerySourceOperations.java | 3 +- .../airbyte/db/jdbc/JdbcSourceOperations.java | 8 +- .../io/airbyte/db/jdbc/TestJdbcUtils.java | 6 +- .../bases/base-normalization/Dockerfile | 4 +- .../macros/cross_db_utils/datatypes.sql | 46 +++++ .../test_simple_streams/first_dbt_project.yml | 5 + .../models/generated/sources.yml | 1 + .../test_simple_streams/dbt_project.yml | 31 ++-- .../models/generated/sources.yml | 1 + .../mysql/test_nested_streams/dbt_project.yml | 8 + .../models/generated/sources.yml | 1 + .../mysql/test_simple_streams/dbt_project.yml | 5 + .../test_normalization/exchange_rate.sql | 44 ++--- .../models/generated/sources.yml | 1 + .../test_normalization/exchange_rate.sql | 44 ++--- .../test_simple_streams/dbt_project.yml | 25 +-- .../models/generated/sources.yml | 1 + .../test_simple_streams/dbt_project.yml | 26 +-- .../test_simple_streams/first_dbt_project.yml | 31 ++-- .../test_normalization/types_testing_scd.sql | 73 ++++++++ .../test_normalization/types_testing.sql | 24 +++ .../test_normalization/types_testing_stg.sql | 53 ++++++ .../test_normalization/types_testing_ab1.sql | 20 +++ .../test_normalization/types_testing_ab2.sql | 20 +++ .../test_normalization/types_testing_scd.sql | 163 ++++++++++++++++++ .../test_normalization/types_testing.sql | 23 +++ .../test_normalization/types_testing_stg.sql | 20 +++ .../models/generated/sources.yml | 1 + .../modified_models/generated/sources.yml | 20 +-- .../test_normalization/types_testing_scd.sql | 15 ++ .../test_normalization/types_testing.sql | 15 ++ .../test_normalization/types_testing_stg.sql | 15 ++ .../test_nested_streams/dbt_project.yml | 158 +++++++++-------- ..._columns_resulting_into_long_names_scd.sql | 6 +- ...plex_columns_resulting_into_long_names.sql | 8 +- ...ns_resulting_into_long_names_partition.sql | 8 +- ...sulting_into_long_names_partition_data.sql | 10 +- ...long_names_partition_double_array_data.sql | 10 +- ..._columns_resulting_into_long_names_ab1.sql | 6 +- ..._columns_resulting_into_long_names_ab2.sql | 2 +- ...esulting_into_long_names_partition_ab1.sql | 2 +- ...ing_into_long_names_partition_data_ab1.sql | 2 +- ..._names_partition_double_array_data_ab1.sql | 2 +- ..._columns_resulting_into_long_names_scd.sql | 8 +- ...plex_columns_resulting_into_long_names.sql | 4 +- ...ns_resulting_into_long_names_partition.sql | 2 +- ...sulting_into_long_names_partition_data.sql | 2 +- ...long_names_partition_double_array_data.sql | 2 +- .../models/generated/sources.yml | 17 +- ..._columns_resulting_into_long_names_scd.sql | 4 +- ...plex_columns_resulting_into_long_names.sql | 4 +- ...ns_resulting_into_long_names_partition.sql | 2 +- ...sulting_into_long_names_partition_data.sql | 2 +- ...long_names_partition_double_array_data.sql | 2 +- .../test_simple_streams/dbt_project.yml | 64 +++---- .../test_simple_streams/first_dbt_project.yml | 99 ++++++----- .../dedup_exchange_rate_scd.sql | 6 +- .../dedup_exchange_rate.sql | 8 +- .../test_normalization/exchange_rate.sql | 8 +- .../dedup_exchange_rate_stg.sql | 6 +- .../multiple_column_names_conflicts_stg.sql | 6 +- .../dedup_exchange_rate_ab1.sql | 6 +- .../dedup_exchange_rate_ab2.sql | 2 +- .../dedup_exchange_rate_scd.sql | 8 +- .../dedup_exchange_rate.sql | 4 +- .../test_normalization/exchange_rate.sql | 4 +- .../dedup_exchange_rate_stg.sql | 2 +- .../models/generated/sources.yml | 3 +- .../dedup_exchange_rate_ab1.sql | 6 +- .../dedup_exchange_rate_ab2.sql | 2 +- .../dedup_exchange_rate_scd.sql | 8 +- .../dedup_exchange_rate.sql | 4 +- .../test_normalization/exchange_rate.sql | 4 +- .../dedup_exchange_rate_stg.sql | 2 +- .../modified_models/generated/sources.yml | 20 +-- .../dedup_exchange_rate_scd.sql | 4 +- .../dedup_exchange_rate.sql | 4 +- .../test_normalization/exchange_rate.sql | 8 +- .../dedup_exchange_rate_stg.sql | 6 +- .../dedup_exchange_rate_scd.sql | 4 +- .../dedup_exchange_rate.sql | 4 +- .../test_normalization/exchange_rate.sql | 8 +- .../dedup_exchange_rate_stg.sql | 6 +- .../test_nested_streams/dbt_project.yml | 34 ++-- .../models/generated/sources.yml | 1 + .../test_simple_streams/dbt_project.yml | 31 ++-- .../models/generated/sources.yml | 1 + .../data_input/catalog.json | 25 +++ .../data_input/messages.txt | 11 ++ .../data_input/replace_identifiers.json | 3 + .../test_types_testing_values.sql | 1 + .../simple_streams_first_run_row_counts.sql | 4 + .../types_testing_incorrect_values.sql | 34 ++++ .../transform_catalog/stream_processor.py | 17 +- .../normalization/transform_catalog/utils.py | 31 +++- .../integrations/debezium/CdcSourceTest.java | 16 +- .../data_type_array_object_test_catalog.json | 4 - .../data_type_array_object_test_messages.txt | 4 +- .../data_type_array_test_catalog.json | 7 - .../data_type_array_test_messages.txt | 4 +- .../data_type_basic_test_catalog.json | 11 -- .../data_type_basic_test_messages.txt | 5 +- .../data_type_object_test_catalog.json | 4 - .../data_type_object_test_messages.txt | 4 +- .../number_data_type_array_test_catalog.json | 7 - .../number_data_type_array_test_messages.txt | 4 +- .../number_data_type_test_catalog.json | 11 -- .../number_data_type_test_messages.txt | 5 +- .../AbstractSourceDatabaseTypeTest.java | 2 +- .../destination/s3/avro/JsonSchemaType.java | 2 +- .../s3/avro/JsonToAvroSchemaConverter.java | 2 +- .../s3/avro/JsonSchemaTypeTest.java | 2 +- .../type_conversion_test_cases.json | 20 +-- .../connectors/source-bigquery/Dockerfile | 2 +- .../bigquery/BigQuerySourceDatatypeTest.java | 2 +- .../jdbc/test/JdbcSourceAcceptanceTest.java | 4 +- .../source-mysql-strict-encrypt/Dockerfile | 2 +- .../connectors/source-mysql/Dockerfile | 2 +- .../source/mysql/MySqlSourceOperations.java | 3 +- .../AbstractMySqlSourceDatatypeTest.java | 22 +-- .../source/mysql/CdcMysqlSourceTest.java | 4 +- .../source-postgres-strict-encrypt/Dockerfile | 2 +- .../connectors/source-postgres/Dockerfile | 2 +- .../postgres/PostgresSourceOperations.java | 5 +- ...stractSshPostgresSourceAcceptanceTest.java | 12 +- .../CdcPostgresSourceAcceptanceTest.java | 25 +-- .../CdcPostgresSourceDatatypeTest.java | 17 +- .../sources/PostgresSourceAcceptanceTest.java | 10 +- .../sources/PostgresSourceDatatypeTest.java | 21 +-- .../PostgresJdbcSourceAcceptanceTest.java | 36 ++-- .../connectors/source-tidb/Dockerfile | 2 +- .../source/tidb/TiDBSourceOperations.java | 3 +- .../source/tidb/TiDBSourceAcceptanceTest.java | 4 +- .../protocol/models/JsonSchemaType.java | 4 +- .../NormalizationRunnerFactory.java | 2 +- docs/integrations/sources/bigquery.md | 1 + docs/integrations/sources/mysql.md | 1 + docs/integrations/sources/postgres.md | 1 + docs/integrations/sources/tidb.md | 1 + .../basic-normalization.md | 3 +- .../supported-data-types.md | 4 +- 143 files changed, 1242 insertions(+), 610 deletions(-) create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/scd/test_normalization/types_testing_scd.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/test_normalization/types_testing.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/test_normalization/types_testing_stg.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_ctes/test_normalization/types_testing_ab1.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_ctes/test_normalization/types_testing_ab2.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/types_testing_scd.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/test_normalization/types_testing.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/test_normalization/types_testing_stg.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/scd/test_normalization/types_testing_scd.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/test_normalization/types_testing.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/test_normalization/types_testing_stg.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests/test_types_testing_values.sql create mode 100644 airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests_tmp/types_testing_incorrect_values.sql diff --git a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml index bc7dc86772f8..9cb1629b45d4 100644 --- a/airbyte-config/init/src/main/resources/seed/source_definitions.yaml +++ b/airbyte-config/init/src/main/resources/seed/source_definitions.yaml @@ -96,7 +96,7 @@ - name: BigQuery sourceDefinitionId: bfd1ddf8-ae8a-4620-b1d7-55597d2ba08c dockerRepository: airbyte/source-bigquery - dockerImageTag: 0.1.9 + dockerImageTag: 0.2.0 documentationUrl: https://docs.airbyte.io/integrations/sources/bigquery icon: bigquery.svg sourceType: database @@ -604,7 +604,7 @@ - name: MySQL sourceDefinitionId: 435bb9a5-7887-4809-aa58-28c27df0d7ad dockerRepository: airbyte/source-mysql - dockerImageTag: 0.5.17 + dockerImageTag: 0.6.0 documentationUrl: https://docs.airbyte.io/integrations/sources/mysql icon: mysql.svg sourceType: database @@ -754,7 +754,7 @@ - name: Postgres sourceDefinitionId: decd338e-5647-4c0b-adf4-da0e75f5a750 dockerRepository: airbyte/source-postgres - dockerImageTag: 0.4.37 + dockerImageTag: 0.4.38 documentationUrl: https://docs.airbyte.io/integrations/sources/postgres icon: postgresql.svg sourceType: database @@ -961,7 +961,7 @@ - name: TiDB sourceDefinitionId: 0dad1a35-ccf8-4d03-b73e-6788c00b13ae dockerRepository: airbyte/source-tidb - dockerImageTag: 0.1.5 + dockerImageTag: 0.2.0 documentationUrl: https://docs.airbyte.io/integrations/sources/tidb icon: tidb.svg sourceType: database diff --git a/airbyte-config/init/src/main/resources/seed/source_specs.yaml b/airbyte-config/init/src/main/resources/seed/source_specs.yaml index c2a780f12a21..4c0d45ff8a79 100644 --- a/airbyte-config/init/src/main/resources/seed/source_specs.yaml +++ b/airbyte-config/init/src/main/resources/seed/source_specs.yaml @@ -829,7 +829,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-bigquery:0.1.9" +- dockerImage: "airbyte/source-bigquery:0.2.0" spec: documentationUrl: "https://docs.airbyte.io/integrations/sources/bigquery" connectionSpecification: @@ -5755,7 +5755,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-mysql:0.5.17" +- dockerImage: "airbyte/source-mysql:0.6.0" spec: documentationUrl: "https://docs.airbyte.io/integrations/sources/mysql" connectionSpecification: @@ -7040,7 +7040,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-postgres:0.4.37" +- dockerImage: "airbyte/source-postgres:0.4.38" spec: documentationUrl: "https://docs.airbyte.com/integrations/sources/postgres" connectionSpecification: @@ -9343,7 +9343,7 @@ supportsNormalization: false supportsDBT: false supported_destination_sync_modes: [] -- dockerImage: "airbyte/source-tidb:0.1.5" +- dockerImage: "airbyte/source-tidb:0.2.0" spec: documentationUrl: "https://docs.airbyte.io/integrations/sources/tidb" connectionSpecification: diff --git a/airbyte-db/db-lib/src/main/java/io/airbyte/db/bigquery/BigQuerySourceOperations.java b/airbyte-db/db-lib/src/main/java/io/airbyte/db/bigquery/BigQuerySourceOperations.java index fdeab84ac1b4..af8a4d1c923f 100644 --- a/airbyte-db/db-lib/src/main/java/io/airbyte/db/bigquery/BigQuerySourceOperations.java +++ b/airbyte-db/db-lib/src/main/java/io/airbyte/db/bigquery/BigQuerySourceOperations.java @@ -119,7 +119,8 @@ public Date getDateValue(final FieldValue fieldValue, final DateFormat dateForma public JsonSchemaType getJsonType(final StandardSQLTypeName bigQueryType) { return switch (bigQueryType) { case BOOL -> JsonSchemaType.BOOLEAN; - case INT64, FLOAT64, NUMERIC, BIGNUMERIC -> JsonSchemaType.NUMBER; + case INT64 -> JsonSchemaType.INTEGER; + case FLOAT64, NUMERIC, BIGNUMERIC -> JsonSchemaType.NUMBER; case STRING, BYTES, TIMESTAMP, DATE, TIME, DATETIME -> JsonSchemaType.STRING; case ARRAY -> JsonSchemaType.ARRAY; case STRUCT -> JsonSchemaType.OBJECT; diff --git a/airbyte-db/db-lib/src/main/java/io/airbyte/db/jdbc/JdbcSourceOperations.java b/airbyte-db/db-lib/src/main/java/io/airbyte/db/jdbc/JdbcSourceOperations.java index ffd584a7c06c..35269a391da0 100644 --- a/airbyte-db/db-lib/src/main/java/io/airbyte/db/jdbc/JdbcSourceOperations.java +++ b/airbyte-db/db-lib/src/main/java/io/airbyte/db/jdbc/JdbcSourceOperations.java @@ -102,12 +102,12 @@ public JDBCType getFieldType(final JsonNode field) { } @Override - public JsonSchemaType getJsonType(JDBCType jdbcType) { + public JsonSchemaType getJsonType(final JDBCType jdbcType) { return switch (jdbcType) { case BIT, BOOLEAN -> JsonSchemaType.BOOLEAN; - case TINYINT, SMALLINT -> JsonSchemaType.NUMBER; - case INTEGER -> JsonSchemaType.NUMBER; - case BIGINT -> JsonSchemaType.NUMBER; + case TINYINT, SMALLINT -> JsonSchemaType.INTEGER; + case INTEGER -> JsonSchemaType.INTEGER; + case BIGINT -> JsonSchemaType.INTEGER; case FLOAT, DOUBLE -> JsonSchemaType.NUMBER; case REAL -> JsonSchemaType.NUMBER; case NUMERIC, DECIMAL -> JsonSchemaType.NUMBER; diff --git a/airbyte-db/db-lib/src/test/java/io/airbyte/db/jdbc/TestJdbcUtils.java b/airbyte-db/db-lib/src/test/java/io/airbyte/db/jdbc/TestJdbcUtils.java index 42a923fa1f2c..82f1151dab9b 100644 --- a/airbyte-db/db-lib/src/test/java/io/airbyte/db/jdbc/TestJdbcUtils.java +++ b/airbyte-db/db-lib/src/test/java/io/airbyte/db/jdbc/TestJdbcUtils.java @@ -301,9 +301,9 @@ private static void assertExpectedOutputTypes(final Connection connection) throw final Map expected = ImmutableMap.builder() .put("bit", JsonSchemaType.BOOLEAN) .put("boolean", JsonSchemaType.BOOLEAN) - .put("smallint", JsonSchemaType.NUMBER) - .put("int", JsonSchemaType.NUMBER) - .put("bigint", JsonSchemaType.NUMBER) + .put("smallint", JsonSchemaType.INTEGER) + .put("int", JsonSchemaType.INTEGER) + .put("bigint", JsonSchemaType.INTEGER) .put("float", JsonSchemaType.NUMBER) .put("double", JsonSchemaType.NUMBER) .put("real", JsonSchemaType.NUMBER) diff --git a/airbyte-integrations/bases/base-normalization/Dockerfile b/airbyte-integrations/bases/base-normalization/Dockerfile index 666808999510..0002e312e0e4 100644 --- a/airbyte-integrations/bases/base-normalization/Dockerfile +++ b/airbyte-integrations/bases/base-normalization/Dockerfile @@ -28,5 +28,5 @@ WORKDIR /airbyte ENV AIRBYTE_ENTRYPOINT "/airbyte/entrypoint.sh" ENTRYPOINT ["/airbyte/entrypoint.sh"] -LABEL io.airbyte.version=0.2.11 -LABEL io.airbyte.name=airbyte/normalization \ No newline at end of file +LABEL io.airbyte.version=0.2.12 +LABEL io.airbyte.name=airbyte/normalization diff --git a/airbyte-integrations/bases/base-normalization/dbt-project-template/macros/cross_db_utils/datatypes.sql b/airbyte-integrations/bases/base-normalization/dbt-project-template/macros/cross_db_utils/datatypes.sql index 762477b24b0d..923ec66fe4f1 100755 --- a/airbyte-integrations/bases/base-normalization/dbt-project-template/macros/cross_db_utils/datatypes.sql +++ b/airbyte-integrations/bases/base-normalization/dbt-project-template/macros/cross_db_utils/datatypes.sql @@ -80,6 +80,10 @@ {# int ------------------------------------------------- #} {% macro default__type_int() %} + int +{% endmacro %} + +{% macro mysql__type_int() %} signed {% endmacro %} @@ -116,6 +120,48 @@ {% endmacro %} +{# very_large_integer --------------------------------------- --#} +{# +Most databases don't have a true unbounded numeric datatype, so we use a really big numeric field. +Our type terminology unfortunately collides with DB terminology (i.e. "big_integer" means different things in different contexts) +so this macro needs to be called very_large_integer. +#} +{%- macro type_very_large_integer() -%} + {{ adapter.dispatch('type_very_large_integer')() }} +{%- endmacro -%} + +{% macro default__type_very_large_integer() %} + numeric +{% endmacro %} + +{% macro snowflake__type_very_large_integer() %} + numeric +{% endmacro %} + +{% macro mysql__type_very_large_integer() %} + decimal(38, 0) +{% endmacro %} + +{% macro clickhouse__type_very_large_integer() %} + decimal128(0) +{% endmacro %} + +{# timestamp ------------------------------------------------- --#} +{% macro mysql__type_timestamp() %} + time +{% endmacro %} + +{%- macro sqlserver__type_timestamp() -%} + {#-- in TSQL timestamp is really datetime --#} + {#-- https://docs.microsoft.com/en-us/sql/t-sql/functions/date-and-time-data-types-and-functions-transact-sql?view=sql-server-ver15#DateandTimeDataTypes --#} + datetime +{%- endmacro -%} + +{% macro clickhouse__type_timestamp() %} + DateTime64 +{% endmacro %} + + {# timestamp with time zone ------------------------------------------------- #} {%- macro type_timestamp_with_timezone() -%} diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/first_dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/first_dbt_project.yml index bd8ede6a646f..12745c37a150 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/first_dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/first_dbt_project.yml @@ -83,3 +83,8 @@ vars: multiple_column_names_conflicts_stg: test_normalization._airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts_scd: test_normalization._airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts: test_normalization._airbyte_raw_multiple_column_names_conflicts + types_testing_ab1: test_normalization._airbyte_raw_types_testing + types_testing_ab2: test_normalization._airbyte_raw_types_testing + types_testing_stg: test_normalization._airbyte_raw_types_testing + types_testing_scd: test_normalization._airbyte_raw_types_testing + types_testing: test_normalization._airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/models/generated/sources.yml index 97bf0d05cbd4..f51802427655 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/models/generated/sources.yml @@ -13,3 +13,4 @@ sources: - name: _airbyte_raw_multiple_column_names_conflicts - name: _airbyte_raw_pos_dedup_cdcx - name: _airbyte_raw_renamed_dedup_cdc_excluded + - name: _airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/dbt_project.yml index 02cf2fd559fd..5d86e8cce453 100755 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/dbt_project.yml @@ -1,25 +1,25 @@ name: airbyte_utils -version: "1.0" +version: '1.0' config-version: 2 profile: normalize model-paths: - - models +- models docs-paths: - - docs +- docs analysis-paths: - - analysis +- analysis test-paths: - - tests +- tests seed-paths: - - data +- data macro-paths: - - macros +- macros target-path: ../build log-path: ../logs packages-install-path: /dbt clean-targets: - - build - - dbt_modules +- build +- dbt_modules quoting: database: true schema: false @@ -42,10 +42,10 @@ models: +tags: airbyte_internal_views +materialized: view dispatch: - - macro_namespace: dbt_utils - search_order: - - airbyte_utils - - dbt_utils +- macro_namespace: dbt_utils + search_order: + - airbyte_utils + - dbt_utils vars: json_column: _airbyte_data models_to_source: @@ -83,3 +83,8 @@ vars: multiple_column_names_conflicts_stg: test_normalization._airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts_scd: test_normalization._airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts: test_normalization._airbyte_raw_multiple_column_names_conflicts + types_testing_ab1: test_normalization._airbyte_raw_types_testing + types_testing_ab2: test_normalization._airbyte_raw_types_testing + types_testing_stg: test_normalization._airbyte_raw_types_testing + types_testing_scd: test_normalization._airbyte_raw_types_testing + types_testing: test_normalization._airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/models/generated/sources.yml index 97bf0d05cbd4..f51802427655 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/models/generated/sources.yml @@ -13,3 +13,4 @@ sources: - name: _airbyte_raw_multiple_column_names_conflicts - name: _airbyte_raw_pos_dedup_cdcx - name: _airbyte_raw_renamed_dedup_cdc_excluded + - name: _airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_nested_streams/dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_nested_streams/dbt_project.yml index 88b2a40e2d11..f187620c7c7c 100755 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_nested_streams/dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_nested_streams/dbt_project.yml @@ -79,6 +79,10 @@ vars: unnest_alias_ab2: test_normalization._airbyte_raw_unnest_alias unnest_alias_ab3: test_normalization._airbyte_raw_unnest_alias unnest_alias: test_normalization._airbyte_raw_unnest_alias + arrays_ab1: test_normalization._airbyte_raw_arrays + arrays_ab2: test_normalization._airbyte_raw_arrays + arrays_ab3: test_normalization._airbyte_raw_arrays + arrays: test_normalization._airbyte_raw_arrays nested_stream_with_co_2g_names_partition_ab1: test_normalization._airbyte_raw_nested_s__lting_into_long_names nested_stream_with_co_2g_names_partition_ab2: test_normalization._airbyte_raw_nested_s__lting_into_long_names nested_stream_with_co_2g_names_partition_ab3: test_normalization._airbyte_raw_nested_s__lting_into_long_names @@ -91,6 +95,10 @@ vars: unnest_alias_children_ab2: test_normalization._airbyte_raw_unnest_alias unnest_alias_children_ab3: test_normalization._airbyte_raw_unnest_alias unnest_alias_children: test_normalization._airbyte_raw_unnest_alias + arrays_nested_array_parent_ab1: test_normalization._airbyte_raw_arrays + arrays_nested_array_parent_ab2: test_normalization._airbyte_raw_arrays + arrays_nested_array_parent_ab3: test_normalization._airbyte_raw_arrays + arrays_nested_array_parent: test_normalization._airbyte_raw_arrays nested_stream_with_co_3double_array_data_ab1: test_normalization._airbyte_raw_nested_s__lting_into_long_names nested_stream_with_co_3double_array_data_ab2: test_normalization._airbyte_raw_nested_s__lting_into_long_names nested_stream_with_co_3double_array_data_ab3: test_normalization._airbyte_raw_nested_s__lting_into_long_names diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_nested_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_nested_streams/models/generated/sources.yml index e34e562ac3fa..50def309c8c4 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_nested_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_nested_streams/models/generated/sources.yml @@ -6,6 +6,7 @@ sources: schema: false identifier: false tables: + - name: _airbyte_raw_arrays - name: _airbyte_raw_conflict_stream_array - name: _airbyte_raw_conflict_stream_name - name: _airbyte_raw_conflict_stream_scalar diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/dbt_project.yml index 5650b278a87c..bc7fa6d50166 100755 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/dbt_project.yml @@ -79,3 +79,8 @@ vars: multiple_column_names_conflicts_stg: test_normalization._airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts_scd: test_normalization._airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts: test_normalization._airbyte_raw_multiple_column_names_conflicts + types_testing_ab1: test_normalization._airbyte_raw_types_testing + types_testing_ab2: test_normalization._airbyte_raw_types_testing + types_testing_stg: test_normalization._airbyte_raw_types_testing + types_testing_scd: test_normalization._airbyte_raw_types_testing + types_testing: test_normalization._airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/first_output/airbyte_tables/test_normalization/exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/first_output/airbyte_tables/test_normalization/exchange_rate.sql index f7848e246cf1..540fc0e7911f 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/first_output/airbyte_tables/test_normalization/exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/first_output/airbyte_tables/test_normalization/exchange_rate.sql @@ -3,41 +3,41 @@ create table test_normalization.`exchange_rate__dbt_tmp` as ( - + with __dbt__cte__exchange_rate_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema -- depends_on: test_normalization._airbyte_raw_exchange_rate select - json_value(_airbyte_data, + json_value(_airbyte_data, '$."id"' RETURNING CHAR) as id, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."currency"' RETURNING CHAR) as currency, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."date"' RETURNING CHAR) as `date`, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."timestamp_col"' RETURNING CHAR) as timestamp_col, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."HKD@spéçiäl & characters"' RETURNING CHAR) as `HKD@spéçiäl & characters`, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."HKD_special___characters"' RETURNING CHAR) as hkd_special___characters, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."NZD"' RETURNING CHAR) as nzd, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."USD"' RETURNING CHAR) as usd, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."column___with__quotes"' RETURNING CHAR) as `column__'with"_quotes`, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."datetime_tz"' RETURNING CHAR) as datetime_tz, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."datetime_no_tz"' RETURNING CHAR) as datetime_no_tz, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."time_tz"' RETURNING CHAR) as time_tz, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."time_no_tz"' RETURNING CHAR) as time_no_tz, _airbyte_ab_id, _airbyte_emitted_at, - + CURRENT_TIMESTAMP as _airbyte_normalized_at from test_normalization._airbyte_raw_exchange_rate as table_alias @@ -48,7 +48,7 @@ where 1 = 1 -- SQL model to cast each column to its adequate SQL type converted from the JSON schema type -- depends_on: __dbt__cte__exchange_rate_ab1 select - cast(id as + cast(id as signed ) as id, cast(currency as char(1024)) as currency, @@ -57,14 +57,14 @@ select end as `date` , cast(nullif(timestamp_col, '') as char(1024)) as timestamp_col, - cast(`HKD@spéçiäl & characters` as + cast(`HKD@spéçiäl & characters` as float ) as `HKD@spéçiäl & characters`, cast(hkd_special___characters as char(1024)) as hkd_special___characters, - cast(nzd as + cast(nzd as float ) as nzd, - cast(usd as + cast(usd as float ) as usd, cast(`column__'with"_quotes` as char(1024)) as `column__'with"_quotes`, @@ -74,12 +74,12 @@ select end as datetime_no_tz , nullif(cast(time_tz as char(1024)), "") as time_tz, - nullif(cast(time_no_tz as + nullif(cast(time_no_tz as time ), "") as time_no_tz, _airbyte_ab_id, _airbyte_emitted_at, - + CURRENT_TIMESTAMP as _airbyte_normalized_at from __dbt__cte__exchange_rate_ab1 @@ -113,7 +113,7 @@ select time_no_tz, _airbyte_ab_id, _airbyte_emitted_at, - + CURRENT_TIMESTAMP as _airbyte_normalized_at, _airbyte_exchange_rate_hashid diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/models/generated/sources.yml index 97bf0d05cbd4..f51802427655 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/models/generated/sources.yml @@ -13,3 +13,4 @@ sources: - name: _airbyte_raw_multiple_column_names_conflicts - name: _airbyte_raw_pos_dedup_cdcx - name: _airbyte_raw_renamed_dedup_cdc_excluded + - name: _airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/second_output/airbyte_tables/test_normalization/exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/second_output/airbyte_tables/test_normalization/exchange_rate.sql index f7848e246cf1..540fc0e7911f 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/second_output/airbyte_tables/test_normalization/exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mysql/test_simple_streams/second_output/airbyte_tables/test_normalization/exchange_rate.sql @@ -3,41 +3,41 @@ create table test_normalization.`exchange_rate__dbt_tmp` as ( - + with __dbt__cte__exchange_rate_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema -- depends_on: test_normalization._airbyte_raw_exchange_rate select - json_value(_airbyte_data, + json_value(_airbyte_data, '$."id"' RETURNING CHAR) as id, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."currency"' RETURNING CHAR) as currency, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."date"' RETURNING CHAR) as `date`, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."timestamp_col"' RETURNING CHAR) as timestamp_col, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."HKD@spéçiäl & characters"' RETURNING CHAR) as `HKD@spéçiäl & characters`, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."HKD_special___characters"' RETURNING CHAR) as hkd_special___characters, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."NZD"' RETURNING CHAR) as nzd, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."USD"' RETURNING CHAR) as usd, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."column___with__quotes"' RETURNING CHAR) as `column__'with"_quotes`, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."datetime_tz"' RETURNING CHAR) as datetime_tz, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."datetime_no_tz"' RETURNING CHAR) as datetime_no_tz, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."time_tz"' RETURNING CHAR) as time_tz, - json_value(_airbyte_data, + json_value(_airbyte_data, '$."time_no_tz"' RETURNING CHAR) as time_no_tz, _airbyte_ab_id, _airbyte_emitted_at, - + CURRENT_TIMESTAMP as _airbyte_normalized_at from test_normalization._airbyte_raw_exchange_rate as table_alias @@ -48,7 +48,7 @@ where 1 = 1 -- SQL model to cast each column to its adequate SQL type converted from the JSON schema type -- depends_on: __dbt__cte__exchange_rate_ab1 select - cast(id as + cast(id as signed ) as id, cast(currency as char(1024)) as currency, @@ -57,14 +57,14 @@ select end as `date` , cast(nullif(timestamp_col, '') as char(1024)) as timestamp_col, - cast(`HKD@spéçiäl & characters` as + cast(`HKD@spéçiäl & characters` as float ) as `HKD@spéçiäl & characters`, cast(hkd_special___characters as char(1024)) as hkd_special___characters, - cast(nzd as + cast(nzd as float ) as nzd, - cast(usd as + cast(usd as float ) as usd, cast(`column__'with"_quotes` as char(1024)) as `column__'with"_quotes`, @@ -74,12 +74,12 @@ select end as datetime_no_tz , nullif(cast(time_tz as char(1024)), "") as time_tz, - nullif(cast(time_no_tz as + nullif(cast(time_no_tz as time ), "") as time_no_tz, _airbyte_ab_id, _airbyte_emitted_at, - + CURRENT_TIMESTAMP as _airbyte_normalized_at from __dbt__cte__exchange_rate_ab1 @@ -113,7 +113,7 @@ select time_no_tz, _airbyte_ab_id, _airbyte_emitted_at, - + CURRENT_TIMESTAMP as _airbyte_normalized_at, _airbyte_exchange_rate_hashid diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/oracle/test_simple_streams/dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/oracle/test_simple_streams/dbt_project.yml index a696787c00ab..e8b2b254e10b 100755 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/oracle/test_simple_streams/dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/oracle/test_simple_streams/dbt_project.yml @@ -1,25 +1,25 @@ name: airbyte_utils -version: "1.0" +version: '1.0' config-version: 2 profile: normalize source-paths: - - models +- models docs-paths: - - docs +- docs analysis-paths: - - analysis +- analysis test-paths: - - tests +- tests data-paths: - - data +- data macro-paths: - - macros +- macros target-path: ../build log-path: ../logs modules-path: /dbt clean-targets: - - build - - dbt_modules +- build +- dbt_modules quoting: database: false schema: false @@ -42,7 +42,7 @@ models: +materialized: view vars: dbt_utils_dispatch_list: - - airbyte_utils + - airbyte_utils json_column: _airbyte_data models_to_source: exchange_rate_ab1: test_normalization.airbyte_raw_exchange_rate @@ -79,3 +79,8 @@ vars: multiple_column_names_conflicts_stg: test_normalization.airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts_scd: test_normalization.airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts: test_normalization.airbyte_raw_multiple_column_names_conflicts + types_testing_ab1: test_normalization.airbyte_raw_types_testing + types_testing_ab2: test_normalization.airbyte_raw_types_testing + types_testing_stg: test_normalization.airbyte_raw_types_testing + types_testing_scd: test_normalization.airbyte_raw_types_testing + types_testing: test_normalization.airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/oracle/test_simple_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/oracle/test_simple_streams/models/generated/sources.yml index b5460fb4d43d..6fc61e6c97e1 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/oracle/test_simple_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/oracle/test_simple_streams/models/generated/sources.yml @@ -13,3 +13,4 @@ sources: - name: airbyte_raw_multiple_column_names_conflicts - name: airbyte_raw_pos_dedup_cdcx - name: airbyte_raw_renamed_dedup_cdc_excluded + - name: airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/dbt_project.yml index 77cd51053747..013a446b320a 100755 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/dbt_project.yml @@ -1,25 +1,25 @@ name: airbyte_utils -version: "1.0" +version: '1.0' config-version: 2 profile: normalize model-paths: - - modified_models +- modified_models docs-paths: - - docs +- docs analysis-paths: - - analysis +- analysis test-paths: - - tests +- tests seed-paths: - - data +- data macro-paths: - - macros +- macros target-path: ../build log-path: ../logs packages-install-path: /dbt clean-targets: - - build - - dbt_modules +- build +- dbt_modules quoting: database: true schema: false @@ -42,10 +42,10 @@ models: +tags: airbyte_internal_views +materialized: view dispatch: - - macro_namespace: dbt_utils - search_order: - - airbyte_utils - - dbt_utils +- macro_namespace: dbt_utils + search_order: + - airbyte_utils + - dbt_utils vars: json_column: _airbyte_data models_to_source: diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_dbt_project.yml index 200e87ca5ea7..12745c37a150 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_dbt_project.yml @@ -1,25 +1,25 @@ name: airbyte_utils -version: "1.0" +version: '1.0' config-version: 2 profile: normalize model-paths: - - models +- models docs-paths: - - docs +- docs analysis-paths: - - analysis +- analysis test-paths: - - tests +- tests seed-paths: - - data +- data macro-paths: - - macros +- macros target-path: ../build log-path: ../logs packages-install-path: /dbt clean-targets: - - build - - dbt_modules +- build +- dbt_modules quoting: database: true schema: false @@ -42,10 +42,10 @@ models: +tags: airbyte_internal_views +materialized: view dispatch: - - macro_namespace: dbt_utils - search_order: - - airbyte_utils - - dbt_utils +- macro_namespace: dbt_utils + search_order: + - airbyte_utils + - dbt_utils vars: json_column: _airbyte_data models_to_source: @@ -83,3 +83,8 @@ vars: multiple_column_names_conflicts_stg: test_normalization._airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts_scd: test_normalization._airbyte_raw_multiple_column_names_conflicts multiple_column_names_conflicts: test_normalization._airbyte_raw_multiple_column_names_conflicts + types_testing_ab1: test_normalization._airbyte_raw_types_testing + types_testing_ab2: test_normalization._airbyte_raw_types_testing + types_testing_stg: test_normalization._airbyte_raw_types_testing + types_testing_scd: test_normalization._airbyte_raw_types_testing + types_testing: test_normalization._airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/scd/test_normalization/types_testing_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/scd/test_normalization/types_testing_scd.sql new file mode 100644 index 000000000000..238d662a00cf --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/scd/test_normalization/types_testing_scd.sql @@ -0,0 +1,73 @@ + + + + create table "postgres".test_normalization."types_testing_scd" + as ( + +-- depends_on: ref('types_testing_stg') +with + +input_data as ( + select * + from "postgres"._airbyte_test_normalization."types_testing_stg" + -- types_testing from "postgres".test_normalization._airbyte_raw_types_testing +), + +scd_data as ( + -- SQL model to build a Type 2 Slowly Changing Dimension (SCD) table for each record identified by their primary key + select + md5(cast(coalesce(cast("id" as text), '') as text)) as _airbyte_unique_key, + "id", + airbyte_integer_column, + nullable_airbyte_integer_column, + _airbyte_emitted_at as _airbyte_start_at, + lag(_airbyte_emitted_at) over ( + partition by "id" + order by + _airbyte_emitted_at is null asc, + _airbyte_emitted_at desc, + _airbyte_emitted_at desc + ) as _airbyte_end_at, + case when row_number() over ( + partition by "id" + order by + _airbyte_emitted_at is null asc, + _airbyte_emitted_at desc, + _airbyte_emitted_at desc + ) = 1 then 1 else 0 end as _airbyte_active_row, + _airbyte_ab_id, + _airbyte_emitted_at, + _airbyte_types_testing_hashid + from input_data +), +dedup_data as ( + select + -- we need to ensure de-duplicated rows for merge/update queries + -- additionally, we generate a unique key for the scd table + row_number() over ( + partition by + _airbyte_unique_key, + _airbyte_start_at, + _airbyte_emitted_at + order by _airbyte_active_row desc, _airbyte_ab_id + ) as _airbyte_row_num, + md5(cast(coalesce(cast(_airbyte_unique_key as text), '') || '-' || coalesce(cast(_airbyte_start_at as text), '') || '-' || coalesce(cast(_airbyte_emitted_at as text), '') as text)) as _airbyte_unique_key_scd, + scd_data.* + from scd_data +) +select + _airbyte_unique_key, + _airbyte_unique_key_scd, + "id", + airbyte_integer_column, + nullable_airbyte_integer_column, + _airbyte_start_at, + _airbyte_end_at, + _airbyte_active_row, + _airbyte_ab_id, + _airbyte_emitted_at, + now() as _airbyte_normalized_at, + _airbyte_types_testing_hashid +from dedup_data where _airbyte_row_num = 1 + ); + \ No newline at end of file diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/test_normalization/types_testing.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/test_normalization/types_testing.sql new file mode 100644 index 000000000000..424c1918935b --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/test_normalization/types_testing.sql @@ -0,0 +1,24 @@ + + + + create table "postgres".test_normalization."types_testing" + as ( + +-- Final base SQL model +-- depends_on: "postgres".test_normalization."types_testing_scd" +select + _airbyte_unique_key, + "id", + airbyte_integer_column, + nullable_airbyte_integer_column, + _airbyte_ab_id, + _airbyte_emitted_at, + now() as _airbyte_normalized_at, + _airbyte_types_testing_hashid +from "postgres".test_normalization."types_testing_scd" +-- types_testing from "postgres".test_normalization._airbyte_raw_types_testing +where 1 = 1 +and _airbyte_active_row = 1 + + ); + \ No newline at end of file diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/test_normalization/types_testing_stg.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/test_normalization/types_testing_stg.sql new file mode 100644 index 000000000000..7eccd56d0609 --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/first_output/airbyte_incremental/test_normalization/types_testing_stg.sql @@ -0,0 +1,53 @@ + + + + create table "postgres"._airbyte_test_normalization."types_testing_stg" + as ( + +with __dbt__cte__types_testing_ab1 as ( + +-- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema +-- depends_on: "postgres".test_normalization._airbyte_raw_types_testing +select + jsonb_extract_path_text(_airbyte_data, 'id') as "id", + jsonb_extract_path_text(_airbyte_data, 'airbyte_integer_column') as airbyte_integer_column, + jsonb_extract_path_text(_airbyte_data, 'nullable_airbyte_integer_column') as nullable_airbyte_integer_column, + _airbyte_ab_id, + _airbyte_emitted_at, + now() as _airbyte_normalized_at +from "postgres".test_normalization._airbyte_raw_types_testing as table_alias +-- types_testing +where 1 = 1 + +), __dbt__cte__types_testing_ab2 as ( + +-- SQL model to cast each column to its adequate SQL type converted from the JSON schema type +-- depends_on: __dbt__cte__types_testing_ab1 +select + cast("id" as + bigint +) as "id", + cast(airbyte_integer_column as + bigint +) as airbyte_integer_column, + cast(nullable_airbyte_integer_column as + bigint +) as nullable_airbyte_integer_column, + _airbyte_ab_id, + _airbyte_emitted_at, + now() as _airbyte_normalized_at +from __dbt__cte__types_testing_ab1 +-- types_testing +where 1 = 1 + +)-- SQL model to build a hash column based on the values of this record +-- depends_on: __dbt__cte__types_testing_ab2 +select + md5(cast(coalesce(cast("id" as text), '') || '-' || coalesce(cast(airbyte_integer_column as text), '') || '-' || coalesce(cast(nullable_airbyte_integer_column as text), '') as text)) as _airbyte_types_testing_hashid, + tmp.* +from __dbt__cte__types_testing_ab2 tmp +-- types_testing +where 1 = 1 + + ); + \ No newline at end of file diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_ctes/test_normalization/types_testing_ab1.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_ctes/test_normalization/types_testing_ab1.sql new file mode 100644 index 000000000000..2fca430a9c39 --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_ctes/test_normalization/types_testing_ab1.sql @@ -0,0 +1,20 @@ +{{ config( + indexes = [{'columns':['_airbyte_emitted_at'],'type':'btree'}], + unique_key = '_airbyte_ab_id', + schema = "_airbyte_test_normalization", + tags = [ "top-level-intermediate" ] +) }} +-- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema +-- depends_on: {{ source('test_normalization', '_airbyte_raw_types_testing') }} +select + {{ json_extract_scalar('_airbyte_data', ['id'], ['id']) }} as {{ adapter.quote('id') }}, + {{ json_extract_scalar('_airbyte_data', ['airbyte_integer_column'], ['airbyte_integer_column']) }} as airbyte_integer_column, + {{ json_extract_scalar('_airbyte_data', ['nullable_airbyte_integer_column'], ['nullable_airbyte_integer_column']) }} as nullable_airbyte_integer_column, + _airbyte_ab_id, + _airbyte_emitted_at, + {{ current_timestamp() }} as _airbyte_normalized_at +from {{ source('test_normalization', '_airbyte_raw_types_testing') }} as table_alias +-- types_testing +where 1 = 1 +{{ incremental_clause('_airbyte_emitted_at', this) }} + diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_ctes/test_normalization/types_testing_ab2.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_ctes/test_normalization/types_testing_ab2.sql new file mode 100644 index 000000000000..da93832f7f77 --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_ctes/test_normalization/types_testing_ab2.sql @@ -0,0 +1,20 @@ +{{ config( + indexes = [{'columns':['_airbyte_emitted_at'],'type':'btree'}], + unique_key = '_airbyte_ab_id', + schema = "_airbyte_test_normalization", + tags = [ "top-level-intermediate" ] +) }} +-- SQL model to cast each column to its adequate SQL type converted from the JSON schema type +-- depends_on: {{ ref('types_testing_ab1') }} +select + cast({{ adapter.quote('id') }} as {{ dbt_utils.type_bigint() }}) as {{ adapter.quote('id') }}, + cast(airbyte_integer_column as {{ dbt_utils.type_bigint() }}) as airbyte_integer_column, + cast(nullable_airbyte_integer_column as {{ dbt_utils.type_bigint() }}) as nullable_airbyte_integer_column, + _airbyte_ab_id, + _airbyte_emitted_at, + {{ current_timestamp() }} as _airbyte_normalized_at +from {{ ref('types_testing_ab1') }} +-- types_testing +where 1 = 1 +{{ incremental_clause('_airbyte_emitted_at', this) }} + diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/types_testing_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/types_testing_scd.sql new file mode 100644 index 000000000000..0a0b409c90b7 --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/types_testing_scd.sql @@ -0,0 +1,163 @@ +{{ config( + indexes = [{'columns':['_airbyte_active_row','_airbyte_unique_key_scd','_airbyte_emitted_at'],'type': 'btree'}], + unique_key = "_airbyte_unique_key_scd", + schema = "test_normalization", + post_hook = [" + {% + set final_table_relation = adapter.get_relation( + database=this.database, + schema=this.schema, + identifier='types_testing' + ) + %} + {# + If the final table doesn't exist, then obviously we can't delete anything from it. + Also, after a reset, the final table is created without the _airbyte_unique_key column (this column is created during the first sync) + So skip this deletion if the column doesn't exist. (in this case, the table is guaranteed to be empty anyway) + #} + {% + if final_table_relation is not none and '_airbyte_unique_key' in adapter.get_columns_in_relation(final_table_relation)|map(attribute='name') + %} + -- Delete records which are no longer active: + -- This query is equivalent, but the left join version is more performant: + -- delete from final_table where unique_key in ( + -- select unique_key from scd_table where 1 = 1 + -- ) and unique_key not in ( + -- select unique_key from scd_table where active_row = 1 + -- ) + -- We're incremental against normalized_at rather than emitted_at because we need to fetch the SCD + -- entries that were _updated_ recently. This is because a deleted record will have an SCD record + -- which was emitted a long time ago, but recently re-normalized to have active_row = 0. + delete from {{ final_table_relation }} where {{ final_table_relation }}._airbyte_unique_key in ( + select recent_records.unique_key + from ( + select distinct _airbyte_unique_key as unique_key + from {{ this }} + where 1=1 {{ incremental_clause('_airbyte_normalized_at', this.schema + '.' + adapter.quote('types_testing')) }} + ) recent_records + left join ( + select _airbyte_unique_key as unique_key, count(_airbyte_unique_key) as active_count + from {{ this }} + where _airbyte_active_row = 1 {{ incremental_clause('_airbyte_normalized_at', this.schema + '.' + adapter.quote('types_testing')) }} + group by _airbyte_unique_key + ) active_counts + on recent_records.unique_key = active_counts.unique_key + where active_count is null or active_count = 0 + ) + {% else %} + -- We have to have a non-empty query, so just do a noop delete + delete from {{ this }} where 1=0 + {% endif %} + ","delete from _airbyte_test_normalization.types_testing_stg where _airbyte_emitted_at != (select max(_airbyte_emitted_at) from _airbyte_test_normalization.types_testing_stg)"], + tags = [ "top-level" ] +) }} +-- depends_on: ref('types_testing_stg') +with +{% if is_incremental() %} +new_data as ( + -- retrieve incremental "new" data + select + * + from {{ ref('types_testing_stg') }} + -- types_testing from {{ source('test_normalization', '_airbyte_raw_types_testing') }} + where 1 = 1 + {{ incremental_clause('_airbyte_emitted_at', this) }} +), +new_data_ids as ( + -- build a subset of _airbyte_unique_key from rows that are new + select distinct + {{ dbt_utils.surrogate_key([ + adapter.quote('id'), + ]) }} as _airbyte_unique_key + from new_data +), +empty_new_data as ( + -- build an empty table to only keep the table's column types + select * from new_data where 1 = 0 +), +previous_active_scd_data as ( + -- retrieve "incomplete old" data that needs to be updated with an end date because of new changes + select + {{ star_intersect(ref('types_testing_stg'), this, from_alias='inc_data', intersect_alias='this_data') }} + from {{ this }} as this_data + -- make a join with new_data using primary key to filter active data that need to be updated only + join new_data_ids on this_data._airbyte_unique_key = new_data_ids._airbyte_unique_key + -- force left join to NULL values (we just need to transfer column types only for the star_intersect macro on schema changes) + left join empty_new_data as inc_data on this_data._airbyte_ab_id = inc_data._airbyte_ab_id + where _airbyte_active_row = 1 +), +input_data as ( + select {{ dbt_utils.star(ref('types_testing_stg')) }} from new_data + union all + select {{ dbt_utils.star(ref('types_testing_stg')) }} from previous_active_scd_data +), +{% else %} +input_data as ( + select * + from {{ ref('types_testing_stg') }} + -- types_testing from {{ source('test_normalization', '_airbyte_raw_types_testing') }} +), +{% endif %} +scd_data as ( + -- SQL model to build a Type 2 Slowly Changing Dimension (SCD) table for each record identified by their primary key + select + {{ dbt_utils.surrogate_key([ + adapter.quote('id'), + ]) }} as _airbyte_unique_key, + {{ adapter.quote('id') }}, + airbyte_integer_column, + nullable_airbyte_integer_column, + _airbyte_emitted_at as _airbyte_start_at, + lag(_airbyte_emitted_at) over ( + partition by {{ adapter.quote('id') }} + order by + _airbyte_emitted_at is null asc, + _airbyte_emitted_at desc, + _airbyte_emitted_at desc + ) as _airbyte_end_at, + case when row_number() over ( + partition by {{ adapter.quote('id') }} + order by + _airbyte_emitted_at is null asc, + _airbyte_emitted_at desc, + _airbyte_emitted_at desc + ) = 1 then 1 else 0 end as _airbyte_active_row, + _airbyte_ab_id, + _airbyte_emitted_at, + _airbyte_types_testing_hashid + from input_data +), +dedup_data as ( + select + -- we need to ensure de-duplicated rows for merge/update queries + -- additionally, we generate a unique key for the scd table + row_number() over ( + partition by + _airbyte_unique_key, + _airbyte_start_at, + _airbyte_emitted_at + order by _airbyte_active_row desc, _airbyte_ab_id + ) as _airbyte_row_num, + {{ dbt_utils.surrogate_key([ + '_airbyte_unique_key', + '_airbyte_start_at', + '_airbyte_emitted_at' + ]) }} as _airbyte_unique_key_scd, + scd_data.* + from scd_data +) +select + _airbyte_unique_key, + _airbyte_unique_key_scd, + {{ adapter.quote('id') }}, + airbyte_integer_column, + nullable_airbyte_integer_column, + _airbyte_start_at, + _airbyte_end_at, + _airbyte_active_row, + _airbyte_ab_id, + _airbyte_emitted_at, + {{ current_timestamp() }} as _airbyte_normalized_at, + _airbyte_types_testing_hashid +from dedup_data where _airbyte_row_num = 1 + diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/test_normalization/types_testing.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/test_normalization/types_testing.sql new file mode 100644 index 000000000000..8f979379656d --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/test_normalization/types_testing.sql @@ -0,0 +1,23 @@ +{{ config( + indexes = [{'columns':['_airbyte_unique_key'],'unique':True}], + unique_key = "_airbyte_unique_key", + schema = "test_normalization", + tags = [ "top-level" ] +) }} +-- Final base SQL model +-- depends_on: {{ ref('types_testing_scd') }} +select + _airbyte_unique_key, + {{ adapter.quote('id') }}, + airbyte_integer_column, + nullable_airbyte_integer_column, + _airbyte_ab_id, + _airbyte_emitted_at, + {{ current_timestamp() }} as _airbyte_normalized_at, + _airbyte_types_testing_hashid +from {{ ref('types_testing_scd') }} +-- types_testing from {{ source('test_normalization', '_airbyte_raw_types_testing') }} +where 1 = 1 +and _airbyte_active_row = 1 +{{ incremental_clause('_airbyte_emitted_at', this) }} + diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/test_normalization/types_testing_stg.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/test_normalization/types_testing_stg.sql new file mode 100644 index 000000000000..3eabf9e4ae69 --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/airbyte_incremental/test_normalization/types_testing_stg.sql @@ -0,0 +1,20 @@ +{{ config( + indexes = [{'columns':['_airbyte_emitted_at'],'type':'btree'}], + unique_key = '_airbyte_ab_id', + schema = "_airbyte_test_normalization", + tags = [ "top-level-intermediate" ] +) }} +-- SQL model to build a hash column based on the values of this record +-- depends_on: {{ ref('types_testing_ab2') }} +select + {{ dbt_utils.surrogate_key([ + adapter.quote('id'), + 'airbyte_integer_column', + 'nullable_airbyte_integer_column', + ]) }} as _airbyte_types_testing_hashid, + tmp.* +from {{ ref('types_testing_ab2') }} tmp +-- types_testing +where 1 = 1 +{{ incremental_clause('_airbyte_emitted_at', this) }} + diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/sources.yml index 97bf0d05cbd4..f51802427655 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/models/generated/sources.yml @@ -13,3 +13,4 @@ sources: - name: _airbyte_raw_multiple_column_names_conflicts - name: _airbyte_raw_pos_dedup_cdcx - name: _airbyte_raw_renamed_dedup_cdc_excluded + - name: _airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/modified_models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/modified_models/generated/sources.yml index 79ad1a1bb5c5..6a5d7bdc09a1 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/modified_models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/modified_models/generated/sources.yml @@ -1,12 +1,12 @@ version: 2 sources: - - name: test_normalization - quoting: - database: true - schema: false - identifier: false - tables: - - name: _airbyte_raw_dedup_cdc_excluded - - name: _airbyte_raw_dedup_exchange_rate - - name: _airbyte_raw_exchange_rate - - name: _airbyte_raw_renamed_dedup_cdc_excluded +- name: test_normalization + quoting: + database: true + schema: false + identifier: false + tables: + - name: _airbyte_raw_dedup_cdc_excluded + - name: _airbyte_raw_dedup_exchange_rate + - name: _airbyte_raw_exchange_rate + - name: _airbyte_raw_renamed_dedup_cdc_excluded diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/scd/test_normalization/types_testing_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/scd/test_normalization/types_testing_scd.sql new file mode 100644 index 000000000000..8388a44777b8 --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/scd/test_normalization/types_testing_scd.sql @@ -0,0 +1,15 @@ + + + delete from "postgres".test_normalization."types_testing_scd" + where (_airbyte_unique_key_scd) in ( + select (_airbyte_unique_key_scd) + from "types_testing_scd__dbt_tmp" + ); + + + insert into "postgres".test_normalization."types_testing_scd" ("_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "airbyte_integer_column", "nullable_airbyte_integer_column", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_types_testing_hashid") + ( + select "_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "airbyte_integer_column", "nullable_airbyte_integer_column", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_types_testing_hashid" + from "types_testing_scd__dbt_tmp" + ) + \ No newline at end of file diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/test_normalization/types_testing.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/test_normalization/types_testing.sql new file mode 100644 index 000000000000..f01bbf894193 --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/test_normalization/types_testing.sql @@ -0,0 +1,15 @@ + + + delete from "postgres".test_normalization."types_testing" + where (_airbyte_unique_key) in ( + select (_airbyte_unique_key) + from "types_testing__dbt_tmp" + ); + + + insert into "postgres".test_normalization."types_testing" ("_airbyte_unique_key", "id", "airbyte_integer_column", "nullable_airbyte_integer_column", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_types_testing_hashid") + ( + select "_airbyte_unique_key", "id", "airbyte_integer_column", "nullable_airbyte_integer_column", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_types_testing_hashid" + from "types_testing__dbt_tmp" + ) + \ No newline at end of file diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/test_normalization/types_testing_stg.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/test_normalization/types_testing_stg.sql new file mode 100644 index 000000000000..1295c519d0d7 --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/postgres/test_simple_streams/second_output/airbyte_incremental/test_normalization/types_testing_stg.sql @@ -0,0 +1,15 @@ + + + delete from "postgres"._airbyte_test_normalization."types_testing_stg" + where (_airbyte_ab_id) in ( + select (_airbyte_ab_id) + from "types_testing_stg__dbt_tmp" + ); + + + insert into "postgres"._airbyte_test_normalization."types_testing_stg" ("_airbyte_types_testing_hashid", "id", "airbyte_integer_column", "nullable_airbyte_integer_column", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at") + ( + select "_airbyte_types_testing_hashid", "id", "airbyte_integer_column", "nullable_airbyte_integer_column", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at" + from "types_testing_stg__dbt_tmp" + ) + \ No newline at end of file diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/dbt_project.yml index 12f7ddeae970..767544968e0b 100755 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/dbt_project.yml @@ -1,25 +1,25 @@ name: airbyte_utils -version: "1.0" +version: '1.0' config-version: 2 profile: normalize model-paths: - - models +- models docs-paths: - - docs +- docs analysis-paths: - - analysis +- analysis test-paths: - - tests +- tests seed-paths: - - data +- data macro-paths: - - macros +- macros target-path: ../build log-path: ../logs packages-install-path: /dbt clean-targets: - - build - - dbt_modules +- build +- dbt_modules quoting: database: true schema: false @@ -44,76 +44,84 @@ models: +tags: airbyte_internal_views +materialized: view dispatch: - - macro_namespace: dbt_utils - search_order: - - airbyte_utils - - dbt_utils +- macro_namespace: dbt_utils + search_order: + - airbyte_utils + - dbt_utils vars: json_column: _airbyte_data models_to_source: - nested_stream_with_complex_columns_resulting_into_long_names_ab1: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_ab2: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_stg: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_scd: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - non_nested_stream_without_namespace_resulting_into_long_names_ab1: test_normalization._airbyte_raw_non_nested_stream_without_namespace_resulting_into_long_names - non_nested_stream_without_namespace_resulting_into_long_names_ab2: test_normalization._airbyte_raw_non_nested_stream_without_namespace_resulting_into_long_names - non_nested_stream_without_namespace_resulting_into_long_names_ab3: test_normalization._airbyte_raw_non_nested_stream_without_namespace_resulting_into_long_names - non_nested_stream_without_namespace_resulting_into_long_names: test_normalization._airbyte_raw_non_nested_stream_without_namespace_resulting_into_long_names - some_stream_that_was_empty_ab1: test_normalization._airbyte_raw_some_stream_that_was_empty - some_stream_that_was_empty_ab2: test_normalization._airbyte_raw_some_stream_that_was_empty - some_stream_that_was_empty_stg: test_normalization._airbyte_raw_some_stream_that_was_empty - some_stream_that_was_empty_scd: test_normalization._airbyte_raw_some_stream_that_was_empty - some_stream_that_was_empty: test_normalization._airbyte_raw_some_stream_that_was_empty + nested_stream_with_complex_columns_resulting_into_long_names_ab1: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_ab2: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_stg: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_scd: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + non_nested_stream_without_namespace_resulting_into_long_names_ab1: test_normalization_xjvlg._airbyte_raw_non_nested_stream_without_namespace_resulting_into_long_names + non_nested_stream_without_namespace_resulting_into_long_names_ab2: test_normalization_xjvlg._airbyte_raw_non_nested_stream_without_namespace_resulting_into_long_names + non_nested_stream_without_namespace_resulting_into_long_names_ab3: test_normalization_xjvlg._airbyte_raw_non_nested_stream_without_namespace_resulting_into_long_names + non_nested_stream_without_namespace_resulting_into_long_names: test_normalization_xjvlg._airbyte_raw_non_nested_stream_without_namespace_resulting_into_long_names + some_stream_that_was_empty_ab1: test_normalization_xjvlg._airbyte_raw_some_stream_that_was_empty + some_stream_that_was_empty_ab2: test_normalization_xjvlg._airbyte_raw_some_stream_that_was_empty + some_stream_that_was_empty_stg: test_normalization_xjvlg._airbyte_raw_some_stream_that_was_empty + some_stream_that_was_empty_scd: test_normalization_xjvlg._airbyte_raw_some_stream_that_was_empty + some_stream_that_was_empty: test_normalization_xjvlg._airbyte_raw_some_stream_that_was_empty simple_stream_with_namespace_resulting_into_long_names_ab1: test_normalization_namespace._airbyte_raw_simple_stream_with_namespace_resulting_into_long_names simple_stream_with_namespace_resulting_into_long_names_ab2: test_normalization_namespace._airbyte_raw_simple_stream_with_namespace_resulting_into_long_names simple_stream_with_namespace_resulting_into_long_names_ab3: test_normalization_namespace._airbyte_raw_simple_stream_with_namespace_resulting_into_long_names simple_stream_with_namespace_resulting_into_long_names: test_normalization_namespace._airbyte_raw_simple_stream_with_namespace_resulting_into_long_names - conflict_stream_name_ab1: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_name_ab2: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_name_ab3: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_name: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_scalar_ab1: test_normalization._airbyte_raw_conflict_stream_scalar - conflict_stream_scalar_ab2: test_normalization._airbyte_raw_conflict_stream_scalar - conflict_stream_scalar_ab3: test_normalization._airbyte_raw_conflict_stream_scalar - conflict_stream_scalar: test_normalization._airbyte_raw_conflict_stream_scalar - conflict_stream_array_ab1: test_normalization._airbyte_raw_conflict_stream_array - conflict_stream_array_ab2: test_normalization._airbyte_raw_conflict_stream_array - conflict_stream_array_ab3: test_normalization._airbyte_raw_conflict_stream_array - conflict_stream_array: test_normalization._airbyte_raw_conflict_stream_array - unnest_alias_ab1: test_normalization._airbyte_raw_unnest_alias - unnest_alias_ab2: test_normalization._airbyte_raw_unnest_alias - unnest_alias_ab3: test_normalization._airbyte_raw_unnest_alias - unnest_alias: test_normalization._airbyte_raw_unnest_alias - nested_stream_with_complex_columns_resulting_into_long_names_partition_ab1: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition_ab2: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition_ab3: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - conflict_stream_name_conflict_stream_name_ab1: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_name_conflict_stream_name_ab2: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_name_conflict_stream_name_ab3: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_name_conflict_stream_name: test_normalization._airbyte_raw_conflict_stream_name - unnest_alias_children_ab1: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children_ab2: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children_ab3: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children: test_normalization._airbyte_raw_unnest_alias - nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab1: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab2: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab3: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab1: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab2: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab3: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - nested_stream_with_complex_columns_resulting_into_long_names_partition_data: test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names - conflict_stream_name_conflict_stream_name_conflict_stream_name_ab1: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_name_conflict_stream_name_conflict_stream_name_ab2: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_name_conflict_stream_name_conflict_stream_name_ab3: test_normalization._airbyte_raw_conflict_stream_name - conflict_stream_name_conflict_stream_name_conflict_stream_name: test_normalization._airbyte_raw_conflict_stream_name - unnest_alias_children_owner_ab1: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children_owner_ab2: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children_owner_ab3: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children_owner: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children_owner_column___with__quotes_ab1: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children_owner_column___with__quotes_ab2: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children_owner_column___with__quotes_ab3: test_normalization._airbyte_raw_unnest_alias - unnest_alias_children_owner_column___with__quotes: test_normalization._airbyte_raw_unnest_alias + conflict_stream_name_ab1: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_name_ab2: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_name_ab3: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_name: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_scalar_ab1: test_normalization_xjvlg._airbyte_raw_conflict_stream_scalar + conflict_stream_scalar_ab2: test_normalization_xjvlg._airbyte_raw_conflict_stream_scalar + conflict_stream_scalar_ab3: test_normalization_xjvlg._airbyte_raw_conflict_stream_scalar + conflict_stream_scalar: test_normalization_xjvlg._airbyte_raw_conflict_stream_scalar + conflict_stream_array_ab1: test_normalization_xjvlg._airbyte_raw_conflict_stream_array + conflict_stream_array_ab2: test_normalization_xjvlg._airbyte_raw_conflict_stream_array + conflict_stream_array_ab3: test_normalization_xjvlg._airbyte_raw_conflict_stream_array + conflict_stream_array: test_normalization_xjvlg._airbyte_raw_conflict_stream_array + unnest_alias_ab1: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_ab2: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_ab3: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias: test_normalization_xjvlg._airbyte_raw_unnest_alias + arrays_ab1: test_normalization_xjvlg._airbyte_raw_arrays + arrays_ab2: test_normalization_xjvlg._airbyte_raw_arrays + arrays_ab3: test_normalization_xjvlg._airbyte_raw_arrays + arrays: test_normalization_xjvlg._airbyte_raw_arrays + nested_stream_with_complex_columns_resulting_into_long_names_partition_ab1: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition_ab2: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition_ab3: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + conflict_stream_name_conflict_stream_name_ab1: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_name_conflict_stream_name_ab2: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_name_conflict_stream_name_ab3: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_name_conflict_stream_name: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + unnest_alias_children_ab1: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children_ab2: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children_ab3: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children: test_normalization_xjvlg._airbyte_raw_unnest_alias + arrays_nested_array_parent_ab1: test_normalization_xjvlg._airbyte_raw_arrays + arrays_nested_array_parent_ab2: test_normalization_xjvlg._airbyte_raw_arrays + arrays_nested_array_parent_ab3: test_normalization_xjvlg._airbyte_raw_arrays + arrays_nested_array_parent: test_normalization_xjvlg._airbyte_raw_arrays + nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab1: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab2: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab3: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab1: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab2: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab3: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + nested_stream_with_complex_columns_resulting_into_long_names_partition_data: test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + conflict_stream_name_conflict_stream_name_conflict_stream_name_ab1: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_name_conflict_stream_name_conflict_stream_name_ab2: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_name_conflict_stream_name_conflict_stream_name_ab3: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + conflict_stream_name_conflict_stream_name_conflict_stream_name: test_normalization_xjvlg._airbyte_raw_conflict_stream_name + unnest_alias_children_owner_ab1: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children_owner_ab2: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children_owner_ab3: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children_owner: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children_owner_column___with__quotes_ab1: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children_owner_column___with__quotes_ab2: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children_owner_column___with__quotes_ab3: test_normalization_xjvlg._airbyte_raw_unnest_alias + unnest_alias_children_owner_column___with__quotes: test_normalization_xjvlg._airbyte_raw_unnest_alias diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql index 753b62319771..9b59d6d77c88 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql @@ -2,7 +2,7 @@ create table - "integrationtests"."test_normalization"."nested_stream_with_complex_columns_resulting_into_long_names_scd__dbt_tmp" + "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_scd" compound sortkey(_airbyte_active_row,_airbyte_unique_key_scd,_airbyte_emitted_at) @@ -14,8 +14,8 @@ with input_data as ( select * - from "integrationtests"._airbyte_test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_stg" - -- nested_stream_with_complex_columns_resulting_into_long_names from "integrationtests".test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names + from "integrationtests"._airbyte_test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_stg" + -- nested_stream_with_complex_columns_resulting_into_long_names from "integrationtests".test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names ), scd_data as ( diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql index 6961ef2198dc..184fa2bf1104 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql @@ -2,7 +2,7 @@ create table - "integrationtests"."test_normalization"."nested_stream_with_complex_columns_resulting_into_long_names__dbt_tmp" + "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names" compound sortkey(_airbyte_unique_key,_airbyte_emitted_at) @@ -10,7 +10,7 @@ as ( -- Final base SQL model --- depends_on: "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_scd" +-- depends_on: "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_scd" select _airbyte_unique_key, id, @@ -20,8 +20,8 @@ select _airbyte_emitted_at, getdate() as _airbyte_normalized_at, _airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid -from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_scd" --- nested_stream_with_complex_columns_resulting_into_long_names from "integrationtests".test_normalization._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names +from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_scd" +-- nested_stream_with_complex_columns_resulting_into_long_names from "integrationtests".test_normalization_xjvlg._airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names where 1 = 1 and _airbyte_active_row = 1 diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql index 8348fdeb8132..4e1c7b1f3942 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql @@ -2,7 +2,7 @@ create table - "integrationtests"."test_normalization"."nested_stream_with_complex_columns_resulting_into_long_names_partition__dbt_tmp" + "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" compound sortkey(_airbyte_emitted_at) @@ -12,7 +12,7 @@ with __dbt__cte__nested_stream_with_complex_columns_resulting_into_long_names_partition_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_scd" +-- depends_on: "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_scd" select _airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid, "partition"."double_array_data" as double_array_data, @@ -20,7 +20,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_scd" as table_alias +from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_scd" as table_alias -- partition at nested_stream_with_complex_columns_resulting_into_long_names/partition where 1 = 1 and "partition" is not null @@ -62,7 +62,7 @@ select getdate() as _airbyte_normalized_at, _airbyte_partition_hashid from __dbt__cte__nested_stream_with_complex_columns_resulting_into_long_names_partition_ab3 --- partition at nested_stream_with_complex_columns_resulting_into_long_names/partition from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_scd" +-- partition at nested_stream_with_complex_columns_resulting_into_long_names/partition from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_scd" where 1 = 1 ); diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql index 3f46c9e431c0..e19271e39a6f 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql @@ -2,7 +2,7 @@ create table - "integrationtests"."test_normalization"."nested_stream_with_complex_columns_resulting_into_long_names_partition_data__dbt_tmp" + "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition_data" compound sortkey(_airbyte_emitted_at) @@ -12,13 +12,13 @@ with __dbt__cte__nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition" +-- depends_on: "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" with joined as ( select table_alias._airbyte_partition_hashid as _airbyte_hashid, _airbyte_nested_data - from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition" as table_alias, table_alias.data as _airbyte_nested_data + from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" as table_alias, table_alias.data as _airbyte_nested_data ) select _airbyte_partition_hashid, @@ -26,7 +26,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition" as table_alias +from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" as table_alias -- data at nested_stream_with_complex_columns_resulting_into_long_names/partition/DATA left join joined on _airbyte_partition_hashid = joined._airbyte_hashid where 1 = 1 @@ -67,7 +67,7 @@ select getdate() as _airbyte_normalized_at, _airbyte_data_hashid from __dbt__cte__nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab3 --- data at nested_stream_with_complex_columns_resulting_into_long_names/partition/DATA from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition" +-- data at nested_stream_with_complex_columns_resulting_into_long_names/partition/DATA from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" where 1 = 1 ); diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql index 84fbebb03b50..7e38b76f87fe 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/first_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql @@ -2,7 +2,7 @@ create table - "integrationtests"."test_normalization"."nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data__dbt_tmp" + "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data" compound sortkey(_airbyte_emitted_at) @@ -12,13 +12,13 @@ with __dbt__cte__nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition" +-- depends_on: "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" with joined as ( select table_alias._airbyte_partition_hashid as _airbyte_hashid, _airbyte_nested_data - from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition" as table_alias, table_alias.double_array_data as _airbyte_nested_data + from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" as table_alias, table_alias.double_array_data as _airbyte_nested_data ) select _airbyte_partition_hashid, @@ -26,7 +26,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition" as table_alias +from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" as table_alias -- double_array_data at nested_stream_with_complex_columns_resulting_into_long_names/partition/double_array_data left join joined on _airbyte_partition_hashid = joined._airbyte_hashid where 1 = 1 @@ -67,7 +67,7 @@ select getdate() as _airbyte_normalized_at, _airbyte_double_array_data_hashid from __dbt__cte__nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab3 --- double_array_data at nested_stream_with_complex_columns_resulting_into_long_names/partition/double_array_data from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition" +-- double_array_data at nested_stream_with_complex_columns_resulting_into_long_names/partition/double_array_data from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" where 1 = 1 ); diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_ab1.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_ab1.sql index c6c4c7bb3973..ed49a5e91606 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_ab1.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_ab1.sql @@ -1,11 +1,11 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_xjvlg", tags = [ "top-level-intermediate" ] ) }} -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: {{ source('test_normalization', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} +-- depends_on: {{ source('test_normalization_xjvlg', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} select {{ json_extract_scalar('_airbyte_data', ['id'], ['id']) }} as id, {{ json_extract_scalar('_airbyte_data', ['date'], ['date']) }} as date, @@ -13,7 +13,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, {{ current_timestamp() }} as _airbyte_normalized_at -from {{ source('test_normalization', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} as table_alias +from {{ source('test_normalization_xjvlg', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} as table_alias -- nested_stream_with_complex_columns_resulting_into_long_names where 1 = 1 {{ incremental_clause('_airbyte_emitted_at', this) }} diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_ab2.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_ab2.sql index 41c1f86de0c7..19ab94bca151 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_ab2.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_ab2.sql @@ -1,7 +1,7 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_xjvlg", tags = [ "top-level-intermediate" ] ) }} -- SQL model to cast each column to its adequate SQL type converted from the JSON schema type diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_ab1.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_ab1.sql index 7d9968fe7170..18a21b472981 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_ab1.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_ab1.sql @@ -1,6 +1,6 @@ {{ config( sort = "_airbyte_emitted_at", - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_xjvlg", tags = [ "nested-intermediate" ] ) }} -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab1.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab1.sql index b3f16a06f544..4cc3285a5f6e 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab1.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data_ab1.sql @@ -1,6 +1,6 @@ {{ config( sort = "_airbyte_emitted_at", - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_xjvlg", tags = [ "nested-intermediate" ] ) }} -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab1.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab1.sql index 7fe25a4c8eca..4876b27d7cc0 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab1.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_ctes/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data_ab1.sql @@ -1,6 +1,6 @@ {{ config( sort = "_airbyte_emitted_at", - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_xjvlg", tags = [ "nested-intermediate" ] ) }} -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql index 627f56e3ad2a..a629e4de4e5d 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql @@ -1,7 +1,7 @@ {{ config( sort = ["_airbyte_active_row", "_airbyte_unique_key_scd", "_airbyte_emitted_at"], unique_key = "_airbyte_unique_key_scd", - schema = "test_normalization", + schema = "test_normalization_xjvlg", post_hook = [" {% set final_table_relation = adapter.get_relation( @@ -48,7 +48,7 @@ -- We have to have a non-empty query, so just do a noop delete delete from {{ this }} where 1=0 {% endif %} - ","drop view _airbyte_test_normalization.nested_stream_with_complex_columns_resulting_into_long_names_stg"], + ","drop view _airbyte_test_normalization_xjvlg.nested_stream_with_complex_columns_resulting_into_long_names_stg"], tags = [ "top-level" ] ) }} -- depends_on: ref('nested_stream_with_complex_columns_resulting_into_long_names_stg') @@ -59,7 +59,7 @@ new_data as ( select * from {{ ref('nested_stream_with_complex_columns_resulting_into_long_names_stg') }} - -- nested_stream_with_complex_columns_resulting_into_long_names from {{ source('test_normalization', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} + -- nested_stream_with_complex_columns_resulting_into_long_names from {{ source('test_normalization_xjvlg', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} where 1 = 1 {{ incremental_clause('_airbyte_emitted_at', this) }} ), @@ -95,7 +95,7 @@ input_data as ( input_data as ( select * from {{ ref('nested_stream_with_complex_columns_resulting_into_long_names_stg') }} - -- nested_stream_with_complex_columns_resulting_into_long_names from {{ source('test_normalization', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} + -- nested_stream_with_complex_columns_resulting_into_long_names from {{ source('test_normalization_xjvlg', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} ), {% endif %} scd_data as ( diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql index 7b608604bbee..f95f159eedc9 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql @@ -1,7 +1,7 @@ {{ config( sort = ["_airbyte_unique_key", "_airbyte_emitted_at"], unique_key = "_airbyte_unique_key", - schema = "test_normalization", + schema = "test_normalization_xjvlg", tags = [ "top-level" ] ) }} -- Final base SQL model @@ -16,7 +16,7 @@ select {{ current_timestamp() }} as _airbyte_normalized_at, _airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid from {{ ref('nested_stream_with_complex_columns_resulting_into_long_names_scd') }} --- nested_stream_with_complex_columns_resulting_into_long_names from {{ source('test_normalization', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} +-- nested_stream_with_complex_columns_resulting_into_long_names from {{ source('test_normalization_xjvlg', '_airbyte_raw_nested_stream_with_complex_columns_resulting_into_long_names') }} where 1 = 1 and _airbyte_active_row = 1 {{ incremental_clause('_airbyte_emitted_at', this) }} diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql index 10a422df08a2..18a73cf63b7f 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql @@ -1,6 +1,6 @@ {{ config( sort = "_airbyte_emitted_at", - schema = "test_normalization", + schema = "test_normalization_xjvlg", tags = [ "nested" ] ) }} -- Final base SQL model diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql index 4551ff86747c..ad3d8a9a61b5 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql @@ -1,6 +1,6 @@ {{ config( sort = "_airbyte_emitted_at", - schema = "test_normalization", + schema = "test_normalization_xjvlg", tags = [ "nested" ] ) }} -- Final base SQL model diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql index 3f7bbf34e3e8..2059cb60a01a 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql @@ -1,6 +1,6 @@ {{ config( sort = "_airbyte_emitted_at", - schema = "test_normalization", + schema = "test_normalization_xjvlg", tags = [ "nested" ] ) }} -- Final base SQL model diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/sources.yml index 92fa4c9a2580..56faa01c65dc 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/models/generated/sources.yml @@ -1,11 +1,19 @@ version: 2 sources: -- name: test_normalization +- name: test_normalization_namespace + quoting: + database: true + schema: false + identifier: false + tables: + - name: _airbyte_raw_simple_stream_with_namespace_resulting_into_long_names +- name: test_normalization_xjvlg quoting: database: true schema: false identifier: false tables: + - name: _airbyte_raw_arrays - name: _airbyte_raw_conflict_stream_array - name: _airbyte_raw_conflict_stream_name - name: _airbyte_raw_conflict_stream_scalar @@ -13,10 +21,3 @@ sources: - name: _airbyte_raw_non_nested_stream_without_namespace_resulting_into_long_names - name: _airbyte_raw_some_stream_that_was_empty - name: _airbyte_raw_unnest_alias -- name: test_normalization_namespace - quoting: - database: true - schema: false - identifier: false - tables: - - name: _airbyte_raw_simple_stream_with_namespace_resulting_into_long_names diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql index a3986d744fa8..45c63e057a5e 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql @@ -1,13 +1,13 @@ - delete from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_scd" + delete from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_scd" where (_airbyte_unique_key_scd) in ( select (_airbyte_unique_key_scd) from "nested_stream_with_complex_columns_resulti__dbt_tmp" ); - insert into "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_scd" ("_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "date", "partition", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid") + insert into "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_scd" ("_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "date", "partition", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid") ( select "_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "date", "partition", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid" from "nested_stream_with_complex_columns_resulti__dbt_tmp" diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql index 842e9a979c68..e32bb140a099 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names.sql @@ -1,13 +1,13 @@ - delete from "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names" + delete from "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names" where (_airbyte_unique_key) in ( select (_airbyte_unique_key) from "nested_stream_with_complex_columns_resulti__dbt_tmp" ); - insert into "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names" ("_airbyte_unique_key", "id", "date", "partition", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid") + insert into "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names" ("_airbyte_unique_key", "id", "date", "partition", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid") ( select "_airbyte_unique_key", "id", "date", "partition", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid" from "nested_stream_with_complex_columns_resulti__dbt_tmp" diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql index 9bf3ef8552d7..9944a91ca642 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition.sql @@ -1,7 +1,7 @@ - insert into "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition" ("_airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid", "double_array_data", "data", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_partition_hashid") + insert into "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition" ("_airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid", "double_array_data", "data", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_partition_hashid") ( select "_airbyte_nested_stream_with_complex_columns_resulting_into_long_names_hashid", "double_array_data", "data", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_partition_hashid" from "nested_stream_with_complex_columns_resulti__dbt_tmp" diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql index ec83ef6267a0..52b4bd4fc5f4 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_data.sql @@ -1,7 +1,7 @@ - insert into "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition_data" ("_airbyte_partition_hashid", "currency", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_data_hashid") + insert into "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition_data" ("_airbyte_partition_hashid", "currency", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_data_hashid") ( select "_airbyte_partition_hashid", "currency", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_data_hashid" from "nested_stream_with_complex_columns_resulti__dbt_tmp" diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql index 05efdbcc1ce2..91aaa5e85cc0 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_nested_streams/second_output/airbyte_incremental/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data.sql @@ -1,7 +1,7 @@ - insert into "integrationtests".test_normalization."nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data" ("_airbyte_partition_hashid", "id", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_double_array_data_hashid") + insert into "integrationtests".test_normalization_xjvlg."nested_stream_with_complex_columns_resulting_into_long_names_partition_double_array_data" ("_airbyte_partition_hashid", "id", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_double_array_data_hashid") ( select "_airbyte_partition_hashid", "id", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_double_array_data_hashid" from "nested_stream_with_complex_columns_resulti__dbt_tmp" diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/dbt_project.yml index 06d2109d3356..c645baf3c5fe 100755 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/dbt_project.yml @@ -1,25 +1,25 @@ name: airbyte_utils -version: "1.0" +version: '1.0' config-version: 2 profile: normalize model-paths: - - modified_models +- modified_models docs-paths: - - docs +- docs analysis-paths: - - analysis +- analysis test-paths: - - tests +- tests seed-paths: - - data +- data macro-paths: - - macros +- macros target-path: ../build log-path: ../logs packages-install-path: /dbt clean-targets: - - build - - dbt_modules +- build +- dbt_modules quoting: database: true schema: false @@ -44,29 +44,29 @@ models: +tags: airbyte_internal_views +materialized: view dispatch: - - macro_namespace: dbt_utils - search_order: - - airbyte_utils - - dbt_utils +- macro_namespace: dbt_utils + search_order: + - airbyte_utils + - dbt_utils vars: json_column: _airbyte_data models_to_source: - exchange_rate_ab1: test_normalization._airbyte_raw_exchange_rate - exchange_rate_ab2: test_normalization._airbyte_raw_exchange_rate - exchange_rate_ab3: test_normalization._airbyte_raw_exchange_rate - exchange_rate: test_normalization._airbyte_raw_exchange_rate - dedup_exchange_rate_ab1: test_normalization._airbyte_raw_dedup_exchange_rate - dedup_exchange_rate_ab2: test_normalization._airbyte_raw_dedup_exchange_rate - dedup_exchange_rate_stg: test_normalization._airbyte_raw_dedup_exchange_rate - dedup_exchange_rate_scd: test_normalization._airbyte_raw_dedup_exchange_rate - dedup_exchange_rate: test_normalization._airbyte_raw_dedup_exchange_rate - renamed_dedup_cdc_excluded_ab1: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - renamed_dedup_cdc_excluded_ab2: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - renamed_dedup_cdc_excluded_stg: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - renamed_dedup_cdc_excluded_scd: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - renamed_dedup_cdc_excluded: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - dedup_cdc_excluded_ab1: test_normalization._airbyte_raw_dedup_cdc_excluded - dedup_cdc_excluded_ab2: test_normalization._airbyte_raw_dedup_cdc_excluded - dedup_cdc_excluded_stg: test_normalization._airbyte_raw_dedup_cdc_excluded - dedup_cdc_excluded_scd: test_normalization._airbyte_raw_dedup_cdc_excluded - dedup_cdc_excluded: test_normalization._airbyte_raw_dedup_cdc_excluded + exchange_rate_ab1: test_normalization_bhhpj._airbyte_raw_exchange_rate + exchange_rate_ab2: test_normalization_bhhpj._airbyte_raw_exchange_rate + exchange_rate_ab3: test_normalization_bhhpj._airbyte_raw_exchange_rate + exchange_rate: test_normalization_bhhpj._airbyte_raw_exchange_rate + dedup_exchange_rate_ab1: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + dedup_exchange_rate_ab2: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + dedup_exchange_rate_stg: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + dedup_exchange_rate_scd: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + dedup_exchange_rate: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + renamed_dedup_cdc_excluded_ab1: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + renamed_dedup_cdc_excluded_ab2: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + renamed_dedup_cdc_excluded_stg: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + renamed_dedup_cdc_excluded_scd: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + renamed_dedup_cdc_excluded: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + dedup_cdc_excluded_ab1: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded + dedup_cdc_excluded_ab2: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded + dedup_cdc_excluded_stg: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded + dedup_cdc_excluded_scd: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded + dedup_cdc_excluded: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_dbt_project.yml index 5b2760dc9d0f..70d0b5b4fa3b 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_dbt_project.yml @@ -1,25 +1,25 @@ name: airbyte_utils -version: "1.0" +version: '1.0' config-version: 2 profile: normalize model-paths: - - models +- models docs-paths: - - docs +- docs analysis-paths: - - analysis +- analysis test-paths: - - tests +- tests seed-paths: - - data +- data macro-paths: - - macros +- macros target-path: ../build log-path: ../logs packages-install-path: /dbt clean-targets: - - build - - dbt_modules +- build +- dbt_modules quoting: database: true schema: false @@ -44,44 +44,49 @@ models: +tags: airbyte_internal_views +materialized: view dispatch: - - macro_namespace: dbt_utils - search_order: - - airbyte_utils - - dbt_utils +- macro_namespace: dbt_utils + search_order: + - airbyte_utils + - dbt_utils vars: json_column: _airbyte_data models_to_source: - exchange_rate_ab1: test_normalization._airbyte_raw_exchange_rate - exchange_rate_ab2: test_normalization._airbyte_raw_exchange_rate - exchange_rate_ab3: test_normalization._airbyte_raw_exchange_rate - exchange_rate: test_normalization._airbyte_raw_exchange_rate - dedup_exchange_rate_ab1: test_normalization._airbyte_raw_dedup_exchange_rate - dedup_exchange_rate_ab2: test_normalization._airbyte_raw_dedup_exchange_rate - dedup_exchange_rate_stg: test_normalization._airbyte_raw_dedup_exchange_rate - dedup_exchange_rate_scd: test_normalization._airbyte_raw_dedup_exchange_rate - dedup_exchange_rate: test_normalization._airbyte_raw_dedup_exchange_rate - renamed_dedup_cdc_excluded_ab1: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - renamed_dedup_cdc_excluded_ab2: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - renamed_dedup_cdc_excluded_stg: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - renamed_dedup_cdc_excluded_scd: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - renamed_dedup_cdc_excluded: test_normalization._airbyte_raw_renamed_dedup_cdc_excluded - dedup_cdc_excluded_ab1: test_normalization._airbyte_raw_dedup_cdc_excluded - dedup_cdc_excluded_ab2: test_normalization._airbyte_raw_dedup_cdc_excluded - dedup_cdc_excluded_stg: test_normalization._airbyte_raw_dedup_cdc_excluded - dedup_cdc_excluded_scd: test_normalization._airbyte_raw_dedup_cdc_excluded - dedup_cdc_excluded: test_normalization._airbyte_raw_dedup_cdc_excluded - pos_dedup_cdcx_ab1: test_normalization._airbyte_raw_pos_dedup_cdcx - pos_dedup_cdcx_ab2: test_normalization._airbyte_raw_pos_dedup_cdcx - pos_dedup_cdcx_stg: test_normalization._airbyte_raw_pos_dedup_cdcx - pos_dedup_cdcx_scd: test_normalization._airbyte_raw_pos_dedup_cdcx - pos_dedup_cdcx: test_normalization._airbyte_raw_pos_dedup_cdcx - 1_prefix_startwith_number_ab1: test_normalization._airbyte_raw_1_prefix_startwith_number - 1_prefix_startwith_number_ab2: test_normalization._airbyte_raw_1_prefix_startwith_number - 1_prefix_startwith_number_stg: test_normalization._airbyte_raw_1_prefix_startwith_number - 1_prefix_startwith_number_scd: test_normalization._airbyte_raw_1_prefix_startwith_number - 1_prefix_startwith_number: test_normalization._airbyte_raw_1_prefix_startwith_number - multiple_column_names_conflicts_ab1: test_normalization._airbyte_raw_multiple_column_names_conflicts - multiple_column_names_conflicts_ab2: test_normalization._airbyte_raw_multiple_column_names_conflicts - multiple_column_names_conflicts_stg: test_normalization._airbyte_raw_multiple_column_names_conflicts - multiple_column_names_conflicts_scd: test_normalization._airbyte_raw_multiple_column_names_conflicts - multiple_column_names_conflicts: test_normalization._airbyte_raw_multiple_column_names_conflicts + exchange_rate_ab1: test_normalization_bhhpj._airbyte_raw_exchange_rate + exchange_rate_ab2: test_normalization_bhhpj._airbyte_raw_exchange_rate + exchange_rate_ab3: test_normalization_bhhpj._airbyte_raw_exchange_rate + exchange_rate: test_normalization_bhhpj._airbyte_raw_exchange_rate + dedup_exchange_rate_ab1: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + dedup_exchange_rate_ab2: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + dedup_exchange_rate_stg: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + dedup_exchange_rate_scd: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + dedup_exchange_rate: test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate + renamed_dedup_cdc_excluded_ab1: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + renamed_dedup_cdc_excluded_ab2: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + renamed_dedup_cdc_excluded_stg: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + renamed_dedup_cdc_excluded_scd: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + renamed_dedup_cdc_excluded: test_normalization_bhhpj._airbyte_raw_renamed_dedup_cdc_excluded + dedup_cdc_excluded_ab1: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded + dedup_cdc_excluded_ab2: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded + dedup_cdc_excluded_stg: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded + dedup_cdc_excluded_scd: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded + dedup_cdc_excluded: test_normalization_bhhpj._airbyte_raw_dedup_cdc_excluded + pos_dedup_cdcx_ab1: test_normalization_bhhpj._airbyte_raw_pos_dedup_cdcx + pos_dedup_cdcx_ab2: test_normalization_bhhpj._airbyte_raw_pos_dedup_cdcx + pos_dedup_cdcx_stg: test_normalization_bhhpj._airbyte_raw_pos_dedup_cdcx + pos_dedup_cdcx_scd: test_normalization_bhhpj._airbyte_raw_pos_dedup_cdcx + pos_dedup_cdcx: test_normalization_bhhpj._airbyte_raw_pos_dedup_cdcx + 1_prefix_startwith_number_ab1: test_normalization_bhhpj._airbyte_raw_1_prefix_startwith_number + 1_prefix_startwith_number_ab2: test_normalization_bhhpj._airbyte_raw_1_prefix_startwith_number + 1_prefix_startwith_number_stg: test_normalization_bhhpj._airbyte_raw_1_prefix_startwith_number + 1_prefix_startwith_number_scd: test_normalization_bhhpj._airbyte_raw_1_prefix_startwith_number + 1_prefix_startwith_number: test_normalization_bhhpj._airbyte_raw_1_prefix_startwith_number + multiple_column_names_conflicts_ab1: test_normalization_bhhpj._airbyte_raw_multiple_column_names_conflicts + multiple_column_names_conflicts_ab2: test_normalization_bhhpj._airbyte_raw_multiple_column_names_conflicts + multiple_column_names_conflicts_stg: test_normalization_bhhpj._airbyte_raw_multiple_column_names_conflicts + multiple_column_names_conflicts_scd: test_normalization_bhhpj._airbyte_raw_multiple_column_names_conflicts + multiple_column_names_conflicts: test_normalization_bhhpj._airbyte_raw_multiple_column_names_conflicts + types_testing_ab1: test_normalization_bhhpj._airbyte_raw_types_testing + types_testing_ab2: test_normalization_bhhpj._airbyte_raw_types_testing + types_testing_stg: test_normalization_bhhpj._airbyte_raw_types_testing + types_testing_scd: test_normalization_bhhpj._airbyte_raw_types_testing + types_testing: test_normalization_bhhpj._airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql index 2d58288b13ea..3c1032d3297f 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql @@ -2,7 +2,7 @@ create table - "integrationtests"."test_normalization"."dedup_exchange_rate_scd__dbt_tmp" + "integrationtests".test_normalization_bhhpj."dedup_exchange_rate_scd" compound sortkey(_airbyte_active_row,_airbyte_unique_key_scd,_airbyte_emitted_at) @@ -14,8 +14,8 @@ with input_data as ( select * - from "integrationtests"._airbyte_test_normalization."dedup_exchange_rate_stg" - -- dedup_exchange_rate from "integrationtests".test_normalization._airbyte_raw_dedup_exchange_rate + from "integrationtests"._airbyte_test_normalization_bhhpj."dedup_exchange_rate_stg" + -- dedup_exchange_rate from "integrationtests".test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate ), scd_data as ( diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql index b7670479fa0c..b6903fe4ceb0 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql @@ -2,7 +2,7 @@ create table - "integrationtests"."test_normalization"."dedup_exchange_rate__dbt_tmp" + "integrationtests".test_normalization_bhhpj."dedup_exchange_rate" compound sortkey(_airbyte_unique_key,_airbyte_emitted_at) @@ -10,7 +10,7 @@ as ( -- Final base SQL model --- depends_on: "integrationtests".test_normalization."dedup_exchange_rate_scd" +-- depends_on: "integrationtests".test_normalization_bhhpj."dedup_exchange_rate_scd" select _airbyte_unique_key, id, @@ -25,8 +25,8 @@ select _airbyte_emitted_at, getdate() as _airbyte_normalized_at, _airbyte_dedup_exchange_rate_hashid -from "integrationtests".test_normalization."dedup_exchange_rate_scd" --- dedup_exchange_rate from "integrationtests".test_normalization._airbyte_raw_dedup_exchange_rate +from "integrationtests".test_normalization_bhhpj."dedup_exchange_rate_scd" +-- dedup_exchange_rate from "integrationtests".test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate where 1 = 1 and _airbyte_active_row = 1 diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_tables/test_normalization/exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_tables/test_normalization/exchange_rate.sql index f2537f70055b..e2bd3830cb42 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_tables/test_normalization/exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_tables/test_normalization/exchange_rate.sql @@ -1,7 +1,7 @@ create table - "integrationtests".test_normalization."exchange_rate__dbt_tmp" + "integrationtests".test_normalization_bhhpj."exchange_rate__dbt_tmp" compound sortkey(_airbyte_emitted_at) @@ -11,7 +11,7 @@ with __dbt__cte__exchange_rate_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization._airbyte_raw_exchange_rate +-- depends_on: "integrationtests".test_normalization_bhhpj._airbyte_raw_exchange_rate select case when _airbyte_data."id" != '' then _airbyte_data."id" end as id, case when _airbyte_data."currency" != '' then _airbyte_data."currency" end as currency, @@ -25,7 +25,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization._airbyte_raw_exchange_rate as table_alias +from "integrationtests".test_normalization_bhhpj._airbyte_raw_exchange_rate as table_alias -- exchange_rate where 1 = 1 ), __dbt__cte__exchange_rate_ab2 as ( @@ -87,6 +87,6 @@ select getdate() as _airbyte_normalized_at, _airbyte_exchange_rate_hashid from __dbt__cte__exchange_rate_ab3 --- exchange_rate from "integrationtests".test_normalization._airbyte_raw_exchange_rate +-- exchange_rate from "integrationtests".test_normalization_bhhpj._airbyte_raw_exchange_rate where 1 = 1 ); \ No newline at end of file diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql index f20d8dbdc37b..903a3141f625 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql @@ -1,11 +1,11 @@ - create view "integrationtests"._airbyte_test_normalization."dedup_exchange_rate_stg__dbt_tmp" as ( + create view "integrationtests"._airbyte_test_normalization_bhhpj."dedup_exchange_rate_stg__dbt_tmp" as ( with __dbt__cte__dedup_exchange_rate_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization._airbyte_raw_dedup_exchange_rate +-- depends_on: "integrationtests".test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate select case when _airbyte_data."id" != '' then _airbyte_data."id" end as id, case when _airbyte_data."currency" != '' then _airbyte_data."currency" end as currency, @@ -18,7 +18,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization._airbyte_raw_dedup_exchange_rate as table_alias +from "integrationtests".test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate as table_alias -- dedup_exchange_rate where 1 = 1 diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_views/test_normalization/multiple_column_names_conflicts_stg.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_views/test_normalization/multiple_column_names_conflicts_stg.sql index 37786e8c5256..b496abf0c5ec 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_views/test_normalization/multiple_column_names_conflicts_stg.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/first_output/airbyte_views/test_normalization/multiple_column_names_conflicts_stg.sql @@ -1,11 +1,11 @@ - create view "integrationtests"._airbyte_test_normalization."multiple_column_names_conflicts_stg__dbt_tmp" as ( + create view "integrationtests"._airbyte_test_normalization_bhhpj."multiple_column_names_conflicts_stg__dbt_tmp" as ( with __dbt__cte__multiple_column_names_conflicts_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization._airbyte_raw_multiple_column_names_conflicts +-- depends_on: "integrationtests".test_normalization_bhhpj._airbyte_raw_multiple_column_names_conflicts select case when _airbyte_data."id" != '' then _airbyte_data."id" end as id, case when _airbyte_data."User Id" != '' then _airbyte_data."User Id" end as "user id", @@ -17,7 +17,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization._airbyte_raw_multiple_column_names_conflicts as table_alias +from "integrationtests".test_normalization_bhhpj._airbyte_raw_multiple_column_names_conflicts as table_alias -- multiple_column_names_conflicts where 1 = 1 diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab1.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab1.sql index 17c4a88a2059..b8200f8bf679 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab1.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab1.sql @@ -1,11 +1,11 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_bhhpj", tags = [ "top-level-intermediate" ] ) }} -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} +-- depends_on: {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} select {{ json_extract_scalar('_airbyte_data', ['id'], ['id']) }} as id, {{ json_extract_scalar('_airbyte_data', ['currency'], ['currency']) }} as currency, @@ -18,7 +18,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, {{ current_timestamp() }} as _airbyte_normalized_at -from {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} as table_alias +from {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} as table_alias -- dedup_exchange_rate where 1 = 1 {{ incremental_clause('_airbyte_emitted_at', this) }} diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab2.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab2.sql index 796d4205f5ae..420c7c986975 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab2.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab2.sql @@ -1,7 +1,7 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_bhhpj", tags = [ "top-level-intermediate" ] ) }} -- SQL model to cast each column to its adequate SQL type converted from the JSON schema type diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql index 683191e161c5..b716e29bdf6e 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql @@ -1,7 +1,7 @@ {{ config( sort = ["_airbyte_active_row", "_airbyte_unique_key_scd", "_airbyte_emitted_at"], unique_key = "_airbyte_unique_key_scd", - schema = "test_normalization", + schema = "test_normalization_bhhpj", post_hook = [" {% set final_table_relation = adapter.get_relation( @@ -48,7 +48,7 @@ -- We have to have a non-empty query, so just do a noop delete delete from {{ this }} where 1=0 {% endif %} - ","drop view _airbyte_test_normalization.dedup_exchange_rate_stg"], + ","drop view _airbyte_test_normalization_bhhpj.dedup_exchange_rate_stg"], tags = [ "top-level" ] ) }} -- depends_on: ref('dedup_exchange_rate_stg') @@ -59,7 +59,7 @@ new_data as ( select * from {{ ref('dedup_exchange_rate_stg') }} - -- dedup_exchange_rate from {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} + -- dedup_exchange_rate from {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} where 1 = 1 {{ incremental_clause('_airbyte_emitted_at', this) }} ), @@ -97,7 +97,7 @@ input_data as ( input_data as ( select * from {{ ref('dedup_exchange_rate_stg') }} - -- dedup_exchange_rate from {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} + -- dedup_exchange_rate from {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} ), {% endif %} scd_data as ( diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_incremental/test_normalization/dedup_exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_incremental/test_normalization/dedup_exchange_rate.sql index d8b57a81b7cf..8f8fd8c8e9bc 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_incremental/test_normalization/dedup_exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_incremental/test_normalization/dedup_exchange_rate.sql @@ -1,7 +1,7 @@ {{ config( sort = ["_airbyte_unique_key", "_airbyte_emitted_at"], unique_key = "_airbyte_unique_key", - schema = "test_normalization", + schema = "test_normalization_bhhpj", tags = [ "top-level" ] ) }} -- Final base SQL model @@ -21,7 +21,7 @@ select {{ current_timestamp() }} as _airbyte_normalized_at, _airbyte_dedup_exchange_rate_hashid from {{ ref('dedup_exchange_rate_scd') }} --- dedup_exchange_rate from {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} +-- dedup_exchange_rate from {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} where 1 = 1 and _airbyte_active_row = 1 {{ incremental_clause('_airbyte_emitted_at', this) }} diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_tables/test_normalization/exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_tables/test_normalization/exchange_rate.sql index dcf6f501bc35..a66a0b168c2e 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_tables/test_normalization/exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_tables/test_normalization/exchange_rate.sql @@ -1,7 +1,7 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "test_normalization", + schema = "test_normalization_bhhpj", tags = [ "top-level" ] ) }} -- Final base SQL model @@ -21,6 +21,6 @@ select {{ current_timestamp() }} as _airbyte_normalized_at, _airbyte_exchange_rate_hashid from {{ ref('exchange_rate_ab3') }} --- exchange_rate from {{ source('test_normalization', '_airbyte_raw_exchange_rate') }} +-- exchange_rate from {{ source('test_normalization_bhhpj', '_airbyte_raw_exchange_rate') }} where 1 = 1 diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql index 8de81a6690f8..db45cc80a67a 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql @@ -1,7 +1,7 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_bhhpj", tags = [ "top-level-intermediate" ] ) }} -- SQL model to build a hash column based on the values of this record diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/sources.yml index 97bf0d05cbd4..6aa768851a80 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/models/generated/sources.yml @@ -1,6 +1,6 @@ version: 2 sources: -- name: test_normalization +- name: test_normalization_bhhpj quoting: database: true schema: false @@ -13,3 +13,4 @@ sources: - name: _airbyte_raw_multiple_column_names_conflicts - name: _airbyte_raw_pos_dedup_cdcx - name: _airbyte_raw_renamed_dedup_cdc_excluded + - name: _airbyte_raw_types_testing diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab1.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab1.sql index eca4c17d59fb..cfb1d029d88f 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab1.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab1.sql @@ -1,11 +1,11 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_bhhpj", tags = [ "top-level-intermediate" ] ) }} -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} +-- depends_on: {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} select {{ json_extract_scalar('_airbyte_data', ['id'], ['id']) }} as id, {{ json_extract_scalar('_airbyte_data', ['currency'], ['currency']) }} as currency, @@ -18,7 +18,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, {{ current_timestamp() }} as _airbyte_normalized_at -from {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} as table_alias +from {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} as table_alias -- dedup_exchange_rate where 1 = 1 {{ incremental_clause('_airbyte_emitted_at', this) }} diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab2.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab2.sql index bf26dc2829f0..2a9275c69a1e 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab2.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_ctes/test_normalization/dedup_exchange_rate_ab2.sql @@ -1,7 +1,7 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_bhhpj", tags = [ "top-level-intermediate" ] ) }} -- SQL model to cast each column to its adequate SQL type converted from the JSON schema type diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql index 2582b1213c70..9f8c382ff834 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql @@ -1,7 +1,7 @@ {{ config( sort = ["_airbyte_active_row", "_airbyte_unique_key_scd", "_airbyte_emitted_at"], unique_key = "_airbyte_unique_key_scd", - schema = "test_normalization", + schema = "test_normalization_bhhpj", post_hook = [" {% set final_table_relation = adapter.get_relation( @@ -48,7 +48,7 @@ -- We have to have a non-empty query, so just do a noop delete delete from {{ this }} where 1=0 {% endif %} - ","drop view _airbyte_test_normalization.dedup_exchange_rate_stg"], + ","drop view _airbyte_test_normalization_bhhpj.dedup_exchange_rate_stg"], tags = [ "top-level" ] ) }} -- depends_on: ref('dedup_exchange_rate_stg') @@ -59,7 +59,7 @@ new_data as ( select * from {{ ref('dedup_exchange_rate_stg') }} - -- dedup_exchange_rate from {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} + -- dedup_exchange_rate from {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} where 1 = 1 {{ incremental_clause('_airbyte_emitted_at', this) }} ), @@ -97,7 +97,7 @@ input_data as ( input_data as ( select * from {{ ref('dedup_exchange_rate_stg') }} - -- dedup_exchange_rate from {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} + -- dedup_exchange_rate from {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} ), {% endif %} scd_data as ( diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_incremental/test_normalization/dedup_exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_incremental/test_normalization/dedup_exchange_rate.sql index 421177e81179..c5fed3b30237 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_incremental/test_normalization/dedup_exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_incremental/test_normalization/dedup_exchange_rate.sql @@ -1,7 +1,7 @@ {{ config( sort = ["_airbyte_unique_key", "_airbyte_emitted_at"], unique_key = "_airbyte_unique_key", - schema = "test_normalization", + schema = "test_normalization_bhhpj", tags = [ "top-level" ] ) }} -- Final base SQL model @@ -21,7 +21,7 @@ select {{ current_timestamp() }} as _airbyte_normalized_at, _airbyte_dedup_exchange_rate_hashid from {{ ref('dedup_exchange_rate_scd') }} --- dedup_exchange_rate from {{ source('test_normalization', '_airbyte_raw_dedup_exchange_rate') }} +-- dedup_exchange_rate from {{ source('test_normalization_bhhpj', '_airbyte_raw_dedup_exchange_rate') }} where 1 = 1 and _airbyte_active_row = 1 {{ incremental_clause('_airbyte_emitted_at', this) }} diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_tables/test_normalization/exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_tables/test_normalization/exchange_rate.sql index e50ad0ad8a92..9a7a498cc375 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_tables/test_normalization/exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_tables/test_normalization/exchange_rate.sql @@ -1,7 +1,7 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "test_normalization", + schema = "test_normalization_bhhpj", tags = [ "top-level" ] ) }} -- Final base SQL model @@ -21,6 +21,6 @@ select {{ current_timestamp() }} as _airbyte_normalized_at, _airbyte_exchange_rate_hashid from {{ ref('exchange_rate_ab3') }} --- exchange_rate from {{ source('test_normalization', '_airbyte_raw_exchange_rate') }} +-- exchange_rate from {{ source('test_normalization_bhhpj', '_airbyte_raw_exchange_rate') }} where 1 = 1 diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql index 59153246fdb5..9d10a9ea9490 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql @@ -1,7 +1,7 @@ {{ config( sort = "_airbyte_emitted_at", unique_key = '_airbyte_ab_id', - schema = "_airbyte_test_normalization", + schema = "_airbyte_test_normalization_bhhpj", tags = [ "top-level-intermediate" ] ) }} -- SQL model to build a hash column based on the values of this record diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/sources.yml index 79ad1a1bb5c5..4daf898b3002 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/modified_models/generated/sources.yml @@ -1,12 +1,12 @@ version: 2 sources: - - name: test_normalization - quoting: - database: true - schema: false - identifier: false - tables: - - name: _airbyte_raw_dedup_cdc_excluded - - name: _airbyte_raw_dedup_exchange_rate - - name: _airbyte_raw_exchange_rate - - name: _airbyte_raw_renamed_dedup_cdc_excluded +- name: test_normalization_bhhpj + quoting: + database: true + schema: false + identifier: false + tables: + - name: _airbyte_raw_dedup_cdc_excluded + - name: _airbyte_raw_dedup_exchange_rate + - name: _airbyte_raw_exchange_rate + - name: _airbyte_raw_renamed_dedup_cdc_excluded diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql index fd784e2bb931..de775a2e5c16 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql @@ -1,13 +1,13 @@ - delete from "integrationtests".test_normalization."dedup_exchange_rate_scd" + delete from "integrationtests".test_normalization_bhhpj."dedup_exchange_rate_scd" where (_airbyte_unique_key_scd) in ( select (_airbyte_unique_key_scd) from "dedup_exchange_rate_scd__dbt_tmp" ); - insert into "integrationtests".test_normalization."dedup_exchange_rate_scd" ("_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "currency", "date", "timestamp_col", "hkd@spéçiäl & characters", "hkd_special___characters", "nzd", "usd", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid") + insert into "integrationtests".test_normalization_bhhpj."dedup_exchange_rate_scd" ("_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "currency", "date", "timestamp_col", "hkd@spéçiäl & characters", "hkd_special___characters", "nzd", "usd", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid") ( select "_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "currency", "date", "timestamp_col", "hkd@spéçiäl & characters", "hkd_special___characters", "nzd", "usd", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid" from "dedup_exchange_rate_scd__dbt_tmp" diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql index 0cda4f2641d5..372889fb42bd 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql @@ -1,13 +1,13 @@ - delete from "integrationtests".test_normalization."dedup_exchange_rate" + delete from "integrationtests".test_normalization_bhhpj."dedup_exchange_rate" where (_airbyte_unique_key) in ( select (_airbyte_unique_key) from "dedup_exchange_rate__dbt_tmp" ); - insert into "integrationtests".test_normalization."dedup_exchange_rate" ("_airbyte_unique_key", "id", "currency", "date", "timestamp_col", "hkd@spéçiäl & characters", "hkd_special___characters", "nzd", "usd", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid") + insert into "integrationtests".test_normalization_bhhpj."dedup_exchange_rate" ("_airbyte_unique_key", "id", "currency", "date", "timestamp_col", "hkd@spéçiäl & characters", "hkd_special___characters", "nzd", "usd", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid") ( select "_airbyte_unique_key", "id", "currency", "date", "timestamp_col", "hkd@spéçiäl & characters", "hkd_special___characters", "nzd", "usd", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid" from "dedup_exchange_rate__dbt_tmp" diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_tables/test_normalization/exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_tables/test_normalization/exchange_rate.sql index f2537f70055b..e2bd3830cb42 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_tables/test_normalization/exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_tables/test_normalization/exchange_rate.sql @@ -1,7 +1,7 @@ create table - "integrationtests".test_normalization."exchange_rate__dbt_tmp" + "integrationtests".test_normalization_bhhpj."exchange_rate__dbt_tmp" compound sortkey(_airbyte_emitted_at) @@ -11,7 +11,7 @@ with __dbt__cte__exchange_rate_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization._airbyte_raw_exchange_rate +-- depends_on: "integrationtests".test_normalization_bhhpj._airbyte_raw_exchange_rate select case when _airbyte_data."id" != '' then _airbyte_data."id" end as id, case when _airbyte_data."currency" != '' then _airbyte_data."currency" end as currency, @@ -25,7 +25,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization._airbyte_raw_exchange_rate as table_alias +from "integrationtests".test_normalization_bhhpj._airbyte_raw_exchange_rate as table_alias -- exchange_rate where 1 = 1 ), __dbt__cte__exchange_rate_ab2 as ( @@ -87,6 +87,6 @@ select getdate() as _airbyte_normalized_at, _airbyte_exchange_rate_hashid from __dbt__cte__exchange_rate_ab3 --- exchange_rate from "integrationtests".test_normalization._airbyte_raw_exchange_rate +-- exchange_rate from "integrationtests".test_normalization_bhhpj._airbyte_raw_exchange_rate where 1 = 1 ); \ No newline at end of file diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql index f20d8dbdc37b..903a3141f625 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/second_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql @@ -1,11 +1,11 @@ - create view "integrationtests"._airbyte_test_normalization."dedup_exchange_rate_stg__dbt_tmp" as ( + create view "integrationtests"._airbyte_test_normalization_bhhpj."dedup_exchange_rate_stg__dbt_tmp" as ( with __dbt__cte__dedup_exchange_rate_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization._airbyte_raw_dedup_exchange_rate +-- depends_on: "integrationtests".test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate select case when _airbyte_data."id" != '' then _airbyte_data."id" end as id, case when _airbyte_data."currency" != '' then _airbyte_data."currency" end as currency, @@ -18,7 +18,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization._airbyte_raw_dedup_exchange_rate as table_alias +from "integrationtests".test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate as table_alias -- dedup_exchange_rate where 1 = 1 diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql index 0e0d5f46f449..a193db25eb23 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql @@ -1,13 +1,13 @@ - delete from "integrationtests".test_normalization."dedup_exchange_rate_scd" + delete from "integrationtests".test_normalization_bhhpj."dedup_exchange_rate_scd" where (_airbyte_unique_key_scd) in ( select (_airbyte_unique_key_scd) from "dedup_exchange_rate_scd__dbt_tmp" ); - insert into "integrationtests".test_normalization."dedup_exchange_rate_scd" ("_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "currency", "new_column", "date", "timestamp_col", "hkd@spéçiäl & characters", "nzd", "usd", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid") + insert into "integrationtests".test_normalization_bhhpj."dedup_exchange_rate_scd" ("_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "currency", "new_column", "date", "timestamp_col", "hkd@spéçiäl & characters", "nzd", "usd", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid") ( select "_airbyte_unique_key", "_airbyte_unique_key_scd", "id", "currency", "new_column", "date", "timestamp_col", "hkd@spéçiäl & characters", "nzd", "usd", "_airbyte_start_at", "_airbyte_end_at", "_airbyte_active_row", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid" from "dedup_exchange_rate_scd__dbt_tmp" diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql index d506183c78f7..6afa610cc721 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_incremental/test_normalization/dedup_exchange_rate.sql @@ -1,13 +1,13 @@ - delete from "integrationtests".test_normalization."dedup_exchange_rate" + delete from "integrationtests".test_normalization_bhhpj."dedup_exchange_rate" where (_airbyte_unique_key) in ( select (_airbyte_unique_key) from "dedup_exchange_rate__dbt_tmp" ); - insert into "integrationtests".test_normalization."dedup_exchange_rate" ("_airbyte_unique_key", "id", "currency", "new_column", "date", "timestamp_col", "hkd@spéçiäl & characters", "nzd", "usd", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid") + insert into "integrationtests".test_normalization_bhhpj."dedup_exchange_rate" ("_airbyte_unique_key", "id", "currency", "new_column", "date", "timestamp_col", "hkd@spéçiäl & characters", "nzd", "usd", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid") ( select "_airbyte_unique_key", "id", "currency", "new_column", "date", "timestamp_col", "hkd@spéçiäl & characters", "nzd", "usd", "_airbyte_ab_id", "_airbyte_emitted_at", "_airbyte_normalized_at", "_airbyte_dedup_exchange_rate_hashid" from "dedup_exchange_rate__dbt_tmp" diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_tables/test_normalization/exchange_rate.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_tables/test_normalization/exchange_rate.sql index 1fa0ba1e56c4..031baa2a7efb 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_tables/test_normalization/exchange_rate.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_tables/test_normalization/exchange_rate.sql @@ -1,7 +1,7 @@ create table - "integrationtests".test_normalization."exchange_rate__dbt_tmp" + "integrationtests".test_normalization_bhhpj."exchange_rate__dbt_tmp" compound sortkey(_airbyte_emitted_at) @@ -11,7 +11,7 @@ with __dbt__cte__exchange_rate_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization._airbyte_raw_exchange_rate +-- depends_on: "integrationtests".test_normalization_bhhpj._airbyte_raw_exchange_rate select case when _airbyte_data."id" != '' then _airbyte_data."id" end as id, case when _airbyte_data."currency" != '' then _airbyte_data."currency" end as currency, @@ -25,7 +25,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization._airbyte_raw_exchange_rate as table_alias +from "integrationtests".test_normalization_bhhpj._airbyte_raw_exchange_rate as table_alias -- exchange_rate where 1 = 1 ), __dbt__cte__exchange_rate_ab2 as ( @@ -89,6 +89,6 @@ select getdate() as _airbyte_normalized_at, _airbyte_exchange_rate_hashid from __dbt__cte__exchange_rate_ab3 --- exchange_rate from "integrationtests".test_normalization._airbyte_raw_exchange_rate +-- exchange_rate from "integrationtests".test_normalization_bhhpj._airbyte_raw_exchange_rate where 1 = 1 ); \ No newline at end of file diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql index c7ee5d552f22..8c9d36dd07d1 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/redshift/test_simple_streams/third_output/airbyte_views/test_normalization/dedup_exchange_rate_stg.sql @@ -1,11 +1,11 @@ - create view "integrationtests"._airbyte_test_normalization."dedup_exchange_rate_stg__dbt_tmp" as ( + create view "integrationtests"._airbyte_test_normalization_bhhpj."dedup_exchange_rate_stg__dbt_tmp" as ( with __dbt__cte__dedup_exchange_rate_ab1 as ( -- SQL model to parse JSON blob stored in a single column and extract into separated field columns as described by the JSON Schema --- depends_on: "integrationtests".test_normalization._airbyte_raw_dedup_exchange_rate +-- depends_on: "integrationtests".test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate select case when _airbyte_data."id" != '' then _airbyte_data."id" end as id, case when _airbyte_data."currency" != '' then _airbyte_data."currency" end as currency, @@ -18,7 +18,7 @@ select _airbyte_ab_id, _airbyte_emitted_at, getdate() as _airbyte_normalized_at -from "integrationtests".test_normalization._airbyte_raw_dedup_exchange_rate as table_alias +from "integrationtests".test_normalization_bhhpj._airbyte_raw_dedup_exchange_rate as table_alias -- dedup_exchange_rate where 1 = 1 diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_nested_streams/dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_nested_streams/dbt_project.yml index 571d5bae581b..8a64d6b8085f 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_nested_streams/dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_nested_streams/dbt_project.yml @@ -1,25 +1,25 @@ name: airbyte_utils -version: "1.0" +version: '1.0' config-version: 2 profile: normalize model-paths: - - models +- models docs-paths: - - docs +- docs analysis-paths: - - analysis +- analysis test-paths: - - tests +- tests seed-paths: - - data +- data macro-paths: - - macros +- macros target-path: ../build log-path: ../logs packages-install-path: /dbt clean-targets: - - build - - dbt_modules +- build +- dbt_modules quoting: database: true schema: false @@ -43,10 +43,10 @@ models: +tags: airbyte_internal_views +materialized: view dispatch: - - macro_namespace: dbt_utils - search_order: - - airbyte_utils - - dbt_utils +- macro_namespace: dbt_utils + search_order: + - airbyte_utils + - dbt_utils vars: json_column: _airbyte_data models_to_source: @@ -84,6 +84,10 @@ vars: UNNEST_ALIAS_AB2: TEST_NORMALIZATION._AIRBYTE_RAW_UNNEST_ALIAS UNNEST_ALIAS_AB3: TEST_NORMALIZATION._AIRBYTE_RAW_UNNEST_ALIAS UNNEST_ALIAS: TEST_NORMALIZATION._AIRBYTE_RAW_UNNEST_ALIAS + ARRAYS_AB1: TEST_NORMALIZATION._AIRBYTE_RAW_ARRAYS + ARRAYS_AB2: TEST_NORMALIZATION._AIRBYTE_RAW_ARRAYS + ARRAYS_AB3: TEST_NORMALIZATION._AIRBYTE_RAW_ARRAYS + ARRAYS: TEST_NORMALIZATION._AIRBYTE_RAW_ARRAYS NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES_PARTITION_AB1: TEST_NORMALIZATION._AIRBYTE_RAW_NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES_PARTITION_AB2: TEST_NORMALIZATION._AIRBYTE_RAW_NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES_PARTITION_AB3: TEST_NORMALIZATION._AIRBYTE_RAW_NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES @@ -96,6 +100,10 @@ vars: UNNEST_ALIAS_CHILDREN_AB2: TEST_NORMALIZATION._AIRBYTE_RAW_UNNEST_ALIAS UNNEST_ALIAS_CHILDREN_AB3: TEST_NORMALIZATION._AIRBYTE_RAW_UNNEST_ALIAS UNNEST_ALIAS_CHILDREN: TEST_NORMALIZATION._AIRBYTE_RAW_UNNEST_ALIAS + ARRAYS_NESTED_ARRAY_PARENT_AB1: TEST_NORMALIZATION._AIRBYTE_RAW_ARRAYS + ARRAYS_NESTED_ARRAY_PARENT_AB2: TEST_NORMALIZATION._AIRBYTE_RAW_ARRAYS + ARRAYS_NESTED_ARRAY_PARENT_AB3: TEST_NORMALIZATION._AIRBYTE_RAW_ARRAYS + ARRAYS_NESTED_ARRAY_PARENT: TEST_NORMALIZATION._AIRBYTE_RAW_ARRAYS NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES_PARTITION_DOUBLE_ARRAY_DATA_AB1: TEST_NORMALIZATION._AIRBYTE_RAW_NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES_PARTITION_DOUBLE_ARRAY_DATA_AB2: TEST_NORMALIZATION._AIRBYTE_RAW_NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES_PARTITION_DOUBLE_ARRAY_DATA_AB3: TEST_NORMALIZATION._AIRBYTE_RAW_NESTED_STREAM_WITH_COMPLEX_COLUMNS_RESULTING_INTO_LONG_NAMES diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_nested_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_nested_streams/models/generated/sources.yml index f03c2b8ae028..b51dbe4cce7a 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_nested_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_nested_streams/models/generated/sources.yml @@ -6,6 +6,7 @@ sources: schema: false identifier: false tables: + - name: _AIRBYTE_RAW_ARRAYS - name: _AIRBYTE_RAW_CONFLICT_STREAM_ARRAY - name: _AIRBYTE_RAW_CONFLICT_STREAM_NAME - name: _AIRBYTE_RAW_CONFLICT_STREAM_SCALAR diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_simple_streams/dbt_project.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_simple_streams/dbt_project.yml index 76a02e4351f0..2b466206f083 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_simple_streams/dbt_project.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_simple_streams/dbt_project.yml @@ -1,25 +1,25 @@ name: airbyte_utils -version: "1.0" +version: '1.0' config-version: 2 profile: normalize model-paths: - - models +- models docs-paths: - - docs +- docs analysis-paths: - - analysis +- analysis test-paths: - - tests +- tests seed-paths: - - data +- data macro-paths: - - macros +- macros target-path: ../build log-path: ../logs packages-install-path: /dbt clean-targets: - - build - - dbt_modules +- build +- dbt_modules quoting: database: true schema: false @@ -43,10 +43,10 @@ models: +tags: airbyte_internal_views +materialized: view dispatch: - - macro_namespace: dbt_utils - search_order: - - airbyte_utils - - dbt_utils +- macro_namespace: dbt_utils + search_order: + - airbyte_utils + - dbt_utils vars: json_column: _airbyte_data models_to_source: @@ -84,3 +84,8 @@ vars: MULTIPLE_COLUMN_NAMES_CONFLICTS_STG: TEST_NORMALIZATION._AIRBYTE_RAW_MULTIPLE_COLUMN_NAMES_CONFLICTS MULTIPLE_COLUMN_NAMES_CONFLICTS_SCD: TEST_NORMALIZATION._AIRBYTE_RAW_MULTIPLE_COLUMN_NAMES_CONFLICTS MULTIPLE_COLUMN_NAMES_CONFLICTS: TEST_NORMALIZATION._AIRBYTE_RAW_MULTIPLE_COLUMN_NAMES_CONFLICTS + TYPES_TESTING_AB1: TEST_NORMALIZATION._AIRBYTE_RAW_TYPES_TESTING + TYPES_TESTING_AB2: TEST_NORMALIZATION._AIRBYTE_RAW_TYPES_TESTING + TYPES_TESTING_STG: TEST_NORMALIZATION._AIRBYTE_RAW_TYPES_TESTING + TYPES_TESTING_SCD: TEST_NORMALIZATION._AIRBYTE_RAW_TYPES_TESTING + TYPES_TESTING: TEST_NORMALIZATION._AIRBYTE_RAW_TYPES_TESTING diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_simple_streams/models/generated/sources.yml b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_simple_streams/models/generated/sources.yml index 1d9d7ae323bb..2932fe914c6c 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_simple_streams/models/generated/sources.yml +++ b/airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/snowflake/test_simple_streams/models/generated/sources.yml @@ -13,3 +13,4 @@ sources: - name: _AIRBYTE_RAW_MULTIPLE_COLUMN_NAMES_CONFLICTS - name: _AIRBYTE_RAW_POS_DEDUP_CDCX - name: _AIRBYTE_RAW_RENAMED_DEDUP_CDC_EXCLUDED + - name: _AIRBYTE_RAW_TYPES_TESTING diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/catalog.json b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/catalog.json index 9ce14ed53079..daef42fdf220 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/catalog.json +++ b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/catalog.json @@ -262,6 +262,31 @@ "cursor_field": [], "destination_sync_mode": "append_dedup", "primary_key": [["id"]] + }, + { + "stream": { + "name": "types_testing", + "json_schema": { + "type": ["null", "object"], + "properties": { + "id": { + "type": "integer" + }, + "airbyte_integer_column": { + "type": "number", + "airbyte_type": "integer" + }, + "nullable_airbyte_integer_column": { + "type": ["null", "number"], + "airbyte_type": "integer" + } + } + } + }, + "sync_mode": "full_refresh", + "cursor_field": [], + "destination_sync_mode": "append_dedup", + "primary_key": [["id"]] } ] } diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/messages.txt b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/messages.txt index 4a92cdb9106f..ed52bb6c8674 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/messages.txt +++ b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/messages.txt @@ -54,3 +54,14 @@ {"type": "RECORD", "record": {"stream": "1_prefix_startwith_number", "emitted_at": 1602637990800, "data": { "id": 2, "date": "2020-09-01", "text": "hi 4"}}} {"type":"RECORD","record":{"stream":"multiple_column_names_conflicts","data":{"id":1,"User Id":"chris","user_id":42,"User id":300,"user id": 102,"UserId":101},"emitted_at":1623959926}} + +# These records are verified in types_testing_incorrect_values.sql. If you add/remove entries, make sure to update that file as well. +# IMPORTANT: big_integer_column and nullable_big_integer_column were removed from catalog.json because of difficulties in implementing NUMERIC support. +# This is fine, because no major sources currently produce big_integer fields. +# After that functionality is completed, we should restore their entries to catalog.json. +# Verify max value for int64, and a 28-digit value for big_integer. (28 is larger than an int64 can handle, but still within bounds for a BigQuery NUMERIC column) +{"type":"RECORD","record":{"stream":"types_testing","data":{"id":1,"airbyte_integer_column":9223372036854775807,"nullable_airbyte_integer_column":9223372036854775807,"big_integer_column":"1234567890123456789012345678","nullable_big_integer_column":"1234567890123456789012345678"},"emitted_at":1623959926}} +# Verify max value for int64, and a negative 28-digit value for big_integer +{"type":"RECORD","record":{"stream":"types_testing","data":{"id":2,"airbyte_integer_column":-9223372036854775808,"nullable_airbyte_integer_column":-9223372036854775808,"big_integer_column":"-1234567890123456789012345678","nullable_big_integer_column":"-1234567890123456789012345678"},"emitted_at":1623959926}} +# Verify nullable values +{"type":"RECORD","record":{"stream":"types_testing","data":{"id":3,"airbyte_integer_column":0,"big_integer_column":0},"emitted_at":1623959926}} diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/replace_identifiers.json b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/replace_identifiers.json index ddb47f1fbbcb..2402097b4425 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/replace_identifiers.json +++ b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/data_input/replace_identifiers.json @@ -25,6 +25,9 @@ }, { "SIMPLE_STREAMS_SECOND_RUN_ROW_COUNTS": "simple_streams_second_run_row_counts" + }, + { + "TYPES_TESTING_INCORRECT_VALUES": "types_testing_incorrect_values" } ], "redshift": [], diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests/test_types_testing_values.sql b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests/test_types_testing_values.sql new file mode 100644 index 000000000000..41eff66fa313 --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests/test_types_testing_values.sql @@ -0,0 +1 @@ +select * from {{ ref('types_testing_incorrect_values') }} diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests_tmp/simple_streams_first_run_row_counts.sql b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests_tmp/simple_streams_first_run_row_counts.sql index c1499e9526eb..9fed54a37ead 100644 --- a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests_tmp/simple_streams_first_run_row_counts.sql +++ b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests_tmp/simple_streams_first_run_row_counts.sql @@ -37,6 +37,10 @@ union all union all select distinct 'pos_dedup_cdcx' as label, count(*) as row_count, 3 as expected_count from {{ ref('pos_dedup_cdcx') }} + +union all + select distinct 'types_testing' as label, count(*) as row_count, 3 as expected_count + from {{ ref('types_testing') }} ) select * from table_row_counts diff --git a/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests_tmp/types_testing_incorrect_values.sql b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests_tmp/types_testing_incorrect_values.sql new file mode 100644 index 000000000000..9a382eda267c --- /dev/null +++ b/airbyte-integrations/bases/base-normalization/integration_tests/resources/test_simple_streams/dbt_test_config/dbt_data_tests_tmp/types_testing_incorrect_values.sql @@ -0,0 +1,34 @@ +-- Note that we cast columns_column to string to avoid any weird numeric equality nonsense. +-- For example, in Postgres, this query returns `true`, even though the two numbers are different: (9223372036854775807 is the max value of a signed 64-bit int) +-- select (9223372036854775807 :: double precision) = (9223372036854775806 :: double precision) +-- Because a double has only 15 decimals of precision, so both values are rounded off to 9.223372036854776e+18 + +select * from {{ ref('types_testing') }} where +( + id = 1 and ( + cast(airbyte_integer_column as {{ dbt_utils.type_string() }}) != '9223372036854775807' + or cast(nullable_airbyte_integer_column as {{ dbt_utils.type_string() }}) != '9223372036854775807' + {# + or cast(big_integer_column as {{ dbt_utils.type_string() }}) != '1234567890123456789012345678' + or cast(nullable_big_integer_column as {{ dbt_utils.type_string() }}) != '1234567890123456789012345678' + #} + ) +) or ( + id = 2 and ( + cast(airbyte_integer_column as {{ dbt_utils.type_string() }}) != '-9223372036854775808' + or cast(nullable_airbyte_integer_column as {{ dbt_utils.type_string() }}) != '-9223372036854775808' + {# + or cast(big_integer_column as {{ dbt_utils.type_string() }}) != '-1234567890123456789012345678' + or cast(nullable_big_integer_column as {{ dbt_utils.type_string() }}) != '-1234567890123456789012345678' + #} + ) +) or ( + id = 3 and ( + cast(airbyte_integer_column as {{ dbt_utils.type_string() }}) != '0' + or nullable_airbyte_integer_column is not null + {# + or cast(big_integer_column as {{ dbt_utils.type_string() }}) != '0' + or nullable_big_integer_column is not null + #} + ) +) diff --git a/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py b/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py index c6a5aa2d5105..29afa625f535 100644 --- a/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py +++ b/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/stream_processor.py @@ -17,13 +17,14 @@ from normalization.transform_catalog.utils import ( is_airbyte_column, is_array, + is_big_integer, is_boolean, is_combining_node, is_date, is_datetime, is_datetime_with_timezone, is_datetime_without_timezone, - is_integer, + is_long, is_number, is_object, is_simple_property, @@ -458,11 +459,11 @@ def extract_json_column(property_name: str, json_column_name: str, definition: D if "type" in definition: if is_array(definition["type"]): json_extract = jinja_call(f"json_extract_array({json_column_name}, {json_path}, {normalized_json_path})") - if is_simple_property(definition.get("items", {"type": "object"}).get("type", "object")): + if is_simple_property(definition.get("items", {"type": "object"})): json_extract = jinja_call(f"json_extract_string_array({json_column_name}, {json_path}, {normalized_json_path})") elif is_object(definition["type"]): json_extract = jinja_call(f"json_extract('{table_alias}', {json_column_name}, {json_path}, {normalized_json_path})") - elif is_simple_property(definition["type"]): + elif is_simple_property(definition): json_extract = jinja_call(f"json_extract_scalar({json_column_name}, {json_path}, {normalized_json_path})") return f"{json_extract} as {column_name}" @@ -511,10 +512,12 @@ def cast_property_type(self, property_name: str, column_name: str, jinja_column: elif is_object(definition["type"]): sql_type = jinja_call("type_json()") # Treat simple types from narrower to wider scope type: boolean < integer < number < string - elif is_boolean(definition["type"]): + elif is_boolean(definition["type"], definition): cast_operation = jinja_call(f"cast_to_boolean({jinja_column})") return f"{cast_operation} as {column_name}" - elif is_integer(definition["type"]): + elif is_big_integer(definition): + sql_type = jinja_call("type_very_large_integer()") + elif is_long(definition["type"], definition): sql_type = jinja_call("dbt_utils.type_bigint()") elif is_number(definition["type"]): sql_type = jinja_call("dbt_utils.type_float()") @@ -713,7 +716,7 @@ def safe_cast_to_string(definition: Dict, column_name: str, destination_type: De if "type" not in definition: col = column_name - elif is_boolean(definition["type"]): + elif is_boolean(definition["type"], definition): col = f"boolean_to_string({column_name})" elif is_array(definition["type"]): col = f"array_to_string({column_name})" @@ -1460,7 +1463,7 @@ def find_properties_object(path: List[str], field: str, properties) -> Dict[str, elif "properties" in properties: # we found a properties object return {current: properties["properties"]} - elif "type" in properties and is_simple_property(properties["type"]): + elif "type" in properties and is_simple_property(properties): # we found a basic type return {current: {}} elif isinstance(properties, dict): diff --git a/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/utils.py b/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/utils.py index 3dec1da8a927..862ae722d9ed 100644 --- a/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/utils.py +++ b/airbyte-integrations/bases/base-normalization/normalization/transform_catalog/utils.py @@ -63,15 +63,26 @@ def is_number(property_type) -> bool: return property_type == "number" or "number" in property_type -def is_integer(property_type) -> bool: +def is_big_integer(definition: dict) -> bool: + return "airbyte_type" in definition and definition["airbyte_type"] == "big_integer" + + +def is_long(property_type, definition: dict) -> bool: + # Check specifically for {type: number, airbyte_type: integer} + if ( + (property_type == "number" or "number" in property_type) + and "airbyte_type" in definition + and definition["airbyte_type"] == "integer" + ): + return True if is_string(property_type) or is_number(property_type): # Handle union type, give priority to wider scope types return False return property_type == "integer" or "integer" in property_type -def is_boolean(property_type) -> bool: - if is_string(property_type) or is_number(property_type) or is_integer(property_type): +def is_boolean(property_type, definition: dict) -> bool: + if is_string(property_type) or is_number(property_type) or is_big_integer(definition) or is_long(property_type, definition): # Handle union type, give priority to wider scope types return False return property_type == "boolean" or "boolean" in property_type @@ -89,8 +100,18 @@ def is_airbyte_column(name: str) -> bool: return name.startswith("_airbyte_") -def is_simple_property(property_type) -> bool: - return is_string(property_type) or is_integer(property_type) or is_number(property_type) or is_boolean(property_type) +def is_simple_property(definition: dict) -> bool: + if "type" not in definition: + property_type = "object" + else: + property_type = definition["type"] + return ( + is_string(property_type) + or is_big_integer(definition) + or is_long(property_type, definition) + or is_number(property_type) + or is_boolean(property_type, definition) + ) def is_combining_node(properties: dict) -> Set[str]: diff --git a/airbyte-integrations/bases/debezium-v1-9-2/src/testFixtures/java/io/airbyte/integrations/debezium/CdcSourceTest.java b/airbyte-integrations/bases/debezium-v1-9-2/src/testFixtures/java/io/airbyte/integrations/debezium/CdcSourceTest.java index c2cce3d8bfc7..f6890d72bfc8 100644 --- a/airbyte-integrations/bases/debezium-v1-9-2/src/testFixtures/java/io/airbyte/integrations/debezium/CdcSourceTest.java +++ b/airbyte-integrations/bases/debezium-v1-9-2/src/testFixtures/java/io/airbyte/integrations/debezium/CdcSourceTest.java @@ -89,8 +89,8 @@ public abstract class CdcSourceTest { CatalogHelpers.createAirbyteStream( MODELS_STREAM_NAME, MODELS_SCHEMA, - Field.of(COL_ID, JsonSchemaType.NUMBER), - Field.of(COL_MAKE_ID, JsonSchemaType.NUMBER), + Field.of(COL_ID, JsonSchemaType.INTEGER), + Field.of(COL_MAKE_ID, JsonSchemaType.INTEGER), Field.of(COL_MODEL, JsonSchemaType.STRING)) .withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)) .withSourceDefinedPrimaryKey(List.of(List.of(COL_ID))))); @@ -474,8 +474,8 @@ void testCdcAndFullRefreshInSameSync() throws Exception { .withStream(CatalogHelpers.createAirbyteStream( MODELS_STREAM_NAME + "_2", MODELS_SCHEMA, - Field.of(COL_ID, JsonSchemaType.NUMBER), - Field.of(COL_MAKE_ID, JsonSchemaType.NUMBER), + Field.of(COL_ID, JsonSchemaType.INTEGER), + Field.of(COL_MAKE_ID, JsonSchemaType.INTEGER), Field.of(COL_MODEL, JsonSchemaType.STRING)) .withSupportedSyncModes( Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)) @@ -599,8 +599,8 @@ protected AirbyteCatalog expectedCatalogForDiscover() { final AirbyteStream streamWithoutPK = CatalogHelpers.createAirbyteStream( MODELS_STREAM_NAME + "_2", MODELS_SCHEMA, - Field.of(COL_ID, JsonSchemaType.NUMBER), - Field.of(COL_MAKE_ID, JsonSchemaType.NUMBER), + Field.of(COL_ID, JsonSchemaType.INTEGER), + Field.of(COL_MAKE_ID, JsonSchemaType.INTEGER), Field.of(COL_MODEL, JsonSchemaType.STRING)); streamWithoutPK.setSourceDefinedPrimaryKey(Collections.emptyList()); streamWithoutPK.setSupportedSyncModes(List.of(SyncMode.FULL_REFRESH)); @@ -609,8 +609,8 @@ protected AirbyteCatalog expectedCatalogForDiscover() { final AirbyteStream randomStream = CatalogHelpers.createAirbyteStream( MODELS_STREAM_NAME + "_random", MODELS_SCHEMA + "_random", - Field.of(COL_ID + "_random", JsonSchemaType.NUMBER), - Field.of(COL_MAKE_ID + "_random", JsonSchemaType.NUMBER), + Field.of(COL_ID + "_random", JsonSchemaType.INTEGER), + Field.of(COL_MAKE_ID + "_random", JsonSchemaType.INTEGER), Field.of(COL_MODEL + "_random", JsonSchemaType.STRING)) .withSourceDefinedCursor(true) .withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)) diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_object_test_catalog.json b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_object_test_catalog.json index 2ce588e732eb..28c7cd0fb7c8 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_object_test_catalog.json +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_object_test_catalog.json @@ -31,10 +31,6 @@ "airbyte_type": "big_number" }, "property_integer": { "type": "integer" }, - "property_big_integer": { - "type": "string", - "airbyte_type": "big_integer" - }, "property_boolean": { "type": "boolean" } } } diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_object_test_messages.txt b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_object_test_messages.txt index 15a6eebf906d..88acd94fbc6f 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_object_test_messages.txt +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_object_test_messages.txt @@ -1,2 +1,2 @@ -{"type": "RECORD", "record": {"stream": "object_array_test_1", "emitted_at": 1602637589100, "data": { "property_string" : "qqq", "property_array" : [ { "property_string": "foo bar", "property_date": "2021-01-23", "property_timestamp_with_timezone": "2022-11-22T01:23:45+00:00", "property_timestamp_without_timezone": "2022-11-22T01:23:45", "property_number": 56.78, "property_big_number": "100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.1234", "property_integer": 42, "property_big_integer": "123141241234124123141241234124123141241234124123141241234124123141241234124", "property_boolean": true } ] }}} -{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} \ No newline at end of file +{"type": "RECORD", "record": {"stream": "object_array_test_1", "emitted_at": 1602637589100, "data": { "property_string" : "qqq", "property_array" : [ { "property_string": "foo bar", "property_date": "2021-01-23", "property_timestamp_with_timezone": "2022-11-22T01:23:45+00:00", "property_timestamp_without_timezone": "2022-11-22T01:23:45", "property_number": 56.78, "property_big_number": "100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.1234", "property_integer": 42, "property_boolean": true } ] }}} +{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_test_catalog.json b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_test_catalog.json index 45e614559a14..3735017a3709 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_test_catalog.json +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_test_catalog.json @@ -52,13 +52,6 @@ "type": "integer" } }, - "array_big_integer": { - "type": ["array"], - "items": { - "type": "string", - "airbyte_type": "big_integer" - } - }, "array_boolean": { "type": ["array"], "items": { diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_test_messages.txt b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_test_messages.txt index 1c0bb5224a0a..a39ce73ac873 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_test_messages.txt +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_array_test_messages.txt @@ -1,2 +1,2 @@ -{"type": "RECORD", "record": {"stream": "array_test_1", "emitted_at": 1602637589100, "data": { "string_array" : ["foo bar", "some random special characters: ࠈൡሗ"], "array_date" : ["2021-01-23", "1504-02-29"], "array_timestamp_with_timezone" : ["2022-11-22T01:23:45+05:00", "9999-12-21T01:23:45-05:00"], "array_timestamp_without_timezone" : ["2022-11-22T01:23:45", "1504-02-29T01:23:45"], "array_number" : [56.78, 0, -12345.678], "array_big_number" : ["-12345.678", "100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.1234"], "array_integer" : [42, 0, 12345], "array_big_integer" : ["0", "123141241234124123141241234124123141241234124123141241234124123141241234124"], "array_boolean" : [true, false] }}} -{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} \ No newline at end of file +{"type": "RECORD", "record": {"stream": "array_test_1", "emitted_at": 1602637589100, "data": { "string_array" : ["foo bar", "some random special characters: ࠈൡሗ"], "array_date" : ["2021-01-23", "1504-02-29"], "array_timestamp_with_timezone" : ["2022-11-22T01:23:45+05:00", "9999-12-21T01:23:45-05:00"], "array_timestamp_without_timezone" : ["2022-11-22T01:23:45", "1504-02-29T01:23:45"], "array_number" : [56.78, 0, -12345.678], "array_big_number" : ["-12345.678", "100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.1234"], "array_integer" : [42, 0, 12345], "array_boolean" : [true, false] }}} +{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_basic_test_catalog.json b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_basic_test_catalog.json index aecc94b9d487..e0479b9030fd 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_basic_test_catalog.json +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_basic_test_catalog.json @@ -76,17 +76,6 @@ } } }, - { - "name": "big_integer_test_1", - "json_schema": { - "properties": { - "data": { - "type": "string", - "airbyte_type": "big_integer" - } - } - } - }, { "name": "boolean_test_1", "json_schema": { diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_basic_test_messages.txt b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_basic_test_messages.txt index 438ba2698dad..ebcee0c19123 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_basic_test_messages.txt +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_basic_test_messages.txt @@ -19,8 +19,5 @@ {"type": "RECORD", "record": {"stream": "integer_test_1", "emitted_at": 1602637589100, "data": { "data" : 42 }}} {"type": "RECORD", "record": {"stream": "integer_test_1", "emitted_at": 1602637589200, "data": { "data" : 0 }}} {"type": "RECORD", "record": {"stream": "integer_test_1", "emitted_at": 1602637589300, "data": { "data" : -12345 }}} -{"type": "RECORD", "record": {"stream": "big_integer_test_1", "emitted_at": 1602637589100, "data": { "data" : "123141241234124123141241234124123141241234124123141241234124123141241234124" }}} -{"type": "RECORD", "record": {"stream": "big_integer_test_1", "emitted_at": 1602637589200, "data": { "data" : "0" }}} -{"type": "RECORD", "record": {"stream": "big_integer_test_1", "emitted_at": 1602637589300, "data": { "data" : "-1234" }}} {"type": "RECORD", "record": {"stream": "boolean_test_1", "emitted_at": 1602637589100, "data": { "data" : true }}} -{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} \ No newline at end of file +{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_object_test_catalog.json b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_object_test_catalog.json index f143cbcea728..4f95a2954188 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_object_test_catalog.json +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_object_test_catalog.json @@ -32,10 +32,6 @@ "property_integer": { "type": "integer" }, - "property_big_integer": { - "type": "string", - "airbyte_type": "big_integer" - }, "property_boolean": { "type": "boolean" } diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_object_test_messages.txt b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_object_test_messages.txt index b42a63188a53..c92ed533f0c7 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_object_test_messages.txt +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/data_type_object_test_messages.txt @@ -1,2 +1,2 @@ -{"type": "RECORD", "record": {"stream": "object_test_1", "emitted_at": 1602637589100, "data": {"property_string": "foo bar", "property_date": "2021-01-23", "property_timestamp_with_timezone": "2022-11-22T01:23:45+00:00", "property_timestamp_without_timezone": "2022-11-22T01:23:45", "property_number": 56.78, "property_big_number": "100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.1234", "property_integer": 42, "property_big_integer": "123141241234124123141241234124123141241234124123141241234124123141241234124", "property_boolean": true }}} -{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} \ No newline at end of file +{"type": "RECORD", "record": {"stream": "object_test_1", "emitted_at": 1602637589100, "data": {"property_string": "foo bar", "property_date": "2021-01-23", "property_timestamp_with_timezone": "2022-11-22T01:23:45+00:00", "property_timestamp_without_timezone": "2022-11-22T01:23:45", "property_number": 56.78, "property_big_number": "100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.1234", "property_integer": 42, "property_boolean": true }}} +{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_array_test_catalog.json b/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_array_test_catalog.json index 77f33c308236..033d7266761c 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_array_test_catalog.json +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_array_test_catalog.json @@ -23,13 +23,6 @@ "type": "number", "airbyte_type": "integer" } - }, - "array_big_integer": { - "type": ["array"], - "items": { - "type": "number", - "airbyte_type": "big_integer" - } } } } diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_array_test_messages.txt b/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_array_test_messages.txt index ce69867d8af0..11662ae71ffc 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_array_test_messages.txt +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_array_test_messages.txt @@ -1,2 +1,2 @@ -{"type": "RECORD", "record": {"stream": "array_test_1", "emitted_at": 1602637589100, "data": { "array_number" : [-12345.678, 100000000000000000.1234],"array_float" : [-12345.678, 0, 1000000000000000000000000000000000000000000000000000.1234], "array_integer" : [42, 0, 12345], "array_big_integer" : [0, 1141241234124123141241234124] }}} -{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} \ No newline at end of file +{"type": "RECORD", "record": {"stream": "array_test_1", "emitted_at": 1602637589100, "data": { "array_number" : [-12345.678, 100000000000000000.1234],"array_float" : [-12345.678, 0, 1000000000000000000000000000000000000000000000000000.1234], "array_integer" : [42, 0, 12345]}} +{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_test_catalog.json b/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_test_catalog.json index 3cdb51d784e0..77f0c9c2faf0 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_test_catalog.json +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_test_catalog.json @@ -11,17 +11,6 @@ } } }, - { - "name": "big_integer_test", - "json_schema": { - "properties": { - "data": { - "type": "number", - "airbyte_type": "big_integer" - } - } - } - }, { "name": "float_test", "json_schema": { diff --git a/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_test_messages.txt b/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_test_messages.txt index 5fdc9da09502..5af9c922a3cc 100644 --- a/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_test_messages.txt +++ b/airbyte-integrations/bases/standard-destination-test/src/main/resources/number_data_type_test_messages.txt @@ -1,13 +1,10 @@ {"type": "RECORD", "record": {"stream": "int_test", "emitted_at": 1602637589100, "data": { "data" : 42 }}} {"type": "RECORD", "record": {"stream": "int_test", "emitted_at": 1602637589200, "data": { "data" : 0 }}} {"type": "RECORD", "record": {"stream": "int_test", "emitted_at": 1602637589300, "data": { "data" : -12345 }}} -{"type": "RECORD", "record": {"stream": "big_integer_test", "emitted_at": 1602637589100, "data": { "data" : 1231123412412314 }}} -{"type": "RECORD", "record": {"stream": "big_integer_test", "emitted_at": 1602637589200, "data": { "data" : 0 }}} -{"type": "RECORD", "record": {"stream": "big_integer_test", "emitted_at": 1602637589300, "data": { "data" : -1234 }}} {"type": "RECORD", "record": {"stream": "float_test", "emitted_at": 1602637589100, "data": { "data" : 56.78 }}} {"type": "RECORD", "record": {"stream": "float_test", "emitted_at": 1602637589200, "data": { "data" : 0 }}} {"type": "RECORD", "record": {"stream": "float_test", "emitted_at": 1602637589300, "data": { "data" : -12345.678 }}} {"type": "RECORD", "record": {"stream": "default_number_test", "emitted_at": 1602637589100, "data": { "data" : 10000000000000000000000.1234 }}} {"type": "RECORD", "record": {"stream": "default_number_test", "emitted_at": 1602637589200, "data": { "data" : 0 }}} {"type": "RECORD", "record": {"stream": "default_number_test", "emitted_at": 1602637589300, "data": { "data" : -12345.678 }}} -{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} \ No newline at end of file +{"type": "STATE", "state": { "data": {"start_date": "2022-02-14"}}} diff --git a/airbyte-integrations/bases/standard-source-test/src/main/java/io/airbyte/integrations/standardtest/source/AbstractSourceDatabaseTypeTest.java b/airbyte-integrations/bases/standard-source-test/src/main/java/io/airbyte/integrations/standardtest/source/AbstractSourceDatabaseTypeTest.java index 43952519bc2e..a804b2dc243b 100644 --- a/airbyte-integrations/bases/standard-source-test/src/main/java/io/airbyte/integrations/standardtest/source/AbstractSourceDatabaseTypeTest.java +++ b/airbyte-integrations/bases/standard-source-test/src/main/java/io/airbyte/integrations/standardtest/source/AbstractSourceDatabaseTypeTest.java @@ -190,7 +190,7 @@ private ConfiguredAirbyteCatalog getConfiguredCatalog() { .withStream(CatalogHelpers.createAirbyteStream( String.format("%s", test.getNameWithTestPrefix()), String.format("%s", getNameSpace()), - Field.of(getIdColumnName(), JsonSchemaType.NUMBER), + Field.of(getIdColumnName(), JsonSchemaType.INTEGER), Field.of(getTestColumnName(), test.getAirbyteType())) .withSourceDefinedCursor(true) .withSourceDefinedPrimaryKey(List.of(List.of(getIdColumnName()))) diff --git a/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaType.java b/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaType.java index 0fa759a6acbf..b9407c94b714 100644 --- a/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaType.java +++ b/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaType.java @@ -17,7 +17,7 @@ public enum JsonSchemaType { STRING("string", true, null, Schema.Type.STRING), NUMBER_INT("number", true, "integer", Schema.Type.INT), - NUMBER_LONG("number", true, "big_integer", Schema.Type.LONG), + NUMBER_BIGINT("string", true, "big_integer", Schema.Type.STRING), NUMBER_FLOAT("number", true, "float", Schema.Type.FLOAT), NUMBER("number", true, null, Schema.Type.DOUBLE), INTEGER("integer", true, null, Schema.Type.INT), diff --git a/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java b/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java index 920ddfbaa1a8..950a068ac2de 100644 --- a/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java +++ b/airbyte-integrations/connectors/destination-s3/src/main/java/io/airbyte/integrations/destination/s3/avro/JsonToAvroSchemaConverter.java @@ -218,7 +218,7 @@ Schema parseSingleType(final String fieldName, final Schema fieldSchema; switch (fieldType) { - case INTEGER, NUMBER, NUMBER_INT, NUMBER_LONG, NUMBER_FLOAT, BOOLEAN -> fieldSchema = Schema.create(fieldType.getAvroType()); + case INTEGER, NUMBER, NUMBER_INT, NUMBER_BIGINT, NUMBER_FLOAT, BOOLEAN -> fieldSchema = Schema.create(fieldType.getAvroType()); case STRING -> { if (fieldDefinition.has("format")) { final String format = fieldDefinition.get("format").asText(); diff --git a/airbyte-integrations/connectors/destination-s3/src/test/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaTypeTest.java b/airbyte-integrations/connectors/destination-s3/src/test/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaTypeTest.java index be27d9802ae4..3ecbb4d416f0 100644 --- a/airbyte-integrations/connectors/destination-s3/src/test/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaTypeTest.java +++ b/airbyte-integrations/connectors/destination-s3/src/test/java/io/airbyte/integrations/destination/s3/avro/JsonSchemaTypeTest.java @@ -29,7 +29,7 @@ public static class JsonSchemaTypeProvider implements ArgumentsProvider { public Stream provideArguments(ExtensionContext context) throws Exception { return Stream.of( Arguments.of("number", "integer", JsonSchemaType.NUMBER_INT), - Arguments.of("number", "big_integer", JsonSchemaType.NUMBER_LONG), + Arguments.of("string", "big_integer", JsonSchemaType.NUMBER_BIGINT), Arguments.of("number", "float", JsonSchemaType.NUMBER_FLOAT), Arguments.of("number", null, JsonSchemaType.NUMBER), Arguments.of("string", null, JsonSchemaType.STRING), diff --git a/airbyte-integrations/connectors/destination-s3/src/test/resources/parquet/json_schema_converter/type_conversion_test_cases.json b/airbyte-integrations/connectors/destination-s3/src/test/resources/parquet/json_schema_converter/type_conversion_test_cases.json index 2262b4ff76e6..d533c0e8da2f 100644 --- a/airbyte-integrations/connectors/destination-s3/src/test/resources/parquet/json_schema_converter/type_conversion_test_cases.json +++ b/airbyte-integrations/connectors/destination-s3/src/test/resources/parquet/json_schema_converter/type_conversion_test_cases.json @@ -17,10 +17,10 @@ { "fieldName": "big_integer_field", "jsonFieldSchema": { - "type": "number", + "type": "string", "airbyte_type": "big_integer" }, - "avroFieldType": ["null", "long"] + "avroFieldType": ["null", "string"] }, { "fieldName": "float_field", @@ -72,7 +72,7 @@ "type": "number" }, { - "type": "number", + "type": "string", "airbyte_type": "big_integer" } ] @@ -81,7 +81,7 @@ "null", { "type": "array", - "items": ["null", "string", "double", "long"] + "items": ["null", "string", "double"] } ] }, @@ -93,8 +93,8 @@ "id": { "type": "integer" }, - "long_id": { - "type": "number", + "big_integer_id": { + "type": "string", "airbyte_type": "big_integer" }, "node_id": { @@ -114,8 +114,8 @@ "default": null }, { - "name": "long_id", - "type": ["null", "long"], + "name": "big_integer_id", + "type": ["null", "string"], "default": null }, { @@ -194,10 +194,10 @@ "oneOf": [ { "type": "string" }, { "type": "integer" }, - { "type": "number", "airbyte_type": "big_integer" } + { "type": "string", "airbyte_type": "big_integer" } ] }, - "avroFieldType": ["null", "string", "int", "long"] + "avroFieldType": ["null", "string", "int"] }, { "fieldName": "logical_type_date_time", diff --git a/airbyte-integrations/connectors/source-bigquery/Dockerfile b/airbyte-integrations/connectors/source-bigquery/Dockerfile index 5a9b95c44483..e6aba9ad1c7b 100644 --- a/airbyte-integrations/connectors/source-bigquery/Dockerfile +++ b/airbyte-integrations/connectors/source-bigquery/Dockerfile @@ -17,5 +17,5 @@ ENV APPLICATION source-bigquery COPY --from=build /airbyte /airbyte # Airbyte's build system uses these labels to know what to name and tag the docker images produced by this Dockerfile. -LABEL io.airbyte.version=0.1.9 +LABEL io.airbyte.version=0.2.0 LABEL io.airbyte.name=airbyte/source-bigquery diff --git a/airbyte-integrations/connectors/source-bigquery/src/test-integration/java/io/airbyte/integrations/source/bigquery/BigQuerySourceDatatypeTest.java b/airbyte-integrations/connectors/source-bigquery/src/test-integration/java/io/airbyte/integrations/source/bigquery/BigQuerySourceDatatypeTest.java index fb7fad3e5231..4d55e4a42241 100644 --- a/airbyte-integrations/connectors/source-bigquery/src/test-integration/java/io/airbyte/integrations/source/bigquery/BigQuerySourceDatatypeTest.java +++ b/airbyte-integrations/connectors/source-bigquery/src/test-integration/java/io/airbyte/integrations/source/bigquery/BigQuerySourceDatatypeTest.java @@ -87,7 +87,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("int64") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .createTablePatternSql(CREATE_SQL_PATTERN) .addInsertValues("null", "-128", "127", "9223372036854775807", "-9223372036854775808") .addExpectedValues(null, "-128", "127", "9223372036854775807", "-9223372036854775808") diff --git a/airbyte-integrations/connectors/source-jdbc/src/testFixtures/java/io/airbyte/integrations/source/jdbc/test/JdbcSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-jdbc/src/testFixtures/java/io/airbyte/integrations/source/jdbc/test/JdbcSourceAcceptanceTest.java index 6ec2a81bcf61..baa0b94a86eb 100644 --- a/airbyte-integrations/connectors/source-jdbc/src/testFixtures/java/io/airbyte/integrations/source/jdbc/test/JdbcSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-jdbc/src/testFixtures/java/io/airbyte/integrations/source/jdbc/test/JdbcSourceAcceptanceTest.java @@ -839,7 +839,7 @@ protected AirbyteCatalog getCatalog(final String defaultNamespace) { CatalogHelpers.createAirbyteStream( TABLE_NAME, defaultNamespace, - Field.of(COL_ID, JsonSchemaType.NUMBER), + Field.of(COL_ID, JsonSchemaType.INTEGER), Field.of(COL_NAME, JsonSchemaType.STRING), Field.of(COL_UPDATED_AT, JsonSchemaType.STRING)) .withSupportedSyncModes(List.of(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)) @@ -847,7 +847,7 @@ protected AirbyteCatalog getCatalog(final String defaultNamespace) { CatalogHelpers.createAirbyteStream( TABLE_NAME_WITHOUT_PK, defaultNamespace, - Field.of(COL_ID, JsonSchemaType.NUMBER), + Field.of(COL_ID, JsonSchemaType.INTEGER), Field.of(COL_NAME, JsonSchemaType.STRING), Field.of(COL_UPDATED_AT, JsonSchemaType.STRING)) .withSupportedSyncModes(List.of(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)) diff --git a/airbyte-integrations/connectors/source-mysql-strict-encrypt/Dockerfile b/airbyte-integrations/connectors/source-mysql-strict-encrypt/Dockerfile index 20eeeeb009d8..2616485a7d1d 100644 --- a/airbyte-integrations/connectors/source-mysql-strict-encrypt/Dockerfile +++ b/airbyte-integrations/connectors/source-mysql-strict-encrypt/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-mysql-strict-encrypt COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.5.17 +LABEL io.airbyte.version=0.6.0 LABEL io.airbyte.name=airbyte/source-mysql-strict-encrypt diff --git a/airbyte-integrations/connectors/source-mysql/Dockerfile b/airbyte-integrations/connectors/source-mysql/Dockerfile index 23f1e5d103a0..8a1c05da29d1 100644 --- a/airbyte-integrations/connectors/source-mysql/Dockerfile +++ b/airbyte-integrations/connectors/source-mysql/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-mysql COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.5.17 +LABEL io.airbyte.version=0.6.0 LABEL io.airbyte.name=airbyte/source-mysql diff --git a/airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSourceOperations.java b/airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSourceOperations.java index c206ac558d72..91de0cf63338 100644 --- a/airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSourceOperations.java +++ b/airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSourceOperations.java @@ -173,7 +173,8 @@ public JsonSchemaType getJsonType(MysqlType mysqlType) { case // TINYINT(1) is boolean, but it should have been converted to MysqlType.BOOLEAN in {@link // getFieldType} - TINYINT, TINYINT_UNSIGNED, SMALLINT, SMALLINT_UNSIGNED, INT, INT_UNSIGNED, MEDIUMINT, MEDIUMINT_UNSIGNED, BIGINT, BIGINT_UNSIGNED, FLOAT, FLOAT_UNSIGNED, DOUBLE, DOUBLE_UNSIGNED, DECIMAL, DECIMAL_UNSIGNED -> JsonSchemaType.NUMBER; + TINYINT, TINYINT_UNSIGNED, SMALLINT, SMALLINT_UNSIGNED, INT, MEDIUMINT, MEDIUMINT_UNSIGNED, INT_UNSIGNED, BIGINT, BIGINT_UNSIGNED -> JsonSchemaType.INTEGER; + case FLOAT, FLOAT_UNSIGNED, DOUBLE, DOUBLE_UNSIGNED, DECIMAL, DECIMAL_UNSIGNED -> JsonSchemaType.NUMBER; case BOOLEAN -> JsonSchemaType.BOOLEAN; case NULL -> JsonSchemaType.NULL; // BIT(1) is boolean, but it should have been converted to MysqlType.BOOLEAN in {@link getFieldType} diff --git a/airbyte-integrations/connectors/source-mysql/src/test-integration/java/io/airbyte/integrations/source/mysql/AbstractMySqlSourceDatatypeTest.java b/airbyte-integrations/connectors/source-mysql/src/test-integration/java/io/airbyte/integrations/source/mysql/AbstractMySqlSourceDatatypeTest.java index db889bc03702..c9972beed503 100644 --- a/airbyte-integrations/connectors/source-mysql/src/test-integration/java/io/airbyte/integrations/source/mysql/AbstractMySqlSourceDatatypeTest.java +++ b/airbyte-integrations/connectors/source-mysql/src/test-integration/java/io/airbyte/integrations/source/mysql/AbstractMySqlSourceDatatypeTest.java @@ -85,7 +85,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("tinyint") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("null", "-128", "127") .addExpectedValues(null, "-128", "127") .build()); @@ -104,7 +104,7 @@ protected void initTests() { TestDataHolder.builder() .sourceType("tinyint") .fullSourceDataType("tinyint(2)") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("null", "-128", "127") .addExpectedValues(null, "-128", "127") .build()); @@ -124,7 +124,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("smallint") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("null", "-32768", "32767") .addExpectedValues(null, "-32768", "32767") .build()); @@ -132,7 +132,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("smallint") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .fullSourceDataType("smallint zerofill") .addInsertValues("1") .addExpectedValues("1") @@ -141,7 +141,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("smallint") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .fullSourceDataType("smallint unsigned") .addInsertValues("null", "0", "65535") .addExpectedValues(null, "0", "65535") @@ -150,7 +150,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("mediumint") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("null", "-8388608", "8388607") .addExpectedValues(null, "-8388608", "8388607") .build()); @@ -158,7 +158,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("mediumint") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .fullSourceDataType("mediumint zerofill") .addInsertValues("1") .addExpectedValues("1") @@ -167,7 +167,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("int") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("null", "-2147483648", "2147483647") .addExpectedValues(null, "-2147483648", "2147483647") .build()); @@ -175,7 +175,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("int") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .fullSourceDataType("int unsigned") .addInsertValues("3428724653") .addExpectedValues("3428724653") @@ -184,7 +184,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("int") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .fullSourceDataType("int zerofill") .addInsertValues("1") .addExpectedValues("1") @@ -193,7 +193,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("bigint") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("null", "9223372036854775807") .addExpectedValues(null, "9223372036854775807") .build()); diff --git a/airbyte-integrations/connectors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMysqlSourceTest.java b/airbyte-integrations/connectors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMysqlSourceTest.java index be34c80c53a3..9a0ffef35398 100644 --- a/airbyte-integrations/connectors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMysqlSourceTest.java +++ b/airbyte-integrations/connectors/source-mysql/src/test/java/io/airbyte/integrations/source/mysql/CdcMysqlSourceTest.java @@ -294,8 +294,8 @@ protected AirbyteCatalog expectedCatalogForDiscover() { AirbyteStream streamWithoutPK = CatalogHelpers.createAirbyteStream( MODELS_STREAM_NAME + "_2", MODELS_SCHEMA, - Field.of(COL_ID, JsonSchemaType.NUMBER), - Field.of(COL_MAKE_ID, JsonSchemaType.NUMBER), + Field.of(COL_ID, JsonSchemaType.INTEGER), + Field.of(COL_MAKE_ID, JsonSchemaType.INTEGER), Field.of(COL_MODEL, JsonSchemaType.STRING)); streamWithoutPK.setSourceDefinedPrimaryKey(Collections.emptyList()); streamWithoutPK.setSupportedSyncModes(List.of(SyncMode.FULL_REFRESH)); diff --git a/airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile b/airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile index 951b6db76d98..43c49007b434 100644 --- a/airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile +++ b/airbyte-integrations/connectors/source-postgres-strict-encrypt/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres-strict-encrypt COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.4.37 +LABEL io.airbyte.version=0.4.38 LABEL io.airbyte.name=airbyte/source-postgres-strict-encrypt diff --git a/airbyte-integrations/connectors/source-postgres/Dockerfile b/airbyte-integrations/connectors/source-postgres/Dockerfile index 29f9277eb349..5c18da098743 100644 --- a/airbyte-integrations/connectors/source-postgres/Dockerfile +++ b/airbyte-integrations/connectors/source-postgres/Dockerfile @@ -16,5 +16,5 @@ ENV APPLICATION source-postgres COPY --from=build /airbyte /airbyte -LABEL io.airbyte.version=0.4.37 +LABEL io.airbyte.version=0.4.38 LABEL io.airbyte.name=airbyte/source-postgres diff --git a/airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSourceOperations.java b/airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSourceOperations.java index 2e683f9498a5..481f82304436 100644 --- a/airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSourceOperations.java +++ b/airbyte-integrations/connectors/source-postgres/src/main/java/io/airbyte/integrations/source/postgres/PostgresSourceOperations.java @@ -244,7 +244,8 @@ public JDBCType getFieldType(final JsonNode field) { public JsonSchemaType getJsonType(final JDBCType jdbcType) { return switch (jdbcType) { case BOOLEAN -> JsonSchemaType.BOOLEAN; - case TINYINT, SMALLINT, INTEGER, BIGINT, FLOAT, DOUBLE, REAL, NUMERIC, DECIMAL -> JsonSchemaType.NUMBER; + case TINYINT, SMALLINT, INTEGER, BIGINT -> JsonSchemaType.INTEGER; + case FLOAT, DOUBLE, REAL, NUMERIC, DECIMAL -> JsonSchemaType.NUMBER; case BLOB, BINARY, VARBINARY, LONGVARBINARY -> JsonSchemaType.STRING_BASE_64; case ARRAY -> JsonSchemaType.ARRAY; case DATE -> JsonSchemaType.STRING_DATE; @@ -257,10 +258,12 @@ public JsonSchemaType getJsonType(final JDBCType jdbcType) { }; } + @Override protected void putBoolean(final ObjectNode node, final String columnName, final ResultSet resultSet, final int index) throws SQLException { node.put(columnName, resultSet.getString(index).equalsIgnoreCase("t")); } + @Override protected void putBigDecimal(final ObjectNode node, final String columnName, final ResultSet resultSet, final int index) { final BigDecimal bigDecimal = DataTypeUtils.returnNullIfInvalid(() -> resultSet.getBigDecimal(index)); if (bigDecimal != null) { diff --git a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/AbstractSshPostgresSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/AbstractSshPostgresSourceAcceptanceTest.java index b25f56c6e0af..2e34b5e0629d 100644 --- a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/AbstractSshPostgresSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/AbstractSshPostgresSourceAcceptanceTest.java @@ -118,18 +118,18 @@ protected ConfiguredAirbyteCatalog getConfiguredCatalog() { .withCursorField(Lists.newArrayList("id")) .withDestinationSyncMode(DestinationSyncMode.APPEND) .withStream(CatalogHelpers.createAirbyteStream( - STREAM_NAME, - Field.of("id", JsonSchemaType.NUMBER), - Field.of("name", JsonSchemaType.STRING)) + STREAM_NAME, + Field.of("id", JsonSchemaType.INTEGER), + Field.of("name", JsonSchemaType.STRING)) .withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL))), new ConfiguredAirbyteStream() .withSyncMode(SyncMode.INCREMENTAL) .withCursorField(Lists.newArrayList("id")) .withDestinationSyncMode(DestinationSyncMode.APPEND) .withStream(CatalogHelpers.createAirbyteStream( - STREAM_NAME2, - Field.of("id", JsonSchemaType.NUMBER), - Field.of("name", JsonSchemaType.STRING)) + STREAM_NAME2, + Field.of("id", JsonSchemaType.INTEGER), + Field.of("name", JsonSchemaType.STRING)) .withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL))))); } diff --git a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/CdcPostgresSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/CdcPostgresSourceAcceptanceTest.java index c9ecad3ffae3..ac7a1762777a 100644 --- a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/CdcPostgresSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/CdcPostgresSourceAcceptanceTest.java @@ -15,6 +15,7 @@ import io.airbyte.integrations.base.ssh.SshHelpers; import io.airbyte.integrations.standardtest.source.SourceAcceptanceTest; import io.airbyte.integrations.standardtest.source.TestDestinationEnv; +import io.airbyte.integrations.util.HostPortResolver; import io.airbyte.protocol.models.CatalogHelpers; import io.airbyte.protocol.models.ConfiguredAirbyteCatalog; import io.airbyte.protocol.models.ConfiguredAirbyteStream; @@ -68,8 +69,8 @@ protected void setupEnvironment(final TestDestinationEnv environment) throws Exc .put("initial_waiting_seconds", INITIAL_WAITING_SECONDS) .build()); config = Jsons.jsonNode(ImmutableMap.builder() - .put(JdbcUtils.HOST_KEY, container.getHost()) - .put(JdbcUtils.PORT_KEY, container.getFirstMappedPort()) + .put(JdbcUtils.HOST_KEY, HostPortResolver.resolveHost(container)) + .put(JdbcUtils.PORT_KEY, HostPortResolver.resolvePort(container)) .put(JdbcUtils.DATABASE_KEY, container.getDatabaseName()) .put(JdbcUtils.SCHEMAS_KEY, List.of(NAMESPACE)) .put(JdbcUtils.USERNAME_KEY, container.getUsername()) @@ -83,8 +84,8 @@ protected void setupEnvironment(final TestDestinationEnv environment) throws Exc config.get(JdbcUtils.PASSWORD_KEY).asText(), DatabaseDriver.POSTGRESQL.getDriverClassName(), String.format(DatabaseDriver.POSTGRESQL.getUrlFormatString(), - config.get(JdbcUtils.HOST_KEY).asText(), - config.get(JdbcUtils.PORT_KEY).asInt(), + container.getHost(), + container.getFirstMappedPort(), config.get(JdbcUtils.DATABASE_KEY).asText()), SQLDialect.POSTGRES)) { final Database database = new Database(dslContext); @@ -141,20 +142,20 @@ protected ConfiguredAirbyteCatalog getConfiguredCatalog() { .withCursorField(Lists.newArrayList("id")) .withDestinationSyncMode(DestinationSyncMode.APPEND) .withStream(CatalogHelpers.createAirbyteStream( - STREAM_NAME, - NAMESPACE, - Field.of("id", JsonSchemaType.NUMBER), - Field.of("name", JsonSchemaType.STRING)) + STREAM_NAME, + NAMESPACE, + Field.of("id", JsonSchemaType.INTEGER), + Field.of("name", JsonSchemaType.STRING)) .withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL))), new ConfiguredAirbyteStream() .withSyncMode(SyncMode.INCREMENTAL) .withCursorField(Lists.newArrayList("id")) .withDestinationSyncMode(DestinationSyncMode.APPEND) .withStream(CatalogHelpers.createAirbyteStream( - STREAM_NAME2, - NAMESPACE, - Field.of("id", JsonSchemaType.NUMBER), - Field.of("name", JsonSchemaType.STRING)) + STREAM_NAME2, + NAMESPACE, + Field.of("id", JsonSchemaType.INTEGER), + Field.of("name", JsonSchemaType.STRING)) .withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL))))); } diff --git a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/CdcPostgresSourceDatatypeTest.java b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/CdcPostgresSourceDatatypeTest.java index bb5936c23364..c31991f42c92 100644 --- a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/CdcPostgresSourceDatatypeTest.java +++ b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/CdcPostgresSourceDatatypeTest.java @@ -14,6 +14,7 @@ import io.airbyte.integrations.standardtest.source.AbstractSourceDatabaseTypeTest; import io.airbyte.integrations.standardtest.source.TestDataHolder; import io.airbyte.integrations.standardtest.source.TestDestinationEnv; +import io.airbyte.integrations.util.HostPortResolver; import io.airbyte.protocol.models.JsonSchemaType; import java.util.List; import org.jooq.DSLContext; @@ -52,8 +53,8 @@ protected Database setupDatabase() throws Exception { .put("initial_waiting_seconds", INITIAL_WAITING_SECONDS) .build()); config = Jsons.jsonNode(ImmutableMap.builder() - .put(JdbcUtils.HOST_KEY, container.getHost()) - .put(JdbcUtils.PORT_KEY, container.getFirstMappedPort()) + .put(JdbcUtils.HOST_KEY, HostPortResolver.resolveHost(container)) + .put(JdbcUtils.PORT_KEY, HostPortResolver.resolvePort(container)) .put(JdbcUtils.DATABASE_KEY, container.getDatabaseName()) .put(JdbcUtils.SCHEMAS_KEY, List.of(SCHEMA_NAME)) .put(JdbcUtils.USERNAME_KEY, container.getUsername()) @@ -67,8 +68,8 @@ protected Database setupDatabase() throws Exception { config.get(JdbcUtils.PASSWORD_KEY).asText(), DatabaseDriver.POSTGRESQL.getDriverClassName(), String.format(DatabaseDriver.POSTGRESQL.getUrlFormatString(), - config.get(JdbcUtils.HOST_KEY).asText(), - config.get(JdbcUtils.PORT_KEY).asInt(), + container.getHost(), + container.getFirstMappedPort(), config.get(JdbcUtils.DATABASE_KEY).asText()), SQLDialect.POSTGRES); final Database database = new Database(dslContext); @@ -119,7 +120,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("bigint") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("-9223372036854775808", "9223372036854775807", "0", "null") .addExpectedValues("-9223372036854775808", "9223372036854775807", "0", null) .build()); @@ -127,7 +128,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("bigserial") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("1", "9223372036854775807", "0", "-9223372036854775808") .addExpectedValues("1", "9223372036854775807", "0", "-9223372036854775808") .build()); @@ -135,7 +136,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("serial") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("1", "2147483647", "0", "-2147483647") .addExpectedValues("1", "2147483647", "0", "-2147483647") .build()); @@ -143,7 +144,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("smallserial") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("1", "32767", "0", "-32767") .addExpectedValues("1", "32767", "0", "-32767") .build()); diff --git a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceAcceptanceTest.java index 0bf811596546..ca5e44db3bf2 100644 --- a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceAcceptanceTest.java @@ -13,8 +13,10 @@ import io.airbyte.db.factory.DatabaseDriver; import io.airbyte.db.jdbc.JdbcUtils; import io.airbyte.integrations.base.ssh.SshHelpers; +import io.airbyte.integrations.standardtest.source.AbstractSourceConnectorTest; import io.airbyte.integrations.standardtest.source.SourceAcceptanceTest; import io.airbyte.integrations.standardtest.source.TestDestinationEnv; +import io.airbyte.integrations.util.HostPortResolver; import io.airbyte.protocol.models.CatalogHelpers; import io.airbyte.protocol.models.ConfiguredAirbyteCatalog; import io.airbyte.protocol.models.ConfiguredAirbyteStream; @@ -46,8 +48,8 @@ protected void setupEnvironment(final TestDestinationEnv environment) throws Exc .put("method", "Standard") .build()); config = Jsons.jsonNode(ImmutableMap.builder() - .put(JdbcUtils.HOST_KEY, container.getHost()) - .put(JdbcUtils.PORT_KEY, container.getFirstMappedPort()) + .put(JdbcUtils.HOST_KEY, HostPortResolver.resolveHost(container)) + .put(JdbcUtils.PORT_KEY, HostPortResolver.resolvePort(container)) .put(JdbcUtils.DATABASE_KEY, container.getDatabaseName()) .put(JdbcUtils.SCHEMAS_KEY, Jsons.jsonNode(List.of("public"))) .put(JdbcUtils.USERNAME_KEY, container.getUsername()) @@ -61,8 +63,8 @@ protected void setupEnvironment(final TestDestinationEnv environment) throws Exc config.get(JdbcUtils.PASSWORD_KEY).asText(), DatabaseDriver.POSTGRESQL.getDriverClassName(), String.format(DatabaseDriver.POSTGRESQL.getUrlFormatString(), - config.get(JdbcUtils.HOST_KEY).asText(), - config.get(JdbcUtils.PORT_KEY).asInt(), + container.getHost(), + container.getFirstMappedPort(), config.get(JdbcUtils.DATABASE_KEY).asText()), SQLDialect.POSTGRES)) { final Database database = new Database(dslContext); diff --git a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceDatatypeTest.java b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceDatatypeTest.java index 40b01ab071df..b3945702cf36 100644 --- a/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceDatatypeTest.java +++ b/airbyte-integrations/connectors/source-postgres/src/test-integration/java/io/airbyte/integrations/io/airbyte/integration_tests/sources/PostgresSourceDatatypeTest.java @@ -14,6 +14,7 @@ import io.airbyte.integrations.standardtest.source.AbstractSourceDatabaseTypeTest; import io.airbyte.integrations.standardtest.source.TestDataHolder; import io.airbyte.integrations.standardtest.source.TestDestinationEnv; +import io.airbyte.integrations.util.HostPortResolver; import io.airbyte.protocol.models.JsonSchemaType; import java.sql.SQLException; import java.util.Set; @@ -36,8 +37,8 @@ protected Database setupDatabase() throws SQLException { .put("method", "Standard") .build()); config = Jsons.jsonNode(ImmutableMap.builder() - .put(JdbcUtils.HOST_KEY, container.getHost()) - .put(JdbcUtils.PORT_KEY, container.getFirstMappedPort()) + .put(JdbcUtils.HOST_KEY, HostPortResolver.resolveHost(container)) + .put(JdbcUtils.PORT_KEY, HostPortResolver.resolvePort(container)) .put(JdbcUtils.DATABASE_KEY, container.getDatabaseName()) .put(JdbcUtils.USERNAME_KEY, container.getUsername()) .put(JdbcUtils.PASSWORD_KEY, container.getPassword()) @@ -50,8 +51,8 @@ protected Database setupDatabase() throws SQLException { config.get(JdbcUtils.PASSWORD_KEY).asText(), DatabaseDriver.POSTGRESQL.getDriverClassName(), String.format(DatabaseDriver.POSTGRESQL.getUrlFormatString(), - config.get(JdbcUtils.HOST_KEY).asText(), - config.get(JdbcUtils.PORT_KEY).asInt(), + container.getHost(), + container.getFirstMappedPort(), config.get(JdbcUtils.DATABASE_KEY).asText()), SQLDialect.POSTGRES); final Database database = new Database(dslContext); @@ -109,7 +110,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("bigint") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("-9223372036854775808", "9223372036854775807", "0", "null") .addExpectedValues("-9223372036854775808", "9223372036854775807", "0", null) .build()); @@ -117,7 +118,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType("bigserial") - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("1", "9223372036854775807", "0", "-9223372036854775808") .addExpectedValues("1", "9223372036854775807", "0", "-9223372036854775808") .build()); @@ -271,7 +272,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType(type) - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("null", "1001", "-2147483648", "2147483647") .addExpectedValues(null, "1001", "-2147483648", "2147483647") .build()); @@ -416,7 +417,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType(type) - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("null", "-32768", "32767") .addExpectedValues(null, "-32768", "32767") .build()); @@ -426,7 +427,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType(type) - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("1", "32767", "0", "-32767") .addExpectedValues("1", "32767", "0", "-32767") .build()); @@ -436,7 +437,7 @@ protected void initTests() { addDataTypeTestData( TestDataHolder.builder() .sourceType(type) - .airbyteType(JsonSchemaType.NUMBER) + .airbyteType(JsonSchemaType.INTEGER) .addInsertValues("1", "2147483647", "0", "-2147483647") .addExpectedValues("1", "2147483647", "0", "-2147483647") .build()); diff --git a/airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresJdbcSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresJdbcSourceAcceptanceTest.java index 0d9300a2b5e5..bb8f6b715bfb 100644 --- a/airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresJdbcSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-postgres/src/test/java/io/airbyte/integrations/source/postgres/PostgresJdbcSourceAcceptanceTest.java @@ -61,6 +61,7 @@ static void init() { setEnv(EnvVariableFeatureFlags.USE_STREAM_CAPABLE_STATE, "true"); } + @Override @BeforeEach public void setup() throws Exception { final String dbName = Strings.addRandomSuffix("db", "_", 10).toLowerCase(); @@ -217,6 +218,7 @@ protected List getAirbyteMessagesSecondSync(final String streamN .collect(Collectors.toList()); } + @Override protected List getAirbyteMessagesSecondStreamWithNamespace(final String streamName2) { return getTestMessages() .stream() @@ -233,6 +235,7 @@ protected List getAirbyteMessagesSecondStreamWithNamespace(final .collect(Collectors.toList()); } + @Override protected List getAirbyteMessagesForTablesWithQuoting(final ConfiguredAirbyteStream streamForTableWithSpaces) { return getTestMessages() .stream() @@ -316,6 +319,7 @@ protected List getTestMessages() { COL_LAST_COMMENT_AT, "2006-01-01T17:23:54.123456"))))); } + @Override protected void executeStatementReadIncrementallyTwice() throws SQLException { database.execute(connection -> { connection.createStatement().execute( @@ -333,25 +337,25 @@ protected void executeStatementReadIncrementallyTwice() throws SQLException { protected AirbyteCatalog getCatalog(final String defaultNamespace) { return new AirbyteCatalog().withStreams(Lists.newArrayList( CatalogHelpers.createAirbyteStream( - TABLE_NAME, - defaultNamespace, - Field.of(COL_ID, JsonSchemaType.NUMBER), - Field.of(COL_NAME, JsonSchemaType.STRING), - Field.of(COL_UPDATED_AT, JsonSchemaType.STRING_DATE), - Field.of(COL_WAKEUP_AT, JsonSchemaType.STRING_TIME_WITH_TIMEZONE), - Field.of(COL_LAST_VISITED_AT, JsonSchemaType.STRING_TIMESTAMP_WITH_TIMEZONE), - Field.of(COL_LAST_COMMENT_AT, JsonSchemaType.STRING_TIMESTAMP_WITHOUT_TIMEZONE)) + TABLE_NAME, + defaultNamespace, + Field.of(COL_ID, JsonSchemaType.INTEGER), + Field.of(COL_NAME, JsonSchemaType.STRING), + Field.of(COL_UPDATED_AT, JsonSchemaType.STRING_DATE), + Field.of(COL_WAKEUP_AT, JsonSchemaType.STRING_TIME_WITH_TIMEZONE), + Field.of(COL_LAST_VISITED_AT, JsonSchemaType.STRING_TIMESTAMP_WITH_TIMEZONE), + Field.of(COL_LAST_COMMENT_AT, JsonSchemaType.STRING_TIMESTAMP_WITHOUT_TIMEZONE)) .withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)) .withSourceDefinedPrimaryKey(List.of(List.of(COL_ID))), CatalogHelpers.createAirbyteStream( - TABLE_NAME_WITHOUT_PK, - defaultNamespace, - Field.of(COL_ID, JsonSchemaType.NUMBER), - Field.of(COL_NAME, JsonSchemaType.STRING), - Field.of(COL_UPDATED_AT, JsonSchemaType.STRING_DATE), - Field.of(COL_WAKEUP_AT, JsonSchemaType.STRING_TIME_WITH_TIMEZONE), - Field.of(COL_LAST_VISITED_AT, JsonSchemaType.STRING_TIMESTAMP_WITH_TIMEZONE), - Field.of(COL_LAST_COMMENT_AT, JsonSchemaType.STRING_TIMESTAMP_WITHOUT_TIMEZONE)) + TABLE_NAME_WITHOUT_PK, + defaultNamespace, + Field.of(COL_ID, JsonSchemaType.INTEGER), + Field.of(COL_NAME, JsonSchemaType.STRING), + Field.of(COL_UPDATED_AT, JsonSchemaType.STRING_DATE), + Field.of(COL_WAKEUP_AT, JsonSchemaType.STRING_TIME_WITH_TIMEZONE), + Field.of(COL_LAST_VISITED_AT, JsonSchemaType.STRING_TIMESTAMP_WITH_TIMEZONE), + Field.of(COL_LAST_COMMENT_AT, JsonSchemaType.STRING_TIMESTAMP_WITHOUT_TIMEZONE)) .withSupportedSyncModes(Lists.newArrayList(SyncMode.FULL_REFRESH, SyncMode.INCREMENTAL)) .withSourceDefinedPrimaryKey(Collections.emptyList()), CatalogHelpers.createAirbyteStream( diff --git a/airbyte-integrations/connectors/source-tidb/Dockerfile b/airbyte-integrations/connectors/source-tidb/Dockerfile index 9606b576426b..4f8df8707587 100755 --- a/airbyte-integrations/connectors/source-tidb/Dockerfile +++ b/airbyte-integrations/connectors/source-tidb/Dockerfile @@ -17,5 +17,5 @@ ENV APPLICATION source-tidb COPY --from=build /airbyte /airbyte # Airbyte's build system uses these labels to know what to name and tag the docker images produced by this Dockerfile. -LABEL io.airbyte.version=0.1.5 +LABEL io.airbyte.version=0.2.0 LABEL io.airbyte.name=airbyte/source-tidb diff --git a/airbyte-integrations/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSourceOperations.java b/airbyte-integrations/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSourceOperations.java index a6d6c6b19313..a568e6ca342e 100644 --- a/airbyte-integrations/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSourceOperations.java +++ b/airbyte-integrations/connectors/source-tidb/src/main/java/io/airbyte/integrations/source/tidb/TiDBSourceOperations.java @@ -143,7 +143,8 @@ public JsonSchemaType getJsonType(MysqlType mysqlType) { case // TINYINT(1) is boolean, but it should have been converted to MysqlType.BOOLEAN in {@link // getFieldType} - TINYINT, TINYINT_UNSIGNED, SMALLINT, SMALLINT_UNSIGNED, INT, INT_UNSIGNED, MEDIUMINT, MEDIUMINT_UNSIGNED, BIGINT, BIGINT_UNSIGNED, FLOAT, FLOAT_UNSIGNED, DOUBLE, DOUBLE_UNSIGNED, DECIMAL, DECIMAL_UNSIGNED -> JsonSchemaType.NUMBER; + TINYINT, TINYINT_UNSIGNED, SMALLINT, SMALLINT_UNSIGNED, INT, MEDIUMINT, MEDIUMINT_UNSIGNED, INT_UNSIGNED, BIGINT, BIGINT_UNSIGNED -> JsonSchemaType.INTEGER; + case FLOAT, FLOAT_UNSIGNED, DOUBLE, DOUBLE_UNSIGNED, DECIMAL, DECIMAL_UNSIGNED -> JsonSchemaType.NUMBER; case BOOLEAN -> JsonSchemaType.BOOLEAN; case NULL -> JsonSchemaType.NULL; // BIT(1) is boolean, but it should have been converted to MysqlType.BOOLEAN in {@link getFieldType} diff --git a/airbyte-integrations/connectors/source-tidb/src/test-integration/java/io/airbyte/integrations/source/tidb/TiDBSourceAcceptanceTest.java b/airbyte-integrations/connectors/source-tidb/src/test-integration/java/io/airbyte/integrations/source/tidb/TiDBSourceAcceptanceTest.java index 011472e4bbcc..48a75e35c22b 100755 --- a/airbyte-integrations/connectors/source-tidb/src/test-integration/java/io/airbyte/integrations/source/tidb/TiDBSourceAcceptanceTest.java +++ b/airbyte-integrations/connectors/source-tidb/src/test-integration/java/io/airbyte/integrations/source/tidb/TiDBSourceAcceptanceTest.java @@ -56,8 +56,8 @@ protected void setupEnvironment(final TestDestinationEnv testEnv) throws Excepti "", DatabaseDriver.MYSQL.getDriverClassName(), String.format(DatabaseDriver.MYSQL.getUrlFormatString(), - config.get(JdbcUtils.HOST_KEY).asText(), - config.get(JdbcUtils.PORT_KEY).asInt(), + container.getHost(), + container.getFirstMappedPort(), config.get(JdbcUtils.DATABASE_KEY).asText()), SQLDialect.MYSQL)) { final Database database = new Database(dslContext); diff --git a/airbyte-protocol/protocol-models/src/main/java/io/airbyte/protocol/models/JsonSchemaType.java b/airbyte-protocol/protocol-models/src/main/java/io/airbyte/protocol/models/JsonSchemaType.java index e4dabfd45ce6..4885bac3fb8c 100644 --- a/airbyte-protocol/protocol-models/src/main/java/io/airbyte/protocol/models/JsonSchemaType.java +++ b/airbyte-protocol/protocol-models/src/main/java/io/airbyte/protocol/models/JsonSchemaType.java @@ -19,12 +19,14 @@ public class JsonSchemaType { public static final String TIME_WITH_TIMEZONE = "time_with_timezone"; public static final String TIMESTAMP_WITH_TIMEZONE = "timestamp_with_timezone"; public static final String TIMESTAMP_WITHOUT_TIMEZONE = "timestamp_without_timezone"; + public static final String AIRYBTE_INT_TYPE = "integer"; public static final String CONTENT_ENCODING = "contentEncoding"; public static final String BASE_64 = "base64"; public static final String AIRBYTE_TYPE = "airbyte_type"; public static final JsonSchemaType STRING = JsonSchemaType.builder(JsonSchemaPrimitive.STRING).build(); public static final JsonSchemaType NUMBER = JsonSchemaType.builder(JsonSchemaPrimitive.NUMBER).build(); + public static final JsonSchemaType INTEGER = JsonSchemaType.builder(JsonSchemaPrimitive.NUMBER).withAirbyteType(AIRYBTE_INT_TYPE).build(); public static final JsonSchemaType BOOLEAN = JsonSchemaType.builder(JsonSchemaPrimitive.BOOLEAN).build(); public static final JsonSchemaType OBJECT = JsonSchemaType.builder(JsonSchemaPrimitive.OBJECT).build(); public static final JsonSchemaType ARRAY = JsonSchemaType.builder(JsonSchemaPrimitive.ARRAY).build(); @@ -48,7 +50,7 @@ public class JsonSchemaType { .withAirbyteType(TIMESTAMP_WITHOUT_TIMEZONE).build(); public static final JsonSchemaType STRING_DATE = JsonSchemaType.builder(JsonSchemaPrimitive.STRING) .withFormat(DATE).build(); - public static final JsonSchemaType NUMBER_BIGINT = JsonSchemaType.builder(JsonSchemaPrimitive.NUMBER).withAirbyteType("big_integer").build(); + public static final JsonSchemaType NUMBER_BIGINT = JsonSchemaType.builder(JsonSchemaPrimitive.STRING).withAirbyteType("big_integer").build(); private final Map jsonSchemaTypeMap; diff --git a/airbyte-workers/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java b/airbyte-workers/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java index 1d321a4dfeeb..3291c7e09909 100644 --- a/airbyte-workers/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java +++ b/airbyte-workers/src/main/java/io/airbyte/workers/normalization/NormalizationRunnerFactory.java @@ -14,7 +14,7 @@ public class NormalizationRunnerFactory { public static final String BASE_NORMALIZATION_IMAGE_NAME = "airbyte/normalization"; - public static final String NORMALIZATION_VERSION = "0.2.11"; + public static final String NORMALIZATION_VERSION = "0.2.12"; static final Map> NORMALIZATION_MAPPING = ImmutableMap.>builder() diff --git a/docs/integrations/sources/bigquery.md b/docs/integrations/sources/bigquery.md index 6be71edaa5f9..56fcc2881e2d 100644 --- a/docs/integrations/sources/bigquery.md +++ b/docs/integrations/sources/bigquery.md @@ -88,6 +88,7 @@ Once you've configured BigQuery as a source, delete the Service Account Key from | Version | Date | Pull Request | Subject | | :--- | :--- | :--- | :--- | +| 0.2.0 | 2022-07-26 | [14362](https://github.com/airbytehq/airbyte/pull/14362) | Integral columns are now discovered as int64 fields. | | 0.1.9 | 2022-07-14 | [14574](https://github.com/airbytehq/airbyte/pull/14574) | Removed additionalProperties:false from JDBC source connectors | | 0.1.8 | 2022-06-17 | [13864](https://github.com/airbytehq/airbyte/pull/13864) | Updated stacktrace format for any trace message errors | | 0.1.7 | 2022-04-11 | [11484](https://github.com/airbytehq/airbyte/pull/11484) | BigQuery connector escape column names | diff --git a/docs/integrations/sources/mysql.md b/docs/integrations/sources/mysql.md index 6b126fa4fd3e..4ab18a83f076 100644 --- a/docs/integrations/sources/mysql.md +++ b/docs/integrations/sources/mysql.md @@ -185,6 +185,7 @@ If you do not see a type in this list, assume that it is coerced into a string. | Version | Date | Pull Request | Subject | |:--------|:-----------|:-----------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------| +| 0.6.0 | 2022-07-26 | [14362](https://github.com/airbytehq/airbyte/pull/14362) | Integral columns are now discovered as int64 fields. | | 0.5.17 | 2022-07-22 | [14714](https://github.com/airbytehq/airbyte/pull/14714) | Clarified error message when invalid cursor column selected | | 0.5.16 | 2022-07-14 | [14574](https://github.com/airbytehq/airbyte/pull/14574) | Removed additionalProperties:false from JDBC source connectors | | 0.5.15 | 2022-06-23 | [14077](https://github.com/airbytehq/airbyte/pull/14077) | Use the new state management | diff --git a/docs/integrations/sources/postgres.md b/docs/integrations/sources/postgres.md index 906da229eaed..5f5b35f57c32 100644 --- a/docs/integrations/sources/postgres.md +++ b/docs/integrations/sources/postgres.md @@ -336,6 +336,7 @@ One optimization on the Airbyte side is to break one large and long sync into mu | Version | Date | Pull Request | Subject | |:--------|:-----------|:---------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------| +| 0.5.0 | 2022-07-26 | [14362](https://github.com/airbytehq/airbyte/pull/14362) | Integral columns are now discovered as int64 fields. | | 0.4.37 | 2022-07-22 | [14714](https://github.com/airbytehq/airbyte/pull/14714) | Clarified error message when invalid cursor column selected | | 0.4.36 | 2022-07-21 | [14451](https://github.com/airbytehq/airbyte/pull/14451) | Make initial CDC waiting time configurable | | 0.4.35 | 2022-07-14 | [14574](https://github.com/airbytehq/airbyte/pull/14574) | Removed additionalProperties:false from JDBC source connectors | diff --git a/docs/integrations/sources/tidb.md b/docs/integrations/sources/tidb.md index 6292c62cf896..d0fad8f5c74a 100644 --- a/docs/integrations/sources/tidb.md +++ b/docs/integrations/sources/tidb.md @@ -120,6 +120,7 @@ Using this feature requires additional configuration, when creating the source. | Version | Date | Pull Request | Subject | | :------ | :--- | :----------- | ------- | +| 0.2.0 | 2022-07-26 | [14362](https://github.com/airbytehq/airbyte/pull/14362) | Integral columns are now discovered as int64 fields. | | 0.1.5 | 2022-07-25 | [14996](https://github.com/airbytehq/airbyte/pull/14996) | Removed additionalProperties:false from spec | | 0.1.4 | 2022-07-22 | [14714](https://github.com/airbytehq/airbyte/pull/14714) | Clarified error message when invalid cursor column selected | | 0.1.3 | 2022-07-04 | [14243](https://github.com/airbytehq/airbyte/pull/14243) | Update JDBC string builder | diff --git a/docs/understanding-airbyte/basic-normalization.md b/docs/understanding-airbyte/basic-normalization.md index c5d9b1d947eb..dce59364e26f 100644 --- a/docs/understanding-airbyte/basic-normalization.md +++ b/docs/understanding-airbyte/basic-normalization.md @@ -353,7 +353,8 @@ Therefore, in order to "upgrade" to the desired normalization version, you need | Airbyte Version | Normalization Version | Date | Pull Request | Subject | |:----------------|:----------------------|:-----------| :--- |:---------------------------------------------------------------------------| -| | 0.2.11 | 2022-07-26 | [\#13591](https://github.com/airbytehq/airbyte/pull/13591) | Handle timezone in date-time format. Parse date correct in clickhouse. | +| | 0.2.11 | 2022-07-26 | [\#13591](https://github.com/airbytehq/airbyte/pull/13591) | Updated support for integer columns. | +| | 0.2.12 | 2022-07-26 | [\#14362](https://github.com/airbytehq/airbyte/pull/14362) | Handle timezone in date-time format. Parse date correct in clickhouse. | | | 0.2.10 | 2022-07-18 | [\#14792](https://github.com/airbytehq/airbyte/pull/14792) | Add support for key pair auth for snowflake | | | 0.2.9 | 2022-07-06 | [\#14485](https://github.com/airbytehq/airbyte/pull/14485) | BigQuery partition pruning otimization | | | 0.2.8 | 2022-07-13 | [\#14522](https://github.com/airbytehq/airbyte/pull/14522) | BigQuery replaces `NULL` array entries with the string value `"NULL"` | diff --git a/docs/understanding-airbyte/supported-data-types.md b/docs/understanding-airbyte/supported-data-types.md index 296905af508d..ae20eab7de2a 100644 --- a/docs/understanding-airbyte/supported-data-types.md +++ b/docs/understanding-airbyte/supported-data-types.md @@ -17,7 +17,7 @@ This table summarizes the available types. See the [Specific Types](#specific-ty | Datetime with timezone | `{"type": "string", "format": "date-time", "airbyte_type": "timestamp_with_timezone"}` | `"2022-11-22T01:23:45+05:00"` | | Datetime without timezone | `{"type": "string", "format": "date-time", "airbyte_type": "timestamp_without_timezone"}` | `"2022-11-22T01:23:45"` | | Integer | `{"type": "integer"}` | `42` | -| Big integer (unrepresentable as a 64-bit two's complement int) | `{"type": "string", "airbyte_type": "big_integer"}` | `"123141241234124123141241234124123141241234124123141241234124123141241234124"` | +| Big integer (unrepresentable as a 64-bit two's complement int) | `{"type": "string", "airbyte_type": "big_integer"}` | `"12345678901234567890123456789012345678"` | | Number | `{"type": "number"}` | `1234.56` | | Big number (unrepresentable as a 64-bit IEEE 754 float) | `{"type": "string", "airbyte_type": "big_number"}` | `"1,000,000,...,000.1234"` with 500 0's | | Array | `{"type": "array"}`; optionally `items` and `additionalItems` | `[1, 2, 3]` | @@ -25,6 +25,8 @@ This table summarizes the available types. See the [Specific Types](#specific-ty | Untyped (i.e. any value is valid) | `{}` | | | Union | `{"anyOf": [...]}` or `{"oneOf": [...]}` | | +Note that some of these may be destination-dependent. For example, Snowflake `NUMERIC` columns can be at most 38 digits wide, but Postgres `NUMERIC` columns may have up to 131072 digits before the decimal point. + ### Record structure As a reminder, sources expose a `discover` command, which returns a list of [`AirbyteStreams`](https://github.com/airbytehq/airbyte/blob/111131a193359027d0081de1290eb4bb846662ef/airbyte-protocol/models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml#L122), and a `read` method, which emits a series of [`AirbyteRecordMessages`](https://github.com/airbytehq/airbyte/blob/111131a193359027d0081de1290eb4bb846662ef/airbyte-protocol/models/src/main/resources/airbyte_protocol/airbyte_protocol.yaml#L46-L66). The type system determines what a valid `json_schema` is for an `AirbyteStream`, which in turn dictates what messages `read` is allowed to emit.