upgrade airbyte to 0.50.44

[fatchat]

Check Airbyte 0.50.40 with Destinations v2

• set up new airbyte on a new machine with GSM

• set up postgres destination with destinations v2

• set up and sync an avni source

• set up and sync a kobo source

• set up and sync a commcare source

• set up and sync a glific source

• set up and sync a surveycto source

• set up and sync a google sheets source

• set up a bigquery destination with destinations v2

• set up and sync an avni source

• set up and sync a kobo source

• set up and sync a commcare source

• set up and sync a glific source

• set up and sync a surveycto source

• set up and sync a google sheets source

Check that we can migrate up to 0.50.40 from an existing installation

• migrate staging to the new airbyte and sync all connections
• check if airbyte will support a postgres on destinations v1 and another on v2 simultaneously
• check if airbyte will support a bigquery on destinations v1 and another on v2 simultaneously

Update dbt code to read from flattened staging tables

• SNEHA
• SHRI
• STiR
• LAHI
• Antarang
• TAP
• Arghyam
• Janaagraha
• superset_usage
• KEF

People need to be notified

• Send out an email on Feb 27 for migration downtime on Sunday March 10

Platform updates

• Create warehouse - check that it works
• Create connection - remove norm op

[fatchat]

airbytehq#30814

[fatchat]

Postgres destination works (v2)
Avni source works

[fatchat]

Kobotoolbox source works

[fatchat]

Commcare source works

[fatchat]

on an airbyte 0.50.21 deployment:

• set up a google sheets -> bigquery sync without normalization
• saw the usual _airbyte_ab_id, _airbyte_emitted_at, _airbyte_data columns in the _airbyte_raw_* tables
• upgraded the destination (not the connector!) to use destinations v2 and ran the sync
• saw the expected columns _airbyte_raw_id, _airbyte_extracted_at, _airbyte_meta and the new dataset airbyte_internal
• also saw the data columns after normalization
• the table names did not beign with _airbyte_raw_
• normalization was still available on the connection!
• running with normalization was like running with destinations v1
• removed normalization and synced
• result was like destinations v2

[fatchat]

• it was possible to downgrade the airbyte destination to destinations v1 and sync
• the destination ID did not change across these actions

this means that we can migrate BigQuery clients one-by-one to destinations v2

BQ destination connector version: 1.9.0

[fatchat]

The Postgres destination connector versions 0.* did not support destinations v2
There were no 1.* versions
Version 2.0.1 is destinations v2, no option for v1

These upgrades therefore need to be done per-connector i.e. per workspace

[fatchat]

Synced to a Postgre destinations connector v 0.4.0 (dest v1) and got the usual _airbyte_raw_* tables with _airbyte_data etc

Upgraded the connector to v 2.0.1 (dest v2)

• the airbyte_internal schema was created
• the normalized tables were created with the new _airbyte_raw_id, _airbyte_extracted_at and _airbyte_meta columns

[fatchat]

staging is now running airbyte 0.50.44
postgres connector is 2.0.9
bigquery is 2.4.12

[fatchat]

production is now running airbyte 0.50.44
postgres connector is 2.0.9
bigquery is 2.4.12

Important: This command can only be run by one of the repository admins:
Usage: ./tools/bin/get_repo_admins.sh <personal_access_token>