feat(data-warehouse): added sql source v2

[Gilbert09]

Problem

• Our SQL source is slow and inefficient
  • We can't sync our own clickhouse as a source due to limitations of how big of a number is allowed in Python/C
  • Bigquery syncs aren't completing in a timely manner and so sessions are expiring mid syncs
• DLT was on an old version too - we're missing out on a bunch of fixes

Changes

• Added a new SQL Source V2
  • Most of the code is from https://github.com/dlt-hub/verified-sources/tree/master/sources/sql_database - but is fairly modified for our circumstances
  • This is only turned on for normal SQL sources (postgres/mysql/etc) for the PostHog team to do some further testing
  • This is also turned on for Bigquery (who we're the only users of) to do some more testing
• Added custom type to support JSON types in BigQuery - see Add support for JSON datatype googleapis/python-bigquery-sqlalchemy#399
• Upgraded DLT version
  • This came with some breaking changes, but all sorted in this PR

Known issues:

• ARRAY types are a bit funky and need further investigating (they're currently turned off)
• When using the new source, we can't drop the DB in tests due to an active session to it - unsure where this session is coming from, we dispose of the engine/connection and so Edit: Fixed in my last commit

Plan:

• Test this out with team 2 postgres source, may sure everything works as expected
• Test out the bigquery stuff
• Fix the above known issues
• Add support for snowflake
• Remove code flag and release to all users

Does this work well for both Cloud and self-hosted?

Hopefully

How did you test this code?

Tested extensively locally all day

[rossgray]

Just out of interest, what changes do we need to make to the default dlt source?

Most of the code is from https://github.com/dlt-hub/verified-sources/tree/master/sources/sql_database - but is fairly modified for our circumstances

Also, just for context; did v1 come from dlt too but they've since updated it, which is why we're now creating a v2?

[Gilbert09]

Just out of interest, what changes do we need to make to the default dlt source?

Most of the code is from dlt-hub/verified-sources@master/sources/sql_database - but is fairly modified for our circumstances

@rossgray From the top of my head

• Adding the ability to materialise empty tables (which DLT doesn't do by default)
• Add JSON support to BigQuery
• Closing a leaky SQL connection
• Adjusting PyArrow types to remove interval and binary columns (they're not supported by either DLT or HogQL)
• Adding in better support for planetscale DBs
• Better handling of incremental values without a default
• Adjusting how we look for primary keys
• Not using the first row of a table to determine types (e.g. if the value of a nullable column is null, then the type gets set to NoneType as opposed to the actual correct type)

I think that's everything. Basically all the fixes we've added over time to the existing SQL source

[rossgray]

Just out of interest, what changes do we need to make to the default dlt source?

Most of the code is from dlt-hub/verified-sources@master/sources/sql_database - but is fairly modified for our circumstances

@rossgray From the top of my head

• Adding the ability to materialise empty tables (which DLT doesn't do by default)
• Add JSON support to BigQuery
• Closing a leaky SQL connection
• Adjusting PyArrow types to remove interval and binary columns (they're not supported by either DLT or HogQL)
• Adding in better support for planetscale DBs
• Better handling of incremental values without a default
• Adjusting how we look for primary keys
• Not using the first row of a table to determine types (e.g. if the value of a nullable column is null, then the type gets set to NoneType as opposed to the actual correct type)

I think that's everything. Basically all the fixes we've added over time to the existing SQL source

Thanks. Wonder if we could contribute some of these changes back to dlt at some point since quite a few of these sound useful to others

[daibhin]

This feels like it could have been multiple PRs but if you're confident that it's all safely behind a FF then

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ PipelineStepFailed: Pipeline execution failed at stage extract when processing package 1730380440.9322343 with except... posthog.temporal.data_imports.pipelines.pipelin... View Issue

_{Did you find this useful? React with a 👍 or 👎}