refactor: Postgres (+ Redshift) batch exports now async #18501
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tomasfarias
changed the title
tomasfarias
changed the title
tomasfarias
force-pushed
the
refactor/async-postgres-batch-exports
branch
from
638d5d1 to
194bce8
Compare
tomasfarias
force-pushed
the
refactor/batch-exports-logging-new
branch
2 times, most recently
from
28ca083 to
403c87d
Compare
Base automatically changed from
refactor/batch-exports-logging-new
to
master
November 13, 2023 15:07
|
Closed in favor of #18631 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
Making PG batch exports async means less blocking the main thread, more concurrent tasks, more performance! Let's do all destinations async and bring down the number of required workers!
Turns out is quite easy to make Postgres batch exports async. What's more, the new COPY API doesn't require a file, so we may try a different buffered approach in the future. I'm very excited for the future of batch exports!
Changes
asyncandawait.""instead ofNone, but I don't think that's very significant: at least semantically we are not losing or gaining anything as these fields are meant to be deprecated, and in terms of schema both are compatible.👉 Stay up-to-date with PostHog coding conventions for a smoother review.
How did you test this code?
Same tests as before, just now async too.