refactor: Redshift batch export uses spmc consumer

[tomasfarias]

Problem

Redshift batch export is not using SPMC abstractions, instead it is doing its own thing. This leads to duplication and harder to read code. The main reason it wasn't migrated is that Redshift doesn't write to a file but instead to a query that then gets executed. We can still adapt the same abstractions as all other batch exports though, so that's what we do here.

Changes

• Create a new BatchExportWriter: RedshiftInsertBatchExportWriter. This new writer will write INSERT queries for Redshift.
• Redshift now runs a RedshiftConsumer that flushes these queries by executing them.
• run_consumer_loop is now run_consumer as we no longer run multiple in a loop, this may change later.
• Also run_consumer now takes an instance of Consumer which simplifies the signature slightly.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

[rossgray]

Looks good. The only slightly confusing thing is that we're trying to fit a SQL-based writer into a temporary file-writer abstraction but think this is fine for now

[posthog-bot]

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week. If you want to permanentely keep it open, use the waiting label.

[tomasfarias]

@rossgray

Looks good. The only slightly confusing thing is that we're trying to fit a SQL-based writer into a temporary file-writer abstraction but think this is fine for now

The abstraction should cover both cases: It's a writer, whether it's writing to a temporary file, or a SQL query buffer, should be left up to the subclass to decide. That being said, I am open to expressing this more clearly in the code, as it was certainly just a temporary file writer in the beginning. Maybe we need more separation between "writer" and "writee" to make it more obvious, and some of the sql buffer logic moved to a new "writee".