chore: drop kafka_skip_broken_messages from kafka_events_json

[bretthoerner]

Problem

We shouldn't be sending any messages that don't deserialize properly, and so we'd like to go back to kafka_skip_broken_messages = 0 in prod.

Changes

Remake virtual Kafka table (and MV that's based on it) without the kafka_skip_broken_messages setting.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

This is modeled after a previous migration and should be reversible. Since it could pause the pipeline we should probably deploy early on a Monday and monitor it (with a revert PR ready).

[bretthoerner]

@fuziontech I'm not sure if I missed a tool to create these, I created the migration manually. I assume our migration script checks for the migration by name.

Speaking of which, there are two 0049 migrations here. I hope that's OK/known?: https://github.com/PostHog/posthog/tree/master/posthog/clickhouse/migrations

[tiina303]

lgtm

[bretthoerner]

Ping @fuziontech - looks like I need your approval.

[fuziontech]

lgtm! sorry for lag
🚢 it!

[fuziontech]

@fuziontech I'm not sure if I missed a tool to create these, I created the migration manually. I assume our migration script checks for the migration by name.

No tool to make these, just manually. Checks for i+1 where i is the current migration id.

Speaking of which, there are two 0049 migrations here. I hope that's OK/known?: https://github.com/PostHog/posthog/tree/master/posthog/clickhouse/migrations

oh, interesting. I would probably increment this to 0051 and bump the last migration to 0050

[fuziontech]

👀

[fuziontech]

Actually. Give me a second to put a PR to correct this. One of those migrations needs to be removed. One sec.

[fuziontech]

Once this lands, let's rebase this one and 🚢 it
#19110

[bretthoerner]

Rebased 🤞

[bretthoerner]

Looks good, just needs your stamp @fuziontech

[fuziontech]

just kept rolling for all green tests.
Ship it!

[bretthoerner]

Actually going to close this in favor of the upcoming _error virtual columns.

This migration will allow bad messages to kill ingestion, where as the _error columns can be forwarded and stored in a table for us to analyze (without killing ingestion).