Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add projection on inserted_at #17466

Merged
merged 4 commits into from
Oct 6, 2023
Merged

Conversation

tomasfarias
Copy link
Contributor

Problem

Since we added inserted_at batch exports are querying events based on inserted_at.

However, we are forced to include timestamp in the query to better utilize the sort key. It would be better if we could use a sort key based on inserted_at to optimize performance.

Moreover, due to the discrepancy between inserted_at and timestamp we are missing events that were historically loaded.

Changes

Add a migration that creates a projection with inserted_at in the sort key.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

@eboswort
Copy link

🙏 this MR would make a big difference for me and my team!

@posthog-bot
Copy link
Contributor

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.

@eboswort
Copy link

eboswort commented Oct 4, 2023

Hi @tomasfarias, would you be able to help get this MR across the finish line? Really looking forward to being able to successfully sync all my posthog data to snowflake without missing stuff.

Copy link
Member

@fuziontech fuziontech left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good to me! thanks for only doing this on 202309 partition

@fuziontech
Copy link
Member

if you wanted to limit this to 202310 I would be even happier tbh!

@tomasfarias
Copy link
Contributor Author

tomasfarias commented Oct 6, 2023

if you wanted to limit this to 202310 I would be even happier tbh!

Updated the materialization to 202310. Will merge if tests are green, thank you.

@tomasfarias tomasfarias merged commit ba61ff1 into master Oct 6, 2023
69 checks passed
@tomasfarias tomasfarias deleted the feat/projection-on-inserted-at branch October 6, 2023 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants