feat(plugin-server): project_id in clickhouse_events_json
#25873
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Twixes
force-pushed
the
clickhouse-events-json-project
branch
from
97f98ce to
c0345d0
Compare
oliverb123
reviewed
Oct 30, 2024
| const { eventUuid: uuid, event, teamId, distinctId, properties, timestamp } = preIngestionEvent | ||
|
|
||
| let team = this.teamManager.getCachedTeam(teamId) |
There was a problem hiding this comment.
Not blocking, but this feels a bit like spooky action at a distance - is there any chance we could do this in the populateTeamData step instead, where we know we're already fetching the team? I know that'd require a bit more wiring, but I'd prefer the "larger" change here I think
There was a problem hiding this comment.
Good point. Have rewritten to avoid any extra teamManager calls. Now we set the project_id here: https://github.com/PostHog/posthog/pull/25873/files#diff-51ee2dd21f9a70e91c6e2bf4a0c856040b7f5b7708cbea1c5d674fe1bd92b843R190
oliverb123
approved these changes
Oct 30, 2024
Twixes
force-pushed
the
clickhouse-events-json-project
branch
from
ea6b373 to
47ca021
Compare
An error occurred while trying to automatically change base from
group-type-project
to
master
October 31, 2024 23:12
Twixes
force-pushed
the
clickhouse-events-json-project
branch
from
47ca021 to
4e41d8a
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
To do project-based taxonomy, rather than team-based,
property-defs-rsneeds to be gettingproject_idas part of event payloads. To do this, I propose we start including propertyproject_idin theclickhouse_events_jsontopic's payloads.Changes
We're now adding
project_idright before producing events toclickhouse_events_json.We are not using this
project_idanywhere yet. The only place I'm planning to use it isproperty-defs-rsin a follow-up PR.How did you test this code?
I saw we don't have any plugin server tests observing the
clickhouse_events_jsontopic – all event processing tests fetch from ClickHouse instead. That's all good for most cases, except this particular one, asproject_idis specifically key in Kafka payloads and ignored in ClickHouse.Hence added Kafka observation to
plugin-server/tests/worker/ingestion/process-event.test.ts– inspired byplugin-server/functional_tests/scheduled-tasks-runner.test.ts– and added aproject_idassertion based on that.