Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(cdp): Use cyclotron part 2 #24746

Merged
merged 91 commits into from
Sep 10, 2024
Merged

feat(cdp): Use cyclotron part 2 #24746

merged 91 commits into from
Sep 10, 2024

Conversation

benjackwhite
Copy link
Contributor

@benjackwhite benjackwhite commented Sep 2, 2024

Problem

The first PR refactored the code to make it easier to enqueue and read from Cyclotron instead of Kafka. This implements it

Changes

  • Adds the extra work needed to allow writing and reading from Cyclotron postgres instead of kafka
  • Works e2e with flags so we can slowly start testing it

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

# Conflicts:
#	plugin-server/tests/cdp/cdp-api.test.ts
#	plugin-server/tests/cdp/cdp-processed-events-consumer.test.ts
#	plugin-server/tests/cdp/hog-executor.test.ts
# Conflicts:
#	plugin-server/src/cdp/hog-executor.ts
# Conflicts:
#	plugin-server/src/cdp/cdp-consumers.ts
#	plugin-server/src/main/pluginsServer.ts
# Conflicts:
#	plugin-server/src/cdp/cdp-consumers.ts
#	plugin-server/src/main/pluginsServer.ts
@benjackwhite benjackwhite requested review from oliverb123, bretthoerner and mariusandra and removed request for oliverb123 and bretthoerner September 5, 2024 11:51
Copy link
Collaborator

@mariusandra mariusandra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested locally, everything seems to work. Two issues worth addressing though:

  • I think I found a bug with the cyclotron/kafka split (comment inline)
  • bin/start-cyclotron should also use the new env.

plugin-server/src/cdp/cdp-consumers.ts Show resolved Hide resolved
plugin-server/src/cdp/cdp-consumers.ts Show resolved Hide resolved
Copy link
Collaborator

@mariusandra mariusandra left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Go forth and multiply!

@benjackwhite benjackwhite merged commit 7fd0fbb into master Sep 10, 2024
87 checks passed
@benjackwhite benjackwhite deleted the feat/cdp-cyclotron-part-2 branch September 10, 2024 07:05
benjackwhite added a commit that referenced this pull request Sep 10, 2024
timgl pushed a commit that referenced this pull request Sep 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants