Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: start of clickhouse server version bump to 24.8 #25189

Merged
merged 111 commits into from
Nov 26, 2024
Merged

feat: start of clickhouse server version bump to 24.8 #25189

merged 111 commits into from
Nov 26, 2024

Conversation

fuziontech
Copy link
Member

@fuziontech fuziontech commented Sep 24, 2024

In flight

Problem

We need to upgrade CH versions that are deployed, this is the first step to make sure things don't break

Changes

  • Update test suite and CI to use ClickHouse 24.8
  • Update dev to ClickHouse 24.8
  • Fix tests that broke on 24.8 or disable the experimental query analyzer when needed
    • These queries will need to be fixed by the teams that own them eventually to get the improvements offered by the experimental query analyzer
  • Update snapshots of queries that were touched

Deployment plan

This is slightly tricky because we will have a period of time where we will error on queries due to query <> DB version incompatibilities. We will manage this by moving quickly.

  • Upgrade Dev (This tests the feasibility of upgrading from 23.12 -> 24.8)
  • Land this PR (We do this now because we can revert if things go poorly)
  • Monitor 👀 (quickly)
  • Upgrade US (This is the first prod cluster because there are fewer nodes, easier to complete quickly and test)
    • Offline first
    • Online next
  • Upgrade new US cluster
  • Monitor 👀
  • Upgrade EU
    • Upgrade Coordinator
  • Monitor 👀

Tests to fix ✅

  • posthog/queries/funnels/test/test_funnel.py <- this one has a lot of failures @fuziontech
  • posthog/queries/funnels/test/test_funnel_strict.py @Daesgar
  • posthog/queries/funnels/test/test_funnel_unordered.py @Daesgar
  • posthog/queries/funnels/test/test_breakdowns_by_current_url.py @fuziontech
  • posthog/api/test/test_person.py @fuziontech - I'm working on this one to figure out why we are getting 400 responses
  • ee/clickhouse/queries/funnels/test/test_funnel_correlation.py @fuziontech
  • posthog/session_recordings/queries/test/test_session_recording_list_from_filters.py @fuziontech
  • ee/clickhouse/views/test/test_clickhouse_experiment_secondary_results.py @fuziontech
  • ee/clickhouse/views/test/test_clickhouse_experiments.py @fuziontech

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

  • Yes

How did you test this code?

With a lot of tests

@fuziontech fuziontech force-pushed the ch-24.3 branch 2 times, most recently from f2a07f1 to a447bd7 Compare September 24, 2024 23:53
@fuziontech fuziontech changed the title feat: start of clickhouse server version bump to 24.3 feat: start of clickhouse server version bump to ~24.3~ -> 24.4 Oct 1, 2024
@fuziontech fuziontech changed the title feat: start of clickhouse server version bump to ~24.3~ -> 24.4 feat: start of clickhouse server version bump to ~~24.3~~ -> 24.4 Oct 1, 2024
@fuziontech fuziontech changed the title feat: start of clickhouse server version bump to ~~24.3~~ -> 24.4 feat: start of clickhouse server version bump to 24.4 Oct 1, 2024
fuziontech and others added 15 commits November 15, 2024 09:31
* master: (66 commits)
  feat: display line context in stack frames (#26254)
  fix(experiments HogQL): fix funnels filter in prepare_query (#26327)
  fix(hogql): Support parsing dw properties with nested values (#26321)
  fix(err): include raw js frame on frame content in pg (#26326)
  fix: mobile project settings network labels (#26318)
  fix: reduce perms from 3 to 2 for test perf (#26322)
  feat: add new alerts feature for product_analytics (#26320)
  feat(experiments): Enable holdouts for everyone (#26301)
  feat: reorg inspector list rows (#26243)
  feat: Split-up batch exports into sync and async (#26294)
  fix(data-warehouse): Fix custom series colors in chart tooltips (#26310)
  feat(cdp): adjust zapier destination description (#26313)
  fix(insights): prevent race condition (#26265)
  feat(product-assistant): evaluation pipeline (#26179)
  chore: session replay project config for mobile - what is possible (#26289)
  feat(cdp): make hash functions return null if the input is null (#26311)
  feat(err): cymbal for all teams (#26312)
  feat(hogql): Allow breakdowns for lazy-joined tables (#26302)
  feat: create events table to store last 7 days of data (#26239)
  chore: Bump batch exports start jitter (#26278)
  ...
@fuziontech fuziontech requested review from Daesgar, a team and tkaemming November 22, 2024 05:47
@fuziontech fuziontech merged commit 1e4d4ec into master Nov 26, 2024
90 checks passed
@fuziontech fuziontech deleted the ch-24.3 branch November 26, 2024 03:09
@macobo
Copy link
Contributor

macobo commented Nov 27, 2024

Word to the wise @fuziontech @tkaemming @Daesgar - you might run into ClickHouse/ClickHouse#68239 upgrading to 24.8. This was fixed in 24.9.

@fuziontech
Copy link
Member Author

@macobo you are my hero - thank you. We've been digging into why CH CPU has been 📈 since the upgrade.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants