feat(data-warehouse): use materialized view in hogql #25187

EDsCODE · 2024-09-24T20:24:23Z

Problem

materialized tables aren't queried right now

Changes

update saved query logic to use materialized tables
include compaction and use deltas3wrapper
branch from feat(data-warehouse): materialized view UI #25186

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

How did you test this code?

posthog-bot · 2024-09-24T20:40:53Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-09-24T20:59:39Z

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

chromium: 0 added, 1 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

…osthog into dw-materialized-use-new-tables

Gilbert09

See my one comment about using DataWarehouseTable - I reckon the pipelines should operate the same between external sources vs materialization - both create a DataWarehouseTable and set it on their equivalent models (Schema vs SavedQuery)

Gilbert09 · 2024-10-01T09:26:53Z

posthog/warehouse/models/datawarehouse_saved_query.py

@@ -63,6 +68,7 @@ class Status(models.TextChoices):
        null=True,
        help_text="The timestamp of this SavedQuery's last run (if any).",
    )
+    credential = models.ForeignKey(DataWarehouseCredential, on_delete=models.CASCADE, null=True, blank=True)


Instead of adding credential's to saved query, lets add a foreign key to DataWarehouseTable instead, this means:

We only have one type of "Table", regardless of whether it's an external data sourced table, or a materialized view, they should both be synonymous as far as the system is concerned

The URL of the table is stored in the DB and not code, making easy changes to how we store the data with backward compatibility (this has been a godsend for all the DLT pipeline changes we've made)

It'll support different S3 formats (parquet, delta, csv, etc) - this will be helpful for when we eventually move to iceberg, and again, this will be stored in the DB and so migrations of tables don't have to be one big migration

good point let me give it a try

posthog/hogql/database/s3_table.py

…osthog into dw-materialized-use-new-tables

EDsCODE · 2024-10-01T20:43:57Z

TODO:

how mat tables overwrite the saved queries (names will be shared)
UI issues (rerunning a model run)
UI: feedback when starting a model run
update tests

posthog-bot · 2024-10-03T18:48:24Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (wasn't pushed!)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-10-03T19:07:49Z

📸 UI snapshots have been updated

4 snapshot changes in total. 0 added, 4 modified, 0 deleted:

chromium: 0 added, 4 modified, 0 deleted (diff for shard 1, diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-10-03T19:26:41Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-10-03T20:46:52Z

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

chromium: 0 added, 2 modified, 0 deleted (diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-10-03T21:06:33Z

📸 UI snapshots have been updated

4 snapshot changes in total. 0 added, 4 modified, 0 deleted:

chromium: 0 added, 4 modified, 0 deleted (diff for shard 1, diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-10-03T21:26:30Z

📸 UI snapshots have been updated

4 snapshot changes in total. 0 added, 4 modified, 0 deleted:

chromium: 0 added, 4 modified, 0 deleted (diff for shard 1, diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-10-03T21:46:02Z

📸 UI snapshots have been updated

4 snapshot changes in total. 0 added, 4 modified, 0 deleted:

chromium: 0 added, 4 modified, 0 deleted (diff for shard 1, diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-10-03T22:05:54Z

📸 UI snapshots have been updated

5 snapshot changes in total. 0 added, 5 modified, 0 deleted:

chromium: 0 added, 5 modified, 0 deleted (diff for shard 1, diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot · 2024-10-03T22:25:28Z

📸 UI snapshots have been updated

3 snapshot changes in total. 0 added, 3 modified, 0 deleted:

chromium: 0 added, 3 modified, 0 deleted (diff for shard 1, diff for shard 2)
webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

sentry-io · 2024-10-04T01:27:58Z

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

‼️ CHQueryErrorTimeoutExceeded: DB::Exception: Timeout exceeded: elapsed 600.082745045 seconds, maximum: 600. Stack trace: posthog.tasks.exporter.export_asset View Issue
‼️ AttributeError: 'AnonymousUser' object has no attribute 'distinct_id' /embedded/{access_token} View Issue
‼️ AttributeError: 'AnonymousUser' object has no attribute 'distinct_id' /shared/{access_token} View Issue
‼️ AttributeError: 'AnonymousUser' object has no attribute 'distinct_id' /embedded/{access_token} View Issue
‼️ AttributeError: 'AnonymousUser' object has no attribute 'distinct_id' /embedded/{access_token} View Issue

_{Did you find this useful? React with a 👍 or 👎}

EDsCODE added 4 commits September 24, 2024 12:58

wip

1cc7e82

put behind flag

eda5fb4

add credential

e6472eb

move where credential is written

a596152

EDsCODE changed the base branch from master to dw-materialized-database-tables-ui September 24, 2024 20:24

github-actions bot added 2 commits September 24, 2024 20:38

Update query snapshots

a80b6f6

Update UI snapshots for chromium (1)

0abd47c

Update UI snapshots for chromium (1)

9334b03

Base automatically changed from dw-materialized-database-tables-ui to master September 30, 2024 20:02

EDsCODE added 4 commits September 30, 2024 16:05

merge main

69a658e

Merge branch 'dw-materialized-use-new-tables' of github.com:PostHog/p…

fb3eaa0

…osthog into dw-materialized-use-new-tables

prune unrelated

15edc5c

restore unrelated

3f7805c

EDsCODE marked this pull request as ready for review September 30, 2024 20:08

EDsCODE added 3 commits September 30, 2024 16:25

add feature flag to backend

d167b88

make migration concurrent

9399565

typing

c1fd904

EDsCODE requested a review from Gilbert09 October 1, 2024 01:42

Merge branch 'master' into dw-materialized-use-new-tables

b686ce3

EDsCODE requested review from a team and removed request for Gilbert09 October 1, 2024 01:42

Gilbert09 requested changes Oct 1, 2024

View reviewed changes

EDsCODE added 4 commits October 1, 2024 13:57

temp refactor to use data warehouse table

9233e1a

change migration

23f996c

add compaction and delta wrapper

529e034

Merge branch 'dw-materialized-use-new-tables' of github.com:PostHog/p…

d552ff8

…osthog into dw-materialized-use-new-tables

EDsCODE marked this pull request as draft October 1, 2024 20:42

EDsCODE and others added 2 commits October 3, 2024 14:28

move clause

56bf024

Merge branch 'master' into dw-materialized-use-new-tables

ba33455

github-actions bot added 2 commits October 3, 2024 19:05

Update UI snapshots for chromium (1)

cc9676c

Update UI snapshots for chromium (2)

ca8abe6

Update UI snapshots for chromium (1)

8208381

EDsCODE and others added 2 commits October 3, 2024 16:25

merge main

d0296fd

Update UI snapshots for chromium (2)

876f114

github-actions bot added 2 commits October 3, 2024 21:04

Update UI snapshots for chromium (1)

a134c65

Update UI snapshots for chromium (2)

0b9e4ba

github-actions bot added 2 commits October 3, 2024 21:23

Update UI snapshots for chromium (1)

32c6933

Update UI snapshots for chromium (2)

230a9d5

github-actions bot added 2 commits October 3, 2024 21:43

Update UI snapshots for chromium (1)

53ef377

Update UI snapshots for chromium (2)

d2a5159

github-actions bot added 2 commits October 3, 2024 22:03

Update UI snapshots for chromium (1)

eabbd9b

Update UI snapshots for chromium (2)

8b13b8c

github-actions bot added 2 commits October 3, 2024 22:22

Update UI snapshots for chromium (1)

74a24b6

Update UI snapshots for chromium (2)

17f986c

Merge branch 'master' into dw-materialized-use-new-tables

b06dbc3

EDsCODE merged commit 6e9e9e8 into master Oct 4, 2024
93 checks passed

EDsCODE deleted the dw-materialized-use-new-tables branch October 4, 2024 01:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(data-warehouse): use materialized view in hogql #25187

feat(data-warehouse): use materialized view in hogql #25187

EDsCODE commented Sep 24, 2024 •

edited

Loading

posthog-bot commented Sep 24, 2024

posthog-bot commented Sep 24, 2024

Gilbert09 left a comment

Gilbert09 Oct 1, 2024

EDsCODE Oct 1, 2024

EDsCODE commented Oct 1, 2024 •

edited

Loading

posthog-bot commented Oct 3, 2024

posthog-bot commented Oct 3, 2024

posthog-bot commented Oct 3, 2024

posthog-bot commented Oct 3, 2024

posthog-bot commented Oct 3, 2024

posthog-bot commented Oct 3, 2024

posthog-bot commented Oct 3, 2024

posthog-bot commented Oct 3, 2024

posthog-bot commented Oct 3, 2024

sentry-io bot commented Oct 4, 2024 •

edited

Loading

feat(data-warehouse): use materialized view in hogql #25187

feat(data-warehouse): use materialized view in hogql #25187

Conversation

EDsCODE commented Sep 24, 2024 • edited Loading

Problem

Changes

Does this work well for both Cloud and self-hosted?

How did you test this code?

posthog-bot commented Sep 24, 2024

📸 UI snapshots have been updated

posthog-bot commented Sep 24, 2024

📸 UI snapshots have been updated

Gilbert09 left a comment

Choose a reason for hiding this comment

Gilbert09 Oct 1, 2024

Choose a reason for hiding this comment

EDsCODE Oct 1, 2024

Choose a reason for hiding this comment

EDsCODE commented Oct 1, 2024 • edited Loading

posthog-bot commented Oct 3, 2024

📸 UI snapshots have been updated

posthog-bot commented Oct 3, 2024

📸 UI snapshots have been updated

posthog-bot commented Oct 3, 2024

📸 UI snapshots have been updated

posthog-bot commented Oct 3, 2024

📸 UI snapshots have been updated

posthog-bot commented Oct 3, 2024

📸 UI snapshots have been updated

posthog-bot commented Oct 3, 2024

📸 UI snapshots have been updated

posthog-bot commented Oct 3, 2024

📸 UI snapshots have been updated

posthog-bot commented Oct 3, 2024

📸 UI snapshots have been updated

posthog-bot commented Oct 3, 2024

📸 UI snapshots have been updated

sentry-io bot commented Oct 4, 2024 • edited Loading

Suspect Issues

EDsCODE commented Sep 24, 2024 •

edited

Loading

EDsCODE commented Oct 1, 2024 •

edited

Loading

sentry-io bot commented Oct 4, 2024 •

edited

Loading