feat: hogql trends #17519

Gilbert09 · 2023-09-19T09:49:03Z

Problem

We want to move insight trends backend to use HogQL as opposed to the current janky filter query

Changes

Started off this work behind the hogql_insights feature flag. Got the trends insight rendering with some basic functionality:

Multiple series
Date range filters
The Total Count and HogQL Expression math operators
Property filters (internal users flag, etc)

Still to do:

(anything else I've forgotten here?)

How did you test this code?

Only been testing in the browser so far. Some testing will be required on the new backend query runner

posthog/hogql_queries/lifecycle_query_runner.py

Gilbert09 · 2023-09-19T09:51:58Z

posthog/hogql_queries/trends_query_runner.py

+        with self.timings.measure("trends_query"):
+            for series in self.query.series:
+                queries.append(
+                    parse_select(


To support multiple series. Looks like we'll have to execute multiple queries (the current trends code does this too, but in parallel with multiple series). I had looked at using a UNION ALL over multiple queries, but we could very easily hit max query limits given a bunch of series. Are there any alternatives that can be used here with Clickhouse / HogQL?

Hmm... Scope creep, but what about just multiple columns in the same query? E.g. select count(distinct person_id) as series1, count(id) as series_2, etc from event

The base from events query is always the same for everyone --> all events + date range + global filters.

The individual event queries filter this further. E.g. "only $pageview with filters 1, 2, 3". This could be accomplished with something like:

select countIf(distinct person_id, series_1_filters) as series1, countIf(id, series_2_filters) as series_2 from event where global_filters

This might break with DAU/MAU queries, but I'm not sure right now..

The global filters could then include the common filters (e.g. event='$pageview') if there are any to speed up the process, otherwise they'd read through all events. Doing a UNION ALL query, or doing multiple queries and aggregating the results, works... but is likely slower as you need to do multiple passes over the data.

posthog/hogql_queries/trends_query_runner.py

posthog-bot · 2023-09-19T10:02:57Z