-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(database): faster queries from the persons table #17811
Conversation
""" | ||
SELECT id FROM raw_persons WHERE (id, version) IN ( | ||
SELECT id, max(version) as version | ||
FROM raw_persons |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be filtered by team id
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
it's automatically, as this is a hogql query, not a clickhouse query
|
||
query = parse_select( | ||
""" | ||
SELECT id FROM raw_persons WHERE (id, version) IN ( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What happens with this id? Does it get replaced with the actual fields we need (properties, created_at) or is this going into another subquery?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
look a few lines lower:
query.select = []
for field_name, field_chain in requested_fields.items():
query.select.append(
ast.Alias(
alias=field_name,
expr=ast.Field(chain=["raw_persons"] + field_chain),
)
)
return query
Problem
The existing
select * from persons
transformation intoselect * from (select ... from raw_persons)
is inefficient and times out for large customers, even on the new cluster.Changes
Swaps it out for a faster version:
How did you test this code?
Tested the query in the SQL editor. Hoping CI will update the snapshots and report of failures.