Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(batch-exports): Move everyone to async arrow streaming #24804

Closed
Closed
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 0 additions & 19 deletions posthog/temporal/batch_exports/batch_exports.py
Original file line number Diff line number Diff line change
Expand Up @@ -178,25 +178,6 @@ async def iter_records_from_model_view(
) -> AsyncRecordsGenerator:
if model_name == "persons":
view = SELECT_FROM_PERSONS_VIEW
elif str(team_id) not in settings.ASYNC_ARROW_STREAMING_TEAM_IDS:
# TODO: Let this model be exported by `astream_query_as_arrow`.
# Just to reduce risk, I don't want to change the function that runs 100% of the exports
# without battle testing it first.
# There are already changes going out to the queries themselves that will impact events in a
# positive way. So, we can come back later and drop this block.
# UPDATE: Will start moving teams over to `astream_query_as_arrow` by setting their ids
# in `ASYNC_ARROW_STREAMING_TEAM_IDS`. If testing goes well, we'll remove this block.
for record_batch in iter_records(
client,
team_id=team_id,
is_backfill=is_backfill,
interval_start=interval_start,
interval_end=interval_end,
fields=fields,
**parameters,
):
yield record_batch
return
else:
if parameters["exclude_events"]:
parameters["exclude_events"] = list(parameters["exclude_events"])
Expand Down
Loading