Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Frontend support for file_format in S3 batch export #21076

Merged

Conversation

tomasfarias
Copy link
Contributor

@tomasfarias tomasfarias commented Mar 21, 2024

Problem

Adds a dropdown to the UI for users to select an export format. Needs #20979 to be merged first.

Changes

Form now has a "Format" dropdown:
image

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

Does this work well for both Cloud and self-hosted?

Yes sir.

How did you test this code?

Properly exported parquet to local MinIO:
image

import pyarrow.parquet as pq

>>> t = pq.read_table("~/Downloads/2024-03-21T00 00 00+00 00-2024-03-22T00 00 00+00 00.parquet")
>>> t
pyarrow.Table
distinct_id: string
inserted_at: timestamp[us, tz=UTC]
uuid: string
event: string
person_id: string
created_at: timestamp[us, tz=UTC]
timestamp: timestamp[us, tz=UTC]
elements_chain: string
properties: string
person_properties: string

@tomasfarias tomasfarias marked this pull request as ready for review March 21, 2024 15:37
@tomasfarias tomasfarias requested a review from a team March 21, 2024 15:41
Copy link
Contributor

github-actions bot commented Mar 21, 2024

Size Change: 0 B

Total Size: 824 kB

ℹ️ View Unchanged
Filename Size
frontend/dist/toolbar.js 824 kB

compressed-size-action

Base automatically changed from feat/support-for-multiple-file-formats to master March 22, 2024 13:20
We now should be more explicit about what is the context in which the
batch export temporary file is alive. The writer outlives this
context, so it can be used by callers to, for example, check how many
records were written.
@tomasfarias tomasfarias force-pushed the feat/frontend-support-for-multiple-s3-file-formats branch from 3a37b40 to 6fa3360 Compare March 22, 2024 13:20
@tomasfarias tomasfarias force-pushed the feat/frontend-support-for-multiple-s3-file-formats branch from 803e61d to c0b695d Compare March 22, 2024 13:23
@tomasfarias tomasfarias enabled auto-merge (squash) March 22, 2024 13:23
@tomasfarias tomasfarias merged commit fcc192e into master Mar 22, 2024
93 checks passed
@tomasfarias tomasfarias deleted the feat/frontend-support-for-multiple-s3-file-formats branch March 22, 2024 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant