Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(data-warehouse): Allow syncing to a single S3 folder for the life of a schema #23155

Merged
merged 3 commits into from
Jun 25, 2024

Conversation

Gilbert09
Copy link
Member

Revert of a revert - see Slack thread

Changes

  • Sets all DLT write_disposition to replace if incremental is turned off
  • Syncs DLT files to the same single S3 folder, instead of writing to a new folder on every sync

Does this work well for both Cloud and self-hosted?

Yep

How did you test this code?

Local browser testing

@Gilbert09 Gilbert09 requested a review from EDsCODE June 21, 2024 15:46
Copy link
Member

@EDsCODE EDsCODE left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice. just keep an eye in prod when this is deployed as this might act up

@Gilbert09 Gilbert09 merged commit 48e0672 into master Jun 25, 2024
84 checks passed
@Gilbert09 Gilbert09 deleted the tom/folder-path-replace branch June 25, 2024 10:39
Copy link

sentry-io bot commented Jun 25, 2024

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ ExternalDataSource.DoesNotExist: ExternalDataSource matching query does not exist. /api/projects/{parent_lookup_team_id}/external_... View Issue

Did you find this useful? React with a 👍 or 👎

timgl pushed a commit that referenced this pull request Jun 27, 2024
…fe of a schema (#23155)

Allow syncing to a single S3 folder for the life of a schema
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants