Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(data-warehouse): external data resource #17554

Merged
merged 42 commits into from
Nov 2, 2023
Merged

feat(data-warehouse): external data resource #17554

merged 42 commits into from
Nov 2, 2023

Conversation

EDsCODE
Copy link
Member

@EDsCODE EDsCODE commented Sep 20, 2023 •
edited
Loading

Problem

  • i want to be able to link data for use in datawarehouse in one click

Changes

  • Adds one click sync capabilities
  • Settings
    • AIRBYTE_API_KEY
    • AIRBYTE_WORKSPACE_ID
    • AIRBYTE_BUCKET_KEY
    • AIRBYTE_BUCKET_SECRET

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

@posthog-bot
Copy link
Contributor

Hey @EDsCODE! 👋
This pull request seems to contain no description. Please add useful context, rationale, and/or any other information that will help make sense of this change now and in the distant Mars-based future.

@posthog-bot
Copy link
Contributor

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.

@posthog-bot
Copy link
Contributor

This PR was closed due to lack of activity. Feel free to reopen if it's still relevant.

@posthog-bot posthog-bot closed this Oct 6, 2023
@EDsCODE EDsCODE reopened this Oct 9, 2023
@posthog-bot posthog-bot removed the stale label Oct 10, 2023
@EDsCODE EDsCODE changed the title feat(data-warehouse): testing airbyte feat(data-warehouse): external data resource Oct 26, 2023
@EDsCODE EDsCODE marked this pull request as ready for review October 26, 2023 20:26
@posthog-bot
Copy link
Contributor

Hey @EDsCODE! 👋
This pull request seems to contain no description. Please add useful context, rationale, and/or any other information that will help make sense of this change now and in the distant Mars-based future.

timgl
timgl reviewed Oct 27, 2023
View reviewed changes
posthog/warehouse/models/external_data_resource.py Outdated
from posthog.models.team import Team


class ExternalDataResource(CreatedMetaFields, UUIDModel):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry to be pedantic about this, though it's a nightmare to fix later: I don't think external data source is the right name. I want to make sure we're consistently using "Source", in contrast to "Destination" which we're using on the pipeline side.

A few suggestions

  • ExternalDataSource
  • DatawarehouseSource
  • ETLSource

Thoughts?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ha, I was in the middle of changing it to option 1 (ExternalDataSource) but then did "resource" mainly because the entities that are created are (source, connection, destination). But I think it's ok to do ExternalDataSource anyway so I'll change it to that

@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

  • chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@EDsCODE EDsCODE requested a review from timgl October 27, 2023 20:52
@posthog-bot
Copy link
Contributor

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

  • chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

@EDsCODE EDsCODE merged commit 199e913 into master Nov 2, 2023
71 checks passed
@EDsCODE EDsCODE deleted the dw-airbyte branch November 2, 2023 19:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@timgl timgl timgl approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@EDsCODE @posthog-bot @timgl