Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(data-warehouse): DLT + temporal #18700

Merged
merged 74 commits into from
Nov 29, 2023
Merged
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
77bf893
testing
EDsCODE Nov 16, 2023
8ac4252
draft
EDsCODE Nov 17, 2023
8cd229d
tests
EDsCODE Nov 17, 2023
b19c331
working rough draft
EDsCODE Nov 17, 2023
1573e10
split workflow and activity workers
EDsCODE Nov 17, 2023
4ae9959
temp
EDsCODE Nov 20, 2023
b33c0a8
merge base
EDsCODE Nov 20, 2023
9a9ef68
split task queues
EDsCODE Nov 20, 2023
b5af2c3
add schedule
EDsCODE Nov 20, 2023
b92c47a
working with schedule
EDsCODE Nov 20, 2023
e205015
add bucket stitching activity
EDsCODE Nov 20, 2023
dd4f5c3
sync data steps for pipeline
EDsCODE Nov 21, 2023
4f2406b
add comment
EDsCODE Nov 21, 2023
0487944
update default endpoints
EDsCODE Nov 21, 2023
95b9276
workflow with all steps
EDsCODE Nov 21, 2023
e79a6ab
update heartbeat
EDsCODE Nov 22, 2023
016e7e0
update env var, update folders so there are common and specific workf…
EDsCODE Nov 22, 2023
877db76
update workflow
EDsCODE Nov 22, 2023
b31fcfc
merge base
EDsCODE Nov 22, 2023
45b1d4d
update imports
EDsCODE Nov 22, 2023
9b243a1
reformat
EDsCODE Nov 22, 2023
5a79f4d
already sync
EDsCODE Nov 22, 2023
83f27a8
merge base
EDsCODE Nov 22, 2023
4236775
format
EDsCODE Nov 22, 2023
dd1098d
adjust async methods
EDsCODE Nov 22, 2023
8b180cf
make it readable
EDsCODE Nov 22, 2023
24922ff
remove extra
EDsCODE Nov 22, 2023
da4c1a1
remove unnecessary config
EDsCODE Nov 22, 2023
5d99007
add invoices to default endpoint
EDsCODE Nov 22, 2023
e4decfc
change heartbeat name
EDsCODE Nov 22, 2023
777cee2
remove
EDsCODE Nov 22, 2023
0193e0f
update dev packages
EDsCODE Nov 22, 2023
ce8ae50
update migration
EDsCODE Nov 22, 2023
a10c9a6
package and tests
EDsCODE Nov 22, 2023
5d1a4e9
update test import path
EDsCODE Nov 22, 2023
4d8c246
restore task queue env var and remove unnecessary ones, update github…
EDsCODE Nov 27, 2023
7424fcc
fix tests
EDsCODE Nov 27, 2023
60a444a
typing
EDsCODE Nov 27, 2023
60b9ea6
revert task queue change
EDsCODE Nov 27, 2023
10c9b30
update github action and env vars
EDsCODE Nov 27, 2023
67cec89
merge base
EDsCODE Nov 27, 2023
78f576c
poll reload
EDsCODE Nov 27, 2023
ab3463f
update retry
EDsCODE Nov 27, 2023
36a760e
change activity stiching function back to sync for now
EDsCODE Nov 27, 2023
6aeb052
typing
EDsCODE Nov 27, 2023
780ffec
try to import differently
EDsCODE Nov 27, 2023
7fc1b2e
Update query snapshots
github-actions[bot] Nov 27, 2023
ee00f3a
Update query snapshots
github-actions[bot] Nov 27, 2023
6f5b90a
remap
EDsCODE Nov 28, 2023
2bff119
remap
EDsCODE Nov 28, 2023
3658c81
Merge branch 'dw-sync-experiment' of github.com:PostHog/posthog into …
EDsCODE Nov 28, 2023
b5842b7
move
EDsCODE Nov 28, 2023
972a16c
move
EDsCODE Nov 28, 2023
d1e9c77
Update UI snapshots for `chromium` (2)
github-actions[bot] Nov 28, 2023
54d5476
Update UI snapshots for `chromium` (1)
github-actions[bot] Nov 28, 2023
87487d3
Update UI snapshots for `chromium` (2)
github-actions[bot] Nov 28, 2023
9728f72
Update UI snapshots for `chromium` (1)
github-actions[bot] Nov 28, 2023
dfe50a8
sort
EDsCODE Nov 28, 2023
a60bb33
Merge branch 'dw-sync-experiment' of github.com:PostHog/posthog into …
EDsCODE Nov 28, 2023
d55e08f
update import
EDsCODE Nov 28, 2023
9d7c3ca
Update UI snapshots for `chromium` (1)
github-actions[bot] Nov 28, 2023
35687fe
Update UI snapshots for `chromium` (1)
github-actions[bot] Nov 28, 2023
42a440e
typing
EDsCODE Nov 28, 2023
6658429
Merge branch 'dw-sync-experiment' of github.com:PostHog/posthog into …
EDsCODE Nov 28, 2023
53d5858
add notes on logging
EDsCODE Nov 28, 2023
0516c65
make sure datawarehousetable filtering works right
EDsCODE Nov 28, 2023
817ff19
update timeout and how datawarehouse tables are linked
EDsCODE Nov 29, 2023
331460b
sort
EDsCODE Nov 29, 2023
d20133b
add delete
EDsCODE Nov 29, 2023
f6015c8
merge base
EDsCODE Nov 29, 2023
3bc50e2
Update UI snapshots for `chromium` (1)
github-actions[bot] Nov 29, 2023
ed3cde8
Update UI snapshots for `chromium` (1)
github-actions[bot] Nov 29, 2023
74d5bd1
merge base
EDsCODE Nov 29, 2023
3a4f29c
Merge branch 'dw-sync-experiment' of github.com:PostHog/posthog into …
EDsCODE Nov 29, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
update github action and env vars
EDsCODE committed Nov 27, 2023

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature.
commit 10c9b3048f5480ff240cb83ad59eb4e81c513f0b
11 changes: 6 additions & 5 deletions .github/workflows/container-images-cd.yml
Original file line number Diff line number Diff line change
@@ -123,13 +123,13 @@ jobs:
"image_tag": "${{ steps.build.outputs.digest }}"
}

- name: Check for changes that affect temporal worker
id: check_changes_temporal_worker
- name: Check for changes that affect batch exports temporal worker
id: check_changes_batch_exports_temporal_worker
run: |
echo "::set-output name=changed::$(git diff --name-only HEAD^ HEAD | grep -E '^posthog/temporal/common|^posthog/temporal/batch_exports|^posthog/batch_exports/|^posthog/management/commands/start_temporal_worker.py$' || true)"

- name: Trigger Temporal Worker Cloud deployment
if: steps.check_changes_temporal_worker.outputs.changed != ''
- name: Trigger Batch Exports Temporal Worker Cloud deployment
if: steps.check_changes_batch_exports_temporal_worker.outputs.changed != ''
uses: mvasigh/dispatch-action@main
with:
token: ${{ steps.deployer.outputs.token }}
@@ -138,5 +138,6 @@ jobs:
event_type: temporal_worker_deploy
message: |
{
"image_tag": "${{ steps.build.outputs.digest }}"
"image_tag": "${{ steps.build.outputs.digest }}",
"worker_name": "temporal-worker"
}
10 changes: 2 additions & 8 deletions posthog/management/commands/start_temporal_worker.py
Original file line number Diff line number Diff line change
@@ -16,11 +16,6 @@ class Command(BaseCommand):
help = "Start Temporal Python Django-aware Worker"

def add_arguments(self, parser):
parser.add_argument(
"--workflow-group",
default=settings.WORKFLOW_GROUP,
help="Group of temporal workflows and activities to execute (batch-exports)",
)
parser.add_argument(
"--temporal-host",
default=settings.TEMPORAL_HOST,
@@ -70,13 +65,12 @@ def handle(self, *args, **options):
server_root_ca_cert = options.get("server_root_ca_cert", None)
client_cert = options.get("client_cert", None)
client_key = options.get("client_key", None)
workflow_group = options["workflow_group"]

if workflow_group == "batch-exports":
if task_queue == "no-sandbox-python-django":
workflows = WORKFLOWS
activities = ACTIVITIES
else:
raise ValueError(f"Unknown workflow group: {workflow_group}")
raise ValueError(f"Unknown task queue: {task_queue}")

if options["client_key"]:
options["client_key"] = "--SECRET--"