Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Heartbeating and asyncio for BigQuery batch exports #18741

Merged
merged 3 commits into from
Nov 21, 2023

Conversation

tomasfarias
Copy link
Contributor

Problem

We have refactored the following destinations to use async calls for blocking io:

This is the final one, BigQuery.

Changes

  • Wrap blocking IO in asyncio.to_thread
  • Add heartbeatting.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Manual tests:

DEBUG=1 pytest posthog/temporal/tests/batch_exports/test_bigquery_batch_export_workflow.py -vv 
===================================================== test session starts =====================================================
platform linux -- Python 3.10.10, pytest-7.4.0, pluggy-0.13.1 -- ./.direnv/python-3.10.10/bin/python
cachedir: .pytest_cache
django: settings: posthog.settings (from ini)
rootdir: .
configfile: pytest.ini
plugins: asyncio-0.21.1, icdiff-0.6, flaky-3.7.0, env-0.8.2, Faker-17.5.0, syrupy-1.7.4, mock-3.11.1, split-0.8.1, django-4.5.2, cov-4.1.0
asyncio: mode=strict
collected 8 items                                                                                                             

posthog/temporal/tests/batch_exports/test_bigquery_batch_export_workflow.py::test_insert_into_bigquery_activity_inserts_data_into_bigquery_table[None] PASSED [ 12%]
posthog/temporal/tests/batch_exports/test_bigquery_batch_export_workflow.py::test_insert_into_bigquery_activity_inserts_data_into_bigquery_table[exclude_events1] PASSED [ 25%]
posthog/temporal/tests/batch_exports/test_bigquery_batch_export_workflow.py::test_bigquery_export_workflow[None-hour] PASSED [ 37%]
posthog/temporal/tests/batch_exports/test_bigquery_batch_export_workflow.py::test_bigquery_export_workflow[None-day] PASSED [ 50%]
posthog/temporal/tests/batch_exports/test_bigquery_batch_export_workflow.py::test_bigquery_export_workflow[exclude_events1-hour] PASSED [ 62%]
posthog/temporal/tests/batch_exports/test_bigquery_batch_export_workflow.py::test_bigquery_export_workflow[exclude_events1-day] PASSED [ 75%]
posthog/temporal/tests/batch_exports/test_bigquery_batch_export_workflow.py::test_bigquery_export_workflow_handles_insert_activity_errors PASSED [ 87%]
posthog/temporal/tests/batch_exports/test_bigquery_batch_export_workflow.py::test_bigquery_export_workflow_handles_cancellation PASSED [100%]

--------------------------------------------------- snapshot report summary ---------------------------------------------------

===================================================== 8 passed in 57.52s ======================================================

Copy link
Contributor

Size Change: 0 B

Total Size: 2.01 MB

ℹ️ View Unchanged
Filename Size
frontend/dist/toolbar.js 2.01 MB

compressed-size-action

@tomasfarias tomasfarias changed the title feat: Heartbeatting support for BigQuery feat: Heartbeating and asyncio for BigQuery batch exports Nov 20, 2023
@tomasfarias tomasfarias merged commit 420d5cc into master Nov 21, 2023
69 checks passed
@tomasfarias tomasfarias deleted the refactor/async-bigquery-batch-export branch November 21, 2023 09:37
Copy link

sentry-io bot commented Dec 1, 2023

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ Forbidden: 403 Quota exceeded: Your table exceeded quota for imports or query appends per table. For more in... google.api_core.future.polling in result View Issue
  • ‼️ RefreshError: ('invalid_grant: Invalid grant: account not found', {'error': 'invalid_grant', 'error_description... google.oauth2._client in _handle_error_response View Issue

Did you find this useful? React with a 👍 or 👎

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant