Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: message size two large #23304

Merged
merged 2 commits into from
Jun 28, 2024
Merged

fix: message size two large #23304

merged 2 commits into from
Jun 28, 2024

Conversation

pauldambra
Copy link
Member

follow up to #23268

the mocks in the capture tests made the MessageSizeTooLarge solution work in tests, but it doesn't work in practice, sad times

this PR:

  • updates the mock so it throws at the right time
  • keeps track of the event args that cause each kafka future
  • so that if the future throws MessageSizeTooLarge
  • we can replace some content and retry the kafka produce

metric in place ready at https://grafana.prod-us.posthog.dev/d/session-replay/session-replay?var-role=recordings-blob-ingestion&var-pod=$__all&var-partition=$__all&from=now-1h&to=now&timezone=utc&tab=queries&viewPanel=panel-74

this set of exception handlers is pretty ugly now but i don't want to spend time refactoring this till i've confirmed it has the desired effect in prod

Copy link

sentry-io bot commented Jun 27, 2024

🔍 Existing Issues For Review

Your pull request is modifying functions with the following pre-existing issues:

📄 File: posthog/api/capture.py

Function Unhandled Issue
get_event UnicodeDecodeError: 'utf-16-le' codec can't decode bytes in position 21438-21439: illegal UTF-16 surrogate ...
Event Count: 4

Did you find this useful? React with a 👍 or 👎

@pauldambra pauldambra requested a review from a team June 27, 2024 23:45
@pauldambra pauldambra merged commit 38cf761 into master Jun 28, 2024
85 checks passed
@pauldambra pauldambra deleted the fix/pd/message-size-two-large branch June 28, 2024 08:29
Copy link

sentry-io bot commented Jun 28, 2024

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ KafkaTimeoutError: KafkaTimeoutError: Timeout after waiting for 8.914257952999833 secs. /s/[#].* View Issue
  • ‼️ KafkaTimeoutError: KafkaTimeoutError: Timeout after waiting for 10 secs. /s/[#].* View Issue
  • ‼️ ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied /s/[#].* View Issue

Did you find this useful? React with a 👍 or 👎

Phanatic pushed a commit that referenced this pull request Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants