kafka.errors.MessageSizeTooLargeError: [Error 10] MessageSizeTooLargeError: The message is 1195221 bytes when serialized which is larger than the maximum request size you have configured with the max_request_size configuration

[jk2K]

Bug description

when visit page loaded with posthog.js, I will got below errors

How to reproduce

posthog-self-hosted-web-1                     | 2024-01-06T05:24:10.332158Z [error    ] kafka_session_recording_produce_failure [posthog.api.capture] host=posthog-demo.dev ip=172.19.0.11 pid=1047 request_id=7c7fbb39-d872-48df-b6a3-739d4d82e672 tid=281471218610656 token=phc_BiG43DXnRpxerFRSApf365VvIbn1ZR90eKaLsZTAf00 x_forwarded_for=192.168.65.1
posthog-self-hosted-web-1                     | Traceback (most recent call last):
posthog-self-hosted-web-1                     |   File "/code/posthog/api/capture.py", line 499, in get_event
posthog-self-hosted-web-1                     |     future.get(timeout=settings.KAFKA_PRODUCE_ACK_TIMEOUT_SECONDS - (time.monotonic() - start_time))
posthog-self-hosted-web-1                     |   File "/python-runtime/kafka/producer/future.py", line 65, in get
posthog-self-hosted-web-1                     |     raise self.exception # pylint: disable-msg=raising-bad-type
posthog-self-hosted-web-1                     |   File "/python-runtime/kafka/producer/kafka.py", line 596, in send
posthog-self-hosted-web-1                     |     self._ensure_valid_record_size(message_size)
posthog-self-hosted-web-1                     |   File "/python-runtime/kafka/producer/kafka.py", line 654, in _ensure_valid_record_size
posthog-self-hosted-web-1                     |     raise Errors.MessageSizeTooLargeError(
posthog-self-hosted-web-1                     | kafka.errors.MessageSizeTooLargeError: [Error 10] MessageSizeTooLargeError: The message is 1195221 bytes when serialized which is larger than the maximum request size you have configured with the max_request_size configuration

Environment

Open-source hobby deploy

Additional context

Thank you for your bug report – we love squashing them!

[pauldambra]

I can see in #19634 you already know how to alter your local config / kafka setup to support larger messages so I'll close this since it's a self-hosted setup problem 👍