Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: enable gossip for flower and to squelch errors #19823

Merged
merged 1 commit into from
Jan 18, 2024
Merged

Conversation

fuziontech
Copy link
Member

@fuziontech fuziontech commented Jan 18, 2024

Problem

The celery logs were filled with this error:

{"event": "pidbox command error: AttributeError(\"'NoneType' object has no attribute 'groups'\")", "timestamp": "2024-01-17T23:03:36.230750Z", "logger": "kombu.pidbox", "level": "error", "pid": 9, "tid": 28147 │
│ 3362922816, "exception": "Traceback (most recent call last):\n  File \"/python-runtime/kombu/pidbox.py\", line 102, in dispatch\n    reply = handle(method, arguments)\n  File \"/python-runtime/kombu/pidbox.py\", line 124, in handle_cast\n    return  │
│ self.handle(method, arguments)\n  File \"/python-runtime/kombu/pidbox.py\", line 118, in handle\n    return self.handlers[method](self.state, **arguments)\n  File \"/python-runtime/celery/worker/control.py\", line 340, in enable_events\n    if dispa │
│ tcher.groups and 'task' not in dispatcher.groups:\nAttributeError: 'NoneType' object has no attribute 'groups'"}

After doing a bit of research it turns out this is related to gossip and flower collecting stats on the workers. I turned this on in K8s and the errors went a way and Flower was happy.

Note: This does increase utilization on redis some, but it should be fine at our levels (only 24 workers max right now)

Note2: We are currently running with gossip enabled manually on prod, probably should be everywhere unless there is a reason not to. It is the default for our version of celery.

Changes

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

@posthog-bot
Copy link
Contributor

Hey @fuziontech! 👋
This pull request seems to contain no description. Please add useful context, rationale, and/or any other information that will help make sense of this change now and in the distant Mars-based future.

Copy link
Contributor

Size Change: 0 B

Total Size: 2 MB

ℹ️ View Unchanged
Filename Size
frontend/dist/toolbar.js 2 MB

compressed-size-action

@webjunkie webjunkie merged commit 6b50aff into master Jan 18, 2024
97 checks passed
@webjunkie webjunkie deleted the gossip branch January 18, 2024 08:01
@webjunkie
Copy link
Contributor

I merged this since I saw these errors all over the place.

I'm wondering if we do need to enable events as well, like explained here
sumitasok/celery#5

Or is the gossip now enough 🤔

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants