Celery / Redis issue

[bwbroersma]

Related:

• Add flushing Redis cache to installation routine in documentation #1281

Currently used versions:

Internet.nl/requirements.txt

Line 27 in 1368a1e

celery==5.3.1

Internet.nl/requirements.txt

Line 99 in 1368a1e

hiredis==2.2.3

$ make exec cmd="redis-server --version" service=redis env=develop 
docker compose  --project-name=internetnl-develop exec --user root redis redis-server --version
Redis server v=7.0.12 ...

Maybe the source is caused by:

• fixing this previous (batch?) problem Redis connection leak on eventlet workers #676 (note my comment, which might be right while also being part of the problem), which also mentions Dockerize application and add integration tests #890 and the probably relevant "Set max tasks per child, to reduce memory issues on ipv6kaartmetingen":
  

  Internet.nl/docker/docker-compose.yml

  Lines 315 to 317 in 1368a1e

  	# even though eventlet is used and childs are not really threads or processes, this settings reduces
  	# issues with high memory usage, probably because some objects are freed?
  	- CELERYD_MAX_TASKS_PER_CHILD=100

Note there are some issues with hiredis in combination with this setting:

• worker getting stuck celery/celery#2606 (comment)
• Celery pool with Redis broker freeze silently or crash & freeze when hiredis package is installed celery/celery#3898
• Worker hangs and stop responding after an undeterminated amount of time (sometimes). celery/celery#4321 (comment)

Question:
With this knowledge, can we trigger this, e.g. by setting a very low CELERYD_MAX_TASKS_PER_CHILD? Also note the steps to reproduce and actual behavior with (redis) error messages in celery/celery#3898.

[aequitas]

The CELERYD_MAX_TASKS_PER_CHILD is already very low with 100 so it should trigger more often I guess? But I think it can't hurt to remove this package from the requirements and see if the issues disappear.

The unbound library also seems to use hiredis [0][1]. I don't know how trivial it would be to disable it in that package as well, but that is only related to the caching/connection test and not web/mail tests and queuing afaik.

[0] https://github.com/search?q=repo%3Ainternetstandards%2Funbound%20hiredis&type=code
[1]

Internet.nl/docker/Dockerfile

Line 46 in 1368a1e

--with-libhiredis \