Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 92.76%. Comparing base (0939eb4) to head (476762c).
Report is 16 commits behind head on main.

@@            Coverage Diff             @@
##             main      #87      +/-   ##
==========================================
+ Coverage   85.62%   92.76%   +7.14%     
==========================================
  Files           8       13       +5     
  Lines         153      304     +151     
==========================================
+ Hits          131      282     +151     
  Misses         22       22              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

There is currently a LOG_LEVEL variable that affects console and django log output and a CELERY_LOGLEVEL that is used the celery commands

If we used CELERY_LOGLEVEL for the celery logger, it will only display at the level of LOG_LEVEL. So is celery is DEBUG and the main level is INFO, it will only log celery INFO.

I think there are two ways to handle this:

• Remove CELERY_LOGLEVEL and just use LOG_LEVEL for everything
• Create separate console and file logs for celery and keep its own level variable.

@annashamray do you think it is fine to CELERY_LOGLEVEL? Or should I create a separate logging for celery?

I don't mind your solution of removing CELERY_LOGLEVEL
But I'm not sure if devops actually use the separate env var for celery logging

@stevenbal What do you thing?

@annashamray might be worth asking Sjoerd, but I'd also be fine with removing CELERY_LOGLEVEL and relying on the other variable LOG_LEVEL

@stevenbal I did ask Sjoerd and he would like both like open-forms.

[They] are used to setting CELERY_LOGLEVEL to debug and just leave LOG_LEVEL on its default

This does require creating separate loggers for CELERY.

if it's easy to make it consistent with OF, then let's do that. Though I don't see a specific logger for celery in OF?

Added the new loggers for console and file.

The way its currently implemented in projects:

• Open Forms celery does not use the Django logger at all.
• OAF uses the Django logger but celery was missing from the loggers.
• Open Inwoner does use Django logger with celery setup but does not have a LOG_LEVEL var and always logs DEBUG to the base handlers. Logs from celery based on a CELERY_LOGLEVEL envvar.

My current approach is to add back the CELERY_LOGLEVEL envvar and create separate celerly loggers, but we can also use the open-forms approach. It depends of if the Django console is used for debugging celery

@annashamray @stevenbal can you re-review the new changes and my above comment? Thanks

Also note, with these changes, CELERY ignores setting the log level on the command line.

edit: this can be fixed in config_loggers in celery.py in all the projects.

Perhaps there should be a common celery.py or celery functions in utils?

Example of above

@setup_logging.connect()
def config_loggers(loglevel, logfile, format, colorize, **kwargs):
    from logging.config import dictConfig

    if logfile:
        settings.LOGGING["handlers"]["celery_file"]["filename"] = logfile
    settings.LOGGING["formatters"]["celery"]["format"] = format
    settings.LOGGING["loggers"]["celery"]["level"] = loglevel
    dictConfig(settings.LOGGING)

Really depends on what we want to allow.

Also, should this change the LOG_STDOUT?

Discussed with Conor, and as the default was that there was output on stdout lets set LOG_STDOUT to True by default. In Docker and K8s installs you want logging on stdout and that is our default way of deployment, if someone doesn't want this they can turn it off via this envvar