Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot rename fields unless they're present in every log record #171

Open
agronholm opened this issue Mar 21, 2023 · 5 comments · May be fixed by #172 or #186
Open

Cannot rename fields unless they're present in every log record #171

agronholm opened this issue Mar 21, 2023 · 5 comments · May be fixed by #172 or #186

Comments

@agronholm
Copy link

agronholm commented Mar 21, 2023

I tried the following configuration to get field names matching the OpenTelemetry semantic conventions:

json:
  (): pythonjsonlogger.jsonlogger.JsonFormatter
  rename_fields:
    otelTraceID: trace_id
    otelSpanID: span_id
    otelServiceName: service.name
     levelname: severity
     message: body
     threadName: thread.name
     exc_type: exception.type
     exc_val: exception.message
     traceback: exception.stacktrace
     pathname: code.filepath
     lineno: code.lineno
     funcName: code.function
  format: '%(levelname)s %(name)s %(threadName)s %(message)s %(pathname)s %(lineno)s %(funcName)s'

It would not work because some of these fields were not present in every log record, but the field rename function requires cannot handle missing fields. In particular, renaming the exception related fields is impossible.

@agronholm
Copy link
Author

Addendum: explicitly adding these fields to format removes the error, but then they have null values where a log record does not contain an exception. Would it be possible to get fields renamed without making them mandatory?

@fiarabbit
Copy link

I have this problem too. I really hope this problem to be fixed 👍

@lippertto
Copy link

I have the problem too. I need to rename fields for GCP's Error Reporting to pick up exceptions.

@madzak - how should we proceed here? The code change looks safe. Is there anyone who can review+merge it? (And publish a new version)

@lippertto
Copy link

lippertto commented Jun 13, 2023

For what it's worth: A custom formatter implementation will avoid the bug. However, it will would be better if we could use the standard mechanisms. Here is what I use:

import logging
from pythonjsonlogger import jsonlogger

class CustomJsonFormatter(jsonlogger.JsonFormatter):
    """As long as https://github.com/madzak/python-json-logger/issues/171 is still open, we need to have our
    own JsonFormatter implementation that fixes the bug."""

    def _perform_rename_log_fields(self, log_record):
        for old_field_name, new_field_name in self.rename_fields.items():
            if old_field_name not in log_record:
                continue
            log_record[new_field_name] = log_record[old_field_name]
            del log_record[old_field_name]

def _setup_logging():
    handler = logging.StreamHandler()
    handler.setFormatter(
        CustomJsonFormatter(
            "%(levelname)s %(message)s",
            rename_fields={"levelname": "severity", "exc_info": "stack_trace"},
        )
    )
    logging.getLogger().setLevel(logging.INFO)
    logging.getLogger().addHandler(handler)

@nhairs
Copy link

nhairs commented May 28, 2024

As it looks like python-json-logger is currently unmaintained, I am working on a maintained fork.

I've fixed this in the v3.1.0 release.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
4 participants