Logs Data Model as Structured Logs

[LeVraiSylvain]

Hi,

My architect is asking me to use the "logs data model" as a structured log format. After analyzing it, my understanding is that the "logs data model" aims to represent logs from various sources and is used internally by the OTEL Collector:
• Receivers (input: well-known log format/protocol, e.g., syslog) => (output: DataModel)
• Processors (input: DataModel) => (output: DataModel)
• Exporters (input: DataModel) => (output: well-known log format/protocol, e.g., syslog)

My architect expected this "logs data model" format to be available in OpenTelemetry SDKs (dotnet, java, etc.). Obviously, it isn’t, since it’s not a recognized protocol format. The closest one is the OTLP protocol.
Here’s an example of what we’re sending as a "message," for instance in the MSG field of RFC5424 Syslog (in LogFmt format and not JSON as in this example):

{
  "timestamp": 1586960586000,
  "traceId": "abcdef1234567890abcdef1234567890",
  "spanId": "1234567890abcdef",
  "traceFlags": 1,
  "severityText": "INFO",
  "severityNumber": 9,
  "body": "This is a sample message",
  "resource": {
    "hostname": "my-hostname",
    "service.name": "my-service",
    "service.version": "1.0.0",
    "deployment.name": "my deployment",
    "deployment.environment.name": "dev"
  },
  "attributes": {
    "my.company.log_type": "security",
    "http.request.method": "GET",
    "url.full": "https://my-hostname/api/endpoint1"
  }
}

There are some odd things:
• An incomprehensible timestamp for humans.
• A severityNumber that is unconventional in tools like Log4J, Log4net, Serilog.
• Attributes that are not really coherent for logs. Most of them come from standardized attributes

• https://opentelemetry.io/docs/specs/semconv/resource/
• https://opentelemetry.io/docs/specs/semconv/attributes-registry/

Personally, I would have expected a simple structured log format, defined based on the company’s needs. I wouldn’t have any issue using an existing format that aims to fulfill that need, but OpenTelemetry's "Logs Data Model" doesn’t seem intended for that.

Here’s the proposal I made, which wasn’t accepted:

{
  "timestamp": "2024-08-01T15:30:45.123Z",
  "traceId": "abcdef1234567890abcdef1234567890",
  "spanId": "1234567890abcdef",
  "severity": "info",
  "message": "This is a sample message",
  "type": "security",
  "hostname": "myserver",
  "appName": "my-app",
  "appVersion": "1.0.0",
  "deploymentName": "my deployment",
  "deploymentEnv": "dev",
  "httpMethod": "get",
  "httpUrl": "https://my-hostname/api/endpoint1"
}

Guess what, in this scenario, we wouldn’t even be using OpenTelemetry!

Questions :

• Is my understanding correct?
• Is it possible to provide further clarifications in the documentation to clearly describe where the "logs data model" is used or how it should be used?
• Do you have any recommendations for me?

Thanks.

[pellared]

My architect is asking me to use the "logs data model" as a structured log format.

This is a vague statement, but I will do my best to highlight some aspects.

Here is the specification of the Logs Data Model: https://opentelemetry.io/docs/specs/otel/logs/data-model/
OTel libraries (e.g. OTel Java, .NET, etc) should give the possibility to emit and process log record as defined in Logs Data Model.

Right now, the recommended approach is to use your logging library of choice (e.g. log4j, Serilog) to emit logs and use a bridge which converts and passes the log records to the SDK (via Logs Bridge API). At this point of time, OpenTelemetry does not offer Logs API which is intended to be used directly by application developers (this may change in future, but not sooner than in a few months).

If you want to export the data in a different format than OTLP, You can create your own custom exporter. More: https://opentelemetry.io/docs/specs/otel/logs/sdk/#logrecordexporter

[LeVraiSylvain]

Hi @pellared,

Thank you for your response, it's really appreciated. I apologize if my statement was vague. I'll try to be clearer.

Let me first go back to your message. You wrote this: “Here is the specification of the Logs Data Model: https://opentelemetry.io/docs/specs/otel/logs/data-model/
OTel libraries (e.g., OTel Java, .NET, etc.) should give the possibility to emit and process log records as defined in the Logs Data Model.”

My understanding is that we don’t emit logs directly in the "Logs Data Model" format, but what comes closest is the OTLP protocol. OTLP doesn’t perfectly implement the "Logs Data Model" contract. For example, there is a timeUnixNano field, while in the "Logs Data Model," this field is called timestamp. The structure is also quite different. That’s why I mentioned that what most resembles the "Logs Data Model" has been implemented in the OTLP protocol.

Regarding the "Logs Bridge API," I understand you are referring to appenders/sinks:

https://opentelemetry.io/docs/specs/otel/logs/bridge-api/
https://github.com/open-telemetry/oteps/blob/main/text/logs/0150-logging-library-sdk.md

Now, let me clarify my previous message.

We want to have standardized messages regardless of the protocol used. To meet this need, we will log a message as a structured data format instead of a simple string.

Let me give you an example in .NET:

logger.LogInformation("This is a sample message")

// If my logger is configured to log using the RFC5424 syslog protocol, I would get something like this:
<14>1 2020-04-15T15:43:06.000Z my-hostname my-service 1234 1234567890abcdefabcdef1234567890abcdef1234567890abcdef - traceId=abcdef1234567890abcdef1234567890 spanId=1234567890abcdef traceFlags=1 severityText=INFO severityNumber=9 body="This is a sample message" service.name="my-service" service.version=1.0.0 deployment.name="my deployment" deployment.environment.name=dev my.company.log_type=security http.request.method=GET url.full="https://my-hostname/api/endpoint1"

// If my logger is configured to log using the OTLP protocol, I would get something like this:
{
  "resourceLogs": [
    {
      "resource": {
        "attributes": []
      },
      "scopeLogs": [
        {
          "logRecords": [
            {
              "timeUnixNano": "1586960586000000000",
              "traceId": "abcdef1234567890abcdef1234567890",
              "spanId": "1234567890abcdef",
              "flags": 1,
              "severityText": "INFO",
              "severityNumber": 9,
              "body": {
                "stringValue": "traceId=abcdef1234567890abcdef1234567890 spanId=1234567890abcdef traceFlags=1 severityText=INFO severityNumber=9 body=\"This is a sample message\" service.name=\"my-service\" service.version=1.0.0 deployment.name=\"my deployment\" deployment.environment.name=dev my.company.log_type=security http.request.method=GET url.full=\"https://my-hostname/api/endpoint1\""
              },
              "attributes": []
            }
          ]
        }
      ]
    }
  ]
}

We could have used any structure to meet our need, but my architect chose to use the structure represented by the "Logs Data Model" and to use the standardized attributes from the registry: https://opentelemetry.io/docs/specs/semconv/attributes-registry/

So, my question is: “Is the Logs Data Model really the right fit or are we trying to fit a square peg into a round hole?”

As I mentioned in my previous message, the format of the "Logs Data Model" and the attributes defined in the registry seem to aim at representing all log formats within a common structure. Its implementation seems to make sense only in the OTEL Collector. I indeed see some code in Go that appears to implement it, e.g.:
type Timestamp uint64
Moreover, the attributes seem to be evolving, and several are marked as deprecated.
Would there really have been a container.id attribute if it hadn't been present in the "Elastic Common Schema"?

Thank you in advance!

[pellared]

My understanding is that we don’t emit logs directly in the "Logs Data Model" format, but what comes closest is the OTLP protocol.

Right now, the recommended approach is to use your logging library of choice (e.g. log4j, Serilog) to emit logs and use a bridge which converts and passes the log records to the SDK (via Logs Bridge API). At this point of time, OpenTelemetry does not offer Logs API which is intended to be used directly by application developers (this may change in future, but not sooner than in a few months).

I will try to rephrase it.
Logs Data Model is the SDKs and Collectors representation of logs. You are supposed to emit logs using logging libraries of choice. The log bridges/appenders are supposed to translate them to the Logs Data Model. Then the SDK will export them using the protocol of your choice e.g. OTLP. Currently, it is not advised to emit logs "directly" in Logs Data Model.

[LeVraiSylvain]

Thank you @pellared for these clarifications