AWS Lambda function logs are cluttered with OpenTelemetry errors

[imanshafiei540]

I am using AWS Lambda with Python 3.9 and OpenTelemetry SDK 1.17.0 to send metrics to an ADOT collector. The collector then exports the metrics to an endpoint. While everything is working as expected, I am seeing a few error logs in the Lambda log group, making it hard to maintain the logs.

I need help with either disabling these logs or fixing the errors. Here are some details of the environment variables and configuration files I'm using:

Tools:

1. AWS Lambda
2. Python==3.9
3. opentelemetry-sdk==1.17.0
4. arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python-amd64-ver-1-17-0:1 (AWS Distro for OpenTelemetry Lambda Support)

Environment variables:

AWS_LAMBDA_EXEC_WRAPPER=/opt/otel-instrument
OPENTELEMETRY_COLLECTOR_CONFIG_FILE=/var/task/src/otel-config.yaml
OTEL_METRICS_EXPORTER=otlp
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OPENTELEMETRY_EXTENSION_LOG_LEVEL=error

Configuration file (otel-config.yaml):

receivers:
  otlp:
    protocols:
      http:

exporters:
  otlp:
    endpoint: https://<ANOTHER_SERVICE>:4317

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [otlp]

Here are the error logs that I'm receiving in the Lambda log group:

[WARNING] 2023-05-01T23:56:03.374Z An instrument with name http.server.duration, type Histogram, unit ms and description measures the duration of the inbound HTTP request has been created already.

[ERROR] 2023-05-01T23:56:04.958Z 42a36f0d-6bde-428d-833a-fc34f295a4a9 Failed to export batch code: 404, reason: 404 page not found

[ERROR] 2023-05-01T23:56:04.959Z 42a36f0d-6bde-428d-833a-fc34f295a4a9 Invalid type <class 'NoneType'> of value None
Traceback (most recent call last):
File "/opt/python/opentelemetry/exporter/otlp/proto/http/metric_exporter/init.py", line 345, in _translate_attributes
output.append(_translate_key_values(key, value))
File "/opt/python/opentelemetry/exporter/otlp/proto/http/metric_exporter/init.py", line 429, in _translate_key_values
return KeyValue(key=key, value=_translate_value(value))
File "/opt/python/opentelemetry/exporter/otlp/proto/http/metric_exporter/init.py", line 423, in _translate_value
raise Exception(f"Invalid type {type(value)} of value {value}")
Exception: Invalid type <class 'NoneType'> of value None

I would appreciate any help or suggestions to resolve these errors or disable the logs. Thank you.

[srikanthccv]

[ERROR] 2023-05-01T23:56:04.959Z 42a36f0d-6bde-428d-833a-fc34f295a4a9 Invalid type <class 'NoneType'> of value None
Traceback (most recent call last):
File "/opt/python/opentelemetry/exporter/otlp/proto/http/metric_exporter/init.py", line 345, in _translate_attributes
output.append(_translate_key_values(key, value))
File "/opt/python/opentelemetry/exporter/otlp/proto/http/metric_exporter/init.py", line 429, in _translate_key_values
return KeyValue(key=key, value=_translate_value(value))
File "/opt/python/opentelemetry/exporter/otlp/proto/http/metric_exporter/init.py", line 423, in _translate_value
raise Exception(f"Invalid type {type(value)} of value {value}")
Exception: Invalid type <class 'NoneType'> of value None

There is lambda instrumentation which adds a few HTTP attributes to the span https://github.com/open-telemetry/opentelemetry-python-contrib/blob/46e4b1da44c534fed8e1002899e9e41e6d668018/instrumentation/opentelemetry-instrumentation-aws-lambda/src/opentelemetry/instrumentation/aws_lambda/__init__.py#L193-L230

It uses .get(<header-name>), which can return None if the header is not set. And None isn't a valid span attribute value. The instrumentation should be updated to fix these warnings.

[WARNING] 2023-05-01T23:56:03.374Z An instrument with name http.server.duration, type Histogram, unit ms and description measures the duration of the inbound HTTP request has been created already.

I suspect this is a bug with instrumentation, but I could be wrong.

[ERROR] 2023-05-01T23:56:04.958Z 42a36f0d-6bde-428d-833a-fc34f295a4a9 Failed to export batch code: 404, reason: 404 page not found

This is probably because of the wrong endpoint configuration.

[imanshafiei540]

Thank you for your response.

I forgot to update my question.

For all of them you're right. I'm going to share my experience with each of them.

For the first one (ERROR), it was my fault, I was sending metrics with None in the attributes.

For the second one (WARNING) (https://github.com/open-telemetry/opentelemetry-python-contrib/blob/46e4b1da44c534fed8e1002899e9e41e6d668018/instrumentation/opentelemetry-instrumentation-aws-lambda/src/opentelemetry/instrumentation/aws_lambda/__init__.py#L193-L230), I'm going to open an issue in opentelemetry-python-contrib repo, is that a right repo to open the issue right?

Also, I realized that I'm creating a couple of Histograms and Counters in my logic and also my Lambda is being instrumented by ote-instrument-fastapi, so I just disabled by: OTEL_PYTHON_DISABLED_INSTRUMENTATIONS=fastapi

Thanks.

[srikanthccv]

Yes, you can create it in the contrib repo. If it turns out to be an issue in the core SDK, we can always transfer the issue.

[imanshafiei540]

Thank you so much. I created the issue. open-telemetry/opentelemetry-python-contrib#1780