Setting event timestamp from OTel observed timestamp

[LikeTheSalad]

Motivation/summary

Setting event timestamp from OTel observed timestamp when needed.

apm-data: elastic/apm-data#166

Checklist

• Update CHANGELOG.asciidoc
• Update package changelog.yml (only if changes to apmpackage have been made)
• Documentation has been updated

For functional changes, consider:

• Is it observable through the addition of either logging or metrics?
• Is its use being published in telemetry to enable product improvement?
• Have system tests been added to avoid regression?

How to test these changes

Related issues

[mergify]

This pull request does not have a backport label. Could you fix it @LikeTheSalad? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:

• backport-7.17 is the label to automatically backport to the 7.17 branch.
• backport-8./d is the label to automatically backport to the 8./d branch. /d is the digit.

NOTE: backport-skip has been added to this pull request.

[mergify]

This pull request is now in conflicts. Could you fix it @LikeTheSalad? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-observed-timestamp upstream/otel-observed-timestamp
git merge upstream/main
git push upstream otel-observed-timestamp

[simitt]

@LikeTheSalad apologies for the review, I'll approve once the conflicts are fixed.

[kruskall]

We might to wait before merging this until we sort out otel partial success PR elastic/apm-data#156

[mergify]

This pull request is now in conflicts. Could you fix it @LikeTheSalad? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-observed-timestamp upstream/otel-observed-timestamp
git merge upstream/main
git push upstream otel-observed-timestamp

[mergify]

This pull request is now in conflicts. Could you fix it @LikeTheSalad? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-observed-timestamp upstream/otel-observed-timestamp
git merge upstream/main
git push upstream otel-observed-timestamp

[LikeTheSalad]

We might to wait before merging this until we sort out otel partial success PR elastic/apm-data#156

@kruskall @simitt What's the status on this?

[mergify]

This pull request is now in conflicts. Could you fix it @LikeTheSalad? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b otel-observed-timestamp upstream/otel-observed-timestamp
git merge upstream/main
git push upstream otel-observed-timestamp

[kruskall]

@LikeTheSalad the breaking change should be fixed after elastic/apm-data#172

We can merge this! Can you fix the conflicts ?

[carsonip]

Testing notes

✔️ test-plan-ok

Using a patched version of python otel exporter that sends observed_timestamp with no timestamp, send the following logs to apm-server 8.12

LogRecord #0
ObservedTimestamp: 2020-01-20 14:57:21 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText: WARNING
SeverityNumber: Warn(13)
Body: Str(Jail zesty vixen who grabbed pay from quack.)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #1
ObservedTimestamp: 2020-01-20 14:57:21 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText: ERROR
SeverityNumber: Error(17)
Body: Str(The five boxing wizards jump quickly.)
Trace ID: 
Span ID: 
Flags: 0
LogRecord #2
ObservedTimestamp: 2020-01-20 14:57:21 +0000 UTC
Timestamp: 1970-01-01 00:00:00 +0000 UTC
SeverityText: ERROR
SeverityNumber: Error(17)
Body: Str(Hyderabad, we have a major problem.)
Trace ID: 1c849e7da1326d97451926b1b1fc15fd
Span ID: 316b290eee80d1d8
Flags: 1
        {"kind": "exporter", "data_type": "logs", "name": "logging"}

In discover