Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify when flush returns unsuccessfully #623

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Specify when flush returns unsuccessfully #623

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
54fd027
Specify when flush returns unsuccessfully
felixbarny Mar 24, 2022
d0b99da
Apply suggestions from code review
felixbarny Mar 24, 2022
dc5350a
Fix typo
felixbarny Mar 24, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions specs/agents/tracing-instrumentation-aws-lambda.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,3 +291,17 @@ Therefore, the Lambda instrumentation has to ensure that data is flushed in a bl

Some Lambda functions will use the custom-built Lambda extension that allows the agent to send its data locally. The extension asynchronously forwards the data it receives from the agent to the APM server so the Lambda function can return its result with minimal delay. In order for the extension to know when it can flush its data, it must receive a signal indicating that the lambda function has completed. There are two possible signals: one is via a subscription to the AWS Lambda Logs API and the other is an agent intake request with the query param `flushed=true`. A signal from the agent is preferrable because there is an inherent delay with the sending of the Logs API signal.
Therefore, the agent must send its final intake request at the end of the function invocation with the query param `flushed=true`. In case there is no more data to send at the end of the function invocation, the agent must send an empty intake request with this query param.

### Flush timeout
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This option is used by the Java agent already. See https://www.elastic.co/guide/en/apm/agent/java/current/config-serverless.html#config-data-flush-timeout

Suggested change
### Flush timeout
### Configuration option `data_flush_timeout`


Agents SHOULD NOT wait for a successful flush indefinitely.
In the edge case where the extension takes too much time to respond (e.g. if there's a lenghy GC pause),
felixbarny marked this conversation as resolved.
Show resolved Hide resolved
the `flush` method should return after a timeout.

The default timeout is 1s.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a data point for what is currently happening in the agents, we use api_request_time as our flush timeout, which defaults to 10s.

That's likely too long, especially for Lambda. But I'm thinking we should create a new config option so this is configurable (and divorced from api_request_time).

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The default timeout is 1s.
| | |
|----------------|---|
| Type | [duration](configuration.md#configuration-value-types) |
| Default | `1s` |
| Dynamic | `true` |


### Flushing during a grace period

When the agent calls `flush` during a [grace period](transport.md#transport-errors),
or when entering a grace period while the agent waits for the flush to succeed,
felixbarny marked this conversation as resolved.
Show resolved Hide resolved
the agent MUST immediately return from the flush method.