Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support SQS message (user) attributes #22

Open
piequi opened this issue Mar 14, 2023 · 0 comments · May be fixed by #23
Open

Support SQS message (user) attributes #22

piequi opened this issue Mar 14, 2023 · 0 comments · May be fixed by #23
Labels
enhancement New feature or request

Comments

@piequi
Copy link

piequi commented Mar 14, 2023

Currently the SQS input plugin only handles message body, no matter there are message attributes defined by the user. There is a need to handle those attributes when receiving log events because they could contain relevant metadata for the event logged.

We, for example, have implemented a Python logging handler that sends JSON records containing log messages to a SQS queue. Those messages are read by Logstash, processed and indexed in Elasticsearch.

The content of those JSON records is defined by https://docs.python.org/3/library/logging.html#logrecord-attributes.

With the current implementation of the SQS input plugin, we cannot enrich those records with context metadata (like the cloud region were the service emitting the logs is running).

Using SQS message user attributes, we could add alongside to the log records some metadata to enrich the corresponding event logged.

The idea could be to handle those user attributes in

def add_sqs_data(event, message)

I already started implementing this and will submit a PR shortly for review.

@piequi piequi added the enhancement New feature or request label Mar 14, 2023
@piequi piequi changed the title Support for SQS message (user) attributes Support SQS message (user) attributes Mar 14, 2023
@piequi piequi linked a pull request Mar 14, 2023 that will close this issue
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant