[BUG] Processing A Nested List as Individual Log Events

[Conklin-Spencer-bah]

Describe the bug
It could potentially possible to do this however I have not been able to find anything in documentation that covers it.

If you have CloudWatch Logs -> Data Firehose -> S3 and want to pull that into DataPrepper it brings in the multi line event.

The structure seems to be like so:

{ "messageType": "DATA_MESSAGE", "owner": "123456789", "logGroup": "foo", "logStream": "bar", "logEvents": [{"id": "123456", "message": "some log message here", "timestamp" 1727880215114}, {"id": "789102", "message": "another log message here", "timestamp" 1727880215114}, {"id": "99999", "message": "yet another log message here", "timestamp" 1727880215114}]}

What I was hoping to do was use DataPrepper to read in the log message from S3 (that is like above) and then parse out the "logEvents" and treat each entry as an individual log message to publish to S3 & OpenSearch alike.

S3 being it will allow me to create a neat structure of a prefix with accountid/log-group/YYYY/MM/DD/HH

However I am not sure that it is possible to extract logEvents dictionary that contains a list of arrays and treat them as separate events.

To Reproduce
Steps to reproduce the behavior:

1. Create fake log event like above.
2. Write to DataPrepper
3. Try to parse

Expected behavior
I was expecting a feature within DataPrepper to support something like so:

  processor:
    - parse_json:
    - split_string:
        entries:
          - source: "/logEvents[0]"
            delimiter: ","

Environment (please complete the following information):

• OS: macOSX

Additional context
AWS Managed OpenSearch & AWS Managed OSIS is being used. I setup a local container deployment to expedite testing and still see the same issue.