Add Community ID ingest pipeline processor

[bodgit]

Is your feature request related to a problem? Please describe.

OpenSearch appears to be missing the Community ID ingest pipeline processor. This generates a portable ID to uniquely identify a network traffic flow based on the source/destination IP/port and transport, so rather than having to always do a five-way join, you can search based on this ID.

I was working through ingesting AWS VPC flow logs to OpenSearch and trying to keep the document mappings as close to the format used by the commercial offering as possible and noticed I can't compute this particular field due to the processor being missing.

It looks like it was added to ElasticSearch in version 7.12 but the specification of how to compute the ID is open.

Describe the solution you'd like

Add the missing processor 😉

Additional context

• https://github.com/corelight/community-id-spec
• https://www.elastic.co/guide/en/elasticsearch/reference/7.12/community-id-processor.html
• https://docs.elastic.co/en/integrations/aws/vpcflow

[dblock]

@bodgit Are you going to try and contribute an implementation to OpenSearch? PRs welcome!

[tybalex]

does Opensearch ever support ingest pipeline processor? Please let me know because I can't find it anywhere..thanks.

[vagimeli]

@heemin32 Is this an issue we need to add to the documentation issue opensearch-project/documentation-website#4193?

[heemin32]

@heemin32 Is this an issue we need to add to the documentation issue opensearch-project/documentation-website#4193?

The community ID ingest pipeline processor is not available in OpenSearch yet. We can create an issue in documentation repo once implementation starts.

[heemin32]

does Opensearch ever support ingest pipeline processor? Please let me know because I can't find it anywhere..thanks.

Opensearch do support ingest pipeline processor. We are just missing documentations for it and currently there is an ongoing effort to add the documentations. opensearch-project/documentation-website#4193

[reta]

@gaobinlong could you please a documentation issue for 2.13.0 for this new processor? thank you

[vagimeli]

@gaobinlong Please tag me in the PR when ready for a doc review or for technical writer support. Thanks!

[reta]

@gaobinlong Please tag me in the PR when ready for a doc review or for technical writer support. Thanks!

@vagimeli thank you, just to reiterate, as of today target is next release (2.13.0) since 2.12.0 is already cut, thank you

[gaobinlong]

@gaobinlong Please tag me in the PR when ready for a doc review or for technical writer support. Thanks!

@vagimeli thank you, just to reiterate, as of today target is next release (2.13.0) since 2.12.0 is already cut, thank you

@reta, @vagimeli Yeah, I've created a document issue about this feature and the label 2.13.0 was tagged on it, I'll open a PR later.