Support allow_duplicate_keys for json processor

[MunifTanjim]

Description

This is to support creating an ingest pipeline for ECS JSON logs.
Due to how some logging frameworks work, ECS JSON logs may contain duplicate keys. Instead of failing, the JSON parser should be more lenient and prefer the last value.

Ported from elastic/elasticsearch#74956

Related Issues

Resolves #[Issue number to be closed when this PR is merged]

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change 7498602

Incompatible components

Incompatible components: [https://github.com/opensearch-project/performance-analyzer.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/security-analytics.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/opensearch-oci-object-storage.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/sql.git]

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29467/
• CommitID: aa08236
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29468/
• CommitID: 59121c1
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29469/
• CommitID: 3d434b0
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29470/
• CommitID: 9492ce9
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29471/
• CommitID: 8d272d2
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29472/
• CommitID: b6326b5
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29473/
• CommitID: e787739
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29474/
• CommitID: d39a702
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: FAILURE ❌
• URL: https://build.ci.opensearch.org/job/gradle-check/29475/
• CommitID: 1a1420e
  Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green.
  Is the failure a flaky test unrelated to your change?

[github-actions]

Gradle Check (Jenkins) Run Completed with:

• RESULT: UNSTABLE ❕
• TEST FAILURES:

      1 org.opensearch.search.SearchWeightedRoutingIT.testMultiGetWithNetworkDisruption_FailOpenEnabled

• URL: https://build.ci.opensearch.org/job/gradle-check/29476/
• CommitID: 7498602
  Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Merging #11059 (7498602) into main (63aff16) will decrease coverage by 0.04%.
Report is 5 commits behind head on main.
The diff coverage is 70.00%.

@@             Coverage Diff              @@
##               main   #11059      +/-   ##
============================================
- Coverage     71.29%   71.25%   -0.04%     
- Complexity    58742    58750       +8     
============================================
  Files          4872     4872              
  Lines        276777   276791      +14     
  Branches      40240    40241       +1     
============================================
- Hits         197316   197218      -98     
- Misses        62943    63157     +214     
+ Partials      16518    16416     -102     

Files	Coverage Δ
...a/org/opensearch/core/xcontent/XContentParser.java	100.00% <ø> (ø)
...earch/common/xcontent/cbor/CborXContentParser.java	100.00% <100.00%> (ø)
...earch/common/xcontent/json/JsonXContentParser.java	83.33% <100.00%> (+0.40%)	⬆️
...rch/common/xcontent/smile/SmileXContentParser.java	100.00% <100.00%> (ø)
...va/org/opensearch/ingest/common/JsonProcessor.java	92.59% <100.00%> (+0.43%)	⬆️
...s/replication/SegmentReplicationTargetService.java	56.17% <100.00%> (+0.56%)	⬆️
...rg/opensearch/core/xcontent/MapXContentParser.java	72.61% <0.00%> (-0.47%)	⬇️
...ch/common/xcontent/JsonToStringXContentParser.java	43.90% <0.00%> (-0.55%)	⬇️
...rg/opensearch/core/xcontent/XContentSubParser.java	44.28% <0.00%> (-1.31%)	⬇️
.../java/org/opensearch/ingest/common/Processors.java	0.00% <0.00%> (ø)

... and 477 files with indirect coverage changes

[opensearch-trigger-bot]

This PR is stalled because it has been open for 30 days with no activity.

[MunifTanjim]

This is still relevant! @opensearch-ci-bot

[dblock]

Thanks, not sure how we missed the PR. Sorry about that. Please iterate to green. Needs a CHANGELOG notably.

I don't know much about the xcontent business, maybe @reta can review this? Who else?

[reta]

I don't know much about the xcontent business, maybe @reta can review this? Who else?

Thanks for the pull request @MunifTanjim but we cannot port features from elasticsearch "as-is" without violating license

[dblock]

Oh I didn't notice the port from Elastic. That PR is from Jul 6, 2021 which is after 7.10.2 I believe, closing this PR.

Anyone wanting this feature in OpenSearch: please ensure that you're not copying any non-APLv2 code.