Add fingerprint ingest processor

[gaobinlong]

Description

Add a new ingest processor named fingerprint which generate a hash value for some specified fields or fields not in the specified excluded list and write the hash value to the target_field, the hash value can be used to deduplicate documents within a index and collapse search results.

The usage of this processor is:

"processors": [
      {
        "fingerprint": {
          "fields": ["foo", "bar"],
          "target_field": "fingerprint",
          "hash_method": "[email protected]",
          "ignore_missing": true
        }
      }
    ]

or

"processors": [
      {
        "fingerprint": {
          "exclude_fields": ["zoo"],
          "target_field": "fingerprint"
        }
      }
    ]

The main parameters in this processor are:

1. fields: fields in the document used to generate hash value, field name and value are concatenated and separated by |, like |field1|value1|field2|value2|, for nested fields, the field name is flattened, like |root_field.sub_field1|value1|root_field.sub_field2|value2|

1. fields: fields in the document used to generate hash value, field name, ${value length}:value are concatenated and separated by |, like |field1|3:value1|field2|10:value2|, for nested fields, the field name is flattened, like |root_field.sub_field1|1:value1|root_field.sub_field2|100:value2|
   2. include_all_fields: whether all fields are included to generate the hash value, either fields or include_all_fields can be set.
2. exclude_fields: fields not in this list are used to generate the hash value, either fields and exclude_fields can be non-empty
3. hash_method: [email protected], [email protected], [email protected] or [email protected], [email protected] is the default hash method. This processor is introduced in 2.16.0, we append the OpenSearch version to the hash method name to ensure that this processor always generates same hash value based on a specific hash method, if the processing logic of this processor changes in future version, then this parameter will support new hash method with new version.
4. target_field: the field to store the hash value
5. ignore_missing: if one of the specified fields is missing, the processor will exit quietly and do nothing.

In addition, if fields and exclude_fields are both empty or null, it means include all fields, all fields are used to generate the hash value.

Related Issues

#13612

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• API changes companion pull request created.
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

❌ Gradle check result for 50cdb84: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for e6b8851: SUCCESS

[codecov]

Codecov Report

Attention: Patch coverage is 80.17241% with 23 lines in your changes missing coverage. Please review.

Project coverage is 71.71%. Comparing base (b15cb0c) to head (1f2d5d0).
Report is 424 commits behind head on main.

Files	Patch %	Lines
...opensearch/ingest/common/FingerprintProcessor.java	80.53%	11 Missing and 11 partials ⚠️
...search/ingest/common/IngestCommonModulePlugin.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #13724      +/-   ##
============================================
+ Coverage     71.42%   71.71%   +0.29%     
- Complexity    59978    62103    +2125     
============================================
  Files          4985     5118     +133     
  Lines        282275   291805    +9530     
  Branches      40946    42186    +1240     
============================================
+ Hits         201603   209277    +7674     
- Misses        63999    65283    +1284     
- Partials      16673    17245     +572     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[github-actions]

❌ Gradle check result for 4a37513: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

✅ Gradle check result for 1f2d5d0: SUCCESS

[linuxpi]

❌ Gradle check result for 62703cb: FAILURE
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Known issue:#11085

@gaobinlong This issue linked to this failure is incorrect. The test failing the build is

[org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElsewhereDoesNotDeleteAnother {p0={"cluster.remote_store.state.enabled":"true"}}](https://build.ci.opensearch.org/job/gradle-check/40909/testReport/junit/org.opensearch.indices.store/IndicesStoreIntegrationIT/testShardActiveElsewhereDoesNotDeleteAnother__p0___cluster_remote_store_state_enabled___true___/)

but the issue linked is for - org.opensearch.remotestore.RemoteStoreRestoreIT

[gaobinlong]

❌ Gradle check result for 62703cb: FAILURE
Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Known issue:#11085

@gaobinlong This issue linked to this failure is incorrect. The test failing the build is

[org.opensearch.indices.store.IndicesStoreIntegrationIT.testShardActiveElsewhereDoesNotDeleteAnother {p0={"cluster.remote_store.state.enabled":"true"}}](https://build.ci.opensearch.org/job/gradle-check/40909/testReport/junit/org.opensearch.indices.store/IndicesStoreIntegrationIT/testShardActiveElsewhereDoesNotDeleteAnother__p0___cluster_remote_store_state_enabled___true___/)

but the issue linked is for - org.opensearch.remotestore.RemoteStoreRestoreIT

Sorry for that, I just saw they have same error message: java.lang.AssertionError: Missing cluster-manager... so I think they are the same issue, actually it should be linked to #12788.