[Backport 2.x] Tracing for deep search path (#12103)

[dzane17]

• Tracing for deep search path

Signed-off-by: David Zane [email protected]

• Refactor search phase tracing instrumentation to detach span creation from the SearchRequestOperationsListener

Signed-off-by: Andriy Redko [email protected]

• Close WrappedPhase span

Signed-off-by: David Zane [email protected]

• Add asserting listener class

Signed-off-by: David Zane [email protected]

• Cleanup AbstractSearchAsyncActionTests tests

Signed-off-by: Andriy Redko [email protected]

• Cleanup SearchAsyncActionTests tests

Signed-off-by: Andriy Redko [email protected]

• Cleanup CanMatchPreFilterSearchPhaseTests tests

Signed-off-by: Andriy Redko [email protected]

---

Signed-off-by: David Zane [email protected]
Signed-off-by: Andriy Redko [email protected]
Co-authored-by: Andriy Redko [email protected]
(cherry picked from commit 77a1193)

Description

2.x Backport for #12103

Check List

• New functionality includes testing.
  • All tests pass
• New functionality has been documented.
  • New functionality has javadoc added
• Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
• Commits are signed per the DCO using --signoff
• Commit changes are listed out in CHANGELOG.md file (See: Changelog)
• Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

Compatibility status:

Checks if related components are compatible with change cea6285

Incompatible components

Incompatible components: [https://github.com/opensearch-project/security-analytics.git]

Skipped components

Compatible components

Compatible components: [https://github.com/opensearch-project/custom-codecs.git, https://github.com/opensearch-project/asynchronous-search.git, https://github.com/opensearch-project/flow-framework.git, https://github.com/opensearch-project/observability.git, https://github.com/opensearch-project/reporting.git, https://github.com/opensearch-project/cross-cluster-replication.git, https://github.com/opensearch-project/job-scheduler.git, https://github.com/opensearch-project/neural-search.git, https://github.com/opensearch-project/common-utils.git, https://github.com/opensearch-project/geospatial.git, https://github.com/opensearch-project/k-nn.git, https://github.com/opensearch-project/anomaly-detection.git, https://github.com/opensearch-project/notifications.git, https://github.com/opensearch-project/sql.git, https://github.com/opensearch-project/ml-commons.git, https://github.com/opensearch-project/performance-analyzer-rca.git, https://github.com/opensearch-project/alerting.git, https://github.com/opensearch-project/index-management.git, https://github.com/opensearch-project/security.git, https://github.com/opensearch-project/performance-analyzer.git]

[github-actions]

❕ Gradle check result for cea6285: UNSTABLE

• TEST FAILURES:

      1 org.opensearch.index.shard.RemoteIndexShardTests.testNoFailuresOnFileReads
      1 org.opensearch.cluster.coordination.AwarenessAttributeDecommissionIT.testConcurrentDecommissionAction

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.