Add verbose pipeline parameter to output each processor's execution details

[junweid62]

Description

Related RFC : #16705

This PR introduces enhancements to OpenSearch's search pipeline functionality, focusing on improving the traceability and debugging of search request and response transformations. It addresses the increasing complexity of search pipeline processors by implementing verbose mode support, which provides detailed insights into processor execution.

1. Adds Verbose Mode for Search Pipelines:

   • Introduced the verbose_pipeline parameter to search requests, default to false.
   • Tracks each processor’s input, output, execution time, and status (success/failure).
   • Provides detailed logs of the data flow through request and response processors.

2. Improves Pipeline Debugging:

   • Captures step-by-step data transformations applied by each processor.
   • Includes execution metadata (e.g., timestamps and elapsed time) in search responses for better analysis.

3. Supports All Pipeline Configurations:

   • Works seamlessly with:
     • Default pipelines configured at the index level.
     • Pipelines explicitly specified in the search request.
     • Ad-hoc pipelines defined inline.

4. Test Framework Enhancements:

   • Added tests for verbose mode to ensure correct functionality and compatibility.

Example output with request processor: filter_query response processor: rename_field and sort

{
    "took": 29,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.9116078,
        "hits": [
            {
                "_index": "my_index",
                "_id": "2",
                "_score": 0.9116078,
                "_source": {
                    "notification": "This is a public message 2",
                    "visibility": "public"
                }
            }
        ]
    },
    "processor_result": [
        {
            "processor_name": "filter_query",
            "duration_millis": 0,
            "input_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "match": {
                                    "message": {
                                        "query": "This is a public message 1",
                                        "operator": "OR",
                                        "prefix_length": 0,
                                        "max_expansions": 50,
                                        "fuzzy_transpositions": true,
                                        "lenient": false,
                                        "zero_terms_query": "NONE",
                                        "auto_generate_synonyms_phrase_query": true,
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            },
            "output_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "bool": {
                                    "must": [
                                        {
                                            "match": {
                                                "message": {
                                                    "query": "This is a public message 1",
                                                    "operator": "OR",
                                                    "prefix_length": 0,
                                                    "max_expansions": 50,
                                                    "fuzzy_transpositions": true,
                                                    "lenient": false,
                                                    "zero_terms_query": "NONE",
                                                    "auto_generate_synonyms_phrase_query": true,
                                                    "boost": 1.0
                                                }
                                            }
                                        }
                                    ],
                                    "adjust_pure_negative": true,
                                    "boost": 1.0
                                }
                            }
                        ],
                        "filter": [
                            {
                                "term": {
                                    "visibility": {
                                        "value": "public",
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            }
        },
        {
            "processor_name": "rename_field",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "message": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        },
        {
            "processor_name": "sort",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        }
    ]
}

Related Issues

Resolves #14745

Check List

• Functionality includes testing.
• API changes companion pull request created, if applicable.
• Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[junweid62]

@joshpalis Hey Josh, can you take a look when you have time? :)

[github-actions]

❌ Gradle check result for 488377f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❌ Gradle check result for 719fa1c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[joshpalis]

Great work @junweid62 on implementing this feature. A few comments from my initial pass

[github-actions]

❌ Gradle check result for e4e30f5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

[github-actions]

❕ Gradle check result for 615a4b6: UNSTABLE

• TEST FAILURES:

      2 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

[codecov]

Codecov Report

Attention: Patch coverage is 42.45810% with 103 lines in your changes missing coverage. Please review.

Project coverage is 72.04%. Comparing base (b67cdf4) to head (615a4b6).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
...arch/search/pipeline/ProcessorExecutionDetail.java	18.44%	84 Missing ⚠️
.../java/org/opensearch/search/pipeline/Pipeline.java	68.18%	6 Missing and 1 partial ⚠️
...a/org/opensearch/action/search/SearchResponse.java	16.66%	4 Missing and 1 partial ⚠️
...opensearch/search/builder/SearchSourceBuilder.java	78.57%	0 Missing and 3 partials ⚠️
...ensearch/action/search/SearchResponseSections.java	77.77%	1 Missing and 1 partial ⚠️
...pensearch/rest/action/search/RestSearchAction.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16843      +/-   ##
============================================
+ Coverage     72.03%   72.04%   +0.01%     
- Complexity    65230    65238       +8     
============================================
  Files          5318     5319       +1     
  Lines        304051   304224     +173     
  Branches      43990    44022      +32     
============================================
+ Hits         219021   219187     +166     
- Misses        67121    67134      +13     
+ Partials      17909    17903       -6     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.