Add verbose pipeline parameter to output each processor's execution details #16843

junweid62 · 2024-12-13T00:00:40Z

Description

Related RFC : #16705

This PR introduces enhancements to OpenSearch's search pipeline functionality, focusing on improving the traceability and debugging of search request and response transformations. It addresses the increasing complexity of search pipeline processors by implementing verbose mode support, which provides detailed insights into processor execution.

Adds Verbose Mode for Search Pipelines:
- Introduced the verbose_pipeline parameter to search requests, default to false.
- Tracks each processor’s input, output, execution time, and status (success/failure).
- Provides detailed logs of the data flow through request and response processors.
Improves Pipeline Debugging:
- Captures step-by-step data transformations applied by each processor.
- Includes execution metadata (e.g., timestamps and elapsed time) in search responses for better analysis.
Supports All Pipeline Configurations:
- Works seamlessly with:
  - Default pipelines configured at the index level.
  - Pipelines explicitly specified in the search request.
  - Ad-hoc pipelines defined inline.
Test Framework Enhancements:
- Added tests for verbose mode to ensure correct functionality and compatibility.

Example output with request processor: filter_query response processor: rename_field and sort

{
    "took": 29,
    "timed_out": false,
    "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
    },
    "hits": {
        "total": {
            "value": 1,
            "relation": "eq"
        },
        "max_score": 0.9116078,
        "hits": [
            {
                "_index": "my_index",
                "_id": "2",
                "_score": 0.9116078,
                "_source": {
                    "notification": "This is a public message 2",
                    "visibility": "public"
                }
            }
        ]
    },
    "processor_result": [
        {
            "processor_name": "filter_query",
            "duration_millis": 0,
            "input_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "match": {
                                    "message": {
                                        "query": "This is a public message 1",
                                        "operator": "OR",
                                        "prefix_length": 0,
                                        "max_expansions": 50,
                                        "fuzzy_transpositions": true,
                                        "lenient": false,
                                        "zero_terms_query": "NONE",
                                        "auto_generate_synonyms_phrase_query": true,
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            },
            "output_data": {
                "query": {
                    "bool": {
                        "must": [
                            {
                                "bool": {
                                    "must": [
                                        {
                                            "match": {
                                                "message": {
                                                    "query": "This is a public message 1",
                                                    "operator": "OR",
                                                    "prefix_length": 0,
                                                    "max_expansions": 50,
                                                    "fuzzy_transpositions": true,
                                                    "lenient": false,
                                                    "zero_terms_query": "NONE",
                                                    "auto_generate_synonyms_phrase_query": true,
                                                    "boost": 1.0
                                                }
                                            }
                                        }
                                    ],
                                    "adjust_pure_negative": true,
                                    "boost": 1.0
                                }
                            }
                        ],
                        "filter": [
                            {
                                "term": {
                                    "visibility": {
                                        "value": "public",
                                        "boost": 1.0
                                    }
                                }
                            }
                        ],
                        "adjust_pure_negative": true,
                        "boost": 1.0
                    }
                },
                "search_pipeline": "my_pipeline"
            }
        },
        {
            "processor_name": "rename_field",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "message": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        },
        {
            "processor_name": "sort",
            "duration_millis": 0,
            "input_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ],
            "output_data": [
                {
                    "_index": "my_index",
                    "_id": "2",
                    "_score": 0.9116078,
                    "_source": {
                        "notification": "This is a public message 2",
                        "visibility": "public"
                    }
                }
            ]
        }
    ]
}

Related Issues

Resolves #14745

Check List

Functionality includes testing.
API changes companion pull request created, if applicable.
Public documentation issue/PR created, if applicable.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

…etails Signed-off-by: Junwei Dai <[email protected]>

Signed-off-by: Junwei Dai <[email protected]>

junweid62 · 2024-12-13T00:49:43Z

@joshpalis Hey Josh, can you take a look when you have time? :)

Signed-off-by: Junwei Dai <[email protected]>

github-actions · 2024-12-13T01:41:41Z

❌ Gradle check result for 488377f: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

github-actions · 2024-12-13T04:10:48Z

❌ Gradle check result for 719fa1c: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Junwei Dai <[email protected]>

joshpalis

Great work @junweid62 on implementing this feature. A few comments from my initial pass

server/src/main/java/org/opensearch/search/SearchHits.java

server/src/main/java/org/opensearch/search/internal/InternalSearchResponse.java

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

github-actions · 2024-12-13T18:38:27Z

❌ Gradle check result for e4e30f5: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

Signed-off-by: Junwei Dai <[email protected]>

github-actions · 2024-12-17T22:29:34Z

❕ Gradle check result for 615a4b6: UNSTABLE

TEST FAILURES:

      2 org.opensearch.cluster.MinimumClusterManagerNodesIT.testThreeNodesNoClusterManagerBlock

Please review all flaky tests that succeeded after retry and create an issue if one does not already exist to track the flaky failure.

codecov · 2024-12-17T22:30:22Z

Codecov Report

Attention: Patch coverage is 42.45810% with 103 lines in your changes missing coverage. Please review.

Project coverage is 72.04%. Comparing base (b67cdf4) to head (615a4b6).
Report is 11 commits behind head on main.

Files with missing lines	Patch %	Lines
...arch/search/pipeline/ProcessorExecutionDetail.java	18.44%	84 Missing ⚠️
.../java/org/opensearch/search/pipeline/Pipeline.java	68.18%	6 Missing and 1 partial ⚠️
...a/org/opensearch/action/search/SearchResponse.java	16.66%	4 Missing and 1 partial ⚠️
...opensearch/search/builder/SearchSourceBuilder.java	78.57%	0 Missing and 3 partials ⚠️
...ensearch/action/search/SearchResponseSections.java	77.77%	1 Missing and 1 partial ⚠️
...pensearch/rest/action/search/RestSearchAction.java	0.00%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main   #16843      +/-   ##
============================================
+ Coverage     72.03%   72.04%   +0.01%     
- Complexity    65230    65238       +8     
============================================
  Files          5318     5319       +1     
  Lines        304051   304224     +173     
  Branches      43990    44022      +32     
============================================
+ Hits         219021   219187     +166     
- Misses        67121    67134      +13     
+ Partials      17909    17903       -6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Signed-off-by: Junwei Dai <[email protected]>

github-actions · 2024-12-18T22:34:50Z

❌ Gradle check result for 0ae4d06: FAILURE

Please examine the workflow log, locate, and copy-paste the failure(s) below, then iterate to green. Is the failure a flaky test unrelated to your change?

junweid62 · 2024-12-19T23:26:51Z

Hi, @dbwiddis Could you please take a moment to review this PR when you have the time? I’d appreciate any feedback you might have :)

martin-gaievski · 2024-12-20T04:04:47Z

CHANGELOG.md

@@ -22,6 +22,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add new configuration setting `synonym_analyzer`, to the `synonym` and `synonym_graph` filters, enabling the specification of a custom analyzer for reading the synonym file ([#16488](https://github.com/opensearch-project/OpenSearch/pull/16488)).
 - Add stats for remote publication failure and move download failure stats to remote methods([#16682](https://github.com/opensearch-project/OpenSearch/pull/16682/))
 - Added a precaution to handle extreme date values during sorting to prevent `arithmetic_exception: long overflow` ([#16812](https://github.com/opensearch-project/OpenSearch/pull/16812)).
+- Add `verbose_pipeline` parameter to output each processor's execution details ([#14745](https://github.com/opensearch-project/OpenSearch/pull/14745)).


link has to point to PR #16843, current one is for the issue

martin-gaievski · 2024-12-20T04:09:58Z

server/src/main/java/org/opensearch/search/builder/SearchSourceBuilder.java

@@ -302,6 +305,9 @@ public SearchSourceBuilder(StreamInput in) throws IOException {
        if (in.getVersion().onOrAfter(Version.V_2_18_0)) {
            searchPipeline = in.readOptionalString();
        }
+        if (in.getVersion().onOrAfter(Version.CURRENT)) {


I think you need to have exact version here, CURRENT will change with every next release

martin-gaievski · 2024-12-20T04:10:22Z

server/src/main/java/org/opensearch/search/builder/SearchSourceBuilder.java

@@ -385,6 +391,9 @@ public void writeTo(StreamOutput out) throws IOException {
        if (out.getVersion().onOrAfter(Version.V_2_18_0)) {
            out.writeOptionalString(searchPipeline);
        }
+        if (out.getVersion().onOrAfter(Version.CURRENT)) {


same as in previous comment

martin-gaievski · 2024-12-20T04:11:36Z

server/src/main/java/org/opensearch/search/builder/SearchSourceBuilder.java

+        return this;
+    }
+
+    public Boolean verbosePipeline() {


why return type is a wrapper around primitive Boolean?

martin-gaievski · 2024-12-20T04:13:47Z

server/src/main/java/org/opensearch/search/internal/InternalSearchResponse.java

+
+    private static void writeProcessorResultOnOrAfter(StreamOutput out, List<ProcessorExecutionDetail> processorResult) throws IOException {
+        if (out.getVersion().onOrAfter(Version.V_2_18_0)) {
+            out.writeCollection(processorResult, (o, detail) -> detail.writeTo(o));


+1 to Josh's comment, please use out.writeList()

martin-gaievski · 2024-12-20T04:18:26Z

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

@@ -146,11 +147,18 @@ void transformRequest(SearchRequest request, ActionListener<SearchRequest> reque
            final ActionListener<SearchRequest> nextListener = currentListener;
            SearchRequestProcessor processor = searchRequestProcessors.get(i);
            currentListener = ActionListener.wrap(r -> {
+                ProcessorExecutionDetail detail = new ProcessorExecutionDetail(processor.getType());
+                detail.addInput(r.source().shallowCopy());


can you add check for verbosePipeline here, we're adding unnecessary overhead of doing shallow copy in case verbose is not enabled

martin-gaievski · 2024-12-20T04:20:12Z

server/src/main/java/org/opensearch/search/pipeline/Pipeline.java

@@ -225,11 +241,19 @@ ActionListener<SearchResponse> transformResponseListener(
            final SearchResponseProcessor processor = searchResponseProcessors.get(i);

            responseListener = ActionListener.wrap(r -> {
+                ProcessorExecutionDetail detail = new ProcessorExecutionDetail(processor.getType());
+                detail.addInput(Arrays.asList(r.getHits().deepCopy().getHits()));


we need to spend cycles on doing shallow copy only in case verbose flag is set, please consider making corresponding changes

martin-gaievski · 2024-12-20T04:29:52Z

server/src/main/java/org/opensearch/search/pipeline/PipelineProcessingContext.java

+        attributes.computeIfAbsent(PROCESSOR_EXECUTION_DETAILS_KEY, k -> new ArrayList<ProcessorExecutionDetail>());
+        List<ProcessorExecutionDetail> details = (List<ProcessorExecutionDetail>) attributes.get(PROCESSOR_EXECUTION_DETAILS_KEY);
+        details.add(detail);
+    }


Few extra cycles here for accessing map, how about this version:

public void addProcessorExecutionDetail(ProcessorExecutionDetail detail) { @SuppressWarnings("unchecked") List<ProcessorExecutionDetail> details = (List<ProcessorExecutionDetail>) attributes.computeIfAbsent( PROCESSOR_EXECUTION_DETAILS_KEY, k -> new ArrayList<>() ); details.add(detail); }

github-actions bot added enhancement Enhancement or improvement to existing feature or request Other Priority-High Search Search query, autocomplete ...etc v2.19.0 Issues and PRs related to version 2.19.0 labels Dec 13, 2024

junweid62 added the backport 2.x Backport to 2.x branch label Dec 13, 2024

Add verbose pipeline parameter to output each processor's execution d…

1c3b946

…etails Signed-off-by: Junwei Dai <[email protected]>

add change log

d931750

Signed-off-by: Junwei Dai <[email protected]>

junweid62 force-pushed the search-pipline-execution branch from d5a2c4c to d931750 Compare December 13, 2024 00:26

Add merge missing change log

1f879c1

Signed-off-by: Junwei Dai <[email protected]>

Add missing line

488377f

Signed-off-by: Junwei Dai <[email protected]>

opensearch-ci-bot mentioned this pull request Dec 13, 2024

[AUTOCUT] Gradle Check Flaky Test Report for IndexingIT #14302

Open

Refactor ProcessorExecutionDetail to improve field handling

e4e30f5

Signed-off-by: Junwei Dai <[email protected]>

junweid62 force-pushed the search-pipline-execution branch from 719fa1c to e4e30f5 Compare December 13, 2024 17:41

joshpalis reviewed Dec 13, 2024

View reviewed changes

opensearch-ci-bot mentioned this pull request Dec 14, 2024

[AUTOCUT] Gradle Check Flaky Test Report for SpecificClusterManagerNodesIT #15944

Open

Fix ITtest Fail

615a4b6

Signed-off-by: Junwei Dai <[email protected]>

opensearch-ci-bot mentioned this pull request Dec 18, 2024

[AUTOCUT] Gradle Check Flaky Test Report for MinimumClusterManagerNodesIT #14289

Open

Add more unit test

0ae4d06

Signed-off-by: Junwei Dai <[email protected]>

junweid62 force-pushed the search-pipline-execution branch from 820849b to 0ae4d06 Compare December 18, 2024 21:59

opensearch-ci-bot mentioned this pull request Dec 19, 2024

[AUTOCUT] Gradle Check Flaky Test Report for VerifyVersionConstantsIT #14585

Open

martin-gaievski reviewed Dec 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add verbose pipeline parameter to output each processor's execution details #16843

Add verbose pipeline parameter to output each processor's execution details #16843

junweid62 commented Dec 13, 2024 •

edited

Loading

junweid62 commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

joshpalis left a comment

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 17, 2024

codecov bot commented Dec 17, 2024 •

edited

Loading

github-actions bot commented Dec 18, 2024

junweid62 commented Dec 19, 2024

martin-gaievski Dec 20, 2024

martin-gaievski Dec 20, 2024

martin-gaievski Dec 20, 2024

martin-gaievski Dec 20, 2024

martin-gaievski Dec 20, 2024

martin-gaievski Dec 20, 2024

martin-gaievski Dec 20, 2024

martin-gaievski Dec 20, 2024

Add verbose pipeline parameter to output each processor's execution details #16843

Are you sure you want to change the base?

Add verbose pipeline parameter to output each processor's execution details #16843

Conversation

junweid62 commented Dec 13, 2024 • edited Loading

Description

Related Issues

Check List

junweid62 commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 13, 2024

joshpalis left a comment

Choose a reason for hiding this comment

github-actions bot commented Dec 13, 2024

github-actions bot commented Dec 17, 2024

codecov bot commented Dec 17, 2024 • edited Loading

Codecov Report

github-actions bot commented Dec 18, 2024

junweid62 commented Dec 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

junweid62 commented Dec 13, 2024 •

edited

Loading

codecov bot commented Dec 17, 2024 •

edited

Loading