Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Search Phase exception in Hybrid Query #10405

Closed
mgdunn2 opened this issue Oct 5, 2023 · 5 comments
Closed

[BUG] Search Phase exception in Hybrid Query #10405

mgdunn2 opened this issue Oct 5, 2023 · 5 comments
Labels
bug Something isn't working untriaged

Comments

@mgdunn2
Copy link

mgdunn2 commented Oct 5, 2023 •
edited
Loading

Describe the bug
When performing a hybrid query with one full text query and one KNN query the search often fails with a search_phase_execution_exception with

To Reproduce
Steps to reproduce the behavior:

  1. Create a new index with a KNN field and a text field
  2. Index 3 documents
  3. Add a normalization processor
  4. Perform a hybrid search with two queries, one with a match query and one with a knn query
  5. See error

The error is intermittent but I see it more often than I don't.

Expected behavior
Hybrid results are returned. Each of the two queries when run individually produce results and the hybrid query sometimes succeeds but more often than not fails.

Plugins
Using neural and KNN plugin. Currently have the security plugin disabled.

Screenshots
Create Index Request:

{
  "settings": {
    "index.knn": true
  },
  "mappings": {
    "properties": {
      "id": {
        "type": "keyword"
      },
      "vector": {
        "type": "knn_vector",
        "dimension": 3,
        "method": {
          "engine": "lucene",
          "space_type": "cosinesimil",
          "name": "hnsw",
          "parameters": {}
        }
      },
      "text": {
        "type": "text"
      }
    }
  }
}

Create Normalization Processor (with path _search/pipeline/nlp-search-pipeline) :

{
  "description": "Post processor for hybrid search",
  "phase_results_processors": [
    {
      "normalization-processor": {
        "normalization": {
          "technique": "min_max"
        },
        "combination": {
          "technique": "arithmetic_mean",
          "parameters": {
            "weights": [
              0.2,
              0.8
            ]
          }
        }
      }
    }
  ]
}

Sample Document:

{
  "text": "This is the third test",
  "id": "3",
  "vector": [0.31, 0.2, -0.11]
}

Sample Query (with params search_pipeline=nlp-search-pipeline):

{
	"query": {
		"hybrid": {
			"queries": [
				{
					"bool": {
						"should": {
							"match": {
								"text": {
									"query": "third"
								}
							}
						}
					}
				},
				{
					"knn": {
						"vector": {
							"vector": [
								-0.1,
								-0.2,
								0.3
							],
							"k": 3
						}
					}
				}
			]
		}
	}
}

Error Response:

{
	"error": {
		"root_cause": [],
		"type": "search_phase_execution_exception",
		"reason": "The phase has failed",
		"phase": "query",
		"grouped": true,
		"failed_shards": [],
		"caused_by": {
			"type": "null_pointer_exception",
			"reason": "Cannot invoke \"org.opensearch.search.SearchHit.score(float)\" because \"searchHit\" is null"
		}
	},
	"status": 500
}

Server error:

"org.opensearch.action.search.SearchPhaseExecutionException: The phase has failed
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseFailure(AbstractSearchAsyncAction.java:677) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.successfulShardExecution(AbstractSearchAsyncAction.java:596) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardResultConsumed(AbstractSearchAsyncAction.java:581) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.lambda$onShardResult$9(AbstractSearchAsyncAction.java:564) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.QueryPhaseResultConsumer$PendingMerges.consume(QueryPhaseResultConsumer.java:373) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.QueryPhaseResultConsumer.consumeResult(QueryPhaseResultConsumer.java:132) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onShardResult(AbstractSearchAsyncAction.java:564) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.SearchQueryThenFetchAsyncAction.onShardResult(SearchQueryThenFetchAsyncAction.java:159) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction$1.innerOnResponse(AbstractSearchAsyncAction.java:286) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:59) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.SearchActionListener.onResponse(SearchActionListener.java:44) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:99) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.SearchExecutionStatsCollector.onResponse(SearchExecutionStatsCollector.java:52) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:70) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.SearchTransportService$ConnectionCountingHandler.handleResponse(SearchTransportService.java:746) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.TransportService$6.handleResponse(TransportService.java:880) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1496) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:394) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.handleResponse(InboundHandler.java:386) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.messageReceived(InboundHandler.java:161) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundHandler.inboundMessage(InboundHandler.java:115) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.TcpTransport.inboundMessage(TcpTransport.java:767) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:175) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:150) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:115) [opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.transport.netty4.Netty4MessageChannelHandler.channelRead(Netty4MessageChannelHandler.java:95) [transport-netty4-client-2.10.0.jar:2.10.0]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) [netty-handler-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) [netty-transport-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) [netty-common-4.1.97.Final.jar:4.1.97.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.97.Final.jar:4.1.97.Final]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.opensearch.search.pipeline.SearchPipelineProcessingException: java.lang.NullPointerException: Cannot invoke "org.opensearch.search.SearchHit.score(float)" because "searchHit" is null
	at org.opensearch.search.pipeline.Pipeline.runSearchPhaseResultsTransformer(Pipeline.java:238) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.search.pipeline.PipelinedRequest.transformSearchPhaseResults(PipelinedRequest.java:40) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:714) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.successfulShardExecution(AbstractSearchAsyncAction.java:594) [opensearch-2.10.0.jar:2.10.0]
	... 43 more
Caused by: java.lang.NullPointerException: Cannot invoke "org.opensearch.search.SearchHit.score(float)" because "searchHit" is null
	at org.opensearch.neuralsearch.processor.NormalizationProcessorWorkflow.lambda$updateOriginalFetchResults$3(NormalizationProcessorWorkflow.java:154) ~[?:?]
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) ~[?:?]
	at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:992) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:575) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260) ~[?:?]
	at java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:616) ~[?:?]
	at org.opensearch.neuralsearch.processor.NormalizationProcessorWorkflow.updateOriginalFetchResults(NormalizationProcessorWorkflow.java:156) ~[?:?]
	at org.opensearch.neuralsearch.processor.NormalizationProcessorWorkflow.execute(NormalizationProcessorWorkflow.java:70) ~[?:?]
	at org.opensearch.neuralsearch.processor.NormalizationProcessor.process(NormalizationProcessor.java:62) ~[?:?]
	at org.opensearch.search.pipeline.Pipeline.runSearchPhaseResultsTransformer(Pipeline.java:219) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.search.pipeline.PipelinedRequest.transformSearchPhaseResults(PipelinedRequest.java:40) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.onPhaseDone(AbstractSearchAsyncAction.java:714) ~[opensearch-2.10.0.jar:2.10.0]
	at org.opensearch.action.search.AbstractSearchAsyncAction.successfulShardExecution(AbstractSearchAsyncAction.java:594) ~[opensearch-2.10.0.jar:2.10.0]
	... 43 more"

Host/Environment (please complete the following information):

  • OS: Linux running in Google Kubernetes Engine installed with Helm Chart
  • Version 2.10.0

Additional context
I did not see this error when testing with a single node local instance in docker but I'm seeing it consistently in the 3 node cluster in kubernetes.
@mgdunn2 mgdunn2 added bug Something isn't working untriaged labels Oct 5, 2023
@saratvemulapalli
Copy link
Member

saratvemulapalli commented Oct 5, 2023 •
edited
Loading

Lets start with with neural-search repo. Moving it over.
I cannot transfer this issue.
Anyway @navneet1v / @martin-gaievski can you folks take a look.

@mgdunn2
Copy link
Author

mgdunn2 commented Oct 5, 2023

Should I resubmit this in the neural-search repo?

@martin-gaievski
Copy link
Member

yes please, we need to check from neural-search perspective, it's seems very unlikely a core OpenSearch issue

@mgdunn2
Copy link
Author

mgdunn2 commented Oct 5, 2023

Submitted opensearch-project/neural-search#393

@navneet1v
Copy link
Contributor

closing this issue in favor of opensearch-project/neural-search#393

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working untriaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@mgdunn2 @navneet1v @saratvemulapalli @martin-gaievski