PoC DO NOT MERGE - Semantic Query #11
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
carlosdelest
commented
Nov 8, 2023
| @@ -71,6 +71,8 @@ protected UpdateByQueryRequest buildRequest(RestRequest request, NamedWriteableR | |||
| consumers.put("script", o -> internal.setScript(Script.parse(o))); | |||
| consumers.put("max_docs", s -> setMaxDocsValidateIdentical(internal, ((Number) s).intValue())); | |||
|
|
|||
| // TODO There surely must be a better way of doing this | |||
| request.params().put("_source_includes", "*"); | |||
There was a problem hiding this comment.
Included source exclusion from the demo. Hacky but gets the job done for now
| * sort them according to the provided order. This can be useful for instance to ensure that shards that contain recent | ||
| * data are executed first when sorting by descending timestamp. | ||
| */ | ||
| final class CoordinatorQueryRewriteSearchPhase extends SearchPhase { |
There was a problem hiding this comment.
New search phase for rewriting queries in the coordinator node
| ThreadPool threadPool, | ||
| SearchResponse.Clusters clusters | ||
| ) { | ||
| if (preFilter) { | ||
| if (runCoordinatorPhase) { |
There was a problem hiding this comment.
Added as a pre-step for search phases
| this.fieldNamesToInferenceModel = Map.of(); | ||
| } | ||
|
|
||
| public CoordinatorRewriteContext( |
There was a problem hiding this comment.
A new constructor for the rewriting in the coordinator search phase
| @@ -63,4 +66,9 @@ public CoordinatorRewriteContext getCoordinatorRewriteContext(Index index) { | |||
|
|
|||
| return new CoordinatorRewriteContext(parserConfig, client, nowInMillis, timestampRange, dateFieldType); | |||
| } | |||
|
|
|||
| @Nullable | |||
| public CoordinatorRewriteContext getCoordinatorRewriteContextForModels(Map<String, Set<String>> fieldToModelIds) { | |||
There was a problem hiding this comment.
New method for obtaining the rewrite context for the new search phase
| if (source == null) { | ||
| return false; | ||
| } | ||
| return source.subSearches().stream().anyMatch(sqwb -> sqwb.getQueryBuilder() instanceof CoordinatorRewriteableQueryBuilder); |
There was a problem hiding this comment.
I've used a marker interface for this - we can change that later
| @@ -124,7 +125,8 @@ public static FetchSourceContext parseFromRestRequest(RestRequest request) { | |||
| if (fetchSource != null || sourceIncludes != null || sourceExcludes != null) { | |||
| return FetchSourceContext.of(fetchSource == null || fetchSource, sourceIncludes, sourceExcludes); | |||
| } | |||
| return null; | |||
|
|
|||
| return FetchSourceContext.of(true, null, new String[]{"*." + SemanticTextFieldMapper.SPARSE_VECTOR_SUBFIELD_NAME}); | |||
There was a problem hiding this comment.
Hacky source exclusion
| return inferenceResultsToQuery(fieldName, inferenceResultsSupplier.get()); | ||
| } | ||
|
|
||
| Set<String> modelNames = coordinatorRewriteContext.inferenceModelsForFieldName(fieldName); |
There was a problem hiding this comment.
Retrieves model ids for the field and performs some validations.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PoC for a
semantic_queryquery builder:Uses a new search phase for performing coordinator level query rewriting.
New information for field to model IDs is added to the coordinator rewrite context so the query can perform the actual inference on rewriting.
You can follow this gist to test.