From d7993ad42eaf0e28b31997fe54a2f09cb9b4211f Mon Sep 17 00:00:00 2001 From: Naveen Tatikonda Date: Tue, 7 May 2024 17:15:47 -0500 Subject: [PATCH] Add documentation for k-NN clear cache api (#7030) * Add documentation for k-NN clear cache api Signed-off-by: Naveen Tatikonda * Update _search-plugins/knn/api.md Signed-off-by: Melissa Vagi * Update api.md Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Co-authored-by: Nathan Bower Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Signed-off-by: Melissa Vagi * Update api.md Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Signed-off-by: Melissa Vagi * Update _search-plugins/knn/api.md Signed-off-by: Melissa Vagi * Update api.md Signed-off-by: Melissa Vagi * Fix broken links Signed-off-by: Melissa Vagi * Fix broken links Signed-off-by: Melissa Vagi --------- Signed-off-by: Naveen Tatikonda Signed-off-by: Melissa Vagi Co-authored-by: Melissa Vagi Co-authored-by: Nathan Bower --- .../supported-field-types/knn-vector.md | 2 +- _search-plugins/knn/api.md | 173 +++++++++++------- _search-plugins/knn/approximate-knn.md | 2 +- _search-plugins/knn/knn-index.md | 4 +- _search-plugins/multimodal-search.md | 2 +- _search-plugins/semantic-search.md | 2 +- 6 files changed, 112 insertions(+), 73 deletions(-) diff --git a/_field-types/supported-field-types/knn-vector.md b/_field-types/supported-field-types/knn-vector.md index 741a86da64..c7f9ec7f2b 100644 --- a/_field-types/supported-field-types/knn-vector.md +++ b/_field-types/supported-field-types/knn-vector.md @@ -68,7 +68,7 @@ PUT test-index ## Model IDs Model IDs are used when the underlying Approximate k-NN algorithm requires a training step. As a prerequisite, the -model has to be created with the [Train API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api#train-model). The +model has to be created with the [Train API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api#train-a-model). The model contains the information needed to initialize the native library segment files. ```json diff --git a/_search-plugins/knn/api.md b/_search-plugins/knn/api.md index 5e8ea39b2e..23678063a8 100644 --- a/_search-plugins/knn/api.md +++ b/_search-plugins/knn/api.md @@ -1,6 +1,6 @@ --- layout: default -title: API +title: k-NN plugin API nav_order: 30 parent: k-NN search grand_parent: Search methods @@ -9,14 +9,12 @@ has_children: false # k-NN plugin API -The k-NN plugin adds several APIs for managing, monitoring and optimizing your k-NN workload. - +The k-NN plugin adds several APIs for managing, monitoring, and optimizing your k-NN workload. ## Stats -Introduced 1.0 -{: .label .label-purple } -The k-NN `stats` API provides information about the current status of the k-NN plugin. The plugin keeps track of both cluster-level and node-level statistics. Cluster-level statistics have a single value for the entire cluster. Node-level statistics have a single value for each node in the cluster. You can filter the query by `nodeId` and `statName`: +The k-NN `stats` API provides information about the current status of the k-NN plugin. The plugin keeps track of both cluster-level and node-level statistics. Cluster-level statistics have a single value for the entire cluster. Node-level statistics have a single value for each node in the cluster. You can filter the query by `nodeId` and `statName`, as shown in the following example: + ``` GET /_plugins/_knn/nodeId1,nodeId2/stats/statName1,statName2 ``` @@ -38,7 +36,7 @@ Statistic | Description `cache_capacity_reached` | Whether `knn.memory.circuit_breaker.limit` has been reached. This statistic is only relevant to approximate k-NN search. `load_success_count` | The number of times k-NN successfully loaded a native library index into the cache. This statistic is only relevant to approximate k-NN search. `load_exception_count` | The number of times an exception occurred when trying to load a native library index into the cache. This statistic is only relevant to approximate k-NN search. -`indices_in_cache` | For each OpenSearch index with a `knn_vector` field and approximate k-NN turned on, this statistic provides the number of native library indexes that OpenSearch index has and the total `graph_memory_usage` that the OpenSearch index is using, in kilobytes. +`indices_in_cache` | For each OpenSearch index with a `knn_vector` field and approximate k-NN turned on, this statistic provides the number of native library indexes that OpenSearch index has and the total `graph_memory_usage` that the OpenSearch index is using, in kilobytes. `script_compilations` | The number of times the k-NN script has been compiled. This value should usually be 1 or 0, but if the cache containing the compiled scripts is filled, the k-NN script might be recompiled. This statistic is only relevant to k-NN score script search. `script_compilation_errors` | The number of errors during script compilation. This statistic is only relevant to k-NN score script search. `script_query_requests` | The total number of script queries. This statistic is only relevant to k-NN score script search. @@ -46,16 +44,18 @@ Statistic | Description `nmslib_initialized` | Boolean value indicating whether the *nmslib* JNI library has been loaded and initialized on the node. `faiss_initialized` | Boolean value indicating whether the *faiss* JNI library has been loaded and initialized on the node. `model_index_status` | Status of model system index. Valid values are "red", "yellow", "green". If the index does not exist, this will be null. -`indexing_from_model_degraded` | Boolean value indicating if indexing from a model is degraded. This will happen if there is not enough JVM memory to cache the models. -`training_requests` | The number of training requests made to the node. +`indexing_from_model_degraded` | Boolean value indicating if indexing from a model is degraded. This happens if there is not enough JVM memory to cache the models. +`ing_requests` | The number of training requests made to the node. `training_errors` | The number of training errors that have occurred on the node. `training_memory_usage` | The amount of native memory training is using on the node in kilobytes. `training_memory_usage_percentage` | The amount of native memory training is using on the node as a percentage of the maximum cache capacity. -**Note**: Some stats contain *graph* in the name. In these cases, *graph* is synonymous with *native library index*. The term *graph* is a legacy detail, coming from when the plugin only supported the HNSW algorithm, which consists of hierarchical graphs. +Some statistics contain *graph* in the name. In these cases, *graph* is synonymous with *native library index*. The term *graph* is reflective of when the plugin only supported the HNSW algorithm, which consists of hierarchical graphs. +{: .note} +#### Usage -### Usage +The following code examples show how to retrieve statistics related to the k-NN plugin. The first example fetches comprehensive statistics for the k-NN plugin across all nodes in the cluster, while the second example retrieves specific metrics (circuit breaker status and graph memory usage) for a single node. ```json GET /_plugins/_knn/stats?pretty @@ -125,21 +125,17 @@ GET /_plugins/_knn/HYMrXXsBSamUkcAjhjeN0w/stats/circuit_breaker_triggered,graph_ } ``` - ## Warmup operation -Introduced 1.0 -{: .label .label-purple } - -The native library indexes used to perform approximate k-Nearest Neighbor (k-NN) search are stored as special files with other Apache Lucene segment files. In order for you to perform a search on these indexes using the k-NN plugin, the plugin needs to load these files into native memory. -If the plugin hasn't loaded the files into native memory, it loads them when it receives a search request. The loading time can cause high latency during initial queries. To avoid this situation, users often run random queries during a warmup period. After this warmup period, the files are loaded into native memory and their production workloads can begin. This loading process is indirect and requires extra effort. +The native library indexes used to perform approximate k-NN search are stored as special files with other Apache Lucene segment files. To perform a search on these indexes using the k-NN plugin, the plugin needs to load these files into native memory. -As an alternative, you can avoid this latency issue by running the k-NN plugin warmup API operation on whatever indexes you're interested in searching. This operation loads all the native library files for all of the shards (primaries and replicas) of all the indexes specified in the request into native memory. +If the plugin has not loaded the files into native memory, then it loads them when it receives a search request. The loading time can cause high latency during initial queries. To avoid this, users often run random queries during a warmup period. After this warmup period, the files are loaded into native memory, and their production workloads can launch. This loading process is indirect and requires extra effort. -After the process finishes, you can start searching against the indexes with no initial latency penalties. The warmup API operation is idempotent, so if a segment's native library files are already loaded into memory, this operation has no impact. It only loads files that aren't currently in memory. +As an alternative, you can avoid this latency issue by running the k-NN plugin warmup API operation on the indexes you want to search. This operation loads all the native library files for all the shards (primaries and replicas) of all the indexes specified in the request into native memory. +After the process is finished, you can search against the indexes without initial latency penalties. The warmup API operation is idempotent, so if a segment's native library files are already loaded into memory, this operation has no effect. It only loads files not currently stored in memory. -### Usage +#### Usage This request performs a warmup on three indexes: @@ -156,7 +152,7 @@ GET /_plugins/_knn/warmup/index1,index2,index3?pretty `total` indicates how many shards the k-NN plugin attempted to warm up. The response also includes the number of shards the plugin succeeded and failed to warm up. -The call doesn't return results until the warmup operation finishes or the request times out. If the request times out, the operation still continues on the cluster. To monitor the warmup operation, use the OpenSearch `_tasks` API: +The call does not return results until the warmup operation finishes or the request times out. If the request times out, then the operation continues on the cluster. To monitor the warmup operation, use the OpenSearch `_tasks` API: ```json GET /_tasks @@ -164,43 +160,89 @@ GET /_tasks After the operation has finished, use the [k-NN `_stats` API operation](#stats) to see what the k-NN plugin loaded into the graph. - ### Best practices For the warmup operation to function properly, follow these best practices: -* Don't run merge operations on indexes that you want to warm up. During merge, the k-NN plugin creates new segments, and old segments are sometimes deleted. For example, you could encounter a situation in which the warmup API operation loads native library indexes A and B into native memory, but segment C is created from segments A and B being merged. The native library indexes for A and B would no longer be in memory, and native library index C would also not be in memory. In this case, the initial penalty for loading native library index C is still present. +* Do not run merge operations on indexes that you want to warm up. During a merge operation, the k-NN plugin creates new segments, and old segments are sometimes deleted. For example, you could encounter a situation in which the warmup API operation loads native library indexes A and B into native memory but segment C is created from segments A and B being merged. Native library indexes A and B would no longer be in memory, and native library index C would also not be in memory. In this case, the initial penalty for loading native library index C still exists. * Confirm that all native library indexes you want to warm up can fit into native memory. For more information about the native memory limit, see the [knn.memory.circuit_breaker.limit statistic]({{site.url}}{{site.baseurl}}/search-plugins/knn/settings#cluster-settings). High graph memory usage causes cache thrashing, which can lead to operations constantly failing and attempting to run again. -* Don't index any documents that you want to load into the cache. Writing new information to segments prevents the warmup API operation from loading the native library indexes until they're searchable. This means that you would have to run the warmup operation again after indexing finishes. +* Do not index any documents that you want to load into the cache. Writing new information to segments prevents the warmup API operation from loading the native library indexes until they are searchable. This means that you would have to run the warmup operation again after indexing. -## Get Model -Introduced 1.2 +## k-NN clear cache +Introduced 2.14 {: .label .label-purple } -Used to retrieve information about models present in the cluster. Some native library index configurations require a -training step before indexing and querying can begin. The output of training is a model that can then be used to -initialize native library index files during indexing. The model is serialized in the k-NN model system index. +During approximate k-NN search or warmup operations, the native library indexes (`nmslib` and `faiss` engines) are loaded into native memory. Currently, you can evict an index from cache or native memory by either deleting the index or setting the k-NN cluster settings `knn.cache.item.expiry.enabled` and `knn.cache.item.expiry.minutes`, which removes the index from the cache if it is idle for a given period of time. However, you cannot evict an index from the cache without deleting the index. To solve this problem, you can use the k-NN clear cache API operation, which clears a given set of indexes from the cache. + +The k-NN clear cache API evicts all native library files for all shards (primaries and replicas) of all indexes specified in the request. Similarly to how the [warmup operation](#warmup-operation) behaves, the k-NN clear cache API is idempotent, meaning that if you try to clear the cache for an index that has already been evicted from the cache, it does not have any additional effect. + +This API operation only works with indexes created using the `nmslib` and `faiss` engines. It has no effect on indexes created using the `lucene` engine. +{: .note} + +#### Usage + +The following request evicts the native library indexes of three indexes from the cache: + +```json +GET /_plugins/_knn/clear_cache/index1,index2,index3?pretty +{ + "_shards" : { + "total" : 6, + "successful" : 6, + "failed" : 0 + } +} +``` + +The `total` parameter indicates the number of shards that the API attempted to clear from the cache. The response includes both the number of cleared shards and the number of shards that the plugin failed to clear. + +The k-NN clear cache API can be used with index patterns to clear one or more indexes that match the given pattern from the cache, as shown in the following example: + +```json +GET /_plugins/_knn/clear_cache/index*?pretty +{ + "_shards" : { + "total" : 6, + "successful" : 6, + "failed" : 0 + } +} +``` + +The API call does not return results until the operation finishes or the request times out. If the request times out, then the operation continues on the cluster. To monitor the request, use the `_tasks` API, as shown in the following example: + +```json +GET /_tasks +``` + +When the operation finishes, use the [k-NN `_stats` API operation](#stats) to see which indexes have been evicted from the cache. + +## Get a model + +The GET model operation retrieves information about models present in the cluster. Some native library index configurations require a training step before indexing and querying can begin. The output of training is a model that can be used to initialize native library index files during indexing. The model is serialized in the k-NN model system index. See the following GET example: ``` GET /_plugins/_knn/models/{model_id} ``` -Response Field | Description +Response field | Description :--- | :--- -`model_id` | The id of the fetched model. +`model_id` | The unique identifier of the fetched model. `model_blob` | The base64 encoded string of the serialized model. -`state` | Current state of the model. Either "created", "failed", "training". -`timestamp` | Time when the model was created. -`description` | User provided description of the model. -`error` | Error message explaining why the model is in the failed state. -`space_type` | Space type this model is trained for. -`dimension` | Dimension this model is for. -`engine` | Native library used to create model. Either "faiss" or "nmslib". +`state` | The model's current state, which can be `created`, `failed`, or `training`. +`timestamp` | The date and time when the model was created. +`description` | A user-provided description of the model. +`error` | An error message explaining why the model is in a failed state. +`space_type` | The space type for which this model is trained, for example, Euclidean or cosine. +`dimension` | The dimensionality of the vector space for which this model is designed. +`engine` | The native library used to create the model, either `faiss` or `nmslib`. ### Usage +The following examples show how to retrieve information about a specific model using the k-NN plugin API. The first example returns all the available information about the model, while the second example shows how to selectively retrieve fields. + ```json GET /_plugins/_knn/models/test-model?pretty { @@ -224,13 +266,14 @@ GET /_plugins/_knn/models/test-model?pretty&filter_path=model_id,state } ``` -## Search Model -Introduced 1.2 -{: .label .label-purple } +## Search for a model -Use an OpenSearch query to search for models in the index. +You can use an OpenSearch query to search for a model in the index. See the following usage example. + +#### Usage + +The following example shows how to search for k-NN models in an OpenSearch cluster and how to retrieve the metadata for those models, excluding the potentially large `model_blob` field: -### Usage ```json GET/POST /_plugins/_knn/models/_search?pretty&_source_excludes=model_blob { @@ -275,13 +318,13 @@ GET/POST /_plugins/_knn/models/_search?pretty&_source_excludes=model_blob } ``` -## Delete Model -Introduced 1.2 -{: .label .label-purple } +## Delete a model -Used to delete a particular model in the cluster. +You can delete a model in the cluster by using the DELETE operation. See the following usage example. -### Usage +#### Usage + +The following example shows how to delete a k-NN model: ```json DELETE /_plugins/_knn/models/{model_id} @@ -291,32 +334,28 @@ DELETE /_plugins/_knn/models/{model_id} } ``` -## Train Model -Introduced 1.2 -{: .label .label-purple } +## Train a model -Create and train a model that can be used for initializing k-NN native library indexes during indexing. This API will -pull training data from a `knn_vector` field in a training index and then create and train a model and then serialize it -to the model system index. Training data must match the dimension passed into the body of the request. This request -will return when training begins. To monitor the state of the model, use the [Get model API](#get-model). +You can create and train a model that can be used for initializing k-NN native library indexes during indexing. This API pulls training data from a `knn_vector` field in a training index, creates and trains a model, and then serializes it to the model system index. Training data must match the dimension passed in the request body. This request is returned when training begins. To monitor the model's state, use the [Get model API](#get-a-model). -Query Parameter | Description +Query parameter | Description :--- | :--- -`model_id` | (Optional) The id of the fetched model. If not specified, a random id will be generated. -`node_id` | (Optional) Preferred node to execute training. If set, this node will be used to perform training if it is deemed to be capable. +`model_id` | The unique identifier of the fetched model. If not specified, then a random ID is generated. Optional. +`node_id` | Specifies the preferred node on which to execute the training process. If provided, the specified node is used for training if it has the necessary capabilities and resources available. Optional. -Request Parameter | Description +Request parameter | Description :--- | :--- -`training_index` | Index from where training data from. -`training_field` | `knn_vector` field from `training_index` to grab training data from. Dimension of this field must match `dimension` passed in to this request. -`dimension` | Dimension this model is for. -`max_training_vector_count` | (Optional) Maximum number of vectors from the training index to use for training. Defaults to all of the vectors in the index. -`search_size` | (Optional) Training data is pulled from the training index with scroll queries. Defines the number of results to return per scroll query. Defaults to 10,000. -`description` | (Optional) User provided description of the model. -`method` | Configuration of ANN method used for search. For more information about possible methods, refer to the [method documentation]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index#method-definitions). Method must require training to be valid. +`training_index` | The index from which the training data is retrieved. +`training_field` | The `knn_vector` field in the `training_index` from which the training data is retrieved. The dimension of this field must match the `dimension` passed in this request. +`dimension` | The dimension of the model being trained. +`max_training_vector_count` | The maximum number of vectors from the training index to be used for training. Defaults to all the vectors in the index. Optional. +`search_size` | The training data is pulled from the training index using scroll queries. This parameter defines the number of results to return per scroll query. Default is `10000`. Optional. +`description` | A user-provided description of the model. Optional. +`method` | The configuration of the approximate k-NN method used for search operations. For more information about the available methods, see [k-NN index method definitions]({{site.url}}{{site.baseurl}}/search-plugins/knn/knn-index#method-definitions). The method requires training to be valid. +#### Usage -### Usage +The following examples show how to initiate the training process for a k-NN model: ```json POST /_plugins/_knn/models/{model_id}/_train?preference={node_id} diff --git a/_search-plugins/knn/approximate-knn.md b/_search-plugins/knn/approximate-knn.md index 16d1a7e686..d2cd550670 100644 --- a/_search-plugins/knn/approximate-knn.md +++ b/_search-plugins/knn/approximate-knn.md @@ -133,7 +133,7 @@ indicates how many results the query actually returns. The plugin returns `k` am ### Building a k-NN index from a model -For some of the algorithms that we support, the native library index needs to be trained before it can be used. It would be expensive to training every newly created segment, so, instead, we introduce the concept of a *model* that is used to initialize the native library index during segment creation. A *model* is created by calling the [Train API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api#train-model), passing in the source of training data as well as the method definition of the model. Once training is complete, the model will be serialized to a k-NN model system index. Then, during indexing, the model is pulled from this index to initialize the segments. +For some of the algorithms that we support, the native library index needs to be trained before it can be used. It would be expensive to training every newly created segment, so, instead, we introduce the concept of a *model* that is used to initialize the native library index during segment creation. A *model* is created by calling the [Train API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api#train-a-model), passing in the source of training data as well as the method definition of the model. Once training is complete, the model will be serialized to a k-NN model system index. Then, during indexing, the model is pulled from this index to initialize the segments. To train a model, we first need an OpenSearch index with training data in it. Training data can come from any `knn_vector` field that has a dimension matching the dimension of the model you want to create. Training data can be the same data that you are going to index or have in a separate set. Let's create a training index: diff --git a/_search-plugins/knn/knn-index.md b/_search-plugins/knn/knn-index.md index 9165ba3338..ab24a0c097 100644 --- a/_search-plugins/knn/knn-index.md +++ b/_search-plugins/knn/knn-index.md @@ -72,7 +72,7 @@ For the ARM64 architecture, only one performance-boosting Faiss library (`libope ## Method definitions -A method definition refers to the underlying configuration of the approximate k-NN algorithm you want to use. Method definitions are used to either create a `knn_vector` field (when the method does not require training) or [create a model during training]({{site.url}}{{site.baseurl}}/search-plugins/knn/api#train-model) that can then be used to [create a `knn_vector` field]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/#building-a-k-nn-index-from-a-model). +A method definition refers to the underlying configuration of the approximate k-NN algorithm you want to use. Method definitions are used to either create a `knn_vector` field (when the method does not require training) or [create a model during training]({{site.url}}{{site.baseurl}}/search-plugins/knn/api#train-a-model) that can then be used to [create a `knn_vector` field]({{site.url}}{{site.baseurl}}/search-plugins/knn/approximate-knn/#building-a-k-nn-index-from-a-model). A method definition will always contain the name of the method, the space_type the method is built for, the engine (the library) to use, and a map of parameters. @@ -137,7 +137,7 @@ For more information about setting these parameters, refer to the [Faiss documen #### IVF training requirements -The IVF algorithm requires a training step. To create an index that uses IVF, you need to train a model with the [Train API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api#train-model), passing the IVF method definition. IVF requires that, at a minimum, there are `nlist` training data points, but it is [recommended that you use more than this](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index#how-big-is-the-dataset). Training data can be composed of either the same data that is going to be ingested or a separate dataset. +The IVF algorithm requires a training step. To create an index that uses IVF, you need to train a model with the [Train API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api#train-a-model), passing the IVF method definition. IVF requires that, at a minimum, there are `nlist` training data points, but it is [recommended that you use more than this](https://github.com/facebookresearch/faiss/wiki/Guidelines-to-choose-an-index#how-big-is-the-dataset). Training data can be composed of either the same data that is going to be ingested or a separate dataset. ### Supported Lucene methods diff --git a/_search-plugins/multimodal-search.md b/_search-plugins/multimodal-search.md index cbc6a64286..6c7ddeed5b 100644 --- a/_search-plugins/multimodal-search.md +++ b/_search-plugins/multimodal-search.md @@ -108,7 +108,7 @@ Before the document is ingested into the index, the ingest pipeline runs the `te ## Step 4: Search the index using neural search -To perform vector search on your index, use the `neural` query clause either in the [k-NN plugin API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api/#search-model) or [Query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/index/) queries. You can refine the results by using a [k-NN search filter]({{site.url}}{{site.baseurl}}/search-plugins/knn/filter-search-knn/). You can search by text, image, or both text and image. +To perform vector search on your index, use the `neural` query clause either in the [k-NN plugin API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api/#search-for-a-model) or [Query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/index/) queries. You can refine the results by using a [k-NN search filter]({{site.url}}{{site.baseurl}}/search-plugins/knn/filter-search-knn/). You can search by text, image, or both text and image. The following example request uses a neural query to search for text and image: diff --git a/_search-plugins/semantic-search.md b/_search-plugins/semantic-search.md index 7c3fbb738f..259685fe3d 100644 --- a/_search-plugins/semantic-search.md +++ b/_search-plugins/semantic-search.md @@ -115,7 +115,7 @@ Before the document is ingested into the index, the ingest pipeline runs the `te ## Step 4: Search the index using neural search -To perform vector search on your index, use the `neural` query clause either in the [k-NN plugin API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api/#search-model) or [Query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/index/) queries. You can refine the results by using a [k-NN search filter]({{site.url}}{{site.baseurl}}/search-plugins/knn/filter-search-knn/). +To perform vector search on your index, use the `neural` query clause either in the [k-NN plugin API]({{site.url}}{{site.baseurl}}/search-plugins/knn/api/#search-for-a-model) or [Query DSL]({{site.url}}{{site.baseurl}}/opensearch/query-dsl/index/) queries. You can refine the results by using a [k-NN search filter]({{site.url}}{{site.baseurl}}/search-plugins/knn/filter-search-knn/). The following example request uses a Boolean query to combine a filter clause and two query clauses---a neural query and a `match` query. The `script_score` query assigns custom weights to the query clauses: