From 51e950e1436bb6c2974e323ed462f4b976b7f097 Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Mon, 25 Nov 2024 23:18:51 +0100 Subject: [PATCH 1/7] Dedupe and clarify "Search your data" (#213) * Dedupe and clarify "Search your data" > Important: These docs now avoid duplicating content from core ES documentation to reduce maintenance overhead and prevent documentation drift. Instead, they focus on serverless-specific content and refer to core docs for implementation details. These docs were manually duplicated a looong time ago and are now incomplete and out of date. - Removed duplicated implementation guides - References core ES docs - Added tip for "bring your own vectors" tutorial - Removed duplicated implementation details - Points to core ES docs - Marked as duplicate/removed from navigation - Added Retrievers section - Simplified Query DSL explanation - References core ES docs - Rewritten for serverless context - Links to quickstart tutorials and notebooks - Points to core docs for concepts * Delete synonyms page (redirect is already in place) * fixes per review * Apply suggestions from code review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --------- Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- .../index-serverless-elasticsearch.asciidoc | 1 - serverless/pages/knn-search.asciidoc | 1078 +---------------- .../pages/search-with-synonyms.asciidoc | 120 -- .../search-your-data-the-search-api.asciidoc | 23 +- serverless/pages/search-your-data.asciidoc | 35 +- 5 files changed, 44 insertions(+), 1213 deletions(-) delete mode 100644 serverless/pages/search-with-synonyms.asciidoc diff --git a/serverless/index-serverless-elasticsearch.asciidoc b/serverless/index-serverless-elasticsearch.asciidoc index 64d53024..a25e8c9c 100644 --- a/serverless/index-serverless-elasticsearch.asciidoc +++ b/serverless/index-serverless-elasticsearch.asciidoc @@ -37,7 +37,6 @@ include::./pages/ingest-your-data-ingest-data-through-integrations-beats.asciido include::./pages/search-your-data.asciidoc[leveloffset=+2] include::./pages/search-your-data-the-search-api.asciidoc[leveloffset=+3] -include::./pages/search-with-synonyms.asciidoc[leveloffset=+3] include::./pages/knn-search.asciidoc[leveloffset=+3] include::./pages/search-your-data-semantic-search.asciidoc[leveloffset=+3] diff --git a/serverless/pages/knn-search.asciidoc b/serverless/pages/knn-search.asciidoc index 0db335bc..b3d723fb 100644 --- a/serverless/pages/knn-search.asciidoc +++ b/serverless/pages/knn-search.asciidoc @@ -13,1081 +13,9 @@ Common use cases for kNN include: * Product recommendations and recommendation engines * Similarity search for images or videos -[discrete] -[[elasticsearch-knn-search-prerequisites]] -== Prerequisites - -* To run a kNN search, you must be able to convert your data into meaningful -vector values. You can -{ml-docs}/ml-nlp-text-emb-vector-search-example.html[create these vectors using -a natural language processing (NLP) model in {es}], or generate them outside -{es}. Vectors can be added to documents as {ref}/dense-vector.html[`dense_vector`] field -values. Queries are represented as vectors with the same dimension. -+ -Design your vectors so that the closer a document's vector is to a query vector, -based on a similarity metric, the better its match. -* To complete the steps in this guide, you must have the following -{ref}/security-privileges.html#privileges-list-indices[index privileges]: -+ -** `create_index` or `manage` to create an index with a `dense_vector` field -** `create`, `index`, or `write` to add data to the index you created -** `read` to search the index - -[discrete] -[[elasticsearch-knn-search-knn-methods]] -== kNN methods - -{es} supports two methods for kNN search: - -* Approximate kNN using the `knn` search -option -* Exact, brute-force kNN using a `script_score` query with a -vector function - -In most cases, you'll want to use approximate kNN. Approximate kNN offers lower -latency at the cost of slower indexing and imperfect accuracy. - -Exact, brute-force kNN guarantees accurate results but doesn't scale well with -large datasets. With this approach, a `script_score` query must scan each -matching document to compute the vector function, which can result in slow -search speeds. However, you can improve latency by using a {ref}/query-dsl.html[query] -to limit the number of matching documents passed to the function. If you -filter your data to a small subset of documents, you can get good search -performance using this approach. - -[discrete] -[[elasticsearch-knn-search-approximate-knn]] -== Approximate kNN - -To run an approximate kNN search, use the {ref}/knn-search.html#approximate-knn[`knn` option] -to search one or more `dense_vector` fields with indexing enabled. - -. Explicitly map one or more `dense_vector` fields. Approximate kNN search -requires the following mapping options: -+ -** A `similarity` value. This value determines the similarity metric used to -score documents based on similarity between the query and document vector. For a -list of available metrics, see the {ref}/dense-vector.html#dense-vector-similarity[`similarity`] -parameter documentation. The `similarity` setting defaults to `cosine`. -+ -[source,bash] ----- -curl -X PUT "${ES_URL}/image-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "image-vector": { - "type": "dense_vector", - "dims": 3, - "similarity": "l2_norm" - }, - "title-vector": { - "type": "dense_vector", - "dims": 5, - "similarity": "l2_norm" - }, - "title": { - "type": "text" - }, - "file-type": { - "type": "keyword" - } - } - } -} -' ----- -. Index your data. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_bulk?refresh=true" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ "index": { "_id": "1" } } -{ "image-vector": [1, 5, -20], "title-vector": [12, 50, -10, 0, 1], "title": "moose family", "file-type": "jpg" } -{ "index": { "_id": "2" } } -{ "image-vector": [42, 8, -15], "title-vector": [25, 1, 4, -12, 2], "title": "alpine lake", "file-type": "png" } -{ "index": { "_id": "3" } } -{ "image-vector": [15, 11, 23], "title-vector": [1, 5, 25, 50, 20], "title": "full moon", "file-type": "jpg" } -... -' ----- -+ -// TEST[continued] -+ -// TEST[s/\.\.\.//] -. Run the search using the {ref}/knn-search.html#approximate-knn[`knn` option]. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "knn": { - "field": "image-vector", - "query_vector": [ - -5, - 9, - -12 - ], - "k": 10, - "num_candidates": 100 - }, - "fields": [ - "title", - "file-type" - ] -} -' ----- -+ -// TEST[continued] -+ -// TEST[s/"k": 10/"k": 3/] -+ -// TEST[s/"num_candidates": 100/"num_candidates": 3/] - -The {ref}/search-search.html#search-api-response-body-score[document `_score`] is determined by -the similarity between the query and document vector. See -{ref}/dense-vector.html#dense-vector-similarity[`similarity`] for more information on how kNN -search scores are computed. - -[discrete] -[[elasticsearch-knn-search-tune-approximate-knn-for-speed-or-accuracy]] -=== Tune approximate kNN for speed or accuracy - -To gather results, the kNN search API finds a `num_candidates` number of -approximate nearest neighbor candidates on each shard. The search computes the -similarity of these candidate vectors to the query vector, selecting the `k` -most similar results from each shard. The search then merges the results from -each shard to return the global top `k` nearest neighbors. - -You can increase `num_candidates` for more accurate results at the cost of -slower search speeds. A search with a high value for `num_candidates` -considers more candidates from each shard. This takes more time, but the -search has a higher probability of finding the true `k` top nearest neighbors. - -Similarly, you can decrease `num_candidates` for faster searches with -potentially less accurate results. - -[discrete] -[[approximate-knn-using-byte-vectors]] -=== Approximate kNN using byte vectors - -The approximate kNN search API supports `byte` value vectors in -addition to `float` value vectors. Use the {ref}/knn-search.html#approximate-knn[`knn` option] -to search a `dense_vector` field with {ref}/dense-vector.html#dense-vector-params[`element_type`] set to -`byte` and indexing enabled. - -. Explicitly map one or more `dense_vector` fields with -{ref}/dense-vector.html#dense-vector-params[`element_type`] set to `byte` and indexing enabled. -+ -[source,bash] ----- -curl -X PUT "${ES_URL}/byte-image-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "byte-image-vector": { - "type": "dense_vector", - "element_type": "byte", - "dims": 2 - }, - "title": { - "type": "text" - } - } - } -} -' ----- -+ -// TEST[continued] -. Index your data ensuring all vector values -are integers within the range [-128, 127]. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/byte-image-index/_bulk?refresh=true" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ "index": { "_id": "1" } } -{ "byte-image-vector": [5, -20], "title": "moose family" } -{ "index": { "_id": "2" } } -{ "byte-image-vector": [8, -15], "title": "alpine lake" } -{ "index": { "_id": "3" } } -{ "byte-image-vector": [11, 23], "title": "full moon" } -' ----- -+ -// TEST[continued] -. Run the search using the {ref}/knn-search.html#approximate-knn[`knn` option] -ensuring the `query_vector` values are integers within the -range [-128, 127]. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/byte-image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "knn": { - "field": "byte-image-vector", - "query_vector": [ - -5, - 9 - ], - "k": 10, - "num_candidates": 100 - }, - "fields": [ - "title" - ] -} -' ----- -+ -// TEST[continued] -+ -// TEST[s/"k": 10/"k": 3/] -+ -// TEST[s/"num_candidates": 100/"num_candidates": 3/] - -[discrete] -[[elasticsearch-knn-search-filtered-knn-search]] -=== Filtered kNN search - -The kNN search API supports restricting the search using a filter. The search -will return the top `k` documents that also match the filter query. - -The following request performs an approximate kNN search filtered by the -`file-type` field: - -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "knn": { - "field": "image-vector", - "query_vector": [54, 10, -2], - "k": 5, - "num_candidates": 50, - "filter": { - "term": { - "file-type": "png" - } - } - }, - "fields": ["title"], - "_source": false -} -' ----- - -// TEST[continued] - -[NOTE] -==== -The filter is applied **during** the approximate kNN search to ensure -that `k` matching documents are returned. This contrasts with a -post-filtering approach, where the filter is applied **after** the approximate -kNN search completes. Post-filtering has the downside that it sometimes -returns fewer than k results, even when there are enough matching documents. -==== - -[discrete] -[[elasticsearch-knn-search-combine-approximate-knn-with-other-features]] -=== Combine approximate kNN with other features - -You can perform 'hybrid retrieval' by providing both the -{ref}/knn-search.html#approximate-knn[`knn` option] and a {ref}/search-search.html#request-body-search-query[`query`]: - -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "query": { - "match": { - "title": { - "query": "mountain lake", - "boost": 0.9 - } - } - }, - "knn": { - "field": "image-vector", - "query_vector": [54, 10, -2], - "k": 5, - "num_candidates": 50, - "boost": 0.1 - }, - "size": 10 -} -' ----- - -// TEST[continued] - -This search finds the global top `k = 5` vector matches, combines them with the matches from the `match` query, and -finally returns the 10 top-scoring results. The `knn` and `query` matches are combined through a disjunction, as if you -took a boolean 'or' between them. The top `k` vector results represent the global nearest neighbors across all index -shards. - -The score of each hit is the sum of the `knn` and `query` scores. You can specify a `boost` value to give a weight to -each score in the sum. In the example above, the scores will be calculated as - -[source,txt] ----- -score = 0.9 * match_score + 0.1 * knn_score ----- - -The `knn` option can also be used with {ref}/search-aggregations.html[aggregations]. -In general, {es} computes aggregations over all documents that match the search. -So for approximate kNN search, aggregations are calculated on the top `k` -nearest documents. If the search also includes a `query`, then aggregations are -calculated on the combined set of `knn` and `query` matches. - -[discrete] -[[elasticsearch-knn-search-perform-semantic-search]] -=== Perform semantic search - -kNN search enables you to perform semantic search by using a previously deployed -{ml-docs}/ml-nlp-search-compare.html#ml-nlp-text-embedding[text embedding model]. -Instead of literal matching on search terms, semantic search retrieves results -based on the intent and the contextual meaning of a search query. - -Under the hood, the text embedding NLP model generates a dense vector from the -input query string called `model_text` you provide. Then, it is searched -against an index containing dense vectors created with the same text embedding -{ml} model. The search results are semantically similar as learned by the model. - -[IMPORTANT] -==== -To perform semantic search: - -* you need an index that contains the dense vector representation of the input -data to search against, -* you must use the same text embedding model for search that you used to create -the dense vectors from the input data, -* the text embedding NLP model deployment must be started. -==== - -Reference the deployed text embedding model or the model deployment in the -`query_vector_builder` object and provide the search query as `model_text`: - -// NOTCONSOLE - -[source,js] ----- -(...) -{ - "knn": { - "field": "dense-vector-field", - "k": 10, - "num_candidates": 100, - "query_vector_builder": { - "text_embedding": { <1> - "model_id": "my-text-embedding-model", <2> - "model_text": "The opposite of blue" <3> - } - } - } -} -(...) ----- - -<1> The {nlp} task to perform. It must be `text_embedding`. - -<2> The ID of the text embedding model to use to generate the dense vectors from -the query string. Use the same model that generated the embeddings from the -input text in the index you search against. You can use the value of the -`deployment_id` instead in the `model_id` argument. - -<3> The query string from which the model generates the dense vector -representation. - -For more information on how to deploy a trained model and use it to create text -embeddings, refer to this -{ml-docs}/ml-nlp-text-emb-vector-search-example.html[end-to-end example]. - -[discrete] -[[elasticsearch-knn-search-search-multiple-knn-fields]] -=== Search multiple kNN fields - -In addition to 'hybrid retrieval', you can search more than one kNN vector field at a time: - -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "query": { - "match": { - "title": { - "query": "mountain lake", - "boost": 0.9 - } - } - }, - "knn": [ { - "field": "image-vector", - "query_vector": [54, 10, -2], - "k": 5, - "num_candidates": 50, - "boost": 0.1 - }, - { - "field": "title-vector", - "query_vector": [1, 20, -52, 23, 10], - "k": 10, - "num_candidates": 10, - "boost": 0.5 - }], - "size": 10 -} -' ----- - -// TEST[continued] - -This search finds the global top `k = 5` vector matches for `image-vector` and the global `k = 10` for the `title-vector`. -These top values are then combined with the matches from the `match` query and the top-10 documents are returned. -The multiple `knn` entries and the `query` matches are combined through a disjunction, -as if you took a boolean 'or' between them. The top `k` vector results represent the global nearest neighbors across -all index shards. - -The scoring for a doc with the above configured boosts would be: - -[source,txt] ----- -score = 0.9 * match_score + 0.1 * knn_score_image-vector + 0.5 * knn_score_title-vector ----- - -[discrete] -[[elasticsearch-knn-search-search-knn-with-expected-similarity]] -=== Search kNN with expected similarity - -While kNN is a powerful tool, it always tries to return `k` nearest neighbors. Consequently, when using `knn` with -a `filter`, you could filter out all relevant documents and only have irrelevant ones left to search. In that situation, -`knn` will still do its best to return `k` nearest neighbors, even though those neighbors could be far away in the -vector space. - -To alleviate this worry, there is a `similarity` parameter available in the `knn` clause. This value is the required -minimum similarity for a vector to be considered a match. The `knn` search flow with this parameter is as follows: - -* Apply any user provided `filter` queries -* Explore the vector space to get `k` vectors -* Do not return any vectors that are further away than the configured `similarity` - -Here is an example. In this example we search for the given `query_vector` for `k` nearest neighbors. However, with -`filter` applied and requiring that the found vectors have at least the provided `similarity` between them. - -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "knn": { - "field": "image-vector", - "query_vector": [1, 5, -20], - "k": 5, - "num_candidates": 50, - "similarity": 36, - "filter": { - "term": { - "file-type": "png" - } - } - }, - "fields": ["title"], - "_source": false -} -' ----- - -// TEST[continued] - -In our data set, the only document with the file type of `png` has a vector of `[42, 8, -15]`. The `l2_norm` distance -between `[42, 8, -15]` and `[1, 5, -20]` is `41.412`, which is greater than the configured similarity of `36`. Meaning, -this search will return no hits. - -[discrete] -[[nested-knn-search]] -=== Nested kNN Search - -It is common for text to exceed a particular model's token limit and requires chunking before building the embeddings -for individual chunks. When using {ref}/nested.html[`nested`] with {ref}/dense-vector.html[`dense_vector`], you can achieve nearest -passage retrieval without copying top-level document metadata. - -Here is a simple passage vectors index that stores vectors and some top-level metadata for filtering. - -[source,bash] ----- -curl -X PUT "${ES_URL}/passage_vectors" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "full_text": { - "type": "text" - }, - "creation_time": { - "type": "date" - }, - "paragraph": { - "type": "nested", - "properties": { - "vector": { - "type": "dense_vector", - "dims": 2 - }, - "text": { - "type": "text", - "index": false - } - } - } - } - } -} -' ----- - -// TEST[continued] - -With the above mapping, we can index multiple passage vectors along with storing the individual passage text. - -[source,bash] ----- -curl -X POST "${ES_URL}/passage_vectors/_bulk?refresh=true" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ "index": { "_id": "1" } } -{ "full_text": "first paragraph another paragraph", "creation_time": "2019-05-04", "paragraph": [ { "vector": [ 0.45, 45 ], "text": "first paragraph", "paragraph_id": "1" }, { "vector": [ 0.8, 0.6 ], "text": "another paragraph", "paragraph_id": "2" } ] } -{ "index": { "_id": "2" } } -{ "full_text": "number one paragraph number two paragraph", "creation_time": "2020-05-04", "paragraph": [ { "vector": [ 1.2, 4.5 ], "text": "number one paragraph", "paragraph_id": "1" }, { "vector": [ -1, 42 ], "text": "number two paragraph", "paragraph_id": "2" } ] } -' ----- - -// TEST[continued] - -// TEST[s/\.\.\.//] - -The query will seem very similar to a typical kNN search: - -[source,bash] ----- -curl -X POST "${ES_URL}/passage_vectors/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "fields": ["full_text", "creation_time"], - "_source": false, - "knn": { - "query_vector": [ - 0.45, - 45 - ], - "field": "paragraph.vector", - "k": 2, - "num_candidates": 2 - } -} -' ----- - -// TEST[continued] - -Note below that even though we have 4 total vectors, we still return two documents. kNN search over nested dense_vectors -will always diversify the top results over the top-level document. Meaning, `"k"` top-level documents will be returned, -scored by their nearest passage vector (e.g. `"paragraph.vector"`). - -[source,console-result] ----- -{ - "took": 4, - "timed_out": false, - "_shards": { - "total": 1, - "successful": 1, - "skipped": 0, - "failed": 0 - }, - "hits": { - "total": { - "value": 2, - "relation": "eq" - }, - "max_score": 1.0, - "hits": [ - { - "_index": "passage_vectors", - "_id": "1", - "_score": 1.0, - "fields": { - "creation_time": [ - "2019-05-04T00:00:00.000Z" - ], - "full_text": [ - "first paragraph another paragraph" - ] - } - }, - { - "_index": "passage_vectors", - "_id": "2", - "_score": 0.9997144, - "fields": { - "creation_time": [ - "2020-05-04T00:00:00.000Z" - ], - "full_text": [ - "number one paragraph number two paragraph" - ] - } - } - ] - } -} ----- - -// TESTRESPONSE[s/"took": 4/"took" : "$body.took"/] - -What if you wanted to filter by some top-level document metadata? You can do this by adding `filter` to your -`knn` clause. - -[NOTE] -==== -`filter` will always be over the top-level document metadata. This means you cannot filter based on `nested` -field metadata. -==== - -[source,bash] ----- -curl -X POST "${ES_URL}/passage_vectors/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "fields": [ - "creation_time", - "full_text" - ], - "_source": false, - "knn": { - "query_vector": [ - 0.45, - 45 - ], - "field": "paragraph.vector", - "k": 2, - "num_candidates": 2, - "filter": { - "bool": { - "filter": [ - { - "range": { - "creation_time": { - "gte": "2019-05-01", - "lte": "2019-05-05" - } - } - } - ] - } - } - } -} -' ----- - -// TEST[continued] - -Now we have filtered based on the top level `"creation_time"` and only one document falls within that range. - -[source,console-result] ----- -{ - "took": 4, - "timed_out": false, - "_shards": { - "total": 1, - "successful": 1, - "skipped": 0, - "failed": 0 - }, - "hits": { - "total": { - "value": 1, - "relation": "eq" - }, - "max_score": 1.0, - "hits": [ - { - "_index": "passage_vectors", - "_id": "1", - "_score": 1.0, - "fields": { - "creation_time": [ - "2019-05-04T00:00:00.000Z" - ], - "full_text": [ - "first paragraph another paragraph" - ] - } - } - ] - } -} ----- - -// TESTRESPONSE[s/"took": 4/"took" : "$body.took"/] - -Additionally, if you wanted to extract the nearest passage for a matched document, you can supply {ref}/inner-hits.html[inner_hits] -to the `knn` clause. - -[NOTE] -==== -`inner_hits` for kNN will only ever return a single hit, the nearest passage vector. -Setting `"size"` to any value greater than `1` will have no effect on the results. -==== - -[source,bash] ----- -curl -X POST "${ES_URL}/passage_vectors/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "fields": [ - "creation_time", - "full_text" - ], - "_source": false, - "knn": { - "query_vector": [ - 0.45, - 45 - ], - "field": "paragraph.vector", - "k": 2, - "num_candidates": 2, - "inner_hits": { - "_source": false, - "fields": [ - "paragraph.text" - ] - } - } -} -' ----- - -// TEST[continued] - -Now the result will contain the nearest found paragraph when searching. - -[source,console-result] ----- -{ - "took": 4, - "timed_out": false, - "_shards": { - "total": 1, - "successful": 1, - "skipped": 0, - "failed": 0 - }, - "hits": { - "total": { - "value": 2, - "relation": "eq" - }, - "max_score": 1.0, - "hits": [ - { - "_index": "passage_vectors", - "_id": "1", - "_score": 1.0, - "fields": { - "creation_time": [ - "2019-05-04T00:00:00.000Z" - ], - "full_text": [ - "first paragraph another paragraph" - ] - }, - "inner_hits": { - "paragraph": { - "hits": { - "total": { - "value": 1, - "relation": "eq" - }, - "max_score": 1.0, - "hits": [ - { - "_index": "passage_vectors", - "_id": "1", - "_nested": { - "field": "paragraph", - "offset": 0 - }, - "_score": 1.0, - "fields": { - "paragraph": [ - { - "text": [ - "first paragraph" - ] - } - ] - } - } - ] - } - } - } - }, - { - "_index": "passage_vectors", - "_id": "2", - "_score": 0.9997144, - "fields": { - "creation_time": [ - "2020-05-04T00:00:00.000Z" - ], - "full_text": [ - "number one paragraph number two paragraph" - ] - }, - "inner_hits": { - "paragraph": { - "hits": { - "total": { - "value": 1, - "relation": "eq" - }, - "max_score": 0.9997144, - "hits": [ - { - "_index": "passage_vectors", - "_id": "2", - "_nested": { - "field": "paragraph", - "offset": 1 - }, - "_score": 0.9997144, - "fields": { - "paragraph": [ - { - "text": [ - "number two paragraph" - ] - } - ] - } - } - ] - } - } - } - } - ] - } -} ----- - -// TESTRESPONSE[s/"took": 4/"took" : "$body.took"/] - -[discrete] -[[knn-indexing-considerations]] -=== Indexing considerations - -For approximate kNN search, {es} stores the dense vector values of each -segment as an https://arxiv.org/abs/1603.09320[HNSW graph]. Indexing vectors for -approximate kNN search can take substantial time because of how expensive it is -to build these graphs. You may need to increase the client request timeout for -index and bulk requests. The {ref}/tune-knn-search.html[approximate kNN tuning guide] -contains important guidance around indexing performance, and how the index -configuration can affect search performance. - -In addition to its search-time tuning parameters, the HNSW algorithm has -index-time parameters that trade off between the cost of building the graph, -search speed, and accuracy. When setting up the `dense_vector` mapping, you -can use the {ref}/dense-vector.html#dense-vector-index-options[`index_options`] argument to adjust -these parameters: - -[source,bash] ----- -curl -X PUT "${ES_URL}/image-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "image-vector": { - "type": "dense_vector", - "dims": 3, - "index": true, - "similarity": "l2_norm", - "index_options": { - "type": "hnsw", - "m": 32, - "ef_construction": 100 - } - } - } - } -} -' ----- - -[discrete] -[[elasticsearch-knn-search-limitations-for-approximate-knn-search]] -=== Limitations for approximate kNN search - -{es} uses the https://arxiv.org/abs/1603.09320[HNSW algorithm] to support -efficient kNN search. Like most kNN algorithms, HNSW is an approximate method -that sacrifices result accuracy for improved search speed. This means the -results returned are not always the true _k_ closest neighbors. - -[NOTE] -==== -Approximate kNN search always uses the -{ref}/search-search.html#dfs-query-then-fetch[`dfs_query_then_fetch`] search type in order to gather -the global top `k` matches across shards. You cannot set the -`search_type` explicitly when running kNN search. -==== - -[discrete] -[[exact-knn]] -== Exact kNN - -To run an exact kNN search, use a `script_score` query with a vector function. - -. Explicitly map one or more `dense_vector` fields. If you don't intend to use -the field for approximate kNN, set the `index` mapping option to `false`. -This can significantly improve indexing speed. -+ -[source,bash] ----- -curl -X PUT "${ES_URL}/product-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "product-vector": { - "type": "dense_vector", - "dims": 5, - "index": false - }, - "price": { - "type": "long" - } - } - } -} -' ----- -. Index your data. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/product-index/_bulk?refresh=true" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ "index": { "_id": "1" } } -{ "product-vector": [230.0, 300.33, -34.8988, 15.555, -200.0], "price": 1599 } -{ "index": { "_id": "2" } } -{ "product-vector": [-0.5, 100.0, -13.0, 14.8, -156.0], "price": 799 } -{ "index": { "_id": "3" } } -{ "product-vector": [0.5, 111.3, -13.0, 14.8, -156.0], "price": 1099 } -... -' ----- -+ -// TEST[continued] -+ -// TEST[s/\.\.\.//] -. Use the search API to run a `script_score` query containing -a {ref}/query-dsl-script-score-query.html#vector-functions[vector function]. +Learn more in the {ref}/knn-search.html[{es} core documentation]. [TIP] ==== -To limit the number of matched documents passed to the vector function, we -recommend you specify a filter query in the `script_score.query` parameter. If -needed, you can use a {ref}/query-dsl-match-all-query.html[`match_all` query] in this -parameter to match all documents. However, matching all documents can -significantly increase search latency. - -[source,bash] ----- -curl -X POST "${ES_URL}/product-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "query": { - "script_score": { - "query": { - "bool": { - "filter": { - "range": { - "price": { - "gte": 1000 - } - } - } - } - }, - "script": { - "source": "cosineSimilarity(params.queryVector, 'product-vector') + 1.0", - "params": { - "queryVector": [ - -0.5, - 90, - -10, - 14.8, - -156 - ] - } - } - } - } -} -' ----- - -// TEST[continued] -==== +Check out our {ref}/bring-your-own-vectors.html[hands-on tutorial] to learn how to ingest dense vector embeddings into Elasticsearch. +==== \ No newline at end of file diff --git a/serverless/pages/search-with-synonyms.asciidoc b/serverless/pages/search-with-synonyms.asciidoc deleted file mode 100644 index e1075134..00000000 --- a/serverless/pages/search-with-synonyms.asciidoc +++ /dev/null @@ -1,120 +0,0 @@ -[[elasticsearch-reference-search-with-synonyms]] -= Full-text search with synonyms - -// :description: Use synonyms to search for words or phrases that have the same or similar meaning. -// :keywords: serverless, elasticsearch, search, synonyms - -Synonyms are words or phrases that have the same or similar meaning. -They are an important aspect of search, as they can improve the search experience and increase the scope of search results. - -Synonyms allow you to: - -* **Improve search relevance** by finding relevant documents that use different terms to express the same concept. -* Make **domain-specific vocabulary** more user-friendly, allowing users to use search terms they are more familiar with. -* **Define common misspellings and typos** to transparently handle common mistakes. - -Synonyms are grouped together using **synonyms sets**. -You can have as many synonyms sets as you need. - -In order to use synonyms sets in {es}, you need to: - -* Store your synonyms set -* Configure synonyms token filters and analyzers - -[discrete] -[[elasticsearch-reference-search-with-synonyms-store-your-synonyms-set]] -== Store your synonyms set - -Your synonyms sets need to be stored in {es} so your analyzers can refer to them. -There are two ways to store your synonyms sets: - -[discrete] -[[elasticsearch-reference-search-with-synonyms-synonyms-api]] -=== Synonyms API - -You can use the {ref}/synonyms-apis.html[synonyms APIs] to manage synonyms sets. -This is the most flexible approach, as it allows to dynamically define and modify synonyms sets. - -Changes in your synonyms sets will automatically reload the associated analyzers. - -[discrete] -[[elasticsearch-reference-search-with-synonyms-inline]] -=== Inline - -You can test your synonyms by adding them directly inline in your token filter definition. - -[WARNING] -==== -Inline synonyms are not recommended for production usage. -A large number of inline synonyms increases cluster size unnecessarily and can lead to performance issues. -==== - -[discrete] -[[synonyms-synonym-token-filters]] -=== Configure synonyms token filters and analyzers - -Once your synonyms sets are created, you can start configuring your token filters and analyzers to use them. - -{es} uses synonyms as part of the {ref}/analysis-overview.html[analysis process]. -You can use two types of {ref}/analysis-tokenfilters.html[token filter]: - -* {ref}/analysis-synonym-graph-tokenfilter.html[Synonym graph token filter]: It is recommended to use it, as it can correctly handle multi-word synonyms ("hurriedly", "in a hurry"). -* {ref}/analysis-synonym-tokenfilter.html[Synonym token filter]: Not recommended if you need to use multi-word synonyms. - -Check each synonym token filter documentation for configuration details and instructions on adding it to an analyzer. - -[discrete] -[[elasticsearch-reference-search-with-synonyms-test-your-analyzer]] -=== Test your analyzer - -You can test an analyzer configuration without modifying your index settings. -Use the {ref}/indices-analyze.html[analyze API] to test your analyzer chain: - -[source,bash] ----- -curl "${ES_URL}/my-index/_analyze?pretty" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d' -{ - "tokenizer": "standard", - "filter" : [ - "lowercase", - { - "type": "synonym_graph", - "synonyms": ["pc => personal computer", "computer, pc, laptop"] - } - ], - "text" : "Check how PC synonyms work" -} ----- - -[discrete] -[[elasticsearch-reference-search-with-synonyms-apply-synonyms-at-index-or-search-time]] -=== Apply synonyms at index or search time - -Analyzers can be applied at {ref}/analysis-index-search-time.html[index time or search time]. - -You need to decide when to apply your synonyms: - -* Index time: Synonyms are applied when the documents are indexed into {es}. This is a less flexible alternative, as changes to your synonyms require {ref}/docs-reindex.html[reindexing]. -* Search time: Synonyms are applied when a search is executed. This is a more flexible approach, which doesn't require reindexing. If token filters are configured with `"updateable": true`, search analyzers can be {ref}/indices-reload-analyzers.html[reloaded] when you make changes to your synonyms. - -Synonyms sets created using the synonyms API can only be used at search time. - -You can specify the analyzer that contains your synonyms set as a {ref}/specify-analyzer.html#specify-search-analyzer[search time analyzer] or as an {ref}/specify-analyzer.html#specify-index-time-analyzer[index time analyzer]. - -The following example adds `my_analyzer` as a search analyzer to the `title` field in an index mapping: - -[source,JSON] ----- - "mappings": { - "properties": { - "title": { - "type": "text", - "search_analyzer": "my_analyzer", - "updateable": true - } - } - } ----- diff --git a/serverless/pages/search-your-data-the-search-api.asciidoc b/serverless/pages/search-your-data-the-search-api.asciidoc index dcf65a95..f0a0f29c 100644 --- a/serverless/pages/search-your-data-the-search-api.asciidoc +++ b/serverless/pages/search-your-data-the-search-api.asciidoc @@ -14,7 +14,24 @@ a specific number of results. You can use the https://www.elastic.co/docs/api/doc/elasticsearch-serverless/group/endpoint-search[search API] to search and aggregate data stored in {es} data streams or indices. -The API's `query` request body parameter accepts queries written in -{ref}/query-dsl.html[Query DSL]. -For more information, refer to {ref}/search-your-data.html[the search API overview] in the classic {es} docs. +For more information, refer to {ref}/search-your-data.html[the search API overview] in the core {es} docs. + +[discrete] +[[elasticsearch-search-your-data-the-query-dsl]] +== Query DSL + +{ref}/query-dsl.html[Query DSL] is full-featured JSON-style query language that enables complex searching, filtering, and aggregations. + +The `_search` API's `query` request body parameter accepts queries written in +Query DSL. + +[discrete] +[[elasticsearch-search-your-data-the-retrievers]] +== Retrievers + +Retrievers are an alternative to Query DSL that allow you to configure complex retrieval pipelines using a simplified syntax. +Retrievers simplify the user experience by allowing entire retrieval pipelines to be configured in a single `_search` API call. + +Learn more in the {ref}/retrievers-overview.html[Retrievers overview] in the core {es} docs. + diff --git a/serverless/pages/search-your-data.asciidoc b/serverless/pages/search-your-data.asciidoc index 196e5b1c..194e88c4 100644 --- a/serverless/pages/search-your-data.asciidoc +++ b/serverless/pages/search-your-data.asciidoc @@ -1,26 +1,33 @@ [[elasticsearch-search-your-data]] = Search your data -// :description: Use the search API to run queries on your data. +// :description: Learn options for searching your data in Elasticsearch Serverless. // :keywords: serverless, elasticsearch, search -A search query, or query, is a request for information about data in {es} data streams or indices. +Searching your data in {es-serverless} works the same way as in other Elasticsearch deployments. +If you haven't used {es} before, you can learn the basics in the {ref}/elasticsearch-intro.html[core {es} documentation]. -You can think of a query as a question, written in a way {es} understands. Depending on your data, you can use a query to get answers to questions like: +You can use the https://www.elastic.co/docs/api/doc/elasticsearch-serverless[{es-serverless} REST APIs] to search your data using any HTTP client, including the <>, or directly in <>. -* What processes on my server take longer than 500 milliseconds to respond? -* What users on my network ran regsvr32.exe within the last week? -* What pages on my website contain a specific word or phrase? +You can also run searches using {kibana-ref}/discover.html[Discover] in your project's UI. -You run search queries using the <>. The API supports several query types and search methods: +[TIP] +==== +Try our hands-on {ref}/quickstart.html#quickstart-list[quick start tutorials] in the core {es} documentation to get started, or check out our https://github.com/elastic/elasticsearch-labs/tree/main/notebooks#readme[Python notebooks]. +==== -**Search for exact values.** -Use {ref}/term-level-queries.html[term-level queries] to filter numbers, dates, IPs, or strings based on exact values or ranges. +[discrete] +[[elasticsearch-search-your-data-query-languages-overview]] +== Query languages -**Full-text search.** -Use {ref}/full-text-queries.html[full-text queries] to query {ref}/analysis.html#analysis[unstructured text] and find documents that best match query terms. Use <> to search for words or phrases that have the same or similar meaning. +Learn about the various query languages you can use to search your data in the {ref}/search-analyze.html[core {es} documentation]. -**Vector search.** -Store dense vectors in {es} and use <> to find similar vectors. +[discrete] +[[elasticsearch-search-your-data-learn-more]] +== Learn more -You can also use Elastic's natural language processing (NLP) model to encode text as sparse or dense vectors. Then use <> to find data based on the intent and contextual meaning rather than matching keywords. +The subpages in this section provide some high-level information about selected important search-related topics: + +* <> +* <> +* <> \ No newline at end of file From e37db79fd920be7c411ca657d55a7077fd2e39b6 Mon Sep 17 00:00:00 2001 From: kosabogi <105062005+kosabogi@users.noreply.github.com> Date: Tue, 26 Nov 2024 16:50:39 +0100 Subject: [PATCH 2/7] Adds notes to Connector clients and Connectors page (#215) --- serverless/pages/action-connectors.asciidoc | 5 +++++ ...ngest-data-through-integrations-connector-client.asciidoc | 5 +++++ 2 files changed, 10 insertions(+) diff --git a/serverless/pages/action-connectors.asciidoc b/serverless/pages/action-connectors.asciidoc index b9210157..8d523714 100644 --- a/serverless/pages/action-connectors.asciidoc +++ b/serverless/pages/action-connectors.asciidoc @@ -4,6 +4,11 @@ // :description: Configure connections to third party systems for use in cases and rules. // :keywords: serverless +[NOTE] +==== +This page is about Kibana connectors that integrate with services like generative AI model providers. If you're looking for Search connectors that synchronize third-party data into {es}, refer to <>. +==== + This content applies to: {es-badge} {obs-badge} {sec-badge} The list of available connectors varies by project type. diff --git a/serverless/pages/ingest-your-data-ingest-data-through-integrations-connector-client.asciidoc b/serverless/pages/ingest-your-data-ingest-data-through-integrations-connector-client.asciidoc index ceb0700f..a9f8a6e7 100644 --- a/serverless/pages/ingest-your-data-ingest-data-through-integrations-connector-client.asciidoc +++ b/serverless/pages/ingest-your-data-ingest-data-through-integrations-connector-client.asciidoc @@ -4,6 +4,11 @@ // :description: Set up and deploy self-managed connectors that run on your own infrastructure. // :keywords: serverless, elasticsearch, ingest, connector, how to +[NOTE] +==== +This page is about Search connectors that synchronize third-party data into {es}. If you're looking for Kibana connectors to integrate with services like generative AI model providers, refer to <>. +==== + [NOTE] ==== This page contains high-level instructions about setting up connector clients in your project's UI. From cf0ee39bc3d97fbc2e7be7307500b1b304af2e69 Mon Sep 17 00:00:00 2001 From: Janeen Mikell Roberts <57149392+jmikell821@users.noreply.github.com> Date: Tue, 26 Nov 2024 22:41:11 -0500 Subject: [PATCH 3/7] [Serverless] Fixes serverless references (#219) --- serverless/index-serverless-general.asciidoc | 2 +- serverless/index.asciidoc | 2 -- serverless/pages/manage-billing-stop-project.asciidoc | 2 +- serverless/pages/ml-nlp-auto-scale.asciidoc | 6 +++--- serverless/pages/service-status.asciidoc | 4 ++-- serverless/pages/welcome-to-serverless.asciidoc | 6 +++--- serverless/pages/what-is-elasticsearch-serverless.asciidoc | 2 +- 7 files changed, 11 insertions(+), 13 deletions(-) diff --git a/serverless/index-serverless-general.asciidoc b/serverless/index-serverless-general.asciidoc index 31bc65c2..210b3d6b 100644 --- a/serverless/index-serverless-general.asciidoc +++ b/serverless/index-serverless-general.asciidoc @@ -1,5 +1,5 @@ [[intro]] -== Welcome to Elastic serverless +== Welcome to {serverless-full} include::{docs-content-root}/serverless/pages/welcome-to-serverless.asciidoc[leveloffset=+2] diff --git a/serverless/index.asciidoc b/serverless/index.asciidoc index a5204d41..b53707a6 100644 --- a/serverless/index.asciidoc +++ b/serverless/index.asciidoc @@ -11,8 +11,6 @@ include::{asciidoc-dir}/../../shared/attributes.asciidoc[] :sec-badge: <> // The values of these attributes are different in stateful vs serverless -:observability: Elastic Observability Serverless -:security: Elastic Security Serverless = Serverless diff --git a/serverless/pages/manage-billing-stop-project.asciidoc b/serverless/pages/manage-billing-stop-project.asciidoc index 4fad3876..50ada38f 100644 --- a/serverless/pages/manage-billing-stop-project.asciidoc +++ b/serverless/pages/manage-billing-stop-project.asciidoc @@ -6,7 +6,7 @@ Got a project you no longer need and don't want to be charged for? Simply delete it. -Warning: All data is lost. Billing for usage is by the hour and any outstanding charges for usage before you deleted the project will still appear on your next bill. +WARNING: All data is lost. Billing for usage is by the hour and any outstanding charges for usage before you deleted the project will still appear on your next bill. To stop being charged for a project: diff --git a/serverless/pages/ml-nlp-auto-scale.asciidoc b/serverless/pages/ml-nlp-auto-scale.asciidoc index 71c81b1c..56f423f1 100644 --- a/serverless/pages/ml-nlp-auto-scale.asciidoc +++ b/serverless/pages/ml-nlp-auto-scale.asciidoc @@ -63,7 +63,7 @@ Increasing the number of threads will make the search processes more performant. [discrete] [[enabling-autoscaling-in-kibana-adaptive-resources]] -== Enabling autoscaling in Kibana - adaptive resources +== Enabling autoscaling in {kib} - adaptive resources You can enable adaptive resources for your models when starting or updating the model deployment. Adaptive resources make it possible for {es} to scale up or down the available resources based on the load on the process. @@ -74,7 +74,7 @@ When the load is low, the number of VCUs that the process can use is automatical You can choose from three levels of resource usage for your trained model deployment; autoscaling will occur within the selected level's range. -Refer to the tables in the auto-scaling-matrix section to find out the setings for the level you selected. +Refer to the tables in the auto-scaling-matrix section to find out the settings for the level you selected. image::images/ml-nlp-deployment.png[ML model deployment with adaptive resources enabled.] @@ -154,7 +154,7 @@ No static allocations for Security and Observability [discrete] [[deployments-on-serverless-optimized-for-search]] -=== Deployments on serverless optimized for search +=== Deployments on serverless optimized for Search [discrete] [[adaptive-resources-enabled-for-search]] diff --git a/serverless/pages/service-status.asciidoc b/serverless/pages/service-status.asciidoc index 0e1610a8..650c4df9 100644 --- a/serverless/pages/service-status.asciidoc +++ b/serverless/pages/service-status.asciidoc @@ -6,7 +6,7 @@ Serverless projects run on cloud platforms, which may undergo changes in availability. When availability changes, Elastic makes sure to provide you with a current service status. -To check current and past service availability, go to the Elastic serverless https://serverless-preview-status.statuspage.io/[service status] page. +To check current and past service availability, go to the {serverless-full} https://serverless-preview-status.statuspage.io/[service status] page. [discrete] [[general-serverless-status-subscribe-to-updates]] @@ -16,7 +16,7 @@ You can be notified about changes to the service status automatically. To receive service status updates: -. Go to the Elastic serverless https://serverless-preview-status.statuspage.io/[service status] page. +. Go to the {serverless-full} https://serverless-preview-status.statuspage.io/[service status] page. . Select **SUBSCRIBE TO UPDATES**. . You can be notified in the following ways: + diff --git a/serverless/pages/welcome-to-serverless.asciidoc b/serverless/pages/welcome-to-serverless.asciidoc index b78f1b32..a61f7641 100644 --- a/serverless/pages/welcome-to-serverless.asciidoc +++ b/serverless/pages/welcome-to-serverless.asciidoc @@ -10,15 +10,15 @@ ++++ -Elastic serverless products allow you to deploy and use Elastic for your use cases without managing the underlying Elastic cluster, +{serverless-full} products allow you to deploy and use Elastic for your use cases without managing the underlying Elastic cluster, such as nodes, data tiers, and scaling. Serverless instances are fully-managed, autoscaled, and automatically upgraded by Elastic so you can focus more on gaining value and insight from your data. Elastic provides three serverless solutions available on {ecloud}: * **{es-serverless}**: Build powerful applications and search experiences using a rich ecosystem of vector search capabilities, APIs, and libraries. -* **{observability}**: Monitor your own platforms and services using powerful machine learning and analytics tools with your logs, metrics, traces, and APM data. -* **{elastic-sec}**: Detect, investigate, and respond to threats, with SIEM, endpoint protection, and AI-powered analytics capabilities. +* **{obs-serverless}**: Monitor your own platforms and services using powerful machine learning and analytics tools with your logs, metrics, traces, and APM data. +* **{sec-serverless}**: Detect, investigate, and respond to threats, with SIEM, endpoint protection, and AI-powered analytics capabilities. Serverless instances of the Elastic Stack that you create in {ecloud} are called **serverless projects**. diff --git a/serverless/pages/what-is-elasticsearch-serverless.asciidoc b/serverless/pages/what-is-elasticsearch-serverless.asciidoc index 262e585d..6ec4e7ac 100644 --- a/serverless/pages/what-is-elasticsearch-serverless.asciidoc +++ b/serverless/pages/what-is-elasticsearch-serverless.asciidoc @@ -12,7 +12,7 @@ If you haven't used {es} before, first learn the basics in the https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html[core {es} documentation]. ==== -{es} is one of the three available project types on <>. +{es-serverless} is one of the three available project types on <>. This project type enables you to use the core functionality of {es}: searching, indexing, storing, and analyzing data of all shapes and sizes. From e95031a05c144eb4745372b237aa1a7e59cde7b8 Mon Sep 17 00:00:00 2001 From: Marci W <333176+marciw@users.noreply.github.com> Date: Wed, 27 Nov 2024 11:50:24 -0500 Subject: [PATCH 4/7] Add logsdb default mode (#218) --- serverless/pages/index-management.asciidoc | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/serverless/pages/index-management.asciidoc b/serverless/pages/index-management.asciidoc index 0d58b569..99a3fe00 100644 --- a/serverless/pages/index-management.asciidoc +++ b/serverless/pages/index-management.asciidoc @@ -80,6 +80,8 @@ on multiple indices, select their checkboxes and then open the **Manage** menu. Investigate your data streams and address lifecycle management needs in the **Data Streams** view. +In {es-serverless}, indices matching the `logs-*-*` pattern use the logsDB index mode by default. The logsDB index mode creates a {ref}/logs-data-stream.html[logs data stream]. + The value in the **Indices** column indicates the number of backing indices. Click this number to drill down into details. A value in the data retention column indicates that the data stream is managed by a data stream lifecycle policy. @@ -115,6 +117,8 @@ image::images/index-management-index-templates.png[Index templates] // TO-DO: This screenshot is missing some tabs that exist in serverless +The default *logs* template uses the logsDB index mode to create a {ref}/logs-data-stream.html[logs data stream]. + If you don't have any templates, you can create one using the **Create template** wizard. //// From a4772acefe52ec683b8bd44a7c5b666dd2daf5cc Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Thu, 28 Nov 2024 17:51:26 +0100 Subject: [PATCH 5/7] Clarify constraints on serverless trial resources (#220) --- serverless/pages/sign-up.asciidoc | 1 + 1 file changed, 1 insertion(+) diff --git a/serverless/pages/sign-up.asciidoc b/serverless/pages/sign-up.asciidoc index 32c8b7c7..9219c363 100644 --- a/serverless/pages/sign-up.asciidoc +++ b/serverless/pages/sign-up.asciidoc @@ -53,6 +53,7 @@ During the free 14 day trial, Elastic provides access to one hosted deployment a * You can have one active serverless project at a time. * Search Power is limited to 100. This setting only exists in {es-serverless} projects * Search Boost Window is limited to 7 days. This setting only exists in {es-serverless} projects +* Scaling is limited for serverless projects in trials. Failures might occur if the workload requires memory or compute beyond what the above search power and search boost window setting limits can provide. **How to remove restrictions?** From bc1ddcb4f457eff0c94ae325cbe8f974b8c0290e Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Fri, 29 Nov 2024 09:37:42 +0100 Subject: [PATCH 6/7] Add index sizing guidelines (#221) Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- .../pages/serverless-differences.asciidoc | 26 +++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/serverless/pages/serverless-differences.asciidoc b/serverless/pages/serverless-differences.asciidoc index 398edcee..8d3a67aa 100644 --- a/serverless/pages/serverless-differences.asciidoc +++ b/serverless/pages/serverless-differences.asciidoc @@ -26,6 +26,32 @@ This guide helps you understand what's different, what's available, and how to w This fully managed approach means many traditional {es} infrastructure APIs and settings are not available to end users, as detailed in the following sections. +[discrete] +[[elasticsearch-differences-serverless-index-size]] +== Index size guidelines + +To ensure optimal performance, follow these recommendations for sizing individual indices on {es-serverless}: + +|=== +|Use case |Maximum index size |Project configuration + +|Vector search +|150GB +|Vector optimized + +|General search (non data-stream) +|300GB +|General purpose + +|Other uses (non data-stream) +|600GB +|General purpose +|=== + +For large datasets that exceed the recommended maximum size for a single index, consider splitting your data across smaller indices and using an alias to search them collectively. + +These recommendations do not apply to indices using better binary quantization (BBQ). Refer to {ref}/dense-vector.html#dense-vector-quantization[vector quantization] in the core {es} docs for more information. + [discrete] [[elasticsearch-differences-serverless-apis-availability]] == API availability From a61c1bfcb1669640a5c0b48b273f85e236e7099b Mon Sep 17 00:00:00 2001 From: shainaraskas Date: Mon, 2 Dec 2024 10:43:54 -0500 Subject: [PATCH 7/7] update service status page links --- serverless/pages/service-status.asciidoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/serverless/pages/service-status.asciidoc b/serverless/pages/service-status.asciidoc index 650c4df9..bed4f343 100644 --- a/serverless/pages/service-status.asciidoc +++ b/serverless/pages/service-status.asciidoc @@ -6,7 +6,7 @@ Serverless projects run on cloud platforms, which may undergo changes in availability. When availability changes, Elastic makes sure to provide you with a current service status. -To check current and past service availability, go to the {serverless-full} https://serverless-preview-status.statuspage.io/[service status] page. +To check current and past service availability, go to the Elastic https://status.elastic.co/[service status] page. [discrete] [[general-serverless-status-subscribe-to-updates]] @@ -16,7 +16,7 @@ You can be notified about changes to the service status automatically. To receive service status updates: -. Go to the {serverless-full} https://serverless-preview-status.statuspage.io/[service status] page. +. Go to the Elastic https://status.elastic.co/[service status] page. . Select **SUBSCRIBE TO UPDATES**. . You can be notified in the following ways: +