diff --git a/serverless/images/discover-find-data-view.png b/serverless/images/discover-find-data-view.png index 869fc9b9..5f9b3107 100644 Binary files a/serverless/images/discover-find-data-view.png and b/serverless/images/discover-find-data-view.png differ diff --git a/serverless/images/semantic-options.svg b/serverless/images/semantic-options.svg new file mode 100644 index 00000000..3bedf530 --- /dev/null +++ b/serverless/images/semantic-options.svg @@ -0,0 +1,62 @@ + + + + Elasticsearch semantic search workflows + + + + + + semantic_text + (Recommended) + + + + Inference API + + + + Model Deployment + + + Complexity: Low + Complexity: Medium + Complexity: High + + + + + + Create Inference Endpoint + + + Define Index Mapping + + + + Create Inference Endpoint + + + Configure Model Settings + + + Define Index Mapping + + + Setup Ingest Pipeline + + + + Select NLP Model + + + Deploy with Eland Client + + + Define Index Mapping + + + Setup Ingest Pipeline + + + diff --git a/serverless/index-serverless-elasticsearch.asciidoc b/serverless/index-serverless-elasticsearch.asciidoc index ef38b36a..6edecc6e 100644 --- a/serverless/index-serverless-elasticsearch.asciidoc +++ b/serverless/index-serverless-elasticsearch.asciidoc @@ -5,6 +5,7 @@ ++++ include::./pages/what-is-elasticsearch-serverless.asciidoc[leveloffset=+2] +include::./pages/serverless-differences.asciidoc[leveloffset=+2] include::./pages/get-started.asciidoc[leveloffset=+2] @@ -37,13 +38,7 @@ include::./pages/search-your-data-the-search-api.asciidoc[leveloffset=+3] include::./pages/search-with-synonyms.asciidoc[leveloffset=+3] include::./pages/knn-search.asciidoc[leveloffset=+3] include::./pages/search-your-data-semantic-search.asciidoc[leveloffset=+3] -include::./pages/search-your-data-semantic-search-elser.asciidoc[leveloffset=+4] include::./pages/explore-your-data.asciidoc[leveloffset=+2] include::./pages/search-playground.asciidoc[leveloffset=+2] - -include::./pages/serverless-differences.asciidoc[leveloffset=+2] - -include::./pages/pricing.asciidoc[leveloffset=+2] -include::./pages/technical-preview-limitations.asciidoc[leveloffset=+2] diff --git a/serverless/pages/apis-http-apis.asciidoc b/serverless/pages/apis-http-apis.asciidoc index 87873d12..8de2a62a 100644 --- a/serverless/pages/apis-http-apis.asciidoc +++ b/serverless/pages/apis-http-apis.asciidoc @@ -4,6 +4,33 @@ // :description: {es} and {kib} expose REST APIs that can be called directly to configure and access {stack} features. // :keywords: serverless, elasticsearch, http, rest, overview -* <>: The {es-serverless} REST APIs have conventions for headers and request bodies. -* <>: The Management APIs for {serverless-short} have request header conventions. -* https://www.elastic.co/docs/api/[API Reference]: Explore the reference information for Elastic Serverless REST APIs +[discrete] +[[elasticsearch-api-references-links]] +== API references + +The following APIs are available for {es-serverless} users. +These links will take you to the autogenerated API reference documentation. + +https://www.elastic.co/docs/api/doc/elasticsearch-serverless[Elasticsearch Serverless APIs →]:: +Use these APIs to index, manage, search, and analyze your data in {es-serverless}. ++ +[TIP] +==== +Learn how to <>. +==== + +https://www.elastic.co/docs/api/doc/serverless[Kibana Serverless APIs →]:: +Use these APIs to manage resources such as connectors, data views, and saved objects for your {serverless-full} project. + +https://www.elastic.co/docs/api/doc/elastic-cloud-serverless[{serverless-full} APIs →]:: +Use these APIs to manage your {serverless-full} projects. + +[discrete] +[[additional-api-details]] +== Additional API information + +<>:: +Reference information about headers and request body conventions for {es-serverless} REST APIs. + +<>:: +Reference information about request header conventions for {serverless-full} REST APIs. diff --git a/serverless/pages/clients.asciidoc b/serverless/pages/clients.asciidoc index e3e42f6e..9590ed24 100644 --- a/serverless/pages/clients.asciidoc +++ b/serverless/pages/clients.asciidoc @@ -20,3 +20,8 @@ Currently, the following language clients are supported: * <> | https://github.com/elastic/elasticsearch-serverless-php[Repository] * <> | https://github.com/elastic/elasticsearch-serverless-python[Repository] * <> | https://github.com/elastic/elasticsearch-serverless-ruby[Repository] + +[TIP] +==== +Learn how to <>. +==== diff --git a/serverless/pages/connecting-to-es-endpoint.asciidoc b/serverless/pages/connecting-to-es-endpoint.asciidoc index 02069ee8..34746236 100644 --- a/serverless/pages/connecting-to-es-endpoint.asciidoc +++ b/serverless/pages/connecting-to-es-endpoint.asciidoc @@ -1,5 +1,8 @@ [[elasticsearch-connecting-to-es-serverless-endpoint]] -= Connecting to your Elasticsearch Serverless endpoint += Connect to your Elasticsearch Serverless endpoint +++++ +Connect to your endpoint +++++ [TIP] ==== diff --git a/serverless/pages/data-views.asciidoc b/serverless/pages/data-views.asciidoc index bfa31dce..3a0d6139 100644 --- a/serverless/pages/data-views.asciidoc +++ b/serverless/pages/data-views.asciidoc @@ -35,12 +35,11 @@ After you've loaded your data, follow these steps to create a {data-source}: // -. Open **Discover** then open the data view menu. +. Go to **{project-settings} → {manage-app} → {data-views-app}**. Alternatively, go to **Discover** and open the data view menu. + [role="screenshot"] image:images/discover-find-data-view.png[How to set the {data-source} in Discover] + -Alternatively, go to **{project-settings} → {manage-app} → {data-views-app}**. . Click **Create a {data-source}**. . Give your {data-source} a name. . Start typing in the **Index pattern** field, and Elastic looks for the names of diff --git a/serverless/pages/ingest-your-data.asciidoc b/serverless/pages/ingest-your-data.asciidoc index aa4f772f..067b3b92 100644 --- a/serverless/pages/ingest-your-data.asciidoc +++ b/serverless/pages/ingest-your-data.asciidoc @@ -4,27 +4,36 @@ // :description: Add data to your {es-serverless} project. // :keywords: serverless, elasticsearch, ingest, overview -You have many options for ingesting, or indexing, data into {es}: +The best ingest option(s) for your use case depends on whether you are indexing general content or time series (timestamped) data. -* <> -* <> -* <> -* <> -* <> -* https://github.com/elastic/crawler[Elastic Open Web Crawler] +[discrete] +[[es-ingestion-overview-apis]] +== Ingest data using APIs -The best ingest option(s) for your use case depends on whether you are indexing general content or time series (timestamped) data. +You can use the <> to add data to your {es} indices, using any HTTP client, including the <>. -**General content** +While the {es} APIs can be used for any data type, Elastic provides specialized tools that optimize ingestion for specific use cases. + +[discrete] +[[es-ingestion-overview-general-content]] +== Ingest general content -General content includes HTML pages, catalogs, files, and other content that does not update continuously. -This data can be updated, but the value of the content remains relatively constant over time. -Use connector clients to sync data from a range of popular data sources to {es}. -You can also send data directly to {es} from your application using the API. +General content is typically text-heavy data that does not have a timestamp. +This could be data like knowledge bases, website content, product catalogs, and more. + +You can use these specialized tools to add general content to {es} indices: + +* <> +* https://github.com/elastic/crawler[Elastic Open Web Crawler] +* <> [discrete] [[elasticsearch-ingest-time-series-data]] -**Times series (timestamped) data** +== Ingest time series data Time series, or timestamped data, describes data that changes frequently and "flows" over time, such as stock quotes, system metrics, and network traffic data. -Use {beats} or {ls} to collect time series data. + +You can use these specialized tools to add timestamped data to {es} data streams: + +* <> +* <> diff --git a/serverless/pages/search-your-data-semantic-search-elser.asciidoc b/serverless/pages/search-your-data-semantic-search-elser.asciidoc deleted file mode 100644 index 5e9d4907..00000000 --- a/serverless/pages/search-your-data-semantic-search-elser.asciidoc +++ /dev/null @@ -1,390 +0,0 @@ -[[elasticsearch-reference-semantic-search-elser]] -= Tutorial: Semantic search with ELSER - -// :description: Perform semantic search using ELSER, an NLP model trained by Elastic. -// :keywords: elasticsearch, elser, semantic search - -Elastic Learned Sparse EncodeR - or ELSER - is an NLP model trained by Elastic -that enables you to perform semantic search by using sparse vector -representation. Instead of literal matching on search terms, semantic search -retrieves results based on the intent and the contextual meaning of a search -query. - -The instructions in this tutorial shows you how to use ELSER to perform semantic -search on your data. - -[NOTE] -==== -Only the first 512 extracted tokens per field are considered during -semantic search with ELSER. Refer to -{ml-docs}/ml-nlp-limitations.html#ml-nlp-elser-v1-limit-512[this page] for more -information. -==== - -[discrete] -[[requirements]] -== Requirements - -To perform semantic search by using ELSER, you must have the NLP model deployed -in your cluster. Refer to the -{ml-docs}/ml-nlp-elser.html[ELSER documentation] to learn how to download and -deploy the model. - -[NOTE] -==== -The minimum dedicated ML node size for deploying and using the ELSER model -is 4 GB in {es} Service if -{cloud}/ec-autoscaling.html[deployment autoscaling] is turned off. Turning on -autoscaling is recommended because it allows your deployment to dynamically -adjust resources based on demand. Better performance can be achieved by using -more allocations or more threads per allocation, which requires bigger ML nodes. -Autoscaling provides bigger nodes when required. If autoscaling is turned off, -you must provide suitably sized nodes yourself. -==== - -[discrete] -[[elser-mappings]] -== Create the index mapping - -First, the mapping of the destination index - the index that contains the tokens -that the model created based on your text - must be created. The destination -index must have a field with the -{ref}/sparse-vector.html[`sparse_vector`] or {ref}/rank-features.html[`rank_features`] field -type to index the ELSER output. - -[NOTE] -==== -ELSER output must be ingested into a field with the `sparse_vector` or -`rank_features` field type. Otherwise, {es} interprets the token-weight pairs as -a massive amount of fields in a document. If you get an error similar to this -`"Limit of total fields [1000] has been exceeded while adding new fields"` then -the ELSER output field is not mapped properly and it has a field type different -than `sparse_vector` or `rank_features`. -==== - -[source,bash] ----- -curl -X PUT "${ES_URL}/my-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d' -{ - "mappings": { - "properties": { - "content_embedding": { <1> - "type": "sparse_vector" <2> - }, - "content": { <3> - "type": "text" <4> - } - } - } -} -' ----- - -<1> The name of the field to contain the generated tokens. It must be refrenced -in the {infer} pipeline configuration in the next step. - -<2> The field to contain the tokens is a `sparse_vector` field. - -<3> The name of the field from which to create the sparse vector representation. -In this example, the name of the field is `content`. It must be referenced in the -{infer} pipeline configuration in the next step. - -<4> The field type which is text in this example. - -To learn how to optimize space, refer to the <> section. - -[discrete] -[[inference-ingest-pipeline]] -== Create an ingest pipeline with an inference processor - -Create an {ref}/ingest.html[ingest pipeline] with an -{ref}/inference-processor.html[inference processor] to use ELSER to infer against the data -that is being ingested in the pipeline. - -[source,bash] ----- -curl -X PUT "${ES_URL}/_ingest/pipeline/elser-v2-test" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d' -{ - "processors": [ - { - "inference": { - "model_id": ".elser_model_2", - "input_output": [ <1> - { - "input_field": "content", - "output_field": "content_embedding" - } - ] - } - } - ] -} -' ----- - -[discrete] -[[load-data]] -== Load data - -In this step, you load the data that you later use in the {infer} ingest -pipeline to extract tokens from it. - -Use the `msmarco-passagetest2019-top1000` data set, which is a subset of the MS -MARCO Passage Ranking data set. It consists of 200 queries, each accompanied by -a list of relevant text passages. All unique passages, along with their IDs, -have been extracted from that data set and compiled into a -https://github.com/elastic/stack-docs/blob/main/docs/en/stack/ml/nlp/data/msmarco-passagetest2019-unique.tsv[tsv file]. - -Download the file and upload it to your cluster using the -{kibana-ref}/connect-to-elasticsearch.html#upload-data-kibana[Data Visualizer] -in the {ml-app} UI. Assign the name `id` to the first column and `content` to -the second column. The index name is `test-data`. Once the upload is complete, -you can see an index named `test-data` with 182469 documents. - -[discrete] -[[reindexing-data-elser]] -== Ingest the data through the {infer} ingest pipeline - -Create the tokens from the text by reindexing the data throught the {infer} -pipeline that uses ELSER as the inference model. - -[source,bash] ----- -curl -X POST "${ES_URL}/_reindex?wait_for_completion=false" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d' -{ - "source": { - "index": "test-data", - "size": 50 <1> - }, - "dest": { - "index": "my-index", - "pipeline": "elser-v2-test" - } -} -' ----- - -<1> The default batch size for reindexing is 1000. Reducing `size` to a smaller -number makes the update of the reindexing process quicker which enables you to -follow the progress closely and detect errors early. - -The call returns a task ID to monitor the progress: - -[source,bash] ----- -curl -X GET "${ES_URL}/_tasks/" \ --H "Authorization: ApiKey ${API_KEY}" \ ----- - -You can also open the Trained Models UI, select the Pipelines tab under ELSER to -follow the progress. - -[discrete] -[[sparse-vector-query]] -== Semantic search by using the `sparse_vector` query - -To perform semantic search, use the `sparse_vector` query, and provide the -query text and the inference ID associated with the ELSER model service. The example below uses the query text "How to -avoid muscle soreness after running?", the `content_embedding` field contains -the generated ELSER output: - -[source,bash] ----- -curl -X GET "${ES_URL}/my-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d' -{ - "query":{ - "sparse_vector":{ - "field": "content_embedding", - "inference_id": "my-elser-endpoint", - "query": "How to avoid muscle soreness after running?" - } - } -} -' ----- - -The result is the top 10 documents that are closest in meaning to your query -text from the `my-index` index sorted by their relevancy. The result also -contains the extracted tokens for each of the relevant search results with their -weights. - -[source,consol-result] ----- -"hits": { - "total": { - "value": 10000, - "relation": "gte" - }, - "max_score": 26.199875, - "hits": [ - { - "_index": "my-index", - "_id": "FPr9HYsBag9jXmT8lEpI", - "_score": 26.199875, - "_source": { - "content_embedding": { - "muscular": 0.2821541, - "bleeding": 0.37929374, - "foods": 1.1718726, - "delayed": 1.2112266, - "cure": 0.6848574, - "during": 0.5886185, - "fighting": 0.35022718, - "rid": 0.2752442, - "soon": 0.2967024, - "leg": 0.37649947, - "preparation": 0.32974035, - "advance": 0.09652356, - (...) - }, - "id": 1713868, - "model_id": ".elser_model_2", - "content": "For example, if you go for a run, you will mostly use the muscles in your lower body. Give yourself 2 days to rest those muscles so they have a chance to heal before you exercise them again. Not giving your muscles enough time to rest can cause muscle damage, rather than muscle development." - } - }, - (...) - ] -} ----- - -[discrete] -[[sparse-vector-compound-query]] -== Combining semantic search with other queries - -You can combine `sparse_vector` with other queries in a -{ref}/compound-queries.html[compound query]. For example using a filter clause in a -{ref}/query-dsl-bool-query.html[Boolean query] or a full text query which may or may not use the same -query text as the `sparse_vector` query. This enables you to combine the search -results from both queries. - -The search hits from the `sparse_vector` query tend to score higher than other -{es} queries. Those scores can be regularized by increasing or decreasing the -relevance scores of each query by using the `boost` parameter. Recall on the -`sparse_vector` query can be high where there is a long tail of less relevant -results. Use the `min_score` parameter to prune those less relevant documents. - -[source,bash] ----- -curl -X GET "${ES_URL}/my-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d' -{ - "query": { - "bool": { <1> - "should": [ - { - "sparse_vector": { - "field": "content_embedding", - "query": "How to avoid muscle soreness after running?", - "inference_id": "my-elser-endpoint", - "boost": 1 <2> - } - } - }, - { - "query_string": { - "query": "toxins", - "boost": 4 <3> - } - } - ] - } - }, - "min_score": 10 <4> -} -' ----- - -<1> Both the `sparse_vector` and the `query_string` queries are in a `should` -clause of a `bool` query. - -<2> The `boost` value is `1` for the `sparse_vector` query which is the default -value. This means that the relevance score of the results of this query are not -boosted. - -<3> The `boost` value is `4` for the `query_string` query. The relevance score -of the results of this query is increased causing them to rank higher in the -search results. - -<4> Only the results with a score equal to or higher than `10` are displayed. - -[discrete] -[[optimization]] -== Optimizing performance - -[discrete] -[[save-space]] -=== Saving disk space by excluding the ELSER tokens from document source - -The tokens generated by ELSER must be indexed for use in the -{ref}/query-dsl-sparse-vector-query.html[sparse_vector query]. However, it is not -necessary to retain those terms in the document source. You can save disk space -by using the {ref}/mapping-source-field.html#include-exclude[source exclude] mapping to remove the ELSER -terms from the document source. - -[WARNING] -==== -Reindex uses the document source to populate the destination index. -Once the ELSER terms have been excluded from the source, they cannot be -recovered through reindexing. Excluding the tokens from the source is a -space-saving optimsation that should only be applied if you are certain that -reindexing will not be required in the future! It's important to carefully -consider this trade-off and make sure that excluding the ELSER terms from the -source aligns with your specific requirements and use case. -==== - -The mapping that excludes `content_embedding` from the `_source` field can be -created by the following API call: - -[source,bash] ----- -curl -X PUT "${ES_URL}/my-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d' -{ - "mappings": { - "_source": { - "excludes": [ - "content_embedding" - ] - }, - "properties": { - "content_embedding": { - "type": "sparse_vector" - }, - "content": { - "type": "text" - } - } - } -} -' ----- - -[discrete] -[[further-reading]] -== Further reading - -* {ml-docs}/ml-nlp-elser.html[How to download and deploy ELSER] -* {ml-docs}/ml-nlp-limitations.html#ml-nlp-elser-v1-limit-512[ELSER limitation] -* https://www.elastic.co/blog/may-2023-launch-information-retrieval-elasticsearch-ai-model[Improving information retrieval in the Elastic Stack: Introducing Elastic Learned Sparse Encoder, our new retrieval model] - -[discrete] -[[interactive-example]] -== Interactive example - -* The `elasticsearch-labs` repo has an interactive example of running https://github.com/elastic/elasticsearch-labs/blob/main/notebooks/search/03-ELSER.ipynb[ELSER-powered semantic search] using the {es} Python client. diff --git a/serverless/pages/search-your-data-semantic-search-elser.asciidoc b/serverless/pages/search-your-data-semantic-search-elser.asciidoc new file mode 100644 index 00000000..02712bc1 --- /dev/null +++ b/serverless/pages/search-your-data-semantic-search-elser.asciidoc @@ -0,0 +1,9 @@ +[[elasticsearch-reference-semantic-search-elser]] += Tutorial: Semantic search with ELSER + +// :description: Perform semantic search using ELSER, an NLP model trained by Elastic. +// :keywords: elasticsearch, elser, semantic search + +// This page is not included in the index file, so it is not visible in the navigation menu anymore. HTTP redirects will be set up. + +ℹ️ Refer to <> for an overview of semantic search in {es-serverless}. \ No newline at end of file diff --git a/serverless/pages/search-your-data-semantic-search.asciidoc b/serverless/pages/search-your-data-semantic-search.asciidoc index efe8670f..de30fe82 100644 --- a/serverless/pages/search-your-data-semantic-search.asciidoc +++ b/serverless/pages/search-your-data-semantic-search.asciidoc @@ -8,136 +8,42 @@ Semantic search is a search method that helps you find data based on the intent and contextual meaning of a search query, instead of a match on query terms (lexical search). -{es} provides semantic search capabilities using {ml-docs}/ml-nlp.html[natural -language processing (NLP)] and vector search. Deploying an NLP model to {es} -enables it to extract text embeddings out of text. Embeddings are vectors that -provide a numeric representation of a text. Pieces of content with similar -meaning have similar representations. +Elasticsearch provides various semantic search capabilities using natural language processing (NLP) and vector search. Using an NLP model enables you to extract text embeddings out of text. Embeddings are vectors that provide a numeric representation of a text. Pieces of content with similar meaning have similar representations. -image::images/vector-search-oversimplification.png[A simplified representation of encoding textual concepts as vectors] +There are three main workflows for implementing semantic search with {es}, arranged in order of increasing complexity: -_A simplified representation of encoding textual concepts as vectors_ +- <> +- <> +- <> -At query time, {es} can use the same NLP model to convert a query into -embeddings, enabling you to find documents with similar text embeddings. +image::images/semantic-options.svg[Overview of semantic search workflows in {es}] -This guide shows you how to implement semantic search with {es}, from selecting -an NLP model, to writing queries. +[NOTE] +==== +Semantic search is available on all Elastic deployment types: self-managed clusters, Elastic Cloud Hosted deployments, and {es-serverless} projects. The links on this page will take you to the {ref}/semantic-search.html[{es} core documentation]. +==== [discrete] -[[semantic-search-select-nlp-model]] -== Select an NLP model - -{es} offers the usage of a -{ml-docs}/ml-nlp-model-ref.html#ml-nlp-model-ref-text-embedding[wide range of NLP models], -including both dense and sparse vector models. Your choice of the language model -is critical for implementing semantic search successfully. - -While it is possible to bring your own text embedding model, achieving good -search results through model tuning is challenging. Selecting an appropriate -model from our third-party model list is the first step. Training the model on -your own data is essential to ensure better search results than using only BM25. -However, the model training process requires a team of data scientists and ML -experts, making it expensive and time-consuming. - -To address this issue, Elastic provides a pre-trained representational model -called {ml-docs}/ml-nlp-elser.html[Elastic Learned Sparse EncodeR (ELSER)]. -ELSER, currently available only for English, is an out-of-domain sparse vector -model that does not require fine-tuning. This adaptability makes it suitable for -various NLP use cases out of the box. Unless you have a team of ML specialists, -it is highly recommended to use the ELSER model. - -In the case of sparse vector representation, the vectors mostly consist of zero -values, with only a small subset containing non-zero values. This representation -is commonly used for textual data. In the case of ELSER, each document in an -index and the query text itself are represented by high-dimensional sparse -vectors. Each non-zero element of the vector corresponds to a term in the model -vocabulary. The ELSER vocabulary contains around 30000 terms, so the sparse -vectors created by ELSER contain about 30000 values, the majority of which are -zero. Effectively the ELSER model is replacing the terms in the original query -with other terms that have been learnt to exist in the documents that best match -the original search terms in a training dataset, and weights to control how -important each is. +[[elasticsearch-reference-semantic-search-semantic-text]] +== Semantic search with `semantic text` -[discrete] -[[semantic-search-deploy-nlp-model]] -== Deploy the model - -After you decide which model you want to use for implementing semantic search, -you need to deploy the model in {es}. +The `semantic_text` field simplifies semantic search by providing inference at ingestion time with sensible default values, eliminating the need for complex configurations. -include::../partials/deploy-nlp-model-widget.asciidoc[] +Learn how to implement semantic search with `semantic text` in the {ref}/semantic-search-semantic-text.html[Elasticsearch docs →]. [discrete] -[[semantic-search-field-mappings]] -== Map a field for the text embeddings +[[elasticsearch-reference-semantic-search-inference-api]] +== Semantic search with the inference API -Before you start using the deployed model to generate embeddings based on your -input text, you need to prepare your index mapping first. The mapping of the -index depends on the type of model. +The inference API workflow enables you to perform semantic search using models from a variety of services, such as Cohere, OpenAI, HuggingFace, Azure AI Studio, and more. -include::../partials/field-mappings-widget.asciidoc[] - -[discrete] -[[semantic-search-generate-embeddings]] -== Generate text embeddings +Learn how to implement semantic search with the inference API in the {ref}/semantic-search-inference.html[Elasticsearch docs →]. -Once you have created the mappings for the index, you can generate text -embeddings from your input text. This can be done by using an -{ref}/ingest.html[ingest pipeline] with an {ref}/inference-processor.html[inference processor]. -The ingest pipeline processes the input data and indexes it into the destination -index. At index time, the inference ingest processor uses the trained model to -infer against the data ingested through the pipeline. After you created the -ingest pipeline with the inference processor, you can ingest your data through -it to generate the model output. - -include::../partials/generate-embeddings-widget.asciidoc[] - -Now it is time to perform semantic search! [discrete] -[[semantic-search-search]] -== Search the data - -Depending on the type of model you have deployed, you can query sparse vectors -with a sparse vector query, or dense vectors with a kNN search. +[[elasticsearch-reference-semantic-search-model-deployment]] +== Semantic search with the model deployment workflow -include::../partials/search-widget.asciidoc[] +The model deployment workflow enables you to deploy custom NLP models in Elasticsearch, giving you full control over text embedding generation and vector search. While this workflow offers advanced flexibility, it requires expertise in NLP and machine learning. -[discrete] -[[semantic-search-hybrid-search]] -== Beyond semantic search with hybrid search - -In some situations, lexical search may perform better than semantic search. For -example, when searching for single words or IDs, like product numbers. - -Combining semantic and lexical search into one hybrid search request using -{ref}/rrf.html[reciprocal rank fusion] provides the best of both worlds. Not only that, -but hybrid search using reciprocal rank fusion {blog-ref}improving-information-retrieval-elastic-stack-hybrid[has been shown to perform better -in general]. - -include::../partials/hybrid-search-widget.asciidoc[] - -[discrete] -[[semantic-search-read-more]] -== Read more - -* Tutorials: -+ -** <> -** {ml-docs}/ml-nlp-text-emb-vector-search-example.html[Semantic search with the msmarco-MiniLM-L-12-v3 sentence-transformer model] -* Blogs: -+ -** {blog-ref}may-2023-launch-sparse-encoder-ai-model[Introducing Elastic Learned Sparse Encoder: Elastic's AI model for semantic search] -** {blog-ref}lexical-ai-powered-search-elastic-vector-database[How to get the best of lexical and AI-powered search with Elastic's vector database] -** Information retrieval blog series: -+ -*** {blog-ref}improving-information-retrieval-elastic-stack-search-relevance[Part 1: Steps to improve search relevance] -*** {blog-ref}improving-information-retrieval-elastic-stack-benchmarking-passage-retrieval[Part 2: Benchmarking passage retrieval] -*** {blog-ref}may-2023-launch-information-retrieval-elasticsearch-ai-model[Part 3: Introducing Elastic Learned Sparse Encoder, our new retrieval model] -*** {blog-ref}improving-information-retrieval-elastic-stack-hybrid[Part 4: Hybrid retrieval] -* Interactive examples: -+ -** The https://github.com/elastic/elasticsearch-labs[`elasticsearch-labs`] repo contains a number of interactive semantic search examples in the form of executable Python notebooks, using the {es} Python client. - -// The include that was here is another page +Learn how to implement semantic search with the model deployment workflow in the {ref}/semantic-search-deployed-nlp-model.html[Elasticsearch docs →]. diff --git a/serverless/pages/serverless-differences.asciidoc b/serverless/pages/serverless-differences.asciidoc index 65533244..398edcee 100644 --- a/serverless/pages/serverless-differences.asciidoc +++ b/serverless/pages/serverless-differences.asciidoc @@ -1,17 +1,111 @@ [[elasticsearch-differences]] = Differences from other {es} offerings +++++ +Serverless differences +++++ // :description: Understand how {es-serverless} differs from Elastic Cloud Hosted and self-managed offerings. // :keywords: serverless, elasticsearch -++++ -Serverless differences -++++ +<> handles all the infrastructure management for you, providing a fully managed {es} service. + +If you've used {es} before, you'll notice some differences in how you work with the service on {serverless-full}, because a number of APIs and settings are not required for serverless projects. + +This guide helps you understand what's different, what's available, and how to work effectively when running {es} on {serverless-full}. + +[discrete] +[[elasticsearch-differences-serverless-infrastructure-management]] +== Fully managed infrastructure + +{es-serverless} manages all infrastructure automatically, including: + +* Cluster scaling and optimization +* Node management and allocation +* Shard distribution and replication +* Resource utilization and monitoring + +This fully managed approach means many traditional {es} infrastructure APIs and settings are not available to end users, as detailed in the following sections. + +[discrete] +[[elasticsearch-differences-serverless-apis-availability]] +== API availability + +Because {es-serverless} manages infrastructure automatically, certain APIs are not available, while others remain fully accessible. + +[TIP] +==== +Refer to the https://www.elastic.co/docs/api/doc/elasticsearch-serverless[{es-serverless} API reference] for a complete list of available APIs. +==== + +The following categories of operations are unavailable: + +Infrastructure operations:: +* All `_nodes/*` operations +* All `_cluster/*` operations +* Most `_cat/*` operations, except for index-related operations such as `/_cat/indices` and `/_cat/aliases` + +Storage and backup:: +* All `_snapshot/*` operations +* Repository management operations + +Index management:: +* `indices/close` operations +* `indices/open` operations +* Recovery and stats operations +* Force merge operations + +When attempting to use an unavailable API, you'll receive a clear error message: + +[source,json] +---- +{ + "error": { + "root_cause": [ + { + "type": "api_not_available_exception", + "reason": "Request for uri [/] with method [] exists but is not available when running in serverless mode" + } + ], + "status": 410 + } +} +---- + +[discrete] +[[elasticsearch-differences-serverless-settings-availability]] +== Settings availability + +In {es-serverless}, you can only configure {ref}/index-modules.html#index-modules-settings[index-level settings]. +Cluster-level settings and node-level settings are not required by end users and the `elasticsearch.yml` file is fully managed by Elastic. + +Available settings:: +*Index-level settings*: Settings that control how {es} documents are processed, stored, and searched are available to end users. These include: +* Analysis configuration +* Mapping parameters +* Search/query settings +* Indexing settings such as `refresh_interval` + +Managed settings:: +*Infrastructure-related settings*: Settings that affect cluster resources or data distribution are not available to end users. These include: +* Node configurations +* Cluster topology +* Shard allocation +* Resource management + +[discrete] +[[elasticsearch-differences-serverless-feature-categories]] +== Feature availability Some features that are available in Elastic Cloud Hosted and self-managed offerings are not available in {es-serverless}. -These features have either been replaced by a new feature, or are not applicable in the new Serverless architecture: +These features have either been replaced by a new feature, are planned to be released in future, or are not applicable in the new serverless architecture. + +[discrete] +[[elasticsearch-differences-serverless-features-replaced]] +=== Replaced features -* **Index lifecycle management ({ilm-init})** is not available, in favor of <>. +These features have been replaced by a new feature and are therefore not available on {es-serverless}: + +* *Index lifecycle management ({ilm-init})* is not available, in favor of <>. + In an Elastic Cloud Hosted or self-managed environment, {ilm-init} lets you automatically transition indices through data tiers according to your performance needs and retention requirements. This allows you to balance hardware costs with performance. {es-serverless} eliminates this @@ -20,21 +114,44 @@ complexity by optimizing your cluster performance for you. Data stream lifecycle is an optimized lifecycle tool that lets you focus on the most common lifecycle management needs, without unnecessary hardware-centric concepts like data tiers. + -* **Watcher** is not available, in favor of Kibana Alerts**. +* *Watcher* is not available, in favor of <>. + Kibana Alerts allows rich integrations across use cases like APM, metrics, security, and uptime. Prepackaged rule types simplify setup and hide the details of complex, domain-specific detections, while providing a consistent interface across Kibana. + +[discrete] +[[elasticsearch-differences-serverless-feature-planned]] +=== Planned features + +The following features are planned for future support in all {serverless-full} projects: + +* Reindexing from remote clusters +* Cross-project search and replication +* Snapshot and restore +* Migrations from non-serverless deployments +* Audit logging +* Authentication realms (native/SAML/OIDC/Kerberos/JWT) +* Clone index API +* Traffic filtering and VPCs + +The following {es-serverless} project-specific features are planned for future support: + +* {ref}/behavioral-analytics-overview.html[Behavioral Analytics] +* {ref}/search-application-overview.html[Search Applications] +* Managed web crawler ++ +You can use the https://github.com/elastic/crawler[self-managed web crawler] in the meantime. + -* Certain APIs, API parameters, index, cluster and node level settings are not available. Refer to our -<> for a list of available APIs. -+ -{es-serverless} manages the underlying Elastic cluster for you, optimizing nodes, shards, and replicas for your use case. -Because of this, various management and monitoring APIs, API parameters and settings are not available on Serverless. -+ -* {ref}/search-aggregations-metrics-scripted-metric-aggregation.html[Scripted metric aggregations] are not available. +* Managed Search connectors ++ +You can use {ref}/es-build-connector.html[self-managed Search connectors] in the meantime. -.Other limitations -[IMPORTANT] -==== -For serverless technical preview limitations, refer to <>. -==== +[discrete] +[[elasticsearch-differences-serverless-feature-unavailable]] +=== Unplanned features + +The following features are not available in {es-serverless} and are not planned for future support: + +* https://www.elastic.co/guide/en/cloud/current/ec-custom-bundles.html[Custom plugins and bundles] +* https://www.elastic.co/guide/en/elasticsearch/hadoop/current/reference.html[{es} for Apache Hadoop] +* {ref}/search-aggregations-metrics-scripted-metric-aggregation.html[Scripted metric aggregations] diff --git a/serverless/pages/technical-preview-limitations.asciidoc b/serverless/pages/technical-preview-limitations.asciidoc index f34629a2..830ff882 100644 --- a/serverless/pages/technical-preview-limitations.asciidoc +++ b/serverless/pages/technical-preview-limitations.asciidoc @@ -1,22 +1,10 @@ [[elasticsearch-technical-preview-limitations]] -= Technical preview limitations += Limitations // :description: Review the limitations that apply to {es-serverless} projects. // :keywords: serverless, elasticsearch -The following are currently not available: -* Custom plugins and custom bundles -* Reindexing from remote clusters -* Cross-cluster search and cross-cluster replication -* Snapshot and restore -* Clone index API -* Migrations from non-serverless {es} deployments. In the interim, you can <> to move data to and from serverless projects. -* Audit logging -* {es} for Apache Hadoop +// This page is not included in the index file, so it is not visible in the navigation menu anymore. HTTP redirects will be set up. -Currently, workloads outside of the following ranges may experience higher latencies (greater than sub-second): - -* Search queries on indices greater than 150GB -* Index queries per second (QPS) greater than 1000 -* Search queries per second (QPS) greater than 1800 +Refer to <> for a comparison of {es-serverless} with other Elastic Cloud offerings. \ No newline at end of file diff --git a/serverless/pages/what-is-elasticsearch-serverless.asciidoc b/serverless/pages/what-is-elasticsearch-serverless.asciidoc index 5f52c06c..262e585d 100644 --- a/serverless/pages/what-is-elasticsearch-serverless.asciidoc +++ b/serverless/pages/what-is-elasticsearch-serverless.asciidoc @@ -4,7 +4,7 @@ // :description: Build search solutions and applications with {es-serverless}. // :keywords: serverless, elasticsearch, overview -{es} is an open source distributed search and analytics engine, scalable data store, and vector database. It's optimized for speed and relevance on production-scale workloads. +{es} is an open source distributed search and analytics engine, scalable data store, and vector database. It’s optimized for speed and relevance on production-scale workloads. .Understanding Elasticsearch on serverless [IMPORTANT] @@ -12,7 +12,6 @@ If you haven't used {es} before, first learn the basics in the https://www.elastic.co/guide/en/elasticsearch/reference/current/elasticsearch-intro.html[core {es} documentation]. ==== - {es} is one of the three available project types on <>. This project type enables you to use the core functionality of {es}: searching, indexing, storing, and analyzing data of all shapes and sizes. @@ -32,12 +31,9 @@ a| [.card-title]#<># + Get started by creating your first {es} project on serverless. | 🔌 -a| [.card-title]#<># + +a| [.card-title]#<># + Learn how to connect your applications to your {es-serverless} endpoint. -// TODO add coming link to new page about connecting to your serverless endpoint -// <> - | ⤵️ a| [.card-title]#<># + Learn how to get your data into {es} and start building your search application.