Merge branch 'main' into stef_testNewAPI

README.asciidoc

@@ -1,20 +1,24 @@
 = Elasticsearch
 
-Elasticsearch is a distributed, RESTful search engine optimized for speed and relevance on production-scale workloads. You can use Elasticsearch to perform real-time search over massive datasets for applications including:
+Elasticsearch is a distributed search and analytics engine optimized for speed and relevance on production-scale workloads. Elasticsearch is the foundation of Elastic's open Stack platform. Search in near real-time over massive datasets, perform vector searches, integrate with generative AI applications, and much more.
 
-* Vector search
+Use cases enabled by Elasticsearch include:
+
+* https://www.elastic.co/search-labs/blog/articles/retrieval-augmented-generation-rag[Retrieval Augmented Generation (RAG)] 
+* https://www.elastic.co/search-labs/blog/categories/vector-search[Vector search]
 * Full-text search
 * Logs
 * Metrics
 * Application performance monitoring (APM)
 * Security logs
 
-
 \... and more!
 
 To learn more about Elasticsearch's features and capabilities, see our
 https://www.elastic.co/products/elasticsearch[product page].
 
+To access information on https://www.elastic.co/search-labs/blog/categories/ml-research[machine learning innovations] and the latest https://www.elastic.co/search-labs/blog/categories/lucene[Lucene contributions from Elastic], more information can be found in https://www.elastic.co/search-labs[Search Labs]. 
+
 [[get-started]]
 == Get started
 

build-tools-internal/version.properties

@@ -1,5 +1,5 @@
 elasticsearch     = 8.14.0
-lucene            = 9.9.2
+lucene            = 9.10.0
 
 bundled_jdk_vendor = openjdk
 bundled_jdk = 21.0.2+13@f2283984656d49d69e91c558476027ac

build-tools/src/main/java/org/elasticsearch/gradle/testclusters/ElasticsearchCluster.java

@@ -433,7 +433,7 @@ private void commonNodeConfig() {
             if (node.getTestDistribution().equals(TestDistribution.INTEG_TEST)) {
                 node.defaultConfig.put("xpack.security.enabled", "false");
             } else {
-                if (node.getVersion().onOrAfter("7.16.0")) {
+                if (hasDeprecationIndexing(node)) {
                     node.defaultConfig.put("cluster.deprecation_indexing.enabled", "false");
                 }
             }
@@ -474,13 +474,17 @@ public void nextNodeToNextVersion() {
         commonNodeConfig();
         nodeIndex += 1;
         if (node.getTestDistribution().equals(TestDistribution.DEFAULT)) {
-            if (node.getVersion().onOrAfter("7.16.0")) {
+            if (hasDeprecationIndexing(node)) {
                 node.setting("cluster.deprecation_indexing.enabled", "false");
             }
         }
         node.start();
     }
 
+    private static boolean hasDeprecationIndexing(ElasticsearchNode node) {
+        return node.getVersion().onOrAfter("7.16.0") && node.getSettingKeys().contains("stateless.enabled") == false;
+    }
+
     @Override
     public void extraConfigFile(String destination, File from) {
         nodes.all(node -> node.extraConfigFile(destination, from));

docs/Versions.asciidoc

@@ -1,8 +1,8 @@
 
 include::{docs-root}/shared/versions/stack/{source_branch}.asciidoc[]
 
-:lucene_version:        9.9.2
-:lucene_version_path:   9_9_2
+:lucene_version:        9.10.0
+:lucene_version_path:   9_10_0
 :jdk:                   11.0.2
 :jdk_major:             11
 :build_type:            tar

docs/changelog/105578.yaml

@@ -0,0 +1,13 @@
+pr: 105578
+summary: Upgrade to Lucene 9.10.0
+area: Search
+type: enhancement
+issues: []
+highlight:
+  title: New Lucene 9.10 release
+  body: |-
+    - https://github.com/apache/lucene/pull/13090: Prevent humongous allocations in ScalarQuantizer when building quantiles.
+    - https://github.com/apache/lucene/pull/12962: Speedup concurrent multi-segment HNSW graph search
+    - https://github.com/apache/lucene/pull/13033: Range queries on numeric/date/ip fields now exit earlier on segments whose values don't intersect with the query range. This should especially help when there are other required clauses in the `bool` query and when the range filter is narrow, e.g. filtering on the last 5 minutes.
+    - https://github.com/apache/lucene/pull/13026: `bool` queries that mix `filter` and `should` clauses will now propagate minimum competitive scores through the `should` clauses. This should yield speedups when sorting by descending score.
+  notable: true

docs/changelog/105772.yaml

@@ -0,0 +1,5 @@
+pr: 105772
+summary: "[ILM] Delete step deletes data stream with only one index"
+area: ILM+SLM
+type: bug
+issues: []

docs/changelog/105789.yaml

@@ -0,0 +1,6 @@
+pr: 105789
+summary: Make Health API more resilient to multi-version clusters
+area: Health
+type: bug
+issues:
+ - 90183

docs/changelog/105848.yaml

@@ -0,0 +1,5 @@
+pr: 105848
+summary: '`ProjectOperator` should not retain references to released blocks'
+area: ES|QL
+type: bug
+issues: []

docs/reference/api-conventions.asciidoc

@@ -302,6 +302,14 @@ Content-Type: application/vnd.elasticsearch+json; compatible-with=7
 Accept: application/vnd.elasticsearch+json; compatible-with=7
 ----------------------------------------------------------------------
 
+[discrete]
+[[api-push-back]]
+=== HTTP `429 Too Many Requests` status code push back
+
+{es} APIs may respond with the HTTP `429 Too Many Requests` status code, indicating that the cluster is too busy
+to handle the request. When this happens, consider retrying after a short delay. If the retry also receives
+a `429 Too Many Requests` response, extend the delay by backing off exponentially before each subsequent retry.
+
 [discrete]
 [[api-url-access-control]]
 === URL-based access control

docs/reference/cluster/get-desired-balance.asciidoc

@@ -7,6 +7,7 @@
 NOTE: {cloud-only}
 
 Exposes:
+
 * the desired balance computation and reconciliation stats
 * balancing stats such as distribution of shards, disk and ingest forecasts
   across nodes and data tiers (based on the current cluster state)

docs/reference/indices/shrink-index.asciidoc

@@ -101,7 +101,8 @@ A shrink operation:
   disks)
 
 . Recovers the target index as though it were a closed index which
-  had just been re-opened.
+  had just been re-opened. Recovers shards to <<indices-get-settings,Index Setting>> 
+  `.routing.allocation.initial_recovery._id`.
 
 
 [[_shrinking_an_index]]

docs/reference/ingest/processors/geoip.asciidoc

@@ -435,6 +435,8 @@ each node's <<es-tmpdir,temporary directory>> at `$ES_TMPDIR/geoip-databases/<no
 Note that {es} will make a GET request to `${ingest.geoip.downloader.endpoint}?elastic_geoip_service_tos=agree`,
 expecting the list of metadata about databases typically found in `overview.json`.
 
+The GeoIP downloader uses the JDK's builtin cacerts. If you're using a custom endpoint, add the custom https endpoint cacert(s) to the JDK's truststore.
+
 [[ingest-geoip-downloader-poll-interval]]
 `ingest.geoip.downloader.poll.interval`::
 (<<dynamic-cluster-setting,Dynamic>>, <<time-units,time value>>)

docs/reference/mapping/params/format.asciidoc

@@ -70,6 +70,11 @@ The following tables lists all the defaults ISO formats supported:
     (separated by `T`), is optional.
     Examples: `yyyy-MM-dd'T'HH:mm:ss.SSSZ` or  `yyyy-MM-dd`.
 
+    NOTE: When using `date_optional_time`, the parsing is lenient and will attempt to parse
+    numbers as a year (e.g. `292278994` will be parsed as a year). This can lead to unexpected results
+    when paired with a numeric focused format like `epoch_second` and `epoch_millis`.
+    It is recommended you use `strict_date_optional_time` when pairing with a numeric focused format.
+
 [[strict-date-time-nanos]]`strict_date_optional_time_nanos`::
 
     A generic ISO datetime parser, where the date must include the year at a minimum, and the time

docs/reference/mapping/types/dense-vector.asciidoc

@@ -232,7 +232,6 @@ expense of slower indexing speed.
 +
 ^*^ This parameter can only be specified when `index` is `true`.
 +
-+
 .Properties of `index_options`
 [%collapsible%open]
 ====