Merge remote-tracking branch 'origin/main' into esql_searchstats_for_…

…indexed_docvalues
craigtaverner · Nov 5, 2024 · edba324 · edba324
2 parents a574391 + dd1b623
commit edba324
Show file tree

Hide file tree

Showing 303 changed files with 6,956 additions and 2,702 deletions.
diff --git a/docs/changelog/114227.yaml b/docs/changelog/114227.yaml
@@ -0,0 +1,6 @@
+pr: 114227
+summary: Ignore conflicting fields during dynamic mapping update
+area: Mapping
+type: bug
+issues:
+ - 114228
diff --git a/docs/changelog/114879.yaml b/docs/changelog/114879.yaml
@@ -0,0 +1,5 @@
+pr: 114879
+summary: Add refresh `.security` index call between security migrations
+area: Security
+type: enhancement
+issues: []
diff --git a/docs/changelog/115779.yaml b/docs/changelog/115779.yaml
@@ -0,0 +1,6 @@
+pr: 115779
+summary: Don't allow secure settings in YML config (109115)
+area: Infra/Settings
+type: bug
+issues:
+ - 109115
diff --git a/docs/changelog/115797.yaml b/docs/changelog/115797.yaml
@@ -0,0 +1,6 @@
+pr: 115797
+summary: Enable `_tier` based coordinator rewrites for all indices (not just mounted
+  indices)
+area: Search
+type: enhancement
+issues: []
diff --git a/docs/changelog/115876.yaml b/docs/changelog/115876.yaml
@@ -0,0 +1,5 @@
+pr: 115876
+summary: Inference duration and error metrics
+area: Machine Learning
+type: enhancement
+issues: []
diff --git a/docs/changelog/115994.yaml b/docs/changelog/115994.yaml
@@ -0,0 +1,5 @@
+pr: 115994
+summary: Add logsdb telemetry
+area: Logs
+type: enhancement
+issues: []
diff --git a/docs/changelog/116015.yaml b/docs/changelog/116015.yaml
@@ -0,0 +1,6 @@
+pr: 116015
+summary: Empty percentile results no longer throw no_such_element_exception in Anomaly Detection jobs
+area: Machine Learning
+type: bug
+issues:
+ - 116013
diff --git a/docs/changelog/116031.yaml b/docs/changelog/116031.yaml
@@ -0,0 +1,6 @@
+pr: 116031
+summary: Resolve pipelines from template on lazy rollover write
+area: Data streams
+type: bug
+issues:
+ - 112781
diff --git a/docs/changelog/116043.yaml b/docs/changelog/116043.yaml
@@ -0,0 +1,6 @@
+pr: 116043
+summary: Support partial sort fields in TopN pushdown
+area: ES|QL
+type: enhancement
+issues:
+ - 114515
diff --git a/docs/changelog/116060.yaml b/docs/changelog/116060.yaml
@@ -0,0 +1,6 @@
+pr: 116060
+summary: Fix leak in `DfsQueryPhase` and introduce search disconnect stress test
+area: Search
+type: bug
+issues:
+ - 115056
diff --git a/docs/changelog/116086.yaml b/docs/changelog/116086.yaml
@@ -0,0 +1,6 @@
+pr: 116086
+summary: "ESQL: Fix DEBUG log of filter"
+area: ES|QL
+type: bug
+issues:
+ - 116055
diff --git a/docs/changelog/116112.yaml b/docs/changelog/116112.yaml
@@ -0,0 +1,13 @@
+pr: 116112
+summary: Remove ignored fallback option on GeoIP processor
+area: Ingest Node
+type: breaking
+issues: []
+breaking:
+  title: Remove ignored fallback option on GeoIP processor
+  area: Ingest
+  details: >-
+    The option fallback_to_default_databases on the geoip ingest processor has been removed.
+    (It was deprecated and ignored since 8.0.0.)
+  impact: Customers should stop remove the noop fallback_to_default_databases option on any geoip ingest processors.
+  notable: false
diff --git a/docs/changelog/116174.yaml b/docs/changelog/116174.yaml
@@ -0,0 +1,7 @@
+pr: 116174
+summary: Handle with `illegalArgumentExceptions` negative values in HDR percentile
+  aggregations
+area: Aggregations
+type: bug
+issues:
+ - 115777
diff --git a/docs/changelog/116211.yaml b/docs/changelog/116211.yaml
@@ -0,0 +1,5 @@
+pr: 116211
+summary: Use underlying `ByteBuf` `refCount` for `ReleasableBytesReference`
+area: Network
+type: bug
+issues: []
diff --git a/docs/changelog/116212.yaml b/docs/changelog/116212.yaml
@@ -0,0 +1,6 @@
+pr: 116212
+summary: Handle status code 0 in S3 CMU response
+area: Snapshot/Restore
+type: bug
+issues:
+ - 102294
diff --git a/docs/changelog/116219.yaml b/docs/changelog/116219.yaml
@@ -0,0 +1,6 @@
+pr: 116219
+summary: "[apm-data] Apply lazy rollover on index template creation"
+area: Data streams
+type: bug
+issues:
+ - 116230
diff --git a/docs/changelog/116259.yaml b/docs/changelog/116259.yaml
@@ -0,0 +1,14 @@
+pr: 116259
+summary: Fix `_type` deprecation on simulate pipeline API
+area: Ingest Node
+type: deprecation
+issues: []
+deprecation:
+  title: Document `_type` deprecated on simulate pipeline API
+  area: REST API
+  details: >-
+    Passing a document with a `_type` property is deprecated in the `/_ingest/pipeline/{id}/_simulate` and
+    `/_ingest/pipeline/_simulate` APIs.
+  impact: >-
+    Users should already have stopped using mapping types, which were deprecated in {es} 7. This deprecation warning
+    will fire if they specify mapping types on documents pass to the simulate pipeline API.
diff --git a/docs/reference/cat/shards.asciidoc b/docs/reference/cat/shards.asciidoc
@@ -9,13 +9,13 @@
 ====
 cat APIs are only intended for human consumption using the command line or {kib}
 console.
-They are _not_ intended for use by applications. For application 
+They are _not_ intended for use by applications. For application
 consumption, use the <<cluster-state,cluster state API>>.
 ====
 
 The `shards` command is the detailed view of all nodes' shard <<shard-allocation-relocation-recovery,allocation>>.
-It will tell you if the shard is a primary or replica, the number of docs, the 
-bytes it takes on disk, the node where it's located, and if the shard is 
+It will tell you if the shard is a primary or replica, the number of docs, the
+bytes it takes on disk, the node where it's located, and if the shard is
 currently <<shard-allocation-relocation-recovery,recovering>>.
 
 For <<data-streams,data streams>>, the API returns information about the stream's backing indices.
@@ -258,9 +258,6 @@ Time spent in suggest, such as `0`.
 `suggest.total`, `suto`, `suggestTotal`::
 Number of suggest operations, such as `0`.
 
-`sync_id`::
-Sync ID of the shard.
-
 `unassigned.at`, `ua`::
 Time at which the shard became unassigned in
 {wikipedia}/List_of_UTC_time_offsets[Coordinated Universal Time (UTC)].

diff --git a/docs/reference/connector/docs/_connectors-create-client.asciidoc b/docs/reference/connector/docs/_connectors-create-client.asciidoc
@@ -3,7 +3,7 @@
 
 To create a new {service-name} connector:
 
-. Navigate to the *Search -> Connectors* page in the Kibana UI.
+. In the Kibana UI, navigate to the *Search -> Content -> Connectors* page from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 . Follow the instructions to create a new  *{service-name}* self-managed connector.
 
 [discrete#es-connectors-{service-name-stub}-client-create-use-the-api]

diff --git a/docs/reference/connector/docs/_connectors-create-native.asciidoc b/docs/reference/connector/docs/_connectors-create-native.asciidoc
@@ -3,7 +3,7 @@
 
 To create a new {service-name} connector:
 
-. Navigate to the *Search -> Connectors* page in the Kibana UI.
+. In the Kibana UI, navigate to the *Search -> Content -> Connectors* page from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 . Follow the instructions to create a new native  *{service-name}* connector.
 
 For additional operations, see <<es-connectors-usage>>.

diff --git a/docs/reference/connector/docs/connectors-hosted-tutorial-mongo.asciidoc b/docs/reference/connector/docs/connectors-hosted-tutorial-mongo.asciidoc
@@ -123,7 +123,7 @@ Once you're deployment is created, navigate to *Search*.
 The Elastic connector will sync your MongoDB data into a search-optimized Elasticsearch index.
 The first step is to create your index in the Kibana UI.
 
-In the main menu navigate to *Search > Content > Indices*.
+In the main menu, navigate to *Search > Content > Indices*, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Follow these steps to create your index:
 
@@ -178,7 +178,7 @@ If all the configuration details are correct, the sync will begin and documents
 
 As soon as your first documents are synced, you can view the documents and inspect the mapping for the index:
 
-* In Kibana, navigate to *Search* > *Content* > *Indices*.
+* In Kibana, navigate to *Search* > *Content* > *Indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 * Select your index, for example `search-mongo-sample`.
 * Choose the *Documents* tab to view the synced documents.
 Expand a document to view its fields.

diff --git a/docs/reference/connector/docs/connectors-managed-service.asciidoc b/docs/reference/connector/docs/connectors-managed-service.asciidoc
@@ -80,7 +80,7 @@ Create a new index to be managed by the connector.
 
 Continue from above, or navigate to the following location within the {kib} UI:
 
-*Search > Content > Elasticsearch indices*
+*Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Choose the index to configure, and then choose the *Configuration* tab.
 

diff --git a/docs/reference/connector/docs/connectors-usage.asciidoc b/docs/reference/connector/docs/connectors-usage.asciidoc
@@ -3,7 +3,7 @@
 
 This document describes operations available to <<es-native-connectors,managed connectors>> and <<es-build-connector,self-managed connectors>>, using the UI.
 
-In the Kibana UI, go to *Search > Content > Connectors* to view a summary of all your connectors and sync jobs, and to create new connectors.
+In the Kibana UI, navigate to *Search > Content > Connectors* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field]. Here, you can view a summary of all your connectors and sync jobs, and to create new connectors.
 
 [TIP]
 ====
@@ -24,7 +24,7 @@ Once you've chosen the data source type you'd like to sync, you'll be prompted t
 
 View and manage all Elasticsearch indices managed by connectors.
 
-In the {kib} UI, navigate to *Search > Content > Connectors* to view a list of connector indices and their attributes, including connector type health and ingestion status.
+In the {kib} UI, navigate to *Search > Content > Connectors* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field]. Here, you can view a list of connector indices and their attributes, including connector type health and ingestion status.
 
 Within this interface, you can choose to view the details for each existing index or delete an index.
 Or, you can <<es-connectors-usage-index-create,create a new connector index>>.
@@ -82,7 +82,7 @@ The workflow for these updates is as follows:
 
 After creating an index to be managed by a connector, you can configure automatic, recurring syncs.
 
-In the {kib} UI, navigate to *Search > Content > Connectors*.
+In the {kib} UI, navigate to *Search > Content > Connectors* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Choose the index to configure, and then choose the *Scheduling* tab.
 
@@ -107,7 +107,7 @@ You may want to <<es-connectors-usage-index-view,view the index details>> to see
 
 After creating the index to be managed by a connector, you can request a single sync at any time.
 
-In the {kib} UI, navigate to *Search > Content > Elasticsearch indices*.
+In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Then choose the index to sync.
 
@@ -128,7 +128,7 @@ This operation requires access to Kibana and the `write` {ref}/security-privileg
 
 After a sync has started, you can cancel the sync before it completes.
 
-In the {kib} UI, navigate to *Search > Content > Elasticsearch indices*.
+In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Then choose the index with the running sync.
 
@@ -144,7 +144,7 @@ This operation requires access to Kibana and the `write` {ref}/security-privileg
 
 View the index details to see a variety of information that communicate the status of the index and connector.
 
-In the {kib} UI, navigate to *Search > Content > Elasticsearch indices*.
+In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Then choose the index to view.
 
@@ -192,7 +192,7 @@ This operation requires access to Kibana and the `read` {ref}/security-privilege
 View the documents the connector has synced from the data.
 Additionally view the index mappings to determine the current document schema.
 
-In the {kib} UI, navigate to *Search > Content > Elasticsearch indices*.
+In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Then choose the index to view.
 
@@ -211,7 +211,7 @@ See <<es-connectors-security>> for security details.
 
 Use <<es-sync-rules,sync rules>> to limit which documents are fetched from the data source, or limit which fetched documents are stored in Elastic.
 
-In the {kib} UI, navigate to *Search > Content > Elasticsearch indices*.
+In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Then choose the index to manage and choose the *Sync rules* tab.
 
@@ -220,6 +220,6 @@ Then choose the index to manage and choose the *Sync rules* tab.
 
 Use {ref}/ingest-pipeline-search.html[ingest pipelines] to transform fetched data before it is stored in Elastic.
 
-In the {kib} UI, navigate to *Search > Content > Elasticsearch indices*.
+In the {kib} UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Then choose the index to manage and choose the *Pipelines* tab.
diff --git a/docs/reference/connector/docs/dls-e2e-guide.asciidoc b/docs/reference/connector/docs/dls-e2e-guide.asciidoc
@@ -54,7 +54,7 @@ To build our search experience for our SharePoint Online data, we need to create
 
 Follow these steps to create a Search Application in the Kibana UI:
 
-. Navigate to *Search > Search Applications*.
+. Navigate to *Search > Search Applications* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 . Select *Create*.
 . *Name* the Search Application.
 . Select the *index* used by the SharePoint Online connector.

diff --git a/docs/reference/connector/docs/postgresql-connector-client-tutorial.asciidoc b/docs/reference/connector/docs/postgresql-connector-client-tutorial.asciidoc
@@ -70,7 +70,7 @@ To complete this tutorial, you'll need to complete the following steps:
 Elastic connectors enable you to create searchable, read-only replicas of your data sources in Elasticsearch.
 The first step in setting up your self-managed connector is to create an index.
 
-In the {kibana-ref}[Kibana^] UI go to *Search > Content > Elasticsearch indices*.
+In the {kibana-ref}[Kibana^] UI, navigate to *Search > Content > Elasticsearch indices* from the main menu, or use the {kibana-ref}/kibana-concepts-analysts.html#_finding_your_apps_and_objects[global search field].
 
 Create a new connector index:
 

diff --git a/docs/reference/esql/esql-functions-operators.asciidoc b/docs/reference/esql/esql-functions-operators.asciidoc
@@ -48,6 +48,12 @@ include::functions/ip-functions.asciidoc[tag=ip_list]
 include::functions/math-functions.asciidoc[tag=math_list]
 ====
 
+.*Search functions*
+[%collapsible]
+====
+include::functions/search-functions.asciidoc[tag=search_list]
+====
+
 .*Spatial functions*
 [%collapsible]
 ====
@@ -89,6 +95,7 @@ include::functions/conditional-functions-and-expressions.asciidoc[]
 include::functions/date-time-functions.asciidoc[]
 include::functions/ip-functions.asciidoc[]
 include::functions/math-functions.asciidoc[]
+include::functions/search-functions.asciidoc[]
 include::functions/spatial-functions.asciidoc[]
 include::functions/string-functions.asciidoc[]
 include::functions/type-conversion-functions.asciidoc[]