Update docs for v8.11.0 release (elastic#100367)

docs/reference/migration/migrate_8_11.asciidoc

@@ -16,5 +16,27 @@ coming::[8.11.0]
 [[breaking-changes-8.11]]
 === Breaking changes
 
-There are no breaking changes in {es} 8.11.
+The following changes in {es} 8.11 might affect your applications
+and prevent them from operating normally.
+Before upgrading to 8.11, review these changes and take the described steps
+to mitigate the impact.
+
+
+There are no notable breaking changes in {es} 8.11.
+But there are some less critical breaking changes.
+
+[discrete]
+[[breaking_811_rest_api_changes]]
+==== REST API changes
+
+[[remove_transport_versions_from_cluster_state_api]]
+.Remove `transport_versions` from cluster state API
+[%collapsible]
+====
+*Details* +
+The `transport_versions` subobject of the response to `GET _cluster/state` has been replaced by the `nodes_versions` subobject.
+
+*Impact* +
+If needed, retrieve the per-node transport versions from the `nodes_versions` subobject.
+====
 

docs/reference/release-notes/8.11.0.asciidoc

@@ -5,4 +5,287 @@ coming[8.11.0]
 
 Also see <<breaking-changes-8.11,Breaking changes in 8.11>>.
 
+[[breaking-8.11.0]]
+[float]
+=== Breaking changes
+
+Infra/Core::
+* Remove `transport_versions` from cluster state API {es-pull}99223[#99223]
+
+[[bug-8.11.0]]
+[float]
+=== Bug fixes
+
+Aggregations::
+* Allow parsing on non-string routing fields {es-pull}97729[#97729]
+* Fix cardinality agg for `const_keyword` {es-pull}99814[#99814] (issue: {es-issue}99776[#99776])
+* Support runtime fields in synthetic source {es-pull}99796[#99796] (issue: {es-issue}98287[#98287])
+
+Allocation::
+* Consider node shutdown in `DataTierAllocationDecider` {es-pull}98824[#98824] (issue: {es-issue}97207[#97207])
+
+Application::
+* [Profiling] Tighten resource creation check {es-pull}99873[#99873]
+
+CRUD::
+* Change `GetFromTranslog` to indices action {es-pull}99300[#99300]
+* Wait for cluster to recover before resolving index template {es-pull}99797[#99797]
+
+Cluster Coordination::
+* Reset `GatewayService` flags before reroute {es-pull}98653[#98653] (issue: {es-issue}98606[#98606])
+
+Data streams::
+* Propagate cancellation in `DataTiersUsageTransportAction` {es-pull}100253[#100253]
+
+Distributed::
+* Skip settings validation during desired nodes updates {es-pull}99946[#99946]
+
+Downsampling::
+* Make downsample target index replicas configurable {es-pull}99712[#99712]
+
+ES|QL::
+* "params" correctly parses the values including an optional "type" {es-pull}99310[#99310] (issue: {es-issue}99294[#99294])
+* Account for an exception being thrown when building a `BytesRefArrayBlock` {es-pull}99726[#99726] (issue: {es-issue}99472[#99472])
+* Add ESQL own flavor of arithmetic operators {es-pull}98628[#98628]
+* Better management of not stored TEXT fiels with synthetic source {es-pull}99695[#99695]
+* Continue resolving attributes for Eval {es-pull}99601[#99601] (issue: {es-issue}99576[#99576])
+* Create a Vector when needed for IN {es-pull}99382[#99382] (issue: {es-issue}99347[#99347])
+* Implement serialization of `InvalidMappedField` {es-pull}98972[#98972] (issue: {es-issue}98851[#98851])
+* Make ESQL more resilient to non-indexed fields {es-pull}99588[#99588] (issue: {es-issue}99506[#99506])
+* Skip synthetic attributes when planning the physical fragment {es-pull}99188[#99188] (issue: {es-issue}99170[#99170])
+* Support queries that don't return underlying fields {es-pull}98759[#98759] (issue: {es-issue}98404[#98404])
+* Use exact attributes for data source extraction {es-pull}99874[#99874] (issue: {es-issue}99183[#99183])
+
+Geo::
+* Use `NamedWritable` to enable `GeoBoundingBox` serialisation {es-pull}99163[#99163] (issue: {es-issue}99089[#99089])
+
+Health::
+* Fix NPE in `StableMasterHealthIndicatorService` {es-pull}98635[#98635]
+
+Highlighting::
+* Correctly handle `ScriptScoreQuery` in plain highlighter {es-pull}99804[#99804] (issue: {es-issue}99700[#99700])
+
+ILM+SLM::
+* Compute SLM retention from `RepositoryData` {es-pull}100092[#100092] (issue: {es-issue}99953[#99953])
+* ILM introduce the `check-ts-end-time-passed` step {es-pull}100179[#100179] (issue: {es-issue}99696[#99696])
+* ILM the delete action waits for a TSDS index time/bounds to lapse {es-pull}100207[#100207]
+
+Infra/Core::
+* Add `java.net.NetPermission` to APM module's permissions {es-pull}99474[#99474]
+* Revert "Kibana system index does not allow user templates to affect it" {es-pull}98888[#98888]
+* Specify correct current `IndexVersion` after 8.10 release {es-pull}98574[#98574] (issue: {es-issue}98555[#98555])
+
+Ingest Node::
+* Update enrich execution to only set index false on fields that support it {es-pull}98038[#98038] (issue: {es-issue}98019[#98019])
+* Validate enrich index before completing policy execution {es-pull}100106[#100106]
+
+Machine Learning::
+* Adding retry logic for start model deployment API {es-pull}99673[#99673]
+* Avoid risk of OOM in datafeeds when memory is constrained {es-pull}98915[#98915] (issue: {es-issue}89769[#89769])
+* Defend against negative datafeed start times {es-pull}100284[#100284]
+* Preserve order of inference results when calling the _infer API with multiple inputs on a model deployment with more than one allocation the output results order was not guaranteed to match the input order. The fix ensures the output order matches the input order. {es-pull}100143[#100143]
+* Safely drain deployment request queues before allowing node to shutdown {es-pull}98406[#98406]
+* Using 1 MB chunks for elser model storage {es-pull}99677[#99677]
+
+Mapping::
+* Automatically disable `ignore_malformed` on datastream `@timestamp` fields {es-pull}99346[#99346]
+* Correct behaviour of `ContentPath::remove()` {es-pull}98332[#98332] (issue: {es-issue}98327[#98327])
+* Fix merges of mappings with `subobjects: false` for composable index templates {es-pull}97317[#97317] (issue: {es-issue}96768[#96768])
+
+Network::
+* Do not report failure after connections are made {es-pull}99117[#99117]
+
+Percolator::
+* Fix percolator query for stored queries that expand on wildcard field names {es-pull}98878[#98878]
+
+Search::
+* Close expired search contexts on SEARCH thread {es-pull}99660[#99660]
+* Fix UnsignedLong field range query gt "0" can get the result equal to 0 {es-pull}98843[#98843]
+* Fix `advanceExact` for doc values from sources {es-pull}99685[#99685]
+* Fix fields API for `geo_point` fields inside other arrays {es-pull}99868[#99868] (issue: {es-issue}99781[#99781])
+* Fork response-sending in `OpenPointInTimeAction` {es-pull}99222[#99222]
+* [CI] `SearchResponseTests#testSerialization` failing resolved {es-pull}100020[#100020] (issue: {es-issue}100005[#100005])
+* fix fuzzy query rewrite parameter not work {es-pull}97642[#97642]
+
+Snapshot/Restore::
+* Fix thread context in `getRepositoryData` {es-pull}99627[#99627]
+* Frozen index input clone copy cache file {es-pull}98930[#98930]
+* Support $ and / in restore rename replacements {es-pull}99892[#99892] (issue: {es-issue}99078[#99078])
+
+TSDB::
+* Don't ignore empty index template that have no template definition {es-pull}98840[#98840] (issue: {es-issue}98834[#98834])
+* Make tsdb settings public in Serverless {es-pull}99567[#99567] (issue: {es-issue}99563[#99563])
+
+Transform::
+* Do not use PIT in the presence of remote indices in source {es-pull}99803[#99803]
+* Ignore "index not found" error when `delete_dest_index` flag is set but the dest index doesn't exist {es-pull}99738[#99738]
+* Let `_stats` internally timeout if checkpoint information can not be retrieved {es-pull}99914[#99914]
+
+Vector Search::
+* Update version range in `jvm.options` for the Panama Vector API {es-pull}99846[#99846]
+
+Watcher::
+* Treating watcher webhook response header names as case-insensitive {es-pull}99717[#99717]
+
+[[enhancement-8.11.0]]
+[float]
+=== Enhancements
+
+Aggregations::
+* Disable `FilterByFilterAggregator` through `ClusterSettings` {es-pull}99417[#99417] (issue: {es-issue}99335[#99335])
+* Represent histogram value count as long {es-pull}99912[#99912] (issue: {es-issue}99820[#99820])
+* Skip `DisiPriorityQueue` on single filter agg {es-pull}99215[#99215] (issue: {es-issue}99202[#99202])
+* Use a competitive iterator in `FiltersAggregator` {es-pull}98360[#98360] (issue: {es-issue}97544[#97544])
+
+Allocation::
+* Report a node's "roles" setting in the /_cluster/allocation/explain response {es-pull}98550[#98550] (issue: {es-issue}97859[#97859])
+
+Application::
+* Add flamegraph API {es-pull}99091[#99091]
+* [Profiling] Allow to customize the ILM policy {es-pull}99909[#99909]
+* [Profiling] Allow to wait until resources created {es-pull}99655[#99655]
+
+Audit::
+* Reduce verbosity of the bulk indexing audit log {es-pull}98470[#98470]
+
+Authentication::
+* Allow native users/roles to be disabled via setting {es-pull}98654[#98654]
+
+Authorization::
+* Add manage permission for fleet managed threat intel indices {es-pull}99231[#99231]
+
+CAT APIs::
+* Add 'dataset' size to cat indices and cat shards {es-pull}98622[#98622] (issue: {es-issue}95092[#95092])
+
+Data streams::
+* Allow explain data stream lifecycle to accept a data stream {es-pull}98811[#98811]
+
+ES|QL::
+* Add `CEIL` function {es-pull}98847[#98847]
+* Add ability to perform date math {es-pull}98870[#98870] (issue: {es-issue}98402[#98402])
+* Add support for TEXT fields in comparison operators and SORT {es-pull}98528[#98528] (issue: {es-issue}98642[#98642])
+* Compact topn {es-pull}99316[#99316]
+* Date math for negatives {es-pull}99711[#99711]
+* Enable arithmetics for durations and periods {es-pull}99432[#99432] (issue: {es-issue}99293[#99293])
+* Enhance SHOW FUNCTIONS command {es-pull}99736[#99736] (issue: {es-issue}99507[#99507])
+* Improve log messages {es-pull}99470[#99470]
+* Log execution time consistently {es-pull}99286[#99286]
+* Log query and execution time {es-pull}99058[#99058]
+* Log start and end of queries {es-pull}99746[#99746]
+* Lower the implicit limit, if none is user-provided {es-pull}99816[#99816] (issue: {es-issue}99458[#99458])
+* Mark counter fields as unsupported {es-pull}99054[#99054]
+* Serialize the source in expressions {es-pull}99956[#99956]
+* Simple check if all blocks get released {es-pull}100199[#100199]
+* Support unsigned long in sqrt and log10 for ESQL {es-pull}98711[#98711]
+* Use DEBUG log level to report ESQL execution steps {es-pull}99303[#99303]
+
+Engine::
+* Use `IndexWriter.flushNextBuffer()` to reclaim memory from indexing buffers {es-pull}94607[#94607]
+
+Health::
+* Avoiding the use of nodes that are no longer in the cluster when computing master stability {es-pull}98809[#98809] (issue: {es-issue}98636[#98636])
+* When a primary is inactive but this is considered expected, the same applies for the replica of this shard. {es-pull}99995[#99995] (issue: {es-issue}99951[#99951])
+
+Highlighting::
+* Implement matches() on `SourceConfirmedTextQuery` {es-pull}100134[#100134]
+
+Infra/Core::
+* APM Metering API {es-pull}99832[#99832]
+* Update the elastic-apm-agent version {es-pull}100064[#100064]
+* Use mappings version to retrieve system index mappings at creation time {es-pull}99555[#99555]
+
+Infra/Node Lifecycle::
+* Add links to docs from failing bootstrap checks {es-pull}99644[#99644] (issue: {es-issue}99614[#99614])
+* Chunk `SingleNodeShutdownStatus` and `ShutdownShardMigrationStatus` (and related action) response {es-pull}99798[#99798] (issue: {es-issue}99678[#99678])
+
+Infra/REST API::
+* Add `IndexVersion` to node info {es-pull}99515[#99515]
+* Add component info versions to node info in a pluggable way {es-pull}99631[#99631]
+* Return a 410 (Gone) status code for unavailable API endpoints {es-pull}97397[#97397]
+
+Ingest Node::
+* Show concrete error when enrich index not exist rather than NPE {es-pull}99604[#99604]
+
+Machine Learning::
+* Add new _inference API {es-pull}99224[#99224]
+* Adding an option for trained models to be platform specific {es-pull}99584[#99584]
+* Log warnings for jobs unassigned for a long time {es-pull}100154[#100154]
+* Simplify the Inference Ingest Processor configuration {es-pull}100205[#100205]
+
+Mapping::
+* Automatically flatten objects when subobjects:false {es-pull}97972[#97972] (issue: {es-issue}88934[#88934])
+* Explicit parsing object capabilities of `FieldMappers` {es-pull}98684[#98684] (issue: {es-issue}98537[#98537])
+* Reintroduce `sparse_vector` mapping {es-pull}98996[#98996]
+
+Network::
+* Chunk the cluster allocation explain response {es-pull}99641[#99641] (issue: {es-issue}97803[#97803])
+
+Recovery::
+* Wait for cluster state in recovery {es-pull}99193[#99193]
+
+Search::
+* Add additional counters to `_clusters` response for all Cluster search states {es-pull}99566[#99566] (issue: {es-issue}98927[#98927])
+* Add checks in term and terms queries that input terms are not too long {es-pull}99818[#99818] (issue: {es-issue}99802[#99802])
+* Adding support for exist queries to `sparse_vector` fields {es-pull}99775[#99775] (issue: {es-issue}99319[#99319])
+* Make `_index` optional for pinned query docs {es-pull}97450[#97450]
+* Optimize ContentPath#pathAsText {es-pull}98244[#98244] (issue: {es-issue}94544[#94544])
+* Reduce copying when creating scroll/PIT ids {es-pull}99219[#99219]
+* Refactor `SearchResponseClusters` to use CHM {es-pull}100129[#100129] (issue: {es-issue}99101[#99101])
+* Support cluster/details for CCS minimize_roundtrips=false {es-pull}98457[#98457]
+
+Security::
+* Support rotatating the JWT shared secret {es-pull}99278[#99278]
+
+Snapshot/Restore::
+* Remove shard data files when they fail to write for snapshot {es-pull}99694[#99694]
+
+Stats::
+* Prune unnecessary information from TransportNodesInfoAction.NodeInfoRequest {es-pull}99938[#99938] (issue: {es-issue}99744[#99744])
+
+TSDB::
+* Add `index.look_back_time` setting for tsdb data streams {es-pull}98518[#98518] (issue: {es-issue}98463[#98463])
+* Improve time-series error and documentation {es-pull}100018[#100018]
+* Trim stored fields for `_id` field in tsdb {es-pull}97409[#97409]
+
+Transform::
+* Add accessors required to recreate `TransformStats` object from the fields {es-pull}98844[#98844]
+
+Vector Search::
+* Add new max_inner_product vector similarity function {es-pull}99445[#99445]
+* Adds `nested` support for indexed `dense_vector` fields {es-pull}99532[#99532]
+* Dense vector field types are indexed by default {es-pull}98268[#98268]
+* Increase the max vector dims to 4096 {es-pull}99682[#99682]
+
+[[feature-8.11.0]]
+[float]
+=== New features
+
+Analysis::
+* Add support for Persian language stemmer {es-pull}99106[#99106] (issue: {es-issue}98911[#98911])
+
+Application::
+* Automatically map float arrays of lengths 128 - 2048 as dense_vector {es-pull}98512[#98512] (issue: {es-issue}97532[#97532])
+
+Data streams::
+* GA the data stream lifecycle {es-pull}100187[#100187]
+* GET `_data_stream` displays both ILM and DSL information {es-pull}99947[#99947]
+
+ES|QL::
+* Integrate Elasticsearch Query Language, ES|QL {es-pull}98309[#98309]
+* LEAST and GREATEST functions {es-pull}98630[#98630]
+* LEFT function {es-pull}98942[#98942]
+* LTRIM, RTRIM and fix unicode whitespace {es-pull}98590[#98590]
+* RIGHT function {es-pull}98974[#98974]
+* TopN sorting with min and max for multi-value fields {es-pull}98337[#98337]
+
+[[upgrade-8.11.0]]
+[float]
+=== Upgrades
+
+Packaging::
+* Upgrade bundled JDK to Java 21 {es-pull}99724[#99724]
+
+Search::
+* Upgrade main to Lucene 9.8.0 {es-pull}100138[#100138]
+
 

docs/reference/release-notes/highlights.asciidoc

@@ -25,13 +25,59 @@ Other versions:
 
 endif::[]
 
-// The notable-highlights tag marks entries that
-// should be featured in the Stack Installation and Upgrade Guide:
 // tag::notable-highlights[]
-// [discrete]
-// === Heading
-//
-// Description.
+
+[discrete]
+[[introducing_es_ql_new_query_language_for_flexible_iterative_analytics]]
+=== Introducing ES|QL — A new query language for flexible, iterative analytics
+As the Elastic Platform has become more widely adopted for search, security, observability, and general analytics,
+analyst users require the ability to take data-as-ingested, transform it to fit their investigative needs
+post-ingestion, and derive insights from underlying Elasticsearch index data. They need a concise, integrated,
+and efficient workflow enabled by rich and expressive queries where search, filter, aggregation,
+and transformation are performed via a single query expression with little-to-no UI context switching.
+
+To solve these challenges, the Elastic team is developing the <<esql,Elasticsearch Query Language (ES|QL)>>.
+ES|QL provides Elastic users with a flexible, powerful, and robust query expression language to interrogate data.
+ES|QL also provides a superior query UX with post-ingest processing capabilities that fundamentally transforms
+and expands the analytics and data processing capabilities of Elasticsearch.
+
+ES|QL introduces distributed compute capabilities to users in disparate roles and with varying skill levels.
+These compute capabilities enable ES|QL to simplify user workflows in several key ways.
+
+* Utilize a superior query UX: ES|QL query expressions support complex analytics and data processing.
+  They are easy to learn, read, and share.
+* Use the filter, aggregation, and transformation capabilities of Elasticsearch with subqueries and lookups,
+  made possible by new Elasticsearch compute and data processing capabilities.
+* Use ES|QL across Kibana in Discover, Kibana Lens, and Elastic Solutions, giving you seamless workflows.
+  You will be able to visualize ES|QL queries, share them with teams on dashboards or as queries,
+  and use queries to create custom alerts.
+
+[role="screenshot"]
+image:images/esql/esql_kibana_discover_logs.png[ES|QL usage in Kibana Discover for Analysing Proxy Logs]
+
+{es-pull}98309[#98309]
+
+[discrete]
+[[data_stream_lifecycle_in_technical_preview]]
+=== The data stream lifecycle is now in Technical Preview
+This marks the data stream lifecycle as available in Technical Preview. Data streams will be able to take advantage of a built-in simplified and resilient lifecycle implementation. Data streams with a configured lifecycle will be automatically rolled over and tail merged (a forcemerge implementation that's lightweight and only merges the long tail of small segments instead of the whole shard). With the shard and index maintenance tasks being handled automatically to ensure optimum performance, and trade-off between indexing and searching, you'll be able to focus on the business related lifecycle aspects like data retention.
+
+{es-pull}100187[#100187]
+
 // end::notable-highlights[]
 
 
+[discrete]
+[[use_indexwriter_flushnextbuffer_to_reclaim_memory_from_indexing_buffers]]
+=== Use `IndexWriter.flushNextBuffer()` to reclaim memory from indexing buffers
+Rather than forcing a refresh to reclaim memory from indexing buffers, which flushes all
+segments no matter how large, Elasticsearch now takes advantage of
+`IndexWriter#flushNextBuffer` which only flushes the largest pending segment. This should smooth
+out indexing allowing for larger segment sizes, fewer merges and higher throughput.
+
+Furthermore, the selection algorithm to pick which shard to reclaim memory from next was
+changed, from picking the shard that uses the most RAM to going over shards in a round-robin
+fashion. This approach has proved to work significantly better in practice.
+
+{es-pull}94607[#94607]
+