From 5a2b61814540026dced477513649bf4781369e0e Mon Sep 17 00:00:00 2001 From: Matt Culbreth Date: Tue, 7 Nov 2023 17:16:41 -0500 Subject: [PATCH] Forward port 8.11.0 docs (#101885) --- .../reference/migration/migrate_8_11.asciidoc | 55 ++- docs/reference/release-notes/8.11.0.asciidoc | 333 +++++++++++++++++- 2 files changed, 383 insertions(+), 5 deletions(-) diff --git a/docs/reference/migration/migrate_8_11.asciidoc b/docs/reference/migration/migrate_8_11.asciidoc index a353d1a6a87fa..098456e1aca42 100644 --- a/docs/reference/migration/migrate_8_11.asciidoc +++ b/docs/reference/migration/migrate_8_11.asciidoc @@ -9,12 +9,61 @@ your application to {es} 8.11. See also <> and <>. -coming::[8.11.0] - [discrete] [[breaking-changes-8.11]] === Breaking changes -There are no breaking changes in {es} 8.11. +The following changes in {es} 8.11 might affect your applications +and prevent them from operating normally. +Before upgrading to 8.11, review these changes and take the described steps +to mitigate the impact. + + +There are no notable breaking changes in {es} 8.11. +But there are some less critical breaking changes. + +[discrete] +[[breaking_811_rest_api_changes]] +==== REST API changes + +[[remove_transport_versions_from_cluster_state_api]] +.Remove `transport_versions` from cluster state API +[%collapsible] +==== +*Details* + +The `transport_versions` subobject of the response to `GET _cluster/state` has been replaced by the `nodes_versions` subobject. + +*Impact* + +If needed, retrieve the per-node transport versions from the `nodes_versions` subobject. +==== + + +[discrete] +[[deprecated-8.11]] +=== Deprecations + +The following functionality has been deprecated in {es} 8.11 +and will be removed in a future version. +While this won't have an immediate impact on your applications, +we strongly encourage you to take the described steps to update your code +after upgrading to 8.11. + +To find out if you are using any deprecated functionality, +enable <>. + +[discrete] +[[deprecations_811_rollup]] +==== Rollup deprecations + +[[rollup_functionality_deprecated]] +.Rollup functionality is now deprecated +[%collapsible] +==== +*Details* + +<> has been deprecated and will be removed in a future release. Previously, rollups were available in technical preview. + +*Impact* + +Use <> to reduce storage costs for time series data by storing it at reduced granularity. +==== diff --git a/docs/reference/release-notes/8.11.0.asciidoc b/docs/reference/release-notes/8.11.0.asciidoc index a5c8c8c73128e..08ddaf5667845 100644 --- a/docs/reference/release-notes/8.11.0.asciidoc +++ b/docs/reference/release-notes/8.11.0.asciidoc @@ -1,8 +1,337 @@ [[release-notes-8.11.0]] == {es} version 8.11.0 -coming[8.11.0] - Also see <>. +[[breaking-8.11.0]] +[float] +=== Breaking changes + +Infra/Core:: +* Remove `transport_versions` from cluster state API {es-pull}99223[#99223] + +[[bug-8.11.0]] +[float] +=== Bug fixes + +Aggregations:: +* Adjust `DateHistogram's` bucket accounting to be iteratively {es-pull}101012[#101012] +* Allow parsing on non-string routing fields {es-pull}97729[#97729] +* Support runtime fields in synthetic source {es-pull}99796[#99796] (issue: {es-issue}98287[#98287]) + +Allocation:: +* Consider node shutdown in `DataTierAllocationDecider` {es-pull}98824[#98824] (issue: {es-issue}97207[#97207]) + +Application:: +* Align look-back with client-side cache {es-pull}101264[#101264] +* Increase K/V look-back time interval {es-pull}101205[#101205] +* Provide stable resampling {es-pull}101255[#101255] +* [Profiling] Tighten resource creation check {es-pull}99873[#99873] + +Authorization:: +* Allow `enrich_user` to read/view enrich indices {es-pull}100707[#100707] +* Grant editor and viewer access to profiling {es-pull}100594[#100594] + +CCR:: +* CCR: Use local cluster state request {es-pull}100323[#100323] + +CRUD:: +* Change `GetFromTranslog` to indices action {es-pull}99300[#99300] +* Wait for cluster to recover before resolving index template {es-pull}99797[#99797] + +Cluster Coordination:: +* Reset `GatewayService` flags before reroute {es-pull}98653[#98653] (issue: {es-issue}98606[#98606]) + +Data streams:: +* DSL waits for the tsdb time boundaries to lapse {es-pull}100470[#100470] (issue: {es-issue}99696[#99696]) +* Propagate cancellation in `DataTiersUsageTransportAction` {es-pull}100253[#100253] +* [DSL] skip deleting indices that have in-progress downsampling operations {es-pull}101495[#101495] + +Downsampling:: +* Make downsample target index replicas configurable {es-pull}99712[#99712] + +ES|QL:: +* "params" correctly parses the values including an optional "type" {es-pull}99310[#99310] (issue: {es-issue}99294[#99294]) +* Account for an exception being thrown when building a `BytesRefArrayBlock` {es-pull}99726[#99726] (issue: {es-issue}99472[#99472]) +* Add arithmetic operators {es-pull}98628[#98628] +* Add identity check in Block equality {es-pull}100377[#100377] (issue: {es-issue}100374[#100374]) +* Adds Enrich implicit `match_fields` to `field_caps` call {es-pull}101456[#101456] (issue: {es-issue}101328[#101328]) +* Better management of not stored TEXT fiels with synthetic source {es-pull}99695[#99695] +* Continue resolving attributes for Eval {es-pull}99601[#99601] (issue: {es-issue}99576[#99576]) +* Create a Vector when needed for IN {es-pull}99382[#99382] (issue: {es-issue}99347[#99347]) +* ESQL: Fix unreleased block in topn {es-pull}101648[#101648] (issue: {es-issue}101588[#101588]) +* ESQL: check type before casting {es-pull}101492[#101492] (issue: {es-issue}101489[#101489]) +* Fix NPE when aggregating literals {es-pull}99827[#99827] +* Fix escaping of backslash in LIKE operator {es-pull}101120[#101120] (issue: {es-issue}101106[#101106]) +* Fix eval of functions on foldable literals {es-pull}101438[#101438] (issue: {es-issue}101425[#101425]) +* Fix non-null value being returned for unsupported data types in `ValueSources` {es-pull}100656[#100656] (issue: {es-issue}100048[#100048]) +* Graceful handling of non-bool condition in the filter {es-pull}100645[#100645] (issues: {es-issue}100049[#100049], {es-issue}100409[#100409]) +* Handle queries with non-existing enrich policies and no field {es-pull}100647[#100647] (issue: {es-issue}100593[#100593]) +* Implement serialization of `InvalidMappedField` {es-pull}98972[#98972] (issue: {es-issue}98851[#98851]) +* Improve verifier error for incorrect agg declaration {es-pull}100650[#100650] (issue: {es-issue}100641[#100641]) +* Limit how many bytes `concat()` can process {es-pull}100360[#100360] +* Make DISSECT parameter `append_separator` case insensitive {es-pull}101358[#101358] (issue: {es-issue}101138[#101138]) +* Page shouldn't close a block twice {es-pull}100370[#100370] (issues: {es-issue}100356[#100356], {es-issue}100365[#100365]) +* Preserve intermediate aggregation output in local relation {es-pull}100866[#100866] (issue: {es-issue}100807[#100807]) +* Properly handle multi-values in fold() and date math {es-pull}100766[#100766] (issue: {es-issue}100497[#100497]) +* Remove aliasing inside Eval {es-pull}100238[#100238] (issue: {es-issue}100174[#100174]) +* Resilience to non-indexed fields {es-pull}99588[#99588] (issue: {es-issue}99506[#99506]) +* Skip synthetic attributes when planning the physical fragment {es-pull}99188[#99188] (issue: {es-issue}99170[#99170]) +* Support date and time intervals as input params {es-pull}101001[#101001] (issue: {es-issue}99570[#99570]) +* Support queries that don't return underlying fields {es-pull}98759[#98759] (issue: {es-issue}98404[#98404]) +* Use exact attributes for data source extraction {es-pull}99874[#99874] (issue: {es-issue}99183[#99183]) +* `mv_expand` pushes down limit and project and keep the limit after it untouched {es-pull}100782[#100782] (issues: {es-issue}99971[#99971], {es-issue}100774[#100774]) +* support metric tsdb fields while querying index patterns {es-pull}100351[#100351] (issue: {es-issue}100144[#100144]) + +Geo:: +* Use `NamedWritable` to enable `GeoBoundingBox` serialisation {es-pull}99163[#99163] (issue: {es-issue}99089[#99089]) + +Health:: +* Fix NPE in `StableMasterHealthIndicatorService` {es-pull}98635[#98635] +* Health report infrastructure doesn't trip the circuit breakers {es-pull}101629[#101629] +* Propagate cancellation in `GetHealthAction` {es-pull}100273[#100273] + +Highlighting:: +* Correctly handle `ScriptScoreQuery` in plain highlighter {es-pull}99804[#99804] (issue: {es-issue}99700[#99700]) +* Disable `weight_matches` when kNN query is present {es-pull}101713[#101713] + +ILM+SLM:: +* Compute SLM retention from `RepositoryData` {es-pull}100092[#100092] (issue: {es-issue}99953[#99953]) +* `WaitForSnapshotStep` verifies if the index belongs to the latest snapshot of that SLM policy {es-pull}100911[#100911] + +Infra/Core:: +* Add `java.net.NetPermission` to APM module's permissions {es-pull}99474[#99474] +* Don't update system index mappings in mixed clusters {es-pull}101778[#101778] (issues: {es-issue}101331[#101331], {es-issue}99778[#99778]) +* Revert "Kibana system index does not allow user templates to affect it" {es-pull}98888[#98888] +* Specify correct current `IndexVersion` after 8.10 release {es-pull}98574[#98574] (issue: {es-issue}98555[#98555]) +* Tracing: Use `doPriv` when working with spans, use `SpanId` {es-pull}100232[#100232] + +Infra/Scripting:: +* Improve painless error wrapping {es-pull}100872[#100872] + +Ingest Node:: +* Improving tika handling {es-pull}101486[#101486] +* Update enrich execution to only set index false on fields that support it {es-pull}98038[#98038] (issue: {es-issue}98019[#98019]) + +Machine Learning:: +* Avoid risk of OOM in datafeeds when memory is constrained {es-pull}98915[#98915] (issue: {es-issue}89769[#89769]) +* Fix for inference requests being sent to every node with a model allocation. If there are more nodes than items in the original request then empty requests were sent. {es-pull}100388[#100388] (issue: {es-issue}100180[#100180]) +* Preserve order of inference results when calling the _infer API with multiple inputs on a model deployment with more than one allocation the output results order was not guaranteed to match the input order. The fix ensures the output order matches the input order. {es-pull}100143[#100143] +* Remove noisy 'Could not find trained model' message {es-pull}100760[#100760] +* Safely drain deployment request queues before allowing node to shutdown {es-pull}98406[#98406] +* Use the correct writable name for model assignment metadata in mixed version clusters. Prevents a node failure due to IllegalArgumentException Unknown NamedWriteable [trained_model_assignment] {es-pull}100886[#100886] +* Wait to gracefully stop deployments until alternative allocation exists {es-pull}99107[#99107] + +Mapping:: +* Automatically disable `ignore_malformed` on datastream `@timestamp` fields {es-pull}99346[#99346] +* Correct behaviour of `ContentPath::remove()` {es-pull}98332[#98332] (issue: {es-issue}98327[#98327]) +* Fix merges of mappings with `subobjects: false` for composable index templates {es-pull}97317[#97317] (issue: {es-issue}96768[#96768]) +* Percolator to support parsing script score query with params {es-pull}101051[#101051] (issue: {es-issue}97377[#97377]) + +Network:: +* Do not report failure after connections are made {es-pull}99117[#99117] + +Percolator:: +* Fix percolator query for stored queries that expand on wildcard field names {es-pull}98878[#98878] + +Query Languages:: +* Preserve subfields for unsupported types {es-pull}100875[#100875] (issue: {es-issue}100869[#100869]) + +Recovery:: +* Fix interruption of `markAllocationIdAsInSync` {es-pull}100610[#100610] (issues: {es-issue}96578[#96578], {es-issue}100589[#100589]) + +Search:: +* Consistent scores for multi-term `SourceConfirmedTestQuery` {es-pull}100846[#100846] (issue: {es-issue}98712[#98712]) +* Fix UnsignedLong field range query gt "0" can get the result equal to 0 {es-pull}98843[#98843] +* Fix `advanceExact` for doc values from sources {es-pull}99685[#99685] +* Fork response-sending in `OpenPointInTimeAction` {es-pull}99222[#99222] +* [CI] `SearchResponseTests#testSerialization` failing resolved {es-pull}100020[#100020] (issue: {es-issue}100005[#100005]) +* fix fuzzy query rewrite parameter not work {es-pull}97642[#97642] + +Security:: +* Fix NullPointerException in RotableSecret {es-pull}100779[#100779] (issue: {es-issue}99759[#99759]) + +Snapshot/Restore:: +* Fix race condition in `SnapshotsService` {es-pull}101652[#101652] +* Fix snapshot double finalization {es-pull}101497[#101497] +* Fix thread context in `getRepositoryData` {es-pull}99627[#99627] +* Frozen index input clone copy cache file {es-pull}98930[#98930] +* Make S3 anti-contention delay configurable {es-pull}101245[#101245] +* More robust timeout for repo analysis {es-pull}101184[#101184] (issue: {es-issue}101182[#101182]) +* Register `repository_s3` settings {es-pull}101344[#101344] +* Reinstate `RepositoryData` BwC {es-pull}100447[#100447] + +TSDB:: +* Don't ignore empty index template that have no template definition {es-pull}98840[#98840] (issue: {es-issue}98834[#98834]) +* Fix painless execute api and tsdb issue {es-pull}101212[#101212] (issue: {es-issue}101072[#101072]) +* Make tsdb settings public in Serverless {es-pull}99567[#99567] (issue: {es-issue}99563[#99563]) + +Transform:: +* Fix possible NPE when getting transform stats for failed transforms {es-pull}98061[#98061] (issue: {es-issue}98052[#98052]) +* Ignore `IndexNotFound` error when refreshing destination index {es-pull}101627[#101627] +* Make Transform Feature Reset really wait for all the tasks {es-pull}100624[#100624] +* Make tasks that calculate checkpoints cancellable {es-pull}100808[#100808] + +Watcher:: +* Treating watcher webhook response header names as case-insensitive {es-pull}99717[#99717] + +[[deprecation-8.11.0]] +[float] +=== Deprecations + +Rollup:: +* Rollup functionality is now deprecated {es-pull}101265[#101265] + +[[enhancement-8.11.0]] +[float] +=== Enhancements + +Aggregations:: +* Disable `FilterByFilterAggregator` through `ClusterSettings` {es-pull}99417[#99417] (issue: {es-issue}99335[#99335]) +* Represent histogram value count as long {es-pull}99912[#99912] (issue: {es-issue}99820[#99820]) +* Skip `DisiPriorityQueue` on single filter agg {es-pull}99215[#99215] (issue: {es-issue}99202[#99202]) +* Use a competitive iterator in `FiltersAggregator` {es-pull}98360[#98360] (issue: {es-issue}97544[#97544]) + +Allocation:: +* Report a node's "roles" setting in the /_cluster/allocation/explain response {es-pull}98550[#98550] (issue: {es-issue}97859[#97859]) + +Application:: +* Add flamegraph API {es-pull}99091[#99091] +* [Profiling] Allow to customize the ILM policy {es-pull}99909[#99909] +* [Profiling] Allow to wait until resources created {es-pull}99655[#99655] + +Audit:: +* Reduce verbosity of the bulk indexing audit log {es-pull}98470[#98470] + +Authentication:: +* Allow native users/roles to be disabled via setting {es-pull}98654[#98654] + +CAT APIs:: +* Add 'dataset' size to cat indices and cat shards {es-pull}98622[#98622] (issue: {es-issue}95092[#95092]) + +Data streams:: +* Allow explain data stream lifecycle to accept a data stream {es-pull}98811[#98811] + +ES|QL:: +* Add `CEIL` function {es-pull}98847[#98847] +* Add ability to perform date math {es-pull}98870[#98870] (issue: {es-issue}98402[#98402]) +* Add support for TEXT fields in comparison operators and SORT {es-pull}98528[#98528] (issue: {es-issue}98642[#98642]) +* Compact topn {es-pull}99316[#99316] +* Date math for negatives {es-pull}99711[#99711] +* Enable arithmetics for durations and periods {es-pull}99432[#99432] (issue: {es-issue}99293[#99293]) +* Enhance SHOW FUNCTIONS command {es-pull}99736[#99736] (issue: {es-issue}99507[#99507]) +* Improve log messages {es-pull}99470[#99470] +* Log execution time consistently {es-pull}99286[#99286] +* Log query and execution time {es-pull}99058[#99058] +* Log start and end of queries {es-pull}99746[#99746] +* Lower the implicit limit, if none is user-provided {es-pull}99816[#99816] (issue: {es-issue}99458[#99458]) +* Make settings dynamic {es-pull}101516[#101516] +* Mark counter fields as unsupported {es-pull}99054[#99054] +* Remove the swapped-args check for date_xxx() {es-pull}101362[#101362] (issue: {es-issue}99562[#99562]) +* Serialize the source in expressions {es-pull}99956[#99956] +* Simple check if all blocks get released {es-pull}100199[#100199] +* Support unsigned long in sqrt and log10 {es-pull}98711[#98711] +* Use DEBUG log level to report execution steps {es-pull}99303[#99303] + +Engine:: +* Use `IndexWriter.flushNextBuffer()` to reclaim memory from indexing buffers {es-pull}94607[#94607] + +Health:: +* Avoiding the use of nodes that are no longer in the cluster when computing master stability {es-pull}98809[#98809] (issue: {es-issue}98636[#98636]) +* When a primary is inactive but this is considered expected, the same applies for the replica of this shard. {es-pull}99995[#99995] (issue: {es-issue}99951[#99951]) + +Infra/Core:: +* APM Metering API {es-pull}99832[#99832] +* Update the elastic-apm-agent version {es-pull}100064[#100064] +* Use mappings version to retrieve system index mappings at creation time {es-pull}99555[#99555] + +Infra/Node Lifecycle:: +* Add links to docs from failing bootstrap checks {es-pull}99644[#99644] (issue: {es-issue}99614[#99614]) +* Chunk `SingleNodeShutdownStatus` and `ShutdownShardMigrationStatus` (and related action) response {es-pull}99798[#99798] (issue: {es-issue}99678[#99678]) + +Infra/REST API:: +* Add `IndexVersion` to node info {es-pull}99515[#99515] +* Add component info versions to node info in a pluggable way {es-pull}99631[#99631] +* Return a 410 (Gone) status code for unavailable API endpoints {es-pull}97397[#97397] + +Machine Learning:: +* Add new _inference API {es-pull}99224[#99224] +* Adding an option for trained models to be platform specific {es-pull}99584[#99584] +* Log warnings for jobs unassigned for a long time {es-pull}100154[#100154] +* Simplify the Inference Ingest Processor configuration {es-pull}100205[#100205] + +Mapping:: +* Automatically flatten objects when subobjects:false {es-pull}97972[#97972] (issue: {es-issue}88934[#88934]) +* Explicit parsing object capabilities of `FieldMappers` {es-pull}98684[#98684] (issue: {es-issue}98537[#98537]) +* Reintroduce `sparse_vector` mapping {es-pull}98996[#98996] + +Network:: +* Chunk the cluster allocation explain response {es-pull}99641[#99641] (issue: {es-issue}97803[#97803]) + +Recovery:: +* Wait for cluster state in recovery {es-pull}99193[#99193] + +Search:: +* Add additional counters to `_clusters` response for all Cluster search states {es-pull}99566[#99566] (issue: {es-issue}98927[#98927]) +* Adding support for exist queries to `sparse_vector` fields {es-pull}99775[#99775] (issue: {es-issue}99319[#99319]) +* Make `_index` optional for pinned query docs {es-pull}97450[#97450] +* Reduce copying when creating scroll/PIT ids {es-pull}99219[#99219] +* Refactor `SearchResponseClusters` to use CHM {es-pull}100129[#100129] (issue: {es-issue}99101[#99101]) +* Support cluster/details for CCS minimize_roundtrips=false {es-pull}98457[#98457] + +Security:: +* Support rotatating the JWT shared secret {es-pull}99278[#99278] + +Snapshot/Restore:: +* Remove shard data files when they fail to write for snapshot {es-pull}99694[#99694] + +Stats:: +* Prune unnecessary information from TransportNodesInfoAction.NodeInfoRequest {es-pull}99938[#99938] (issue: {es-issue}99744[#99744]) + +TSDB:: +* Add `index.look_back_time` setting for tsdb data streams {es-pull}98518[#98518] (issue: {es-issue}98463[#98463]) +* Improve time-series error and documentation {es-pull}100018[#100018] +* Trim stored fields for `_id` field in tsdb {es-pull}97409[#97409] + +Transform:: +* Add accessors required to recreate `TransformStats` object from the fields {es-pull}98844[#98844] + +Vector Search:: +* Add new max_inner_product vector similarity function {es-pull}99445[#99445] +* Adds `nested` support for indexed `dense_vector` fields {es-pull}99532[#99532] +* Dense vector field types are indexed by default {es-pull}98268[#98268] +* Increase the max vector dims to 4096 {es-pull}99682[#99682] + +[[feature-8.11.0]] +[float] +=== New features + +Analysis:: +* Add support for Persian language stemmer {es-pull}99106[#99106] (issue: {es-issue}98911[#98911]) + +Application:: +* Automatically map float arrays of lengths 128 - 2048 as dense_vector {es-pull}98512[#98512] (issue: {es-issue}97532[#97532]) + +Data streams:: +* GA the data stream lifecycle {es-pull}100187[#100187] +* GET `_data_stream` displays both ILM and DSL information {es-pull}99947[#99947] + +ES|QL:: +* Integrate Elasticsearch Query Language, ES|QL {es-pull}98309[#98309] +* LEAST and GREATEST functions {es-pull}98630[#98630] +* LEFT function {es-pull}98942[#98942] +* LTRIM, RTRIM and fix unicode whitespace {es-pull}98590[#98590] +* RIGHT function {es-pull}98974[#98974] +* TopN sorting with min and max for multi-value fields {es-pull}98337[#98337] + +[[upgrade-8.11.0]] +[float] +=== Upgrades + +Packaging:: +* Update bundled JDK to 21.0.1 {es-pull}101133[#101133] + +Search:: +* Upgrade main to Lucene 9.8.0 {es-pull}100138[#100138] +