diff --git a/docs/reference/migration/migrate_8_14.asciidoc b/docs/reference/migration/migrate_8_14.asciidoc index 2e6cd439ebed0..bdff8bef246b5 100644 --- a/docs/reference/migration/migrate_8_14.asciidoc +++ b/docs/reference/migration/migrate_8_14.asciidoc @@ -21,25 +21,8 @@ and prevent them from operating normally. Before upgrading to 8.14, review these changes and take the described steps to mitigate the impact. - -There are no notable breaking changes in {es} 8.14. -But there are some less critical breaking changes. - [discrete] -[[breaking_814_rest_api_changes]] -==== REST API changes - -[[prevent_dls_fls_if_replication_assigned]] -.Prevent DLS/FLS if `replication` is assigned -[%collapsible] -==== -*Details* + -For cross-cluster API keys, {es} no longer allows specifying document-level security (DLS) or field-level security (FLS) in the `search` field, if `replication` is also specified. {es} likewise blocks the use of any existing cross-cluster API keys that meet this condition. - -*Impact* + -Remove any document-level security (DLS) or field-level security (FLS) definitions from the `search` field for cross-cluster API keys that also have a `replication` field, or create two separate cross-cluster API keys, one for search and one for replication. -==== - +[[breaking-changes-8.14-0]] [discrete] [[breaking_814_dls_changes]] @@ -58,33 +41,3 @@ When Document Level Security (DLS) is applied to the validate query API with the *Impact* + If needed, test workflows with DLS enabled to ensure that the stricter security rules do not impact your application. ==== - - -[discrete] -[[deprecated-8.14]] -=== Deprecations - -The following functionality has been deprecated in {es} 8.14 -and will be removed in a future version. -While this won't have an immediate impact on your applications, -we strongly encourage you to take the described steps to update your code -after upgrading to 8.14. - -To find out if you are using any deprecated functionality, -enable <>. - -[discrete] -[[deprecations_814_mapping]] -==== Mapping deprecations - -[[deprecate_allowing_fields_in_scenarios_where_it_ignored]] -.Deprecate allowing `fields` in scenarios where it is ignored -[%collapsible] -==== -*Details* + -The following mapped types have always ignored `fields` when using multi-fields. This deprecation makes this clearer and we will completely disallow `fields` for these mapped types in the future. - -*Impact* + -In the future, `join`, `aggregate_metric_double`, and `constant_keyword`, will all disallow supplying `fields` as a parameter in the mapping. -==== - diff --git a/docs/reference/release-notes/8.14.0.asciidoc b/docs/reference/release-notes/8.14.0.asciidoc index 58a5dae73997d..a203c983927cd 100644 --- a/docs/reference/release-notes/8.14.0.asciidoc +++ b/docs/reference/release-notes/8.14.0.asciidoc @@ -5,348 +5,4 @@ coming[8.14.0] Also see <>. -[[breaking-8.14.0]] -[float] -=== Breaking changes - -Security:: -* Prevent DLS/FLS if `replication` is assigned {es-pull}108600[#108600] -* Apply stricter Document Level Security (DLS) rules for the validate query API with the rewrite parameter {es-pull}105709[#105709] -* Apply stricter Document Level Security (DLS) rules for terms aggregations when min_doc_count is set to 0 {es-pull}105714[#105714] - -[[bug-8.14.0]] -[float] -=== Bug fixes - -Aggregations:: -* Cross check livedocs for terms aggs when index access control list is non-null {es-pull}105714[#105714] -* ESQL: Enable VALUES agg for datetime {es-pull}107016[#107016] -* Fix IOOBE in TTest aggregation when using filters {es-pull}109034[#109034] -* Validate stats formatting in standard `InternalStats` constructor {es-pull}107678[#107678] (issue: {es-issue}107671[#107671]) - -Application:: -* [Bugfix] Connector API - fix status serialisation issue in termquery {es-pull}108365[#108365] -* [Connector API] Fix bug with filtering validation toXContent {es-pull}107467[#107467] -* [Connector API] Fix bug with parsing *_doc_count nullable fields {es-pull}108854[#108854] -* [Connector API] Fix bug with with wrong target index for access control sync {es-pull}109097[#109097] - -Authorization:: -* Users with monitor privileges can access async_search/status endpoint even when setting keep_alive {es-pull}107383[#107383] - -CAT APIs:: -* Fix numeric sorts in `_cat/nodes` {es-pull}106189[#106189] (issue: {es-issue}48070[#48070]) - -CCR:: -* Add ?master_timeout query parameter to ccr apis {es-pull}105168[#105168] - -CRUD:: -* Fix `noop_update_total` is not being updated when using the `_bulk` {es-pull}105745[#105745] (issue: {es-issue}105742[#105742]) -* Use correct system index bulk executor {es-pull}106150[#106150] - -Cluster Coordination:: -* Fix support for infinite `?master_timeout` {es-pull}107050[#107050] - -Data streams:: -* Add non-indexed fields to ecs templates {es-pull}106714[#106714] -* Fix bulk NPE when retrying failure redirect after cluster block {es-pull}107598[#107598] -* Improve error message when rolling over DS alias {es-pull}106708[#106708] (issue: {es-issue}106137[#106137]) -* Only skip deleting a downsampled index if downsampling is in progress as part of DSL retention {es-pull}109020[#109020] - -Downsampling:: -* Fix downsample action request serialization {es-pull}106919[#106919] (issue: {es-issue}106917[#106917]) - -EQL:: -* Use #addWithoutBreaking when adding a negative number of bytes to the circuit breaker in `SequenceMatcher` {es-pull}107655[#107655] - -ES|QL:: -* ESQL: Allow reusing BUCKET grouping expressions in aggs {es-pull}107578[#107578] -* ESQL: Disable quoting in FROM command {es-pull}108431[#108431] -* ESQL: Fix MV_DEDUPE when using data from an index {es-pull}107577[#107577] (issue: {es-issue}104745[#104745]) -* ESQL: Fix error message when failing to resolve aggregate groupings {es-pull}108101[#108101] (issue: {es-issue}108053[#108053]) -* ESQL: Fix treating all fields as MV in COUNT pushdown {es-pull}106720[#106720] -* ESQL: Re-enable logical dependency check {es-pull}105860[#105860] -* ESQL: median, count and `count_distinct` over constants {es-pull}107414[#107414] (issues: {es-issue}105248[#105248], {es-issue}104900[#104900]) -* ES|QL fix no-length substring with supplementary (4-byte) character {es-pull}107183[#107183] -* ES|QL: Fix usage of IN operator with TEXT fields {es-pull}106654[#106654] (issue: {es-issue}105379[#105379]) -* ES|QL: Improve support for TEXT fields in functions {es-pull}106810[#106810] -* Fix docs generation of signatures for variadic functions {es-pull}107865[#107865] -* [ESQL] Mark `date_diff` as requiring all three arguments {es-pull}108834[#108834] (issue: {es-issue}108383[#108383]) - -Health:: -* Don't stop checking if the `HealthNode` persistent task is present {es-pull}105449[#105449] (issue: {es-issue}98926[#98926]) -* Health monitor concurrency fixes {es-pull}105674[#105674] (issue: {es-issue}105065[#105065]) - -Highlighting:: -* Check preTags and postTags params for empty values {es-pull}106396[#106396] (issue: {es-issue}69009[#69009]) -* added fix for inconsistent text trimming in Unified Highlighter {es-pull}99961[#99961] (issue: {es-issue}101803[#101803]) - -Infra/CLI:: -* Workaround G1 bug for JDK 22 and 22.0.1 {es-pull}108571[#108571] - -Infra/Core:: -* Add a check for the same feature being declared regular and historical {es-pull}106285[#106285] -* Fix `AffixSetting.exists` to include secure settings {es-pull}106745[#106745] -* Fix regression in get index settings (human=true) where the version was not displayed in human-readable format {es-pull}107447[#107447] -* Nativeaccess: try to load all located libsystemds {es-pull}108238[#108238] (issue: {es-issue}107878[#107878]) -* Update several references to `IndexVersion.toString` to use `toReleaseVersion` {es-pull}107828[#107828] (issue: {es-issue}107821[#107821]) -* Update several references to `TransportVersion.toString` to use `toReleaseVersion` {es-pull}107902[#107902] - -Infra/Logging:: -* Log when update AffixSetting using addAffixMapUpdateConsumer {es-pull}97072[#97072] - -Infra/Node Lifecycle:: -* Consider `ShardRouting` roles when calculating shard copies in shutdown status {es-pull}106063[#106063] -* Wait indefintely for http connections on shutdown by default {es-pull}106511[#106511] - -Infra/Scripting:: -* Guard against a null scorer in painless execute {es-pull}109048[#109048] (issue: {es-issue}43541[#43541]) -* Painless: Apply true regex limit factor with FIND and MATCH operation {es-pull}105670[#105670] - -Ingest Node:: -* Catching `StackOverflowErrors` from bad regexes in `GsubProcessor` {es-pull}106851[#106851] -* Fix `uri_parts` processor behaviour for missing extensions {es-pull}105689[#105689] (issue: {es-issue}105612[#105612]) -* Remove leading is_ prefix from Enterprise geoip docs {es-pull}108518[#108518] -* Slightly better geoip `databaseType` validation {es-pull}106889[#106889] - -License:: -* Fix lingering license warning header {es-pull}108031[#108031] (issue: {es-issue}107573[#107573]) - -Machine Learning:: -* Fix NPE in ML assignment notifier {es-pull}107312[#107312] -* Fix `startOffset` must be non-negative error in XLMRoBERTa tokenizer {es-pull}107891[#107891] (issue: {es-issue}104626[#104626]) -* Fix the position of spike, dip and distribution changes bucket when the sibling aggregation includes empty buckets {es-pull}106472[#106472] -* Make OpenAI embeddings parser more flexible {es-pull}106808[#106808] - -Mapping:: -* Dedupe terms in terms queries {es-pull}106381[#106381] -* Extend support of `allowedFields` to `getMatchingFieldNames` and `getAllFields` {es-pull}106862[#106862] -* Fix for raw mapping merge of fields named "properties" {es-pull}108867[#108867] (issue: {es-issue}108866[#108866]) -* Handle infinity during synthetic source construction for scaled float field {es-pull}107494[#107494] (issue: {es-issue}107101[#107101]) -* Handle pass-through subfields with deep nesting {es-pull}106767[#106767] -* Wrap "Pattern too complex" exception into an `IllegalArgumentException` {es-pull}109173[#109173] - -Network:: -* Fix HTTP corner-case response leaks {es-pull}105617[#105617] - -Search:: -* Add `internalClusterTest` for and fix leak in `ExpandSearchPhase` {es-pull}108562[#108562] (issue: {es-issue}108369[#108369]) -* Avoid attempting to load the same empty field twice in fetch phase {es-pull}107551[#107551] -* Bugfix: Disable eager loading `BitSetFilterCache` on Indexing Nodes {es-pull}105791[#105791] -* Cross-cluster painless/execute actions should check permissions only on target remote cluster {es-pull}105360[#105360] -* Fix error 500 on invalid `ParentIdQuery` {es-pull}105693[#105693] (issue: {es-issue}105366[#105366]) -* Fix range queries for float/half_float fields when bounds are out of type's range {es-pull}106691[#106691] -* Fixing NPE when requesting [_none_] for `stored_fields` {es-pull}104711[#104711] -* Fork when handling remote field-caps responses {es-pull}107370[#107370] -* Handle parallel calls to `createWeight` when profiling is on {es-pull}108041[#108041] (issues: {es-issue}104131[#104131], {es-issue}104235[#104235]) -* Harden field-caps request dispatcher {es-pull}108736[#108736] -* Replace `UnsupportedOperationException` with `IllegalArgumentException` for non-existing columns {es-pull}107038[#107038] -* Unable to retrieve multiple stored field values {es-pull}106575[#106575] -* Validate `model_id` is required when using the `learning_to_rank` rescorer {es-pull}107743[#107743] - -Security:: -* Disable validate when rewrite parameter is sent and the index access control list is non-null {es-pull}105709[#105709] -* Fix field caps and field level security {es-pull}106731[#106731] - -Snapshot/Restore:: -* Fix double-pausing shard snapshot {es-pull}109148[#109148] (issue: {es-issue}109143[#109143]) -* Treat 404 as empty register in `AzureBlobStore` {es-pull}108900[#108900] (issue: {es-issue}108504[#108504]) -* `SharedBlobCacheService.maybeFetchRegion` should use `computeCacheFileRegionSize` {es-pull}106685[#106685] - -TSDB:: -* Flip dynamic mapping condition when create tsid {es-pull}105636[#105636] - -Transform:: -* Consolidate permissions checks {es-pull}106413[#106413] (issue: {es-issue}105794[#105794]) -* Disable PIT for remote clusters {es-pull}107969[#107969] -* Make force-stopping the transform always remove persistent task from cluster state {es-pull}106989[#106989] (issue: {es-issue}106811[#106811]) -* Only trigger action once per thread {es-pull}107232[#107232] (issue: {es-issue}107215[#107215]) -* [Transform] Auto retry Transform start {es-pull}106243[#106243] - -Vector Search:: -* Fix multithreading copies in lib vec {es-pull}108802[#108802] -* [8.14] Fix multithreading copies in lib vec {es-pull}108810[#108810] - -[[deprecation-8.14.0]] -[float] -=== Deprecations - -Mapping:: -* Deprecate allowing `fields` in scenarios where it is ignored {es-pull}106031[#106031] - -[[enhancement-8.14.0]] -[float] -=== Enhancements - -Aggregations:: -* Add a `PriorityQueue` backed by `BigArrays` {es-pull}106361[#106361] -* All new `shard_seed` parameter for `random_sampler` agg {es-pull}104830[#104830] - -Allocation:: -* Add allocation stats {es-pull}105894[#105894] -* Add index forecasts to /_cat/allocation output {es-pull}97561[#97561] - -Application:: -* [Profiling] Add TopN Functions API {es-pull}106860[#106860] -* [Profiling] Allow to override index settings {es-pull}106172[#106172] -* [Profiling] Speed up serialization of flamegraph {es-pull}105779[#105779] - -Authentication:: -* Support Profile Activate with JWTs with client authn {es-pull}105439[#105439] (issue: {es-issue}105342[#105342]) - -Authorization:: -* Allow users to get status of own async search tasks {es-pull}106638[#106638] -* [Security Solution] Add `read` permission for third party agent indices for `kibana_system` {es-pull}107046[#107046] - -Data streams:: -* Add data stream lifecycle to kibana reporting template {es-pull}106259[#106259] - -ES|QL:: -* Add ES|QL Locate function {es-pull}106899[#106899] (issue: {es-issue}106818[#106818]) -* Add ES|QL signum function {es-pull}106866[#106866] -* Add status for enrich operator {es-pull}106036[#106036] -* Add two new OGC functions ST_X and ST_Y {es-pull}105768[#105768] -* Adjust array resizing in block builder {es-pull}106934[#106934] -* Bulk loading enrich fields in ESQL {es-pull}106796[#106796] -* ENRICH support for TEXT fields {es-pull}106435[#106435] (issue: {es-issue}105384[#105384]) -* ESQL: Add timers to many status results {es-pull}105421[#105421] -* ESQL: Allow grouping key inside stats expressions {es-pull}106579[#106579] -* ESQL: Introduce expression validation phase {es-pull}105477[#105477] (issue: {es-issue}105425[#105425]) -* ESQL: Log queries at debug level {es-pull}108257[#108257] -* ESQL: Regex improvements {es-pull}106429[#106429] -* ESQL: Sum of constants {es-pull}105454[#105454] -* ESQL: Support ST_DISJOINT {es-pull}107007[#107007] -* ESQL: Support partially folding CASE {es-pull}106094[#106094] -* ESQL: Use faster field caps {es-pull}105067[#105067] -* ESQL: extend BUCKET with spans {es-pull}107272[#107272] -* ESQL: perform a reduction on the data node {es-pull}106516[#106516] -* Expand support for ENRICH to full set supported by ES ingest processors {es-pull}106186[#106186] (issue: {es-issue}106162[#106162]) -* Introduce ordinal bytesref block {es-pull}106852[#106852] (issue: {es-issue}106387[#106387]) -* Leverage ordinals in enrich lookup {es-pull}107449[#107449] -* Serialize big array blocks {es-pull}106373[#106373] -* Serialize big array vectors {es-pull}106327[#106327] -* Specialize serialization for `ArrayVectors` {es-pull}105893[#105893] -* Specialize serialization of array blocks {es-pull}106102[#106102] -* Speed up serialization of `BytesRefArray` {es-pull}106053[#106053] -* Support ST_CONTAINS and ST_WITHIN {es-pull}106503[#106503] -* Support ST_INTERSECTS between geometry column and other geometry or string {es-pull}104907[#104907] (issue: {es-issue}104874[#104874]) - -Engine:: -* Add metric for calculating index flush time excluding waiting on locks {es-pull}107196[#107196] - -Highlighting:: -* Enable 'encoder' and 'tags_schema' highlighting settings at field level {es-pull}107224[#107224] (issue: {es-issue}94028[#94028]) - -ILM+SLM:: -* Add a flag to re-enable writes on the final index after an ILM shrink action. {es-pull}107121[#107121] (issue: {es-issue}106599[#106599]) - -Indices APIs:: -* Wait forever for `IndexTemplateRegistry` asset installation {es-pull}105985[#105985] - -Infra/CLI:: -* Enhance search tier GC options {es-pull}106526[#106526] -* Increase KDF iteration count in `KeyStoreWrapper` {es-pull}107107[#107107] - -Infra/Core:: -* Add pluggable `BuildVersion` in `NodeMetadata` {es-pull}105757[#105757] - -Infra/Metrics:: -* Infrastructure for metering the update requests {es-pull}105063[#105063] -* `DocumentParsingObserver` to accept an `indexName` to allow skipping system indices {es-pull}107041[#107041] - -Infra/Scripting:: -* String sha512() painless function {es-pull}99048[#99048] (issue: {es-issue}97691[#97691]) - -Ingest Node:: -* Add support for the 'Anonymous IP' database to the geoip processor {es-pull}107287[#107287] (issue: {es-issue}90789[#90789]) -* Add support for the 'Enterprise' database to the geoip processor {es-pull}107377[#107377] -* Adding `cache_stats` to geoip stats API {es-pull}107334[#107334] -* Support data streams in enrich policy indices {es-pull}107291[#107291] (issue: {es-issue}98836[#98836]) - -Machine Learning:: -* Add GET `_inference` for all inference endpoints {es-pull}107517[#107517] -* Added a timeout parameter to the inference API {es-pull}107242[#107242] -* Enable retrying on 500 error response from Cohere text embedding API {es-pull}105797[#105797] - -Mapping:: -* Make int8_hnsw our default index for new dense-vector fields {es-pull}106836[#106836] - -Ranking:: -* Add retrievers using the parser-only approach {es-pull}105470[#105470] - -Search:: -* Add Lucene spanish plural stemmer {es-pull}106952[#106952] -* Add `modelId` and `modelText` to `KnnVectorQueryBuilder` {es-pull}106068[#106068] -* Add a SIMD (Neon) optimised vector distance function for int8 {es-pull}106133[#106133] -* Add transport version for search load autoscaling {es-pull}106377[#106377] -* CCS with `minimize_roundtrips` performs incremental merges of each `SearchResponse` {es-pull}105781[#105781] -* Track ongoing search tasks {es-pull}107129[#107129] - -Security:: -* Invalidating cross cluster API keys requires `manage_security` {es-pull}107411[#107411] -* Show owner `realm_type` for returned API keys {es-pull}105629[#105629] - -Snapshot/Restore:: -* Add setting for max connections to S3 {es-pull}107533[#107533] -* Distinguish different snapshot failures by log level {es-pull}105622[#105622] - -Stats:: -* (API+) CAT Nodes alias for shard header to match CAT Allocation {es-pull}105847[#105847] -* Add total size in bytes to doc stats {es-pull}106840[#106840] (issue: {es-issue}97670[#97670]) - -TSDB:: -* Improve short-circuiting downsample execution {es-pull}106563[#106563] -* Support non-keyword dimensions as routing fields in TSDB {es-pull}105501[#105501] -* Text fields are stored by default in TSDB indices {es-pull}106338[#106338] (issue: {es-issue}97039[#97039]) - -Transform:: -* Check node shutdown before fail {es-pull}107358[#107358] (issue: {es-issue}100891[#100891]) -* Do not log error on node restart when the transform is already failed {es-pull}106171[#106171] (issue: {es-issue}106168[#106168]) - -[[feature-8.14.0]] -[float] -=== New features - -Application:: -* Allow `typed_keys` for search application Search API {es-pull}108007[#108007] -* [Connector API] Support cleaning up sync jobs when deleting a connector {es-pull}107253[#107253] - -ES|QL:: -* ESQL: Values aggregation function {es-pull}106065[#106065] (issue: {es-issue}103600[#103600]) -* ESQL: allow sorting by expressions and not only regular fields {es-pull}107158[#107158] -* Support ES|QL requests through the `NodeClient::execute` {es-pull}106244[#106244] - -Indices APIs:: -* Add granular error list to alias action response {es-pull}106514[#106514] (issue: {es-issue}94478[#94478]) - -Machine Learning:: -* Add Cohere rerank to `_inference` service {es-pull}106378[#106378] -* Add support for Azure OpenAI embeddings to inference service {es-pull}107178[#107178] -* Create default word based chunker {es-pull}107303[#107303] -* Text structure endpoints to determine the structure of a list of messages and of an indexed field {es-pull}105660[#105660] - -Mapping:: -* Flatten object mappings when subobjects is false {es-pull}103542[#103542] (issues: {es-issue}99860[#99860], {es-issue}103497[#103497]) - -Security:: -* Get and Query API Key with profile uid {es-pull}106531[#106531] - -Vector Search:: -* Adding support for hex-encoded byte vectors on knn-search {es-pull}105393[#105393] - -[[upgrade-8.14.0]] -[float] -=== Upgrades - -Infra/Core:: -* Upgrade jna to 5.12.1 {es-pull}105717[#105717] - -Ingest Node:: -* Updating the tika version to 2.9.1 in the ingest attachment plugin {es-pull}106315[#106315] - -Network:: -* Upgrade to Netty 4.1.107 {es-pull}105517[#105517] - -Packaging:: -* Update bundled JDK to Java 22 (again) {es-pull}108654[#108654] -