[8.x] [Data Views] Backport Upgrade Assistant changes to warn about s…

…cripted field creation removal in 9.0 (#205561)

## Summary

This PR backports the Upgrade Assistant changes from #202250 to warn
about the upcoming scripted field creation removal in 9.0. It also
includes some of the documentation changes and cleanup in the Data Views
Management UI.

### Checklist

- [x] Any text added follows [EUI's writing
guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses
sentence case text and includes [i18n
support](https://github.com/elastic/kibana/blob/main/packages/kbn-i18n/README.md)
- [x]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
- [ ] If a plugin configuration key changed, check if it needs to be
allowlisted in the cloud and added to the [docker
list](https://github.com/elastic/kibana/blob/main/src/dev/build/tasks/os_packages/docker_generator/resources/base/bin/kibana-docker)
- [ ] This was checked for breaking HTTP API changes, and any breaking
changes have been approved by the breaking-change committee. The
`release_note:breaking` label should be applied in these situations.
- [ ] [Flaky Test
Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner/1) was
used on any tests changed
- [x] The PR description includes the appropriate Release Notes section,
and the correct `release_note:*` label is applied per the
[guidelines](https://www.elastic.co/guide/en/kibana/master/contributing.html#kibana-release-notes-process)

docs/management/manage-data-views.asciidoc

@@ -170,7 +170,7 @@ Edit the settings for runtime fields, or remove runtime fields from data views.
 [[scripted-fields]]
 === Add scripted fields to data views
 
-deprecated::[7.13,Use {ref}/runtime.html[runtime fields] instead of scripted fields. Runtime fields support Painless scripts and provide greater flexibility.]
+deprecated::[7.13,Use {ref}/runtime.html[runtime fields] instead of scripted fields. Runtime fields support Painless scripting and provide greater flexibility. You can also use the {ref}/esql.html[Elasticsearch Query Language (ES|QL)] to compute values directly at query time.]
 
 Scripted fields compute data on the fly from the data in your {es} indices. The data is shown on
 the Discover tab as part of the document data, and you can use scripted fields in your visualizations. You query scripted fields with the <<kuery-query, {kib} query language>>, and can filter them using the filter bar. The scripted field values are computed at query time, so they aren't indexed and cannot be searched using the {kib} default
@@ -192,6 +192,110 @@ doc['field_name'].value
 For more information on scripted fields and additional examples, refer to
 https://www.elastic.co/blog/using-painless-kibana-scripted-fields[Using Painless in {kib} scripted fields]
 
+[float]
+[[migrate-off-scripted-fields]]
+==== Migrate to runtime fields or ES|QL queries
+
+The following code snippets demonstrate how an example scripted field called `computed_values` on the Kibana Sample Data Logs data view could be migrated to either a runtime field or an ES|QL query, highlighting the differences between each approach.
+
+[float]
+[[scripted-field-example]]
+===== Scripted field
+
+In the scripted field example, variables are created to track all values the script will need to access or return. Since scripted fields can only return a single value, the created variables must be returned together as an array at the end of the script.
+
+[source,text]
+----
+def hour_of_day = $('@timestamp', ZonedDateTime.parse('1970-01-01T00:00:00Z')).getHour();
+def time_of_day = '';
+
+if (hour_of_day >= 22 || hour_of_day < 5)
+  time_of_day = 'Night';
+else if (hour_of_day < 12)
+  time_of_day = 'Morning';
+else if (hour_of_day < 18)
+  time_of_day = 'Afternoon';
+else
+  time_of_day = 'Evening';
+
+def response_int = Integer.parseInt($('response.keyword', '200'));
+def response_category = '';
+
+if (response_int < 200)
+  response_category = 'Informational';
+else if (response_int < 300)
+  response_category = 'Successful';
+else if (response_int < 400)
+  response_category = 'Redirection';
+else if (response_int < 500)
+  response_category = 'Client Error';
+else
+  response_category = 'Server Error';
+
+return [time_of_day, response_category];
+----
+
+[float]
+[[runtime-field-example]]
+===== Runtime field
+
+Unlike scripted fields, runtime fields do not need to return a single value and can emit values at any point in the script, which will be combined and returned as a multi-value field. This allows for more flexibility in the script logic and removes the need to manually manage an array of values.
+
+[source,text]
+----
+def hour_of_day = $('@timestamp', ZonedDateTime.parse('1970-01-01T00:00:00Z')).getHour();
+
+if (hour_of_day >= 22 || hour_of_day < 5)
+  emit('Night');
+else if (hour_of_day < 12)
+  emit('Morning');
+else if (hour_of_day < 18)
+  emit('Afternoon');
+else
+  emit('Evening');
+
+def response_int = Integer.parseInt($('response.keyword', '200'));
+
+if (response_int < 200)
+  emit('Informational');
+else if (response_int < 300)
+  emit('Successful');
+else if (response_int < 400)
+  emit('Redirection');
+else if (response_int < 500)
+  emit('Client Error');
+else
+  emit('Server Error');
+----
+
+[float]
+[[esql-example]]
+===== ES|QL query
+
+Alternatively, ES|QL can be used to skip the need for data view management entirely and simply compute the values you need at query time. ES|QL supports computing multiple field values in a single query, using computed values with its rich set of commands and functions, and even aggregations against computed values. This makes it an excellent solution for one-off queries and realtime data analysis.
+
+[source,esql]
+----
+FROM kibana_sample_data_logs
+  | EVAL hour_of_day = DATE_EXTRACT("HOUR_OF_DAY", @timestamp)
+  | EVAL time_of_day = CASE(
+      hour_of_day >= 22 OR hour_of_day < 5, "Night",
+      hour_of_day < 12, "Morning",
+      hour_of_day < 18, "Afternoon",
+      "Evening"
+    )
+  | EVAL response_int = TO_INTEGER(response)
+  | EVAL response_category = CASE(
+      response_int < 200, "Informational",
+      response_int < 300, "Successful",
+      response_int < 400, "Redirection",
+      response_int < 500, "Client Error",
+      "Server Error"
+    )
+  | EVAL computed_values = MV_APPEND(time_of_day, response_category)
+  | DROP hour_of_day, time_of_day, response_int, response_category
+----
+
 [float]
 [[create-scripted-field]]
 ==== Create scripted fields
@@ -214,6 +318,8 @@ For more information about scripted fields in {es}, refer to {ref}/modules-scrip
 [[update-scripted-field]]
 ==== Manage scripted fields
 
+WARNING: The ability to create new scripted fields has been removed from the *Data Views* management page in 9.0. Existing scripted fields can still be edited or deleted, and the creation UI can be accessed by navigating directly to `/app/management/kibana/dataViews/dataView/{dataViewId}/create-field`, but we recommend migrating to runtime fields or ES|QL queries instead to prepare for removal.
+
 . Go to the *Data Views* management page using the navigation menu or the <<kibana-navigation-search,global search field>>.
 
 . Select the data view that contains the scripted field you want to manage.

src/core/packages/deprecations/common/index.ts

@@ -8,6 +8,7 @@
  */
 
 export type {
+  DeprecationDetailsMessage,
   BaseDeprecationDetails,
   ConfigDeprecationDetails,
   FeatureDeprecationDetails,

src/core/packages/deprecations/common/src/types.ts

@@ -7,6 +7,11 @@
  * License v3.0 only", or the "Server Side Public License, v 1".
  */
 
+export interface DeprecationDetailsMessage {
+  type: 'markdown' | 'text';
+  content: string;
+}
+
 /**
  * Base properties shared by all types of deprecations
  *
@@ -22,7 +27,7 @@ export interface BaseDeprecationDetails {
    * The description message to be displayed for the deprecation.
    * Check the README for writing deprecations in `src/core/server/deprecations/README.mdx`
    */
-  message: string | string[];
+  message: string | DeprecationDetailsMessage | Array<string | DeprecationDetailsMessage>;
   /**
    * levels:
    * - warning: will not break deployment upon upgrade

src/platform/packages/shared/kbn-doc-links/src/get_doc_links.ts

@@ -349,6 +349,7 @@ export const getDocLinks = ({ kibanaBranch, buildFlavor }: GetDocLinkOptions): D
       fieldFormattersNumber: `${KIBANA_DOCS}numeral.html`,
       fieldFormattersString: `${KIBANA_DOCS}managing-data-views.html#string-field-formatters`,
       runtimeFields: `${KIBANA_DOCS}managing-data-views.html#runtime-fields`,
+      migrateOffScriptedFields: `${KIBANA_DOCS}managing-data-views.html#migrate-off-scripted-fields`,
     },
     addData: `${KIBANA_DOCS}connect-to-elasticsearch.html`,
     kibana: {

src/platform/packages/shared/kbn-doc-links/src/types.ts

@@ -307,6 +307,7 @@ export interface DocLinks {
     readonly fieldFormattersNumber: string;
     readonly fieldFormattersString: string;
     readonly runtimeFields: string;
+    readonly migrateOffScriptedFields: string;
   };
   readonly addData: string;
   readonly kibana: {

src/platform/plugins/shared/data_view_management/public/components/edit_index_pattern/scripted_fields_table/components/call_outs/call_outs.tsx

@@ -24,11 +24,12 @@ export const CallOuts = ({ deprecatedLangsInUse, painlessDocLink }: CallOutsProp
 
   return (
     <>
+      <EuiSpacer size="m" />
       <EuiCallOut
         title={
           <FormattedMessage
-            id="indexPatternManagement.editIndexPattern.scripted.deprecationLangHeader"
-            defaultMessage="Deprecation languages in use"
+            id="indexPatternManagement.editIndexPattern.scripted.deprecatedLangHeader"
+            defaultMessage="Deprecated languages in use"
           />
         }
         color="danger"
@@ -38,7 +39,7 @@ export const CallOuts = ({ deprecatedLangsInUse, painlessDocLink }: CallOutsProp
           <FormattedMessage
             id="indexPatternManagement.editIndexPattern.scripted.deprecationLangLabel.deprecationLangDetail"
             defaultMessage="The following deprecated languages are in use: {deprecatedLangsInUse}. Support for these languages will be
-            removed in the next major version of Kibana and Elasticsearch. Convert you scripted fields to {link} to avoid any problems."
+            removed in the next major version of Kibana and Elasticsearch. Convert your scripted fields to {link} to avoid any problems."
             values={{
               deprecatedLangsInUse: deprecatedLangsInUse.join(', '),
               link: (
@@ -53,7 +54,6 @@ export const CallOuts = ({ deprecatedLangsInUse, painlessDocLink }: CallOutsProp
           />
         </p>
       </EuiCallOut>
-      <EuiSpacer size="m" />
     </>
   );
 };

src/platform/plugins/shared/data_view_management/public/components/edit_index_pattern/scripted_fields_table/components/header/header.tsx

@@ -9,9 +9,19 @@
 
 import React from 'react';
 import { withRouter, RouteComponentProps } from 'react-router-dom';
-import { EuiButton, EuiFlexGroup, EuiFlexItem, EuiText, EuiLink, EuiIcon } from '@elastic/eui';
+import {
+  EuiCallOut,
+  EuiButton,
+  EuiFlexGroup,
+  EuiFlexItem,
+  EuiText,
+  EuiLink,
+  EuiSpacer,
+} from '@elastic/eui';
 
 import { FormattedMessage } from '@kbn/i18n-react';
+import { i18n } from '@kbn/i18n';
+
 import { ScopedHistory } from '@kbn/core/public';
 
 import { reactRouterNavigate, useKibana } from '@kbn/kibana-react-plugin/public';
@@ -27,32 +37,47 @@ export const Header = withRouter(({ indexPatternId, history }: HeaderProps) => {
   const links = docLinks?.links;
   const userEditPermission = dataViews.getCanSaveSync();
   return (
-    <EuiFlexGroup alignItems="center">
+    <EuiFlexGroup alignItems="flexStart">
       <EuiFlexItem>
+        <EuiCallOut
+          title={i18n.translate('indexPatternManagement.editIndexPattern.deprecation.title', {
+            defaultMessage: 'Scripted fields are deprecated',
+          })}
+          color="warning"
+          iconType="warning"
+        >
+          <FormattedMessage
+            id="indexPatternManagement.editIndexPattern.deprecation.message"
+            tagName="span"
+            defaultMessage="Use {runtimeFieldsLink} instead of scripted fields. Runtime fields support Painless scripting and provide greater flexibility. You can also use the {esqlLink} to compute values directly at query time."
+            values={{
+              runtimeFieldsLink: (
+                <EuiLink target="_blank" href={links.indexPatterns.runtimeFields}>
+                  <FormattedMessage
+                    id="indexPatternManagement.header.runtimeLink"
+                    defaultMessage="runtime fields"
+                  />
+                </EuiLink>
+              ),
+              esqlLink: (
+                <EuiLink target="_blank" href={links.query.queryESQL}>
+                  <FormattedMessage
+                    id="indexPatternManagement.header.esqlLink"
+                    defaultMessage="Elasticsearch Query Language (ES|QL)"
+                  />
+                </EuiLink>
+              ),
+            }}
+          />
+        </EuiCallOut>
+        <EuiSpacer size="m" />
         <EuiText size="s">
           <p>
             <FormattedMessage
               tagName="span"
               id="indexPatternManagement.editIndexPattern.scriptedLabel"
               defaultMessage="Scripted fields can be used in visualizations and displayed in documents. However, they cannot be searched."
             />
-            <br />
-            <EuiIcon type="warning" color="warning" css={{ marginRight: '4px' }} />
-            <FormattedMessage
-              id="indexPatternManagement.editIndexPattern.deprecation"
-              tagName="span"
-              defaultMessage="Scripted fields are deprecated. Use {runtimeDocs} instead."
-              values={{
-                runtimeDocs: (
-                  <EuiLink target="_blank" href={links.runtimeFields.overview}>
-                    <FormattedMessage
-                      id="indexPatternManagement.header.runtimeLink"
-                      defaultMessage="runtime fields"
-                    />
-                  </EuiLink>
-                ),
-              }}
-            />
           </p>
         </EuiText>
       </EuiFlexItem>