Skip to content

Commit

Permalink
[8.11] [Security Solution][Elastic AI Assistant] Updates ESQL Knowled…
Browse files Browse the repository at this point in the history
…ge Base docs to latest (#169593) (#169950)

# Backport

This will backport the following commits from `main` to `8.11`:
- [[Security Solution][Elastic AI Assistant] Updates ESQL Knowledge Base
docs to latest (#169593)](#169593)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Garrett
Spong","email":"[email protected]"},"sourceCommit":{"committedDate":"2023-10-26T14:45:05Z","message":"[Security
Solution][Elastic AI Assistant] Updates ESQL Knowledge Base docs to
latest (#169593)\n\n## Summary\r\n\r\nIn preparation for the final
`8.11` BC, this PR updates the Assistant\r\nESQL Knowledge Base docs to
match that of the latest [Elasticsearch
ESQL\r\ndocumentation](https://github.com/elastic/elasticsearch/tree/main/docs/reference/esql)\r\nand
[language\r\nfiles](https://github.com/elastic/elasticsearch/tree/f883dd98566c1f8ffa34779c9949eaeb27596014/x-pack/plugin/esql/src/main/antlr).\r\n\r\n\r\n##
Update process\r\n\r\nTo update, I deleted
the\r\n`x-pack/plugins/elastic_assistant/server/knowledge_base/documentation`\r\nfolder
contents, then copied the latest [Elasticsearch
ESQL\r\ndocumentation](https://github.com/elastic/elasticsearch/tree/main/docs/reference/esql)\r\nfiles
over. Then ran the below script from that directory via terminal\r\nto
rename all files/directories from `kebab-case` to `snake_case`
as\r\nrequired by the Kibana repo:\r\n\r\n> find . -depth -name '*-*'
-exec bash -c 'for f; do base=${f##*/}; mv\r\n-- \"$f\"
\"${f%/*}/${base//-/_}\"; done' bash {} +\r\n\r\n\r\nFor the language
files, I checked those manually and copied over the\r\nentire file
contents since there are only 4.\r\n\r\nWhen we do the next update, (and
confirm these raw `asciidocs` are the\r\nbest format), I will create a
single script that can be run for\r\nmaintenance
purposes.","sha":"b0f8c5bcf02bd6f6494b3a7ddf6ba757f6f630c8","branchLabelMapping":{"^v8.12.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","Team:
SecuritySolution","Feature:Elastic AI
Assistant","v8.11.0","v8.12.0"],"number":169593,"url":"https://github.com/elastic/kibana/pull/169593","mergeCommit":{"message":"[Security
Solution][Elastic AI Assistant] Updates ESQL Knowledge Base docs to
latest (#169593)\n\n## Summary\r\n\r\nIn preparation for the final
`8.11` BC, this PR updates the Assistant\r\nESQL Knowledge Base docs to
match that of the latest [Elasticsearch
ESQL\r\ndocumentation](https://github.com/elastic/elasticsearch/tree/main/docs/reference/esql)\r\nand
[language\r\nfiles](https://github.com/elastic/elasticsearch/tree/f883dd98566c1f8ffa34779c9949eaeb27596014/x-pack/plugin/esql/src/main/antlr).\r\n\r\n\r\n##
Update process\r\n\r\nTo update, I deleted
the\r\n`x-pack/plugins/elastic_assistant/server/knowledge_base/documentation`\r\nfolder
contents, then copied the latest [Elasticsearch
ESQL\r\ndocumentation](https://github.com/elastic/elasticsearch/tree/main/docs/reference/esql)\r\nfiles
over. Then ran the below script from that directory via terminal\r\nto
rename all files/directories from `kebab-case` to `snake_case`
as\r\nrequired by the Kibana repo:\r\n\r\n> find . -depth -name '*-*'
-exec bash -c 'for f; do base=${f##*/}; mv\r\n-- \"$f\"
\"${f%/*}/${base//-/_}\"; done' bash {} +\r\n\r\n\r\nFor the language
files, I checked those manually and copied over the\r\nentire file
contents since there are only 4.\r\n\r\nWhen we do the next update, (and
confirm these raw `asciidocs` are the\r\nbest format), I will create a
single script that can be run for\r\nmaintenance
purposes.","sha":"b0f8c5bcf02bd6f6494b3a7ddf6ba757f6f630c8"}},"sourceBranch":"main","suggestedTargetBranches":["8.11"],"targetPullRequestStates":[{"branch":"8.11","label":"v8.11.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.12.0","labelRegex":"^v8.12.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/169593","number":169593,"mergeCommit":{"message":"[Security
Solution][Elastic AI Assistant] Updates ESQL Knowledge Base docs to
latest (#169593)\n\n## Summary\r\n\r\nIn preparation for the final
`8.11` BC, this PR updates the Assistant\r\nESQL Knowledge Base docs to
match that of the latest [Elasticsearch
ESQL\r\ndocumentation](https://github.com/elastic/elasticsearch/tree/main/docs/reference/esql)\r\nand
[language\r\nfiles](https://github.com/elastic/elasticsearch/tree/f883dd98566c1f8ffa34779c9949eaeb27596014/x-pack/plugin/esql/src/main/antlr).\r\n\r\n\r\n##
Update process\r\n\r\nTo update, I deleted
the\r\n`x-pack/plugins/elastic_assistant/server/knowledge_base/documentation`\r\nfolder
contents, then copied the latest [Elasticsearch
ESQL\r\ndocumentation](https://github.com/elastic/elasticsearch/tree/main/docs/reference/esql)\r\nfiles
over. Then ran the below script from that directory via terminal\r\nto
rename all files/directories from `kebab-case` to `snake_case`
as\r\nrequired by the Kibana repo:\r\n\r\n> find . -depth -name '*-*'
-exec bash -c 'for f; do base=${f##*/}; mv\r\n-- \"$f\"
\"${f%/*}/${base//-/_}\"; done' bash {} +\r\n\r\n\r\nFor the language
files, I checked those manually and copied over the\r\nentire file
contents since there are only 4.\r\n\r\nWhen we do the next update, (and
confirm these raw `asciidocs` are the\r\nbest format), I will create a
single script that can be run for\r\nmaintenance
purposes.","sha":"b0f8c5bcf02bd6f6494b3a7ddf6ba757f6f630c8"}}]}]
BACKPORT-->

Co-authored-by: Garrett Spong <[email protected]>
  • Loading branch information
kibanamachine and spong authored Oct 26, 2023
1 parent 86a86dd commit 34ea138
Show file tree
Hide file tree
Showing 148 changed files with 1,606 additions and 304 deletions.

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,10 +1,28 @@
[[esql-processing-commands]]
== {esql} processing commands
[[esql-commands]]
=== {esql} commands

++++
<titleabbrev>Processing commands</titleabbrev>
<titleabbrev>Commands</titleabbrev>
++++

// tag::source_commands[]
==== Source commands

An {esql} source command produces a table, typically with data from {es}. An {esql} query must start with a source command.

image::images/esql/source-command.svg[A source command producing a table from {es},align="center"]

{esql} supports these source commands:

* <<esql-from>>
* <<esql-row>>
* <<esql-show>>

// end::source_command[]

// tag::proc_commands[]
==== Processing commands

{esql} processing commands change an input table by adding, removing, or changing
rows and columns.

Expand All @@ -25,6 +43,12 @@ image::images/esql/processing-command.svg[A processing command changing an input
* <<esql-stats-by>>
* <<esql-where>>

// end::proc_command[]

include::source-commands/from.asciidoc[]
include::source-commands/row.asciidoc[]
include::source-commands/show.asciidoc[]

include::processing-commands/dissect.asciidoc[]
include::processing-commands/drop.asciidoc[]
include::processing-commands/enrich.asciidoc[]
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
[[esql-enrich-data]]
=== Enrich data

++++
<titleabbrev>Enrich data</titleabbrev>
++++

You can use {esql}'s <<esql-enrich>> processing command to enrich a table with
data from indices in {es}.

For example, you can use `ENRICH` to:

* Identify web services or vendors based on known IP addresses
* Add product information to retail orders based on product IDs
* Supplement contact information based on an email address

[[esql-how-enrich-works]]
==== How the `ENRICH` command works

The `ENRICH` command adds new columns to a table, with data from {es} indices.
It requires a few special components:

image::images/esql/esql-enrich.png[align="center"]

[[esql-enrich-policy]]
Enrich policy::
+
--
A set of configuration options used to add the right enrich data to the input
table.

An enrich policy contains:

include::../ingest/enrich.asciidoc[tag=enrich-policy-fields]

After <<esql-create-enrich-policy,creating a policy>>, it must be
<<esql-execute-enrich-policy,executed>> before it can be used. Executing an
enrich policy uses data from the policy's source indices to create a streamlined
system index called the _enrich index_. The `ENRICH` command uses this index to
match and enrich an input table.
--

[[esql-source-index]]
Source index::
An index which stores enrich data that the `ENRICH` command can add to input
tables. You can create and manage these indices just like a regular {es} index.
You can use multiple source indices in an enrich policy. You also can use the
same source index in multiple enrich policies.

[[esql-enrich-index]]
Enrich index::
+
--
A special system index tied to a specific enrich policy.

Directly matching rows from input tables to documents in source indices could be
slow and resource intensive. To speed things up, the `ENRICH` command uses an
enrich index.

include::../ingest/enrich.asciidoc[tag=enrich-index]
--

[[esql-set-up-enrich-policy]]
==== Set up an enrich policy

To start using `ENRICH`, follow these steps:

. Check the <<enrich-prereqs, prerequisites>>.
. <<esql-create-enrich-source-index>>.
. <<esql-create-enrich-policy>>.
. <<esql-execute-enrich-policy>>.
. <<esql-use-enrich>>

Once you have enrich policies set up, you can <<esql-update-enrich-data,update
your enrich data>> and <<esql-update-enrich-policies, update your enrich
policies>>.

[IMPORTANT]
====
The `ENRICH` command performs several operations and may impact the speed of
your query.
====

[[esql-enrich-prereqs]]
==== Prerequisites

include::{es-repo-dir}/ingest/apis/enrich/put-enrich-policy.asciidoc[tag=enrich-policy-api-prereqs]

[[esql-create-enrich-source-index]]
==== Add enrich data

include::../ingest/enrich.asciidoc[tag=create-enrich-source-index]

[[esql-create-enrich-policy]]
==== Create an enrich policy

include::../ingest/enrich.asciidoc[tag=create-enrich-policy]

[[esql-execute-enrich-policy]]
==== Execute the enrich policy

include::../ingest/enrich.asciidoc[tag=execute-enrich-policy1]

image::images/esql/esql-enrich-policy.png[align="center"]

include::../ingest/enrich.asciidoc[tag=execute-enrich-policy2]

[[esql-use-enrich]]
==== Use the enrich policy

After the policy has been executed, you can use the <<esql-enrich,`ENRICH`
command>> to enrich your data.

image::images/esql/esql-enrich-command.png[align="center",width=50%]

include::processing-commands/enrich.asciidoc[tag=examples]

[[esql-update-enrich-data]]
==== Update an enrich index

include::{es-repo-dir}/ingest/apis/enrich/execute-enrich-policy.asciidoc[tag=update-enrich-index]

[[esql-update-enrich-policies]]
==== Update an enrich policy

include::../ingest/enrich.asciidoc[tag=update-enrich-policy]
Original file line number Diff line number Diff line change
Expand Up @@ -26,16 +26,17 @@ these functions:
* <<esql-date_parse>>
* <<esql-date_trunc>>
* <<esql-e>>
* <<esql-ends_with>>
* <<esql-floor>>
* <<esql-greatest>>
* <<esql-is_finite>>
* <<esql-is_infinite>>
* <<esql-is_nan>>
* <<esql-least>>
* <<esql-left>>
* <<esql-length>>
* <<esql-log10>>
* <<esql-ltrim>>
* <<esql-rtrim>>
* <<esql-mv_avg>>
* <<esql-mv_concat>>
* <<esql-mv_count>>
Expand All @@ -47,13 +48,15 @@ these functions:
* <<esql-now>>
* <<esql-pi>>
* <<esql-pow>>
* <<esql-replace>>
* <<esql-right>>
* <<esql-round>>
* <<esql-rtrim>>
* <<esql-sin>>
* <<esql-sinh>>
* <<esql-split>>
* <<esql-starts_with>>
* <<esql-substring>>
* <<esql-left>>
* <<esql-tan>>
* <<esql-tanh>>
* <<esql-tau>>
Expand Down Expand Up @@ -88,16 +91,17 @@ include::functions/date_format.asciidoc[]
include::functions/date_parse.asciidoc[]
include::functions/date_trunc.asciidoc[]
include::functions/e.asciidoc[]
include::functions/ends_with.asciidoc[]
include::functions/floor.asciidoc[]
include::functions/greatest.asciidoc[]
include::functions/is_finite.asciidoc[]
include::functions/is_infinite.asciidoc[]
include::functions/is_nan.asciidoc[]
include::functions/least.asciidoc[]
include::functions/left.asciidoc[]
include::functions/length.asciidoc[]
include::functions/log10.asciidoc[]
include::functions/ltrim.asciidoc[]
include::functions/rtrim.asciidoc[]
include::functions/mv_avg.asciidoc[]
include::functions/mv_concat.asciidoc[]
include::functions/mv_count.asciidoc[]
Expand All @@ -109,14 +113,16 @@ include::functions/mv_sum.asciidoc[]
include::functions/now.asciidoc[]
include::functions/pi.asciidoc[]
include::functions/pow.asciidoc[]
include::functions/replace.asciidoc[]
include::functions/right.asciidoc[]
include::functions/round.asciidoc[]
include::functions/rtrim.asciidoc[]
include::functions/sin.asciidoc[]
include::functions/sinh.asciidoc[]
include::functions/split.asciidoc[]
include::functions/sqrt.asciidoc[]
include::functions/starts_with.asciidoc[]
include::functions/substring.asciidoc[]
include::functions/left.asciidoc[]
include::functions/tan.asciidoc[]
include::functions/tanh.asciidoc[]
include::functions/tau.asciidoc[]
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
[[esql-functions-operators]]
=== {esql} functions and operators

++++
<titleabbrev>Functions and operators</titleabbrev>
++++

{esql} provides a comprehensive set of functions and operators for working with data.
The functions are divided into the following categories:

[[esql-functions]]
<<esql-agg-functions>>::
include::functions/aggregation-functions.asciidoc[tag=agg_list]

<<esql-math-functions>>::
include::functions/math-functions.asciidoc[tag=math_list]

<<esql-string-functions>>::
include::functions/string-functions.asciidoc[tag=string_list]

<<esql-date-time-functions>>::
include::functions/date-time-functions.asciidoc[tag=date_list]

<<esql-type-conversion-functions>>::
include::functions/type-conversion-functions.asciidoc[tag=type_list]

<<esql-conditional-functions-and-expressions>>::
include::functions/conditional-functions-and-expressions.asciidoc[tag=cond_list]

<<esql-mv-functions>>::
include::functions/mv-functions.asciidoc[tag=mv_list]

<<esql-operators>>::
include::functions/operators.asciidoc[tag=op_list]

include::functions/aggregation-functions.asciidoc[]
include::functions/math-functions.asciidoc[]
include::functions/string-functions.asciidoc[]
include::functions/date-time-functions.asciidoc[]
include::functions/type-conversion-functions.asciidoc[]
include::functions/conditional-functions-and-expressions.asciidoc[]
include::functions/mv-functions.asciidoc[]
include::functions/operators.asciidoc[]
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
[[esql-getting-started]]
== Getting started with {esql}

++++
<titleabbrev>Getting started</titleabbrev>
++++

coming::[8.11]
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
[[esql-kibana]]
== Using {esql} in {kib}

++++
<titleabbrev>Kibana</titleabbrev>
++++


Use {esql} in Discover to explore a data set. From the data view dropdown,
select *Try {esql}* to get started.

NOTE: {esql} queries in Discover and Lens are subject to the time range selected
with the time filter.


Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[[esql-language]]
== Working with the {esql} language

++++
<titleabbrev>Working with the {esql} language</titleabbrev>
++++

Detailed information about the {esql} language:

* <<esql-syntax>>
* <<esql-commands>>
* <<esql-functions>>
* <<esql-multivalued-fields>>
* <<esql-metadata-fields>>
* <<esql-enrich-data>>

include::esql-syntax.asciidoc[]
include::esql-commands.asciidoc[]
include::esql-functions-operators.asciidoc[]
include::multivalued-fields.asciidoc[]
include::metadata-fields.asciidoc[]
include::esql-enrich-data.asciidoc[]

Loading

0 comments on commit 34ea138

Please sign in to comment.