From 1430ae41472b9f426383eff60356fe4455519220 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Wed, 29 May 2024 16:17:23 -0700 Subject: [PATCH 01/15] Adds LLM performance matrix doc to ESS & serverless --- docs/assistant/llm-performance-matrix.asciidoc | 14 ++++++++++++++ docs/assistant/security-assistant.asciidoc | 1 + .../assistant/llm-performance-matrix.mdx | 17 +++++++++++++++++ docs/serverless/serverless-security.docnav.json | 3 +++ 4 files changed, 35 insertions(+) create mode 100644 docs/assistant/llm-performance-matrix.asciidoc create mode 100644 docs/serverless/assistant/llm-performance-matrix.mdx diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc new file mode 100644 index 0000000000..ed2964493d --- /dev/null +++ b/docs/assistant/llm-performance-matrix.asciidoc @@ -0,0 +1,14 @@ +[[llm-performance-matrix]] += Large language model performance matrix + +This table describes the performance of various LLMs for different use-cases in {elastic-sec}, based on our internal testing. To learn more about how to implement these use-cases, refer to <> or <>. + +[cols="1,1,1,1,1,1,1", options="header"] +|=== +| Feature/Model | Claude 3: Opus | Claude 3: Sonnet | Claude 3: Haiku | GPT-4o | GPT-4 Turbo | GPT-4 32K + +| Assistant - General | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| Assistant - {{esql}} generation | Great | Great | Poor | Excellent | Poor | Excellent +| Assistant - Alert questions | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) +| Attack Discovery | Excellent | Great | Poor | Poor | Good | Good (limited context) +|=== diff --git a/docs/assistant/security-assistant.asciidoc b/docs/assistant/security-assistant.asciidoc index 9f0066833c..7c94ff23db 100644 --- a/docs/assistant/security-assistant.asciidoc +++ b/docs/assistant/security-assistant.asciidoc @@ -194,6 +194,7 @@ In addition to practical advice, AI Assistant can offer conceptual advice, tips, include::ai-alert-triage.asciidoc[leveloffset=+1] +include::llm-performance-matrix.asciidoc[leveloffset=+1] include::azure-openai-setup.asciidoc[leveloffset=+1] include::connect-to-openai.asciidoc[leveloffset=+1] include::connect-to-bedrock.asciidoc[leveloffset=+1] diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx new file mode 100644 index 0000000000..7b51834a09 --- /dev/null +++ b/docs/serverless/assistant/llm-performance-matrix.mdx @@ -0,0 +1,17 @@ +--- +id: llm-performance-matrix +slug: /serverless/security/llm-performance-matrix +title: Large language model performance matrix +description: Learn how different models perform on different tasks in ((elastic-sec)). +tags: ["security", "overview", "get-started"] +status: in review +--- + +This table describes the performance of various LLMs for different use-cases in ((elastic-sec)), based on our internal testing. To learn more about how to implement these use-cases, refer to or . + +| Feature/Model | Claude 3: Opus | Claude 3: Sonnet | Claude 3: Haiku | GPT-4o | GPT-4 Turbo | GPT-4 32K | +|------------------------------|-----------------|-------------------|------------------|---------|--------------|-----------| +| Assistant - General | Excellent | Excellent | Excellent | Excellent| Excellent | Excellent | +| Assistant - ((esql)) generation| Great | Great | Poor | Excellent| Poor | Excellent | +| Assistant - Alert questions | Excellent | Excellent | Excellent | Excellent| Poor | Good (limited context) | +| Attack Discovery | Excellent | Great | Poor | Poor | Good | Good (limited context) | diff --git a/docs/serverless/serverless-security.docnav.json b/docs/serverless/serverless-security.docnav.json index e2b0da2e04..fbc65af420 100644 --- a/docs/serverless/serverless-security.docnav.json +++ b/docs/serverless/serverless-security.docnav.json @@ -30,6 +30,9 @@ { "id":"serverlessSecurityAssistantAlertTriage" }, + { + "id":"llm-performance-matrix" + }, { "id": "serverlessSecurityConnectBedrock" }, From 6475b168baaa1d7a5c584ba7d4b36d32f2a9120d Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Wed, 29 May 2024 16:36:20 -0700 Subject: [PATCH 02/15] experimenting with matrix formatting --- docs/serverless/assistant/llm-performance-matrix.mdx | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx index 7b51834a09..62c04f1b63 100644 --- a/docs/serverless/assistant/llm-performance-matrix.mdx +++ b/docs/serverless/assistant/llm-performance-matrix.mdx @@ -15,3 +15,11 @@ This table describes the performance of various LLMs for different use-cases in | Assistant - ((esql)) generation| Great | Great | Poor | Excellent| Poor | Excellent | | Assistant - Alert questions | Excellent | Excellent | Excellent | Excellent| Poor | Good (limited context) | | Attack Discovery | Excellent | Great | Poor | Poor | Good | Good (limited context) | + +| **Feature** | **Model** | | | | | | +|-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|----------------| +| | **Claude 3: Opus** | **Claude 3: Sonnet** | **Claude 3: Haiku** | **GPT-4o** | **GPT-4 Turbo** | **GPT-4 32K** | +| **Assistant - General** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | +| **Assistant - ES\|QL Generation** | Great | Great | Poor | Excellent | Poor | Excellent | +| **Assistant - Alert Questions** | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | +| **Attack Discovery** | Excellent | Great | Poor | Poor | Good | Good (limited context) | From 24c807e4cf10204753c0c1ef57ebc0b40f709e94 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Wed, 29 May 2024 17:15:22 -0700 Subject: [PATCH 03/15] uses alternative matrix formatting --- .../assistant/llm-performance-matrix.asciidoc | 2 +- .../assistant/llm-performance-matrix.mdx | 28 ++++++++++--------- 2 files changed, 16 insertions(+), 14 deletions(-) diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc index ed2964493d..19401de783 100644 --- a/docs/assistant/llm-performance-matrix.asciidoc +++ b/docs/assistant/llm-performance-matrix.asciidoc @@ -1,7 +1,7 @@ [[llm-performance-matrix]] = Large language model performance matrix -This table describes the performance of various LLMs for different use-cases in {elastic-sec}, based on our internal testing. To learn more about how to implement these use-cases, refer to <> or <>. +This table describes the performance of various LLMs for different use-cases in {elastic-sec}, based on our internal testing. To learn more about these use-cases, refer to <> or <>. [cols="1,1,1,1,1,1,1", options="header"] |=== diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx index 62c04f1b63..129b49abd5 100644 --- a/docs/serverless/assistant/llm-performance-matrix.mdx +++ b/docs/serverless/assistant/llm-performance-matrix.mdx @@ -7,19 +7,21 @@ tags: ["security", "overview", "get-started"] status: in review --- -This table describes the performance of various LLMs for different use-cases in ((elastic-sec)), based on our internal testing. To learn more about how to implement these use-cases, refer to or . +This table describes the performance of various LLMs for different use-cases in ((elastic-sec)), based on our internal testing. To learn more about these use-cases, refer to or . -| Feature/Model | Claude 3: Opus | Claude 3: Sonnet | Claude 3: Haiku | GPT-4o | GPT-4 Turbo | GPT-4 32K | -|------------------------------|-----------------|-------------------|------------------|---------|--------------|-----------| -| Assistant - General | Excellent | Excellent | Excellent | Excellent| Excellent | Excellent | -| Assistant - ((esql)) generation| Great | Great | Poor | Excellent| Poor | Excellent | -| Assistant - Alert questions | Excellent | Excellent | Excellent | Excellent| Poor | Good (limited context) | -| Attack Discovery | Excellent | Great | Poor | Poor | Good | Good (limited context) | - -| **Feature** | **Model** | | | | | | +| | **Model** | | | | | | |-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|----------------| -| | **Claude 3: Opus** | **Claude 3: Sonnet** | **Claude 3: Haiku** | **GPT-4o** | **GPT-4 Turbo** | **GPT-4 32K** | -| **Assistant - General** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | -| **Assistant - ES\|QL Generation** | Great | Great | Poor | Excellent | Poor | Excellent | -| **Assistant - Alert Questions** | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | +| **Feature:** | **Claude 3: Opus** | **Claude 3: Sonnet** | **Claude 3: Haiku** | **GPT-4o** | **GPT-4 Turbo** | **GPT-4 32K** | +| **Assistant: general** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | +| **Assistant: ((esql)) generation** | Great | Great | Poor | Excellent | Poor | Excellent | +| **Assistant: alert questions** | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | | **Attack Discovery** | Excellent | Great | Poor | Poor | Good | Good (limited context) | + +| | **Model** | **Claude 3: Opus** |**Claude 3: Sonnet**| **Claude 3: Haiku** |**GPT-4o**|**GPT-4 Turbo** | **GPT-4 32K** | +|-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|----------------|--------------------| +| **Feature:** | | | | | | | | +| | | | | | | | +| **Assistant: general** | |Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | +| **Assistant: ((esql)) generation** | |Great | Great | Poor | Excellent | Poor | Excellent | +| **Assistant: alert questions** | | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | +| **Attack Discovery** | |Excellent | Great | Poor | Poor | Good | Good (limited context) | From e7c4ea12152bd5c9a7d0048082be17f2d607876b Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Wed, 29 May 2024 17:16:04 -0700 Subject: [PATCH 04/15] updates matrix format, adds alternative version --- docs/serverless/assistant/llm-performance-matrix.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx index 129b49abd5..73e4cbc5da 100644 --- a/docs/serverless/assistant/llm-performance-matrix.mdx +++ b/docs/serverless/assistant/llm-performance-matrix.mdx @@ -9,9 +9,9 @@ status: in review This table describes the performance of various LLMs for different use-cases in ((elastic-sec)), based on our internal testing. To learn more about these use-cases, refer to or . -| | **Model** | | | | | | +| **Feature:** | **Model** | | | | | | |-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|----------------| -| **Feature:** | **Claude 3: Opus** | **Claude 3: Sonnet** | **Claude 3: Haiku** | **GPT-4o** | **GPT-4 Turbo** | **GPT-4 32K** | +| | **Claude 3: Opus** | **Claude 3: Sonnet** | **Claude 3: Haiku** | **GPT-4o** | **GPT-4 Turbo** | **GPT-4 32K** | | **Assistant: general** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **Assistant: ((esql)) generation** | Great | Great | Poor | Excellent | Poor | Excellent | | **Assistant: alert questions** | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | From fdea39348eb0a285fd2e24cbc4484af91f8264cc Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Thu, 30 May 2024 08:03:22 -0700 Subject: [PATCH 05/15] uses updated table format --- docs/serverless/assistant/llm-performance-matrix.mdx | 8 -------- 1 file changed, 8 deletions(-) diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx index 73e4cbc5da..531d4b9c0f 100644 --- a/docs/serverless/assistant/llm-performance-matrix.mdx +++ b/docs/serverless/assistant/llm-performance-matrix.mdx @@ -17,11 +17,3 @@ This table describes the performance of various LLMs for different use-cases in | **Assistant: alert questions** | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | | **Attack Discovery** | Excellent | Great | Poor | Poor | Good | Good (limited context) | -| | **Model** | **Claude 3: Opus** |**Claude 3: Sonnet**| **Claude 3: Haiku** |**GPT-4o**|**GPT-4 Turbo** | **GPT-4 32K** | -|-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|----------------|--------------------| -| **Feature:** | | | | | | | | -| | | | | | | | -| **Assistant: general** | |Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | -| **Assistant: ((esql)) generation** | |Great | Great | Poor | Excellent | Poor | Excellent | -| **Assistant: alert questions** | | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | -| **Attack Discovery** | |Excellent | Great | Poor | Poor | Good | Good (limited context) | From ce767afe368fd9a29b5729ca0470c3a188028a14 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Thu, 30 May 2024 08:07:04 -0700 Subject: [PATCH 06/15] updates ESS version of table --- docs/assistant/llm-performance-matrix.asciidoc | 11 +++++++++++ 1 file changed, 11 insertions(+) diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc index 19401de783..375ff6ed53 100644 --- a/docs/assistant/llm-performance-matrix.asciidoc +++ b/docs/assistant/llm-performance-matrix.asciidoc @@ -12,3 +12,14 @@ This table describes the performance of various LLMs for different use-cases in | Assistant - Alert questions | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | Attack Discovery | Excellent | Great | Poor | Poor | Good | Good (limited context) |=== + +[cols="1,1,1,1,1,1,1", options="header"] +|=== +| *Feature* | *Model* | | | | | +| | *Claude 3: Opus* | *Claude 3: Sonnet* | *Claude 3: Haiku* | *GPT-4o* | *GPT-4 Turbo*| *GPT-4 32K* + +| *Assistant - General* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent +| *Assistant - ES|QL Generation*| Great | Great | Poor | Excellent | Poor | Excellent +| *Assistant - Alert Questions* | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) +| *Attack Discovery* | Excellent | Great | Poor | Poor | Good | Good (limited context) +|=== From 97b53d5f5a9f15a4c6ae39199b99b0e005225fa5 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Fri, 31 May 2024 10:55:05 -0700 Subject: [PATCH 07/15] fixes table --- docs/assistant/llm-performance-matrix.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc index 375ff6ed53..665e205a81 100644 --- a/docs/assistant/llm-performance-matrix.asciidoc +++ b/docs/assistant/llm-performance-matrix.asciidoc @@ -19,7 +19,7 @@ This table describes the performance of various LLMs for different use-cases in | | *Claude 3: Opus* | *Claude 3: Sonnet* | *Claude 3: Haiku* | *GPT-4o* | *GPT-4 Turbo*| *GPT-4 32K* | *Assistant - General* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| *Assistant - ES|QL Generation*| Great | Great | Poor | Excellent | Poor | Excellent +| *Assistant - {{esql}} Generation*| Great | Great | Poor | Excellent | Poor | Excellent | *Assistant - Alert Questions* | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | *Attack Discovery* | Excellent | Great | Poor | Poor | Good | Good (limited context) |=== From 3c5a2c431a0dc945cb26530c5e5d9d6e695127e2 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein Date: Fri, 31 May 2024 13:35:07 -0700 Subject: [PATCH 08/15] removes outdated table formatting --- docs/assistant/llm-performance-matrix.asciidoc | 12 +----------- 1 file changed, 1 insertion(+), 11 deletions(-) diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc index 665e205a81..a29fe97b10 100644 --- a/docs/assistant/llm-performance-matrix.asciidoc +++ b/docs/assistant/llm-performance-matrix.asciidoc @@ -1,17 +1,7 @@ [[llm-performance-matrix]] = Large language model performance matrix -This table describes the performance of various LLMs for different use-cases in {elastic-sec}, based on our internal testing. To learn more about these use-cases, refer to <> or <>. - -[cols="1,1,1,1,1,1,1", options="header"] -|=== -| Feature/Model | Claude 3: Opus | Claude 3: Sonnet | Claude 3: Haiku | GPT-4o | GPT-4 Turbo | GPT-4 32K - -| Assistant - General | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| Assistant - {{esql}} generation | Great | Great | Poor | Excellent | Poor | Excellent -| Assistant - Alert questions | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) -| Attack Discovery | Excellent | Great | Poor | Poor | Good | Good (limited context) -|=== +This table describes the performance of various LLMs for different use-cases in {elastic-sec}, based on our internal testing. To learn more about these use-cases, refer to <> or <>. [cols="1,1,1,1,1,1,1", options="header"] |=== From db7df88c1ab4e1785e9901aefe898488de7adfa2 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Fri, 31 May 2024 14:35:11 -0700 Subject: [PATCH 09/15] Update docs/assistant/llm-performance-matrix.asciidoc Co-authored-by: Joe Peeples --- docs/assistant/llm-performance-matrix.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc index a29fe97b10..85e1c5d637 100644 --- a/docs/assistant/llm-performance-matrix.asciidoc +++ b/docs/assistant/llm-performance-matrix.asciidoc @@ -9,7 +9,7 @@ This table describes the performance of various LLMs for different use-cases in | | *Claude 3: Opus* | *Claude 3: Sonnet* | *Claude 3: Haiku* | *GPT-4o* | *GPT-4 Turbo*| *GPT-4 32K* | *Assistant - General* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent -| *Assistant - {{esql}} Generation*| Great | Great | Poor | Excellent | Poor | Excellent +| *Assistant - {esql} Generation*| Great | Great | Poor | Excellent | Poor | Excellent | *Assistant - Alert Questions* | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | *Attack Discovery* | Excellent | Great | Poor | Poor | Good | Good (limited context) |=== From eda012e4ddb34c95a1418be08a7968686e1519d6 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Fri, 31 May 2024 14:35:22 -0700 Subject: [PATCH 10/15] Update docs/serverless/assistant/llm-performance-matrix.mdx Co-authored-by: Joe Peeples --- docs/serverless/assistant/llm-performance-matrix.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx index 531d4b9c0f..b9ff35b559 100644 --- a/docs/serverless/assistant/llm-performance-matrix.mdx +++ b/docs/serverless/assistant/llm-performance-matrix.mdx @@ -15,5 +15,5 @@ This table describes the performance of various LLMs for different use-cases in | **Assistant: general** | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | | **Assistant: ((esql)) generation** | Great | Great | Poor | Excellent | Poor | Excellent | | **Assistant: alert questions** | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) | -| **Attack Discovery** | Excellent | Great | Poor | Poor | Good | Good (limited context) | +| **Attack discovery** | Excellent | Great | Poor | Poor | Good | Good (limited context) | From 56ae0a94d3ff367bd2cac7b4981f608f5a59fafc Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Fri, 31 May 2024 14:35:27 -0700 Subject: [PATCH 11/15] Update docs/serverless/assistant/llm-performance-matrix.mdx Co-authored-by: Joe Peeples --- docs/serverless/assistant/llm-performance-matrix.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx index b9ff35b559..14f8a35267 100644 --- a/docs/serverless/assistant/llm-performance-matrix.mdx +++ b/docs/serverless/assistant/llm-performance-matrix.mdx @@ -7,7 +7,7 @@ tags: ["security", "overview", "get-started"] status: in review --- -This table describes the performance of various LLMs for different use-cases in ((elastic-sec)), based on our internal testing. To learn more about these use-cases, refer to or . +This table describes the performance of various LLMs for different use cases in ((elastic-sec)), based on our internal testing. To learn more about these use cases, refer to or . | **Feature:** | **Model** | | | | | | |-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|----------------| From cc8ee83ce8a928035069c619bbe498378e04dd9c Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Mon, 3 Jun 2024 08:52:45 -0700 Subject: [PATCH 12/15] Update docs/assistant/llm-performance-matrix.asciidoc Co-authored-by: Joe Peeples --- docs/assistant/llm-performance-matrix.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc index 85e1c5d637..d6e1fd565a 100644 --- a/docs/assistant/llm-performance-matrix.asciidoc +++ b/docs/assistant/llm-performance-matrix.asciidoc @@ -11,5 +11,5 @@ This table describes the performance of various LLMs for different use-cases in | *Assistant - General* | Excellent | Excellent | Excellent | Excellent | Excellent | Excellent | *Assistant - {esql} Generation*| Great | Great | Poor | Excellent | Poor | Excellent | *Assistant - Alert Questions* | Excellent | Excellent | Excellent | Excellent | Poor | Good (limited context) -| *Attack Discovery* | Excellent | Great | Poor | Poor | Good | Good (limited context) +| *Attack discovery* | Excellent | Great | Poor | Poor | Good | Good (limited context) |=== From b34b0a68fed9d925832ed97114a77397e421e39c Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Mon, 3 Jun 2024 08:52:59 -0700 Subject: [PATCH 13/15] Update docs/assistant/llm-performance-matrix.asciidoc Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> --- docs/assistant/llm-performance-matrix.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc index d6e1fd565a..117bb5a8c0 100644 --- a/docs/assistant/llm-performance-matrix.asciidoc +++ b/docs/assistant/llm-performance-matrix.asciidoc @@ -1,7 +1,7 @@ [[llm-performance-matrix]] = Large language model performance matrix -This table describes the performance of various LLMs for different use-cases in {elastic-sec}, based on our internal testing. To learn more about these use-cases, refer to <> or <>. +This table describes the performance of various large language models (LLMs) for different use-cases in {elastic-sec}, based on our internal testing. To learn more about these use-cases, refer to <> or <>. [cols="1,1,1,1,1,1,1", options="header"] |=== From 3ce73729eea108576460cf6512dd1814c09436e5 Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Mon, 3 Jun 2024 08:53:13 -0700 Subject: [PATCH 14/15] Update docs/serverless/assistant/llm-performance-matrix.mdx Co-authored-by: natasha-moore-elastic <137783811+natasha-moore-elastic@users.noreply.github.com> --- docs/serverless/assistant/llm-performance-matrix.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx index 14f8a35267..bec3ea79a5 100644 --- a/docs/serverless/assistant/llm-performance-matrix.mdx +++ b/docs/serverless/assistant/llm-performance-matrix.mdx @@ -7,7 +7,7 @@ tags: ["security", "overview", "get-started"] status: in review --- -This table describes the performance of various LLMs for different use cases in ((elastic-sec)), based on our internal testing. To learn more about these use cases, refer to or . +This table describes the performance of various large language models (LLMs) for different use cases in ((elastic-sec)), based on our internal testing. To learn more about these use cases, refer to or . | **Feature:** | **Model** | | | | | | |-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|----------------| From 83c0e94b1c246c6fbb0ef82d8dbd0b6fd3cc4b8d Mon Sep 17 00:00:00 2001 From: Benjamin Ironside Goldstein <91905639+benironside@users.noreply.github.com> Date: Mon, 3 Jun 2024 09:00:43 -0700 Subject: [PATCH 15/15] Update docs/assistant/llm-performance-matrix.asciidoc --- docs/assistant/llm-performance-matrix.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc index 117bb5a8c0..e36b7bf01d 100644 --- a/docs/assistant/llm-performance-matrix.asciidoc +++ b/docs/assistant/llm-performance-matrix.asciidoc @@ -1,7 +1,7 @@ [[llm-performance-matrix]] = Large language model performance matrix -This table describes the performance of various large language models (LLMs) for different use-cases in {elastic-sec}, based on our internal testing. To learn more about these use-cases, refer to <> or <>. +This table describes the performance of various large language models (LLMs) for different use cases in {elastic-sec}, based on our internal testing. To learn more about these use cases, refer to <> or <>. [cols="1,1,1,1,1,1,1", options="header"] |===