Adds LLM performance matrix doc to ESS & serverless (#5286)

* Adds LLM performance matrix doc to ESS & serverless * experimenting with matrix formatting * uses alternative matrix formatting * updates matrix format, adds alternative version * uses updated table format * updates ESS version of table * fixes table * removes outdated table formatting * Update docs/assistant/llm-performance-matrix.asciidoc Co-authored-by: Joe Peeples <[email protected]> * Update docs/serverless/assistant/llm-performance-matrix.mdx Co-authored-by: Joe Peeples <[email protected]> * Update docs/serverless/assistant/llm-performance-matrix.mdx Co-authored-by: Joe Peeples <[email protected]> * Update docs/assistant/llm-performance-matrix.asciidoc Co-authored-by: Joe Peeples <[email protected]> * Update docs/assistant/llm-performance-matrix.asciidoc Co-authored-by: natasha-moore-elastic <[email protected]> * Update docs/serverless/assistant/llm-performance-matrix.mdx Co-authored-by: natasha-moore-elastic <[email protected]> * Update docs/assistant/llm-performance-matrix.asciidoc --------- Co-authored-by: Joe Peeples <[email protected]> Co-authored-by: natasha-moore-elastic <[email protected]>
elastic · Jun 3, 2024 · adb07fa · adb07fa
1 parent ac82dee
commit adb07fa
Show file tree

Hide file tree

Showing 4 changed files with 38 additions and 0 deletions.
diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc
@@ -0,0 +1,15 @@
+[[llm-performance-matrix]]
+= Large language model performance matrix
+
+This table describes the performance of various large language models (LLMs) for different use cases in {elastic-sec}, based on our internal testing. To learn more about these use cases, refer to <<attack-discovery, Attack discovery>> or <<security-assistant, AI Assistant>>.
+
+[cols="1,1,1,1,1,1,1", options="header"]
+|===
+| *Feature*                     | *Model*               |                    |                   |         |              |             
+|                               | *Claude 3: Opus*      | *Claude 3: Sonnet* | *Claude 3: Haiku* | *GPT-4o* | *GPT-4 Turbo*| *GPT-4 32K* 
+
+| *Assistant - General*         | Excellent             | Excellent          | Excellent         | Excellent | Excellent     | Excellent
+| *Assistant - {esql} Generation*| Great                 | Great              | Poor              | Excellent | Poor          | Excellent
+| *Assistant - Alert Questions* | Excellent             | Excellent          | Excellent         | Excellent | Poor          | Good (limited context)
+| *Attack discovery*            | Excellent             | Great              | Poor              | Poor      | Good          | Good (limited context)
+|===
diff --git a/docs/assistant/security-assistant.asciidoc b/docs/assistant/security-assistant.asciidoc
@@ -194,6 +194,7 @@ In addition to practical advice, AI Assistant can offer conceptual advice, tips,
 
 
 include::ai-alert-triage.asciidoc[leveloffset=+1]
+include::llm-performance-matrix.asciidoc[leveloffset=+1]
 include::azure-openai-setup.asciidoc[leveloffset=+1]
 include::connect-to-openai.asciidoc[leveloffset=+1]
 include::connect-to-bedrock.asciidoc[leveloffset=+1]
diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx
@@ -0,0 +1,19 @@
+---
+id: llm-performance-matrix
+slug: /serverless/security/llm-performance-matrix
+title: Large language model performance matrix
+description: Learn how different models perform on different tasks in ((elastic-sec)).
+tags: ["security", "overview", "get-started"]
+status: in review
+---
+
+This table describes the performance of various large language models (LLMs) for different use cases in ((elastic-sec)), based on our internal testing. To learn more about these use cases, refer to <DocLink id="attackDiscovery" text="Attack discovery"/> or <DocLink id="serverlessSecurityAIAssistant" text="AI Assistant"/>.
+
+|           **Feature:**        | **Model**             |                    |                    |            |                 |                |
+|-------------------------------|-----------------------|--------------------|--------------------|------------|-----------------|----------------|
+|                               | **Claude 3: Opus**    | **Claude 3: Sonnet** | **Claude 3: Haiku** | **GPT-4o** | **GPT-4 Turbo** | **GPT-4 32K**  |
+| **Assistant: general**       | Excellent             | Excellent          | Excellent          | Excellent  | Excellent       | Excellent      |
+| **Assistant: ((esql)) generation** | Great           | Great              | Poor               | Excellent  | Poor            | Excellent      |
+| **Assistant: alert questions** | Excellent          | Excellent          | Excellent          | Excellent  | Poor            | Good (limited context) |
+| **Attack discovery**          | Excellent             | Great              | Poor               | Poor       | Good            | Good (limited context) |
+
diff --git a/docs/serverless/serverless-security.docnav.json b/docs/serverless/serverless-security.docnav.json
@@ -30,6 +30,9 @@
         {
           "id":"serverlessSecurityAssistantAlertTriage"
         },
+        {
+          "id":"llm-performance-matrix"
+        },
         {
           "id": "serverlessSecurityConnectBedrock"
         },
-Original file line number
+Diff line change
@@ Expand Up / @@ -30,6 +30,9 @@ @@
             {
               "id":"serverlessSecurityAssistantAlertTriage"
             },
+            {
+              "id":"llm-performance-matrix"
+            },
             {
               "id": "serverlessSecurityConnectBedrock"
             },
@@ Expand Down @@