Adds LLM performance matrix doc to ESS & serverless

elastic · May 29, 2024 · 1430ae4 · 1430ae4
1 parent cdd1e6e
commit 1430ae4
Show file tree

Hide file tree

Showing 4 changed files with 35 additions and 0 deletions.
diff --git a/docs/assistant/llm-performance-matrix.asciidoc b/docs/assistant/llm-performance-matrix.asciidoc
@@ -0,0 +1,14 @@
+[[llm-performance-matrix]]
+= Large language model performance matrix
+
+This table describes the performance of various LLMs for different use-cases in {elastic-sec}, based on our internal testing. To learn more about how to implement these use-cases, refer to <<attack-discovery>> or <<security-assistant>>.
+
+[cols="1,1,1,1,1,1,1", options="header"]
+|===
+| Feature/Model                | Claude 3: Opus  | Claude 3: Sonnet  | Claude 3: Haiku  | GPT-4o  | GPT-4 Turbo  | GPT-4 32K
+
+| Assistant - General          | Excellent       | Excellent          | Excellent        | Excellent | Excellent   | Excellent
+| Assistant - {{esql}} generation | Great           | Great              | Poor             | Excellent | Poor       | Excellent
+| Assistant - Alert questions  | Excellent       | Excellent          | Excellent        | Excellent | Poor       | Good (limited context)
+| Attack Discovery             | Excellent       | Great              | Poor             | Poor     | Good       | Good (limited context)
+|===
diff --git a/docs/assistant/security-assistant.asciidoc b/docs/assistant/security-assistant.asciidoc
@@ -194,6 +194,7 @@ In addition to practical advice, AI Assistant can offer conceptual advice, tips,
 
 
 include::ai-alert-triage.asciidoc[leveloffset=+1]
+include::llm-performance-matrix.asciidoc[leveloffset=+1]
 include::azure-openai-setup.asciidoc[leveloffset=+1]
 include::connect-to-openai.asciidoc[leveloffset=+1]
 include::connect-to-bedrock.asciidoc[leveloffset=+1]
diff --git a/docs/serverless/assistant/llm-performance-matrix.mdx b/docs/serverless/assistant/llm-performance-matrix.mdx
@@ -0,0 +1,17 @@
+---
+id: llm-performance-matrix
+slug: /serverless/security/llm-performance-matrix
+title: Large language model performance matrix
+description: Learn how different models perform on different tasks in ((elastic-sec)).
+tags: ["security", "overview", "get-started"]
+status: in review
+---
+
+This table describes the performance of various LLMs for different use-cases in ((elastic-sec)), based on our internal testing. To learn more about how to implement these use-cases, refer to <DocLink id="attackDiscovery" text="Attack discovery"/> or <DocLink id="serverlessSecurityAIAssistant" text="AI Assistant"/>.
+
+| Feature/Model                | Claude 3: Opus  | Claude 3: Sonnet  | Claude 3: Haiku  | GPT-4o  | GPT-4 Turbo  | GPT-4 32K |
+|------------------------------|-----------------|-------------------|------------------|---------|--------------|-----------|
+| Assistant - General          | Excellent       | Excellent         | Excellent        | Excellent| Excellent    | Excellent |
+| Assistant - ((esql)) generation| Great           | Great             | Poor             | Excellent| Poor        | Excellent |
+| Assistant - Alert questions  | Excellent       | Excellent         | Excellent        | Excellent| Poor        | Good (limited context) |
+| Attack Discovery             | Excellent       | Great             | Poor             | Poor     | Good        | Good (limited context) |
diff --git a/docs/serverless/serverless-security.docnav.json b/docs/serverless/serverless-security.docnav.json
@@ -30,6 +30,9 @@
         {
           "id":"serverlessSecurityAssistantAlertTriage"
         },
+        {
+          "id":"llm-performance-matrix"
+        },
         {
           "id": "serverlessSecurityConnectBedrock"
         },
-Original file line number
+Diff line change
@@ Expand Up / @@ -30,6 +30,9 @@ @@
             {
               "id":"serverlessSecurityAssistantAlertTriage"
             },
+            {
+              "id":"llm-performance-matrix"
+            },
             {
               "id": "serverlessSecurityConnectBedrock"
             },
@@ Expand Down @@