[Security solution] Fix gemini streaming #201299
Merged
+133
−14
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
stephmilovic
added
release_note:skip
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
stephmilovic
added
backport:prev-major
|
@elasticmachine merge upstream |
💛 Build succeeded, but was flaky
Failed CI StepsTest Failures
Metrics [docs]
|
patrykkopycinski
approved these changes
Nov 22, 2024
|
Starting backport for target branches: 8.15, 8.16, 8.17, 8.x |
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 22, 2024
## Summary
Fixes streaming for Gemini in Security Assistant.
Content can appear in the `finishReason` block. I'm not sure when this
started happening. Updates our streaming logic to support content being
in the `finishReason` block.
Example of `finishReason` block with content:
```
`data: {"candidates": [{"content": {"role": "model","parts": [{"text": " are 170 critical and 20 high open alerts."}]},"finishReason": "STOP","safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.060086742,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17106095},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.16776322,"severity": "HARM_SEVERITY_LOW","severityScore": 0.37113687},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.124212936,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17441037},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.05419875,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03461887}]}],"usageMetadata": {"promptTokenCount": 1062,"candidatesTokenCount": 15,"totalTokenCount": 1077},"modelVersion": "gemini-1.5-pro-002"}`
```
## To test
1. Have alerts
2. Have a Gemini connector (`gemini-1.5-pro-002`)
3. Have streaming on in the assistant with the Gemini connector selected
4. Ask the assistant: "How many open alerts do I have?"
### Previously
A response begin to streams and then the response gets cut off.
<img width="963" alt="Screenshot 2024-11-21 at 4 18 06 PM"
src="https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84">
### Now
The response streams in full as expected.
<img width="965" alt="Screenshot 2024-11-21 at 4 25 13 PM"
src="https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554">
---------
Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit 855e03e)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 22, 2024
## Summary
Fixes streaming for Gemini in Security Assistant.
Content can appear in the `finishReason` block. I'm not sure when this
started happening. Updates our streaming logic to support content being
in the `finishReason` block.
Example of `finishReason` block with content:
```
`data: {"candidates": [{"content": {"role": "model","parts": [{"text": " are 170 critical and 20 high open alerts."}]},"finishReason": "STOP","safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.060086742,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17106095},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.16776322,"severity": "HARM_SEVERITY_LOW","severityScore": 0.37113687},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.124212936,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17441037},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.05419875,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03461887}]}],"usageMetadata": {"promptTokenCount": 1062,"candidatesTokenCount": 15,"totalTokenCount": 1077},"modelVersion": "gemini-1.5-pro-002"}`
```
## To test
1. Have alerts
2. Have a Gemini connector (`gemini-1.5-pro-002`)
3. Have streaming on in the assistant with the Gemini connector selected
4. Ask the assistant: "How many open alerts do I have?"
### Previously
A response begin to streams and then the response gets cut off.
<img width="963" alt="Screenshot 2024-11-21 at 4 18 06 PM"
src="https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84">
### Now
The response streams in full as expected.
<img width="965" alt="Screenshot 2024-11-21 at 4 25 13 PM"
src="https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554">
---------
Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit 855e03e)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Nov 22, 2024
## Summary
Fixes streaming for Gemini in Security Assistant.
Content can appear in the `finishReason` block. I'm not sure when this
started happening. Updates our streaming logic to support content being
in the `finishReason` block.
Example of `finishReason` block with content:
```
`data: {"candidates": [{"content": {"role": "model","parts": [{"text": " are 170 critical and 20 high open alerts."}]},"finishReason": "STOP","safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.060086742,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17106095},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.16776322,"severity": "HARM_SEVERITY_LOW","severityScore": 0.37113687},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.124212936,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17441037},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.05419875,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03461887}]}],"usageMetadata": {"promptTokenCount": 1062,"candidatesTokenCount": 15,"totalTokenCount": 1077},"modelVersion": "gemini-1.5-pro-002"}`
```
## To test
1. Have alerts
2. Have a Gemini connector (`gemini-1.5-pro-002`)
3. Have streaming on in the assistant with the Gemini connector selected
4. Ask the assistant: "How many open alerts do I have?"
### Previously
A response begin to streams and then the response gets cut off.
<img width="963" alt="Screenshot 2024-11-21 at 4 18 06 PM"
src="https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84">
### Now
The response streams in full as expected.
<img width="965" alt="Screenshot 2024-11-21 at 4 25 13 PM"
src="https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554">
---------
Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit 855e03e)
💔 Some backports could not be created
Note: Successful backport PRs will be merged automatically after passing CI. Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
kibanamachine
added a commit
that referenced
this pull request
Nov 22, 2024
# Backport This will backport the following commits from `main` to `8.17`: - [[Security solution] Fix gemini streaming (#201299)](#201299) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Steph Milovic","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-22T12:52:40Z","message":"[Security solution] Fix gemini streaming (#201299)\n\n## Summary\r\n\r\nFixes streaming for Gemini in Security Assistant. \r\n\r\nContent can appear in the `finishReason` block. I'm not sure when this\r\nstarted happening. Updates our streaming logic to support content being\r\nin the `finishReason` block.\r\n\r\nExample of `finishReason` block with content:\r\n\r\n```\r\n`data: {\"candidates\": [{\"content\": {\"role\": \"model\",\"parts\": [{\"text\": \" are 170 critical and 20 high open alerts.\"}]},\"finishReason\": \"STOP\",\"safetyRatings\": [{\"category\": \"HARM_CATEGORY_HATE_SPEECH\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.060086742,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17106095},{\"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.16776322,\"severity\": \"HARM_SEVERITY_LOW\",\"severityScore\": 0.37113687},{\"category\": \"HARM_CATEGORY_HARASSMENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.124212936,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17441037},{\"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.05419875,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.03461887}]}],\"usageMetadata\": {\"promptTokenCount\": 1062,\"candidatesTokenCount\": 15,\"totalTokenCount\": 1077},\"modelVersion\": \"gemini-1.5-pro-002\"}`\r\n```\r\n\r\n## To test\r\n\r\n1. Have alerts\r\n2. Have a Gemini connector (`gemini-1.5-pro-002`) \r\n3. Have streaming on in the assistant with the Gemini connector selected\r\n4. Ask the assistant: \"How many open alerts do I have?\"\r\n\r\n### Previously\r\nA response begin to streams and then the response gets cut off.\r\n<img width=\"963\" alt=\"Screenshot 2024-11-21 at 4 18 06 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84\">\r\n\r\n\r\n### Now \r\nThe response streams in full as expected.\r\n<img width=\"965\" alt=\"Screenshot 2024-11-21 at 4 25 13 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554\">\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"855e03e0b61bf6d1a340dd35dd43f05f8c5709b3","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team: SecuritySolution","backport:prev-major","Team:Security Generative AI"],"title":"[Security solution] Fix gemini streaming","number":201299,"url":"https://github.com/elastic/kibana/pull/201299","mergeCommit":{"message":"[Security solution] Fix gemini streaming (#201299)\n\n## Summary\r\n\r\nFixes streaming for Gemini in Security Assistant. \r\n\r\nContent can appear in the `finishReason` block. I'm not sure when this\r\nstarted happening. Updates our streaming logic to support content being\r\nin the `finishReason` block.\r\n\r\nExample of `finishReason` block with content:\r\n\r\n```\r\n`data: {\"candidates\": [{\"content\": {\"role\": \"model\",\"parts\": [{\"text\": \" are 170 critical and 20 high open alerts.\"}]},\"finishReason\": \"STOP\",\"safetyRatings\": [{\"category\": \"HARM_CATEGORY_HATE_SPEECH\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.060086742,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17106095},{\"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.16776322,\"severity\": \"HARM_SEVERITY_LOW\",\"severityScore\": 0.37113687},{\"category\": \"HARM_CATEGORY_HARASSMENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.124212936,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17441037},{\"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.05419875,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.03461887}]}],\"usageMetadata\": {\"promptTokenCount\": 1062,\"candidatesTokenCount\": 15,\"totalTokenCount\": 1077},\"modelVersion\": \"gemini-1.5-pro-002\"}`\r\n```\r\n\r\n## To test\r\n\r\n1. Have alerts\r\n2. Have a Gemini connector (`gemini-1.5-pro-002`) \r\n3. Have streaming on in the assistant with the Gemini connector selected\r\n4. Ask the assistant: \"How many open alerts do I have?\"\r\n\r\n### Previously\r\nA response begin to streams and then the response gets cut off.\r\n<img width=\"963\" alt=\"Screenshot 2024-11-21 at 4 18 06 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84\">\r\n\r\n\r\n### Now \r\nThe response streams in full as expected.\r\n<img width=\"965\" alt=\"Screenshot 2024-11-21 at 4 25 13 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554\">\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"855e03e0b61bf6d1a340dd35dd43f05f8c5709b3"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201299","number":201299,"mergeCommit":{"message":"[Security solution] Fix gemini streaming (#201299)\n\n## Summary\r\n\r\nFixes streaming for Gemini in Security Assistant. \r\n\r\nContent can appear in the `finishReason` block. I'm not sure when this\r\nstarted happening. Updates our streaming logic to support content being\r\nin the `finishReason` block.\r\n\r\nExample of `finishReason` block with content:\r\n\r\n```\r\n`data: {\"candidates\": [{\"content\": {\"role\": \"model\",\"parts\": [{\"text\": \" are 170 critical and 20 high open alerts.\"}]},\"finishReason\": \"STOP\",\"safetyRatings\": [{\"category\": \"HARM_CATEGORY_HATE_SPEECH\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.060086742,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17106095},{\"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.16776322,\"severity\": \"HARM_SEVERITY_LOW\",\"severityScore\": 0.37113687},{\"category\": \"HARM_CATEGORY_HARASSMENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.124212936,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17441037},{\"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.05419875,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.03461887}]}],\"usageMetadata\": {\"promptTokenCount\": 1062,\"candidatesTokenCount\": 15,\"totalTokenCount\": 1077},\"modelVersion\": \"gemini-1.5-pro-002\"}`\r\n```\r\n\r\n## To test\r\n\r\n1. Have alerts\r\n2. Have a Gemini connector (`gemini-1.5-pro-002`) \r\n3. Have streaming on in the assistant with the Gemini connector selected\r\n4. Ask the assistant: \"How many open alerts do I have?\"\r\n\r\n### Previously\r\nA response begin to streams and then the response gets cut off.\r\n<img width=\"963\" alt=\"Screenshot 2024-11-21 at 4 18 06 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84\">\r\n\r\n\r\n### Now \r\nThe response streams in full as expected.\r\n<img width=\"965\" alt=\"Screenshot 2024-11-21 at 4 25 13 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554\">\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"855e03e0b61bf6d1a340dd35dd43f05f8c5709b3"}}]}] BACKPORT--> Co-authored-by: Steph Milovic <[email protected]>
kibanamachine
added a commit
that referenced
this pull request
Nov 22, 2024
# Backport This will backport the following commits from `main` to `8.x`: - [[Security solution] Fix gemini streaming (#201299)](#201299) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Steph Milovic","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-11-22T12:52:40Z","message":"[Security solution] Fix gemini streaming (#201299)\n\n## Summary\r\n\r\nFixes streaming for Gemini in Security Assistant. \r\n\r\nContent can appear in the `finishReason` block. I'm not sure when this\r\nstarted happening. Updates our streaming logic to support content being\r\nin the `finishReason` block.\r\n\r\nExample of `finishReason` block with content:\r\n\r\n```\r\n`data: {\"candidates\": [{\"content\": {\"role\": \"model\",\"parts\": [{\"text\": \" are 170 critical and 20 high open alerts.\"}]},\"finishReason\": \"STOP\",\"safetyRatings\": [{\"category\": \"HARM_CATEGORY_HATE_SPEECH\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.060086742,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17106095},{\"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.16776322,\"severity\": \"HARM_SEVERITY_LOW\",\"severityScore\": 0.37113687},{\"category\": \"HARM_CATEGORY_HARASSMENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.124212936,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17441037},{\"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.05419875,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.03461887}]}],\"usageMetadata\": {\"promptTokenCount\": 1062,\"candidatesTokenCount\": 15,\"totalTokenCount\": 1077},\"modelVersion\": \"gemini-1.5-pro-002\"}`\r\n```\r\n\r\n## To test\r\n\r\n1. Have alerts\r\n2. Have a Gemini connector (`gemini-1.5-pro-002`) \r\n3. Have streaming on in the assistant with the Gemini connector selected\r\n4. Ask the assistant: \"How many open alerts do I have?\"\r\n\r\n### Previously\r\nA response begin to streams and then the response gets cut off.\r\n<img width=\"963\" alt=\"Screenshot 2024-11-21 at 4 18 06 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84\">\r\n\r\n\r\n### Now \r\nThe response streams in full as expected.\r\n<img width=\"965\" alt=\"Screenshot 2024-11-21 at 4 25 13 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554\">\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"855e03e0b61bf6d1a340dd35dd43f05f8c5709b3","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","Team: SecuritySolution","backport:prev-major","Team:Security Generative AI"],"title":"[Security solution] Fix gemini streaming","number":201299,"url":"https://github.com/elastic/kibana/pull/201299","mergeCommit":{"message":"[Security solution] Fix gemini streaming (#201299)\n\n## Summary\r\n\r\nFixes streaming for Gemini in Security Assistant. \r\n\r\nContent can appear in the `finishReason` block. I'm not sure when this\r\nstarted happening. Updates our streaming logic to support content being\r\nin the `finishReason` block.\r\n\r\nExample of `finishReason` block with content:\r\n\r\n```\r\n`data: {\"candidates\": [{\"content\": {\"role\": \"model\",\"parts\": [{\"text\": \" are 170 critical and 20 high open alerts.\"}]},\"finishReason\": \"STOP\",\"safetyRatings\": [{\"category\": \"HARM_CATEGORY_HATE_SPEECH\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.060086742,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17106095},{\"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.16776322,\"severity\": \"HARM_SEVERITY_LOW\",\"severityScore\": 0.37113687},{\"category\": \"HARM_CATEGORY_HARASSMENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.124212936,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17441037},{\"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.05419875,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.03461887}]}],\"usageMetadata\": {\"promptTokenCount\": 1062,\"candidatesTokenCount\": 15,\"totalTokenCount\": 1077},\"modelVersion\": \"gemini-1.5-pro-002\"}`\r\n```\r\n\r\n## To test\r\n\r\n1. Have alerts\r\n2. Have a Gemini connector (`gemini-1.5-pro-002`) \r\n3. Have streaming on in the assistant with the Gemini connector selected\r\n4. Ask the assistant: \"How many open alerts do I have?\"\r\n\r\n### Previously\r\nA response begin to streams and then the response gets cut off.\r\n<img width=\"963\" alt=\"Screenshot 2024-11-21 at 4 18 06 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84\">\r\n\r\n\r\n### Now \r\nThe response streams in full as expected.\r\n<img width=\"965\" alt=\"Screenshot 2024-11-21 at 4 25 13 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554\">\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"855e03e0b61bf6d1a340dd35dd43f05f8c5709b3"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201299","number":201299,"mergeCommit":{"message":"[Security solution] Fix gemini streaming (#201299)\n\n## Summary\r\n\r\nFixes streaming for Gemini in Security Assistant. \r\n\r\nContent can appear in the `finishReason` block. I'm not sure when this\r\nstarted happening. Updates our streaming logic to support content being\r\nin the `finishReason` block.\r\n\r\nExample of `finishReason` block with content:\r\n\r\n```\r\n`data: {\"candidates\": [{\"content\": {\"role\": \"model\",\"parts\": [{\"text\": \" are 170 critical and 20 high open alerts.\"}]},\"finishReason\": \"STOP\",\"safetyRatings\": [{\"category\": \"HARM_CATEGORY_HATE_SPEECH\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.060086742,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17106095},{\"category\": \"HARM_CATEGORY_DANGEROUS_CONTENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.16776322,\"severity\": \"HARM_SEVERITY_LOW\",\"severityScore\": 0.37113687},{\"category\": \"HARM_CATEGORY_HARASSMENT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.124212936,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.17441037},{\"category\": \"HARM_CATEGORY_SEXUALLY_EXPLICIT\",\"probability\": \"NEGLIGIBLE\",\"probabilityScore\": 0.05419875,\"severity\": \"HARM_SEVERITY_NEGLIGIBLE\",\"severityScore\": 0.03461887}]}],\"usageMetadata\": {\"promptTokenCount\": 1062,\"candidatesTokenCount\": 15,\"totalTokenCount\": 1077},\"modelVersion\": \"gemini-1.5-pro-002\"}`\r\n```\r\n\r\n## To test\r\n\r\n1. Have alerts\r\n2. Have a Gemini connector (`gemini-1.5-pro-002`) \r\n3. Have streaming on in the assistant with the Gemini connector selected\r\n4. Ask the assistant: \"How many open alerts do I have?\"\r\n\r\n### Previously\r\nA response begin to streams and then the response gets cut off.\r\n<img width=\"963\" alt=\"Screenshot 2024-11-21 at 4 18 06 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84\">\r\n\r\n\r\n### Now \r\nThe response streams in full as expected.\r\n<img width=\"965\" alt=\"Screenshot 2024-11-21 at 4 25 13 PM\"\r\nsrc=\"https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554\">\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"855e03e0b61bf6d1a340dd35dd43f05f8c5709b3"}}]}] BACKPORT--> Co-authored-by: Steph Milovic <[email protected]>
kibanamachine
added
v8.18.0
backport missing
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
1 similar comment
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
paulinashakirova
pushed a commit
to paulinashakirova/kibana
that referenced
this pull request
Nov 26, 2024
## Summary
Fixes streaming for Gemini in Security Assistant.
Content can appear in the `finishReason` block. I'm not sure when this
started happening. Updates our streaming logic to support content being
in the `finishReason` block.
Example of `finishReason` block with content:
```
`data: {"candidates": [{"content": {"role": "model","parts": [{"text": " are 170 critical and 20 high open alerts."}]},"finishReason": "STOP","safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.060086742,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17106095},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.16776322,"severity": "HARM_SEVERITY_LOW","severityScore": 0.37113687},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.124212936,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17441037},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.05419875,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03461887}]}],"usageMetadata": {"promptTokenCount": 1062,"candidatesTokenCount": 15,"totalTokenCount": 1077},"modelVersion": "gemini-1.5-pro-002"}`
```
## To test
1. Have alerts
2. Have a Gemini connector (`gemini-1.5-pro-002`)
3. Have streaming on in the assistant with the Gemini connector selected
4. Ask the assistant: "How many open alerts do I have?"
### Previously
A response begin to streams and then the response gets cut off.
<img width="963" alt="Screenshot 2024-11-21 at 4 18 06 PM"
src="https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84">
### Now
The response streams in full as expected.
<img width="965" alt="Screenshot 2024-11-21 at 4 25 13 PM"
src="https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554">
---------
Co-authored-by: Elastic Machine <[email protected]>
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
3 similar comments
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
|
Looks like this PR has backport PRs but they still haven't been merged. Please merge them ASAP to keep the branches relatively in sync. |
stephmilovic
pushed a commit
that referenced
this pull request
Dec 2, 2024
kibanamachine
added
v8.16.2
and removed
backport missing
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this pull request
Dec 12, 2024
## Summary
Fixes streaming for Gemini in Security Assistant.
Content can appear in the `finishReason` block. I'm not sure when this
started happening. Updates our streaming logic to support content being
in the `finishReason` block.
Example of `finishReason` block with content:
```
`data: {"candidates": [{"content": {"role": "model","parts": [{"text": " are 170 critical and 20 high open alerts."}]},"finishReason": "STOP","safetyRatings": [{"category": "HARM_CATEGORY_HATE_SPEECH","probability": "NEGLIGIBLE","probabilityScore": 0.060086742,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17106095},{"category": "HARM_CATEGORY_DANGEROUS_CONTENT","probability": "NEGLIGIBLE","probabilityScore": 0.16776322,"severity": "HARM_SEVERITY_LOW","severityScore": 0.37113687},{"category": "HARM_CATEGORY_HARASSMENT","probability": "NEGLIGIBLE","probabilityScore": 0.124212936,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.17441037},{"category": "HARM_CATEGORY_SEXUALLY_EXPLICIT","probability": "NEGLIGIBLE","probabilityScore": 0.05419875,"severity": "HARM_SEVERITY_NEGLIGIBLE","severityScore": 0.03461887}]}],"usageMetadata": {"promptTokenCount": 1062,"candidatesTokenCount": 15,"totalTokenCount": 1077},"modelVersion": "gemini-1.5-pro-002"}`
```
## To test
1. Have alerts
2. Have a Gemini connector (`gemini-1.5-pro-002`)
3. Have streaming on in the assistant with the Gemini connector selected
4. Ask the assistant: "How many open alerts do I have?"
### Previously
A response begin to streams and then the response gets cut off.
<img width="963" alt="Screenshot 2024-11-21 at 4 18 06 PM"
src="https://github.com/user-attachments/assets/3f25b682-64ea-4878-83de-17180886eb84">
### Now
The response streams in full as expected.
<img width="965" alt="Screenshot 2024-11-21 at 4 25 13 PM"
src="https://github.com/user-attachments/assets/162a6ccc-08c5-4108-bb1e-5df2b37ce554">
---------
Co-authored-by: Elastic Machine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes streaming for Gemini in Security Assistant.
Content can appear in the
finishReasonblock. I'm not sure when this started happening. Updates our streaming logic to support content being in thefinishReasonblock.Example of
finishReasonblock with content:To test
gemini-1.5-pro-002)Previously
A response begin to streams and then the response gets cut off.
Now
The response streams in full as expected.
