From 354bec4ca4587a8d6562138fcae79bbc7d32ade6 Mon Sep 17 00:00:00 2001 From: Viduni Wickramarachchi Date: Wed, 30 Oct 2024 12:48:04 -0400 Subject: [PATCH] [Obs AI Assistant] Fix chat on the Alerts page (#197126) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes https://github.com/elastic/kibana/issues/184214 ## Summary ### Problem The Observability AI Assistant doesn't work on the Alerts page - errors out with a 400 status code from OpenAI. The reason for this is that the description for the function `get_data_on_screen` is too long, and there is a token limit for function descriptions by OpenAI. **Note:** This error does not occur with Gemini or Bedrock because simulated function calling is enabled by default for them. With simulated function calling, all functions and their descriptions are appended to the system message, therefore doesn't run into a token limit error as opposed to OpenAI. ### Solution Append the function description to the system message instead of sending it with the function. The implementation includes: - Registering an AdHoc instruction - Retrieving AdHoc instructions - Combine the retrieved AdHoc instructions with the other adHoc instructions passed to the chat and pass all AdHoc instructions to the `getSystemMessageFromInstructions` function. This correctly orders the description for the `get_data_on_screen` function at the end of the system message _OpenAI request object **before** the above update:_
Click to expand JSON ```json { "messages": [ { "role": "system", "content": "You are a helpful assistant for Elastic Observability. Your goal is to help the Elastic Observability users to quickly assess what is happening in their observed systems. You can help them visualise and analyze data, investigate their systems, perform root cause analysis or identify optimisation opportunities.\n\n It's very important to not assume what the user is meaning. Ask them for clarification if needed.\n\n If you are unsure about which function should be used and with what arguments, ask the user for clarification or confirmation.\n\n In KQL (\"kqlFilter\")) escaping happens with double quotes, not single quotes. Some characters that need escaping are: ':()\\ /\". Always put a field value in double quotes. Best: service.name:\"opbeans-go\". Wrong: service.name:opbeans-go. This is very important!\n\n You can use Github-flavored Markdown in your responses. If a function returns an array, consider using a Markdown table to format the response.\n\n Note that ES|QL (the Elasticsearch Query Language which is a new piped language) is the preferred query language.\n\n If you want to call a function or tool, only call it a single time per message. Wait until the function has been executed and its results\n returned to you, before executing the same tool or another tool again if needed.\n\n DO NOT UNDER ANY CIRCUMSTANCES USE ES|QL syntax (`service.name == \"foo\"`) with \"kqlFilter\" (`service.name:\"foo\"`).\n\n The user is able to change the language which they want you to reply in on the settings page of the AI Assistant for Observability, which can be found in the Stack Management app under the option AI Assistants.\n If the user asks how to change the language, reply in the same language the user asked in.\n\nYou MUST use the \"query\" function when the user wants to:\n - visualize data\n - run any arbitrary query\n - breakdown or filter ES|QL queries that are displayed on the current page\n - convert queries from another language to ES|QL\n - asks general questions about ES|QL\n\n DO NOT UNDER ANY CIRCUMSTANCES generate ES|QL queries or explain anything about the ES|QL query language yourself.\n DO NOT UNDER ANY CIRCUMSTANCES try to correct an ES|QL query yourself - always use the \"query\" function for this.\n\n If the user asks for a query, and one of the dataset info functions was called and returned no results, you should still call the query function to generate an example query.\n\n Even if the \"query\" function was used before that, follow it up with the \"query\" function. If a query fails, do not attempt to correct it yourself. Again you should call the \"query\" function,\n even if it has been called before.\n\n When the \"visualize_query\" function has been called, a visualization has been displayed to the user. DO NOT UNDER ANY CIRCUMSTANCES follow up a \"visualize_query\" function call with your own visualization attempt.\n If the \"execute_query\" function has been called, summarize these results for the user. The user does not see a visualization in this case.\n\nYou MUST use the \"get_dataset_info\" function before calling the \"query\" or the \"changes\" functions.\n\nIf a function requires an index, you MUST use the results from the dataset info functions.\n\nYou have access to data on the screen by calling the \"get_data_on_screen\" function.\nUse it to help the user understand what they are looking at. A short summary of what they are looking at is available in the return of the \"context\" function.\nData that is compact enough automatically gets included in the response for the \"context\" function.\n\nYou can use the \"summarize\" function to store new information you have learned in a knowledge database.\nOnly use this function when the user asks for it.\nAll summaries MUST be created in English, even if the conversation was carried out in a different language." }, { "role": "user", "content": "Can you explain this page?" }, { "role": "assistant", "content": "", "function_call": { "name": "context", "arguments": "{}" } }, { "role": "user", "content": "{\"screen_description\":\"The user is looking at http://localhost:5601/phq/app/observability/alerts?_a=(filters:!(),kuery:%27%27,rangeFrom:now-24h,rangeTo:now,status:all). The current time range is 2024-10-21T18:23:54.539Z - 2024-10-21T18:38:54.539Z.\",\"learnings\":[],\"data_on_screen\":[{\"name\":\".es-query\",\"value\":\"Elasticsearch query Alert when matches are found during the latest query run.\",\"description\":\"An available rule is Elasticsearch query.\"},{\"name\":\"observability.rules.custom_threshold\",\"value\":\"Custom threshold Alert when any Observability data type reaches or exceeds a given value.\",\"description\":\"An available rule is Custom threshold.\"},{\"name\":\"xpack.ml.anomaly_detection_alert\",\"value\":\"Anomaly detection Alert when anomaly detection jobs results match the condition.\",\"description\":\"An available rule is Anomaly detection.\"},{\"name\":\"slo.rules.burnRate\",\"value\":\"SLO burn rate Alert when your SLO burn rate is too high over a defined period of time.\",\"description\":\"An available rule is SLO burn rate.\"},{\"name\":\"metrics.alert.threshold\",\"value\":\"Metric threshold Alert when the metrics aggregation exceeds the threshold.\",\"description\":\"An available rule is Metric threshold.\"},{\"name\":\"metrics.alert.inventory.threshold\",\"value\":\"Inventory Alert when the inventory exceeds a defined threshold.\",\"description\":\"An available rule is Inventory.\"},{\"name\":\"logs.alert.document.count\",\"value\":\"Log threshold Alert when the log aggregation exceeds the threshold.\",\"description\":\"An available rule is Log threshold.\"},{\"name\":\"xpack.uptime.alerts.tlsCertificate\",\"value\":\"Uptime TLS Alert when the TLS certificate of an Uptime monitor is about to expire.\",\"description\":\"An available rule is Uptime TLS.\"},{\"name\":\"xpack.uptime.alerts.monitorStatus\",\"value\":\"Uptime monitor status Alert when a monitor is down or an availability threshold is breached.\",\"description\":\"An available rule is Uptime monitor status.\"},{\"name\":\"xpack.uptime.alerts.durationAnomaly\",\"value\":\"Uptime Duration Anomaly Alert when the Uptime monitor duration is anomalous.\",\"description\":\"An available rule is Uptime Duration Anomaly.\"},{\"name\":\"xpack.synthetics.alerts.monitorStatus\",\"value\":\"Synthetics monitor status Alert when a monitor is down.\",\"description\":\"An available rule is Synthetics monitor status.\"},{\"name\":\"xpack.synthetics.alerts.tls\",\"value\":\"Synthetics TLS certificate Alert when the TLS certificate of a Synthetics monitor is about to expire.\",\"description\":\"An available rule is Synthetics TLS certificate.\"},{\"name\":\"apm.error_rate\",\"value\":\"Error count threshold Alert when the number of errors in a service exceeds a defined threshold.\",\"description\":\"An available rule is Error count threshold.\"},{\"name\":\"apm.transaction_error_rate\",\"value\":\"Failed transaction rate threshold Alert when the rate of transaction errors in a service exceeds a defined threshold.\",\"description\":\"An available rule is Failed transaction rate threshold.\"},{\"name\":\"apm.transaction_duration\",\"value\":\"Latency threshold Alert when the latency of a specific transaction type in a service exceeds a defined threshold.\",\"description\":\"An available rule is Latency threshold.\"},{\"name\":\"apm.anomaly\",\"value\":\"APM Anomaly Alert when either the latency, throughput, or failed transaction rate of a service is anomalous.\",\"description\":\"An available rule is APM Anomaly.\"}]}", "name": "context" } ], "stream": true, "tools": [ { "function": { "name": "get_data_on_screen", "description": "Get data that is on the screen:\n.es-query: An available rule is Elasticsearch query.\nobservability.rules.custom_threshold: An available rule is Custom threshold.\nxpack.ml.anomaly_detection_alert: An available rule is Anomaly detection.\nslo.rules.burnRate: An available rule is SLO burn rate.\nmetrics.alert.threshold: An available rule is Metric threshold.\nmetrics.alert.inventory.threshold: An available rule is Inventory.\nlogs.alert.document.count: An available rule is Log threshold.\nxpack.uptime.alerts.tlsCertificate: An available rule is Uptime TLS.\nxpack.uptime.alerts.monitorStatus: An available rule is Uptime monitor status.\nxpack.uptime.alerts.durationAnomaly: An available rule is Uptime Duration Anomaly.\nxpack.synthetics.alerts.monitorStatus: An available rule is Synthetics monitor status.\nxpack.synthetics.alerts.tls: An available rule is Synthetics TLS certificate.\napm.error_rate: An available rule is Error count threshold.\napm.transaction_error_rate: An available rule is Failed transaction rate threshold.\napm.transaction_duration: An available rule is Latency threshold.\napm.anomaly: An available rule is APM Anomaly.", "parameters": { "type": "object", "properties": { "data": { "type": "array", "description": "The pieces of data you want to look at it. You can request one, or multiple", "items": { "type": "string", "enum": [ ".es-query", "observability.rules.custom_threshold", "xpack.ml.anomaly_detection_alert", "slo.rules.burnRate", "metrics.alert.threshold", "metrics.alert.inventory.threshold", "logs.alert.document.count", "xpack.uptime.alerts.tlsCertificate", "xpack.uptime.alerts.monitorStatus", "xpack.uptime.alerts.durationAnomaly", "xpack.synthetics.alerts.monitorStatus", "xpack.synthetics.alerts.tls", "apm.error_rate", "apm.transaction_error_rate", "apm.transaction_duration", "apm.anomaly" ] } } }, "required": ["data"] } }, "type": "function" }, { "function": { "name": "query", "description": "This function generates, executes and/or visualizes a query\n based on the user's request. It also explains how ES|QL works and how to\n convert queries from one language to another. Make sure you call one of\n the get_dataset functions first if you need index or field names. This\n function takes no input.", "parameters": { "type": "object", "properties": {} } }, "type": "function" }, { "function": { "name": "get_alerts_dataset_info", "description": "Use this function to get information about alerts data.", "parameters": { "type": "object", "properties": { "start": { "type": "string", "description": "The start of the current time range, in datemath, like now-24h or an ISO timestamp" }, "end": { "type": "string", "description": "The end of the current time range, in datemath, like now-24h or an ISO timestamp" } } } }, "type": "function" }, { "function": { "name": "alerts", "description": "Get alerts for Observability. Make sure get_alerts_dataset_info was called before.\n Use this to get open (and optionally recovered) alerts for Observability assets, like services,\n hosts or containers.\n Display the response in tabular format if appropriate.\n ", "parameters": { "type": "object", "properties": { "start": { "type": "string", "description": "The start of the time range, in Elasticsearch date math, like `now`." }, "end": { "type": "string", "description": "The end of the time range, in Elasticsearch date math, like `now-24h`." }, "kqlFilter": { "type": "string", "description": "Filter alerts by field:value pairs" }, "includeRecovered": { "type": "boolean", "description": "Whether to include recovered/closed alerts. Defaults to false, which means only active alerts will be returned" } }, "required": ["start", "end"] } }, "type": "function" }, { "function": { "name": "changes", "description": "Returns change points like spikes and dips for logs and metrics.", "parameters": { "type": "object", "properties": { "start": { "type": "string", "description": "The beginning of the time range, in datemath, like now-24h, or an ISO timestamp" }, "end": { "type": "string", "description": "The end of the time range, in datemath, like now, or an ISO timestamp" }, "logs": { "description": "Analyze changes in log patterns. If no index is given, the default logs index pattern will be used", "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string", "description": "The name of this set of logs" }, "index": { "type": "string", "description": "The index or index pattern where to find the logs" }, "kqlFilter": { "type": "string", "description": "A KQL filter to filter the log documents by, e.g. my_field:foo" }, "field": { "type": "string", "description": "The text field that contains the message to be analyzed, usually `message`. ONLY use field names from the conversation." } }, "required": ["name"] } }, "metrics": { "description": "Analyze changes in metrics. DO NOT UNDER ANY CIRCUMSTANCES use date or metric fields for groupBy, leave empty unless needed.", "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string", "description": "The name of this set of metrics" }, "index": { "type": "string", "description": "The index or index pattern where to find the metrics" }, "kqlFilter": { "type": "string", "description": "A KQL filter to filter the log documents by, e.g. my_field:foo" }, "field": { "type": "string", "description": "Metric field that contains the metric. Only use if the metric aggregation type is not count." }, "type": { "type": "string", "description": "The type of metric aggregation to perform. Defaults to count", "enum": ["count", "avg", "sum", "min", "max", "p95", "p99"] }, "groupBy": { "type": "array", "description": "Optional keyword fields to group metrics by.", "items": { "type": "string" } } }, "required": ["index", "name"] } } }, "required": ["start", "end"] } }, "type": "function" }, { "function": { "name": "summarize", "description": "Use this function to store facts in the knowledge database if the user requests it.\n You can score the learnings with a confidence metric, whether it is a correction on a previous learning.\n An embedding will be created that you can recall later with a semantic search.\n When you create this summarisation, make sure you craft it in a way that can be recalled with a semantic\n search later, and that it would have answered the user's original request.", "parameters": { "type": "object", "properties": { "id": { "type": "string", "description": "An id for the document. This should be a short human-readable keyword field with only alphabetic characters and underscores, that allow you to update it later." }, "text": { "type": "string", "description": "A human-readable summary of what you have learned, described in such a way that you can recall it later with semantic search, and that it would have answered the user's original request." }, "is_correction": { "type": "boolean", "description": "Whether this is a correction for a previous learning." }, "confidence": { "type": "string", "description": "How confident you are about this being a correct and useful learning", "enum": ["low", "medium", "high"] }, "public": { "type": "boolean", "description": "Whether this information is specific to the user, or generally applicable to any user of the product" } }, "required": ["id", "text", "is_correction", "confidence", "public"] } }, "type": "function" }, { "function": { "name": "elasticsearch", "description": "Call Elasticsearch APIs on behalf of the user. Make sure the request body is valid for the API that you are using. Only call this function when the user has explicitly requested it.", "parameters": { "type": "object", "properties": { "method": { "type": "string", "description": "The HTTP method of the Elasticsearch endpoint", "enum": ["GET", "PUT", "POST", "DELETE", "PATCH"] }, "path": { "type": "string", "description": "The path of the Elasticsearch endpoint, including query parameters" }, "body": { "type": "object", "description": "The body of the request" } }, "required": ["method", "path"] } }, "type": "function" }, { "function": { "name": "kibana", "description": "Call Kibana APIs on behalf of the user. Only call this function when the user has explicitly requested it, and you know how to call it, for example by querying the knowledge base or having the user explain it to you. Assume that pathnames, bodies and query parameters may have changed since your knowledge cut off date.", "parameters": { "type": "object", "properties": { "method": { "type": "string", "description": "The HTTP method of the Kibana endpoint", "enum": ["GET", "PUT", "POST", "DELETE", "PATCH"] }, "pathname": { "type": "string", "description": "The pathname of the Kibana endpoint, excluding query parameters" }, "query": { "type": "object", "description": "The query parameters, as an object" }, "body": { "type": "object", "description": "The body of the request" } }, "required": ["method", "pathname"] } }, "type": "function" }, { "function": { "name": "get_dataset_info", "description": "Use this function to get information about indices/datasets available and the fields available on them.\n\n providing empty string as index name will retrieve all indices\n else list of all fields for the given index will be given. if no fields are returned this means no indices were matched by provided index pattern.\n wildcards can be part of index name.", "parameters": { "type": "object", "properties": { "index": { "type": "string", "description": "index pattern the user is interested in or empty string to get information about all available indices" } }, "required": ["index"] } }, "type": "function" }, { "function": { "name": "execute_connector", "description": "Use this function when user explicitly asks to call a kibana connector.", "parameters": { "type": "object", "properties": { "id": { "type": "string", "description": "The id of the connector" }, "params": { "type": "object", "description": "The connector parameters" } }, "required": ["id", "params"] } }, "type": "function" } ], "temperature": 0 } ```
_OpenAI request object **after** the above update:_
Click to expand JSON ```json { "messages": [ { "role": "system", "content": "You are a helpful assistant for Elastic Observability. Your goal is to help the Elastic Observability users to quickly assess what is happening in their observed systems. You can help them visualise and analyze data, investigate their systems, perform root cause analysis or identify optimisation opportunities.\n\n It's very important to not assume what the user is meaning. Ask them for clarification if needed.\n\n If you are unsure about which function should be used and with what arguments, ask the user for clarification or confirmation.\n\n In KQL (\"kqlFilter\")) escaping happens with double quotes, not single quotes. Some characters that need escaping are: ':()\\ /\". Always put a field value in double quotes. Best: service.name:\"opbeans-go\". Wrong: service.name:opbeans-go. This is very important!\n\n You can use Github-flavored Markdown in your responses. If a function returns an array, consider using a Markdown table to format the response.\n\n Note that ES|QL (the Elasticsearch Query Language which is a new piped language) is the preferred query language.\n\n If you want to call a function or tool, only call it a single time per message. Wait until the function has been executed and its results\n returned to you, before executing the same tool or another tool again if needed.\n\n DO NOT UNDER ANY CIRCUMSTANCES USE ES|QL syntax (`service.name == \"foo\"`) with \"kqlFilter\" (`service.name:\"foo\"`).\n\n The user is able to change the language which they want you to reply in on the settings page of the AI Assistant for Observability, which can be found in the Stack Management app under the option AI Assistants.\n If the user asks how to change the language, reply in the same language the user asked in.\n\nYou MUST use the \"query\" function when the user wants to:\n - visualize data\n - run any arbitrary query\n - breakdown or filter ES|QL queries that are displayed on the current page\n - convert queries from another language to ES|QL\n - asks general questions about ES|QL\n\n DO NOT UNDER ANY CIRCUMSTANCES generate ES|QL queries or explain anything about the ES|QL query language yourself.\n DO NOT UNDER ANY CIRCUMSTANCES try to correct an ES|QL query yourself - always use the \"query\" function for this.\n\n If the user asks for a query, and one of the dataset info functions was called and returned no results, you should still call the query function to generate an example query.\n\n Even if the \"query\" function was used before that, follow it up with the \"query\" function. If a query fails, do not attempt to correct it yourself. Again you should call the \"query\" function,\n even if it has been called before.\n\n When the \"visualize_query\" function has been called, a visualization has been displayed to the user. DO NOT UNDER ANY CIRCUMSTANCES follow up a \"visualize_query\" function call with your own visualization attempt.\n If the \"execute_query\" function has been called, summarize these results for the user. The user does not see a visualization in this case.\n\nYou MUST use the \"get_dataset_info\" function before calling the \"query\" or the \"changes\" functions.\n\nIf a function requires an index, you MUST use the results from the dataset info functions.\n\nYou have access to data on the screen by calling the \"get_data_on_screen\" function.\nUse it to help the user understand what they are looking at. A short summary of what they are looking at is available in the return of the \"context\" function.\nData that is compact enough automatically gets included in the response for the \"context\" function.\n\nYou can use the \"summarize\" function to store new information you have learned in a knowledge database.\nOnly use this function when the user asks for it.\nAll summaries MUST be created in English, even if the conversation was carried out in a different language.\nThe \"get_data_on_screen\" function will Get data that is on the screen:\n.es-query: An available rule is Elasticsearch query.\nobservability.rules.custom_threshold: An available rule is Custom threshold.\nxpack.ml.anomaly_detection_alert: An available rule is Anomaly detection.\nslo.rules.burnRate: An available rule is SLO burn rate.\nmetrics.alert.threshold: An available rule is Metric threshold.\nmetrics.alert.inventory.threshold: An available rule is Inventory.\nlogs.alert.document.count: An available rule is Log threshold.\nxpack.uptime.alerts.tlsCertificate: An available rule is Uptime TLS.\nxpack.uptime.alerts.monitorStatus: An available rule is Uptime monitor status.\nxpack.uptime.alerts.durationAnomaly: An available rule is Uptime Duration Anomaly.\nxpack.synthetics.alerts.monitorStatus: An available rule is Synthetics monitor status.\nxpack.synthetics.alerts.tls: An available rule is Synthetics TLS certificate.\napm.error_rate: An available rule is Error count threshold.\napm.transaction_error_rate: An available rule is Failed transaction rate threshold.\napm.transaction_duration: An available rule is Latency threshold.\napm.anomaly: An available rule is APM Anomaly." }, { "role": "user", "content": "Can you explain this page?" }, { "role": "assistant", "content": "", "function_call": { "name": "context", "arguments": "{}" } }, { "role": "user", "content": "{\"screen_description\":\"The user is looking at http://localhost:5601/phq/app/observability/alerts?_a=(filters:!(),kuery:%27%27,rangeFrom:now-24h,rangeTo:now,status:all). The current time range is 2024-10-21T18:23:54.539Z - 2024-10-21T18:38:54.539Z.\",\"learnings\":[],\"data_on_screen\":[{\"name\":\".es-query\",\"value\":\"Elasticsearch query Alert when matches are found during the latest query run.\",\"description\":\"An available rule is Elasticsearch query.\"},{\"name\":\"observability.rules.custom_threshold\",\"value\":\"Custom threshold Alert when any Observability data type reaches or exceeds a given value.\",\"description\":\"An available rule is Custom threshold.\"},{\"name\":\"xpack.ml.anomaly_detection_alert\",\"value\":\"Anomaly detection Alert when anomaly detection jobs results match the condition.\",\"description\":\"An available rule is Anomaly detection.\"},{\"name\":\"slo.rules.burnRate\",\"value\":\"SLO burn rate Alert when your SLO burn rate is too high over a defined period of time.\",\"description\":\"An available rule is SLO burn rate.\"},{\"name\":\"metrics.alert.threshold\",\"value\":\"Metric threshold Alert when the metrics aggregation exceeds the threshold.\",\"description\":\"An available rule is Metric threshold.\"},{\"name\":\"metrics.alert.inventory.threshold\",\"value\":\"Inventory Alert when the inventory exceeds a defined threshold.\",\"description\":\"An available rule is Inventory.\"},{\"name\":\"logs.alert.document.count\",\"value\":\"Log threshold Alert when the log aggregation exceeds the threshold.\",\"description\":\"An available rule is Log threshold.\"},{\"name\":\"xpack.uptime.alerts.tlsCertificate\",\"value\":\"Uptime TLS Alert when the TLS certificate of an Uptime monitor is about to expire.\",\"description\":\"An available rule is Uptime TLS.\"},{\"name\":\"xpack.uptime.alerts.monitorStatus\",\"value\":\"Uptime monitor status Alert when a monitor is down or an availability threshold is breached.\",\"description\":\"An available rule is Uptime monitor status.\"},{\"name\":\"xpack.uptime.alerts.durationAnomaly\",\"value\":\"Uptime Duration Anomaly Alert when the Uptime monitor duration is anomalous.\",\"description\":\"An available rule is Uptime Duration Anomaly.\"},{\"name\":\"xpack.synthetics.alerts.monitorStatus\",\"value\":\"Synthetics monitor status Alert when a monitor is down.\",\"description\":\"An available rule is Synthetics monitor status.\"},{\"name\":\"xpack.synthetics.alerts.tls\",\"value\":\"Synthetics TLS certificate Alert when the TLS certificate of a Synthetics monitor is about to expire.\",\"description\":\"An available rule is Synthetics TLS certificate.\"},{\"name\":\"apm.error_rate\",\"value\":\"Error count threshold Alert when the number of errors in a service exceeds a defined threshold.\",\"description\":\"An available rule is Error count threshold.\"},{\"name\":\"apm.transaction_error_rate\",\"value\":\"Failed transaction rate threshold Alert when the rate of transaction errors in a service exceeds a defined threshold.\",\"description\":\"An available rule is Failed transaction rate threshold.\"},{\"name\":\"apm.transaction_duration\",\"value\":\"Latency threshold Alert when the latency of a specific transaction type in a service exceeds a defined threshold.\",\"description\":\"An available rule is Latency threshold.\"},{\"name\":\"apm.anomaly\",\"value\":\"APM Anomaly Alert when either the latency, throughput, or failed transaction rate of a service is anomalous.\",\"description\":\"An available rule is APM Anomaly.\"}]}", "name": "context" } ], "stream": true, "tools": [ { "function": { "name": "get_data_on_screen", "parameters": { "type": "object", "properties": { "data": { "type": "array", "description": "The pieces of data you want to look at it. You can request one, or multiple", "items": { "type": "string", "enum": [ ".es-query", "observability.rules.custom_threshold", "xpack.ml.anomaly_detection_alert", "slo.rules.burnRate", "metrics.alert.threshold", "metrics.alert.inventory.threshold", "logs.alert.document.count", "xpack.uptime.alerts.tlsCertificate", "xpack.uptime.alerts.monitorStatus", "xpack.uptime.alerts.durationAnomaly", "xpack.synthetics.alerts.monitorStatus", "xpack.synthetics.alerts.tls", "apm.error_rate", "apm.transaction_error_rate", "apm.transaction_duration", "apm.anomaly" ] } } }, "required": ["data"] } }, "type": "function" }, { "function": { "name": "query", "description": "This function generates, executes and/or visualizes a query\n based on the user's request. It also explains how ES|QL works and how to\n convert queries from one language to another. Make sure you call one of\n the get_dataset functions first if you need index or field names. This\n function takes no input.", "parameters": { "type": "object", "properties": {} } }, "type": "function" }, { "function": { "name": "get_alerts_dataset_info", "description": "Use this function to get information about alerts data.", "parameters": { "type": "object", "properties": { "start": { "type": "string", "description": "The start of the current time range, in datemath, like now-24h or an ISO timestamp" }, "end": { "type": "string", "description": "The end of the current time range, in datemath, like now-24h or an ISO timestamp" } } } }, "type": "function" }, { "function": { "name": "alerts", "description": "Get alerts for Observability. Make sure get_alerts_dataset_info was called before.\n Use this to get open (and optionally recovered) alerts for Observability assets, like services,\n hosts or containers.\n Display the response in tabular format if appropriate.\n ", "parameters": { "type": "object", "properties": { "start": { "type": "string", "description": "The start of the time range, in Elasticsearch date math, like `now`." }, "end": { "type": "string", "description": "The end of the time range, in Elasticsearch date math, like `now-24h`." }, "kqlFilter": { "type": "string", "description": "Filter alerts by field:value pairs" }, "includeRecovered": { "type": "boolean", "description": "Whether to include recovered/closed alerts. Defaults to false, which means only active alerts will be returned" } }, "required": ["start", "end"] } }, "type": "function" }, { "function": { "name": "changes", "description": "Returns change points like spikes and dips for logs and metrics.", "parameters": { "type": "object", "properties": { "start": { "type": "string", "description": "The beginning of the time range, in datemath, like now-24h, or an ISO timestamp" }, "end": { "type": "string", "description": "The end of the time range, in datemath, like now, or an ISO timestamp" }, "logs": { "description": "Analyze changes in log patterns. If no index is given, the default logs index pattern will be used", "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string", "description": "The name of this set of logs" }, "index": { "type": "string", "description": "The index or index pattern where to find the logs" }, "kqlFilter": { "type": "string", "description": "A KQL filter to filter the log documents by, e.g. my_field:foo" }, "field": { "type": "string", "description": "The text field that contains the message to be analyzed, usually `message`. ONLY use field names from the conversation." } }, "required": ["name"] } }, "metrics": { "description": "Analyze changes in metrics. DO NOT UNDER ANY CIRCUMSTANCES use date or metric fields for groupBy, leave empty unless needed.", "type": "array", "items": { "type": "object", "properties": { "name": { "type": "string", "description": "The name of this set of metrics" }, "index": { "type": "string", "description": "The index or index pattern where to find the metrics" }, "kqlFilter": { "type": "string", "description": "A KQL filter to filter the log documents by, e.g. my_field:foo" }, "field": { "type": "string", "description": "Metric field that contains the metric. Only use if the metric aggregation type is not count." }, "type": { "type": "string", "description": "The type of metric aggregation to perform. Defaults to count", "enum": ["count", "avg", "sum", "min", "max", "p95", "p99"] }, "groupBy": { "type": "array", "description": "Optional keyword fields to group metrics by.", "items": { "type": "string" } } }, "required": ["index", "name"] } } }, "required": ["start", "end"] } }, "type": "function" }, { "function": { "name": "summarize", "description": "Use this function to store facts in the knowledge database if the user requests it.\n You can score the learnings with a confidence metric, whether it is a correction on a previous learning.\n An embedding will be created that you can recall later with a semantic search.\n When you create this summarisation, make sure you craft it in a way that can be recalled with a semantic\n search later, and that it would have answered the user's original request.", "parameters": { "type": "object", "properties": { "id": { "type": "string", "description": "An id for the document. This should be a short human-readable keyword field with only alphabetic characters and underscores, that allow you to update it later." }, "text": { "type": "string", "description": "A human-readable summary of what you have learned, described in such a way that you can recall it later with semantic search, and that it would have answered the user's original request." }, "is_correction": { "type": "boolean", "description": "Whether this is a correction for a previous learning." }, "confidence": { "type": "string", "description": "How confident you are about this being a correct and useful learning", "enum": ["low", "medium", "high"] }, "public": { "type": "boolean", "description": "Whether this information is specific to the user, or generally applicable to any user of the product" } }, "required": ["id", "text", "is_correction", "confidence", "public"] } }, "type": "function" }, { "function": { "name": "elasticsearch", "description": "Call Elasticsearch APIs on behalf of the user. Make sure the request body is valid for the API that you are using. Only call this function when the user has explicitly requested it.", "parameters": { "type": "object", "properties": { "method": { "type": "string", "description": "The HTTP method of the Elasticsearch endpoint", "enum": ["GET", "PUT", "POST", "DELETE", "PATCH"] }, "path": { "type": "string", "description": "The path of the Elasticsearch endpoint, including query parameters" }, "body": { "type": "object", "description": "The body of the request" } }, "required": ["method", "path"] } }, "type": "function" }, { "function": { "name": "kibana", "description": "Call Kibana APIs on behalf of the user. Only call this function when the user has explicitly requested it, and you know how to call it, for example by querying the knowledge base or having the user explain it to you. Assume that pathnames, bodies and query parameters may have changed since your knowledge cut off date.", "parameters": { "type": "object", "properties": { "method": { "type": "string", "description": "The HTTP method of the Kibana endpoint", "enum": ["GET", "PUT", "POST", "DELETE", "PATCH"] }, "pathname": { "type": "string", "description": "The pathname of the Kibana endpoint, excluding query parameters" }, "query": { "type": "object", "description": "The query parameters, as an object" }, "body": { "type": "object", "description": "The body of the request" } }, "required": ["method", "pathname"] } }, "type": "function" }, { "function": { "name": "get_dataset_info", "description": "Use this function to get information about indices/datasets available and the fields available on them.\n\n providing empty string as index name will retrieve all indices\n else list of all fields for the given index will be given. if no fields are returned this means no indices were matched by provided index pattern.\n wildcards can be part of index name.", "parameters": { "type": "object", "properties": { "index": { "type": "string", "description": "index pattern the user is interested in or empty string to get information about all available indices" } }, "required": ["index"] } }, "type": "function" }, { "function": { "name": "execute_connector", "description": "Use this function when user explicitly asks to call a kibana connector.", "parameters": { "type": "object", "properties": { "id": { "type": "string", "description": "The id of the connector" }, "params": { "type": "object", "description": "The connector parameters" } }, "required": ["id", "params"] } }, "type": "function" } ], "temperature": 0 } ```
--------- Co-authored-by: Søren Louv-Jansen --- .../server/routes/functions/route.ts | 2 +- .../chat_function_client/index.test.ts | 52 ++++++++++++++++++- .../service/chat_function_client/index.ts | 28 ++++++++-- .../server/service/client/index.test.ts | 2 + .../server/service/client/index.ts | 7 ++- .../client/operators/continue_conversation.ts | 7 ++- .../server/service/types.ts | 3 ++ .../get_system_message_from_instructions.ts | 2 +- .../server/rule_connector/index.test.ts | 1 + .../server/rule_connector/index.ts | 2 +- 10 files changed, 94 insertions(+), 12 deletions(-) diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant/server/routes/functions/route.ts b/x-pack/plugins/observability_solution/observability_ai_assistant/server/routes/functions/route.ts index 8a61248d4e70e..c402a0506736f 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant/server/routes/functions/route.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant/server/routes/functions/route.ts @@ -68,7 +68,7 @@ const getFunctionsRoute = createObservabilityAIAssistantServerRoute({ systemMessage: getSystemMessageFromInstructions({ applicationInstructions: functionClient.getInstructions(), userInstructions, - adHocInstructions: [], + adHocInstructions: functionClient.getAdhocInstructions(), availableFunctionNames, }), }; diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/chat_function_client/index.test.ts b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/chat_function_client/index.test.ts index 3d83c470de0c5..0d911b497cbbb 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/chat_function_client/index.test.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/chat_function_client/index.test.ts @@ -7,6 +7,7 @@ import dedent from 'dedent'; import { ChatFunctionClient, GET_DATA_ON_SCREEN_FUNCTION_NAME } from '.'; import { FunctionVisibility } from '../../../common/functions/types'; +import { AdHocInstruction } from '../../../common/types'; describe('chatFunctionClient', () => { describe('when executing a function with invalid arguments', () => { @@ -86,6 +87,7 @@ describe('chatFunctionClient', () => { ]); const functions = client.getFunctions(); + const adHocInstructions = client.getAdhocInstructions(); expect(functions[0]).toEqual({ definition: { @@ -97,7 +99,7 @@ describe('chatFunctionClient', () => { respond: expect.any(Function), }); - expect(functions[0].definition.description).toContain( + expect(adHocInstructions[0].text).toContain( dedent(`my_dummy_data: My dummy data my_other_dummy_data: My other dummy data `) @@ -128,4 +130,52 @@ describe('chatFunctionClient', () => { }); }); }); + + describe('when adhoc instructions are provided', () => { + let client: ChatFunctionClient; + + beforeEach(() => { + client = new ChatFunctionClient([]); + }); + + describe('register an adhoc Instruction', () => { + it('should register a new adhoc instruction', () => { + const adhocInstruction: AdHocInstruction = { + text: 'Test adhoc instruction', + instruction_type: 'application_instruction', + }; + + client.registerAdhocInstruction(adhocInstruction); + + expect(client.getAdhocInstructions()).toContainEqual(adhocInstruction); + }); + }); + + describe('retrieve adHoc instructions', () => { + it('should return all registered adhoc instructions', () => { + const firstAdhocInstruction: AdHocInstruction = { + text: 'First adhoc instruction', + instruction_type: 'application_instruction', + }; + + const secondAdhocInstruction: AdHocInstruction = { + text: 'Second adhoc instruction', + instruction_type: 'application_instruction', + }; + + client.registerAdhocInstruction(firstAdhocInstruction); + client.registerAdhocInstruction(secondAdhocInstruction); + + const adhocInstructions = client.getAdhocInstructions(); + + expect(adhocInstructions).toEqual([firstAdhocInstruction, secondAdhocInstruction]); + }); + + it('should return an empty array if no adhoc instructions are registered', () => { + const adhocInstructions = client.getAdhocInstructions(); + + expect(adhocInstructions).toEqual([]); + }); + }); + }); }); diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/chat_function_client/index.ts b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/chat_function_client/index.ts index 97def121e8593..ad4b7f0a4fc92 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/chat_function_client/index.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/chat_function_client/index.ts @@ -10,13 +10,18 @@ import Ajv, { type ErrorObject, type ValidateFunction } from 'ajv'; import dedent from 'dedent'; import { compact, keyBy } from 'lodash'; import { FunctionVisibility, type FunctionResponse } from '../../../common/functions/types'; -import type { Message, ObservabilityAIAssistantScreenContextRequest } from '../../../common/types'; +import type { + AdHocInstruction, + Message, + ObservabilityAIAssistantScreenContextRequest, +} from '../../../common/types'; import { filterFunctionDefinitions } from '../../../common/utils/filter_function_definitions'; import type { FunctionCallChatFunction, FunctionHandler, FunctionHandlerRegistry, InstructionOrCallback, + RegisterAdHocInstruction, RegisterFunction, RegisterInstruction, } from '../types'; @@ -35,6 +40,8 @@ export const GET_DATA_ON_SCREEN_FUNCTION_NAME = 'get_data_on_screen'; export class ChatFunctionClient { private readonly instructions: InstructionOrCallback[] = []; + private readonly adhocInstructions: AdHocInstruction[] = []; + private readonly functionRegistry: FunctionHandlerRegistry = new Map(); private readonly validators: Map = new Map(); @@ -49,9 +56,7 @@ export class ChatFunctionClient { this.registerFunction( { name: GET_DATA_ON_SCREEN_FUNCTION_NAME, - description: dedent(`Get data that is on the screen: - ${allData.map((data) => `${data.name}: ${data.description}`).join('\n')} - `), + description: `Retrieve the structured data of content currently visible on the user's screen. Use this tool to understand what the user is viewing at this moment to provide more accurate and context-aware responses to their questions.`, visibility: FunctionVisibility.AssistantOnly, parameters: { type: 'object', @@ -75,6 +80,13 @@ export class ChatFunctionClient { }; } ); + + this.registerAdhocInstruction({ + text: `The ${GET_DATA_ON_SCREEN_FUNCTION_NAME} function will retrieve specific content from the user's screen by specifying a data key. Use this tool to provide context-aware responses. Available data: ${dedent( + allData.map((data) => `${data.name}: ${data.description}`).join('\n') + )}`, + instruction_type: 'application_instruction', + }); } this.actions.forEach((action) => { @@ -95,6 +107,10 @@ export class ChatFunctionClient { this.instructions.push(instruction); }; + registerAdhocInstruction: RegisterAdHocInstruction = (instruction: AdHocInstruction) => { + this.adhocInstructions.push(instruction); + }; + validate(name: string, parameters: unknown) { const validator = this.validators.get(name)!; if (!validator) { @@ -111,6 +127,10 @@ export class ChatFunctionClient { return this.instructions; } + getAdhocInstructions(): AdHocInstruction[] { + return this.adhocInstructions; + } + hasAction(name: string) { return !!this.actions.find((action) => action.name === name)!; } diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.test.ts b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.test.ts index 0476bda1af8a2..8da2a0d843b11 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.test.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.test.ts @@ -124,6 +124,7 @@ describe('Observability AI Assistant client', () => { getActions: jest.fn(), validate: jest.fn(), getInstructions: jest.fn(), + getAdhocInstructions: jest.fn(), } as any; let llmSimulator: LlmSimulator; @@ -173,6 +174,7 @@ describe('Observability AI Assistant client', () => { knowledgeBaseServiceMock.getUserInstructions.mockResolvedValue([]); functionClientMock.getInstructions.mockReturnValue(['system']); + functionClientMock.getAdhocInstructions.mockReturnValue([]); return new ObservabilityAIAssistantClient({ actionsClient: actionsClientMock, diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.ts b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.ts index a050edc8008fb..162220ec7a7f1 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/index.ts @@ -47,12 +47,12 @@ import { } from '../../../common/conversation_complete'; import { CompatibleJSONSchema } from '../../../common/functions/types'; import { + AdHocInstruction, type Conversation, type ConversationCreateRequest, type ConversationUpdateRequest, type KnowledgeBaseEntry, type Message, - type AdHocInstruction, } from '../../../common/types'; import { withoutTokenCountEvents } from '../../../common/utils/without_token_count_events'; import { CONTEXT_FUNCTION_NAME } from '../../functions/context'; @@ -210,6 +210,9 @@ export class ObservabilityAIAssistantClient { const userInstructions$ = from(this.getKnowledgeBaseUserInstructions()).pipe(shareReplay()); + const registeredAdhocInstructions = functionClient.getAdhocInstructions(); + const allAdHocInstructions = adHocInstructions.concat(registeredAdhocInstructions); + // from the initial messages, override any system message with // the one that is based on the instructions (registered, request, kb) const messagesWithUpdatedSystemMessage$ = userInstructions$.pipe( @@ -219,7 +222,7 @@ export class ObservabilityAIAssistantClient { getSystemMessageFromInstructions({ applicationInstructions: functionClient.getInstructions(), userInstructions, - adHocInstructions, + adHocInstructions: allAdHocInstructions, availableFunctionNames: functionClient .getFunctions() .map((fn) => fn.definition.name), diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/operators/continue_conversation.ts b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/operators/continue_conversation.ts index 66204c96f31cb..4c55d32362878 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/operators/continue_conversation.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/client/operators/continue_conversation.ts @@ -178,7 +178,7 @@ export function continueConversation({ chat, signal, functionCallsLeft, - adHocInstructions, + adHocInstructions = [], userInstructions, logger, disableFunctions, @@ -213,11 +213,14 @@ export function continueConversation({ disableFunctions, }); + const registeredAdhocInstructions = functionClient.getAdhocInstructions(); + const allAdHocInstructions = adHocInstructions.concat(registeredAdhocInstructions); + const messagesWithUpdatedSystemMessage = replaceSystemMessage( getSystemMessageFromInstructions({ applicationInstructions: functionClient.getInstructions(), userInstructions, - adHocInstructions, + adHocInstructions: allAdHocInstructions, availableFunctionNames: definitions.map((def) => def.name), }), initialMessages diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/types.ts b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/types.ts index b00da8d6518fa..2df3f36163972 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/types.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/types.ts @@ -18,6 +18,7 @@ import type { Message, ObservabilityAIAssistantScreenContextRequest, InstructionOrPlainText, + AdHocInstruction, } from '../../common/types'; import type { ObservabilityAIAssistantRouteHandlerResources } from '../routes/types'; import { ChatFunctionClient } from './chat_function_client'; @@ -76,6 +77,8 @@ export type RegisterInstructionCallback = ({ export type RegisterInstruction = (...instruction: InstructionOrCallback[]) => void; +export type RegisterAdHocInstruction = (...instruction: AdHocInstruction[]) => void; + export type RegisterFunction = < TParameters extends CompatibleJSONSchema = any, TResponse extends FunctionResponse = any, diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/util/get_system_message_from_instructions.ts b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/util/get_system_message_from_instructions.ts index b2797577883ba..570449673084b 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/util/get_system_message_from_instructions.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant/server/service/util/get_system_message_from_instructions.ts @@ -45,7 +45,7 @@ export function getSystemMessageFromInstructions({ const adHocInstructionsWithId = adHocInstructions.map((adHocInstruction) => ({ ...adHocInstruction, - doc_id: adHocInstruction.doc_id ?? v4(), + doc_id: adHocInstruction?.doc_id ?? v4(), })); // split ad hoc instructions into user instructions and application instructions diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.test.ts b/x-pack/plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.test.ts index 479ffeaa40f4f..de02e4cf841ce 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.test.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.test.ts @@ -91,6 +91,7 @@ describe('observabilityAIAssistant rule_connector', () => { getFunctionClient: async () => ({ getFunctions: () => [], getInstructions: () => [], + getAdhocInstructions: () => [], }), }, context: { diff --git a/x-pack/plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.ts b/x-pack/plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.ts index 19f1408275e1f..59b883fef9c18 100644 --- a/x-pack/plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.ts +++ b/x-pack/plugins/observability_solution/observability_ai_assistant_app/server/rule_connector/index.ts @@ -230,7 +230,7 @@ If available, include the link of the conversation at the end of your answer.` availableFunctionNames: functionClient.getFunctions().map((fn) => fn.definition.name), applicationInstructions: functionClient.getInstructions(), userInstructions: [], - adHocInstructions: [], + adHocInstructions: functionClient.getAdhocInstructions(), }), }, },