[Observability Onboarding] Fix duplicate AWS Services in Firehose flow #200931
Assignees
Labels
Comments
mykolaharmash
added
Feature: Observability Onboarding
Team:obs-ux-logs
|
Pinging @elastic/obs-ux-logs-team (Team:obs-ux-logs) |
|
Pinging @elastic/obs-ux-onboarding-team (Feature: Observability Onboarding) |
mykolaharmash
added a commit
that referenced
this issue
Dec 4, 2024
…Firehose flow (#201613) Closes #200931 Switched to using AWS service list as a base to showing the detected services in the UI instead of the list of populated indices as multiple indices can be related to a single service. ### How to test 1. Go to Firehose flow `/observabilityOnboarding/firehose` 2. Open Kibana dev tools in another tab 3. Ingest documents related into multiple data streams which related to a single AWS service: ``` POST logs-aws.apigateway_logs-default/_doc { "@timestamp": "2024-11-25T13:32:01.000Z", "some": 111, "aws.kinesis.name": "Elastic-CloudwatchLogs" } POST metrics-aws.apigateway_metrics-default/_doc { "@timestamp": "2024-11-25T13:31:01.000Z", "agent": { "type": "firehose" }, "aws": { "cloudwatch": { "namespace": "AWS/ApiGateway" }, "exporter": { "arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J" } }, "cloud": { "account": { "id": "975050175126" }, "provider": "aws", "region": "us-west-2" } } ``` 4. Make sure you see only one entry for the service appear in the Firehose flow
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Dec 4, 2024
…Firehose flow (elastic#201613) Closes elastic#200931 Switched to using AWS service list as a base to showing the detected services in the UI instead of the list of populated indices as multiple indices can be related to a single service. ### How to test 1. Go to Firehose flow `/observabilityOnboarding/firehose` 2. Open Kibana dev tools in another tab 3. Ingest documents related into multiple data streams which related to a single AWS service: ``` POST logs-aws.apigateway_logs-default/_doc { "@timestamp": "2024-11-25T13:32:01.000Z", "some": 111, "aws.kinesis.name": "Elastic-CloudwatchLogs" } POST metrics-aws.apigateway_metrics-default/_doc { "@timestamp": "2024-11-25T13:31:01.000Z", "agent": { "type": "firehose" }, "aws": { "cloudwatch": { "namespace": "AWS/ApiGateway" }, "exporter": { "arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J" } }, "cloud": { "account": { "id": "975050175126" }, "provider": "aws", "region": "us-west-2" } } ``` 4. Make sure you see only one entry for the service appear in the Firehose flow (cherry picked from commit a2fb9b7)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Dec 4, 2024
…Firehose flow (elastic#201613) Closes elastic#200931 Switched to using AWS service list as a base to showing the detected services in the UI instead of the list of populated indices as multiple indices can be related to a single service. ### How to test 1. Go to Firehose flow `/observabilityOnboarding/firehose` 2. Open Kibana dev tools in another tab 3. Ingest documents related into multiple data streams which related to a single AWS service: ``` POST logs-aws.apigateway_logs-default/_doc { "@timestamp": "2024-11-25T13:32:01.000Z", "some": 111, "aws.kinesis.name": "Elastic-CloudwatchLogs" } POST metrics-aws.apigateway_metrics-default/_doc { "@timestamp": "2024-11-25T13:31:01.000Z", "agent": { "type": "firehose" }, "aws": { "cloudwatch": { "namespace": "AWS/ApiGateway" }, "exporter": { "arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J" } }, "cloud": { "account": { "id": "975050175126" }, "provider": "aws", "region": "us-west-2" } } ``` 4. Make sure you see only one entry for the service appear in the Firehose flow (cherry picked from commit a2fb9b7)
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this issue
Dec 4, 2024
…Firehose flow (elastic#201613) Closes elastic#200931 Switched to using AWS service list as a base to showing the detected services in the UI instead of the list of populated indices as multiple indices can be related to a single service. ### How to test 1. Go to Firehose flow `/observabilityOnboarding/firehose` 2. Open Kibana dev tools in another tab 3. Ingest documents related into multiple data streams which related to a single AWS service: ``` POST logs-aws.apigateway_logs-default/_doc { "@timestamp": "2024-11-25T13:32:01.000Z", "some": 111, "aws.kinesis.name": "Elastic-CloudwatchLogs" } POST metrics-aws.apigateway_metrics-default/_doc { "@timestamp": "2024-11-25T13:31:01.000Z", "agent": { "type": "firehose" }, "aws": { "cloudwatch": { "namespace": "AWS/ApiGateway" }, "exporter": { "arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J" } }, "cloud": { "account": { "id": "975050175126" }, "provider": "aws", "region": "us-west-2" } } ``` 4. Make sure you see only one entry for the service appear in the Firehose flow (cherry picked from commit a2fb9b7)
hop-dev
pushed a commit
to hop-dev/kibana
that referenced
this issue
Dec 5, 2024
…Firehose flow (elastic#201613) Closes elastic#200931 Switched to using AWS service list as a base to showing the detected services in the UI instead of the list of populated indices as multiple indices can be related to a single service. ### How to test 1. Go to Firehose flow `/observabilityOnboarding/firehose` 2. Open Kibana dev tools in another tab 3. Ingest documents related into multiple data streams which related to a single AWS service: ``` POST logs-aws.apigateway_logs-default/_doc { "@timestamp": "2024-11-25T13:32:01.000Z", "some": 111, "aws.kinesis.name": "Elastic-CloudwatchLogs" } POST metrics-aws.apigateway_metrics-default/_doc { "@timestamp": "2024-11-25T13:31:01.000Z", "agent": { "type": "firehose" }, "aws": { "cloudwatch": { "namespace": "AWS/ApiGateway" }, "exporter": { "arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J" } }, "cloud": { "account": { "id": "975050175126" }, "provider": "aws", "region": "us-west-2" } } ``` 4. Make sure you see only one entry for the service appear in the Firehose flow
kibanamachine
added a commit
that referenced
this issue
Dec 6, 2024
…ces in Firehose flow (#201613) (#202893) # Backport This will backport the following commits from `main` to `8.17`: - [[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)](#201613) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Mykola Harmash","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-04T10:49:47Z","message":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)\n\nCloses #200931 \r\n\r\nSwitched to using AWS service list as a base to showing the detected\r\nservices in the UI instead of the list of populated indices as multiple\r\nindices can be related to a single service.\r\n\r\n### How to test\r\n\r\n1. Go to Firehose flow `/observabilityOnboarding/firehose`\r\n2. Open Kibana dev tools in another tab\r\n3. Ingest documents related into multiple data streams which related to\r\na single AWS service:\r\n```\r\nPOST logs-aws.apigateway_logs-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:32:01.000Z\",\r\n \"some\": 111,\r\n \"aws.kinesis.name\": \"Elastic-CloudwatchLogs\"\r\n}\r\n\r\nPOST metrics-aws.apigateway_metrics-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:31:01.000Z\",\r\n \"agent\": {\r\n \"type\": \"firehose\"\r\n },\r\n \"aws\": {\r\n \"cloudwatch\": {\r\n \"namespace\": \"AWS/ApiGateway\"\r\n },\r\n \"exporter\": {\r\n \"arn\": \"arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J\"\r\n }\r\n },\r\n \"cloud\": {\r\n \"account\": {\r\n \"id\": \"975050175126\"\r\n },\r\n \"provider\": \"aws\",\r\n \"region\": \"us-west-2\"\r\n }\r\n}\r\n```\r\n4. Make sure you see only one entry for the service appear in the\r\nFirehose flow","sha":"a2fb9b736cd8007a0ba04b9b1051d7988c690b5b","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","backport:prev-major","ci:project-deploy-observability"],"title":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow","number":201613,"url":"https://github.com/elastic/kibana/pull/201613","mergeCommit":{"message":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)\n\nCloses #200931 \r\n\r\nSwitched to using AWS service list as a base to showing the detected\r\nservices in the UI instead of the list of populated indices as multiple\r\nindices can be related to a single service.\r\n\r\n### How to test\r\n\r\n1. Go to Firehose flow `/observabilityOnboarding/firehose`\r\n2. Open Kibana dev tools in another tab\r\n3. Ingest documents related into multiple data streams which related to\r\na single AWS service:\r\n```\r\nPOST logs-aws.apigateway_logs-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:32:01.000Z\",\r\n \"some\": 111,\r\n \"aws.kinesis.name\": \"Elastic-CloudwatchLogs\"\r\n}\r\n\r\nPOST metrics-aws.apigateway_metrics-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:31:01.000Z\",\r\n \"agent\": {\r\n \"type\": \"firehose\"\r\n },\r\n \"aws\": {\r\n \"cloudwatch\": {\r\n \"namespace\": \"AWS/ApiGateway\"\r\n },\r\n \"exporter\": {\r\n \"arn\": \"arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J\"\r\n }\r\n },\r\n \"cloud\": {\r\n \"account\": {\r\n \"id\": \"975050175126\"\r\n },\r\n \"provider\": \"aws\",\r\n \"region\": \"us-west-2\"\r\n }\r\n}\r\n```\r\n4. Make sure you see only one entry for the service appear in the\r\nFirehose flow","sha":"a2fb9b736cd8007a0ba04b9b1051d7988c690b5b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201613","number":201613,"mergeCommit":{"message":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)\n\nCloses #200931 \r\n\r\nSwitched to using AWS service list as a base to showing the detected\r\nservices in the UI instead of the list of populated indices as multiple\r\nindices can be related to a single service.\r\n\r\n### How to test\r\n\r\n1. Go to Firehose flow `/observabilityOnboarding/firehose`\r\n2. Open Kibana dev tools in another tab\r\n3. Ingest documents related into multiple data streams which related to\r\na single AWS service:\r\n```\r\nPOST logs-aws.apigateway_logs-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:32:01.000Z\",\r\n \"some\": 111,\r\n \"aws.kinesis.name\": \"Elastic-CloudwatchLogs\"\r\n}\r\n\r\nPOST metrics-aws.apigateway_metrics-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:31:01.000Z\",\r\n \"agent\": {\r\n \"type\": \"firehose\"\r\n },\r\n \"aws\": {\r\n \"cloudwatch\": {\r\n \"namespace\": \"AWS/ApiGateway\"\r\n },\r\n \"exporter\": {\r\n \"arn\": \"arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J\"\r\n }\r\n },\r\n \"cloud\": {\r\n \"account\": {\r\n \"id\": \"975050175126\"\r\n },\r\n \"provider\": \"aws\",\r\n \"region\": \"us-west-2\"\r\n }\r\n}\r\n```\r\n4. Make sure you see only one entry for the service appear in the\r\nFirehose flow","sha":"a2fb9b736cd8007a0ba04b9b1051d7988c690b5b"}}]}] BACKPORT--> Co-authored-by: Mykola Harmash <[email protected]>
kibanamachine
added a commit
that referenced
this issue
Dec 6, 2024
…ces in Firehose flow (#201613) (#202892) # Backport This will backport the following commits from `main` to `8.16`: - [[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)](#201613) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Mykola Harmash","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-04T10:49:47Z","message":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)\n\nCloses #200931 \r\n\r\nSwitched to using AWS service list as a base to showing the detected\r\nservices in the UI instead of the list of populated indices as multiple\r\nindices can be related to a single service.\r\n\r\n### How to test\r\n\r\n1. Go to Firehose flow `/observabilityOnboarding/firehose`\r\n2. Open Kibana dev tools in another tab\r\n3. Ingest documents related into multiple data streams which related to\r\na single AWS service:\r\n```\r\nPOST logs-aws.apigateway_logs-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:32:01.000Z\",\r\n \"some\": 111,\r\n \"aws.kinesis.name\": \"Elastic-CloudwatchLogs\"\r\n}\r\n\r\nPOST metrics-aws.apigateway_metrics-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:31:01.000Z\",\r\n \"agent\": {\r\n \"type\": \"firehose\"\r\n },\r\n \"aws\": {\r\n \"cloudwatch\": {\r\n \"namespace\": \"AWS/ApiGateway\"\r\n },\r\n \"exporter\": {\r\n \"arn\": \"arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J\"\r\n }\r\n },\r\n \"cloud\": {\r\n \"account\": {\r\n \"id\": \"975050175126\"\r\n },\r\n \"provider\": \"aws\",\r\n \"region\": \"us-west-2\"\r\n }\r\n}\r\n```\r\n4. Make sure you see only one entry for the service appear in the\r\nFirehose flow","sha":"a2fb9b736cd8007a0ba04b9b1051d7988c690b5b","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","backport:prev-major","ci:project-deploy-observability"],"title":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow","number":201613,"url":"https://github.com/elastic/kibana/pull/201613","mergeCommit":{"message":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)\n\nCloses #200931 \r\n\r\nSwitched to using AWS service list as a base to showing the detected\r\nservices in the UI instead of the list of populated indices as multiple\r\nindices can be related to a single service.\r\n\r\n### How to test\r\n\r\n1. Go to Firehose flow `/observabilityOnboarding/firehose`\r\n2. Open Kibana dev tools in another tab\r\n3. Ingest documents related into multiple data streams which related to\r\na single AWS service:\r\n```\r\nPOST logs-aws.apigateway_logs-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:32:01.000Z\",\r\n \"some\": 111,\r\n \"aws.kinesis.name\": \"Elastic-CloudwatchLogs\"\r\n}\r\n\r\nPOST metrics-aws.apigateway_metrics-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:31:01.000Z\",\r\n \"agent\": {\r\n \"type\": \"firehose\"\r\n },\r\n \"aws\": {\r\n \"cloudwatch\": {\r\n \"namespace\": \"AWS/ApiGateway\"\r\n },\r\n \"exporter\": {\r\n \"arn\": \"arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J\"\r\n }\r\n },\r\n \"cloud\": {\r\n \"account\": {\r\n \"id\": \"975050175126\"\r\n },\r\n \"provider\": \"aws\",\r\n \"region\": \"us-west-2\"\r\n }\r\n}\r\n```\r\n4. Make sure you see only one entry for the service appear in the\r\nFirehose flow","sha":"a2fb9b736cd8007a0ba04b9b1051d7988c690b5b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201613","number":201613,"mergeCommit":{"message":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)\n\nCloses #200931 \r\n\r\nSwitched to using AWS service list as a base to showing the detected\r\nservices in the UI instead of the list of populated indices as multiple\r\nindices can be related to a single service.\r\n\r\n### How to test\r\n\r\n1. Go to Firehose flow `/observabilityOnboarding/firehose`\r\n2. Open Kibana dev tools in another tab\r\n3. Ingest documents related into multiple data streams which related to\r\na single AWS service:\r\n```\r\nPOST logs-aws.apigateway_logs-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:32:01.000Z\",\r\n \"some\": 111,\r\n \"aws.kinesis.name\": \"Elastic-CloudwatchLogs\"\r\n}\r\n\r\nPOST metrics-aws.apigateway_metrics-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:31:01.000Z\",\r\n \"agent\": {\r\n \"type\": \"firehose\"\r\n },\r\n \"aws\": {\r\n \"cloudwatch\": {\r\n \"namespace\": \"AWS/ApiGateway\"\r\n },\r\n \"exporter\": {\r\n \"arn\": \"arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J\"\r\n }\r\n },\r\n \"cloud\": {\r\n \"account\": {\r\n \"id\": \"975050175126\"\r\n },\r\n \"provider\": \"aws\",\r\n \"region\": \"us-west-2\"\r\n }\r\n}\r\n```\r\n4. Make sure you see only one entry for the service appear in the\r\nFirehose flow","sha":"a2fb9b736cd8007a0ba04b9b1051d7988c690b5b"}}]}] BACKPORT--> Co-authored-by: Mykola Harmash <[email protected]>
kibanamachine
added a commit
that referenced
this issue
Dec 6, 2024
…es in Firehose flow (#201613) (#202894) # Backport This will backport the following commits from `main` to `8.x`: - [[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)](#201613) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Mykola Harmash","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-04T10:49:47Z","message":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)\n\nCloses #200931 \r\n\r\nSwitched to using AWS service list as a base to showing the detected\r\nservices in the UI instead of the list of populated indices as multiple\r\nindices can be related to a single service.\r\n\r\n### How to test\r\n\r\n1. Go to Firehose flow `/observabilityOnboarding/firehose`\r\n2. Open Kibana dev tools in another tab\r\n3. Ingest documents related into multiple data streams which related to\r\na single AWS service:\r\n```\r\nPOST logs-aws.apigateway_logs-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:32:01.000Z\",\r\n \"some\": 111,\r\n \"aws.kinesis.name\": \"Elastic-CloudwatchLogs\"\r\n}\r\n\r\nPOST metrics-aws.apigateway_metrics-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:31:01.000Z\",\r\n \"agent\": {\r\n \"type\": \"firehose\"\r\n },\r\n \"aws\": {\r\n \"cloudwatch\": {\r\n \"namespace\": \"AWS/ApiGateway\"\r\n },\r\n \"exporter\": {\r\n \"arn\": \"arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J\"\r\n }\r\n },\r\n \"cloud\": {\r\n \"account\": {\r\n \"id\": \"975050175126\"\r\n },\r\n \"provider\": \"aws\",\r\n \"region\": \"us-west-2\"\r\n }\r\n}\r\n```\r\n4. Make sure you see only one entry for the service appear in the\r\nFirehose flow","sha":"a2fb9b736cd8007a0ba04b9b1051d7988c690b5b","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","v9.0.0","backport:prev-minor","backport:prev-major","ci:project-deploy-observability"],"title":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow","number":201613,"url":"https://github.com/elastic/kibana/pull/201613","mergeCommit":{"message":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)\n\nCloses #200931 \r\n\r\nSwitched to using AWS service list as a base to showing the detected\r\nservices in the UI instead of the list of populated indices as multiple\r\nindices can be related to a single service.\r\n\r\n### How to test\r\n\r\n1. Go to Firehose flow `/observabilityOnboarding/firehose`\r\n2. Open Kibana dev tools in another tab\r\n3. Ingest documents related into multiple data streams which related to\r\na single AWS service:\r\n```\r\nPOST logs-aws.apigateway_logs-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:32:01.000Z\",\r\n \"some\": 111,\r\n \"aws.kinesis.name\": \"Elastic-CloudwatchLogs\"\r\n}\r\n\r\nPOST metrics-aws.apigateway_metrics-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:31:01.000Z\",\r\n \"agent\": {\r\n \"type\": \"firehose\"\r\n },\r\n \"aws\": {\r\n \"cloudwatch\": {\r\n \"namespace\": \"AWS/ApiGateway\"\r\n },\r\n \"exporter\": {\r\n \"arn\": \"arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J\"\r\n }\r\n },\r\n \"cloud\": {\r\n \"account\": {\r\n \"id\": \"975050175126\"\r\n },\r\n \"provider\": \"aws\",\r\n \"region\": \"us-west-2\"\r\n }\r\n}\r\n```\r\n4. Make sure you see only one entry for the service appear in the\r\nFirehose flow","sha":"a2fb9b736cd8007a0ba04b9b1051d7988c690b5b"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/201613","number":201613,"mergeCommit":{"message":"[Observability Onboarding] Prevent showing duplcated AWS services in Firehose flow (#201613)\n\nCloses #200931 \r\n\r\nSwitched to using AWS service list as a base to showing the detected\r\nservices in the UI instead of the list of populated indices as multiple\r\nindices can be related to a single service.\r\n\r\n### How to test\r\n\r\n1. Go to Firehose flow `/observabilityOnboarding/firehose`\r\n2. Open Kibana dev tools in another tab\r\n3. Ingest documents related into multiple data streams which related to\r\na single AWS service:\r\n```\r\nPOST logs-aws.apigateway_logs-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:32:01.000Z\",\r\n \"some\": 111,\r\n \"aws.kinesis.name\": \"Elastic-CloudwatchLogs\"\r\n}\r\n\r\nPOST metrics-aws.apigateway_metrics-default/_doc\r\n{\r\n \"@timestamp\": \"2024-11-25T13:31:01.000Z\",\r\n \"agent\": {\r\n \"type\": \"firehose\"\r\n },\r\n \"aws\": {\r\n \"cloudwatch\": {\r\n \"namespace\": \"AWS/ApiGateway\"\r\n },\r\n \"exporter\": {\r\n \"arn\": \"arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J\"\r\n }\r\n },\r\n \"cloud\": {\r\n \"account\": {\r\n \"id\": \"975050175126\"\r\n },\r\n \"provider\": \"aws\",\r\n \"region\": \"us-west-2\"\r\n }\r\n}\r\n```\r\n4. Make sure you see only one entry for the service appear in the\r\nFirehose flow","sha":"a2fb9b736cd8007a0ba04b9b1051d7988c690b5b"}}]}] BACKPORT--> Co-authored-by: Mykola Harmash <[email protected]>
SoniaSanzV
pushed a commit
to SoniaSanzV/kibana
that referenced
this issue
Dec 9, 2024
…Firehose flow (elastic#201613) Closes elastic#200931 Switched to using AWS service list as a base to showing the detected services in the UI instead of the list of populated indices as multiple indices can be related to a single service. ### How to test 1. Go to Firehose flow `/observabilityOnboarding/firehose` 2. Open Kibana dev tools in another tab 3. Ingest documents related into multiple data streams which related to a single AWS service: ``` POST logs-aws.apigateway_logs-default/_doc { "@timestamp": "2024-11-25T13:32:01.000Z", "some": 111, "aws.kinesis.name": "Elastic-CloudwatchLogs" } POST metrics-aws.apigateway_metrics-default/_doc { "@timestamp": "2024-11-25T13:31:01.000Z", "agent": { "type": "firehose" }, "aws": { "cloudwatch": { "namespace": "AWS/ApiGateway" }, "exporter": { "arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J" } }, "cloud": { "account": { "id": "975050175126" }, "provider": "aws", "region": "us-west-2" } } ``` 4. Make sure you see only one entry for the service appear in the Firehose flow
SoniaSanzV
pushed a commit
to SoniaSanzV/kibana
that referenced
this issue
Dec 9, 2024
…Firehose flow (elastic#201613) Closes elastic#200931 Switched to using AWS service list as a base to showing the detected services in the UI instead of the list of populated indices as multiple indices can be related to a single service. ### How to test 1. Go to Firehose flow `/observabilityOnboarding/firehose` 2. Open Kibana dev tools in another tab 3. Ingest documents related into multiple data streams which related to a single AWS service: ``` POST logs-aws.apigateway_logs-default/_doc { "@timestamp": "2024-11-25T13:32:01.000Z", "some": 111, "aws.kinesis.name": "Elastic-CloudwatchLogs" } POST metrics-aws.apigateway_metrics-default/_doc { "@timestamp": "2024-11-25T13:31:01.000Z", "agent": { "type": "firehose" }, "aws": { "cloudwatch": { "namespace": "AWS/ApiGateway" }, "exporter": { "arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J" } }, "cloud": { "account": { "id": "975050175126" }, "provider": "aws", "region": "us-west-2" } } ``` 4. Make sure you see only one entry for the service appear in the Firehose flow
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Dec 9, 2024
…Firehose flow (elastic#201613) Closes elastic#200931 Switched to using AWS service list as a base to showing the detected services in the UI instead of the list of populated indices as multiple indices can be related to a single service. ### How to test 1. Go to Firehose flow `/observabilityOnboarding/firehose` 2. Open Kibana dev tools in another tab 3. Ingest documents related into multiple data streams which related to a single AWS service: ``` POST logs-aws.apigateway_logs-default/_doc { "@timestamp": "2024-11-25T13:32:01.000Z", "some": 111, "aws.kinesis.name": "Elastic-CloudwatchLogs" } POST metrics-aws.apigateway_metrics-default/_doc { "@timestamp": "2024-11-25T13:31:01.000Z", "agent": { "type": "firehose" }, "aws": { "cloudwatch": { "namespace": "AWS/ApiGateway" }, "exporter": { "arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J" } }, "cloud": { "account": { "id": "975050175126" }, "provider": "aws", "region": "us-west-2" } } ``` 4. Make sure you see only one entry for the service appear in the Firehose flow
CAWilson94
pushed a commit
to CAWilson94/kibana
that referenced
this issue
Dec 12, 2024
…Firehose flow (elastic#201613) Closes elastic#200931 Switched to using AWS service list as a base to showing the detected services in the UI instead of the list of populated indices as multiple indices can be related to a single service. ### How to test 1. Go to Firehose flow `/observabilityOnboarding/firehose` 2. Open Kibana dev tools in another tab 3. Ingest documents related into multiple data streams which related to a single AWS service: ``` POST logs-aws.apigateway_logs-default/_doc { "@timestamp": "2024-11-25T13:32:01.000Z", "some": 111, "aws.kinesis.name": "Elastic-CloudwatchLogs" } POST metrics-aws.apigateway_metrics-default/_doc { "@timestamp": "2024-11-25T13:31:01.000Z", "agent": { "type": "firehose" }, "aws": { "cloudwatch": { "namespace": "AWS/ApiGateway" }, "exporter": { "arn": "arn:aws:cloudwatch:us-west-2:975050175126:metric-stream/Elastic-CloudwatchLogsAndMetricsToFirehose-CloudWatchMetricStream-Nhb4NhzPdL4J" } }, "cloud": { "account": { "id": "975050175126" }, "provider": "aws", "region": "us-west-2" } } ``` 4. Make sure you see only one entry for the service appear in the Firehose flow
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
AWS Services that have both metrics and logs indices are duplicated. The service list should be based on the services, not on the underlying index.
The text was updated successfully, but these errors were encountered: