Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate obs-infraobs-integrations to package-spec v3 #1 #8170

Merged
merged 52 commits into from
Oct 18, 2023
+484 −370
Merged

Migrate obs-infraobs-integrations to package-spec v3 #1 #8170

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
b909cc5
[activemq]: migration with yq
shmsr Oct 11, 2023
b16b5b3
[airflow]: migration with yq
shmsr Oct 11, 2023
48ff78f
[apache]: migration with yq
shmsr Oct 11, 2023
264e1bd
[apache_spark]: migration with yq
shmsr Oct 11, 2023
0cc6abb
[cassandra]: migration with yq
shmsr Oct 11, 2023
079d4c2
[cockroachdb]: migration with yq
shmsr Oct 11, 2023
82f1cee
[couchbase]: migration with yq
shmsr Oct 11, 2023
1d94721
[couchdb]: migration with yq
shmsr Oct 11, 2023
a1a8b01
[etcd]: migration with yq
shmsr Oct 11, 2023
b7aad09
[activemq] - removed dotted YAML keys from manifest
shmsr Oct 11, 2023
ca95568
[apache_spark] - removed dotted YAML keys from manifest
shmsr Oct 11, 2023
c0bb399
[apache_tomcat] - removed dotted YAML keys from manifest
shmsr Oct 11, 2023
4e56a84
[cassandra] - removed dotted YAML keys from manifest
shmsr Oct 11, 2023
11c1b97
[couchdb] - removed dotted YAML keys from manifest
shmsr Oct 11, 2023
d1eca9b
[airflow] - removed dotted YAML keys from manifest
shmsr Oct 12, 2023
958c974
[azure_functions] - removed dotted YAML keys from manifest
shmsr Oct 12, 2023
7475ef9
[ceph] - removed dotted YAML keys from manifest
shmsr Oct 12, 2023
9f3ebda
[cockroachdb] - removed dotted YAML keys from manifest
shmsr Oct 12, 2023
c664f18
[coredns] - removed dotted YAML keys from manifest
shmsr Oct 12, 2023
3062b70
[couchbase]: migration with yq
shmsr Oct 12, 2023
75e0ffe
[couchbase] - removed dotted YAML keys from manifest
shmsr Oct 12, 2023
95c1151
[etcd] - removed dotted YAML keys from manifest
shmsr Oct 12, 2023
bf0c44e
Add validation.yml
shmsr Oct 12, 2023
b61f518
Set correct PR number in changelog
shmsr Oct 12, 2023
8d0dd55
Revert "[apache]: migration with yq"
shmsr Oct 12, 2023
30b6a67
[apache_spark]: normalize fields
shmsr Oct 12, 2023
375475e
[cassandra]: normalize fields
shmsr Oct 12, 2023
834f77d
[apache_spark]: fix formatting to make ci happy
shmsr Oct 12, 2023
bb567c7
[cassandra]: fix formatting to make ci happy
shmsr Oct 12, 2023
7ffb828
Format packages and quote fields where necessary
shmsr Oct 12, 2023
80470c1
[couchbase]: s/metrics_type/metric_type
shmsr Oct 12, 2023
d7380dc
[airflow]: remove duplicates from agent.yml instead of ecs.yml
shmsr Oct 12, 2023
b3f91e5
[cockroachdb]: remove duplicates from agent.yml instead of ecs.yml
shmsr Oct 12, 2023
769e5f6
revert changes for packages for which v3 migration failed
shmsr Oct 12, 2023
2018c88
Format packages and quote fields where necessary
shmsr Oct 12, 2023
fddd083
Merge branch 'main' into migrate-infraobs-ps-v3-batch-1-serverless
shmsr Oct 12, 2023
dcd7757
Fix inconsistency w/ elastic-build
shmsr Oct 12, 2023
10973a1
quote fields where necessary
shmsr Oct 16, 2023
b689df1
Move container.labels to ecs.yml
shmsr Oct 16, 2023
3e586a8
Change type from object to unsinged_long
shmsr Oct 16, 2023
7782e02
[ceph]: Explicit mapping for state.count and state.state_name
shmsr Oct 16, 2023
d1d9fd0
Merge branch 'main' into migrate-infraobs-ps-v3-batch-1-serverless
shmsr Oct 16, 2023
3829cd6
[apache_spark]: migration with yq
shmsr Oct 16, 2023
87f32a7
[apache_spark]: pass elastic-packge test static
shmsr Oct 16, 2023
d464dcb
[apache_spark]: format
shmsr Oct 16, 2023
b2f9b8c
[apache_spark]: format
shmsr Oct 16, 2023
6fdcc5d
Make event.type's value formatting consistent
shmsr Oct 17, 2023
b8b7c0b
Merge branch 'main' into migrate-infraobs-ps-v3-batch-1-serverless
shmsr Oct 17, 2023
0c30a24
Change type from nested to group
shmsr Oct 17, 2023
3869925
Change type from unsigned_long to long
shmsr Oct 17, 2023
5ca725b
Quote index.mapping.dimension_fields.limit to avoid getting changed b…
shmsr Oct 17, 2023
155a5fa
Revert "Quote index.mapping.dimension_fields.limit to avoid getting c…
shmsr Oct 17, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion packages/activemq/_dev/deploy/variants.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
variants:
v5.17.1:
"v5.17.1":
SERVICE_VERSION: 5.17.1
default: v5.17.1
5 changes: 5 additions & 0 deletions packages/activemq/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: 0.14.0
changes:
- description: Update the package format_version to 3.0.0.
type: enhancement
link: https://github.com/elastic/integrations/pull/8170
- version: "0.13.1"
changes:
- description: Remove forwarded tag from metrics data streams.
Expand Down
8 changes: 4 additions & 4 deletions packages/activemq/data_stream/audit/_dev/test/pipeline/test-audit.log-expected.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
"version": "8.5.1"
},
tommyers-elastic marked this conversation as resolved.
Show resolved Hide resolved
"event": {
"ingested": "2022-12-08T15:06:10.692324051Z",
"ingested": "2023-10-11T20:53:28.142058918Z",
"kind": "event",
"module": "activemq",
"original": "INFO | anonymous called org.apache.activemq.broker.jmx.QueueView.retryMessages[] at 27-11-2019 08:45:57,213 | qtp443290224-47",
Expand Down Expand Up @@ -41,7 +41,7 @@
"version": "8.5.1"
},
"event": {
"ingested": "2022-12-08T15:06:10.692336343Z",
"ingested": "2023-10-11T20:53:28.142069543Z",
"kind": "event",
"module": "activemq",
"original": "INFO | admin called org.apache.activemq.broker.jmx.QueueView.retryMessages[] at 27-11-2019 08:45:57,229 | qtp443290224-45",
Expand Down Expand Up @@ -71,7 +71,7 @@
"version": "8.5.1"
},
"event": {
"ingested": "2022-12-08T15:06:10.692337760Z",
"ingested": "2023-10-11T20:53:28.142070835Z",
"kind": "event",
"module": "activemq",
"original": "WARN | admin requested /admin/createDestination.action [JMSDestination='test' JMSDestinationType='queue' secret='4eb0bc3e-9d7a-4256-844c-24f40fda98f1' ] from 127.0.0.1 | qtp12205619-39",
Expand Down Expand Up @@ -101,7 +101,7 @@
"version": "8.5.1"
},
"event": {
"ingested": "2022-12-08T15:06:10.692338926Z",
"ingested": "2023-10-11T20:53:28.142071877Z",
"kind": "event",
"module": "activemq",
"original": "INFO | guest requested /admin/purgeDestination.action [JMSDestination='test' JMSDestinationType='queue' secret='eff6a932-1b58-45da-a64a-1b30b246cfc9' ] from 127.0.0.1 | qtp12205619-36",
Expand Down
2 changes: 1 addition & 1 deletion packages/activemq/data_stream/audit/_dev/test/pipeline/test-common-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
dynamic_fields:
event.ingested: ".*"
"event.ingested": ".*"
fields:
"@timestamp": "2020-04-28T11:07:58.223Z"
tags:
Expand Down
40 changes: 20 additions & 20 deletions packages/activemq/data_stream/log/_dev/test/pipeline/test-activemq.log-expected.json
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion packages/activemq/data_stream/log/_dev/test/pipeline/test-common-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
dynamic_fields:
event.ingested: ".*"
"event.ingested": ".*"
fields:
"@timestamp": "2020-04-28T11:07:58.223Z"
tags:
Expand Down
2 changes: 0 additions & 2 deletions packages/activemq/data_stream/topic/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,6 @@
name: service.type
- external: ecs
name: tags
- external: ecs
name: tags
- external: ecs
name: agent.id
dimension: true
Expand Down
11 changes: 7 additions & 4 deletions packages/activemq/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
name: activemq
title: ActiveMQ
version: "0.13.1"
version: "0.14.0"
description: Collect logs and metrics from ActiveMQ instances with Elastic Agent.
type: integration
icons:
- src: /img/activemq.svg
title: activemq
size: 32x32
type: image/svg+xml
format_version: 1.0.0
license: basic
format_version: "3.0.0"
categories:
- message_queue
- observability
conditions:
kibana.version: ^8.8.0
kibana:
version: ^8.8.0
elastic:
subscription: basic
screenshots:
- src: /img/filebeat-activemq-log.png
title: filebeat ActiveMQ application events
Expand Down Expand Up @@ -116,3 +118,4 @@ policy_templates:
description: Collecting broker, queue and topic metrics from ActiveMQ instances.
owner:
github: elastic/obs-infraobs-integrations
type: elastic
4 changes: 4 additions & 0 deletions packages/activemq/validation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
errors:
exclude_checks:
- SVR00004
- SVR00002
5 changes: 5 additions & 0 deletions packages/airflow/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: 0.5.0
changes:
- description: Update the package format_version to 3.0.0.
type: enhancement
link: https://github.com/elastic/integrations/pull/8170
- version: "0.4.0"
changes:
- description: Enable time series data streams for the metrics datasets. This dramatically reduces storage for metrics and is expected to progressively improve query performance. For more details, see https://www.elastic.co/guide/en/elasticsearch/reference/current/tsds.html.
Expand Down
27 changes: 0 additions & 27 deletions packages/airflow/data_stream/statsd/fields/agent.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,33 +57,6 @@
- name: image.id
type: keyword
description: Image ID for the cloud instance.
- name: container
title: Container
group: 2
description: "Container fields are used for meta information about the specific container that is the source of information.\nThese fields help correlate data based containers from any runtime."
type: group
fields:
- name: id
level: core
type: keyword
ignore_above: 1024
description: Unique container id.
dimension: true
- name: image.name
level: extended
type: keyword
ignore_above: 1024
description: Name of the image the container was built on.
- name: labels
level: extended
type: object
object_type: keyword
description: Image labels.
- name: name
level: extended
type: keyword
ignore_above: 1024
description: Container name.
- name: host
title: Host
group: 2
Expand Down
2 changes: 2 additions & 0 deletions packages/airflow/data_stream/statsd/fields/ecs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,5 +14,7 @@
external: ecs
- name: container.image.name
external: ecs
- name: container.labels
external: ecs
- name: host
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tommyers-elastic , @andrewkroh - The host group ECS field reference is included in some integrations and some integrations doesn't include the ECS reference. As the host field is common, Is it worth adding validations for this field as well? Just a thought.
e.g. Integration not having host group field reference.

external: ecs
11 changes: 7 additions & 4 deletions packages/airflow/manifest.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
name: airflow
title: Airflow
version: "0.4.0"
version: "0.5.0"
description: Airflow Integration.
type: integration
format_version: 1.0.0
license: basic
format_version: "3.0.0"
categories:
- observability
conditions:
kibana.version: "^8.9.0"
kibana:
version: "^8.9.0"
elastic:
subscription: basic
icons:
- src: /img/airflow.svg
title: Airflow logo
Expand Down Expand Up @@ -42,3 +44,4 @@ policy_templates:
input_group: metrics
owner:
github: elastic/obs-infraobs-integrations
type: elastic
3 changes: 3 additions & 0 deletions packages/airflow/validation.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
errors:
exclude_checks:
- SVR00002
2 changes: 1 addition & 1 deletion packages/apache_spark/_dev/deploy/variants.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
variants:
v3.2.0:
"v3.2.0":
SERVICE_VERSION: 3.2.0
default: v3.2.0
5 changes: 5 additions & 0 deletions packages/apache_spark/changelog.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "0.8.0"
changes:
- description: Update the package format_version to 3.0.0.
type: enhancement
link: https://github.com/elastic/integrations/pull/8170
- version: "0.7.9"
changes:
- description: Add filters in visualizations.
Expand Down
2 changes: 1 addition & 1 deletion packages/apache_spark/data_stream/application/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ processors:
ignore_missing: true
- set:
field: event.type
value: info
value: [info]
tommyers-elastic marked this conversation as resolved.
Show resolved Hide resolved
- set:
field: event.kind
value: metric
Expand Down
8 changes: 6 additions & 2 deletions packages/apache_spark/data_stream/application/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,14 +34,18 @@
"ingested": "2023-09-28T09:24:37Z",
"kind": "metric",
"module": "apache_spark",
"type": "info"
"type": [
"info"
]
},
"host": {
"architecture": "x86_64",
"containerized": true,
"hostname": "docker-fleet-agent",
"id": "e8978f2086c14e13b7a0af9ed0011d19",
"ip": "172.20.0.7",
"ip": [
"172.20.0.7"
],
"mac": "02-42-AC-14-00-07",
"name": "docker-fleet-agent",
"os": {
Expand Down
2 changes: 1 addition & 1 deletion packages/apache_spark/data_stream/driver/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ processors:
ignore_missing: true
- set:
field: event.type
value: info
value: [info]
- set:
field: event.kind
value: metric
Expand Down
8 changes: 6 additions & 2 deletions packages/apache_spark/data_stream/driver/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,14 +38,18 @@
"ingested": "2023-09-29T12:04:41Z",
"kind": "metric",
"module": "apache_spark",
"type": "info"
"type": [
"info"
]
},
"host": {
"architecture": "x86_64",
"containerized": true,
"hostname": "docker-fleet-agent",
"id": "e8978f2086c14e13b7a0af9ed0011d19",
"ip": "172.26.0.7",
"ip": [
"172.26.0.7"
],
"mac": "02-42-AC-1A-00-07",
"name": "docker-fleet-agent",
"os": {
Expand Down
2 changes: 1 addition & 1 deletion packages/apache_spark/data_stream/executor/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ processors:
ignore_missing: true
- set:
field: event.type
value: info
value: [info]
- set:
field: event.kind
value: metric
Expand Down
8 changes: 6 additions & 2 deletions packages/apache_spark/data_stream/executor/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,14 +37,18 @@
"ingested": "2023-09-28T09:26:49Z",
"kind": "metric",
"module": "apache_spark",
"type": "info"
"type": [
"info"
]
},
"host": {
"architecture": "x86_64",
"containerized": true,
"hostname": "docker-fleet-agent",
"id": "e8978f2086c14e13b7a0af9ed0011d19",
"ip": "172.20.0.7",
"ip": [
"172.20.0.7"
],
"mac": "02-42-AC-14-00-07",
"name": "docker-fleet-agent",
"os": {
Expand Down
2 changes: 1 addition & 1 deletion packages/apache_spark/data_stream/node/elasticsearch/ingest_pipeline/default.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ processors:
ignore_missing: true
- set:
field: event.type
value: info
value: [info]
- set:
field: event.kind
value: metric
Expand Down
4 changes: 3 additions & 1 deletion packages/apache_spark/data_stream/node/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@
"ingested": "2022-04-12T04:42:53Z",
"kind": "metric",
"module": "apache_spark",
"type": "info"
"type": [
"info"
]
},
"host": {
"architecture": "x86_64",
Expand Down
28 changes: 21 additions & 7 deletions packages/apache_spark/docs/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -112,14 +112,18 @@ An example event for `application` looks as following:
"ingested": "2023-09-28T09:24:37Z",
"kind": "metric",
"module": "apache_spark",
"type": "info"
"type": [
"info"
]
},
"host": {
"architecture": "x86_64",
"containerized": true,
"hostname": "docker-fleet-agent",
"id": "e8978f2086c14e13b7a0af9ed0011d19",
"ip": "172.20.0.7",
"ip": [
"172.20.0.7"
],
"mac": "02-42-AC-14-00-07",
"name": "docker-fleet-agent",
"os": {
Expand Down Expand Up @@ -223,14 +227,18 @@ An example event for `driver` looks as following:
"ingested": "2023-09-29T12:04:41Z",
"kind": "metric",
"module": "apache_spark",
"type": "info"
"type": [
"info"
]
},
"host": {
"architecture": "x86_64",
"containerized": true,
"hostname": "docker-fleet-agent",
"id": "e8978f2086c14e13b7a0af9ed0011d19",
"ip": "172.26.0.7",
"ip": [
"172.26.0.7"
],
"mac": "02-42-AC-1A-00-07",
"name": "docker-fleet-agent",
"os": {
Expand Down Expand Up @@ -400,14 +408,18 @@ An example event for `executor` looks as following:
"ingested": "2023-09-28T09:26:49Z",
"kind": "metric",
"module": "apache_spark",
"type": "info"
"type": [
"info"
]
},
"host": {
"architecture": "x86_64",
"containerized": true,
"hostname": "docker-fleet-agent",
"id": "e8978f2086c14e13b7a0af9ed0011d19",
"ip": "172.20.0.7",
"ip": [
"172.20.0.7"
],
"mac": "02-42-AC-14-00-07",
"name": "docker-fleet-agent",
"os": {
Expand Down Expand Up @@ -579,7 +591,9 @@ An example event for `node` looks as following:
"ingested": "2022-04-12T04:42:53Z",
"kind": "metric",
"module": "apache_spark",
"type": "info"
"type": [
"info"
]
},
"host": {
"architecture": "x86_64",
Expand Down
Loading