Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add system tests to logstash package #4443

Merged
merged 9 commits into from
Nov 1, 2022
Merged

Add system tests to logstash package #4443

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
03670c2
Add system tests for logstash package
crespocarlos Oct 11, 2022
72dfc90
Update README
crespocarlos Oct 11, 2022
6f69d52
Test .env file
crespocarlos Oct 12, 2022
9b9429d
Set default env var value
crespocarlos Oct 12, 2022
3fdd75c
Fix alias mapping
crespocarlos Oct 12, 2022
8bb0994
Fix mapping and use Hostname var on test config
crespocarlos Oct 12, 2022
3b4ffe8
Add missing filed to node_stats mapping
crespocarlos Oct 12, 2022
ec8d0b5
Fix docker-compose file
crespocarlos Oct 14, 2022
30fc344
Fix test config and replaces .env with variantes.yml
crespocarlos Oct 14, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion packages/logstash/_dev/deploy/docker/.env
View file
Open in desktop

This file was deleted.

7 changes: 4 additions & 3 deletions packages/logstash/_dev/deploy/docker/docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,11 @@
version: '2.3'
version: "2.3"
services:
logstash:
image: "docker.elastic.co/logstash/logstash:${ELASTIC_VERSION}"
user: root
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The container needs to be accessed by root user to have permission to write in the ${SERVICE_LOGS_DIR} folder

image: "docker.elastic.co/logstash/logstash:${ELASTIC_VERSION:-8.5.0-SNAPSHOT}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.env file doesn't work in the CI, 8.5.0-SNAPSHOT is the default value, in case the var is null. We need to figure out how to make this a dynamic value.

Copy link
Contributor

@klacabane klacabane Oct 14, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we can create a variants.yml file that defines all the versions we want to run the test suite against. The variable defined in the variants file are then available in the docker compose - see #4013 (comment)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah. nice! I'll try this

volumes:
- "./pipeline:/usr/share/logstash/pipeline"
- "./config:/usr/share/logstash/config"
- ${SERVICE_LOGS_DIR}/logstash:/usr/share/logstash/logs
- ${SERVICE_LOGS_DIR}:/usr/share/logstash/logs
ports:
- "127.0.0.1:9600:9600"
4 changes: 4 additions & 0 deletions packages/logstash/_dev/deploy/variants.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
variants:
logstash_8.5.0:
ELASTIC_VERSION: 8.5.0-SNAPSHOT
default: logstash_8.5.0
5 changes: 5 additions & 0 deletions packages/logstash/data_stream/log/_dev/test/system/test-default-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
input: logfile
data_stream:
vars:
paths:
- "{{SERVICE_LOGS_DIR}}/logstash-json.log"
21 changes: 11 additions & 10 deletions packages/logstash/data_stream/log/sample_event.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{
"@timestamp": "2022-09-13T19:08:48.030Z",
"@timestamp": "2022-10-11T14:03:32.641Z",
"agent": {
"ephemeral_id": "c028b260-d373-4eab-978c-08bc42b9dc7c",
"id": "aeed8481-2d37-45f5-989f-daa7c2173ca0",
"ephemeral_id": "e1088945-f8e4-466b-83fd-f636ffbf9bdf",
"id": "79e48fe3-2ecd-4021-aed5-6e7e69d47606",
"name": "docker-fleet-agent",
"type": "filebeat",
"version": "8.5.0"
Expand All @@ -16,33 +16,34 @@
"version": "1.10.0"
},
"elastic_agent": {
"id": "aeed8481-2d37-45f5-989f-daa7c2173ca0",
"id": "79e48fe3-2ecd-4021-aed5-6e7e69d47606",
"snapshot": true,
"version": "8.5.0"
},
"event": {
"agent_id_status": "verified",
"created": "2022-10-11T14:03:32.641Z",
"dataset": "logstash.log",
"ingested": "2022-09-13T19:08:58Z",
"ingested": "2022-10-11T14:03:44Z",
"kind": "event",
"type": "info"
},
"host": {
"architecture": "x86_64",
"containerized": true,
"containerized": false,
"hostname": "docker-fleet-agent",
"id": "8127511256f0493fa1abf625ca3e0609",
"id": "b6bc6723e51b43959ce07f0c3105c72d",
"ip": [
"172.20.0.8"
"192.168.0.7"
],
"mac": [
"02-42-AC-14-00-08"
"02-42-C0-A8-00-07"
],
"name": "docker-fleet-agent",
"os": {
"codename": "focal",
"family": "debian",
"kernel": "5.10.47-linuxkit",
"kernel": "5.10.124-linuxkit",
"name": "Ubuntu",
"platform": "ubuntu",
"type": "linux",
Expand Down
6 changes: 6 additions & 0 deletions packages/logstash/data_stream/node/_dev/test/system/test-default-config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
type: logstash/metrics
dataset: logstash.stack_monitoring.node
vars:
hosts:
- "http://{{Hostname}}:9600"
data_stream: ~
2 changes: 2 additions & 0 deletions packages/logstash/data_stream/node/fields/fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
- name: logstash
type: group
fields:
- name: cluster.id
type: keyword
- name: elasticsearch.cluster.id
type: keyword
- name: node
Expand Down
86 changes: 1 addition & 85 deletions packages/logstash/data_stream/node/fields/package-fields.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,91 +3,7 @@
path: logstash.elasticsearch.cluster.id
- name: timestamp
type: alias
path: '@timestamp'
- name: logstash_stats
type: group
fields:
- name: timestamp
type: alias
path: '@timestamp'
- name: jvm
type: group
fields:
- name: mem
type: group
fields:
- name: heap_used_in_bytes
type: alias
path: logstash.node.stats.jvm.mem.heap_used_in_bytes
- name: heap_max_in_bytes
type: alias
path: logstash.node.stats.jvm.mem.heap_max_in_bytes
- name: uptime_in_millis
type: alias
path: logstash.node.stats.jvm.uptime_in_millis
- name: events
type: group
fields:
- name: in
type: alias
path: logstash.node.stats.events.in
- name: out
type: alias
path: logstash.node.stats.events.out
- name: duration_in_millis
type: alias
path: logstash.node.stats.events.duration_in_millis
- name: logstash
type: group
fields:
- name: uuid
type: alias
path: logstash.node.stats.logstash.uuid
- name: version
type: alias
path: logstash.node.stats.logstash.version
- name: os
type: group
fields:
- name: cpu
type: group
fields:
- name: stat
type: group
fields:
- name: number_of_elapsed_periods
type: alias
path: logstash.node.stats.os.cgroup.cpu.stat.number_of_elapsed_periods
- name: time_throttled_nanos
type: alias
path: logstash.node.stats.os.cgroup.cpu.stat.time_throttled_nanos
- name: number_of_times_throttled
type: alias
path: logstash.node.stats.os.cgroup.cpu.stat.number_of_times_throttled
- name: load_average
type: group
fields:
- name: 15m
type: alias
path: logstash.node.stats.os.cpu.load_average.15m
- name: 1m
type: alias
path: logstash.node.stats.os.cpu.load_average.1m
- name: 5m
type: alias
path: logstash.node.stats.os.cpu.load_average.5m
- name: cgroup
type: group
fields:
- name: cpuacct.usage_nanos
type: alias
path: logstash.node.stats.os.cgroup.cpuacct.usage_nanos
- name: process.cpu.percent
type: alias
path: logstash.node.stats.process.cpu.percent
- name: queue.events_count
type: alias
path: logstash.node.stats.queue.events_count
path: "@timestamp"
- name: logstash_state
type: group
fields:
Expand Down
Loading