[apache_tomcat] [microsoft_sqlserver] [stan] Errors when testing with Elastic Agent wolfi images

[mrodm]

Testing to run system tests using Elastic Agent docker images based on Wolfi images #10933, raised the issue that system tests are failing for these packages (apache_tomcat, microsoft_sqlserver and stan).

The errors that comes from the buildkite build are that elastic-package could not find hits in the data stream:

test case failed: could not find hits in logs-apache_tomcat.catalina-84034 data stream
test case failed: could not find hits in logs-apache_tomcat.access-60721 data stream
test case failed: could not find hits in logs-microsoft_sqlserver.log-20239 data stream
test case failed: could not find hits in logs-stan.log-82655 data stream

For these packages, documents were not ingested due to permission errors while reading those files by Elastic Agent.

• apache_tomcat package:

{"log.level":"error","@timestamp":"2024-08-29T16:48:47.669Z","message":"File could not be opened for reading: failed opening /tmp/service_logs/catalina.2024-08-29.log: open /tmp/service_logs/catalina.2024-08-29.log: permission denied","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-default","type":"filestream"},"log":{"source":"filestream-default"},"source_file":"filestream::filestream-apache_tomcat.catalina-e5df2680-3558-4118-a15a-804e746d2ae9::native::9454105-64515","path":"/tmp/service_logs/catalina.2024-08-29.log","ecs.version":"1.6.0","log.logger":"input.filestream","log.origin":{"file.line":146,"file.name":"filestream/input.go","function":"github.com/elastic/beats/v7/filebeat/input/filestream.(*filestream).Run"},"service.name":"filebeat","id":"filestream-apache_tomcat.catalina-e5df2680-3558-4118-a15a-804e746d2ae9","state-id":"native::9454105-64515","ecs.version":"1.6.0"}

• microsoft_sqlserver:

{"log.level":"error","@timestamp":"2024-09-04T10:59:01.500Z","message":"Harvester could not be started on new file: /tmp/service_logs/errorlog, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /tmp/service_logs/errorlog: open /tmp/service_logs/errorlog: permission denied","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"finished":false,"log.logger":"input","source_file":"/tmp/service_logs/errorlog","state_id":"native::9844199-64515","ecs.version":"1.6.0","log.origin":{"file.line":570,"file.name":"log/input.go","function":"github.com/elastic/beats/v7/filebeat/input/log.(*Input).scan"},"service.name":"filebeat","input_id":"940fda73-b249-4ae7-b07d-5757cb30ecdb","os_id":"9844199-64515","ecs.version":"1.6.0"}

• stan package:

{"log.level":"warn","@timestamp":"2024-08-29T17:01:36.831Z","log.origin":{"function":"github.com/elastic/elastic-agent/internal/pkg/agent/application/coordinator.(*Coordinator).watchRuntimeComponents","file.name":"coordinator/coordinator.go","file.line":663},"message":"Unit state changed log-default-logfile-stan-2dd61177-25e0-47e0-9d06-9aa86ef6b41e (HEALTHY->DEGRADED): Harvester could not be started on new file: /tmp/service_logs/stan.log, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening /tmp/service_logs/stan.log: open /tmp/service_logs/stan.log: permission denied","log":{"source":"elastic-agent"},"component":{"id":"log-default","state":"HEALTHY"},"unit":{"id":"log-default-logfile-stan-2dd61177-25e0-47e0-9d06-9aa86ef6b41e","type":"input","state":"DEGRADED","old_state":"HEALTHY"},"ecs.version":"1.6.0"}

Permission of the files for easch package:

• apache_tomcat

bash-5.2$ ls -l /tmp/service_logs/
total 16
-rw-r-----    1 root     root          5855 Aug 29 16:48 catalina.2024-08-29.log
-rw-r--r--    1 root     root           319 Aug 29 16:48 localhost.log
-rw-r-----    1 root     root           980 Aug 29 16:55 localhost_access_log.2024-08-29.txt

• microsoft_sqlserver

 $ docker exec -it elastic-package-agent-microsoft_sqlserver-log-37113-elastic-agent-1 /bin/bash 
bash-5.2$ ls -l /tmp/service_logs/ |grep errorlog
-rw-r-----    1 root     root         10754 Sep  4 10:54 errorlog
-rw-r-----    1 root     root             0 Sep  4 10:54 errorlog.1

• stan

 $ docker exec -it elastic-package-agent-stan-log-64803-elastic-agent-1 /bin/bash 
bash-5.2$ ls -l /tmp/service_logs/
total 1940
-rw-r-----    1 root     root       1985018 Aug 29 17:02 stan.log

It is needed to check if the permissions of these files could be changed somehow in tests so Elastic Agent user could read those.

As a workarounds, there could be some different approaches:

• Force to use Elastic Agent Ubuntu image by updating the configuration file of the given system tests (agent.base_image: systemd).
  • To avoid failing these packages for now, this is the option used in Update packages to use Elastic Agent images based on Ubuntu #11007
• Run system tests with user root by updating the configuration file of the given system tests (agent.user: root).

This elastic-package issue elastic/elastic-package#1852 could be related to avoid permission errors related to the parent folder, but probably the packages would have to change the user that writes the logs.

[cmacknz]

It should be possible to fix this with Wolfi by doing docker run --user elastic-agent:root or docker run --user root. The wolfi image is elastic-agent:elastic-agent by default now. It previously was elastic-agent:root.

This will be the same thing users need to do it make this work.

[cmacknz]

We could set these data streams as requiring root (see the relevant part of the package spec here) but I don't think this is strictly required, it depends on the user the services are run as most likely. We could also look at changing the service users to not be root, although for Apache it may legitimately be root when run to bind to privileged ports.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pkoutsovasilis]

hi 👋 just thinking out loud here since elastic-agent needs to read files owned by root, and given that we would like to avoid making part of the root group, can't we just give to it the permission DAC_READ_SEARCH?!

[mrodm]

We could set these data streams as requiring root (see the relevant part of the package spec here) but I don't think this is strictly required, it depends on the user the services are run as most likely. We could also look at changing the service users to not be root, although for Apache it may legitimately be root when run to bind to privileged ports.

@cmacknz It could be set that just for those tests elastic-package configures the elastic-agent container to run with user root (via docker-compose user key). That could be done without requiring root privileges for the data stream (or the package) by adding in the system test configuration files this setting (just in the failing tests):

agent:
  user: root

I could update the PR #11007 to apply those settings. That would still be using the Wolfi images for tests in those packages (but the Elastic Agent container running as root).

Tested here (commit 17fdeab) : https://buildkite.com/elastic/integrations/builds/15513

WDYT @jlind23 @cmacknz ?

[jlind23]

@mrodm using root instead of any other user make sense to me here.

[mrodm]

Perfect, I'll update then the PR, thanks @jlind23 !