[system_audit] Errors when testing with Elastic Agent wolfi images

[mrodm]

Testing to run system tests using Elastic Agent docker images based on Wolfi images #10933, raised the issue that system tests are failing for system_audit package.

The error that comes from the buildkite build are that elastic-package could not find hits in the data stream:

test case failed: could not find hits in logs-system_audit.package-86982 data stream

This package has already defined the root permissions in their manifest, but it still does not ingest any docs in Elasticsearch.

Is this package using this beat https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-dataset-system-package.html ? If that is the case, Wolfi images use apk as its package manager and it would not be supported.

Should this package be using the Elastic Agent docker image based on Ubuntu to run the system tests? @elastic/sec-linux-platform

[jlind23]

@andrewkroh wolfi images are soon going to be the default for our all tests, shall we remove the system audit tests from the pipeline as we know they are going to fail anyway?
cc @norrietaylor

[cmacknz]

The Wolfi image changed the default user:group from elastic-agent:root to elastic-agent:elastic-agent, depending on what this does it may be fixable by explicitly running the container with docker run --user elastic-agent:root to elevate into the root group. This is what we'd expect users to do. This puts this in the same category as #10999.

It could also be like #10998 for journald where were assuming the Ubuntu container is approximately equal to a native Ubuntu machine, which is no longer true on purpose with Wolfi. In that case the test setup needs to change, especially if it isn't reasonably expected that this integration would be run in a container (much like most people won't expect to have journald in a container and we just test it there out of convenience).

[andrewkroh]

Is this package using this beat https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-dataset-system-package.html ? If that is the case, Wolfi images use apk as its package manager and it would not be supported.

Yes, system_audit includes the Auditbeat system.package dataset. If it doesn't have any rpm/deb/homebrew then it will not produce any package events, so then the system test would fail.

Is the ubuntu image going to be completely removed? If not, then could we run this test with a separate agent using the ubuntu image? If not, let's mark the test as skipped. It would be nice to add apk support to auditbeat at some point and then we can re-enable this test.

[jlind23]

Is the ubuntu image going to be completely removed? If not, then could we run this test with a separate agent using the ubuntu image? If not, let's mark the test as skipped. It would be nice to add apk support to auditbeat at some point and then we can re-enable this test.

It will most probably be removed by 9.X so we can still run those tests on ubuntu until then but they will eventually fail afterwards.