Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[rawhide][branched] SELinux AVC denials causes the afterburn write to fail in cloud platforms #1784

Open
aaradhak opened this issue Aug 20, 2024 · 2 comments
Labels
area/selinux kind/bug pipeline failure This issue or pull request is derived from CI failures

Comments

@aaradhak
Copy link
Member

aaradhak commented Aug 20, 2024

Describe the bug

In the recent rawhide & branched cloud platform builds, the kola tests are failing due to an error in the afterburn service.

harness.go:1823: mach.Start() failed: machine "i-0c58eeb79e70a7d44" failed basic checks: detected failed or stuck systemd units: some systemd units failed: [email protected]; <nil>

On further debugging, it is found that the afterburn process attempted to write to the /var/home/core/.ssh/authorized_keys.d/ directory but was denied by SELinux . This denial caused the afterburn-sshkeys service to fail with a "Permission denied (os error 13)" error.

This seems to be like a selinux-policy issue.

Aug 20 17:00:43.703000 audit[1592]: AVC avc:  denied  { write } for  pid=1592 comm="afterburn" name="authorized_keys.d" dev="nvme0n1p4" ino=20971648 scontext=system_u:system_r:afterburn_t:s0 tcontext=unconfined_u:object_r:ssh_home_t:s0 tclass=dir permissive=0
Aug 20 17:00:43.705504 [email protected][1592]: Error: failed to run
Aug 20 17:00:43.705504 [email protected][1592]: Caused by:
Aug 20 17:00:43.705504 [email protected][1592]:     0: writing ssh keys
Aug 20 17:00:43.705504 [email protected][1592]:     1: failed to create temporary file
Aug 20 17:00:43.707051 [email protected][1592]:     2: Permission denied (os error 13) at path "/var/home/core/.ssh/authorized_keys.d/.afterburn-gRJMKD"
Aug 20 17:00:43.707269 init.scope[1]: [email protected]: Main process exited, code=exited, status=1/FAILURE
Aug 20 17:00:43.707481 init.scope[1]: [email protected]: Failed with result 'exit-code'.
Aug 20 17:00:43.707727 init.scope[1]: Failed to start [email protected] - Afterburn (SSH Keys).

Apart from the above AVC denials, came across few other AVC denials in the journal log as below:

Aug 20 17:00:37.025779 kernel: audit: type=1400 audit(1724173234.341:4): avc:  denied  { getattr } for  pid=1367 comm="coreos-boot-mou" path="/run/coreos/bootfs_uuid" dev="tmpfs" ino=870 scontext=system_u:system_r:coreos_boot_mount_generator_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
Aug 20 17:00:37.025790 kernel: audit: type=1400 audit(1724173234.357:5): avc:  denied  { read } for  pid=1403 comm="cat" name="bootfs_uuid" dev="tmpfs" ino=870 scontext=system_u:system_r:coreos_boot_mount_generator_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
Aug 20 17:00:37.025801 kernel: audit: type=1400 audit(1724173234.357:6): avc:  denied  { open } for  pid=1403 comm="cat" path="/run/coreos/bootfs_uuid" dev="tmpfs" ino=870 scontext=system_u:system_r:coreos_boot_mount_generator_t:s0 tcontext=system_u:object_r:var_run_t:s0 tclass=file permissive=1
Aug 20 17:00:37.025813 systemd[1]: Populated /etc with preset unit settings.
Aug 20 17:00:37.029860 systemd[1]: initrd-switch-root.service: Deactivated successfully.

Reproduction steps

Start a pipeline job build of the kola cloud platforms.

  • kola-aws
  • kola-gcp
  • kola-azure
  • kola-openstack

Expected behavior

The SELinux policy to allow the afterburn process to write to the directory in question.

Actual behavior

kola tests fails with this error:

harness.go:1823: mach.Start() failed: machine "i-0c58eeb79e70a7d44" failed basic checks: detected failed or stuck systemd units: some systemd units failed: [email protected]; <nil>

System details

Kola cloud platform pipeline jobs.
Streams - rawhide & branched

  • kola-aws
  • kola-gcp
  • kola-azure
  • kola-openstack

Butane or Ignition config

No response

Additional information

There's a similar afterburn issue that was filed against c9s that was fixed, but possibly the fixes there need to be brought to Fedora too: https://issues.redhat.com/browse/RHEL-49735

@aaradhak aaradhak added kind/bug pipeline failure This issue or pull request is derived from CI failures labels Aug 20, 2024
@aaradhak
Copy link
Member Author

aaradhak commented Aug 20, 2024

Opened a BZ for this selinux-policy avc denial - https://bugzilla.redhat.com/show_bug.cgi?id=2306352

marmijo added a commit to marmijo/fedora-coreos-config that referenced this issue Sep 3, 2024
Recent changes in the SELinux policy have broken a lot of our code.
Revert the affected domains back to permissive mode so we can
continue to build and test `releasever >= 41` until
fedora-selinux/selinux-policy#2257 merges
and the domains are reverted upstream or until the issue is resolved
altogether.

Add the workaround for `afterburn_t` as well so we can unblock
coreos/fedora-coreos-tracker#1784
marmijo added a commit to marmijo/fedora-coreos-config that referenced this issue Sep 3, 2024
Recent changes in the SELinux policy have broken a lot of our code.
Revert the affected domains back to permissive mode so we can
continue to build and test `releasever >= 41` until
fedora-selinux/selinux-policy#2257 merges
and the domains are reverted upstream or until the issue is resolved
altogether.

Add the workaround for `afterburn_t` as well so we can unblock
coreos/fedora-coreos-tracker#1784
marmijo added a commit to coreos/fedora-coreos-config that referenced this issue Sep 4, 2024
Recent changes in the SELinux policy have broken a lot of our code.
Revert the affected domains back to permissive mode so we can
continue to build and test `releasever >= 41` until
fedora-selinux/selinux-policy#2257 merges
and the domains are reverted upstream or until the issue is resolved
altogether.

Add the workaround for `afterburn_t` as well so we can unblock
coreos/fedora-coreos-tracker#1784
@marmijo
Copy link
Member

marmijo commented Sep 5, 2024

A workaround was added for this in: coreos/fedora-coreos-config#3127.
We're now able to run the affected kola cloud tests in rawhide and branched.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/selinux kind/bug pipeline failure This issue or pull request is derived from CI failures
Projects
None yet
Development

No branches or pull requests

3 participants