Ansible won't run on a Fedora CoreOS host

[beyondbill]

Fedora CoreOS info

[core@ip-10-10-8-158 ~]$ rpm-ostree status
State: idle
Deployments:
* ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200629.3.0 (2020-07-10T17:58:03Z)
                    Commit: 6df95bdb2fe2d36e091d4d18e3844fa84ce4b80ea3bd0947db5d7a286ff41890
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0

Ansible detects whether SELinux is enabled by checking the existence of /usr/sbin/selinuxenabled.
https://github.com/ansible/ansible/blob/v2.9.10/lib/ansible/module_utils/basic.py#L832
Ansible will throw an error if SELinux is enabled but python bindings (libselinux-python) aren't installed, which is the case in
Fedora CoreOS today. This prevents us from using Ansible to manage hosts that are on Fedora CoreOS.

Proposed solution
Remove /usr/sbin/selinuxenabled assuming it is not in use. This was the solution for the same issue on CoreOS.
coreos/coreos-overlay#1486

[lucab]

Thanks for the report. Contrarily to Container Linux, SELinux on Fedora CoreOS is enabled and in enforcing mode:

$ grep -o selinux /proc/filesystems
selinux
$ cat /sys/fs/selinux/enforce
1

Thus, removing the binary would result in Ansible (and possibly other tools) misdetecting the environment. A better outcome would be for all these tools to recognize that they are running under SELinux enforcing, and behave accordingly.

FCOS does not ship Ansible, nor python interpreters, nor python libraries by design. Assuming you are bringing your own executable and interpreter to the FCOS machine, it would be good for the bundle to take care of its python dependencies too (e.g. python-selinux).

[beyondbill]

@lucab Thanks for the quick response! If SELinux is enforced, then yes, the binary needs to stay. I managed to get Pypi and Pip working on FCOS but couldn't find a way to install libselinux-python. Do you have any suggestion that may help?

[lucab]

@beyondbill I'm sorry, I don't have personal experience/feedback on this (I'm not a python/ansible person), but from a quick google search this seems a common issue. Google also brings up a few workarounds like https://github.com/pycontribs/selinux.
You may want to ask on Ansible channel what's the recommend way to bring up a portable SELinux-aware python environment for their project (without installing stuff directly on the host).

[beyondbill]

@lucab I did see similar issues get a lot of mentions on Google. But so far I have not found any solution without installing libselinux-python package. The link you posted above is a solution for python with virtualenv and according to the readme,

You still need to have libselinux python bindings package installed on your system for it to work

I'll ask this on Ansible channel as you suggested. In the meantime, I also want to explore the following options (in time order):

1. Downloading a pre-compiled libselinux-python binary
2. Compiling libselinux-python binary from source code on FCOS
3. Installing libselinux-python package in a privileged way on FCOS

These are just thoughts and I do not know whether they are feasible. If you know any one is absolutely out of picture, please call out. Thank you!

[jlebon]

Ansible itself is great! For FCOS though, the model we're going for is that all configuration should be part of the Ignition config. So ideally you wouldn't have to run Ansible at all as a post-processing step. I understand that may be difficult to do though if you already have lots of playbooks written up. You might be able to get this working by running a container with the Python bindings and sshd running on a different port and the root filesystem mounted in.

That said, long-term adapting your workflow to use Ignition would be the best approach. See e.g. https://fedoramagazine.org/introducing-fedora-coreos/ for more details.

Hmm, would be interesting actually to have a tool which tries to convert Ansible playbooks to Ignition configs. That will fall over for the more complex stuff, but things like creating/appending to files and managing systemd services, etc... would work fine.

[beyondbill]

Thank you @jlebon! I really like the suggestion to abandon Ansible and move on with FCC / Ignition! Our Ansible playbooks are not terribly complex, mostly sshd security hardening and sudo authorization (via pam_ssh_agent_auth, which is a shared object at least portable on Container Linux). The feasibility of converting these playbooks to FCC / Ignition looks good to me.

[DirkTheDaring]

we rely heavily on ansible, if it doesn't work anymore for CoreOS intentionally, then this causes heavy workload here.
There are happening so many things, that are not covered by ignition.
e.g. we fought against SELinux disabling execution of scripts created by ignition, but more troubles are there, which are now covered by ansible.

**Please make this decision very clear in the docs ** --> then we will drop Fedora CoreOS.

[bgilbert]

Ansible isn't our recommended deployment model and we don't dedicate time to making it work, but we certainly don't intentionally break it either. If anyone has a chance to experiment with running Ansible on Fedora CoreOS in a container, and would like to write up some documentation, we'd be happy to add it to the Fedora CoreOS docs site.

[DirkTheDaring]

@bgilbert
See here How to make ansible work in Fedora CoreOS again

It is quite tedious and really is now fighting against Fedora CoreOS, so a decision to move away, might be required soon.