cannot install "nfs-utils" any longer - conflict with nfs-utils-coreos

[kai-uwe-rommel]

# rpm-ostree status
State: idle
Deployments:
* ostree://fedora:fedora/x86_64/coreos/stable
                   Version: 32.20200615.3.0 (2020-06-30T09:14:54Z)
                    Commit: abb4b978999bd689ff36114e99c737be16026bc154a4428d4c84a340758c5b01
              GPGSignature: Valid signature by 97A1AE57C3A2372CCA3A4ABA6C13026D12C944D0
# rpm-ostree install nfs-utils
Checking out tree abb4b97... done
Resolving dependencies... done
Checking out packages... done
error: Checkout nfs-utils-1:2.5.1-0.fc32.x86_64: Hardlinking f1/ebafd6e675712eea051b7c8b17624a93b6f77589eecb5cf258f869e35227ab.file to mount.nfs: File exists

This is not the newest release of FCOS, but it fails with the newest release as well.
I explicitly went back one release because I DEFINITELY was able to do this successfully only a couple of days ago.

Please help, I do need this rather urgently as I we use this as part of a larger cluster deployment and I am basically stalled ... :-/
I don't know a workaround either.
Does this still work for someone else?

(This is in instances deployed with the VMware OVA.)

[cgwalters]

We already include nfs-utils-coreos - that should have basically the same functionality.

[kai-uwe-rommel]

But I need an NFS server, not only client.

[cgwalters]

The error message is not very helpful here I know, we probably want nfs-utils-coreos to Conflicts: nfs-utils.

[kai-uwe-rommel]

You are certainly right.
But how can I solve the problem?
Any idea?
It has worked until only a couple of days ago.

The install of nfs-utils does not even work any longer on instances which I installed two weeks ago at a time where it definitely worked. I have, for example, two instances installed two weeks ago. On one I installed nfs-utils at this time. No OS-updates have been applied since then (I disabled zincati). But I cannot install nfs-utils NOW on the other one of that age where I did not install it yet then.

So it must be a problem in the package in the repository online.
Can I somehow go back to an older repository version?

[bgilbert]

Package layering is subject to these sorts of problems. Layering of arbitrary packages is not recommended. Can you run your NFS server in a container?

[kai-uwe-rommel]

I can try this ...
But it nevertheless should be solved. There is a OS and there is this standard repository configured into it by default.
So I should be able to install the packages of that repository without such problems ...
I guess it is easy to fix (it has worked fine before) but who can do it? And when?

[cgwalters]

Agreed, using containers for servers would be recommended. That said rpm-ostree override remove nfs-utils-coreos --install nfs-utils works for me locally. I'd need to dig into why this conflict happened, I think it's a variant of coreos/rpm-ostree#415 AKA #401

[bgilbert]

@kai-uwe-rommel, for context, Fedora CoreOS is primarily an image-based distribution. We recommend that any software not provided with the OS should be run in containers. #401 discusses providing some mechanism for allowing users to install certain curated packages, but in general, we reserve the right to make changes which might break layered packages.

[kai-uwe-rommel]

Great! Thanks. I'm not that long into FCOS so that I was not aware of that I can "remove" a base package this way.
Yes, this helps as a workaround!
Many Thanks, you saved my day. :-)

[kai-uwe-rommel]

@bgilbert You are right. But I think that NFS server should be possible to run natively, not as a container, for performance reasons?

[cgwalters]

@bgilbert You are right. But I think that NFS server should be possible to run natively, not as a container, for performance reasons?

Linux containers are just Linux processes. If you use a container with the host network namespace, there is (basically) zero overhead to networking for example.

[kai-uwe-rommel]

One of these days I will try to build a suitable slim container.

BTW, my use case is to provide a lightweight NFS server for OCP/OKD clusters.
And to use FCOS for this purpose to avoid using a separate (heavy weight) standard OS.
And be able to deploy it quickly like the base OS VMs for vSphere UPI OCP/OKD clusters.

(And another such VM with a HAproxy load balancer.)

[jlebon]

See #121 (comment) for an example config to create a containerized NFS server.

[kai-uwe-rommel]

Creating just the systemd unit is trival. But I would not rely on someone else's NFS server container from Docker hub. So the actual work needed (for me) is to create a container that is generic and simple enough. Not rocket science either, agreed. But it is work that could be saved. Just because we can run containers does not mean that it makes sense to run everything in containers. "If all you have is a hammer, everything looks like a nail" ... not.

[LorbusChris]

related: https://src.fedoraproject.org/rpms/nfs-utils/pull-request/9 -- I think @stevedd's plan is to eventually split nfs into a client and a server package.

[jlebon]

The base packaging issue here reported in the original post is detailed in https://src.fedoraproject.org/rpms/nfs-utils/pull-request/9#comment-59201.

Nodes shouldn't be stuck on this forever. Practically, this is a perennial issue that'll hit users layering nfs-utils anytime we have skew between it in the stable repos and nfs-utils-coreos in the target version, i.e. the time window between when a new nfs-utils hits stable and when we do a release containing it.

[LorbusChris]

https://src.fedoraproject.org/rpms/nfs-utils/pull-request/14

[aliencrash]

This still happen if you install cockpit-fileshare too.

[m666m]

Can't auto-update FCOS on my raspberry-pi 4B

  $ rpm-ostree status -b                                                                                                                                    
  State: idle                                                                                                                                                  
  AutomaticUpdatesDriver: Zincati                                                                                                                              
    DriverState: active; trying to stage 40.20240808.3.0 (failed attempts: 4)                                                                                  
  BootedDeployment:                                                                                                                                            
  ● fedora:fedora/aarch64/coreos/stable                                                                                                                        
                    Version: 40.20240709.3.1 (2024-07-29T22:29:40Z)                                                                                            
                 BaseCommit: c8f502b11881597a1386a090a80f6eb680871a9afc4c625796f2ae0f6a1bf7c4                                                                  
               GPGSignature: Valid signature by 115DF9AEF857853EE8445D0A0727707EA15B79CC                                                                       
            LayeredPackages: ack bc btop cmatrix cockpit cockpit-machines cockpit-networkmanager cockpit-packagekit cockpit-podman cockpit-storaged figlet     
                             google-authenticator nmon policycoreutils-python-utils qrencode-libs tmux toilet tree wol                                         
                         
   $ systemctl status zincati.service                                                                                                                        
  ● zincati.service - Zincati Update Agent                                                                                                                     
       Loaded: loaded (/etc/systemd/system/zincati.service; enabled; preset: enabled)                                                                          
      Drop-In: /usr/lib/systemd/system/service.d                                                                                                               
               └─10-timeout-abort.conf                                                                                                                         
       Active: active (running) since Sat 2024-06-29 08:00:41 +08; 2 months 2 days ago                                                                         
         Docs: https://github.com/coreos/zincati                                                                                                               
     Main PID: 1538 (zincati)                                                                                                                                  
       Status: "trying to stage 40.20240808.3.0 (failed attempts: 3)"                                                                                          
        Tasks: 8 (limit: 4097)                                                                                                                                 
       Memory: 13.7M (peak: 18.6M)                                                                                                                             
          CPU: 4min 653ms                                                                                                                                      
       CGroup: /system.slice/zincati.service
               └─1538 /usr/libexec/zincati agent -v
  
  Aug 31 14:58:26 banq.local zincati[1538]:     error: Checkout nfs-utils-1:2.7.1-0.fc40.aarch64: Hardlinking f9/cf981a02b734e235da6d3338a6a9990a7f9735976c64a>
  Aug 31 14:58:26 banq.local zincati[1538]:     
  Aug 31 15:03:53 banq.local zincati[1538]: [INFO  zincati::update_agent::actor] target release '40.20240808.3.0' selected, proceeding to stage it
  Aug 31 15:04:13 banq.local zincati[1538]: [ERROR zincati::update_agent::actor] failed to stage deployment: rpm-ostree deploy failed:
  Aug 31 15:04:13 banq.local zincati[1538]:     error: Checkout nfs-utils-1:2.7.1-0.fc40.aarch64: Hardlinking f9/cf981a02b734e235da6d3338a6a9990a7f9735976c64a>
  Aug 31 15:04:13 banq.local zincati[1538]:     
  Aug 31 15:09:31 banq.local zincati[1538]: [INFO  zincati::update_agent::actor] target release '40.20240808.3.0' selected, proceeding to stage it
  Aug 31 15:09:55 banq.local zincati[1538]: [ERROR zincati::update_agent::actor] failed to stage deployment: rpm-ostree deploy failed:
  Aug 31 15:09:55 banq.local zincati[1538]:     error: Checkout nfs-utils-1:2.7.1-0.fc40.aarch64: Hardlinking f9/cf981a02b734e235da6d3338a6a9990a7f9735976c64a>
  Aug 31 15:09:55 banq.local zincati[1538]: