IPA CA replica installation fails in fips mode on fedora 42

[flo-renaud]

Installation of a CA clone fails in FIPS mode on fedora 42.

Reproducer:

1. Prepare a rawhide server in FIPS mode + se linux permissive mode
2. Prepare a rawhide replica in FIPS mode + se linux permissive mode
3. Enable the copr repo @freeipa/freeipa-master-nightly on both machines, install freeipa-server-dns packages on both machines
4. on the server: ipa-server-install --domain ipa.test --realm IPA.TEST --setup-dns --auto-forwarders -a Secret123 -p Secret123 -U
5. on the replica: ipa-replica-install --domain ipa.test --realm IPA.TEST --server server.ipa.test --setup-dns --auto-forwarders --principal admin --password Secret123 -U
6. on the replica: perform CA clone installation with ipa-ca-install

The Ca clone installation fails:

# ipa-ca-install
Directory Manager (existing master) password: 

Running ipa-certupdate...done
Run connection check to master
Connection check OK
Configuring certificate server (pki-tomcatd). Estimated time: 3 minutes
  [1/31]: creating certificate server db
  [2/31]: ignore time skew for initial replication
  [3/31]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 9 seconds elapsed
Update succeeded

  [4/31]: revert time skew after initial replication
  [5/31]: creating ACIs for admin
  [6/31]: creating installation admin user
  [7/31]: configuring certificate server instance
ipaserver.install.dogtaginstance: CRITICAL Failed to configure CA instance
ipaserver.install.dogtaginstance: CRITICAL See the installation logs and the following files/directories for more information:
ipaserver.install.dogtaginstance: CRITICAL   /var/log/pki/pki-tomcat
  [error] RuntimeError: CA configuration failed.

Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

CA configuration failed.

The log file /var/log/ipareplica-ca-install.log shows that the issue happens in pkispawn in a call to openssl pkcs12:

ERROR: CalledProcessError: Command '['openssl', 'pkcs12', '-in', '/tmp/ca.p12', '-out', '/etc/pki/pki-tomcat/alias/ca.crt', '-nodes', '-nokeys', '-passin', 'pass:XXXXXXXX']' returned non-zero exit status 1.

In IPA source code we had to modify our calls to openssl pkcs12 in FIPS mode:

• add the -nomac arg to openssl pkcs12 -export to export to a p12
• add the -nomacver argument to openssl pkcs12 to export from p12
  PKI may need to perform the same changes.

[flo-renaud]

ipareplica-ca-install.log

[flo-renaud]

pki-ca-spawn.20240911040934.log

[flo-renaud]

Installed packages:
pki-resteasy-core-3.0.26-31.fc41.noarch
dogtag-pki-ca-11.5.0-3.fc41.2.noarch