Verify with X.509 Certificate

[tri-adam]

Description of the Pull Request (PR):

Add test certificate chain. Add X.509 flags and environment variables to singularity verify command.

This fixes or addresses the following GitHub issues:

• Closes Support sign/verify with X.509 certificates #1095

[fnikolai]

@tri-adam

The verification seems to ignore the difference between root and intermediate certificates.

For instance, the following snippet succeeds while it shouldn't (the certificates are swapped).
Is that the desired behavior?

singularity verify  \
--certificate ./test/certs/leaf.pem  \
--certificate-intermediates ./test/certs/root.pem \
--certificate-roots ./test/certs/intermediate.pem \
../images/busybox_latest.sif

[tri-adam]

The verification seems to ignore the difference between root and intermediate certificates.

For instance, the following snippet succeeds while it shouldn't (the certificates are swapped). Is that the desired behavior?

singularity verify  \
--certificate ./test/certs/leaf.pem  \
--certificate-intermediates ./test/certs/root.pem \
--certificate-roots ./test/certs/intermediate.pem \
../images/busybox_latest.sif

I'd expect verify to succeed with those arguments, since it is instructing the command to trust any certificate in intermediate.pem as a root. So the chain of trust would just be leaf->intermediate. Since no intermediate certificate is required, you would get the same behaviour if you drop the --certificate-intermediates flag in that example:

singularity verify  \
--certificate ./test/certs/leaf.pem  \
--certificate-roots ./test/certs/intermediate.pem \
../images/busybox_latest.sif

To demonstrate that they're treated differently, note that this is not equivalent to:

singularity verify  \
--certificate ./test/certs/leaf.pem  \
--certificate-intermediates ./test/certs/intermediate.pem \
../images/busybox_latest.sif

Here, the same chain leaf->intermediate can be built, but intermediate is not a root, and verification fails.

If you take it the other way and specify --certificate-roots to be root.pem, a chain cannot be built at all, and fails for that reason:

singularity verify  \
--certificate ./test/certs/leaf.pem  \
--certificate-roots ./test/certs/root.pem \
../images/busybox_latest.sif

All that being said... this code is quite green and there's always a possibility I've messed something up! Let me know if the above makes sense... or if you feel there's an issue with it.

[dtrudg]

This LGTM subject to any further input from @fnikolai

One request - as we now have useful functionality exposed here, please could you add an entry / entries to the CHANGELOG.md covering the sign/verify additions?

[fnikolai]

@tri-adam I see your point. However, I 'm still not sure if this is the desired behavior.

In the sense that if I can assign any of the intermediates to be the root, then I can effectively skip the rest of the certificate chain. As a result, if any certificate higher in the chain is revoked, the revocation check can be easily bypassed.

Going a bit further, I think the issue originates from gen_certs, where all the certificates are signed from the same CA key. (Shouldn't we use the intermediate's key for signing the leaf certificate ?)

	// load a private key for the Root CA
	pem, err := os.ReadFile(filepath.Join("..", "keys", "private.pem"))
	if err != nil {
		return err
	}

	pri, err := cryptoutils.UnmarshalPEMToPrivateKey(pem, cryptoutils.SkipPassword)
	if err != nil {
		return err
	}

	pub := pri.(crypto.Signer).Public()

	// use the private key to create a self-signed Root CA certificate.
	root, err := createRoot(start, pri)
	if err != nil {
		return err
	}

	// create a new CSR for the OCSP and sign it with the Root CA key.
	intermediate, err := createIntermediate(start, pub, root, pri)
	if err != nil {
		return err
	}

	// create a new CSR for the client and sign it with the Root CA key.
	leaf, err := createLeaf(start, pub, intermediate, pri)
	if err != nil {
		return err
	}

On another note related to #1152, if we use the chain produced by Certificate.Verify (here) as hinted on https://github.com/sylabs/singularity/pull/1160, then we cannot perform end-to-end testing of OCSP by using the google' certificates. (The signature verification fails before the OCSP validation is invoked).

For that, I 'm now working on a tiny OCSP_Responder to complete the gen_certs.go

[fnikolai]

@tri-adam
@dtrudg

fnikolai@458f461

On the good side, we now have:

• client-side logic of OCSP within singularity.
• tests for stressing the client-side OCSP logic using third-party certificates (without signature verification).
• an OCSP server based on openssl for the end-to-end validation of the testing certificates (with signature verification).

On the bad side, the hashing algorithms of the original testing keys (leaf, intermediate, private) appear to be incompatible with OCSP. For OCSP only SHA1, SHA224, SHA256, SHA384, and SHA512 are allowed.

That given, I had to create new keys, which means that now I have to re-sign the testing images. Is there any scripted way to do that?

Once the images are re-signed, I 'll make PR of the commit mentioned above.

[tri-adam]

@tri-adam I see your point. However, I 'm still not sure if this is the desired behavior.

In the sense that if I can assign any of the intermediates to be the root, then I can effectively skip the rest of the certificate chain. As a result, if any certificate higher in the chain is revoked, the revocation check can be easily bypassed.

If the user specifies certificate(s) via --certificate-roots, they are explicitly specifying that as a set of "trust anchors" for chaining. By definition, I don't think there is any certificate "higher in the chain"? This "trust anchor information" is an input to the path validation, and as specified in RFC 5280 §6.1.1 [d]:

The trust anchor information is trusted because it was delivered to the path processing procedure by trustworthy out-of-band procedure.

Thus, as long as it is possible to build a sequence of certificates that satisfy the algorithm in RFC 5820 § 6.1:

  (a)  for all x in {1, ..., n-1}, the subject of certificate x is
       the issuer of certificate x+1;

  (b)  certificate 1 is issued by the trust anchor;

  (c)  certificate n is the certificate to be validated (i.e., the
       target certificate); and

  (d)  for all x in {1, ..., n}, the certificate was valid at the
       time in question.

using a certificate specified in --certificate-roots as "certificate 1", and certificates specified in --certificate-intermediates can be used for {2, ..., n-1}, it should be treated as valid?

Going a bit further, I think the issue originates from gen_certs, where all the certificates are signed from the same CA key. (Shouldn't we use the intermediate's key for signing the leaf certificate ?)

Yeah, I was being a bit lazy there and just using the same key since we only have one in our corpus. I'll add a couple more keys and update the cert generation code to use different keys for each cert it generates.

On another note related to #1152, if we use the chain produced by Certificate.Verify (here) as hinted on https://github.com/sylabs/singularity/pull/1160, then we cannot perform end-to-end testing of OCSP by using the google' certificates. (The signature verification fails before the OCSP validation is invoked).

For that, I 'm now working on a tiny OCSP_Responder to complete the gen_certs.go

Makes sense. I don't see an easy way to test end-to-end without a container that was signed with a private key for which we have a valid cert chain.

[tri-adam]

@tri-adam @dtrudg

fnikolai@458f461

On the good side, we now have:

• client-side logic of OCSP within singularity.
• tests for stressing the client-side OCSP logic using third-party certificates (without signature verification).
• an OCSP server based on openssl for the end-to-end validation of the testing certificates (with signature verification).

On the bad side, the hashing algorithms of the original testing keys (leaf, intermediate, private) appear to be incompatible with OCSP. For OCSP only SHA1, SHA224, SHA256, SHA384, and SHA512 are allowed.

That given, I had to create new keys, which means that now I have to re-sign the testing images. Is there any scripted way to do that?

Once the images are re-signed, I 'll make PR of the commit mentioned above.

Hmm, that's interesting. The current cert is generated using Ed25519 keys, which always uses SHA512 when signing (ref).

What encryption algorithm are you using for the keys you have generated? Can you share the error(s) are you getting when using OCSP with the current cert?

Changelog entries added as requested.

[fnikolai]

What encryption algorithm are you using for the keys you have generated? Can you share the error(s) are you getting when using OCSP with the current cert?

• With Ed25519 keys

>> singularity verify --certificate ./test/certs/leaf.pem --certificate-intermediates ./test/certs/intermediate.pem --certificate-roots ./test/certs/root.pem --ocsp-verify ../images/busybox_latest.sif

FATAL:   Failed to verify container: OCSP verification has failed: certificate: 'CN=leaf,O=Sylabs Inc.,C=US': OCSP Query: certificate error: bad signature on embedded certificate: x509: cannot verify signature: algorithm unimplemented

• With RSA keys

To generate keys:

openssl genrsa -out private.pem 3072
openssl rsa -in private.pem -pubout -out public.pem

and then:

>> singularity verify --certificate ./test/certs/leaf.pem --certificate-intermediates ./test/certs/intermediate.pem --certificate-roots ./test/certs/root.pem --ocsp-verify ../images/busybox_latest.sif

INFO:    Validate: cert:leaf  issuer:root
INFO:    skip self-signed certificate (CN=root,O=Sylabs Inc.,C=US)
INFO:    skip self-signed certificate (CN=root,O=Sylabs Inc.,C=US)
INFO:    OCSP validation has passed
Objects verified:
ID  |GROUP   |LINK    |TYPE
------------------------------------------------
1   |1       |NONE    |Def.FILE
2   |1       |NONE    |JSON.Generic
3   |1       |NONE    |JSON.Generic
4   |1       |NONE    |FS
Container verified: ../images/busybox_latest.sif

In both cases, I 've removed and re-created the certs using go run gen_certs.go.

[tri-adam]

What encryption algorithm are you using for the keys you have generated? Can you share the error(s) are you getting when using OCSP with the current cert?

• With Ed25519 keys

>> singularity verify --certificate ./test/certs/leaf.pem --certificate-intermediates ./test/certs/intermediate.pem --certificate-roots ./test/certs/root.pem --ocsp-verify ../images/busybox_latest.sif

FATAL:   Failed to verify container: OCSP verification has failed: certificate: 'CN=leaf,O=Sylabs Inc.,C=US': OCSP Query: certificate error: bad signature on embedded certificate: x509: cannot verify signature: algorithm unimplemented

• With RSA keys

To generate keys:

openssl genrsa -out private.pem 3072
openssl rsa -in private.pem -pubout -out public.pem

and then:

>> singularity verify --certificate ./test/certs/leaf.pem --certificate-intermediates ./test/certs/intermediate.pem --certificate-roots ./test/certs/root.pem --ocsp-verify ../images/busybox_latest.sif

INFO:    Validate: cert:leaf  issuer:root
INFO:    skip self-signed certificate (CN=root,O=Sylabs Inc.,C=US)
INFO:    skip self-signed certificate (CN=root,O=Sylabs Inc.,C=US)
INFO:    OCSP validation has passed
Objects verified:
ID  |GROUP   |LINK    |TYPE
------------------------------------------------
1   |1       |NONE    |Def.FILE
2   |1       |NONE    |JSON.Generic
3   |1       |NONE    |JSON.Generic
4   |1       |NONE    |FS
Container verified: ../images/busybox_latest.sif

In both cases, I 've removed and re-created the certs using go run gen_certs.go.

Weird. Difficult to imagine OCSP doesn't support Ed25519, but we can switch to RSA if it's easier in the short term. Once you've got OCSP working with RSA, I would like to figure out why Ed25519 isn't working...

One more note... the DSSE image in the corpus (test/images/one-group-signed-dsse.sif) already contains an RSA signature (as well as an Ed25519 signature.) The relevant RSA public key will be added when #1183 is merged, at which point I'll re-base this PR and update gen_certs.go to generate both Ed25519 and RSA certs. Make sense?

Thanks again for all the help reviewing this. Very much appreciated!

[tri-adam]

Going a bit further, I think the issue originates from gen_certs, where all the certificates are signed from the same CA key. (Shouldn't we use the intermediate's key for signing the leaf certificate ?)

Yeah, I was being a bit lazy there and just using the same key since we only have one in our corpus. I'll add a couple more keys and update the cert generation code to use different keys for each cert it generates.

I've proposed adding RSA and ECDSA keys in #1183 and rebased this PR on top of that. Different keys are now used for each of the three certs in gen_certs.go. The leaf cert now uses RSA, intermediate ECDSA, and root Ed25519. Hope this helps!

[fnikolai]

I've proposed adding RSA and ECDSA keys in #1183 and rebased this PR on top of that. Different keys are now used for each of the three certs in gen_certs.go. The leaf cert now uses RSA, intermediate ECDSA, and root Ed25519. Hope this helps!

I have the feeling it will fail :(. The reason is that OCSP server responds with the status of the certificate in question, and the certificate of the server itself (which must be also validated).

In this case, the response for (leaf) will be (status, root). And if we try to validate the root, it will fail because the certificate is signed using Ed25519 keys.

I ll check it tomorrow.

[tri-adam]

I've proposed adding RSA and ECDSA keys in #1183 and rebased this PR on top of that. Different keys are now used for each of the three certs in gen_certs.go. The leaf cert now uses RSA, intermediate ECDSA, and root Ed25519. Hope this helps!

I have the feeling it will fail :(. The reason is that OCSP server responds with the status of the certificate in question, and the certificate of the server itself (which must be also validated).

In this case, the response for (leaf) will be (status, root). And if we try to validate the root, it will fail because the certificate is signed using Ed25519 keys.

I ll check it tomorrow.

OK, yeah let me know how it goes. The test cert chain is now, for better or worse, going to exercise the three main types of key material that people will realistically use in the wild. Hopefully we can get OCSP working with all three, but if there's some technical reason why this isn't possible, we'll figure out a path forward.

The verify end-to-end tests are working using an Ed25519 key (ref), FWIW. So I'm hopeful it'll be possible to solve any signature issues that might come up.

[dtrudg]

@tri-adam - are we in a position to get this merged at this point? If anything is needed around test keys later on, with regard to OCSP, we can adapt it in a later PR. I think this PR is internally consistent and LGTM

[tri-adam]

@tri-adam - are we in a position to get this merged at this point? If anything is needed around test keys later on, with regard to OCSP, we can adapt it in a later PR. I think this PR is internally consistent and LGTM

From my perspective, yes.

The main question left is around the cert chain being suitable for OCSP, I believe. From my perspective, it might be better to have those changes in a PR next to the code that requires them, for ease of review. @fnikolai does that sound good to you?

[fnikolai]

Yeap, I agree that it's time to merge this PR, and continue the OCSP discussion on #1152