Probable issue parsing identity docs in aws-ec2 backend

[lstoll]

I'm not directly a vault user, but we've been using identity documents as well and noticed failures validating them for a small number of instances (<0.5%). It seems like there might be a bug in the PKCS7 lib in use, more info on that here fullsailor/pkcs7#10 . I spoke to a vault user I know, and they seem to have seen some symptoms as well.

I figured I'd give you a heads up on this, in case it comes up. Also worth noting, instead of using the PKCS7 document it's also possible to use the separate document + signature combination. This can be validated using stdlib with different certs from AWS. We implemented that and it seems to be sorting us out. The certs aren't public (yet), but I've asked if they can be shared and will update when I hear back.

[jefferai]

Thanks for the heads-up! Let us know about the other combination when possible.

[vishalnayak]

@lstoll Any chance that you could privately send me a sample of a bad instance identity document? Please be assured that it will be kept confidential and that this request is purely for testing purposes. Its very difficult to debug without being able to reproduce the issue.

[jefferai]

@lstoll You could send to vault (a) hashicorp [dot] com

[lstoll]

Hi! Sorry I can't share the documents, I didn't save the failing cases locally anywhere and we're not using the PKCS7 version anymore. Being that it's an issue in ASN.1 parsing rather than anything cryptographic a synthetic case should be OK for testing - take a valid doc, switch some chars in the signature to 0000, and it should fail at the parsing phase. Otherwise, if you run through enough instances you should eventually get one that fails.

Alternatively, as mentioned I made another setup, that's here: https://github.com/lstoll/grpce/tree/master/identitydoc . It's missing certs for ap-south-1" and "ap-northeast-2", which is why I hadn't mentioned it - I'll follow up on where they're at now.

[jefferai]

Closing for now as the original docs aren't available and #1961 provides a workaround.

[bmonkman]

@jefferai @vishalnayak I had this happen the other day too and did save the identity doc and signature. I'll send it over.

[jefferai]

@bmonkman You saved the identity doc and signature, or the PKCS#7 doc? The latter is the issue but it won't be seen by the identity doc/signature combo.

If you have the PKCS#7 doc that caused problems, that would be great.

[gobins]

@jefferai @vishalnayak Faced this issue with multiple instances with 0.6.4 recently. Were you able to find out more from the doc provided by @bmonkman

[jefferai]

@gobins We haven't yet, but you can simply use the identity doc method instead.

[gobins]

Thanks @jefferai. Got the identity doc method as the fallback option if pkcs7 does not work.

I kept seeing this error if I converted the signature to base64:

• instance identity verification using SHA256 RSA signature is unsuccessful

This worked:
SIGNATURE=$(curl -s --connect-timeout 30 --retry 30 --retry-delay 30 http://169.254.169.254/latest/dynamic/instance-identity/signature | tr -d '\n')

[louism517]

@jefferai @vishalnayak I am seeing this issue with a few instances too.

I have fallen back to using the identity document as per the workaround.

However I have kept the pkcs7 document of a failing instance and am happy to share if you still need it.