Backup to S3 using IRSA does not work

[jeffgus]

Overview

I'm unable to get the backup to S3 to work with a service account and IAM role (IRSA).

Environment

• Platform: OpenShift (ROSA)
• Platform Version: 4.14.x
• PGO Image Tag: registry.connect.redhat.com/crunchydata/crunchy-postgres
• Postgres Version: 15
• Storage: gp3

Steps to Reproduce

Create an IAM role in AWS with a Trust Relationship.
Make sure that the ServiceAccounts are annotated.
set: repo2-s3-key-type = web-id
set bucket name, region, and endpoint.

I set s3.conf to be:

[global]
repo2-retention-full = 14
repo2-retention-full-type = time
repo2-s3-key-type = web-id

I'm not sure if these settings belong in the s3.conf file or the main config file. I've tried both.

EXPECTED

The pgbackrest should be able to find the token to commicate with the s3 bucket.

ACTUAL

I get one of two errors. I get an error saying that AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE env vars are missing. If I override the metadata for all serviceaccounts and edit the StatefulSet for the repo-host settings the serviceAccountName, that error goes away. It is replaced with:

command terminated with exit code 29: ERROR: [029]: unable to find child 'AssumeRoleWithWebIdentityResult':0 in node 'ErrorResponse'

Logs

command terminated with exit code 31: ERROR: [031]: option 'repo2-s3-key-type' is 'web-id' but 'AWS_ROLE_ARN' and 'AWS_WEB_IDENTITY_TOKEN_FILE' are not set

or

command terminated with exit code 29: ERROR: [029]: unable to find child 'AssumeRoleWithWebIdentityResult':0 in node 'ErrorResponse'

Additional Information

This is similar to #3135 and #3472, but these issues are old and things have changed.

I tried to tweak the role trust relationship rule and it doesn't seem to make a difference. I can run a container with awscli with the same serviceAccount and it works fine.

I can also try to run pgbackrest on the repo-node manually. It fails to properly backup (which is expected), bit it DOES communicate with S3 and creates the backup.info file.

What is the correct configuration for this to work?