[EKS] [eks-pod-identity] [bug]: Setting the STS Session name in eks-pod-identity-agent

[taer]

I think this is probably more of a bug report, but I cannot find the proper channels.
Version Info:
EKS: 1.29
Eks Pod ID agent: v1.2.0-eksbuild.1

We are using eks-pod-identity. It's been working great till we started using IAM based kafka.

I directly hit the eks-pod-identity pod inside a container agent via

       AWS_CONTAINER_CREDENTIALS_FULL_URI:      http://169.254.170.23/v1/credentials                                                                                                        
       AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE:  /var/run/secrets/pods.eks.amazonaws.com/serviceaccount/eks-pod-identity-token                                                               │

I took the resulting AccessKey, SecertKey, and Token, and used them to do a aws sts get-caller-identity

{
    "UserId": "AROAXYZP66II6MSBPLOUU:eks-k8s-wl-dev-engine-boo-5af5e7ac-5754-49ea-b28f-2c2a2eb95fbb",
    "Account": "BLAH",
    "Arn": "arn:aws:sts::BLAH:assumed-role/dev-use1-default-bookingApiPod-20240529171038526700000001/eks-k8s-wl-dev-engine-boo-5af5e7ac-5754-49ea-b28f-2c2a2eb95fbb"
}

The SessionName is non static, and I can't find any way to force set it. The issue comes from the MSK IAM usage. MSK doesn't allow "reauthentication". We get this error when the session name changes

failed authentication due to: Cannot change principals during re-authentication from IAM.arn:aws:sts::BLAH:assumed-role/prd-use1-default-bookingApiPod-20240529174141238800000002/eks-k8s-wl-prd-engine-boo-064f1ed1-2349-4774-b895-9a69ccc3eeb1: IAM.arn:aws:sts::BLAH:assumed-role/prd-use1-default-bookingApiPod-20240529174141238800000002/eks-k8s-wl-prd-engine-boo-875fdfcd-19cd-4ac3-8544-7077f94a6e39

Most services we've used IAM for to date don't care. The solution normally would be to set AWS_ROLE_SESSION_NAME when calling STS:AssumeRole. But we're not calling that, the eks-pod-identity pod is. The ContainerCredentialProvider in the SDK is just calling the AWS_CONTAINER_CREDENTIALS_FULL_URI URL with the contents of AWS_CONTAINER_AUTHORIZATION_TOKEN_FILE as the Authorization header. There is no option as far as I've found to have that include a constant session name to satisfy MSK.

Thanks!

[prateekgogia]

I directly hit the eks-pod-identity pod inside a container agent via

IIUC, you are manually calling the AWS_CONTAINER_CREDENTIALS_FULL_URI with the token from inside a pod running on your node and use the credentials to make this aws sts get-caller-identity cli call.

Trying to understand if these credentials ever work from inside a Pod when an application is trying to reach Kafka and fail when using these credentials manual with aws cli? or they never work with Kafka?

[taer]

Yeah. basically, I was doing what the SDK does manually just to validate.

The credentials given by the pod-identity work perfectly w/ Kafka until they reach their expiration. When the SDK goes back to the pod-identity-agent near the expiration, the agent refreshes the tokens for it. The issue though is it refreshes it with a new sessionName. And the MSK IAM auth scheme hates that. It considers the session-name change to be a change in principal, and screams
failed authentication due to: Cannot change principals during re-authentication from IAM.arn:aws:sts::BLAH:assumed-role/prd-use1-default-bookingApiPod-20240529174141238800000002/eks-k8s-wl-prd-engine-boo-064f1ed1-2349-4774-b895-9a69ccc3eeb1: IAM.arn:aws:sts::BLAH:assumed-role/prd-use1-default-bookingApiPod-20240529174141238800000002/eks-k8s-wl-prd-engine-boo-875fdfcd-19cd-4ac3-8544-7077f94a6e39

important parts in there is the change here

arn:aws:sts......roleName/eks-k8s-wl-prd-engine-boo-064f1ed1-2349-4774-b895-9a69ccc3eeb1

arn:aws:sts......roleName/eks-k8s-wl-prd-engine-boo-875fdfcd-19cd-4ac3-8544-7077f94a6e39

That sessionName of pod-name-uuid is generated by the eks-pod-identity-agent. Ideally, we could configure the agent to just use the pod-name and not include the uuid, and we'd be golden.

[dims]

fyi code is here now - https://github.com/aws/eks-pod-identity-agent - can we please move this to an issue there? 🙏🏾

[taraspos]

I've opened feature request asking for support of Custom Session Tags and Custom Session Names in the above repo:

• [Feature request] Support for custom Session Tags and Session Name eks-pod-identity-agent#14

[metropolis-ameer]

following updates to this issue