[QUESTION] Credentials priority

[vdksystem]

I have EKS cluster in prod account (within AWS Organization). It has OIDC enabled and I manage permissions for pods through OIDC.
Configured custom runner on this cluster (with no role attached -> assume that it will get node default).
ECR is in another account.
Configured IAM identity provider for GitHub.
Pipeline should build docker image and push to ECR using kaniko.
On custom-runner it fails with unauthorised, on default ubuntu-latest from GitHub it works fine.

jobs:
  build:
    runs-on: custom-runner
    # These permissions are needed to interact with GitHub's OIDC Token endpoint.
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v3
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: 'arn:aws:iam::ACCOUNT:role/github-action-test-role'
          aws-region: us-east-1
      - uses: aws-actions/amazon-ecr-login@v1
        id: ecr
        with:
          registries: 'ACCOUNT'
      - uses: docker/metadata-action@v4
        id: metadata
        with:
          images: ${{ steps.ecr.outputs.registry }}/${{ github.event.repository.name }}
      - name: debug
        run: |
          pip install awscli
          aws sts get-caller-identity # Here I get expected assumed role
      - uses: int128/kaniko-action@v1
        with:
          push: true
          tags: ${{ steps.metadata.outputs.tags }}
          labels: ${{ steps.metadata.outputs.labels }}
          cache: false

Is it an issue or something is not configured properly?
Thanks

[FlorinTar]

I'm having a similar issue on self managed runners, I'd love to know the answer to this question.

[arjraman]

Could you paste the exact error from your logs here, covering any sensitive information?

[vdksystem]

There is no error for ecr-login step, because kubernetes node has default IAM role attached with read permissions to ECR.
Error is in kaniko step in my case

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "***.dkr.ecr.us-east-1.amazonaws.com/data-transformation:main-f949f95-1677833600": POST https://***.dkr.ecr.us-east-1.amazonaws.com/v2/REPO/blobs/uploads/: unexpected status code 401 Unauthorized: Not Authorized
[40](https://github.com/REPO/actions/runs/4321772096/jobs/7543412816#step:7:42)

[40](https://github.com/REPO/actions/runs/4321772096/jobs/7543412816#step:7:42)
Error: Error: The process '/usr/local/bin/docker' failed with exit code 1

[arjraman]

What OS is your runner?

[vdksystem]

Amazon linux 2