Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[EKS] [BAD-DECISION]: EKS Pod Identity agent daemonset mapped to node-port 80 #2356

Closed
ChrisMcKee opened this issue May 17, 2024 · 6 comments
Labels
EKS Amazon Elastic Kubernetes Service Proposed Community submitted issue

Comments

@ChrisMcKee
Copy link

ChrisMcKee commented May 17, 2024

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request
What do you want us to build?

Which service(s) is this request for?
EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?
What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem.

We tried to install the eks-pod-identity-agent addon so that we could set the auth config to allow both options.
The addon installs as a daemonset with HostNetwork set to true, pod permissions to map to the node, and a default port set to 80.
The instant that the service started to install, all of our Haproxy ingress pods were evicted so that the identity service could map to port 80.

I'd love to know the rationale that went into choosing to map the node-port to what is literally the main http port; and then not to document how to change it to avoid collisions. Through all the documentation that mentions it the only warning is here https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html#pod-id-considerations and it's a note rather than informative. The majority of links go straight to https://docs.aws.amazon.com/eks/latest/userguide/pod-id-agent-setup.html which doesnt mention it at all.

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: eks-pod-identity-agent
  namespace: kube-system
spec:
  template:    
    spec:
      containers:
        - name: eks-pod-identity-agent
          image: >-
            602401143452.dkr.ecr.eu-central-1.amazonaws.com/eks/eks-pod-identity-agent:0.1.6
          command:
            - /go-runner
            - /eks-pod-identity-agent
            - server
          args:
            - '--port'
            - '80'
            - '--cluster-name'
            - prod-eks-cluster
            - '--probe-port'
            - '2703'
          ports:
            - name: proxy
              containerPort: 80
              protocol: TCP
            - name: probes-port
              containerPort: 2703
              protocol: TCP
          securityContext:
            capabilities:
              add:
                - CAP_NET_BIND_SERVICE
      hostNetwork: true

Are you currently working around this issue?
How are you currently solving this problem?

Uninstalled the Addon

@ChrisMcKee ChrisMcKee added the Proposed Community submitted issue label May 17, 2024
@ChrisMcKee ChrisMcKee changed the title [EKS] [BUG]: EKS Pod Identity agent daemonset mapped to node-port 80 [EKS] [BAD-DECISION]: EKS Pod Identity agent daemonset mapped to node-port 80 May 17, 2024
@mikestef9 mikestef9 added the EKS Amazon Elastic Kubernetes Service label May 18, 2024
@vpineda1996
Copy link

vpineda1996 commented May 21, 2024

The EKS Pod Identity Agent only binds to the address on link-local interface created by the initial setup. Specifically, the call to bind binds to port 80 on the following IP addresses:

  • fd00:ec2::23
  • 169.254.170.23

You can get around this limitation by specifying the address of the interface you are trying to bind to port 80. eg if you are doing a bind 0.0.0.0:80, it will fail but if you know the IP of your instance (eg 10.0.163.96) then you can do a direct bind 10.0.163.96:80.

To know the primary IP that your instance has you can run ip addr and find the interface that has the primary IP (normally named ens6)

5: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 0a:24:fe:7c:d3:99 brd ff:ff:ff:ff:ff:ff
    altname enp0s6
    altname ens6
    inet 10.0.163.96/19 brd 10.0.191.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::824:feff:fe7c:d399/64 scope link proto kernel_ll
       valid_lft forever preferred_lft forever

Alternatively you can use EC2's DescribeInstances API to find the primary network interface IP.

@ChrisMcKee
Copy link
Author

@vpineda1996 it hijacks port-80 on the node; If I deploy haproxy ingress daemonset on hostnetwork and use a random port e.g. 30680 it will work fine.
If I have haproxy installed already attached to port 80; installing the AWS Identity addon will kick the haproxy pods causing an outage.
It sounds like you're saying it doesn't use the eks-node host-network / 80? but it is.
image

@ryangraham
Copy link

This just took down all my nginx ingress the same way. Had to remove the addon and delete the daemonset to fix.

@dims
Copy link
Member

dims commented Jun 17, 2024

fyi code is here now - https://github.com/aws/eks-pod-identity-agent - can we please move this to an issue there? 🙏🏾

@ChrisMcKee
Copy link
Author

@dims
Copy link
Member

dims commented Jun 18, 2024

thanks @ChrisMcKee

@dims dims closed this as completed Jun 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
EKS Amazon Elastic Kubernetes Service Proposed Community submitted issue
Projects
None yet
Development

No branches or pull requests

5 participants