Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Connectivity tests pods may be unschedulable in a two nodes cluster #2204

Open
giorio94 opened this issue Dec 21, 2023 · 2 comments
Open

Connectivity tests pods may be unschedulable in a two nodes cluster #2204

giorio94 opened this issue Dec 21, 2023 · 2 comments
Labels
kind/bug Something isn't working

Comments

@giorio94
Copy link
Member

Hit on a CI run: https://github.com/cilium/cilium-cli/actions/runs/7259322671/job/19826588396?pr=2194

The test cluster has four nodes, but Cilium is deployed on only two, while the other two are reserved for extra tests. Which is equivalent to a two nodes cluster.

Pods:

NAMESPACE            NAME                                                  READY   STATUS    RESTARTS   AGE     IP             NODE                          
test-namespace       client-5f8f776644-pxh7p                               0/1     Pending   0          5m1s    <none>         <none>                        
test-namespace       client2-868c49bf66-48rfv                              0/1     Pending   0          5m1s    <none>         <none>                        
test-namespace       client3-674cf46fd5-cf9d2                              1/1     Running   0          5m2s    10.244.0.89    chart-testing-control-plane   
test-namespace       echo-external-node-6c447f645f-xwx4l                   1/1     Running   0          5m1s    172.18.0.4     chart-testing-worker3         
test-namespace       echo-other-node-757cbff7b6-pq9bv                      2/2     Running   0          5m2s    10.244.3.46    chart-testing-worker          
test-namespace       echo-same-node-6cc6494564-sxngz                       0/2     Pending   0          5m1s    <none>         <none>                        

Specifically:

  • The echo-other-node pod is scheduled on chart-testing-worker, with a required anti-affinity targeting the client pod;
  • The client3 pod is scheduled on chart-testing-control-plane, with a required anti-affinity targeting the client pod;

Which makes it impossible to schedule the client pod, as both ready nodes are forbidden by the anti-affinity rules:

0/4 nodes are available: 2 node(s) didn''t match pod affinity/anti-affinity,
2 node(s) didn''t satisfy existing pods anti-affinity rules, 2 node(s) had
taint {node.kubernetes.io/not-ready: }, that the pod didn''t tolerate.

And in turn the client2 and echo-same-node pods are also unschedulable, because of the required affinity targeting the client pod.

The `client3 pod got recently introduced in #2183.

Copy link

This issue has been automatically marked as stale because it has not
had recent activity. It will be closed if no further activity occurs.

@github-actions github-actions bot added the stale label Sep 28, 2024
Copy link

This issue has not seen any activity since it was marked stale.
Closing.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Oct 13, 2024
@giorio94 giorio94 reopened this Oct 14, 2024
@giorio94 giorio94 removed the stale label Oct 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant