Deploying Redis Cluster on multiple Nodes while Preventing Followers from Sharing Nodes with Same-Numbered Leaders

[ahmadjz]

Describe your question

I am aiming to configure pod placement for my Redis cluster to ensure high availability. Specifically, I want to prevent Follower pods from being scheduled on the same nodes as their corresponding Leader pods with the same index. This setup is intended to enhance resilience in case a server goes down.

Desired Pod Placement:

• Nodes:

  • Node1
  • Node2
  • Node3

• Pods:

  • Leaders:
    • Leader1
    • Leader2
    • Leader3
  • Followers:
    • Follower1
    • Follower2
    • Follower3

• Desired Assignment:

  • Node1: Leader1 and Follower2 or Follower3
  • Node2: Leader2 and Follower3 or Follower1
  • Node3: Leader3 and Follower1 or Follower2

Current Configuration Attempts:

I attempted to set up podAntiAffinity rules in my redis-cluster-values.yaml as follows:

redisCluster:
  follower:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                statefulset.kubernetes.io/pod-name: redis-cluster-leader-0
            topologyKey: "kubernetes.io/hostname"

However, this configuration prevents any follower from being scheduled on the same node as redis-cluster-leader-0, which is not the desired behavior.

I also tried another variation:

redisCluster:
  follower:
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: "statefulset.kubernetes.io/pod-name"
                  operator: In
                  values:
                    - redis-cluster-leader-0
            topologyKey: "kubernetes.io/hostname"

This configuration exhibits the same issue, restricting all followers from being placed on the node hosting redis-cluster-leader-0.

I also tried this solution by @sondrelg

    # Prevent leader from being co-hosted on the same node
    # as another leader or its follower
    leader:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: 'kubernetes.io/arch'
                    operator: In
                    values:
                      - 'amd64'

        podAntiAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            # Avoid leaders being scheduled with other leaders
            - labelSelector:
                matchExpressions:
                  - key: 'role'
                    operator: In
                    values:
                      - leader
              topologyKey: 'kubernetes.io/hostname'
            # Prevent leader-1 from being co-hosted with follower-1
            - labelSelector:
                matchExpressions:
                  - key: 'statefulset.kubernetes.io/pod-name'
                    operator: In
                    values:
                      - <name-of-our-deployment>-follower-0
                      - <name-of-our-deployment>-leader-0
              # Applied to the node-level
              topologyKey: 'kubernetes.io/hostname'
            # Prevent leader-2 from being co-hosted with follower-2
            - labelSelector:
                matchExpressions:
                  - key: 'statefulset.kubernetes.io/pod-name'
                    operator: In
                    values:
                      - <name-of-our-deployment>-follower-1
                      - <name-of-our-deployment>-leader-1
              topologyKey: 'kubernetes.io/hostname'
            # Prevent leader-3 from being co-hosted with follower-3
            - labelSelector:
                matchExpressions:
                  - key: 'statefulset.kubernetes.io/pod-name'
                    operator: In
                    values:
                      - <name-of-our-deployment>-follower-2
                      - <name-of-our-deployment>-leader-2
              topologyKey: 'kubernetes.io/hostname'

    follower:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
              - matchExpressions:
                  - key: 'kubernetes.io/arch'
                    operator: In
                    values:
                      - 'amd64'

but the followers stopped working after that

~ ❯ kubectl get pods -n ot-operators -o wide                                                                                            ⎈ kind-redis-test 11:18:47 AM
NAME                              READY   STATUS    RESTARTS   AGE     IP            NODE                        NOMINATED NODE   READINESS GATES
redis-cluster-follower-0          0/1     Pending   0          3m13s   <none>        <none>                      <none>           <none>
redis-cluster-leader-0            1/1     Running   0          3m35s   10.244.1.9    redis-test-control-plane2   <none>           <none>
redis-cluster-leader-1            1/1     Running   0          3m25s   10.244.0.12   redis-test-control-plane    <none>           <none>
redis-cluster-leader-2            1/1     Running   0          3m19s   10.244.2.10   redis-test-control-plane3   <none>           <none>
redis-operator-64785dcd78-hnkwf   1/1     Running   0          8m36s   10.244.2.2    redis-test-control-plane3   <none>           <none>
~ ❯ kubectl describe pod redis-cluster-follower-0 -n ot-operators                                                                       ⎈ kind-redis-test 11:21:12 AM
Name:             redis-cluster-follower-0
Namespace:        ot-operators
Priority:         0
Service Account:  default
Node:             <none>
Labels:           app=redis-cluster-follower
                  app.kubernetes.io/component=middleware
                  app.kubernetes.io/instance=redis-cluster
                  app.kubernetes.io/managed-by=Helm
                  app.kubernetes.io/name=redis-cluster
                  app.kubernetes.io/version=0.16.1
                  controller-revision-hash=redis-cluster-follower-799f97766d
                  helm.sh/chart=redis-cluster-0.16.1
                  redis_setup_type=cluster
                  role=follower
                  statefulset.kubernetes.io/pod-name=redis-cluster-follower-0
Annotations:      meta.helm.sh/release-name: redis-cluster
                  meta.helm.sh/release-namespace: ot-operators
                  redis.opstreelabs.in: true
                  redis.opstreelabs.instance: redis-cluster
Status:           Pending
IP:               
IPs:              <none>
Controlled By:    StatefulSet/redis-cluster-follower
Containers:
  redis-cluster-follower:
    Image:      quay.io/opstree/redis:v7.0.15
    Port:       <none>
    Host Port:  <none>
    Liveness:   exec [sh -c redis-cli -h $(hostname) -p ${REDIS_PORT} ping] delay=0s timeout=1s period=10s #success=1 #failure=3
    Readiness:  exec [sh -c redis-cli -h $(hostname) -p ${REDIS_PORT} ping] delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:
      PERSISTENCE_ENABLED:  true
      REDIS_ADDR:           redis://localhost:6379
      REDIS_MAJOR_VERSION:  v7
      REDIS_PORT:           6379
      SERVER_MODE:          cluster
      SETUP_MODE:           cluster
    Mounts:
      /data from redis-cluster-follower (rw)
      /node-conf from node-conf (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6rshm (ro)
Conditions:
  Type           Status
  PodScheduled   False 
Volumes:
  node-conf:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  node-conf-redis-cluster-follower-0
    ReadOnly:   false
  redis-cluster-follower:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  redis-cluster-follower-redis-cluster-follower-0
    ReadOnly:   false
  kube-api-access-6rshm:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  3m42s  default-scheduler  0/3 nodes are available: 3 node(s) didn't satisfy existing pods anti-affinity rules. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod..

Question:

Is there a way to configure podAntiAffinity rules to match follower pods with their corresponding leader pods based on their index? For example, to prevent deploying FollowerN on a node that already has LeaderN, while allowing FollowerN to coexist with LeaderM where M ≠ N.

Additional context

• Redis Operator Version: 0.18.3
• Redis Cluster Version: 0.16.1

Any guidance or examples on how to configure the podAntiAffinity rules to achieve this selective pod placement would be greatly appreciated

This configuration exhibits the same issue, restricting all followers from being placed on the node hosting redis-cluster-leader-0.

Question:

Is there a way to configure podAntiAffinity rules to match follower pods with their corresponding leader pods based on their index? For example, to prevent deploying FollowerN on a node that already has LeaderN, while allowing FollowerN to coexist with LeaderM where M ≠ N.

Additional context

• Redis Operator Version: 0.18.3
• Redis Cluster Version: 0.16.1

I checked this issue #500 and tried to implement the same solution but it didn't work for me, even though we have a similar problem

Any guidance or examples on how to configure the podAntiAffinity rules to achieve this selective pod placement would be greatly appreciated.