redis pods not coming up

[doronl]

The generated redis pod node looks like this, it is not being schduled, see below that failure-domain.beta.kubernetes.io/node is used for toplogy... we don't have this label on our EKS nodes, I think this is the issue (at least one of them) can we avoid using it?

Kind: Pod
apiVersion: v1
metadata:
  name: redis-node-2
  namespace: redis2
  uid: 255923cc-5efe-4fa6-84fd-b18c1c001962
  resourceVersion: '3459362'
  creationTimestamp: '2022-07-07T12:43:00Z'
  labels:
    app: redis-cluster-pod
    leader-number: '2'
    node-number: '2'
    redis-cluster: dev-rdc
    redis-node-role: leader
  annotations:
    kubernetes.io/psp: eks.privileged
    meta.helm.sh/release-name: redis-poc
    meta.helm.sh/release-namespace: redis2
  ownerReferences:
    - apiVersion: db.payu.com/v1
      kind: RedisCluster
      name: dev-rdc
      uid: 8733f72e-7786-47bb-bace-ad8313f09018
      controller: true
      blockOwnerDeletion: true
  managedFields:
    - manager: kube-scheduler
      operation: Update
      apiVersion: v1
      time: '2022-07-07T12:43:00Z'
      fieldsType: FieldsV1
      fieldsV1:
spec:
  volumes:
    - name: redis-configuration
      configMap:
        name: dev-rdc-redisconfig
        defaultMode: 420
    - name: redis-users
      configMap:
        name: dev-rdc-users-acl
        defaultMode: 420
    - name: host-sys
      hostPath:
        path: /sys
        type: ''
    - name: kube-api-access-wjdgb
      projected:
        sources:
          - serviceAccountToken:
              expirationSeconds: 3607
              path: token
          - configMap:
              name: kube-root-ca.crt
              items:
                - key: ca.crt
                  path: ca.crt
          - downwardAPI:
              items:
                - path: namespace
                  fieldRef:
                    apiVersion: v1
                    fieldPath: metadata.namespace
        defaultMode: 420
  containers:
    - name: redis-container
      image: 'quay.io/opstree/redis:v6.2.5'
      ports:
        - containerPort: 6379
          protocol: TCP
      env:
        - name: PORT
          value: '6379'
        - name: CONF_PATH
          value: /usr/local/etc/redis/redis.conf
      resources:
        limits:
          cpu: '1'
          memory: 2500Mi
        requests:
          cpu: '1'
          memory: 50Mi
      volumeMounts:
        - name: redis-configuration
          mountPath: /usr/local/etc/redis
        - name: redis-users
          mountPath: /usr/local/etc/redis/acl
        - name: kube-api-access-wjdgb
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
    - name: redis-metric-exporter
      image: 'quay.io/opstree/redis-exporter:1.0'
      ports:
        - containerPort: 9121
          protocol: TCP
      resources: {}
      volumeMounts:
        - name: kube-api-access-wjdgb
          readOnly: true
          mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      terminationMessagePath: /dev/termination-log
      terminationMessagePolicy: File
      imagePullPolicy: IfNotPresent
  restartPolicy: Always
  terminationGracePeriodSeconds: 30
  dnsPolicy: ClusterFirst
  nodeSelector:
    role: redis-memdb-agents
  serviceAccountName: default
  serviceAccount: default
  securityContext: {}
  imagePullSecrets:
    - name: regcred
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app
                operator: In
                values:
                  - redis-cluster-pod
          topologyKey: failure-domain.beta.kubernetes.io/node
      preferredDuringSchedulingIgnoredDuringExecution:
        - weight: 100
          podAffinityTerm:
            labelSelector:
              matchExpressions:
                - key: redis-node-role
                  operator: In
                  values:
                    - leader
            topologyKey: failure-domain.beta.kubernetes.io/zone
  schedulerName: default-scheduler
  tolerations:
    - key: node.kubernetes.io/not-ready
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
    - key: node.kubernetes.io/unreachable
      operator: Exists
      effect: NoExecute
      tolerationSeconds: 300
  priority: 0
  enableServiceLinks: true
  preemptionPolicy: PreemptLowerPriority
status:
  phase: Pending
  conditions:
    - type: PodScheduled
      status: 'False'
      lastProbeTime: null
      lastTransitionTime: '2022-07-07T12:43:00Z'
      reason: Unschedulable
  qosClass: Burstable

[voltbit]

Unfortunately the topology key is not configurable as it is right now and the operator uses the legacy key. The whole affinity config is generated by the operator instead of being provided as yaml manifest because they depend on labels generated by operator - the leader/follower labels.

As a workaround you can disable the default affinity and inject your own config using mutating webhooks. Or use the mutating hook to change only the topology key.

https://github.com/PayU/redis-operator/blob/main/api/v1/rediscluster_types.go#L39

However, the problem of pods not being scheduled is likely not caused by the missing label, the anti-affinity makes sure you dont schedule two pods on the same topology area - node/zone/region, since you dont have the label the pods will always be scheduled.

Release 2.0 will contain the new topology keys. And eventually the whole affinity spec will be configurable.

[doronl]

Thanks

1. What is the ETA of 2.0?
2. In 2.0, are you going to separate the spec of the Leader and Follower? today it seems they need to run on the same node group and only the redis-node-role label distinguishes them.

[doronl]

solved by code change + chart change:

1.Replace the topologyKey: failure-domain.beta.kubernetes.io/node with kubernetes.io/hostname
2. Add tolerations to redisPodSpec