Pods stuck on Init:CrashLoopBackOff

[ShenghaoHuang]

Hello,

Hoping someone can shed some light on this issue I've been running into. I am running Airflow on Kubernetes installed via Helm using KuberentesExecutor. Everything was running fine until I started to try and get gitSync up and running. I saw this problem was posted couple of times before without any solutions so hopefully someone has some insight on why the pods are stuck in Init:CrashLoopBackOff.

Here is my helm values.yaml
values.txt

Here are the description of my pods

Name:             airflow-scheduler-7b8c4bc7cb-8r6f2
Namespace:        airflow
Priority:         0
Service Account:  airflow-scheduler
Node:             airflow-cluster-worker2/172.18.0.2
Start Time:       Thu, 06 Oct 2022 02:05:04 -0400
Labels:           component=scheduler
                  pod-template-hash=7b8c4bc7cb
                  release=airflow
                  tier=airflow
Annotations:      checksum/airflow-config: fef4a53378439735b46962de1409a30fc3006a8be5a36351cea83e2e4db9182f
                  checksum/extra-configmaps: 2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
                  checksum/extra-secrets: bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
                  checksum/metadata-secret: dcbb26b06a9d686bf5fedceff6d4024447053fded58a37271cdfef14f8c8c800
                  checksum/pgbouncer-config-secret: da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
                  checksum/result-backend-secret: 74e3e99feee51248d44224665d60fab543dd6b25ba95f04e6fcb0e5758342056
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:           Pending
IP:               10.244.1.10
IPs:
  IP:           10.244.1.10
Controlled By:  ReplicaSet/airflow-scheduler-7b8c4bc7cb
Init Containers:
  wait-for-airflow-migrations:
    Container ID:  containerd://46f16025227f4996f8d30402ef45058cba62cb7264db116ff126248abebfcbdf
    Image:         airflow-custom:1.0.0
    Image ID:      docker.io/library/import-2022-10-06@sha256:79d0523e4735f6aafee324331388958e64923e11a4d5756937ae1298b55a2178
    Port:          <none>
    Host Port:     <none>
    Args:
      airflow
      db
      check-migrations
      --migration-wait-timeout=60
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 06 Oct 2022 02:05:05 -0400
      Finished:     Thu, 06 Oct 2022 02:05:11 -0400
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      airflow-variables  ConfigMap  Optional: false
    Environment:
      AIRFLOW__CORE__FERNET_KEY:            <set to the key 'fernet-key' in secret 'airflow-fernet-key'>                      Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:      <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:              <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__WEBSERVER__SECRET_KEY:       <set to the key 'webserver-secret-key' in secret 'airflow-webserver-secret-key'>  Optional: false
    Mounts:
      /opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pwx8b (ro)
  git-sync-init:
    Container ID:   containerd://a5cfefaf1b32389f96a503269a64705211dbd705efdb185132f22e90f0e96bda
    Image:          k8s.gcr.io/git-sync/git-sync:v3.4.0
    Image ID:       k8s.gcr.io/git-sync/git-sync@sha256:a470676e946f1060815f89dadad4f2c3e4f9d1ab36a46f4423e00f44170fc80c
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 06 Oct 2022 02:21:04 -0400
      Finished:     Thu, 06 Oct 2022 02:21:04 -0400
    Ready:          False
    Restart Count:  8
    Environment:
      GIT_SSH_KEY_FILE:            /etc/git-secret/ssh
      GIT_SYNC_SSH:                true
      GIT_KNOWN_HOSTS:             false
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REPO:               ssh://[email protected]/ShenghaoHuang/airflow-dags.git
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_ROOT:               /git
      GIT_SYNC_DEST:               repo
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_WAIT:               60
      GIT_SYNC_MAX_SYNC_FAILURES:  0
      GIT_SYNC_ONE_TIME:           true
    Mounts:
      /etc/git-secret/ssh from git-sync-ssh-key (ro,path="gitSshKey")
      /git from dags (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pwx8b (ro)
Containers:
  scheduler:
    Container ID:
    Image:         airflow-custom:1.0.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      bash
      -c
      exec airflow scheduler
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       exec [sh -c CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
airflow jobs check --job-type SchedulerJob --hostname $(hostname)
] delay=10s timeout=20s period=60s #success=1 #failure=5
    Environment Variables from:
      airflow-variables  ConfigMap  Optional: false
    Environment:
      AIRFLOW__CORE__FERNET_KEY:            <set to the key 'fernet-key' in secret 'airflow-fernet-key'>                      Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:      <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:              <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__WEBSERVER__SECRET_KEY:       <set to the key 'webserver-secret-key' in secret 'airflow-webserver-secret-key'>  Optional: false
    Mounts:
      /opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
      /opt/airflow/config/airflow_local_settings.py from config (ro,path="airflow_local_settings.py")
      /opt/airflow/dags from dags (ro)
      /opt/airflow/logs from logs (rw)
      /opt/airflow/pod_templates/pod_template_file.yaml from config (ro,path="pod_template_file.yaml")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pwx8b (ro)
  git-sync:
    Container ID:
    Image:          k8s.gcr.io/git-sync/git-sync:v3.4.0
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      GIT_SSH_KEY_FILE:            /etc/git-secret/ssh
      GIT_SYNC_SSH:                true
      GIT_KNOWN_HOSTS:             false
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REPO:               ssh://[email protected]/ShenghaoHuang/airflow-dags.git
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_ROOT:               /git
      GIT_SYNC_DEST:               repo
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_WAIT:               60
      GIT_SYNC_MAX_SYNC_FAILURES:  0
    Mounts:
      /etc/git-secret/ssh from git-sync-ssh-key (ro,path="gitSshKey")
      /git from dags (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pwx8b (ro)
  scheduler-log-groomer:
    Container ID:
    Image:         airflow-custom:1.0.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      bash
      /clean-logs
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      AIRFLOW__LOG_RETENTION_DAYS:  15
    Mounts:
      /opt/airflow/logs from logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-pwx8b (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      airflow-airflow-config
    Optional:  false
  dags:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  git-sync-ssh-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-ssh-git-secret
    Optional:    false
  logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-pwx8b:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  17m                   default-scheduler  Successfully assigned airflow/airflow-scheduler-7b8c4bc7cb-8r6f2 to airflow-cluster-worker2  
  Normal   Pulled     17m                   kubelet            Container image "airflow-custom:1.0.0" already present on machine
  Normal   Created    17m                   kubelet            Created container wait-for-airflow-migrations
  Normal   Started    17m                   kubelet            Started container wait-for-airflow-migrations
  Normal   Pulling    17m                   kubelet            Pulling image "k8s.gcr.io/git-sync/git-sync:v3.4.0"
  Normal   Pulled     17m                   kubelet            Successfully pulled image "k8s.gcr.io/git-sync/git-sync:v3.4.0" in 6.864346064s
  Normal   Created    15m (x5 over 17m)     kubelet            Created container git-sync-init
  Normal   Started    15m (x5 over 17m)     kubelet            Started container git-sync-init
  Normal   Pulled     15m (x4 over 17m)     kubelet            Container image "k8s.gcr.io/git-sync/git-sync:v3.4.0" already present on machine
  Warning  BackOff    2m10s (x69 over 17m)  kubelet            Back-off restarting failed container

Name:             airflow-triggerer-5b459bcf8f-sm4lx
Namespace:        airflow
Priority:         0
Service Account:  airflow-triggerer
Node:             airflow-cluster-worker3/172.18.0.3
Start Time:       Thu, 06 Oct 2022 02:05:04 -0400
Labels:           component=triggerer
                  pod-template-hash=5b459bcf8f
                  release=airflow
                  tier=airflow
Annotations:      checksum/airflow-config: fef4a53378439735b46962de1409a30fc3006a8be5a36351cea83e2e4db9182f
                  checksum/extra-configmaps: 2e44e493035e2f6a255d08f8104087ff10d30aef6f63176f1b18f75f73295598
                  checksum/extra-secrets: bb91ef06ddc31c0c5a29973832163d8b0b597812a793ef911d33b622bc9d1655
                  checksum/metadata-secret: dcbb26b06a9d686bf5fedceff6d4024447053fded58a37271cdfef14f8c8c800
                  checksum/pgbouncer-config-secret: da52bd1edfe820f0ddfacdebb20a4cc6407d296ee45bcb500a6407e2261a5ba2
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
Status:           Pending
IP:               10.244.2.6
IPs:
  IP:           10.244.2.6
Controlled By:  ReplicaSet/airflow-triggerer-5b459bcf8f
Init Containers:
  wait-for-airflow-migrations:
    Container ID:  containerd://66e26990d4f9ad54cbdc3301e79988f5afd398918014ebe6276f54fa52273e4f
    Image:         airflow-custom:1.0.0
    Image ID:      docker.io/library/import-2022-10-06@sha256:79d0523e4735f6aafee324331388958e64923e11a4d5756937ae1298b55a2178
    Port:          <none>
    Host Port:     <none>
    Args:
      airflow
      db
      check-migrations
      --migration-wait-timeout=60
    State:          Terminated
      Reason:       Completed
      Exit Code:    0
      Started:      Thu, 06 Oct 2022 02:05:05 -0400
      Finished:     Thu, 06 Oct 2022 02:05:12 -0400
    Ready:          True
    Restart Count:  0
    Environment Variables from:
      airflow-variables  ConfigMap  Optional: false
    Environment:
      AIRFLOW__CORE__FERNET_KEY:            <set to the key 'fernet-key' in secret 'airflow-fernet-key'>                      Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:      <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:              <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__WEBSERVER__SECRET_KEY:       <set to the key 'webserver-secret-key' in secret 'airflow-webserver-secret-key'>  Optional: false
    Mounts:
      /opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-67549 (ro)
  git-sync-init:
    Container ID:   containerd://3684745a652efc21827f53cee69f5e680825c11194af87f10290c644a4ebd1d9
    Image:          k8s.gcr.io/git-sync/git-sync:v3.4.0
    Image ID:       k8s.gcr.io/git-sync/git-sync@sha256:a470676e946f1060815f89dadad4f2c3e4f9d1ab36a46f4423e00f44170fc80c
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Thu, 06 Oct 2022 02:21:16 -0400
      Finished:     Thu, 06 Oct 2022 02:21:16 -0400
    Ready:          False
    Restart Count:  8
    Environment:
      GIT_SSH_KEY_FILE:            /etc/git-secret/ssh
      GIT_SYNC_SSH:                true
      GIT_KNOWN_HOSTS:             false
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REPO:               ssh://[email protected]/ShenghaoHuang/airflow-dags.git
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_ROOT:               /git
      GIT_SYNC_DEST:               repo
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_WAIT:               60
      GIT_SYNC_MAX_SYNC_FAILURES:  0
      GIT_SYNC_ONE_TIME:           true
    Mounts:
      /etc/git-secret/ssh from git-sync-ssh-key (ro,path="gitSshKey")
      /git from dags (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-67549 (ro)
Containers:
  triggerer:
    Container ID:
    Image:         airflow-custom:1.0.0
    Image ID:
    Port:          <none>
    Host Port:     <none>
    Args:
      bash
      -c
      exec airflow triggerer
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Liveness:       exec [sh -c CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
airflow jobs check --job-type TriggererJob --hostname $(hostname)
] delay=10s timeout=20s period=60s #success=1 #failure=5
    Environment Variables from:
      airflow-variables  ConfigMap  Optional: false
    Environment:
      AIRFLOW__CORE__FERNET_KEY:            <set to the key 'fernet-key' in secret 'airflow-fernet-key'>                      Optional: false
      AIRFLOW__CORE__SQL_ALCHEMY_CONN:      <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__DATABASE__SQL_ALCHEMY_CONN:  <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW_CONN_AIRFLOW_DB:              <set to the key 'connection' in secret 'airflow-airflow-metadata'>                Optional: false
      AIRFLOW__WEBSERVER__SECRET_KEY:       <set to the key 'webserver-secret-key' in secret 'airflow-webserver-secret-key'>  Optional: false
    Mounts:
      /opt/airflow/airflow.cfg from config (ro,path="airflow.cfg")
      /opt/airflow/config/airflow_local_settings.py from config (ro,path="airflow_local_settings.py")
      /opt/airflow/dags from dags (ro)
      /opt/airflow/logs from logs (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-67549 (ro)
  git-sync:
    Container ID:
    Image:          k8s.gcr.io/git-sync/git-sync:v3.4.0
    Image ID:
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Environment:
      GIT_SSH_KEY_FILE:            /etc/git-secret/ssh
      GIT_SYNC_SSH:                true
      GIT_KNOWN_HOSTS:             false
      GIT_SYNC_REV:                HEAD
      GIT_SYNC_BRANCH:             main
      GIT_SYNC_REPO:               ssh://[email protected]/ShenghaoHuang/airflow-dags.git
      GIT_SYNC_DEPTH:              1
      GIT_SYNC_ROOT:               /git
      GIT_SYNC_DEST:               repo
      GIT_SYNC_ADD_USER:           true
      GIT_SYNC_WAIT:               60
      GIT_SYNC_MAX_SYNC_FAILURES:  0
    Mounts:
      /etc/git-secret/ssh from git-sync-ssh-key (ro,path="gitSshKey")
      /git from dags (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-67549 (ro)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      airflow-airflow-config
    Optional:  false
  dags:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  git-sync-ssh-key:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  airflow-ssh-git-secret
    Optional:    false
  logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  kube-api-access-67549:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  19m                   default-scheduler  Successfully assigned airflow/airflow-triggerer-5b459bcf8f-sm4lx to airflow-cluster-worker3  
  Normal   Pulled     19m                   kubelet            Container image "airflow-custom:1.0.0" already present on machine
  Normal   Created    19m                   kubelet            Created container wait-for-airflow-migrations
  Normal   Started    19m                   kubelet            Started container wait-for-airflow-migrations
  Normal   Pulling    19m                   kubelet            Pulling image "k8s.gcr.io/git-sync/git-sync:v3.4.0"
  Normal   Pulled     19m                   kubelet            Successfully pulled image "k8s.gcr.io/git-sync/git-sync:v3.4.0" in 9.322030396s
  Normal   Started    18m (x4 over 19m)     kubelet            Started container git-sync-init
  Normal   Created    17m (x5 over 19m)     kubelet            Created container git-sync-init
  Normal   Pulled     17m (x4 over 19m)     kubelet            Container image "k8s.gcr.io/git-sync/git-sync:v3.4.0" already present on machine
  Warning  BackOff    4m37s (x68 over 19m)  kubelet            Back-off restarting failed container

I've tried running logs but this is what I get

I saw this issue was posted recently #638
With other people in the comments also running into the same issue without any solution :(

[tkansara]

Hey I had to overrider the resources and waitformigrations for it to work:

scheduler:
  resources:
    requests:
      cpu: 500m
      memory: 2G
    limits:
      cpu: 500m
      memory: 2G
  waitForMigrations:
    enabled: false

[MikeKulls2]

It will just be something causing it to not be able to connect to github repo. Is the private key in the secret? Is the public key installed at github? Is the secret the same name as specified in values file? Is the secret named id_rsa in the secret? Is the secret in the same namespace? Did you specify "id_rsa" for dags.gitSync.sshSecretKey? (I think that is likely the default anyone). Can you try from the command line with git clone and see if that works?