Error and Completed state on fresh pods with negligible load

[solsson]

While working on #35 I suddenly had a depressing status report:

NAME                               READY     STATUS      RESTARTS   AGE
consumer-test1-1588418567-r245l    1/1       Running     0          2h
kafka-0                            2/2       Running     3          3h
kafka-1                            2/2       Running     0          3h
kafka-2                            0/2       Completed   0          3h
monitoring-test-3979390023-c5nwm   1/1       Running     0          7m
pzoo-0                             0/2       Error       0          21h
pzoo-1                             2/2       Running     0          21h
pzoo-2                             2/2       Running     2          21h
testclient                         1/1       Running     1          2h
zoo-0                              2/2       Running     0          20h
zoo-1                              2/2       Running     0          20h

Logs show nothing unusual, but kubectl describe says:

  FirstSeen	LastSeen	Count	From							SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----							-------------	--------	------		-------
  2h		36s		64	kubelet, gke-eu-west-3-default-pool-c36cf3c6-nfgg			Warning		FailedMount	Unable to mount volumes for pod "pzoo-0_kafka(bd1a380f-5a60-11e7-93e7-42010a84002d)": timeout expired waiting for volumes to attach/mount for pod "kafka"/"pzoo-0". list of unattached/unmounted volumes=[data]
  2h		36s		64	kubelet, gke-eu-west-3-default-pool-c36cf3c6-nfgg			Warning		FailedSync	Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "kafka"/"pzoo-0". list of unattached/unmounted volumes=[data]

  FirstSeen	LastSeen	Count	From							SubObjectPath	Type		Reason		Message
  ---------	--------	-----	----							-------------	--------	------		-------
  2h		24s		65	kubelet, gke-eu-west-3-default-pool-c36cf3c6-nfgg			Warning		FailedMount	Unable to mount volumes for pod "kafka-2_kafka(247ecd2f-5afa-11e7-93e7-42010a84002d)": timeout expired waiting for volumes to attach/mount for pod "kafka"/"kafka-2". list of unattached/unmounted volumes=[data]
  2h		24s		65	kubelet, gke-eu-west-3-default-pool-c36cf3c6-nfgg			Warning		FailedSync	Error syncing pod, skipping: timeout expired waiting for volumes to attach/mount for pod "kafka"/"kafka-2". list of unattached/unmounted volumes=[data]

All PVCs are bound.

Persistence setup was changed in #33.

[solsson]

Nodes where at 50-60 percent memory usage so that isn't the problem.

After some time I saw kubectl delete -f resulting in error: error when stopping "zookeeper/50pzoo.yml": timed out waiting for "pzoo" to be synced.

Interestingly I got no effect whatsoever from deletion of the statefulsets, nor from scale down, while pods were in this state. Just showed "0 desired | 3 total".

kubectl delete pod caused proper termination of healthy pod.

[solsson]

I haven't seen Completed since this happened. Did have a lot of problems with restarting pods due to resource limits in #49.