failure in image push does not stop progress and fails later during yaml apply

[Schnitzel]

see the following log output:

##############################################
STEP Push images to registry
The push refers to repository [[harbor-url]/[project]/master/nginx]
d3175c183819: Pushed
3ac28263aa9a: Pushed
668e618b696e: Retrying in 5 seconds
205ad92006fc: Retrying in 5 seconds
668e618b696e: Retrying in 4 seconds
205ad92006fc: Retrying in 4 seconds
668e618b696e: Retrying in 3 seconds
205ad92006fc: Retrying in 3 seconds
668e618b696e: Retrying in 2 seconds
205ad92006fc: Retrying in 2 seconds
668e618b696e: Retrying in 1 second
205ad92006fc: Retrying in 1 second
668e618b696e: Retrying in 10 seconds
668e618b696e: Retrying in 9 seconds
668e618b696e: Retrying in 8 seconds
205ad92006fc: Retrying in 10 seconds
668e618b696e: Retrying in 7 seconds
8097f9967fbe: Pushed
205ad92006fc: Retrying in 9 seconds
8180101eab5e: Retrying in 5 seconds
668e618b696e: Retrying in 6 seconds
205ad92006fc: Retrying in 8 seconds
8180101eab5e: Retrying in 4 seconds
668e618b696e: Retrying in 5 seconds
205ad92006fc: Retrying in 7 seconds
8180101eab5e: Retrying in 3 seconds
668e618b696e: Retrying in 4 seconds
205ad92006fc: Retrying in 6 seconds
8180101eab5e: Retrying in 2 seconds
668e618b696e: Retrying in 3 seconds
205ad92006fc: Retrying in 5 seconds
8180101eab5e: Retrying in 1 second
668e618b696e: Retrying in 2 seconds
205ad92006fc: Retrying in 4 seconds
668e618b696e: Retrying in 1 second
205ad92006fc: Retrying in 3 seconds
668e618b696e: Retrying in 15 seconds
205ad92006fc: Retrying in 2 seconds
8180101eab5e: Pushed
668e618b696e: Retrying in 14 seconds
205ad92006fc: Retrying in 1 second
668e618b696e: Retrying in 13 seconds
668e618b696e: Retrying in 12 seconds
668e618b696e: Retrying in 11 seconds
205ad92006fc: Pushed
760644ab4609: Pushed
668e618b696e: Retrying in 10 seconds
668e618b696e: Retrying in 9 seconds
36e146b1be7f: Pushed
668e618b696e: Retrying in 8 seconds
94bb71355a48: Pushed
668e618b696e: Retrying in 7 seconds
fb6dd2829db8: Pushed
8c0e64ddf02b: Pushed
668e618b696e: Retrying in 6 seconds
fcf1a39f7c22: Pushed
668e618b696e: Retrying in 5 seconds
668e618b696e: Retrying in 4 seconds
b6125321ec6b: Pushed
d87c4a5168b3: Pushed
668e618b696e: Retrying in 3 seconds
668e618b696e: Retrying in 2 seconds
f9c4a6e61f49: Pushed
668e618b696e: Retrying in 1 second
642c39271710: Pushed
7ef7c977e3d5: Pushed
44f6bf79ccd7: Pushed
fa2402820adf: Pushed
668e618b696e: Pushed
unknown blob
The push refers to repository [[harbor-url]/[project]/master/cli]
9c85719de831: Pushed
b0fab4eaefb9: Pushed
a87e74b21319: Pushed
d4c4a8b21ce6: Pushed
d496a8b5052d: Pushed
a56b4203c3ee: Pushed
345a1e802f35: Pushed
ff964fc97284: Pushed
59b039a93ce5: Pushed
39ecedb74bbd: Pushed
8fb5ca324974: Pushed
56d0199a9540: Pushed
07cc6800e2df: Pushed
8e7e98167183: Pushed
491f84c6b570: Pushed
e2daa804af84: Pushed
75b4e07143ab: Pushed
08633d833402: Pushed
e20fc254c0ff: Pushed
5883b2c022c1: Pushed
041aab66fc95: Pushed
6f6c14a459be: Pushed
b4c2e7b867ff: Pushed
8141187d8864: Pushed
21eee3978bbd: Pushed
d136254a3046: Pushed
b7e9fe5f3a94: Pushed
611b973d07cb: Pushed
a70d50891cbd: Pushed
c3f67d46e025: Pushed
6a6b6403af59: Pushed
facd2ae711c9: Pushed
1ad92fdc06e1: Pushed
d71c5aa9cdb7: Pushed
3fde21036de7: Pushed
f89b79d5c609: Pushed
24c371b81e06: Pushed
d995fb9d3e6a: Pushed
f1f8b389e80b: Pushed
c9cf36aa228c: Pushed
31df03794e61: Pushed
7a5011dccb87: Pushed
dffeda9fe4d2: Pushed
4b920958bfc4: Pushed
f4c77de414e2: Pushed
ccbb2da0299c: Pushed
e87f7c4073ed: Pushed
fe59a87308b1: Pushed
345b55c90b38: Pushed
8e8fa5b7f461: Pushed
eca656915cb8: Pushed
731d8204f814: Pushed
1d94ebbe7fac: Pushed
38d6fe389fca: Pushed
9d71a44d6779: Pushed
07b191451f27: Pushed
fea45fae0dd2: Pushed
e650ddc47703: Pushed
ac7a641a6299: Pushed
309a6428997e: Pushed
744437e91ddb: Pushed
909979f517a2: Pushed
49a1a2a80ba6: Pushed
994393dc58e7: Pushed
cd3f5d78e589: Pushed
latest: digest: sha256:460144afd2abaf5415fd066854d08f5d7af72cc6d3c4470a3b08fdfb853072f8 size: 14099
The push refers to repository [[harbor-url]/[project]/master/php]
309a85ee452a: Pushed
f1f8b389e80b: Mounted from [project]/master/cli
24c371b81e06: Mounted from [project]/master/cli
d995fb9d3e6a: Mounted from [project]/master/cli
c9cf36aa228c: Mounted from [project]/master/cli
8e8fa5b7f461: Mounted from [project]/master/cli
31df03794e61: Mounted from [project]/master/cli
dffeda9fe4d2: Mounted from [project]/master/cli
7a5011dccb87: Mounted from [project]/master/cli
4b920958bfc4: Mounted from [project]/master/cli
f4c77de414e2: Mounted from [project]/master/cli
ccbb2da0299c: Mounted from [project]/master/cli
e87f7c4073ed: Mounted from [project]/master/cli
345b55c90b38: Mounted from [project]/master/cli
1d94ebbe7fac: Mounted from [project]/master/cli
731d8204f814: Mounted from [project]/master/cli
eca656915cb8: Mounted from [project]/master/cli
07b191451f27: Mounted from [project]/master/cli
fea45fae0dd2: Mounted from [project]/master/cli
38d6fe389fca: Mounted from [project]/master/cli
e650ddc47703: Mounted from [project]/master/cli
9d71a44d6779: Mounted from [project]/master/cli
909979f517a2: Mounted from [project]/master/cli
ac7a641a6299: Mounted from [project]/master/cli
309a6428997e: Mounted from [project]/master/cli
744437e91ddb: Mounted from [project]/master/cli
49a1a2a80ba6: Mounted from [project]/master/cli
cd3f5d78e589: Mounted from [project]/master/cli
994393dc58e7: Mounted from [project]/master/cli
d178e1a5aecc: Pushed
latest: digest: sha256:2febe79e38dbc1852788cd4c8ae6f30bb1cd1fb5c1e11ad327a287788ab2f621 size: 6797
##############################################
STEP Image Push to Registry: Completed at 2022-09-07 12:20:52 (UTC) Duration 00:02:35 Elapsed 00:03:24
##############################################
##############################################
STEP Create PVC, deployments, and cronjobs
cli: => Type: cli-persistent
cli: => Image: [harbor-url]/[project]/master/cli@sha256:460144afd2abaf5415fd066854d08f5d7af72cc6d3c4470a3b08fdfb853072f8
cli: => Persistent storage path: /app/web/sites/default/files/
cli: => Persistent storage name: nginx
cli: => Cronjob: drush-cron
cli: => Cronjob: drush-cron schedule: 0,15,30,45 * * * *
cli: => Cronjob: drush-cron command: drush cron
cli: => Cronjob: drush-cron created to run inside pod
inPodCronjobs: 0,15,30,45 * * * * drush cron
nginx: => Type: nginx-php-persistent
nginx: => Image: 
nginx: => Persistent storage path: /app/web/sites/default/files/
nginx: => Persistent storage name: nginx
mariadb: => Type: mariadb-dbaas
mariadb: => Image: 
environment: production
inPodCronjobs: ""
##############################################
STEP Deployment Templating: Completed at 2022-09-07 12:20:54 (UTC) Duration 00:00:02 Elapsed 00:03:26
##############################################
##############################################
STEP Apply resources to Kubernetes
deployment.apps/cli configured
mariadbconsumer.mariadb.amazee.io/mariadb unchanged
prebackuppod.backup.appuio.ch/mariadb-prebackuppod configured
persistentvolumeclaim/nginx unchanged
service/nginx unchanged
The Deployment "nginx" is invalid: spec.template.spec.containers[0].image: Required value

the push of the nginx image failed with unknown blob, but the script continued running and then failed when applying the changes to the cluster.

This could lead to unwanted consequences, like the cli deployment has now been updated with a new image, which the nginx deployment is still an older image.

I would expect the script to stop if the pushing of an image fails.
Actually I would expect the script to retry if the pushing fails, as we did before (see the --retries 4) here:

build-deploy-tool/legacy/build-deploy-docker-compose.sh

Line 1228 in d318736

parallel --retries 4 < /kubectl-build-deploy/lagoon/push

[shreddedbacon]

This is one that uses the build log verbosity branch to cut down on log noise #88

Will do some checks

[shreddedbacon]

closing as this has been solved in the buildlog verbosity pr now