Pushing to GitHub fails with BLOB_UPLOAD_UNKNOWN

[mihalyr]

Environment:

• Jib version: 3.4.4
• Build tool: gradle (8.10.2)
• OS: Linux (GitHub Actions, runs-on: ubuntu-latest)

Description of the issue:

When using Jib Gradle plugin to push a container image to GitHub Container Registry (ghcr.io), it fails frequently in the middle of the operation with the following error:

 > com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<hash> but failed because: other: blob upload unknown to registry

The stacktrace contains the following error message:

Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
PUT https://ghcr.io/v2/<org>/<repo>/blobs/upload/<uuid>?digest=sha256:<hash>
{"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}

	at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)
	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:349)
	... 176 more

So far every time when I manually re-run the failed job (GitHub Actions workflow), it succeeds on retry without problems.

Expected behavior:

Running Jib to upload containers to GitHub registry without errors and manual retries.

Steps to reproduce:

1. Create a GitHub Action workflow that uses gradle jib plugin to push to the GitHub container registry in the same repository (jib.to.image=ghcr.io/org/repo, jib.to.auth.username=github.actor, jib.to.auth.password=github.token, jib.container.labels=org.opencontainers.image.source=github.server_url/github.repository)
2. Run the workflow

Log output:

Build failure logs

Executing tasks:
[======================        ] 72.7% complete
> building images to registry


Executing tasks:
[======================        ] 72.7% complete
> checking existence of manifest for sha256:74e...


Executing tasks:
[=========================     ] 81.8% complete
> building images to registry


Executing tasks:
[=========================     ] 81.8% complete
> scheduling pushing manifests


Executing tasks:
[===========================   ] 90.9% complete
> building images to registry


FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':server:jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<hash> but failed because: other: blob upload unknown to registry

* Try:


A new version of jib-gradle-plugin (3.4.3) is available (currently using 3.4.2). Update your build configuration to use the latest features and fixes!
https://github.com/GoogleContainerTools/jib/blob/master/jib-gradle-plugin/CHANGELOG.md

Please see https://github.com/GoogleContainerTools/jib/blob/master/docs/privacy.md for info on disabling this update check.

Executing tasks:
[===========================   ] 90.9% complete
> building images to registry


> Task :server:jib FAILED
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':server:jib'.
	at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:130)
	at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:127)
	at org.gradle.composite.internal.DefaultBuildController.doRun(DefaultBuildController.java:181)
	at org.gradle.composite.internal.DefaultBuildController.access$000(DefaultBuildController.java:50)
	at org.gradle.composite.internal.DefaultBuildController$BuildOpRunnable.lambda$run$0(DefaultBuildController.java:198)
	at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:85)
	at org.gradle.composite.internal.DefaultBuildController$BuildOpRunnable.run(DefaultBuildController.java:198)
	at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64)
	at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:48)
Caused by: com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<hash> but failed because: other: blob upload unknown to registry
	at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild(JibBuildRunner.java:280)
	at com.google.cloud.tools.jib.gradle.BuildImageTask.buildImage(BuildImageTask.java:122)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at org.gradle.internal.reflect.JavaMethod.invoke(JavaMethod.java:125)
	... 145 more
Caused by: com.google.cloud.tools.jib.registry.RegistryErrorException: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<hash> but failed because: other: blob upload unknown to registry
	at com.google.cloud.tools.jib.registry.RegistryErrorExceptionBuilder.build(RegistryErrorExceptionBuilder.java:101)
	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.newRegistryErrorException(RegistryEndpointCaller.java:212)
	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:153)
	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:114)
	at com.google.cloud.tools.jib.registry.RegistryClient.callRegistryEndpoint(RegistryClient.java:623)
	at com.google.cloud.tools.jib.registry.RegistryClient.pushBlob(RegistryClient.java:564)
	at com.google.cloud.tools.jib.builder.steps.PushBlobStep.call(PushBlobStep.java:89)
	at com.google.cloud.tools.jib.builder.steps.PushLayerStep.call(PushLayerStep.java:92)
	at com.google.cloud.tools.jib.builder.steps.PushLayerStep.call(PushLayerStep.java:33)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:79)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.scheduleCallables(StepsRunner.java:648)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$pushApplicationLayers$13(StepsRunner.java:507)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:76)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:327)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:79)
	at com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:37)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.pushApplicationLayers(StepsRunner.java:505)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.lambda$run$2(StepsRunner.java:218)
	at com.google.cloud.tools.jib.builder.steps.StepsRunner.run(StepsRunner.java:218)
	at com.google.cloud.tools.jib.api.Containerizer.run(Containerizer.java:406)
	at com.google.cloud.tools.jib.api.JibContainerBuilder.containerize(JibContainerBuilder.java:610)
	at com.google.cloud.tools.jib.plugins.common.JibBuildRunner.runBuild(JibBuildRunner.java:229)
	... 148 more
Caused by: com.google.cloud.tools.jib.http.ResponseException: 404 Not Found
PUT https://ghcr.io/v2/<org>/<repo>/blobs/upload/<uploadid>?digest=sha256:<hash>
{"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}

	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:355)
	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:266)
	at com.google.cloud.tools.jib.registry.RegistryEndpointCaller.call(RegistryEndpointCaller.java:138)
	... 174 more
Caused by: com.google.api.client.http.HttpResponseException: 404 Not Found
PUT https://ghcr.io/v2/<org>/<repo>/blobs/upload/<uploadid>?digest=sha256:<hash>
{"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}

	at com.google.api.client.http.HttpResponseException$Builder.build(HttpResponseException.java:293)
	at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1118)
	at com.google.cloud.tools.jib.http.FailoverHttpClient.call(FailoverHttpClient.java:349)
	... 176 more


BUILD FAILED in 2m 50s
22 actionable tasks: 15 executed, 7 from cache
Error: Process completed with exit code 1.

[MrPowerGamerBR]

I can confirm the same issue, it is very annoying. There are other users talking about the same issue on #2372, and @KRK1ST said that he talked with GitHub Support and they said that GitHub belives that it is a issue in Jib.

"one theory being that Jib may be retrying an upload before the previous upload has actually failed. This could also explain the inconsistency of failure you are seeing, as different uploads would take different amounts of time, and thus some would fit within the retry window (thus not failing), while a few might exceed the retry window and thus fail."

I have tried running it with -Djib.serialize=true but it doesn't fix the issue. What I'm doing right now is running all jib tasks in different build steps with -Djib.serialize=true, it doesn't 100% fix the issue, but it does make it less common.

[martintomac]

It happens with maven also. We have this issue for long time now over many versions.

Error:  Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default-cli) on project download-gateway: Tried to push BLOB for ghcr.io/<org>/<repo> with digest sha256:<sha> but failed because: other: blob upload unknown to registry: 404 Not Found
Error:  PUT https://ghcr.io/v2/<org>/<repo>/blobs/upload/<uuid>?digest=sha256:<sha>
Error: [ERROR] {"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}

Some of our builds push multiple images sequentially so this fails our build every other or third time. We tried -Djib.serialize=true but it doesn't fix the issue.

[nmwael]

Hmm, we've been hit by this.. We also have a multi module project. In our case I would say its an GH issue since we can verify that the final image are uploaded correctly. Could it be that GH's have nodes that are lagging behind and it sometimes fails because of that?

I'm also a bit puzzled about this digest, it does not appear from our final manifest:
`
https://ghcr.io/v2/REDACTED/blobs/upload/9471d4b5-1ad4-40cc-8940-3c3f25a08079?digest=sha256:e48db31723748e5b38d8d8f257ee3c37f012fb9eef3b2125460af3dc59967b2c

`
Error for reference

Warning: ARNING] PUT https://ghcr.io/v2/REDACTED/blobs/upload/9471d4b5-1ad4-40cc-8940-3c3f25a08079?digest=sha256:e48db31723748e5b38d8d8f257ee3c37f012fb9eef3b2125460af3dc59967b2c failed and will be retried
[INFO] [INFO] 
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] Reactor Summary for REDACTED:
[INFO] [INFO] 
[INFO] [INFO] REDACTED ...... SUCCESS [ 51.362 s]
[INFO] [INFO] REDACTED ...... SUCCESS [02:25 min]
[INFO] [INFO] REDACTED ...... SUCCESS [01:33 min]
[INFO] [INFO] REDACTED ...... SUCCESS [ 20.827 s]
[INFO] [INFO] REDACTED ...... SUCCESS [ 50.092 s]
[INFO] [INFO] REDACTED ...... SUCCESS [ 53.515 s]
[INFO] [INFO] REDACTED ...... SUCCESS [02:05 min]
[INFO] [INFO] REDACTED ...... FAILURE [ 56.547 s]
[INFO] [INFO] REDACTED ...... SKIPPED
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] BUILD FAILURE
[INFO] [INFO] ------------------------------------------------------------------------
[INFO] [INFO] Total time:  09:56 min
[INFO] [INFO] Finished at: 2024-10-21T10:20:16Z
[INFO] [INFO] ------------------------------------------------------------------------
Error: [ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.4.0:build (default) on project REDACTED: Tried to push BLOB for ghcr.io/REDACTED with digest sha256:e48db31723748e5b38d8d8f257ee3c37f012fb9eef3b2125460af3dc59967b2c but failed because: other: blob upload unknown to registry: 404 Not Found
Error: [ERROR] PUT https://ghcr.io/v2/REDACTED/blobs/upload/9471d4b5-1ad4-40cc-8940-3c3f25a08079?digest=sha256:e48db31723748e5b38d8d8f257ee3c37f012fb9eef3b2125460af3dc59967b2c
Error: [INFO] [ERROR] {"errors":[{"code":"BLOB_UPLOAD_UNKNOWN","message":"blob upload unknown to registry"}]}

[mihalyr]

Still an issue on 3.4.4

[murugslendistry]

We are using jib 3.4.3 and seeing same issue for uploading images to ghcr.io. As mentioned earlier rerunning the job works sometimes but its annoying in a multi module project. Hoping for faster resolution.

Its happening for both maven and gradle.

[gustavoteixeirah]

I'm encountering the same error:

* What went wrong:
Execution failed for task ':jib'.
> com.google.cloud.tools.jib.plugins.common.BuildStepsExecutionException: 
Tried to push BLOB for ghcr.io/agro-amazonia/ibi-backend/job with digest 
sha256:ed843f10b0546cf424265cfad5e33199a1dfdba4ec1636e1438a6495cc01ad99
 but failed because: other: blob upload unknown to registry

I've attempted upgrading both Gradle (to 8.5) and Jib (to 3.4.4), but the issue still persists.

[mihalyr]

It's very unfortunate, we just recently moved to use jib from our custom layered approach with docker push to publish, but in some cases it's like one step forward, two steps back.

For this particular publish issue, I was thinking about two workarounds:

1. Use jibDockerBuild and then just use docker push for the image upload
2. Retry

I didn't want to just abandon hope yet before I hear back from google on this ticket, so I am testing retries with https://github.com/marketplace/actions/retry-step

      - name: Build and push image
        # Jib push is very unreliable with GitHub registry
        # See https://github.com/GoogleContainerTools/jib/issues/4301
        # -Djib.serialize=true was supposed to help here based on some issue comments, but it did not help much
        # If jib would continue to give us issues here, we can switch to jibDockerBuild and then use regular docker push
        # For now we retry 3 times, usually the second retry works.
        uses: nick-fields/retry@v3
        with:
          timeout_minutes: 10
          max_attempts: 3
          command: |
            SOURCE_LABEL="org.opencontainers.image.source"
            SOURCE_VALUE="$GITHUB_SERVER_URL/$GITHUB_REPOSITORY"
            ./gradlew :server:jib \
              -Djib.to.image="${REGISTRY}/${IMAGE_NAME}:latest" \
              -Djib.to.auth.username="${{ github.actor }}" \
              -Djib.to.auth.password="${{ github.token }}" \
              -Djib.to.tags="sha-${GITHUB_SHA},${VERSION_TAG}" \
              -Djib.container.labels="$SOURCE_LABEL"="$SOURCE_VALUE" \
              -Djib.serialize=true