failure pushing to on-cluster registry using minio + minikube

[ryane]

• workflow-v2.16.0
• minikube
• default install via helm: helm install deis/workflow --namespace deis --set router.host_port.enabled=true.

I installed deis on minikube by following the guide here: https://deis.com/docs/workflow/quickstart/provider/minikube/boot/

Everything has been working correctly until I tried a Dockerfile build: https://deis.com/docs/workflow/applications/using-dockerfiles/

...
Removing intermediate container 50774bbc9eb9
Successfully built fa44e0887d04
Pushing to registry
received unexpected HTTP status: 500 Internal Server Error
remote: 2017/07/21 21:16:52 Error running git receive hook [Build pod exited with code 1, stopping build.]
To ssh://deis-builder.192.168.99.100.nip.io:2222/sample.git
 ! [remote rejected] deis-test -> deis-test (pre-receive hook declined)
error: failed to push some refs to 'ssh://[email protected]:2222/sample.git'

Looking at the registry logs, I see the following error:

time="2017-07-21T21:13:35.815856518Z" level=error msg="unknown error completing upload: s3aws: NoSuchKey: The specified key does not exist.\n\tstatus code: 404, request id: YDEIGMCM63C8BO0V" go.version=go1.7.3 http.request.host="127.0.0.1:5555" http.request.id=167566f8-c073-4fe5-8643-b03ae9a6cc34 http.request.method=PUT http.request.remoteaddr=172.17.0.1 http.request.uri="/v2/sample/blobs/uploads/e912e92d-29f5-4325-b0da-492c95b76ebb?_state=DNMrhG65J4OOz5Mqboyv6V-aOOjrhuRA4c-ltSN19bd7Ik5hbWUiOiJhcmNoIiwiVVVJRCI6ImU5MTJlOTJkLTI5ZjUtNDMyNS1iMGRhLTQ5MmM5NWI3NmViYiIsIk9mZnNldCI6NzQ3MzcwODQsIlN0YXJ0ZWRBdCI6IjIwMTctMDctMjFUMjE6MTM6MDNaIn0%3D&digest=sha256%3Ac018073abd268f5850152cebe650b0356e8f9990f4dbc5d3110612251d210664" http.request.useragent="docker/1.11.1 go/go1.5.4 git-commit/5604cbe kernel/4.9.13 os/linux arch/amd64 UpstreamClient(docker-py/1.10.6)" instance.id=bd2ea05e-b664-45ed-960a-a9cb5db29bf2 service=registry vars.name=sample vars.uuid=e912e92d-29f5-4325-b0da-492c95b76ebb version=v2.6.0

I do not see anything of interest in the minio logs:

$ kubectl logs -f deis-minio-3370481340-glkd6
minio server /home/minio/
2017/07/21 20:40:51 starting Minio server
2017/07/21 20:40:51 starting health check server on 0.0.0.0:8082
Migration from version ‘2’ to ‘3’ completed successfully.
Migration from version ‘3’ to ‘4’ completed successfully.

AccessKey: xxxxx  SecretKey: xxxxx  Region: us-east-1

Minio Object Storage:
    http://127.0.0.1:9000
    http://172.17.0.13:9000

Minio Browser:
    http://127.0.0.1:9000
    http://172.17.0.13:9000

To configure Minio Client:
    $ wget https://dl.minio.io/client/mc/release/linux-amd64/mc
    $ chmod 755 mc
    $ ./mc config host add myminio http://127.0.0.1:9000 xxxxx xxxxx

Again, this is with a default installation following the guides above. Did I miss a step somewhere or are there any additional undocumented steps I need to perform to get the minio-backed registry working correctly? This looks similar to #826 as well.

[wikkid]

Same problems here, but on bare metal K8S cluster + Deis workflow

[bacongobbler]

I assume this is the same issue we are seeing at #826. I'm going to close this ticket as a duplicate of that one so we can keep the discussion all in one ticket, but please re-open if this seems like a completely separate issue. Thanks!