pack build cannot authenticate against docker registries that require client certificate

[tgquan67]

Summary

I have a private docker registry placed behind a proxy that requires client certificate. I setup my docker CLI according to https://docs.docker.com/engine/security/certificates/, and I can login to, pull from and push to it as normal. However with pack-cli, I can only connect to the registry if I bypass the proxy and connect to the registry directly.

The builder is named private-registry.example.com/builder/test:latest with the following config:

# Buildpacks to include in builder
[[buildpacks]]
id = "test-buildpack"
uri = "../buildpacks/test-buildpack"

# Order used for detection
[[order]]
    # This buildpack will display build-time information (as a dependency)
    [[order.group]]
    id = "test-buildpack"

# Stack that will be used by the builder
[stack]
id = "com.example.buildpacks.stacks.test"
# This image is used at runtime
run-image = "private-registry.example.com/images/test-run-image:latest"
# This image is used at build-time
build-image = "private-registry.example.com/images/test-build-image:latest"

---

Reproduction

Steps

1. run pack build my-image --builder private-registry.example.com/builder/test:latest --path .

Current behavior

latest: Pulling from builder/test
Digest: sha256:4d7b765590695584fbf039cd80335a3e2f3c743c1ef133821d9fc48c1b300828
Status: Downloaded newer image for private-registry.example.com/builder/test:latest
panic: runtime error: index out of range [0] with length 0

goroutine 1 [running]:
github.com/buildpacks/pack/pkg/client.getBestRunMirror({0x7ffdd9aa8397, 0x16}, {0xc000526680, 0x40}, {0x0, 0x0, 0x0?}, {0x0, 0x0, 0x0}, ...)
        github.com/buildpacks/pack/pkg/client/common.go:123 +0x348
github.com/buildpacks/pack/pkg/client.(*Client).resolveRunImage(0xc00003f600, {0x0?, 0xc00040a280?}, {0x12ca812?, 0x47?}, {0x7ffdd9aa8397?, 0x0?}, {{0xc000526680, 0x40}, {0x0, ...}}, ...)
        github.com/buildpacks/pack/pkg/client/common.go:42 +0xeb
github.com/buildpacks/pack/pkg/client.(*Client).Build(_, {_, _}, {{0x0, 0x0}, {0x7ffdd9aa837c, 0x10}, {0x7ffdd9aa8397, 0x47}, {0x0, ...}, ...})
        github.com/buildpacks/pack/pkg/client/build.go:343 +0x868
github.com/buildpacks/pack/internal/commands.Build.func1(0xc000004f00, {0xc000154900?, 0x0?, 0x0?})
        github.com/buildpacks/pack/internal/commands/build.go:157 +0x12f8
github.com/buildpacks/pack/internal/commands.Build.logError.func3(0xc0001dc200?, {0xc000154900?, 0x4?, 0x12bcc9c?})
        github.com/buildpacks/pack/internal/commands/commands.go:58 +0x44
github.com/spf13/cobra.(*Command).execute(0xc000004f00, {0xc000154870, 0x9, 0x9})
        github.com/spf13/[email protected]/command.go:983 +0xabc
github.com/spf13/cobra.(*Command).ExecuteC(0xc000004c00)
        github.com/spf13/[email protected]/command.go:1115 +0x3ff
github.com/spf13/cobra.(*Command).Execute(...)
        github.com/spf13/[email protected]/command.go:1039
github.com/spf13/cobra.(*Command).ExecuteContext(0x14a0da0?, {0x1498bd8?, 0xc00040a280?})
        github.com/spf13/[email protected]/command.go:1032 +0x47
main.main()
        github.com/buildpacks/pack/cmd/pack/main.go:26 +0xd5

Expected behavior

This is what happens when I bypass the proxy

latest: Pulling from builder/test
Digest: sha256:4d7b765590695584fbf039cd80335a3e2f3c743c1ef133821d9fc48c1b300828
Status: Downloaded newer image for private-registry.example.com/builder/test:latest
latest: Pulling from images/test-run-image:latest
Digest: sha256:928a3445cff2821f5e1e4f20ec808d22a5a9690dae545f97e44f15af7575a0e3
Status: Downloaded newer image for private-registry.example.com/images/test-run-image:latest
0.16.0: Pulling from buildpacksio/lifecycle
Digest: sha256:f75a04887fced3ae0504a37edb2c0d29d366511cd9ede34dbb90c5282b106e79
Status: Image is up to date for buildpacksio/lifecycle:0.16.0
===> ANALYZING
[other stuff happened as normal]

---

Environment

pack info

Pack:
  Version:  0.33.1
  OS/Arch:  linux/amd64

Default Lifecycle Version:  0.18.4

Supported Platform APIs:  0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 0.10, 0.11, 0.12

Config:
(no config file found at /home/me/.pack/config.toml)

docker info

Client: Docker Engine - Community
 Version:    24.0.9
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.12.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.24.5
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 1
  Running: 0
  Paused: 0
  Stopped: 1
 Images: 34
 Server Version: 24.0.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-171-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 62.83GiB
 Name: quan-work-ubuntu
 ID: DPTV:QXJW:TN6C:CVLT:2DU7:DGNV:7IRF:UGNG:AOPX:T3VE:JS3X:6KIP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  localhost:32000
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

[natalieparellano]

I believe this relates to google/go-containerregistry#211 as we use GGCR to access the registry image. Setting SSL_CERT_FILE on the host should allow pack to access the image, but you would also need to set SSL_CERT_FILE in the builder image and mount in the cert directory (pack build --volume) for the lifecycle to access it.

Related lifecycle issue: buildpacks/lifecycle#1077

[tgquan67]

@natalieparellano I believe that SSL_CERT_FILE points to a root certificate in the case that your server is using a self-signed cert. What I need in my case is support for client certificate which is used by the proxy to verify the user identity. It was mentioned in this comment google/go-containerregistry#211 (comment)

[natalieparellano]

What I need in my case is support for client certificate

Apologies for my hasty reading of the linked issue. I think we'd need the changes implemented upstream before we can fix it on our end

[natalieparellano]

Blocking on needed changes upstream