Error when trying to clone a repository in container volume

[tk0miya]

• VSCode Version: 1.91.1
• Local OS Version: Darwin x64 23.5.0
• Remote OS Version: N/A (alpine-3.16?)
• Remote Extension/Connection Type: Containers

• Logs:

[72 ms] Dev Containers 0.375.1 in VS Code 1.91.1 (f1e16e1e6214d7c44d078b1f0607b2388f29d729).
[71 ms] Start: リモートの解決
[93 ms] Start: Check Docker is running
[94 ms] Start: Run: docker version
[200 ms] Client:
 Version:           26.1.4
 API version:       1.45
 Go version:        go1.21.11
 Git commit:        5650f9b
 Built:             Wed Jun  5 11:26:02 2024
 OS/Arch:           darwin/amd64
 Context:           orbstack

Server: Docker Engine - Community
 Engine:
  Version:          26.1.4
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.11
  Git commit:       de5c9cf
  Built:            Wed Jun  5 11:29:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.1.13
  GitCommit:        58aa9203c123022138b22cf96540c284876a7910
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
[210 ms] Start: Run: docker volume ls -q
[398 ms] ProxyResolver#loadSystemCertificates count
[402 ms] ProxyResolver#loadSystemCertificates count filtered
[427 ms] Start: Run: docker build -f /var/folders/nz/kb5b16w96bb3f66zc4k784z40000gn/T/vsch/bootstrap-image/0.375.1/bootstrap.Dockerfile -t vsc-volume-bootstrap /var/folders/nz/kb5b16w96bb3f66zc4k784z40000gn/T/vsch/bootstrap-image/0.375.1
[+] Building 0.6s (9/12)                                        docker:orbstack
[+] Building 0.7s (9/12)                                        docker:orbstack
 => [internal] load build definition from bootstrap.Dockerfile             0.0s
 => => transferring dockerfile: 1.02kB                                     0.0s
[+] Building 0.8s (9/12)                                        docker:orbstack
 => [internal] load build definition from bootstrap.Dockerfile             0.0s
 => => transferring dockerfile: 1.02kB                                     0.0s
 => [internal] load metadata for mcr.microsoft.com/devcontainers/base:0-a  0.0s
 => [internal] load .dockerignore                                          0.0s
[+] Building 1.0s (10/12)                                       docker:orbstack
 => [internal] load build definition from bootstrap.Dockerfile             0.0s
 => => transferring dockerfile: 1.02kB                                     0.0s
 => [internal] load metadata for mcr.microsoft.com/devcontainers/base:0-a  0.0s
 => [internal] load .dockerignore                                          0.0s
[+] Building 1.1s (10/12)                                       docker:orbstack
 => [internal] load build definition from bootstrap.Dockerfile             0.0s
 => => transferring dockerfile: 1.02kB                                     0.0s
 => [internal] load metadata for mcr.microsoft.com/devcontainers/base:0-a  0.0s
 => [internal] load .dockerignore                                          0.0s
 => => transferring context: 2B                                            0.0s
 => [1/8] FROM mcr.microsoft.com/devcontainers/base:0-alpine-3.16          0.0s
 => [internal] load build context                                          0.1s
 => => transferring context: 8.87kB                                        0.0s
 => CACHED [2/8] COPY host-ca-certificates.crt /tmp/host-ca-certificates.  0.0s 
 => CACHED [3/8] RUN cat /tmp/host-ca-certificates.crt >> /etc/ssl/certs/  0.0s 
 => CACHED [4/8] RUN csplit -f /usr/local/share/ca-certificates/host-ca-c  0.0s 
 => CACHED [5/8] RUN echo "@old https://dl-cdn.alpinelinux.org/alpine/v3.  0.0s 
 => ERROR [6/8] RUN apk add --no-cache  git-lfs  nodejs  python3  npm  ma  0.8s 
------                                                                          
 > [6/8] RUN apk add --no-cache         git-lfs         nodejs  python3        npm      make    g++     docker-cli      docker-cli-buildx       docker-cli-compose      openssh-client-default@old      ;:
0.420 fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/main/x86_64/APKINDEX.tar.gz
0.530 140352832768840:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:
0.540 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.16/main: Permission denied
0.542 fetch https://dl-cdn.alpinelinux.org/alpine/v3.16/community/x86_64/APKINDEX.tar.gz
0.608 140352832768840:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:
0.614 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.16/community: Permission denied
0.614 fetch https://dl-cdn.alpinelinux.org/alpine/v3.15/main/x86_64/APKINDEX.tar.gz
0.661 140352832768840:error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:ssl/statem/statem_clnt.c:1919:
0.664 WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.15/main: Permission denied
0.665 WARNING: The repository tag for world dependency 'openssh-client-default@old' does not exist
0.665 ERROR: Not committing changes due to missing repository tags. Use --force-broken-world to override.
------
bootstrap.Dockerfile:11
--------------------
  10 |     
  11 | >>> RUN apk add --no-cache \
  12 | >>>      git-lfs \
  13 | >>>      nodejs \
  14 | >>>      python3 \
  15 | >>>      npm \
  16 | >>>      make \
  17 | >>>      g++ \
  18 | >>>      docker-cli \
  19 | >>>      docker-cli-buildx \
  20 | >>>      docker-cli-compose \
  21 | >>>      openssh-client-default@old \
  22 | >>>      ;
  23 |     
--------------------
ERROR: failed to solve: process "/bin/sh -c apk add --no-cache \tgit-lfs \tnodejs \tpython3 \tnpm \tmake \tg++ \tdocker-cli \tdocker-cli-buildx \tdocker-cli-compose \topenssh-client-default@old \t;" did not complete successfully: exit code: 99
[2262 ms] Command failed: docker build -f /var/folders/nz/kb5b16w96bb3f66zc4k784z40000gn/T/vsch/bootstrap-image/0.375.1/bootstrap.Dockerfile -t vsc-volume-bootstrap /var/folders/nz/kb5b16w96bb3f66zc4k784z40000gn/T/vsch/bootstrap-image/0.375.1
[2262 ms] Exit code 1

Steps to Reproduce:

1. Browse https://github.com/microsoft/vscode-remote-try-java
2. Click "Dev Containers: Open" banner

Does this issue occur when you try this locally?: Yes
Does this issue occur when you try this locally and all extensions are disabled?: Yes

[tk0miya]

The error will be resolved and the dev container will invoked well if I modify the bootstrap.Dockerfile to run apk add before modifying TLS certs.

diff -u bootstrap.Dockerfile.orig bootstrap.Dockerfile                                       /var/folders/nz/kb5b16w96bb3f66zc4k784z40000gn/T/vsch/bootstrap-image/0.375.1
--- bootstrap.Dockerfile.orig	2024-07-29 03:18:00
+++ bootstrap.Dockerfile	2024-07-29 03:18:10
@@ -1,10 +1,5 @@
 FROM mcr.microsoft.com/devcontainers/base:0-alpine-3.16

-COPY host-ca-certificates.crt /tmp/host-ca-certificates.crt
-RUN cat /tmp/host-ca-certificates.crt >> /etc/ssl/certs/ca-certificates.crt
-RUN csplit -f /usr/local/share/ca-certificates/host-ca-certificate- -b '%02d.pem' -z -s /tmp/host-ca-certificates.crt '/-----BEGIN CERTIFICATE-----/' '{*}'
-ENV NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt
-
 # Avoiding OpenSSH >8.8 for compatibility for now: https://github.com/microsoft/vscode-remote-release/issues/7482
 RUN echo "@old https://dl-cdn.alpinelinux.org/alpine/v3.15/main" >> /etc/apk/repositories

@@ -21,6 +16,11 @@
 	openssh-client-default@old \
 	;

+COPY host-ca-certificates.crt /tmp/host-ca-certificates.crt
+RUN cat /tmp/host-ca-certificates.crt >> /etc/ssl/certs/ca-certificates.crt
+RUN csplit -f /usr/local/share/ca-certificates/host-ca-certificate- -b '%02d.pem' -z -s /tmp/host-ca-certificates.crt '/-----BEGIN CERTIFICATE-----/' '{*}'
+ENV NODE_EXTRA_CA_CERTS=/etc/ssl/certs/ca-certificates.crt
+
 RUN npm config set cafile /etc/ssl/certs/ca-certificates.crt && cd && npm i node-pty || echo "Continuing without node-pty."

 COPY .vscode-remote-containers /root/.vscode-remote-containers

I'm not sure where this certs file came from and why it's needed. But it seems broken on my environment.

[chrmarti]

We read the OS certificates with /usr/bin/security find-certificate -a -p on macOS and add these to the container to support corporate environments with custom certificates. Not sure why these certificates break access to dl-cdn.alpinelinux.org for you.

[tk0miya]

Thank you for the hint.

As a trial, I removed one of the certificate entries from /etc/ssl/certs/ca-certificates.crt. Then "apk add" works fine.
The certificate I removed is issued by Apple (CN=com.apple.kerberos.kdc).

$ openssl x509 -text -noout -in out.crt
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1982988364 (0x7632004c)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=com.apple.kerberos.kdc, O=System Identity
        Validity
            Not Before: Jun 12 13:35:19 2020 GMT
            Not After : Jun  7 13:35:19 2040 GMT
        Subject: CN=com.apple.kerberos.kdc, O=System Identity
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:b2:8e:09:08:50:76:16:59:62:b4:16:b3:cf:49:
                    06:1f:fa:94:83:f4:f2:e2:25:6f:5d:e3:7c:18:04:
                    56:03:b9:e5:a1:b4:11:2f:38:1c:64:2d:84:e1:2b:
                    11:b5:e8:4b:73:27:73:6f:42:a7:ac:47:6c:ea:68:
                    db:98:07:d5:40:01:c7:8b:2b:20:7a:6f:9b:11:eb:
                    06:54:bd:22:d0:bd:85:63:c5:e6:8a:9d:78:24:bf:
                    88:77:8e:1a:d4:7a:6e:f3:82:e8:47:d5:3c:2b:9a:
                    87:de:ba:25:5d:09:71:fa:16:37:e6:1f:d3:67:c5:
                    a4:19:a9:4c:28:0b:7b:83:70:1b:63:6c:cd:4b:a2:
                    2f:5f:0a:db:64:c4:32:c1:82:0e:f1:48:60:6f:73:
                    b0:37:59:cc:8f:5c:18:f6:76:c0:ec:f9:b9:b8:11:
                    c7:22:f3:c4:b3:00:ec:c3:b7:b4:10:e0:57:ca:00:
                    dd:8b:e2:4c:05:d7:75:8b:dd:c0:32:77:2b:03:95:
                    fe:0d:5e:34:2d:31:63:4a:69:1d:40:88:ef:b7:b8:
                    ae:0e:7c:8d:e4:72:e5:3d:95:b6:df:bc:9e:8a:d9:
                    c8:ec:ef:6e:db:ef:21:38:a6:0f:93:90:e7:62:62:
                    48:6c:f0:8a:eb:02:e6:57:db:05:c1:37:2a:8d:8c:
                    70:b1
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Subject Alternative Name:
                DNS:com.apple.kerberos.kdc
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, Signing KDC Response
    Signature Algorithm: sha256WithRSAEncryption
    Signature Value:
        93:ce:68:37:bb:dc:4b:3d:2c:96:64:04:ca:a0:5f:7a:65:63:
        e6:1c:d7:4d:7d:13:61:d4:69:4a:23:ec:30:b1:53:fa:6b:4e:
        96:b4:2e:6f:fe:87:b9:80:d2:de:f9:2c:cc:17:0f:4a:c0:ff:
        ac:f5:ff:f0:d6:53:e9:62:ab:cb:ba:bf:80:b0:a9:c4:f1:de:
        d3:01:e0:70:d3:53:17:ab:33:2b:03:37:47:b5:27:dd:09:3f:
        76:e4:4e:13:57:56:34:78:b0:54:97:e5:57:50:c9:be:2f:05:
        ff:53:c0:23:13:e4:d7:ab:26:46:e0:84:a9:ae:3a:ac:48:74:
        80:3e:a1:9a:dc:7c:a6:43:11:9d:43:06:4c:d4:73:f4:d6:9c:
        8a:7b:28:d0:ab:2f:58:3b:33:a2:bf:16:ac:f0:6e:f5:61:69:
        9d:4b:0b:0a:29:b8:bd:75:16:23:da:58:4b:b1:fc:94:6c:a4:
        2f:ad:bf:26:cb:2b:fa:57:10:a3:ee:35:1c:43:28:d5:8a:f8:
        bd:82:fa:e9:45:60:db:ce:04:31:fa:c3:f5:a0:69:02:e9:9f:
        b6:40:36:41:fd:34:3b:36:97:dc:9a:22:d6:39:5f:ea:ff:a1:
        55:2f:16:e6:9c:93:b7:01:0e:da:8f:25:e5:29:48:e1:ed:8b:
        16:ca:ee:89

$ openssl verify out.crt
CN=com.apple.kerberos.kdc, O=System Identity
error 18 at 0 depth lookup: self-signed certificate
error out.crt: verification failed

[chrmarti]

That shouldn't affect verification of the certificate for dl-cdn.alpinelinux.org. What are the certificates in ca-certificates.crt right before and right after this certificate? Does removing some other certificate instead of this also make it work or did you try one after the other to find this one? What is the size of ca-certificates.crt?

[tk0miya]

What are the certificates in ca-certificates.crt right before and right after this certificate?

This is the last entry of my ca-certificates.crt. The before entry is CN=com.apple.systemdefault.

Does removing some other certificate instead of this also make it work or did you try one after the other to find this one?

Surprisingly, it also works fine if I remove another one instead (ex. second from last, 3rd from last, ...)

What is the size of ca-certificates.crt?

root ➜ / $ ls -l /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 229217 Jul 25 16:48 /etc/ssl/certs/ca-certificates.crt

[chrmarti]

Does running update-ca-certificates show any relevant output? Does it also make it work (without removing any certificate)?

[tk0miya]

Does running update-ca-certificates show any relevant output?

I got a warning. But I'm not sure this is related or not.

root ➜ / $ update-ca-certificates
WARNING: Skipping duplicate certificate in file ca-cert-ISRG_Root_X1.pem

Does it also make it work (without removing any certificate)?

Unfortunately, no. The TLS error still happens.

BTW, there are 5 certificates copied from the host to /usr/local/share/ca-certificates on my PC.
Two are for root CA, one was installed for tech conf (unused now), and two are for Apple/macOS.

root ➜ /usr/local/share/ca-certificates $ ls
host-ca-certificate-00.pem  host-ca-certificate-01.pem	host-ca-certificate-02.pem  host-ca-certificate-03.pem	host-ca-certificate-04.pem
root ➜ /usr/local/share/ca-certificates $ openssl x509 -text -noout -in host-ca-certificate-00.pem  | grep CN
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
root ➜ /usr/local/share/ca-certificates $ openssl x509 -text -noout -in host-ca-certificate-01.pem  | grep CN
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Subject: C = US, O = Let's Encrypt, CN = R3
root ➜ /usr/local/share/ca-certificates $ openssl x509 -text -noout -in host-ca-certificate-02.pem  | grep CN
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
root ➜ /usr/local/share/ca-certificates $ openssl x509 -text -noout -in host-ca-certificate-03.pem  | grep CN
        Issuer: CN = com.apple.systemdefault, O = System Identity
        Subject: CN = com.apple.systemdefault, O = System Identity
root ➜ /usr/local/share/ca-certificates $ openssl x509 -text -noout -in host-ca-certificate-04.pem  | grep CN
        Issuer: CN = com.apple.kerberos.kdc, O = System Identity
        Subject: CN = com.apple.kerberos.kdc, O = System Identity

In my case, removing the unused cert resolves the error. But I feel the true problem is not resolved yet.
Is it still needed to investigate my error for other people?