Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PoC: Implement optional host proxy for lima cache #2367

Draft
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

afbjorklund
Copy link
Member

@afbjorklund afbjorklund commented May 27, 2024

Implements both http_proxy and https_proxy (with mitm cert)

Does not actually cache anything (yet), just verbose INFO logging.

The use case is to use the regular $LIMA_CACHE also for apt/ctr.

But instead of doing custom mirrors, use a brute force http proxy.

# The host proxy implements a HTTP and HTTPS proxy that can cache downloads on the host.
hostProxy:
  # 🟢 Builtin default: false
  enabled: null

TODO: Add some more options, for configuring apt and containerd

The target is to be able to cache the packages and images for k8s


Tested with curl, and with ctr by using a custom certs.d/docker.io

server = "https://docker.io"

[host."https://registry-1.docker.io"]
  capabilities = ["pull", "resolve"]
  ca = "proxy.crt"

And similar also for https://registry.k8s.io (and https://pkgs.k8s.io)

https://github.com/containerd/nerdctl/blob/main/docs/registry.md

/etc/containerd

/etc/containerd/certs.d/_default/hosts.toml

/etc/apt/apt.conf.d/50proxy.conf

Acquire::https::CaInfo "/path/to/proxy.crt";


@afbjorklund
Copy link
Member Author

afbjorklund commented May 27, 2024

So far, so good.

~/.cache/lima/download/by-url-sha256/1edaca0c14499031c860d2986cc44975d0b5d68883ba871c244e283124526373/url:https://pkgs.k8s.io:443/core:/stable:/v1.30/deb/InRelease
~/.cache/lima/download/by-url-sha256/ec91009c025c3a3b9408f69f4c2e1f95325fdf26e8418aaccde1771548f328c1/url:https://pkgs.k8s.io:443/core:/stable:/v1.30/deb/Packages.gz
~/.cache/lima/download/by-url-sha256/95f784a6907b4370a39638b13ecaa83c048abb42df7dd398a0fffc5aede89980/url:https://pkgs.k8s.io:443/core:/stable:/v1.30/deb/amd64/cri-tools_1.30.0-1.1_amd64.deb
~/.cache/lima/download/by-url-sha256/30cf604c01d11ee9598bc1a891fb9df3633a588b57f5c814a81381b11c2186a4/url:https://registry.k8s.io:443/v2/pause/blobs/sha256:1bf148e3d3560190242f1be53d4d144f430e1d613b1cfa3d9cd38099fee06866
~/.cache/lima/download/by-url-sha256/ada724f6ef036b0d37900cd854cb4f2f10fe217ab1cafbc52d178dabab8e4c4d/url:https://registry.k8s.io:443/v2/pause/manifests/3.9

sudo http_proxy=http://host.lima.internal:47979 https_proxy=http://host.lima.internal:47979 apt update
sudo http_proxy=http://host.lima.internal:47979 https_proxy=http://host.lima.internal:47979 apt install cri-tools

kubeadm config images list
sudo http_proxy=http://host.lima.internal:47979 https_proxy=http://host.lima.internal:47979 nerdctl pull registry.k8s.io/pause:3.9

Must use --tlscacert with ctr call (not /etc/containerd), or must set environment variables when starting containerd.

Using nerdctl avoids these issue with ctr or crictl, so is probably the preferred way to go (since it is installed)

@afbjorklund
Copy link
Member Author

afbjorklund commented May 27, 2024

sudo apt-get install -y kubelet kubeadm kubectl

The following additional packages will be installed:
  conntrack ebtables kubernetes-cni socat
The following NEW packages will be installed:
  conntrack ebtables kubeadm kubectl kubelet kubernetes-cni socat
0 upgraded, 7 newly installed, 0 to remove and 10 not upgraded.
Need to get 72.7 MB of archives.
After this operation, 285 MB of additional disk space will be used.

kubeadm config images list | xargs -n 1 sudo nerdctl pull

REPOSITORY                                 TAG         IMAGE ID        CREATED           PLATFORM       SIZE         BLOB SIZE
registry.k8s.io/coredns/coredns            v1.11.1     1eeb4c7316ba    23 seconds ago    linux/amd64    60.9 MiB     17.3 MiB
registry.k8s.io/etcd                       3.5.12-0    44a8e24dcbba    5 seconds ago     linux/amd64    146.3 MiB    54.6 MiB
registry.k8s.io/kube-apiserver             v1.30.1     0d4a30512343    2 hours ago       linux/amd64    114.3 MiB    31.2 MiB
registry.k8s.io/kube-controller-manager    v1.30.1     0c34190fbf80    2 hours ago       linux/amd64    109.1 MiB    29.7 MiB
registry.k8s.io/kube-proxy                 v1.30.1     a1754e5a3387    2 hours ago       linux/amd64    84.4 MiB     27.7 MiB
registry.k8s.io/kube-scheduler             v1.30.1     74d02f6debc5    2 hours ago       linux/amd64    62.2 MiB     18.4 MiB
registry.k8s.io/pause                      3.9         7031c1b28338    16 seconds ago    linux/amd64    732.0 KiB    314.0 KiB

@afbjorklund
Copy link
Member Author

afbjorklund commented May 27, 2024

Needs better error handling, since those redirects seem to be temporary (missing the remap of apt-cacher-ng)

INFO: resp 307 Temporary Redirect
INFO: resp 404 Not Found

@afbjorklund afbjorklund changed the title PoC: Implement optional host proxy for cache PoC: Implement optional host proxy for lima cache Sep 28, 2024
@afbjorklund

This comment was marked as resolved.

Implements both http_proxy and https_proxy (with mitm cert)

Does not actually cache anything, just verbose INFO logging.

Signed-off-by: Anders F Björklund <[email protected]>
Signed-off-by: Anders F Björklund <[email protected]>
Signed-off-by: Anders F Björklund <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant