Fetch pause container image from ECR before starting `kubelet` #382

etungsten · 2019-10-08T22:57:30Z

Issue #, if available: Fixes #351

Description of changes:
Adds new sandbox_image setting to set the pause container image for containerd/cri plugin.
Converts /etc/containerd/config.toml to a template
Updated pluto to accommodate new naming changes with pause container setting.

Updated host-ctr with new command line options:

-pull-image-only to just pull and unpack the specified image at -source
-containerd-sock to specify the containerd socket
-namespace to specify the containerd namespace.

Adds new oneshot service called pause-ctr-image-fetcher to pull the pause container image from ECR in k8s.io namespace.

Testing:
Launched Thar instance and the sandbox_image option is set successfully:

bash-5.0# cat /etc/containerd/config.toml        
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
disabled_plugins = [
    "io.containerd.snapshotter.v1.aufs",
    "io.containerd.snapshotter.v1.zfs",
    "io.containerd.snapshotter.v1.devmapper",
]

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri"]
# Pause container image is specified here, shares the same image as kubelet's pod-infra-container-image
sandbox_image = "602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/pause-amd64:3.1"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

[plugins."io.containerd.internal.v1.opt"]
path = "/opt/containerd"

pause-ctr-image-fetcher runs successfully:

bash-5.0# systemctl status pause-ctr-image-fetcher
● pause-ctr-image-fetcher.service - Fetches pause container image from ECR before kubelet starts
   Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/pause-ctr-image-fetcher.service; enabled; vendor preset: enabled)
   Active: active (exited) since Tue 2019-10-15 00:22:07 UTC; 6min ago
  Process: 2654 ExecStart=/usr/bin/host-ctr -source ${POD_INFRA_CONTAINER_IMAGE} -pull-image-only -containerd-socket /run/containerd/containerd.sock -namespac
e k8s.io (code=exited, status=0/SUCCESS)
 Main PID: 2654 (code=exited, status=0/SUCCESS)

Oct 15 00:22:07 ip-192-168-40-77.us-west-2.compute.internal systemd[1]: Starting Fetches pause container image from ECR before kubelet starts...
Oct 15 00:22:07 ip-192-168-40-77.us-west-2.compute.internal host-ctr[2654]: time="2019-10-15T00:22:07Z" level=info msg="Pulling from Amazon ECR" ref="ecr.aws/
arn:aws:ecr:us-west-2:602401143452:repository/eks/pause-amd64:3.1"
Oct 15 00:22:07 ip-192-168-40-77.us-west-2.compute.internal host-ctr[2654]: time="2019-10-15T00:22:07Z" level=info msg="Pulled successfully" img="ecr.aws/arn:
aws:ecr:us-west-2:602401143452:repository/eks/pause-amd64:3.1"
Oct 15 00:22:07 ip-192-168-40-77.us-west-2.compute.internal host-ctr[2654]: time="2019-10-15T00:22:07Z" level=info msg=Unpacking... img="ecr.aws/arn:aws:ecr:u
s-west-2:602401143452:repository/eks/pause-amd64:3.1"
Oct 15 00:22:07 ip-192-168-40-77.us-west-2.compute.internal host-ctr[2654]: time="2019-10-15T00:22:07Z" level=info msg="Tagging image" image name="60240114345
2.dkr.ecr.us-west-2.amazonaws.com/eks/pause-amd64:3.1"
Oct 15 00:22:07 ip-192-168-40-77.us-west-2.compute.internal systemd[1]: Started Fetches pause container image from ECR before kubelet starts.

kubelet also starts successfully

bash-5.0# systemctl status kubelet
● kubelet.service - Kubelet
   Loaded: loaded (/x86_64-thar-linux-gnu/sys-root/usr/lib/systemd/system/kubelet.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-10-15 00:22:07 UTC; 5min ago
     Docs: https://github.com/kubernetes/kubernetes
  Process: 2787 ExecStartPre=/sbin/iptables -P FORWARD ACCEPT (code=exited, status=0/SUCCESS)
 Main PID: 2818 (kubelet)
    Tasks: 16
   Memory: 128.9M
      CPU: 4.261s
   CGroup: /system.slice/kubelet.service
           └─2818 /usr/bin/kubelet --cloud-provider aws --config /etc/kubernetes/kubelet/config --kubeconfig /etc/kubernetes/kubelet/kubeconfig --container-runt

And I see the worker node registered and pods are scheduled on it successfully:

$ kubectl describe node ip-192-168-52-231.us-west-2.compute.internal
Name:               ip-192-168-52-231.us-west-2.compute.internal
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/instance-type=c5.large
                    beta.kubernetes.io/os=linux
                    failure-domain.beta.kubernetes.io/region=us-west-2
                    failure-domain.beta.kubernetes.io/zone=us-west-2a
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=ip-192-168-52-231.us-west-2.compute.internal
                    kubernetes.io/os=linux
                    testLabel=foo
                    testLabel2=bar
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 16 Oct 2019 13:05:05 -0700
Taints:             dedicated=experimental:PreferNoSchedule
                    special=true:PreferNoSchedule
Unschedulable:      false
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Wed, 16 Oct 2019 13:08:35 -0700   Wed, 16 Oct 2019 13:05:04 -0700   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 16 Oct 2019 13:08:35 -0700   Wed, 16 Oct 2019 13:05:04 -0700   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 16 Oct 2019 13:08:35 -0700   Wed, 16 Oct 2019 13:05:04 -0700   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            True    Wed, 16 Oct 2019 13:08:35 -0700   Wed, 16 Oct 2019 13:05:25 -0700   KubeletReady                 kubelet is posting ready status
Addresses:
  InternalIP:   192.168.52.231
  ExternalIP:   52.13.23.132
  InternalDNS:  ip-192-168-52-231.us-west-2.compute.internal
  ExternalDNS:  ec2-52-13-23-132.us-west-2.compute.amazonaws.com
  Hostname:     ip-192-168-52-231.us-west-2.compute.internal
Capacity:
 attachable-volumes-aws-ebs:  25
 cpu:                         2
 ephemeral-storage:           20624592Ki
 hugepages-1Gi:               0
 hugepages-2Mi:               0
 memory:                      3792268Ki
 pods:                        110
Allocatable:
 attachable-volumes-aws-ebs:  25
 cpu:                         2
 ephemeral-storage:           19007623956
 hugepages-1Gi:               0
 hugepages-2Mi:               0
 memory:                      3689868Ki
 pods:                        110
System Info:
 Machine ID:                 ec2ae5856748d54644a0d989642e343b
 System UUID:                ec2ae585-6748-d546-44a0-d989642e343b
 Boot ID:                    34552b04-35df-4d5f-84b3-03869b381d15
 Kernel Version:             4.19.72
 OS Image:                   Thar, The Operating System
 Operating System:           linux
 Architecture:               amd64
 Container Runtime Version:  containerd://1.3.0+unknown
 Kubelet Version:            v1.14.6
 Kube-Proxy Version:         v1.14.6
ProviderID:                  aws:///us-west-2a/i-00ecf5a29e918e1bf
Non-terminated Pods:         (3 in total)
  Namespace                  Name                CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                  ----                ------------  ----------  ---------------  -------------  ---
  default                    fluentbit-6pn2c     500m (25%)    0 (0%)      100Mi (2%)       500Mi (13%)    3m37s
  kube-system                aws-node-9947p      10m (0%)      0 (0%)      0 (0%)           0 (0%)         3m57s
  kube-system                kube-proxy-8lljn    100m (5%)     0 (0%)      0 (0%)           0 (0%)         3m57s
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests    Limits
  --------                    --------    ------
  cpu                         610m (30%)  0 (0%)
  memory                      100Mi (2%)  500Mi (13%)
  ephemeral-storage           0 (0%)      0 (0%)
  attachable-volumes-aws-ebs  0           0
Events:
  Type     Reason                   Age                    From                                                      Message
  ----     ------                   ----                   ----                                                      -------
  Normal   Starting                 3m58s                  kubelet, ip-192-168-52-231.us-west-2.compute.internal     Starting kubelet.
  Warning  InvalidDiskCapacity      3m58s                  kubelet, ip-192-168-52-231.us-west-2.compute.internal     invalid capacity 0 on image filesystem
  Normal   NodeHasSufficientMemory  3m58s (x2 over 3m58s)  kubelet, ip-192-168-52-231.us-west-2.compute.internal     Node ip-192-168-52-231.us-west-2.compute.internal status is now: NodeHasSufficientMemory
  Normal   NodeHasNoDiskPressure    3m58s (x2 over 3m58s)  kubelet, ip-192-168-52-231.us-west-2.compute.internal     Node ip-192-168-52-231.us-west-2.compute.internal status is now: NodeHasNoDiskPressure
  Normal   NodeHasSufficientPID     3m58s (x2 over 3m58s)  kubelet, ip-192-168-52-231.us-west-2.compute.internal     Node ip-192-168-52-231.us-west-2.compute.internal status is now: NodeHasSufficientPID
  Normal   NodeAllocatableEnforced  3m58s                  kubelet, ip-192-168-52-231.us-west-2.compute.internal     Updated Node Allocatable limit across pods
  Normal   Starting                 3m48s                  kube-proxy, ip-192-168-52-231.us-west-2.compute.internal  Starting kube-proxy.
  Normal   NodeReady                3m37s                  kubelet, ip-192-168-52-231.us-west-2.compute.internal     Node ip-192-168-52-231.us-west-2.compute.internal status is now: NodeReady

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

etungsten · 2019-10-09T00:08:56Z

Updating PR to implement image fetch from ECR before kubelet starts so containerd is able to start pause containers without needing to auth to ECR (which it cannot do),

etungsten · 2019-10-11T00:18:34Z

Rebase develop

etungsten · 2019-10-14T18:13:47Z

Fixed the issue above, I didn't include the repository path in addition to the pause container image name when building the ECR resolvable reference.
Added a new unit test to cover cases with image sources that have additional repository path (e.g./eks/image:tag).

etungsten · 2019-10-15T00:24:49Z

Adds new namespace command line option for host-ctr to specify the containerd namespace.
After pulling the image, host-ctr will also tag the image with the original image name if it was converted into an ECR resolver image reference.

etungsten · 2019-10-15T01:13:45Z

Updates README with information about settings.kubernetes.sandbox-image

workspaces/api/storewolf/defaults.toml

workspaces/host-ctr/cmd/host-ctr/main.go

etungsten · 2019-10-15T21:53:55Z

1st force push: Rebase develop.
2nd force push: Addresses subset of @jahkeup 's comments.

jahkeup

Changes LGTM!

workspaces/api/storewolf/defaults.toml

etungsten · 2019-10-16T18:39:19Z

Ran go fmt and added comment per @jahkeup

packages/release/pause-ctr-image-fetcher.service

README.md

etungsten · 2019-10-16T20:13:32Z

Addresses @bcressey 's comments.

Removed pause-ctr-image-fetcher.service
Added ExecStartPre= directive to pull the pause container image in kubelet.service
Removed settings.containerd.* and now use kubernetes.pod-infra-container-image to set the sandbox_image option in containerd/config

etungsten · 2019-10-16T20:20:21Z

Rebase develop. Resolves conflicts in pluto/src/main.rs

etungsten · 2019-10-16T22:00:33Z

Rebase develop, fixes conflict from #420

bcressey

LGTM

workspaces/api/pluto/src/main.rs

Converts the containerd/config.toml to a template, and specifies `sandbox-image` option in containerd Renamed pod_infra_container_image to pause_container_image in `pluto`.

Adds new command line options for `host-ctr` to explicitly specify containerd socket, namespace, and optionally to only pull the container image `kubelet.service` will invoke host-ctr to fetch the pause container image before actually starting kubelet.

etungsten · 2019-10-16T22:49:11Z

Addresses @bcressey 's comment.

Testing again, things still work!

Fixes a bug introduced in #382 where `ref` isn't set when pulling images from non-ECR registries

etungsten requested review from tjkirch, bcressey and patraw October 8, 2019 22:57

etungsten changed the title ~~settings: replace pod-infra-container-image setting with new sandbox_image setting~~ WIP: Fetch pause container from ECR Oct 8, 2019

etungsten force-pushed the sandbox-image-setting branch from 935a70b to 7825d00 Compare October 9, 2019 00:07

etungsten force-pushed the sandbox-image-setting branch from 7825d00 to 536a315 Compare October 11, 2019 00:18

etungsten force-pushed the sandbox-image-setting branch from a5d1d49 to ef88499 Compare October 12, 2019 00:05

This comment has been minimized.

Sign in to view

etungsten force-pushed the sandbox-image-setting branch from ef88499 to 9bf6849 Compare October 14, 2019 18:08

etungsten force-pushed the sandbox-image-setting branch from 9bf6849 to 4fd7c40 Compare October 15, 2019 00:22

etungsten changed the title ~~WIP: Fetch pause container from ECR~~ Fetch pause container image from ECR before starting kubelet Oct 15, 2019

etungsten marked this pull request as ready for review October 15, 2019 00:33

etungsten requested a review from jahkeup October 15, 2019 00:35

etungsten force-pushed the sandbox-image-setting branch from 4fd7c40 to 5b5c658 Compare October 15, 2019 01:13

jahkeup suggested changes Oct 15, 2019

View reviewed changes

workspaces/api/storewolf/defaults.toml Outdated Show resolved Hide resolved

workspaces/host-ctr/cmd/host-ctr/main.go Show resolved Hide resolved

workspaces/host-ctr/cmd/host-ctr/main.go Show resolved Hide resolved

workspaces/host-ctr/cmd/host-ctr/main.go Outdated Show resolved Hide resolved

etungsten force-pushed the sandbox-image-setting branch 2 times, most recently from db763f4 to 8041fb6 Compare October 15, 2019 21:52

jahkeup approved these changes Oct 16, 2019

View reviewed changes

workspaces/api/storewolf/defaults.toml Outdated Show resolved Hide resolved

etungsten force-pushed the sandbox-image-setting branch from 8041fb6 to 10c3b20 Compare October 16, 2019 18:38

bcressey reviewed Oct 16, 2019

View reviewed changes

packages/release/pause-ctr-image-fetcher.service Outdated Show resolved Hide resolved

packages/release/pause-ctr-image-fetcher.service Outdated Show resolved Hide resolved

README.md Outdated Show resolved Hide resolved

etungsten force-pushed the sandbox-image-setting branch from 10c3b20 to 0fa9677 Compare October 16, 2019 20:10

etungsten force-pushed the sandbox-image-setting branch from 0fa9677 to 055f512 Compare October 16, 2019 20:15

etungsten force-pushed the sandbox-image-setting branch from 055f512 to 8da97e3 Compare October 16, 2019 20:19

etungsten force-pushed the sandbox-image-setting branch 2 times, most recently from 12677ce to 2ceacb9 Compare October 16, 2019 22:00

bcressey reviewed Oct 16, 2019

View reviewed changes

workspaces/api/pluto/src/main.rs Outdated Show resolved Hide resolved

etungsten added 2 commits October 16, 2019 15:28

settings: templatize containerd/config.toml, specify sandbox_image

f9784cb

Converts the containerd/config.toml to a template, and specifies `sandbox-image` option in containerd Renamed pod_infra_container_image to pause_container_image in `pluto`.

etungsten force-pushed the sandbox-image-setting branch from 2ceacb9 to 241ed42 Compare October 16, 2019 22:49

etungsten merged commit 2924141 into develop Oct 16, 2019

etungsten deleted the sandbox-image-setting branch October 16, 2019 23:48

etungsten mentioned this pull request Oct 17, 2019

host-ctr: Implement exponential backoff for image pulls #433

Merged

etungsten added a commit that referenced this pull request Oct 21, 2019

host-ctr: fix references for non-ECR images

1423f50

Fixes a bug introduced in #382 where `ref` isn't set when pulling images from non-ECR registries

etungsten mentioned this pull request Oct 21, 2019

host-ctr: fix references for non-ECR images #439

Merged

stmcginnis mentioned this pull request Jun 27, 2023

Investigate need for k8s prestart-pull-pause-ctr #3088

Open

rifelpet mentioned this pull request Aug 31, 2024

kubelet cannot pull images when using ECR containerProxy asset repository kubernetes/kops#16762

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fetch pause container image from ECR before starting `kubelet` #382

Fetch pause container image from ECR before starting `kubelet` #382

etungsten commented Oct 8, 2019 •

edited

Loading

etungsten commented Oct 9, 2019 •

edited

Loading

etungsten commented Oct 11, 2019

This comment has been minimized.

etungsten commented Oct 14, 2019

etungsten commented Oct 15, 2019

etungsten commented Oct 15, 2019

etungsten commented Oct 15, 2019

jahkeup left a comment

etungsten commented Oct 16, 2019

etungsten commented Oct 16, 2019

etungsten commented Oct 16, 2019

etungsten commented Oct 16, 2019

bcressey left a comment

etungsten commented Oct 16, 2019 •

edited

Loading

Fetch pause container image from ECR before starting kubelet #382

Fetch pause container image from ECR before starting kubelet #382

Conversation

etungsten commented Oct 8, 2019 • edited Loading

etungsten commented Oct 9, 2019 • edited Loading

etungsten commented Oct 11, 2019

This comment has been minimized.

etungsten commented Oct 14, 2019

etungsten commented Oct 15, 2019

etungsten commented Oct 15, 2019

etungsten commented Oct 15, 2019

jahkeup left a comment

Choose a reason for hiding this comment

etungsten commented Oct 16, 2019

etungsten commented Oct 16, 2019

etungsten commented Oct 16, 2019

etungsten commented Oct 16, 2019

bcressey left a comment

Choose a reason for hiding this comment

etungsten commented Oct 16, 2019 • edited Loading

Fetch pause container image from ECR before starting `kubelet` #382

Fetch pause container image from ECR before starting `kubelet` #382

etungsten commented Oct 8, 2019 •

edited

Loading

etungsten commented Oct 9, 2019 •

edited

Loading

etungsten commented Oct 16, 2019 •

edited

Loading