Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add containerd-wasm-shims to enable Wasm / WASI workloads #1037

Merged
merged 1 commit into from
Dec 22, 2022
Merged

add containerd-wasm-shims to enable Wasm / WASI workloads #1037

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions images/capi/Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,7 @@ endif
COMMON_NODE_VAR_FILES := packer/config/kubernetes.json \
packer/config/cni.json \
packer/config/containerd.json \
packer/config/wasm-shims.json \
packer/config/ansible-args.json \
packer/config/goss-args.json \
packer/config/common.json \
Expand Down
35 changes: 35 additions & 0 deletions images/capi/ansible/roles/containerd/tasks/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,14 @@
dest: /tmp/containerd.tar.gz
mode: 0600

- name: download containerd-wasm-shims
devigned marked this conversation as resolved.
Show resolved Hide resolved
get_url:
url: "{{ containerd_wasm_shims_url }}"
checksum: "sha256:{{ containerd_wasm_shims_sha256 }}"
dest: /tmp/containerd_wasm_shims.tar.gz
mode: 0600
when: containerd_wasm_shims_runtimes | length > 0

- name: Create a directory if it does not exist
file:
path: "{{ sysusr_prefix }}/bin"
Expand All @@ -47,6 +55,17 @@
- --no-overwrite-dir
when: ansible_os_family != "Flatcar"

# install containerd Wasm shims when the runtimes are not empty -- current known runtimes are 'slight' and 'spin'
# see: https://github.com/kubernetes-sigs/image-builder/pull/1037
- name: unpack containerd-wasm-shims
unarchive:
remote_src: True
src: /tmp/containerd_wasm_shims.tar.gz
dest: "{{ sysusr_prefix }}/bin"
extra_opts:
- --no-overwrite-dir
when: ansible_os_family != "Flatcar" and (containerd_wasm_shims_runtimes | length > 0)

- name: unpack containerd for Flatcar to /opt/bin
unarchive:
remote_src: True
Expand All @@ -62,6 +81,17 @@
- 's@opt/local@opt@'
when: ansible_os_family == "Flatcar"

# install containerd Wasm shims when the runtimes are not empty -- current known runtimes are 'slight' and 'spin'
# see: https://github.com/kubernetes-sigs/image-builder/pull/1037
- name: unpack containerd-wasm-shims for Flatcar to /opt/bin
unarchive:
remote_src: True
src: /tmp/containerd_wasm_shims.tar.gz
dest: "{{ sysusr_prefix }}/bin"
extra_opts:
- --no-overwrite-dir
when: ansible_os_family == "Flatcar" and (containerd_wasm_shims_runtimes | length > 0)

# Remove /opt/cni directory, as we will install cni later
- name: delete /opt/cni directory
file:
Expand Down Expand Up @@ -132,3 +162,8 @@
file:
path: /tmp/containerd.tar.gz
state: absent

- name: delete tarball
file:
path: /tmp/containerd_wasm_shims.tar.gz
state: absent
8 changes: 8 additions & 0 deletions images/capi/ansible/roles/containerd/templates/etc/containerd/config.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,14 @@ imports = ["/etc/containerd/conf.d/*.toml"]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
{% if 'spin' in containerd_wasm_shims_runtimes %}
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.spin]
runtime_type = "io.containerd.spin.v1"
{% endif %}
{% if 'slight' in containerd_wasm_shims_runtimes %}
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.slight]
runtime_type = "io.containerd.slight.v1"
{% endif %}
{% endif %}
{% if packer_builder_type.startswith('azure') %}
[plugins."io.containerd.grpc.v1.cri".registry.headers]
Expand Down
1 change: 1 addition & 0 deletions images/capi/packer/azure/azure-config.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
"azure_location": "{{env `AZURE_LOCATION`}}",
"client_id": "{{env `AZURE_CLIENT_ID`}}",
"client_secret": "{{env `AZURE_CLIENT_SECRET`}}",
"containerd_wasm_shims_runtimes": "spin,slight",
"subscription_id": "{{env `AZURE_SUBSCRIPTION_ID`}}",
"vm_size": "Standard_B2ms"
}
2 changes: 2 additions & 0 deletions images/capi/packer/azure/packer.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -181,6 +181,7 @@
"OS": "{{user `distribution` | lower}}",
"PROVIDER": "azure",
"containerd_version": "{{user `containerd_version`}}",
"containerd_wasm_shims_runtimes": "{{user `containerd_wasm_shims_runtimes` }}",
"kubernetes_cni_deb_version": "{{ user `kubernetes_cni_deb_version` }}",
"kubernetes_cni_rpm_version": "{{ split (user `kubernetes_cni_rpm_version`) \"-\" 0 }}",
"kubernetes_cni_source_type": "{{user `kubernetes_cni_source_type`}}",
Expand Down Expand Up @@ -219,6 +220,7 @@
"containerd_sha256": null,
"containerd_url": "https://github.com/containerd/containerd/releases/download/v{{user `containerd_version`}}/cri-containerd-cni-{{user `containerd_version`}}-linux-amd64.tar.gz",
"containerd_version": null,
"containerd_wasm_shims_runtimes": null,
"crictl_url": "https://github.com/kubernetes-sigs/cri-tools/releases/download/v{{user `crictl_version`}}/crictl-v{{user `crictl_version`}}-linux-amd64.tar.gz",
"crictl_version": null,
"direct_shared_gallery_image_id": "",
Expand Down
2 changes: 1 addition & 1 deletion images/capi/packer/config/ansible-args.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"ansible_common_ssh_args": "-o IdentitiesOnly=yes -o HostKeyAlgorithms=+ssh-rsa -o PubkeyAcceptedKeyTypes=+ssh-rsa",
"ansible_common_vars": "containerd_url={{user `containerd_url`}} containerd_sha256={{user `containerd_sha256`}} pause_image={{user `pause_image`}} containerd_additional_settings={{user `containerd_additional_settings`}} containerd_cri_socket={{user `containerd_cri_socket`}} containerd_version={{user `containerd_version`}} crictl_url={{user `crictl_url`}} crictl_sha256={{user `crictl_sha256`}} crictl_source_type={{user `crictl_source_type`}} custom_role_names=\"{{user `custom_role_names`}}\" firstboot_custom_roles_pre=\"{{user `firstboot_custom_roles_pre`}}\" firstboot_custom_roles_post=\"{{user `firstboot_custom_roles_post`}}\" node_custom_roles_pre=\"{{user `node_custom_roles_pre`}}\" node_custom_roles_post=\"{{user `node_custom_roles_post`}}\" disable_public_repos={{user `disable_public_repos`}} extra_debs=\"{{user `extra_debs`}}\" extra_repos=\"{{user `extra_repos`}}\" extra_rpms=\"{{user `extra_rpms`}}\" http_proxy={{user `http_proxy`}} https_proxy={{user `https_proxy`}} kubeadm_template={{user `kubeadm_template`}} kubernetes_cni_http_source={{user `kubernetes_cni_http_source`}} kubernetes_cni_http_checksum={{user `kubernetes_cni_http_checksum`}} kubernetes_http_source={{user `kubernetes_http_source`}} kubernetes_container_registry={{user `kubernetes_container_registry`}} kubernetes_rpm_repo={{user `kubernetes_rpm_repo`}} kubernetes_rpm_gpg_key={{user `kubernetes_rpm_gpg_key`}} kubernetes_rpm_gpg_check={{user `kubernetes_rpm_gpg_check`}} kubernetes_deb_repo={{user `kubernetes_deb_repo`}} kubernetes_deb_gpg_key={{user `kubernetes_deb_gpg_key`}} kubernetes_cni_deb_version={{user `kubernetes_cni_deb_version`}} kubernetes_cni_rpm_version={{user `kubernetes_cni_rpm_version`}} kubernetes_cni_semver={{user `kubernetes_cni_semver`}} kubernetes_cni_source_type={{user `kubernetes_cni_source_type`}} kubernetes_semver={{user `kubernetes_semver`}} kubernetes_source_type={{user `kubernetes_source_type`}} kubernetes_load_additional_imgs={{user `kubernetes_load_additional_imgs`}} kubernetes_deb_version={{user `kubernetes_deb_version`}} kubernetes_rpm_version={{user `kubernetes_rpm_version`}} no_proxy={{user `no_proxy`}} pip_conf_file={{user `pip_conf_file`}} python_path={{user `python_path`}} redhat_epel_rpm={{user `redhat_epel_rpm`}} epel_rpm_gpg_key={{user `epel_rpm_gpg_key`}} reenable_public_repos={{user `reenable_public_repos`}} remove_extra_repos={{user `remove_extra_repos`}} systemd_prefix={{user `systemd_prefix`}} sysusr_prefix={{user `sysusr_prefix`}} sysusrlocal_prefix={{user `sysusrlocal_prefix`}} load_additional_components={{ user `load_additional_components`}} additional_registry_images={{ user `additional_registry_images`}} additional_registry_images_list={{ user `additional_registry_images_list`}} additional_url_images={{ user `additional_url_images`}} additional_url_images_list={{ user `additional_url_images_list`}} additional_executables={{ user `additional_executables`}} additional_executables_list={{ user `additional_executables_list`}} additional_executables_destination_path={{ user `additional_executables_destination_path`}} build_target={{ user `build_target`}} amazon_ssm_agent_rpm={{ user `amazon_ssm_agent_rpm` }}",
"ansible_common_vars": "containerd_url={{user `containerd_url`}} containerd_sha256={{user `containerd_sha256`}} pause_image={{user `pause_image`}} containerd_additional_settings={{user `containerd_additional_settings`}} containerd_cri_socket={{user `containerd_cri_socket`}} containerd_version={{user `containerd_version`}} containerd_wasm_shims_url={{user `containerd_wasm_shims_url`}} containerd_wasm_shims_version={{user `containerd_wasm_shims_version`}} containerd_wasm_shims_sha256={{user `containerd_wasm_shims_sha256`}} containerd_wasm_shims_runtimes=\"{{user `containerd_wasm_shims_runtimes`}}\" crictl_url={{user `crictl_url`}} crictl_sha256={{user `crictl_sha256`}} crictl_source_type={{user `crictl_source_type`}} custom_role_names=\"{{user `custom_role_names`}}\" firstboot_custom_roles_pre=\"{{user `firstboot_custom_roles_pre`}}\" firstboot_custom_roles_post=\"{{user `firstboot_custom_roles_post`}}\" node_custom_roles_pre=\"{{user `node_custom_roles_pre`}}\" node_custom_roles_post=\"{{user `node_custom_roles_post`}}\" disable_public_repos={{user `disable_public_repos`}} extra_debs=\"{{user `extra_debs`}}\" extra_repos=\"{{user `extra_repos`}}\" extra_rpms=\"{{user `extra_rpms`}}\" http_proxy={{user `http_proxy`}} https_proxy={{user `https_proxy`}} kubeadm_template={{user `kubeadm_template`}} kubernetes_cni_http_source={{user `kubernetes_cni_http_source`}} kubernetes_cni_http_checksum={{user `kubernetes_cni_http_checksum`}} kubernetes_http_source={{user `kubernetes_http_source`}} kubernetes_container_registry={{user `kubernetes_container_registry`}} kubernetes_rpm_repo={{user `kubernetes_rpm_repo`}} kubernetes_rpm_gpg_key={{user `kubernetes_rpm_gpg_key`}} kubernetes_rpm_gpg_check={{user `kubernetes_rpm_gpg_check`}} kubernetes_deb_repo={{user `kubernetes_deb_repo`}} kubernetes_deb_gpg_key={{user `kubernetes_deb_gpg_key`}} kubernetes_cni_deb_version={{user `kubernetes_cni_deb_version`}} kubernetes_cni_rpm_version={{user `kubernetes_cni_rpm_version`}} kubernetes_cni_semver={{user `kubernetes_cni_semver`}} kubernetes_cni_source_type={{user `kubernetes_cni_source_type`}} kubernetes_semver={{user `kubernetes_semver`}} kubernetes_source_type={{user `kubernetes_source_type`}} kubernetes_load_additional_imgs={{user `kubernetes_load_additional_imgs`}} kubernetes_deb_version={{user `kubernetes_deb_version`}} kubernetes_rpm_version={{user `kubernetes_rpm_version`}} no_proxy={{user `no_proxy`}} pip_conf_file={{user `pip_conf_file`}} python_path={{user `python_path`}} redhat_epel_rpm={{user `redhat_epel_rpm`}} epel_rpm_gpg_key={{user `epel_rpm_gpg_key`}} reenable_public_repos={{user `reenable_public_repos`}} remove_extra_repos={{user `remove_extra_repos`}} systemd_prefix={{user `systemd_prefix`}} sysusr_prefix={{user `sysusr_prefix`}} sysusrlocal_prefix={{user `sysusrlocal_prefix`}} load_additional_components={{ user `load_additional_components`}} additional_registry_images={{ user `additional_registry_images`}} additional_registry_images_list={{ user `additional_registry_images_list`}} additional_url_images={{ user `additional_url_images`}} additional_url_images_list={{ user `additional_url_images_list`}} additional_executables={{ user `additional_executables`}} additional_executables_list={{ user `additional_executables_list`}} additional_executables_destination_path={{ user `additional_executables_destination_path`}} build_target={{ user `build_target`}} amazon_ssm_agent_rpm={{ user `amazon_ssm_agent_rpm` }}",
"ansible_scp_extra_args": "{{env `ANSIBLE_SCP_EXTRA_ARGS`}}"
}
6 changes: 6 additions & 0 deletions images/capi/packer/config/wasm-shims.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"containerd_wasm_shims_runtimes": "",
"containerd_wasm_shims_sha256": "da84b1c065a58f95a841d39e143cd7115d43e6faedcce7a8782f2942388260d7",
"containerd_wasm_shims_url": "https://github.com/deislabs/containerd-wasm-shims/releases/download/{{user `containerd_wasm_shims_version`}}/containerd-wasm-shims-v1-linux-x86_64.tar.gz",
"containerd_wasm_shims_version": "v0.3.3"
}
17 changes: 17 additions & 0 deletions images/capi/packer/goss/goss-command.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,23 @@ command:
stdout: []
stderr: []
timeout: 0
{{if ne .Vars.containerd_wasm_shims_runtimes ""}}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It isn't garmented both runtime would be installed for this check so this could fail if only one is installed. I am not sure this is a big deal at this point for the test.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's unfortunate that the containerd wasm shims are packaged together into one release artifact. When you download one, you get both.

I think in the future, I'd like to change the shims to be packaged individually and change the variables / tests in image-builder to allow for individual installation and verification.

I've also opened an issue to add --version support for the shims. I didn't like the error test. The test does the job of verifying the shim is installed, on the path, and responding in a reasonable way, but I would much rather see something closer to the containerd version verification, so we can ensure we have the specific version installed that we expect.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think in the future, I'd like to change the shims to be packaged individually and change the variables / tests in image-builder to allow for individual installation and verification.

I think it's acceptable to assume both shims for now, and to fix this when their packaging changes.

containerd-shim-slight-v1:
exit-status: 1
stdout: [ ]
stderr: ["io.containerd.slight.v1: InvalidArgument(\"Shim namespace cannot be empty\")"]
timeout: 0
containerd-shim-spin-v1:
exit-status: 1
stdout: [ ]
stderr: ["io.containerd.spin.v1: InvalidArgument(\"Shim namespace cannot be empty\")"]
timeout: 0
grep -E 'io\.containerd\.(slight|spin)\.v1' /etc/containerd/config.toml:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: I think this wouldn't verify both are there are present only one of them (similar to comment above)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are 100% correct. I was having a hard time dealing with .Vars.containerd_wasm_shims_runtime as it's a comma delimited string ("slight, spin"). It's much easier to deal with it in Python in Ansible using a string contains.

Do you have a suggestion of how to handle it better in Goss?

exit-status: 0
stdout: [ ]
stderr: [ ]
timeout: 0
{{end}}
{{if eq .Vars.kubernetes_source_type "pkg"}}
{{if eq .Vars.kubernetes_cni_source_type "pkg"}}
crictl images | grep -v 'IMAGE ID' | awk -F'[ /]' '{print $2}' | sed 's/-{{ .Vars.arch }}//g' | sort:
Expand Down
1 change: 1 addition & 0 deletions images/capi/packer/goss/goss-vars.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ common_photon_rpms: &common_photon_rpms

arch: "amd64"
containerd_version: ""
containerd_wasm_shims_runtimes: ""
kubernetes_cni_source_type: ""
kubernetes_cni_version: ""
kubernetes_source_type: ""
Expand Down