Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

set annotations when creating sanbodx and containers #369

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

cncal
Copy link
Contributor

@cncal cncal commented May 27, 2024

Proposed Changes

Docker now allows client to set annotaion when creating sandbox and containers that is significant for some low-level runtimes(e.g. kata-containers) as they use it to determine whether the creating is a pod_sandbox or a pod_container.

Docker now allows client to set annotaion when creating sandbox and containers
that is significant for some low-level runtimes(e.g. kata-containers) as
they use it to determine whether the creating is a pod_sandbox or a pod_container.

Signed-off-by: cncal <[email protected]>
@cncal
Copy link
Contributor Author

cncal commented May 27, 2024

@corhere @nwneisen PTAL.

Comment on lines +205 to +206
// TODO: Currently docker ContainerList api doesn't return annotations, so annotations
// still need to be extracted from labels here.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not super comfortable with having different sources of truth for annotations depending on the CRI API endpoint queried. It would be straightforward to extend the ContainerList API to return container annotations; perhaps we should do that first so support can be added properly to cri-dockerd?

Copy link
Contributor Author

@cncal cncal May 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually I'm working on it moby/moby#47866. May I mark this PR a WIP util docker-side supports it ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moby/moby#47866 has been merged into master branch, but released in Pre-release https://github.com/moby/moby/releases/tag/v27.0.0-rc.1. May I bump github.com/docker/docker from 26.1.2+incompatible to v27.0.0-rc.1+incompatible? @corhere

Comment on lines +50 to +51
// TODO: Docker supports annotations for now, so labels and annotations might be separated
// in the future. Keeping them merged is just for backward compatibility.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Backward compatibility with what? AIUI no containers will be persisted across an upgrade (daemon restart) so there is no need for containers started by cri-dockerd v0.3.x to be understandable by v0.4.x, or vice versa.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There are two main considerations for backward compatibility:

  • I‘m not sure if some runtimes use the label to make some judgments
  • Docker ContainerList api doesn't return annotations, so keeping them merged is needed for later extraction.

Copy link
Contributor

@AkihiroSuda AkihiroSuda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems vulnerable to container breakout attacks via systemd properties

Same as GHSA-2cgq-h8xw-2v5j

containerd/CRI needs an explicit allow list for specifying OCI annotations
https://github.com/containerd/containerd/blob/v1.7.17/docs/cri/config.md?plain=1#L310

@cncal
Copy link
Contributor Author

cncal commented May 28, 2024

@AkihiroSuda Yes, I noticed that, but docker does not support setting up pod_annotations or container_annotations for runtime. Also, filetering annotations should be supported at docker side or cri-dockerd?

@AkihiroSuda
Copy link
Contributor

@AkihiroSuda Yes, I noticed that, but docker does not support setting up pod_annotations or container_annotations for runtime.

So, cri-dockerd will have to implement the allow list, or at least, check potentiallyUnsafeConfigAnnotations https://github.com/opencontainers/runc/blob/v1.2.0-rc.1/features.go#L67-L71

Also, filetering annotations should be supported at docker side or cri-dockerd?

cri-dockerd side.

@cncal
Copy link
Contributor Author

cncal commented May 28, 2024

How about adding potentiallyUnsafeConfigAnnotations as a cri-dockerd flag used to filter annotations from k8s ?

@AkihiroSuda
Copy link
Contributor

AkihiroSuda commented May 28, 2024

How about adding potentiallyUnsafeConfigAnnotations as a cri-dockerd flag used to filter annotations from k8s ?

Yes, cri-dockerd may run runc features and check the potentiallyUnsafeConfigAnnotations field to filter unsafe annotations that are passed via Kubernetes annotations.
This needs runc v1.2, which is still RC, though.

@cncal
Copy link
Contributor Author

cncal commented May 28, 2024

Yes, cri-dockerd may run runc features and check the potentiallyUnsafeConfigAnnotations field to filter unsafe annotations that are passed via Kubernetes annotations.

Sorry, I don't get it. You mean we get the RuntimeHandler(e.g. runc) from k8s request at first and run $runtime features command to fetch the potentiallyUnsafeConfigAnnotations value? What if RuntimeHandler is user-defined(may be runc with some special options named runc-1) ? As far as know, kata and gvisor do not have features subcommand for now.

@AkihiroSuda
Copy link
Contributor

Yes, cri-dockerd may run runc features and check the potentiallyUnsafeConfigAnnotations field to filter unsafe annotations that are passed via Kubernetes annotations.

Sorry, I don't get it. You mean we get the RuntimeHandler(e.g. runc) from k8s request at first and run $runtime features command to fetch the potentiallyUnsafeConfigAnnotations value?

Right. $runtime features can be invoked before handling Kubernetes requests though.

What if RuntimeHandler is user-defined(may be runc with some special options named runc-1) ? As far as know, kata and gvisor do not have features subcommand for now.

Every annotation should be assumed unsafe, and should not be propagated in this case.

@cncal
Copy link
Contributor Author

cncal commented May 28, 2024

Yes, cri-dockerd may run runc features and check the potentiallyUnsafeConfigAnnotations field to filter unsafe annotations that are passed via Kubernetes annotations.

Sorry, I don't get it. You mean we get the RuntimeHandler(e.g. runc) from k8s request at first and run $runtime features command to fetch the potentiallyUnsafeConfigAnnotations value?

Right. $runtime features can be invoked before handling Kubernetes requests though.

What if RuntimeHandler is user-defined(may be runc with some special options named runc-1) ? As far as know, kata and gvisor do not have features subcommand for now.

Every annotation should be assumed unsafe, and should not be propagated in this case.

@corhere @nwneisen Any suggestions?

@corhere
Copy link
Collaborator

corhere commented Jun 6, 2024

Runtime features are surfaced in the Info API, keyed by the runtime name. moby/moby#46647

@cncal
Copy link
Contributor Author

cncal commented Jun 7, 2024

Runtime features are surfaced in the Info API, keyed by the runtime name. moby/moby#46647

Thanks @corhere , it sounds good.

@cncal cncal marked this pull request as draft June 9, 2024 08:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants