Spin App that uses WASI:sockets does not work

[kate-goldenring]

TCP tests were added to the conformance tests, which caught a gap in the shim: it cannot successfully run Spin apps that use TCP.

When running this Spin app, requests fail:

curl -i -H "Host: example.com" -H "address: 127.0.0.1:38735" http://127.0.0.1:80
HTTP/1.1 500 Internal Server Error
transfer-encoding: chunked
date: Tue, 25 Jun 2024 17:05:38 GMT

I've also tried running it with the --privileged flag:

sudo ctr run --rm --privileged --env SPIN_VARIABLE_PORT=38735 --net-host --runtime io.containerd.spin.v2 ttl.sh/tcp-spin-test-app-123:48h  myapp bogus-arg

The spin up equivalent runs and passes.

[squillace]

we're talking about this: https://github.com/WebAssembly/wasi-sockets/tree/main/wit yes?

[squillace]

the shim should not enable lower level protocols by default; it should be something you "turn on" when compiling or configuration. opening socket holes in the sandbox is a path to perdition.

That said, it's sometimes necessary!

[squillace]

In fact, I don't think on further thought that spinKUBE should expose everything spin itself can do! it should target the highest-level protocols and apis that radically constrain "most" of the supported component apis -- but no more. a centralized service should not merely expose OS calls by default -- that is definitely not the way to glory.

However, it should be possible to build the shim for yourself and enable what spin can do, definitely. That supply chain would look like

• git clone from commit
• modify build params, possibly pulling in other commits/repos that you are pinning
• build THAT to enable specifically and only what you want to add support for
• tag and sign that specific artifact
• deploy it with runtime-class-manager like any other shim
• profit????

But I would argue strongly against opening up networks and files IN the shim that aren't run through the normal k8s protocols like http.

Not sure that helps, but it's a very interesting area to keep cracking our heads on, as it's precisely the boundary between "more generalized usefulness" and "constrained abilities that make everything safer" that the wasm component world offers.

[squillace]

Thinking through it even more, this kind of thing is where the Fermyon Cloud CRD solution should shine, in reality. sockets means you definitely own the world, and if so, something like cyclotron is the way to go for that feature set. I would even think about it AS something that we would NOT enable in the k8s integration shim (as I argue above, I don't think it's a good thing to do generally) but it absolutely could be enabled for fermyon cloud-type solutions, or even other shims.

[squillace]

anyway, just take this all as food for thought!

[radu-matei]

For context, the Spin application explicitly asks for permission to access a specific network host, it's not implicitly granted -- in Kate's test — https://github.com/kate-goldenring/conformance-tests/blob/ce3d03198089b8fe54afcb69d1b93c4d0288db5a/components/tcp-sockets/spin.toml#L18

[radu-matei]

To your point, @squillace, this could be where something like executor level configuration could control whether you allow things like this, but today, if the application definition enables access to files or network hosts, the embedder should respect that.

[rylev]

I agree with @radu-matei. If we want to go down the road of not having the shim able to run applications which access resources like sockets (at least by default), then we need to do the work of surfacing that earlier in the application lifecycle (e.g., by erroring when the application is loaded). Right not, from a user's perspective, this very much feels like a bug and not a deliberate security/sandboxing decision.

[squillace]

NO to #147 (comment). The dev can do whatever they want, but that in no way means the host environment needs to respect that. This is the same lesson that components give us; you can implement wasi:sockets, but if they host doesn't support it, it can't work. This is merely the capabilities lesson even at the apparmor/seccomp level.

HOWEVER, if a host environment wants to support it, it should have a path to doing so. The question is what path should that be?

Per @rylev, what should spin's dev ux be when the dev has an opinion, but the host may have another? My take is that a shim ("host") might have a wit profile it can project? not sure. But yes, erroring immediately would at the very least be a good thing ("cannot find wasi:socket imports to use! Does the host support wasi:sockets?")

[squillace]

also, while I'm interested in the general principle, I do want to ensure we're talking about the same thing. What should the experience of the spin dev be when they use wasi:sockets but a spin-host does not support it?

[squillace]

this shim is a distration; it's just a spin host configured in a certain way.

[mikkelhegn]

This is the current thinking of what a Spin Dev ex should look like to validate apps targeting worlds: fermyon/spin#2556.

One thing we've "skipped" is to defined a containerd-spin-shim world. The current Spin world is not what the shim supports, if I'm not mistaken.

[squillace]

Thanks @mikkelhegn those are two great links. Per spin 2556, I'm seeing the match between what the shim could be configured to support and what it should by default support should be seen in terms of the controls that the hosting env has over it. The fact that validating spin worlds is a wonderful idea, and I do like the idea of having a spinkube-core world that is the default, and spinkube advanced that could be things like sockets.

To a greater or lesser degree, I'm in line with the questions that Ivan was asking in that spin issue. So yes, there should be a way to "query" or at a minimum understand when a world that you built something on doesn't exist in spinkube. AND there should be a way to expand spinkube capabilities supported at the shim level. These are the right objectives.

[devigned]

We discussed this topic during the SpinKube office hours meeting on Wednesday 6/26. I'm going to do my best to provide a synopsis of the discussion.

We have a few competing concerns to consider.

1. WebAssembly has a default deny security posture.
2. Kubernetes / containers provide means for controlling network access
3. Spin provides a developer experience which allows a app developer to specify constrained network access

We have a couple of design concepts to balance.

1. Spin apps that are developed locally should work the same when deployed
2. A pod workload should behave approximately the same be it Wasm or any other container workload

With these ideas in mind, we discussed the way that egress in Spin apps are defined in the spin.toml. As Radu points out in #147 (comment), the network access is explicitly requested by the developer. We believe this fits with the default deny security posture.

However, we recognized that if there were a case where a cluster administrator wanted to deny any egress from a running Spin application, regardless to what was requested in the spin.toml, there are means to do this today using seccomp filters on the pod, and an assortment of other Kubernetes networking features.

I believe if we determine a case where the Kubernetes / container level remediations I've described are insufficient, we do have other means to deny functionality at the Wasm layer. Perhaps, we could add some containerd configuration for the shim which would allow a cluster operator to inform the shim that it should deny, for example, socket functionality. I believe we want to make sure we have a clear use case for this, because it may be able to be handled in the Kubernete / container layer.

In summary, we believe that defining constrained network access in the spin.toml gives us enough information to determine the app developers specific intent. We are able to ensure the runtime is configured to serve the app developers intent in the shim by providing constrained network access. I believe we can and should provide the same developer experience as running the app locally. We've also described options for cluster operators to deny a workload networking capabilities, and described an option for adding containerd configuration to the shim to deny socket functionality at the Wasm runtime layer.

Please correct me if I've misrepresented any of the discussion.