Containerd runtime detection for wasmedge?

[plaes]

OS: Fedora 41
k3s version v1.32.0+k3s1 (cca8fac)
go version go1.23.3

Is the "K3s will automatically detect alternative container runtimes" as it is advertised in documentation actually working?

Although I have installed wasmedge 0.14.1 from Fedora packages, containerd doesn't seem to be automatically detecting it when restarting k3s (systemctl restart k3s).

I have verified that webassembly examples are working:

ctr i pull docker.io/wasmedge/example-wasi:latest
ctr run --rm \
  --runc-binary crun \
  --runtime io.containerd.runc.v2 \
  --label module.wasm.image/variant=compat-smart \
  docker.io/wasmedge/example-wasi:latest wasm-example \
  /wasi_example_main.wasm 50000000 Hello WasmEdge

And example-wasi even works works when running k3s ctr run ... .

[brandond]

1. The runtime binary needs to be in the $PATH in order to be detected, is it?
2. You need to specify the correct runtimeclassname in your pod spec. Have you?

You've provided no information on what you are doing or what specifically is not working so it's hard to give any specific suggestions on what might be wrong.

[plaes]

You've provided no information on what you are doing or what specifically is not working so it's hard to give any specific suggestions on what might be wrong.

Well, extrapolating from documentation based on nvidia example, I should just install the required binaries (which are in path, see below) and restart k3s (even tried with full stop and start) and I should see something with wasm (in my case wasmedge ?) in the containerd configuration, which I don't, even though wasmedge is in the path.

1. The runtime binary needs to be in the $PATH in order to be detected, is it?

$ which wasmedge
/usr/bin/wasmedge

$ crun --version
crun version 1.19.1
commit: 3e32a70c93f5aa5fea69b50256cca7fd4aa23c80
rundir: /run/crun
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL

2. You need to specify the correct runtimeclassname in your pod spec. Have you?

# wasmedge-sample-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: wasm-demo-app
spec:
  runtimeClassName: wasmedge
  containers:
    - name: wasm-demo-app
      image: docker.io/wasmedge/example-wasi:latest

Not sure whether I also need to install some containerd-shim-... plugin as well so autodiscovery works?

[brandond]

That looks correct. What happens when you create that pod?

[plaes]

In the events I get following:

FailedCreatePodSandBox    Pod/wasm-demo-app
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox runtime: no runtime for "wasmedge" is configured

I have also have the wasmedge runtimeclass. It might look a bit "terse" because I even tried adding the handler: wasmedge there:

# kubectl get runtimeclass wasmedge -oyaml
apiVersion: node.k8s.io/v1
handler: wasmedge
kind: RuntimeClass
metadata:
  creationTimestamp: "2024-09-16T18:34:06Z"
  labels:
    objectset.rio.cattle.io/hash: 57231bea9658cf25fcdac23f1c5685ac7bbcf983
  name: wasmedge
  resourceVersion: "136936303"
  uid: 096b551b-30c9-4151-86c4-9910039e2f1b

Anyway, my suspicion is that whatever is supposed to detect wasmedge and configure containerd automatically, is not working, as grep wasm -r /var/lib/rancher/k3s/agent/etc/containerd/ gives nothing.

[brandond]

Not sure whether I also need to install some containerd-shim-... plugin as well so autodiscovery works?

It is the wasmedge shim that is the runtime, as far as containerd is concerned. Not the wasmedge binary. You can see what k3s is looking for here:

k3s/pkg/agent/containerd/runtimes.go

Lines 101 to 104 in 0a47df6

	"wasmedge": {
	RuntimeType: "io.containerd.wasmedge.v1",
	BinaryName: "containerd-shim-wasmedge-v1",
	},

[plaes]

Well, documentation could be updated to reflect the need for shim when dealing with wasi.

I was hoping crun built with wasmedge extensions would be autodetected as well (as this seems to actually work, but needs to be configured from containerd side).

Sadly there isn't much documentation about this feature upstream either - WasmEdge/WasmEdge#2132

I think this issue could be closed for now...

[plaes]

I got it working using only crun's wasmedge plugin (which is enabled by default in Fedora) without any extra containerd shims!!!

# /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
{{- template "base" . -}}

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."wasmedge"]
  runtime_type = "io.containerd.runc.v2"
  pod_annotations = ["*.wasm.*", "wasm.*", "module.wasm.image/*", "*.module.wasm.image", "module.wasm.image/variant.*"]

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes."wasmedge".options]                                                   
  BinaryName = "crun"
  SystemdCgroup = true

# wasmedge-demo-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: wasm-demo-app
  annotations:
    module.wasm.image/variant: compat
spec:
  runtimeClassName: wasmedge
  containers:
    - name: wasm-demo-app
      image: docker.io/cr7258/wasm-demo-app:v1

[brandond]

I mean, you're using crun as the shim... but cool, glad it works for you! I'm not sure why the general recommendation is to use containerd-shim-wasmedge-v1.