Fail to start several privileged pods on centos7 / vfio / privileged_without_host_devices set to true

[thbtcllt]

Description of problem

I install kubernetes on centos7 and I use containerd. I have vfio devices and i want to start several pods with privileged mode
I have set privileged_without_host_devices to true to avoid pods get all vfio devices at the startup
The complete description of my system is in the info.txt file (output of kata-collect-data.sh)
info.txt

Expected result

privileged pods starts without taking all available vfio

Actual result

privileged pods gets all available vfio and it is not possible to launch several privileged pods. Start of the second pod fails with error like
Warning Failed 8s (x4 over 58s) kubelet, centos7 Error: failed to create containerd task: QMP command failed: vfio 0000:81:10.0: failed to open /dev/vfio/61: Device or resource busy: unknown

pods are launched with containerd-shim-kata-v2 and not by kata-runtime. I have tried to set the privileged_without_host_devices for any kata runtimes in /etc/containerd/config.toml but with no success

[grahamwhaley]

/cc @amshinde

[amshinde]

@thbtcllt I see that you are using containerd 1.2.13, while the change in containerd to support privileged_without_host_devices was introduced in 1.3.0 version:
https://github.com/containerd/cri/blame/release/1.3/docs/config.md#L120

You will need the newer containerd version. I shall update our docs to reflect this as well.

[amshinde]

@thbtcllt I have opened a PR to document the containerd versions:
#639

[thbtcllt]

@amshinde unfortunately i have still the problem with the latest version of containerd (1.3.3). The description of my environment (from kata-collect-data.sh) is: info.txt

I have tried containerd 1.3 version or to set the privileged_without_host_devices option to any kata section and not only the kata-runtime one but it does not change anything.

I have noticed that format of containerd configuration file has changed in 1.3 As i used kata-deploy that updates the containerd configuration file by using the version 1 format I force it.
Does the privileged_without_host_devices option supported in version 1 or must I used the version 2 (example for containerd uses version 1 format so I think that I can use it) ?

How can I check that the privileged_without_host_devices option is correctly sent to kata-runtime ? I have tried to set some logs but I have not found clear evidence.

[thbtcllt]

@amshinde i have found my issue. I have to set the privileged_without_host_devices options to kata-qemu as it is the runtimeClassName used to deploy my pods. On my previous check to add this option to kata-qemu I have forgotten to restart containerd.
Nevertheless I would like to use sriov-network-device-plugin to pass one vf to my pods in privilege mode but when i set the privileged_without_host_devices options no vf are set to my pods.
It seems that when you set the privileged_without_host_devices option you are not able to set after a vf to the pods.
Could you confirm this point ?

[amshinde]

@thbtcllt The option privileged_without_host_devices should not prevent you from passing devices you explicitly pass with a device plugin.
Can you describe the steps you followed, with the results of execing into the container?

[thbtcllt]

@amshinde the plugin always provide the requested devices but regarding the pod mode (privilege or not) and presence of option privileged_without_host_devices this list of requested devices can be modified.

My platform has 4 vf and I use https://github.com/intel/sriov-network-device-plugin to manage them
I use the last containerd version (1.3.3)
My pod configuration is: my-pods.txt
As the used runtime class for this pod is kata-qemu I have set the option privileged_without_host_devices for this plugin in the containerd configuration: I
config.txt

With this configuration when I start my pod the vf is not available.
If I try to launch the pod in a non privilege mode I have the vf in the pod
If I remove the option privileged_without_host_devices for kata-qemu plugin I have the 4 vf in my pod.

I have set the log level to debug in containerd and it seems that containerd change the list of requested devices.
vfio-issue.log is an extract of containerd log for the different cases with my current analysis

Do you have any ideas how I can keep my requested device by the plugin when the option privileged_without_host_devices is set ?

[thbtcllt]

@amshinde
thanks to the trace i have found the origin of the issue in containerd.
Issue is due to https://github.com/containerd/containerd/blob/master/vendor/github.com/containerd/cri/pkg/server/container_create_unix.go#L168
By applying this patch
containerd.txt my problem is solved.
I will do same additional test to be sure that this patch has no side effect but if you have comments on this patch let me know

[amshinde]

@thbtcllt I see the issue now. containerd is not passing any devices in case the option privileged_without_host_devices is set.
Your patch makes sense and looks good to me.

[amshinde]

cc @awprice

[awprice]

Patch seems reasonable to me.

[thbtcllt]

@awprice @amshinde I have tried to push my correction to containerd but i don't clearly understand the process to do it
containerd/containerd#4201: not the correct github repo
containerd/cri#1455: i do not understand the error during the check process

and then as soon as patch is accepted on https://github.com/containerd/cri when it is replicated to https://github.com/containerd/containerd ?

any hints to do it correctly is welcome

[amshinde]

@thbtcllt Yes the containerd/cri repo is the correct repo to submit the patch. Once that PR is merged, it will need to be vendored in the containerd/containerd repo. Since this is a bug fix, it can be vendored on the master as well as 1.3 branch of containerd.
Lets wait for the PR to be merged first. cc @chavafg Who recently submitted a vendoring change in containerd can help you out if you have any questions about vendoring the change in the containerd/containerd repo.

[chavafg]

@thbtcllt were you able to vendor in the containerd/containerd repo?

[thbtcllt]

@chavafg my fix has been vendor by commit https://github.com/containerd/containerd/pull/4219/files
Everything is ok now with the master branch