Fix io limiting with cgroup v2

[Furisto]

Description

Fix io limiting with cgroup v2.

Related Issue(s)

n.a.

How to test

https://www.loom.com/share/3fc4a931fe3c4541b673650f86d56430

Release Notes

Ensure io limiting of workspaces works with cgroup v2

Documentation

[Furisto]

/werft run

👍 started the job as gitpod-build-fo-fix-v2-io-limit.1

[aledbf]

@Furisto please refactor v1 and this plugin to share the logic of the devices?

[csweichel]

Also, the user cgroup is modifiable by the user. I.e. they can disable their own limits

[Furisto]

@csweichel Good point. Actually even if we would hide that within another delegated cgroup namespace you could always create another child cgroup and move your io intensive process in that cgroup 🤔

[csweichel]

Aren't cgroups hierarchical? I.e. you can only act within the context/limits of your parent cgroup?

[Furisto]

I would have expected that but that is not what it looks like in regards to io.max limits.

[csweichel]

kernel bug? :P

[utam0k]

😭

[utam0k]

Worst case, why not check the tree structure of all cgroupfs and apply it? In cgroup v2 the process belongs only to the terminal cgroup. However, we will need to check if the search can be realistically terminated in a finite time.

Or, monitor the cgroup notifications. Wecan use cgroup.events to monitor when a process is registered or deleted. The implementation will be tough, but it should be possible to use this to detect when a user creates a new cgroup and moves a process.

[Furisto]

Worst case, why not check the tree structure of all cgroupfs and apply it? In cgroup v2 the process belongs only to the terminal cgroup. However, we will need to check if the search can be realistically terminated in a finite time.

Or, monitor the cgroup notifications. Wecan use cgroup.events to monitor when a process is registered or deleted. The implementation will be tough, but it should be possible to use this to detect when a user creates a new cgroup and moves a process.

• When the limit is removed from the cgroup the process resides in, it will not have any limits. The limits of the parent do not have any effect as far as I can tell, so applying it to the whole tree structure is not effective.

• I am pretty sure the user could make the modification and then make the file read only or mount tmpfs over it or other shenanigans so that subsequent writes will have no effect.

[kylos101]

@Furisto can we close this for now, and reopen we decide to resume the work? I ask to remove it from our list of active PRs. 🙏

[aledbf]

@Furisto another thing we need to test is the switch to systems instead cgroupfs for cgroups v2. Not sure if the behavior is the same.

[aledbf]

If we want to upgrade to k8s 1.24, we need to switch to systemd k3s-io/k3s#5462

[Furisto]

The most significant change will be that the location of the cgroup for the container will change. As the cgroup is created by systemd it will be located in the systemd managed subtree of the cgroup fs. Considering that we can get the cgroup path from containerd the change will probably transparent to us though and no code will need to be adapted.

From a resource control standpoint nothing will change as runc will connect to systemd via dbus to instruct it to write the resource limits into the cgroup fs. So this is just an indirection, everything will still end up in the cgroup fs, just in a different location. We would still write to the cgroup fs directly in order to control resources.