Unable to create containers when cgroup subsystems are mounted in multiple places

[teddyking]

We recently ran into an issue whereby runc was not able to create containers and was failing with the following error:

EOF
container_linux.go:259: starting container process caused "process_linux.go:283: applying cgroup configuration for process caused \"open /sys/fs/cpuset.cpus: no such file or directory\""

This was happening due to an additional cpu,memory cgroup subsystem mounted somewhere on the system (at /cgroups in this case):

# cat /proc/self/mountinfo | grep cgroup
24 18 0:19 / /sys/fs/cgroup rw,relatime - tmpfs none rw
32 23 0:25 / /cgroups rw,relatime - cgroup cgroup rw,cpu,memory
33 24 0:26 / /sys/fs/cgroup/devices rw,relatime - cgroup cgroup rw,devices
34 24 0:27 / /sys/fs/cgroup/cpuset rw,relatime - cgroup cgroup rw,cpuset
35 24 0:25 / /sys/fs/cgroup/cpu rw,relatime - cgroup cgroup rw,cpu,memory
36 24 0:25 / /sys/fs/cgroup/memory rw,relatime - cgroup cgroup rw,cpu,memory

It seems that this is only a problem when two mounts of the same subsystem are performed in a certain order (i.e in the example above, the /cgroups mount appears first in /proc/self/mountinfo).

Is runc able/supposed to support this sort of setup?

Cheers,
Ed & @craigfurman

[cyphar]

Yeah, this is related to #798 and several other issues we've had in the past. IMO we need to rework how mountpoint checking is done (the code is basically unreadable and is [in my mind] fragile). In particular this code also had similar issues with mounting different cgroup versions (which has now been fixed but I don't like the fix 😉).

In my mind, the nice way for this to be handled (so that it can also accomodate #774) is this:

• Have a list of "implementations" of cgroup (or rather resource) handlers [which includes both cgroupv2 and cgroupv1].
• Go through the list, trying to see if they can set the limit for $resource. continue if any of them succeeds.
• If none of them could set the limit, check whether the limit is the "default" and output an error if it isn't.

The Set and Apply split probably won't work with the above idea, but I'm unsure whether exposing that cgroup interface in our Go interface makes much sense. Then there's the whole kmemcg fiasco.

The above idea could be extended to check (with syscall.Access) whether we have Apply privileges in the rootless context.

[teddyking]

Ok so iiuc this a known issue and it looks like it may require a fairly large change in order to address it. Has anyone started to look at this? If not we could maybe try to get a PR organised. Is this something that would be likely to be accepted?

[cyphar]

I had started to look at it, but I've been swamped by other things recently. IMO if you put together a patch that managed to clean up the mountpoint-finding magic we're currently using to be more resistant to different setups I'd be okay with merging it (though I can't speak for the other maintainers). I wouldn't recommend going straight for the entire rewrite that I proposed above -- simply because it would touch a lot of code that would make people nervous (kmemcg being a prime example).

Though I would like to point out that having multiple mountpoints of the same subsystem is a bit of an odd thing to do (they track the same information) -- however as I mentioned above this is a symptom of a bigger issue we have in the current cgroup manager implementation in runC.

[teddyking]

Ok cool, that sounds reasonable to me.

Yeah I agree it is a bit strange... The background here is that a user was attempting to deploy Garden alongside some other software that was doing its own independent cgroup setup, which then caused runc to start erroring with the above message.

[craigfurman]

@cyphar Opened a PR for your review.

[hqhq]

Fixed by #1372