What this is and isn't

This is a proof of concet for a shared hosting environment where users are allowed to execute arbitrary code and bind their apps to all non-privileged ports in a secure manner. During the development of a new
version of Uberspace I was trying to come up with a solution to the following problems:

Resource limits

Current Uberspace systems are persistently checking for processes that use more than 600MB of memory for a longer period to ensure that there are no excessive noisy neighbors on the system that could limit the experience of other users on the system. These processes are then sent a SIGTERM and the owner of the process is informed via mail about the reasons for doing so. This approach doesn't work for I/O and CPU shares though.

This prototype demonstrates how to create unprivileged users with specified resource limitations by creating a Control Group (this is why it's called cgroups) with pre-defined values for CPUShares and MemoryLimit.

Port assignment

Uberspace currently allows all users to bind to any unpriviled ports. This is a security risk as another user on the system can try to bind to a previously used port in case of a reboot or a crash of the service and collect all incoming traffic to this port or even impersonate the service, if it isn't properly secured, e.g. by using selfsigned certificates or no TLS at all.

I tried to leverage SELinux to assign ports to SELinux-users so that all users are still able to connect to assigned ports without being able to bind their own apps to any port that isn't assigned to them. According to my tests I was successful.

Note: To the best of my knowledge the scenario I described never happened in all of my 4 years at Uberspace.

Increased Security

Uberspace currently disables SELinux. Everyone is scared of the complexity of SELinux and therefore just disables SELinux. According to my own experience, documentation of SELinux is also pretty overwhelming, outdated and often distribution-vendor-§specific.

My approach is based on a blog post by Major Hayden.

This attempt should allow users to do anything that unprivileged users are do on a Linux system while limiting the scope of the impact of e.g. a local root exploit on a shared hosting system like Uberspace, as long as the attack isn't targeted against SELinux in the Linux kernel.

Why not use containers?

To the best of my knowledge, there is no way to allow unprivileged users to create containers without either adding them to a privileged group that is root-equivalent while also exposing control over all containers to any user on the system. Due to the fact that Uberspace is still primarily shared hosting and not a PAAS-solution, containers where not an option for the version of Uberspace that I was working on.

Will this be used for Uberspace?

I do not know if any of this will end up in the next version of Uberspace because I am no longer working for Uberspace. Please direct all questions regarding the new version of Uberspace to either hallo@uberspace.de or to @ubernauten on Twitter.

Please consider this as a proof of concept. Do not use this in production without further testing!

Control Groups

We can confine users to only use specified resources.

Users created using site.yml will be created with a limit of 512CPU shares (out of 1024, which equals to 50% of all cores) and 1GB of RAM (cumulated virtual memory of the user). The user variable defaults to poc_testuser.

Test & Verify

Login is as our test user. Defaults to poc_testuser:
$ ssh poc_testuser@127.0.0.1 -p 2222 -i .vagrant/machines/selinux/virtualbox/private_key

Run the following command as poc_testuser:
$ stress --vm 10 --vm-bytes 120M --timeout 60s

Run the following command as user vagrant. Login via vagrant ssh:
$ systemd-cgtop

Pay close attention to the %CPU and Memory columns showing the Slice user-1001:

/user.slice/user-1001.slice 14 327.5 276.3M - -

The CPU percentage should never grow beyond 512 while the Memory usage should never grow beyond 1GB of RAM.

Note: Both values can vary during verification.

You can also cross-check the configured limits using the following commands:

$ systemctl show -p CPUShares user-1001.slice
CPUShares=512
$ systemctl show -p MemoryLimit user-1001.slice
MemoryLimit=1073741824

SELinux Basic Usage

Uberspace-related

Ports

Our current default policy per user allows connections to all outgoing ports. The policy is defined in the template files shelluser_u.j2 and shelluser.te.j2. A detailed explanation of both template files will follow.

A user cannot bind to a local port, without first adding it to her user-specific uberspace_poc_testuser_port_t-label.

To ensure that a user cannot elevate their privileges by creating a Uberspace account like system or unconfined, we add a prefix called uberspace_. This also allows for easier management of Uberspace-user's ports.

semanage can modify user context modules and port types. Each modification of an existing user context is being merged into a SELinux-managed diff-table that is being applied to each user's existing policy. This means that we do not need to edit individual policies to add or remove ports. We can also modify a user's context without having to manually re-apply port modifications of a user's context.

In simpler words: semanage keeps track of changes to individual user's security contexts outside of each policy template. Each change is applied to the base context, even if the context-module of the user is changed.

Add a port:
semanage port -a -t uberspace_poc_testuser_port_t -p $protocol $port

Delete a port:
semanage port -d -t uberspace_poc_testuser_port_t -p $protocol $port

Note: If a deleted port is still in use, the user will be able to use the port as long as her process is bound to the port.

List all ports enabled for a user:
semanage port --list | grep uberspace_poc_testuser_port_t

List all ports enabled for all Uberspace users:
semanage port --list | grep uberspace_

Test & Verify

Create a second user by replacing the user-variable in site.yml. In this example, the user is ubertest.

Allow ubertest to bind to port 31337/UDP, which is the default port of netcat.

[vagrant@selinux ~]$ sudo semanage port -a -t uberspace_ubertest_port_t -p udp 31337

Login as ubertest:

$ ssh ubertest@127.0.0.1 -p 2222 -i .vagrant/machines/selinux/virtualbox/private_key  
$ nc -u -l

Now log in as poc_testuser:

[poc_testuser@selinux ~]$ nc -u -l
Ncat: bind to :::31337: Permission denied. QUITTING.

As the example demonstrates, the user poc_testuser cannot bind to the port that has just been assigned to ubertest.

Now we send data via netcat from poc_testuser to ubertest:

[poc_testuser@selinux ~]$ echo "isthisevenworking?" > foo.txt && nc -u localhost 31337 < foo.txt
[uberspace@selinux ~]$ nc -u -l
isthisevenworking?

Note: As you can see, it is still possible to locally connect to a port that has been assigned to a user! If a user wants to ensure that access to her process is restricted to her account only, she must use a UNIX socket. Applications still need authentication when binding to a TCP/UDP-port, even if the port is not opened in iptables!

Sockets

Binding to sockets is possible without any limitations.

Useful information about SELinux and its tools

NOTE: I haven't checked all of this in months. Please feel free to correct me wherever I am wrong!

New files and directories inherent the parent directory's SELinux type... tbc.

Naming

Identity: An identity is not the traditional UNIX UID. An SELinux identity can be assigned to one or more specific UNIX UIDs. The identity can be switched. tbc.

id -Z

• Shows current user's SELinux context

ls -Z

• Show SELinux context of files or directories

useradd -Z zusermod -Z selinux_policy user

Trivia

The following SELinux User Capabilities are available:

• User
• Role
• Domain

An overview of all SELinux-context associations with Linux accounts can be found in �/etc/selinux/targeted/modules/active/seusers�.

Basics

Labeling

• Files, processes, ports
• user:role:type:level(optional) Type Enforcement tbc.

Tools

setenforce

• set Enforce/Permissive

getenforce

• get current mode

semanage login -l

• show User/SELinux User configuration

semanage fcontext -l

• list available contexts, use grep to filter

sesearch

• search policy types

seinfo

• verbose information on policies

restorecon

• reset security context of files

chcon

• change security context of files temporarily
• will be reset during a relabel of the filesystem (restorecon)

semodule

• import compiled policy module
• list active policy modules

matchpathcon

• compare current security context with default security context

sepolicy

• New tool introduced in RHEL7. Allows for easier policy creation.