At home I run a small Proxmox VE server. When you create a new guest there, Proxmox offers two buttons right next to each other: “Create VM” and “Create CT”. On my host, both kinds are running side by side, and from the inside they feel surprisingly similar: each has a hostname, an IP address, a root user, its own ps output. But a CT boots in about a second and costs almost no memory, and a VM does not.
A CT is a Linux container, the same family of technology that Docker is built on. This post is my attempt to answer properly what such a container actually is, and why it is not a small virtual machine. I will rebuild a (small, far from complete) container by hand, one isolation mechanism at a time, and let a tiny test program show what changes at each step.
Everything is Linux
To see why a container is not a virtual machine, I first need to recall how a program gets anything done at all.
The kernel is the core of the operating system and the only software running with full privileges on the CPU. It manages memory, schedules processes onto cores, talks to disks and network cards, and owns the file systems. Everything else, your shell, a web server, a database, runs in user space, a restricted CPU mode in which code cannot touch hardware or global system state directly. A running program is called a process.
Whenever a process needs something only the kernel can provide (open a file, create a network connection, read the machine’s name, start another process), it makes a system call (syscall): it deliberately traps into the kernel, the kernel does the work on its behalf and returns the result. C functions like fopen() or gethostname() are, at the bottom, thin wrappers around syscalls. The important consequence: a process only ever sees what the kernel answers.
With that in place, the difference between the two technologies becomes simple:
- A virtual machine runs on virtualized hardware (virtual CPUs, virtual disks, virtual network cards), and on top of it boots a complete guest operating system with its own kernel. The guest’s processes make syscalls into the guest kernel.
- A container has no kernel of its own. Its processes run directly on the host kernel and make syscalls into it like every other process on the machine. The only difference is that the kernel answers them with a restricted, private view of the system.
You can see this with a single command: uname -r prints the version of the running kernel, and inside a container it prints the host’s kernel version, because there is no other kernel it could report.
This has a direct consequence. The features that produce the private view (namespaces, cgroups and root file system switching, all explained below) are Linux kernel features, and container images contain Linux binaries that make Linux syscalls. The macOS and Windows kernels offer neither. So to run containers on a Mac or on Windows, you need a Linux VM, and the containers run inside it. A nice implementation of this on macOS is Colima (“Containers on Lima”): it starts a lightweight Linux VM with Docker inside, and afterwards the normal docker command on the Mac just works, because it talks to the Docker daemon inside that VM through a socket.
A sandbox, not a machine
So a container is not a small VM. But there is a mental model that does fit: a container is a sandbox. By sandbox I mean an isolation mechanism that keeps a process and its children separated from the host system and from other sandboxes: it should not see what it does not need to see, not change what belongs to others, and not use more than it is allowed to.
On Linux, this isolation is built from three independent mechanisms, and the rest of the post goes through them in this order:
- What a process can see of the system: hostname, other processes, network, users, mounts. This is the job of namespaces.
- Which files it sees: its own root file system instead of the host’s. This is the job of pivot_root(and its older, weaker relativechroot).
- How much it may use: CPU time, memory, number of processes. This is the job of control groups (cgroups).
The test subject
To make the isolation visible, I use a tiny C program, prog.c. It writes a file info.md into its current working directory containing:
- its hostname, the machine name the kernel reports,
- its PID, the process ID the kernel assigned to it,
- its UID, the numeric user ID it runs as,
- the output of ls /tmp, a directory in which basically every program on a system leaves its files.
Compile it with gcc -o prog prog.c and run it directly on a Linux host first:
./prog
cat info.md# Process information
Hostname: comp
PID: 194871
UID: 1000
## Contents of /tmp
vscode-ipc-f1a53cad-4344-4344-4344-a40186a1a98c.sock
vscode-ipc-aeae5bd7-4344-4344-4344-f72e59036f49.sock
zellij-1000
...The real hostname, a large PID (a lot of processes have been started since boot), my own user ID, and a /tmp full of other programs’ files. This process sees the whole machine. Now let’s take that view away, piece by piece.
First step: a hostname of its own
Linux ships a small command line tool called unshare (part of util-linux). It starts a program inside new namespaces, with one flag per namespace type. What a namespace is exactly, I explain right after this experiment; for now it is enough to know that --uts gives the program its own hostname.
There is one obstacle. On Linux, every user is identified by a number, the UID, and the user with UID 0, root, is the one allowed to do administrative things. Creating most namespaces is such an administrative thing. Instead of using sudo, I add two more flags, --user --map-root-user, which let an ordinary user do this. How that can possibly be allowed is also explained further down.
unshare --user --map-root-user --uts sh -c 'hostname container && ./prog'info.md now reads:
Hostname: container
PID: 240302
UID: 0Two things changed. The hostname is container, but running hostname on the host afterwards still prints the old name: the rename happened, but only inside the sandbox. And the UID is 0: the program believes it runs as root, although I started it as a normal user.
If you are feeling brave, try the same command without --uts:
unshare --user --map-root-user sh -c 'hostname container && ./prog'This fails with hostname: you must be root to change the host name. Inside, the process is “root”, but that root is not allowed to rename the host. Keep both observations in mind; the next section explains them.
What a namespace actually is
A process runs in user space, but the kernel keeps a record about every process: its memory, its open files, its credentials, its scheduling state. In the Linux source code, this record is a struct called task_struct. It has hundreds of fields, but a few of them are exactly what this post is about (include/linux/sched.h, shortened, with my comments):
struct task_struct {
/* ... */
const struct cred __rcu *cred; /* UIDs, capabilities, user namespace */
/* ... */
/* Filesystem information: */
struct fs_struct *fs; /* root directory and working directory */
/* ... */
/* Namespaces: */
struct nsproxy *nsproxy;
/* ... */
/* Control Group info protected by css_set_lock: */
struct css_set __rcu *cgroups;
/* ... */
};Part of this context is which namespaces the process belongs to. The nsproxy pointer leads to a small struct with one pointer per namespace type (include/linux/nsproxy.h):
struct nsproxy {
refcount_t count;
struct uts_namespace *uts_ns;
struct ipc_namespace *ipc_ns;
struct mnt_namespace *mnt_ns;
struct pid_namespace *pid_ns_for_children;
struct net *net_ns;
struct time_namespace *time_ns;
struct time_namespace *time_ns_for_children;
struct cgroup_namespace *cgroup_ns;
};A namespace is therefore a kernel object holding one kind of system resource (a hostname, a list of mounts, a network stack, a PID numbering), plus a pointer from each process to the instance it uses. Normally all processes point to the same instances, which is why they all see the same machine. When a process forks a child, the child gets a copy of the parent’s context, including these pointers, so children inherit the namespaces of their parent. That is why hostname and ./prog, both started by the sh inside unshare, see the same hostname. unshare --uts does nothing more than create a new uts_namespace (initialized as a copy of the current one, which is why the hostname does not change by itself), point itself at it, and then start the program.
The actual trick happens whenever such a process makes a syscall. The kernel executes the syscall on behalf of the process, and to answer it, it looks at exactly this context: in kernel code, current always points to the task_struct of the process that made the call. Here is how the kernel answers the uname syscall, which gethostname() uses under the hood (kernel/sys.c, include/linux/utsname.h):
SYSCALL_DEFINE1(newuname, struct new_utsname __user *, name)
{
struct new_utsname tmp;
down_read(&uts_sem);
memcpy(&tmp, utsname(), sizeof(tmp));
up_read(&uts_sem);
/* ... copy tmp back to user space ... */
}
static inline struct new_utsname *utsname(void)
{
return ¤t->nsproxy->uts_ns->name;
}current->nsproxy->uts_ns: there is no global hostname that gets faked for containers. There is only “the hostname in the UTS namespace of the process that is asking”. The same pattern appears everywhere in the kernel. When a process calls socket(), the socket is created in the caller’s network namespace (net/socket.c):
int sock_create(int family, int type, int protocol, struct socket **res)
{
return __sock_create(current->nsproxy->net_ns, family, type, protocol, res, 0);
}And when a process calls open("/tmp/info.md", ...), the kernel resolves the path starting at the root directory stored in current->fs and walks through the mounts of the caller’s mount namespace. (A later read() on the returned file descriptor does not need this anymore: the file was already found, so the isolation happens at the moment a name is turned into a file.)
The process never learns about any of this. It makes the same syscalls as always and simply gets different answers. You can even look at the pointers from user space: every process has a directory /proc/self/ns/ with one entry per namespace, and the number in brackets identifies the namespace instance.
$ ls -l /proc/self/ns/uts
... /proc/self/ns/uts -> uts:[4026531838]
$ unshare --user --map-root-user --uts ls -l /proc/self/ns/uts
... /proc/self/ns/uts -> uts:[4026532840]Different number, different namespace.
The namespaces that matter most
Linux has eight namespace types. The UTS namespace (hostname; the name is a leftover from “UNIX Time-sharing System” in the uname struct) was a nice first experiment, but for a sandbox, these four are the important ones:
- Mount namespace (mnt): its own list of mounts. A mount attaches a file system (a disk partition, a RAM-backed file system, a kernel pseudo file system) to a directory in the file tree. In a new mount namespace, mounting and unmounting does not affect the host.
- PID namespace (pid): its own process ID numbering. Processes inside can only see processes in their own namespace (and those below it).
- Network namespace (net): its own complete network stack: interfaces, addresses, routing tables, firewall rules, ports.
- User namespace (user): its own mapping of user and group IDs, and with it, its own notion of who is privileged.
The remaining ones are IPC (System V shared memory and message queues), cgroup (which part of the cgroup tree a process sees), time (offsets for some system clocks) and UTS. I go through the four important ones in the next sections, while building up the container, starting with the one that explains the two surprises from above.
User namespaces: root, but only at home
To understand the user namespace, I first need to be more precise about “root”. Historically, UID 0 simply passed every permission check. Modern Linux splits root’s power into about 40 individual capabilities, for example CAP_SYS_ADMIN (mounting, setting the hostname, and a lot more), CAP_NET_ADMIN (configuring network interfaces) or CAP_SYS_CHROOT. The kernel checks for the specific capability an operation needs.
A user namespace does two things. First, it has a UID mapping: a table that translates IDs inside the namespace to IDs outside. --map-root-user writes a single line into this table, which you can read back from inside:
$ unshare --user --map-root-user cat /proc/self/uid_map
0 1000 1“UID 0 inside corresponds to UID 1000 outside, for a range of 1 ID.” So when the kernel reports my UID to a process inside, it translates 1000 to 0, and prog prints UID: 0. All IDs that are not mapped appear as 65534 (“nobody”): ls -l /etc/shadow inside shows the file as owned by 65534, and reading it still fails with “Permission denied”, because towards the real file system, the process is still UID 1000.
Second, the creator of a user namespace gets the full set of capabilities, but only relative to that namespace: every other namespace is owned by the user namespace it was created in, and a capability only counts for resources owned by a namespace where the process holds it. You can see this check directly in the sethostname syscall (kernel/sys.c):
SYSCALL_DEFINE2(sethostname, char __user *, name, int, len)
{
/* ... */
if (!ns_capable(current->nsproxy->uts_ns->user_ns, CAP_SYS_ADMIN))
return -EPERM;
/* ... */
}“Does the caller have CAP_SYS_ADMIN in the user namespace that owns its UTS namespace?” With --uts, the new UTS namespace was created inside my new user namespace, where I hold all capabilities: allowed. Without --uts, the process still points to the host’s UTS namespace, owned by the initial user namespace, where I am just UID 1000: EPERM. That is the brave experiment from above, explained in two lines of kernel code.
This is also why unshare needs no sudo in this post: the user namespace is created first, and all other namespaces created together with it are owned by it. A small detail: the user namespace is not part of nsproxy at all, it lives in the process credentials (cred->user_ns), right next to the UIDs it translates.
Building the container, flag by flag
Now I stack namespaces until the program sees a (nearly) empty machine.
Processes: PID namespace and /proc
A new PID namespace starts its own numbering, and its first process gets PID 1. Note the field name pid_ns_for_children in nsproxy: unsharing a PID namespace does not move the calling process itself (its PID cannot change while it runs), only children created afterwards land in the new namespace. That is why --pid needs the companion flag --fork: unshare forks and runs the command as its child, which becomes PID 1.
PID 1 is special, in a container just like on a normal Linux system, where it is the init process (usually systemd):
- Processes whose parent dies are re-parented to PID 1, which has to collect (“reap”) them when they exit. Otherwise they stay around as zombies.
- PID 1 only receives signals from inside its namespace for which it has installed a handler. A program that never registered a handler for SIGTERMsimply ignores it when it runs as PID 1.
- If PID 1 exits, the kernel kills every other process in the namespace.
This is where the two kinds of containers on my Proxmox host differ. An LXC container (“CT” in Proxmox terms) is a system container: it boots a complete distribution with its own init system, usually systemd, as PID 1, which takes care of all of the above, and then starts services like a normal machine. A Docker container is an application container: usually the application itself runs as PID 1, and most applications were never written to be an init process. That is why docker run has an --init flag, which starts a tiny init (tini) as PID 1 that forwards signals to the application and reaps zombies.
One more problem: tools like ps or htop do not ask the kernel for a process list through a dedicated syscall. They read /proc, a pseudo file system in which the kernel presents one directory per process. A /proc mount shows the PID namespace of whoever mounted it, so the inherited /proc still shows all host processes. --mount-proc mounts a fresh /proc for the new PID namespace right before starting the program, and it implies a new mount namespace, so the host’s /proc stays untouched.
But how can one and the same process have a small PID inside and a large one on the host? Because in the kernel, a PID is not just a number. It is an object of its own, struct pid, which task_struct points to via thread_pid, and it holds one number per PID namespace level (include/linux/pid.h, shortened, comments mine):
struct upid {
int nr; /* the number ... */
struct pid_namespace *ns; /* ... and the namespace it is valid in */
};
struct pid {
refcount_t count;
unsigned int level; /* depth of the namespace the process was created in */
/* ... */
struct upid numbers[]; /* one entry per level: [0] = host, [level] = innermost */
};PID namespaces are nested: every new one has a parent, and the host’s is level 0. When a process forks, alloc_pid() (kernel/pid.c) walks from the new process’s namespace up to the host and allocates a free number on every level. And when a process asks for a PID, for example with getpid(), the function pid_nr_ns() picks the entry that belongs to the caller’s PID namespace. The same pattern as with the hostname: the answer depends on who is asking.
/proc works the same way, with one twist: it does not use the namespace of the process that reads it, but the one of the process that mounted it. procfs stores that namespace when it is mounted (fs/proc/root.c), and when you list /proc, the kernel only returns processes that have a number in this namespace and names their directories with exactly that number (fs/proc/base.c). That is the kernel-level reason why the inherited /proc keeps showing host PIDs, and why --mount-proc is needed.
You can watch both numbers at once. The NSpid line in /proc/<pid>/status lists all numbers of a process, as seen from the namespace of the /proc mount:
unshare --user --map-root-user --pid --fork --mount-proc sh -c 'grep NSpid /proc/$$/status; exec sleep 60' &
grep NSpid /proc/$(pgrep -n sleep)/statusNSpid: 1
NSpid: 377333 1The first line comes from inside: $$ is the PID of the shell, which is PID 1 in the new namespace, and from within the container only this one number is visible, because the levels above are unknown there. exec then replaces the shell with sleep without creating a new process, so sleep keeps PID 1. The second line is read on the host for exactly this process: host PID 377333 and container PID 1, both stored in the same struct pid.
Files: mount namespace, tmpfs and propagation
With a new mount namespace (--mount), we can mount something over /tmp that only the sandbox sees. A good candidate is tmpfs, a file system that lives purely in RAM and starts out empty. Mounting it on /tmp hides the host’s files from the sandbox without deleting anything.
There is a subtlety here. Mounts have a propagation type, which controls whether mount events travel between mount namespaces:
- shared: mount and unmount events on this mount are propagated to all its “peers”, including copies of it in other mount namespaces. A new mount namespace starts as a copy of the old one, so its mounts are peers of the host’s.
- private: events are not propagated in either direction.
systemd marks the whole file tree as shared at boot. If that stayed so, my tmpfs on /tmp would propagate back and show up on the host, which is the opposite of isolation. unshare therefore sets all mounts in the new namespace to private by default. You can check both with findmnt:
$ findmnt -o TARGET,PROPAGATION /
TARGET PROPAGATION
/ shared
$ unshare --user --map-root-user --mount findmnt -o TARGET,PROPAGATION /
TARGET PROPAGATION
/ privateNetwork: its own struct net
--net creates a new network namespace. In the kernel, a network namespace is literally one struct net, the thing nsproxy->net_ns points to. It contains more or less everything that makes up a network stack (include/net/net_namespace.h, shortened, comments without quotes are mine):
struct net {
/* ... */
struct user_namespace *user_ns; /* Owning user namespace */
/* ... */
struct list_head dev_base_head; /* list of all network interfaces */
/* ... */
struct net_device *loopback_dev; /* The loopback */
/* ... */
struct netns_unix unx; /* UNIX domain sockets */
struct netns_ipv4 ipv4; /* IPv4 state, incl. routing tables */
struct netns_ipv6 ipv6; /* the same for IPv6 */
/* ... */
struct netns_nf nf; /* netfilter hooks, i.e. the firewall */
/* ... */
};And the assignment also works the other way around: every network interface (struct net_device) has a field nd_net pointing to the one namespace it lives in, and every socket stores the namespace it was created in (that was the current->nsproxy->net_ns in sock_create above). An interface can only be in exactly one network namespace at a time.
A fresh network namespace contains exactly one interface, the loopback device lo, and it is even switched off: ping 127.0.0.1 fails with “Network is unreachable” until you run ip link set lo up. There is no routing table entry, no physical network card and therefore no way out. Real container runtimes connect the namespace to the outside with a veth pair, a virtual Ethernet cable whose one end is moved into the container’s namespace and whose other end stays on the host, attached to a bridge.
A nice real-world example of how powerful a separate struct net is: gluetun, a Docker container that is a VPN client. Inside its container, all traffic goes through a VPN, while the host, running on the same kernel at the same time, has no VPN at all. This works because gluetun changes only things that live in its own struct net:
- A tunnel interface. With OpenVPN, gluetun uses a TUN device called tun0. A TUN device is a virtual network interface without hardware behind it: IP packets that are routed to it are handed to a user space program instead of a network card. Here, that program is the VPN client, which encrypts each packet and sends it on as normal traffic via the container’s regular interface to the VPN server.
- Routing tables. Inside the namespace, the traffic is routed into the tunnel, while the encrypted packets to the VPN server itself still take the original route out. For WireGuard, the source shows exactly how: the tunnel’s own packets are marked, and a policy routing rule sends all unmarked traffic to a separate routing table (number 51820) whose routes lead into wg0.
- Firewall rules. gluetun sets the default policy of its iptables chains to DROP and only allows traffic through the VPN interface, plus the connection to the VPN server itself. If the tunnel breaks, nothing leaks out unencrypted: a kill switch (gluetun wiki).
To do this, the gluetun container needs the NET_ADMIN capability and access to /dev/net/tun, both added explicitly in its compose file. And other containers can use the VPN by joining gluetun’s network namespace with network_mode: "service:gluetun" (gluetun wiki); Docker then simply points their network namespace to gluetun’s /proc/<pid>/ns/net instead of creating a new one. The host is not affected by any of it, because its interfaces, routing tables and firewall rules are in a different struct net.
IPC
--ipc gives the sandbox its own System V message queues, semaphores and shared memory segments. It is cheap, so I add it too.
Everything together
unshare
--mount
--uts
--ipc
--net
--pid
--fork
--mount-proc
--user
--map-root-user
sh -c '
hostname container
mount -t tmpfs tmpfs /tmp
ip link set lo up
./prog
'And info.md now reads:
# Process information
Hostname: container
PID: 5
UID: 0
## Contents of /tmp
- Hostname container, from the private UTS namespace.
- PID 5 instead of a six-digit number. Why not 1? PID 1 is the shthat runs the script, and before./progit already started three helper processes (hostname,mount,ip), which got PIDs 2 to 4. The container has its own small world of processes, and we are the fifth one in it.
- UID 0, mapped from my own user.
- /tmpis empty, thanks to the tmpfs that only this mount namespace sees.
A file system of its own: pivot_root
We now have namespace isolation, but the sandbox still sees the host’s file system: /usr/bin, /etc, my home directory. Only /tmp is covered up. A real container has its own root file system (rootfs): a directory tree that looks like a complete, small Linux installation, and that is all the container ever sees. This is essentially what a container image is: a packed root file system plus some metadata.
For this experiment I use the Alpine Linux “mini root filesystem”, a tarball of a few megabytes with a shell, the BusyBox tools (ls, mount, ps, …) and a C library. Unpack it into a directory rootfs and copy the program in:
mkdir rootfs
tar -xzf alpine-minirootfs-*.tar.gz -C rootfs
gcc -static -o rootfs/prog prog.c(A small side note on -static: a normally compiled program is dynamically linked, and at startup a loader from the file system loads the C library into it. Those files belong to the host’s file system and do not exist in the Alpine rootfs, so a dynamically linked prog would fail there with a confusing “No such file or directory”. A statically linked binary carries everything it needs.)
The oldest tool to switch into such a directory is chroot. Every process has a root directory (the root in current->fs from above), where the resolution of absolute paths starts, and chroot simply changes it. But that is all it does: the man page itself says that chroot “is not intended to be used for any kind of security purpose” (chroot(2)). The host’s file tree stays mounted, and a process with the CAP_SYS_CHROOT capability can walk back out, for example by chrooting into a subdirectory while its working directory stays outside of it.
Container runtimes therefore use pivot_root, which works one level deeper: it does not change a per-process pointer, but swaps the root mount of the whole mount namespace. The new root becomes /, the old root is moved to a directory inside it, and from there it can be unmounted completely. After that, the host’s file tree is not just hidden, it is simply not part of this mount namespace anymore. There is nothing left to escape to.
pivot_root has a few requirements, and two of them explain the extra lines in the command below: the new root must be a mount point (so the rootfs directory is bind-mounted onto itself, which turns it into one), and neither it nor the current root may be a shared mount (which unshare already took care of, see above).
unshare
--mount
--uts
--ipc
--net
--pid
--fork
--user
--map-root-user
sh -c '
hostname container
mount --bind ./rootfs ./rootfs
cd ./rootfs
mkdir -p old_root
pivot_root . old_root
cd /
mount -t proc proc /proc
mount -t tmpfs tmpfs /tmp
umount -l /old_root
/prog
'A few details:
- --mount-procis gone. It would mount the new- /proconto the host’s- /procpath before the root switch. Instead,- /procis mounted after- pivot_root, onto the rootfs’ own- /procdirectory. Since we are in the new PID namespace,- psinside lists only a handful of processes.
- The old root is unmounted with umount -l(“lazy”) only after/procwas mounted. The other way around, themountfails with “permission denied”. The reason is a check in the kernel (mount_too_revealing()infs/namespace.c): root in a user namespace may (for a normal, unrestricted/proc) only mount a new one if a fully visible/procalready exists in its mount namespace, so that mounting cannot reveal anything the host had hidden. As long as/old_root/procis still there, that condition holds.
- After the switch, mount,umountand/progare resolved in the new root: these are Alpine’s BusyBox tools now, not the host’s.
- info.mdends up in- rootfs/info.mdwith- Hostname: container,- PID: 9,- UID: 0and an empty- /tmp, and- ls /old_rootshows an empty directory.
At this point, the process has its own hostname, its own process tree, its own network stack, its own file system, and thinks it is root. That is namespace isolation and file system isolation. What we have not covered yet is the third point from the beginning: nothing so far stops this process from eating the whole machine.
Limiting resources: cgroups
A namespace controls what a process sees, not how much CPU time or memory it uses. To show the difference, I use a second program, stress.c. It asks how many CPU cores are online and forks one worker per core, each of which increments a counter in an endless loop.
./stressOn my 16-core machine, htop shows every core at 100%:
The same happens inside all the namespaces we built above: the scheduler happily hands the sandboxed workers every core.
To limit this, Linux has control groups (cgroups). A cgroup is a group of processes that the kernel accounts for and limits together. In the current version, cgroup v2, the groups form a tree that is visible as directories under /sys/fs/cgroup: creating a directory creates a cgroup, and the files inside are its settings and statistics. Every process belongs to exactly one cgroup (the cgroups field in task_struct), and children start in the cgroup of their parent.
For CPU, the relevant file is cpu.max. It holds two numbers in microseconds, a quota and a period: all processes in the group together may use at most “quota” microseconds of CPU time within each period. If the quota is used up, the scheduler does not run any of the group’s processes until the next period begins.
sudo mkdir /sys/fs/cgroup/mycontainer
echo "100000 100000" | sudo tee /sys/fs/cgroup/mycontainer/cpu.max
sudo sh -c '
echo $$ > /sys/fs/cgroup/mycontainer/cgroup.procs
exec unshare
--mount
--uts
--ipc
--net
--pid
--fork
--mount-proc
sh -c "
hostname container
mount -t tmpfs tmpfs /tmp
ip link set lo up
./stress
"
'(This one needs sudo, because creating a cgroup directly under /sys/fs/cgroup is an administrative task on the host.) The order matters: the outer shell first writes its own PID ($$) into cgroup.procs, which moves it into the cgroup. exec then replaces the shell with unshare, keeping the same process and therefore the same cgroup, and everything started from there inherits the membership: the namespaced shell, stress and all its workers.
100000 100000 means 100 ms of CPU time per 100 ms period, so “one CPU’s worth” for the whole group. And htop now looks like this:
What I find important about this picture: the limit does not give the container one dedicated core. The workers still run on all 16 cores, because the scheduler keeps distributing them over all CPUs exactly like before. The only difference is that the group runs out of its time budget after 100 ms of combined CPU time per period, and then all of its workers wait. Spread over 16 cores, that is about 6% per core (a bit more in the screenshot, since the bars also include everything else running on the machine). If you wanted to pin a container to specific cores, that would be a different controller, cpuset.
cgroups can do much more than CPU: memory.max sets a hard memory limit, pids.max limits the number of processes (a nice defense against fork bombs), and io.max throttles disk I/O. When you run docker run --cpus=1 --memory=512m, Docker writes exactly these kinds of files for you. Afterwards, stop stress with Ctrl+C and remove the group with sudo rmdir /sys/fs/cgroup/mycontainer.
What Docker adds on top
Our hand-built container is missing a lot. Here is what Docker does beyond it.
The runtime stack. The docker command talks to the Docker daemon (dockerd), which hands containers over to containerd, which in turn uses runc to actually create them. runc is the part that does what we did by hand: create namespaces, pivot_root into the rootfs, set up cgroups.
Layered images with overlayfs. Unpacking a full rootfs for every container would waste a lot of disk space. Docker images consist of read-only layers, and the kernel’s overlayfs stacks them into one merged view: files in upper layers hide files with the same name in lower ones. Every container gets an additional, empty writable layer on top. When the container modifies a file from an image layer, the file is first copied up into its writable layer, (Docker docs on the overlay2 driver). So a hundred containers from the same image share the image layers on disk, and each only adds its own thin writable layer (Docker storage drivers).
Capabilities. Docker starts containers with a restricted set of capabilities: 14 of them in the default list. CAP_SYS_ADMIN or CAP_NET_ADMIN, for example, are not part of it, so “root” inside a container “has much less privileges than the real ‘root’” (Docker engine security).
No user namespace by default. This one is easy to miss. Our toy container used a user namespace, so its root was my normal user outside. Docker does not do this by default: unless the daemon is configured with --userns-remap, root inside a Docker container is UID 0 on the host as well. Docker’s own docs describe the remapping as “available but not enabled by default” (Docker engine security), and it is kept in check by the reduced capabilities and the next point.
seccomp. Docker installs a seccomp filter, a list of allowed syscalls that the kernel enforces for every process in the container. The default profile disables around 44 of more than 300 syscalls, which a normal application never needs anyway (Docker seccomp docs).
Networking. Docker’s default bridge network is the veth approach from above: one end of a veth pair in the container’s network namespace, the other end attached to the docker0 bridge on the host, plus NAT rules so containers can reach the outside world.
How safe is root in a user namespace?
Everything we did in this post ran without sudo, and the whole time the program believed it was root. The usual argument for why this is fine goes: “root in a user namespace only has power over its own namespaces, so it cannot harm the host.” I think it is worth spelling out which assumptions hide in that sentence.
- Every syscall is interpreted correctly in the namespace context. Every single permission check in the kernel has to ask the right question, like the ns_capable(current->nsproxy->uts_ns->user_ns, ...)insethostname. One check that asks the wrong namespace, or none, and “root at home” becomes root on the host.
- The kernel has no exploitable bugs. There is only one kernel for all containers and the host. A memory corruption bug anywhere in it gives an attacker control over that kernel, and the namespaces are just data structures inside it.
The second assumption gets worse through user namespaces themselves: with namespaced capabilities, an unprivileged user can suddenly reach kernel code that used to be reserved for root, like mounting file systems or configuring firewall rules in its own network namespace. Canonical put it this way: “even if unprivileged user namespaces are bug free, as long as any privileged kernel interface or combination of interfaces has a bug, an unprivileged user can try to exploit that bug” (Canonical).
That this is not theoretical shows in two well-documented examples:
- CVE-2022-0185: an integer underflow in the kernel’s file system context parsing, reachable through the fsconfigsyscall. It requiresCAP_SYS_ADMIN, but only in the current namespace, which any user gets withunshare. The researchers used it for a local privilege escalation on Ubuntu and to escape from a container on Google’s hardened kCTF Kubernetes environment (write-up by the finders).
- CVE-2024-1086: a double free in nf_tables, the kernel’s firewall subsystem. The author of the exploit lists as its only requirements “that nf_tables is enabled and unprivileged user namespaces are enabled”, notes that the latter should be enabled by default on Debian and Ubuntu, and considers kernels from at least v5.14.21 up to v6.6.14 exploitable (Notselwyn’s write-up).
So “root in a user namespace cannot harm the host” is really “root in a user namespace cannot harm the host, as long as the kernel is correct”. This is exactly the difference to a VM, where an attacker has to break out of a whole guest kernel and then through the much smaller interface of the hypervisor.
A different kind of sandbox: iOS apps
While reading about all this, I wondered how my iPhone does it: apps there are clearly isolated from each other too. It turns out iOS takes a fundamentally different approach.
On iOS, all third-party apps run as the same unprivileged user, mobile (Apple Platform Security). The user ID can therefore not separate apps at all, and there are no per-app namespaces either: all apps live in one PID space, one network stack and one file system tree.
Instead, the kernel (XNU) has a framework of hooks in its code paths, the TrustedBSD Mandatory Access Control Framework (MACF). Before an operation like opening a file or sending a network packet, the kernel asks the registered policy modules whether it is allowed. The iOS and macOS sandbox is such a policy module: a kernel extension that looks up the rules attached to the calling process and answers allow or deny. The rules are written in a Scheme-like profile language and compiled in user space before they are handed to the kernel. Dionysus Blazakis reverse engineered this whole chain in his paper “The Apple Sandbox” (2011, analyzed on Mac OS X 10.6, so the details have certainly changed since, but the design has not), and his demos show nicely what this looks like from the program’s side: the file is right there, but opening it fails with “Operation not permitted”.
Each app gets its own container directories when it is installed: a bundle container with the signed app itself, and a data container for its documents and caches, and an app is generally not allowed to access or create files outside of them (Apple File System Programming Guide). Note that these are just directories in the shared file tree, with rules on top, not a separate root. Exceptions (for example an additional iCloud container) are granted via entitlements, key-value permissions embedded in the app’s signature. And because iOS, like Apple’s other device operating systems, only executes code signed with an Apple-issued certificate (Apple Platform Security), neither the app’s identity nor its entitlements can be changed after the fact.
The contrast to Linux containers fits in one sentence: iOS denies access in one shared world, while Linux gives every container its own world. An iOS app could in principle name another app’s files, it just gets a “no” when it tries to open them. A process in a Linux container cannot even name the host’s files after pivot_root, because in its mount namespace they do not exist. Both approaches end up with a single kernel making all the decisions, and both are only as strong as that kernel.
Back to my Proxmox host
So what is the difference between the two buttons on my Proxmox host? A VM gets virtual hardware and runs its own kernel; the host only sees a QEMU process. A CT is an LXC system container: a set of ordinary processes on the Proxmox host’s kernel, starting with its own systemd as PID 1, with its own namespaces, its own root file system, and cgroup limits for the CPU and memory I configured in the web interface. Proxmox runs CTs unprivileged by default, which means with exactly the kind of user namespace mapping from this post: root in the CT is an unprivileged user on the host.
That explains why a CT starts in a second (there is no kernel to boot) and why it can only run Linux (there is only the host’s kernel to use). And it explains why the Proxmox documentation says that “in general, full virtual machines provide better isolation”, and recommends nesting application containers inside a VM for use cases “demanding maximum isolation” (Proxmox wiki): a container is a sandbox built from kernel data structures, and a VM is a separate machine.