Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Namespaces for Container Security
Transcript
- Lucas: So we talk about containers constantly — Docker, Podman, orchestration — but I think the actual kernel primitive that makes them work, Linux namespaces, doesn't get nearly enough attention from the people running them in production. Luna: You mean the thing that actually isolates one container from another? Because I've definitely seen setups where people assume Docker handles all that and they never look under the hood. Lucas: Exactly. Namespaces are the kernel feature that gives a process its own view of the system — its own process tree, its own network stack, its own mounts. Without them, a container is just a regular process with a fancy filesystem. Luna: Wait — so when I run 'docker run', what's happening under the hood? Does Docker create a new namespace for every container? Lucas: Yes, but it's more nuanced. Docker creates multiple namespaces by default — usually a mount namespace, a PID namespace, a network namespace, an IPC namespace, a UTS namespace, and a user namespace if you configure it. But the kernel itself supports seven types, and each one isolates a different global resource. Luna: Seven? I think I know the big ones. What are all of them? Lucas: Let's run through them quickly. Mount namespace — isolates filesystem mount points. PID namespace — gives the process its own set of process IDs starting at 1. Network namespace — own network interfaces, routing, firewall rules. IPC namespace — isolates System V IPC and POSIX message queues. UTS namespace — own hostname and domain name. User namespace — maps user IDs inside the namespace to different UIDs outside. And the seventh, which is relatively new, is the cgroup namespace — it virtualizes the view of the cgroup hierarchy so a container sees only its own cgroup. Luna: That's a lot of isolation. But the thing I've always wondered — can a container break out of a namespace? Like, if there's a kernel bug, is the namespace a real barrier or just a suggestion? Lucas: It's a real barrier enforced by the kernel, but it's not foolproof. If a process inside a namespace has the CAP_SYS_ADMIN capability and the kernel has a vulnerability, it might be able to escape. That's why you never run containers as root inside the namespace unless you absolutely have to, and why you drop capabilities. Luna: Right, so defense in depth. Namespaces are one layer, then capabilities, then seccomp, then AppArmor or SELinux. Lucas: Exactly. But I want to focus on a specific scenario that I see a lot in smaller shops: running a container with '--privileged' or with '--pid=host'. That breaks the PID namespace isolation — now the container can see all processes on the host. And if you also share the network namespace with '--network=host', the container basically has no isolation. Luna: I've definitely seen people do that in dev because it's easier. 'I need to see host processes for debugging.' But in production, that's a huge risk. Lucas: Let's talk about how to actually audit this. On any Linux system, you can use the 'lsns' command to list all namespaces currently in use. Run 'lsns -t pid' to see only PID namespaces, for example. You'll see the namespace type, the number of processes, and the command that started it. Luna: And if you want to inspect a specific namespace? Like, if you see a container and you want to verify its isolation? Lucas: That's where 'nsenter' comes in. You can enter a namespace from the host — say, 'nsenter -t <pid> -n ip addr' to see the network interfaces inside that process's network namespace. It's incredibly powerful for debugging, but also a security tool: you can verify that a container's network namespace doesn't have access to the host's interfaces. Luna: So you're saying if I suspect a misconfigured container, I can use nsenter to confirm it's not leaking? Lucas: Exactly. And you can script it. For example, if you're using Docker, you can get the container's PID from 'docker inspect', then run 'nsenter -t $PID -n -- ip link' to see its interfaces. If you see the host's eth0, something's wrong. Luna: What about user namespaces? I know they're recommended for rootless containers, but they add complexity because UIDs get mapped. Lucas: User namespaces are the gold standard for container security because they let you run the container as root inside the namespace while mapping to an unprivileged user outside. But they require careful configuration. For instance, you need to set up subordinate UID and GID ranges in /etc/subuid and /etc/subgid. Docker does this automatically if you enable userns-remap, but it's not the default. Luna: And I've heard that user namespaces can break certain applications that expect real root — like those that need to mount filesystems or create device nodes. Lucas: That's the trade-off. Some applications need privileges that even root inside a user namespace can't get. But for most workloads, it's the right call. Now, let me give you a concrete example of how to use 'unshare' to create your own namespace from scratch, because I think that demystifies the whole thing. Luna: You mean the command-line tool that let's you run a program in a new namespace? That's something I've never actually tried. Lucas: Exactly. 'unshare' is the low-level tool that Docker and Podman use internally. If you run 'unshare -r -n -m --fork --pid bash', you'll get a shell in a new mount, network, and PID namespace, with a user namespace for mapping. It's a minimal container. Luna: So I can test namespace isolation without Docker installed? That's cool. Lucas: And it's a great way to understand what's actually happening. Once you're inside, run 'mount -t proc none /proc' to see only your own processes. Then try 'ip link' — you'll see only a loopback interface, no eth0. That's the power of network namespaces. Luna: But you can also add interfaces to that namespace from outside, right? Like with 'ip link set eth0 netns <pid>'? Lucas: Exactly. That's how Docker connects containers to a bridge network — it creates a veth pair, puts one end in the host namespace and the other in the container's namespace. It's all about namespace manipulation. Luna: Alright, so let's talk about auditing. If I have a bunch of running containers, how do I quickly verify they're properly namespaced? Is there a single command? Lucas: Not a single command, but you can use 'lsns' to list all namespaces, then cross-reference with your container runtime. For Docker, 'docker inspect --format '{{.State.Pid}}' <container>' gives you the PID, then you can check which namespaces that process is in via /proc/PID/ns/. Each namespace has an inode number — if two processes share the same inode for a namespace type, they're in the same namespace. Luna: So I could write a script that loops over containers, gets their PID, reads the symlink target of /proc/PID/ns/net, and compares it to the host's /proc/1/ns/net. If they match, the container is sharing the network namespace. Lucas: Exactly right. And that's a good audit to run periodically, especially after someone inevitably deploys a container with '--network=host' because they couldn't figure out port mapping. Luna: It's funny because it's true. I've been that person. But okay, you mentioned seven namespace types. Are there any that are commonly overlooked? Lucas: IPC and UTS namespaces are often neglected. UTS isolation means each container can have its own hostname — that's usually fine. But IPC isolation prevents containers from accessing each other's shared memory or semaphores, which can be a security risk if you're running untrusted workloads. I've seen setups where people disable IPC namespace sharing for performance, but that's dangerous. Luna: And the cgroup namespace — that one's newer. What does it add? Lucas: It prevents a container from seeing the host's entire cgroup hierarchy. Without it, a process inside a container could potentially modify cgroup settings for other containers or even the host, if it had the right capabilities. It's a defense-in-depth layer. Luna: So if I'm running a Kubernetes cluster, should I verify that the container runtime is creating all seven namespaces? Lucas: Absolutely. By default, Docker creates PID, mount, network, IPC, UTS, and sometimes user. But cgroup namespace requires kernel 4.6 or later and Docker 20.10+. You can check with 'docker info' under 'SecurityOptions'. If you see 'name=cgroupns' you're good. Luna: What about Podman? I know it's designed to be rootless by default. Lucas: Podman uses the same underlying kernel namespaces, but because it's often rootless, it relies heavily on user namespaces. In fact, a rootless container always creates a user namespace, and then all other namespaces are created inside that user namespace. That actually adds an extra layer of security because the root inside the container is mapped to an unprivileged user outside. Luna: So for production, would you recommend Podman over Docker for security? Lucas: It's not that simple. Docker has a larger ecosystem and better orchestration integration. But if you're running containers on a single server and you care deeply about namespace isolation, rootless Podman is a strong choice. The key is understanding what each runtime does with namespaces. Luna: Let's talk about something that's bitten me before: when a container crashes and leaves behind a network namespace. I've had to clean up stale ones. Lucas: That happens when a process that created a namespace exits but something — like a file descriptor — is still holding it open. You can see stale namespaces with 'lsns' — they'll have no processes. To clean them up, you need to kill the process holding the reference, which is often the container runtime itself. Restarting the runtime usually does it, but it's not ideal. Luna: Or you can use 'nsenter' to enter it and unmount anything inside? I've heard of that. Lucas: You can, but it's tricky. If it's a mount namespace, you might need to bind-mount something to keep it alive while you clean up. Honestly, the best practice is to use a container runtime that manages namespace lifecycle properly, and to monitor 'lsns' as part of your system health checks. Luna: So namespaces are powerful but they're not magic. They're a kernel feature that requires understanding. Lucas: And they're one part of a larger security model. Namespaces plus cgroups plus capabilities plus seccomp plus Mandatory Access Control — that's how you build a secure container platform. If any one of those is misconfigured, you have a gap. Luna: It reminds me of something we talked about a while ago — defense in depth. No single layer is perfect. Lucas: Exactly. And I think that's a good place to mention something that keeps these conversations going. The show is ad-free, and it stays that way because of listeners who chip in small amounts. If you've gotten something useful out of our episodes, a couple of dollars a month at buy me a coffee dot com slash fexingo genuinely makes a difference. It's how we keep doing deep dives like this without sponsors. Luna: Yeah, it's a simple way to support the kind of technical content that doesn't always get attention elsewhere. We appreciate everyone who's contributed. Lucas: So back to namespaces — one last thing I want to leave you with. If you're a sysadmin and you haven't played with 'unshare' on a test VM, do it. Create your own PID namespace, mount proc, and watch a process tree that starts at 1. It gives you a visceral understanding of what containers actually are. Luna: And then try 'nsenter' to jump into that namespace from outside. It's like having a backdoor into a container without using Docker exec. Lucas: Exactly. And that's the kind of knowledge that helps you both build secure systems and debug them when something goes wrong. Namespaces are the foundation — know them, and you'll understand containers at a much deeper level.