Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / Linux Server Virtual File System Internals with FUSE
Transcript
- Lucas: So I want to talk about something that sits in this weird middle ground of Linux storage — FUSE. Filesystem in Userspace. Luna: FUSE. I've used sshfs plenty of times, but I've never really thought about how it actually works under the hood. Lucas: Right. And that's exactly the appeal. Normally, if you want to create a new filesystem on Linux, you write a kernel module. That means working in C, dealing with memory allocation rules, locking, and if you mess up, you can crash the whole box. Luna: Yeah, kernel space is unforgiving. One null pointer and you're looking at a panic. Lucas: Exactly. FUSE lets you run that filesystem logic as a regular user-space process. Your filesystem is just a program that responds to read, write, open, readdir calls over a special character device — /dev/fuse. Luna: So the kernel still does the heavy lifting of VFS — virtual file system — integration, but the actual data handling is in user space. Lucas: Right. The kernel module — fuse.ko — acts as a bridge. When a process does, say, ls on a FUSE mount, the VFS layer forwards the readdir request to fuse.ko, which then writes it to a control file descriptor that the user-space daemon is reading from. Luna: So there's a context switch every time. That has to add latency. Lucas: It does. The overhead is measurable — we're talking tens to hundreds of microseconds per operation, depending on how the daemon is implemented. For most use cases, that's fine. But if you're doing millions of small I/Os per second, like a database, FUSE will crush your throughput. Luna: Right. So it's not for PostgreSQL. But for something like mounting an S3 bucket as a local directory? That's a perfect use case. Lucas: Exactly. Let's use s3fs-fuse as the concrete example. That's a FUSE filesystem that maps S3 objects to files and directories. You install it, run s3fs mybucket /mnt/s3 -o passwd_file=/etc/s3pass, and suddenly you have a remote object store appearing as a local file tree. Luna: And because it's FUSE, you didn't need to compile a kernel module. It's all just a binary you can install with apt or yum. Lucas: That's the killer feature. The development cycle is also way faster. You're writing Python, Go, Rust — whatever you want — and you can test the filesystem logic without rebooting. If your daemon crashes, the mount just stops responding, but the kernel stays up. Luna: So what's actually happening when I run 'cat /mnt/s3/myfile.txt'? Walk me through the full path. Lucas: Sure. First, the cat command calls open on the path. The VFS sees it's on a FUSE mount and forwards the call to fuse.ko. The kernel module packages the request — it includes the file path, the PID of the calling process, flags — and writes it to a ring buffer that the s3fs daemon is polling. Luna: The daemon reads that, then makes an actual HTTP GET request to S3. Lucas: Right. It authenticates, downloads the object — possibly into a local cache — then returns the data to the kernel module, which copies it into the buffer cat expects. The whole round trip includes a network call, so it's slow compared to local ext4. But for archiving, media serving, or backup restoration, it's totally usable. Luna: I've also seen people use FUSE for encryption filesystems like EncFS or gocryptfs. That way the encrypted data lives on disk, but the decrypted view is served through FUSE. Lucas: Yes. And that's actually a great example of separation of concerns. The encryption logic is completely separate from the underlying storage. You can put the encrypted directory on Dropbox, on NFS, on a USB drive — the FUSE daemon doesn't care. It just reads ciphertext and returns plaintext. Luna: And because it's user space, you can use libraries like libsodium or OpenSSL directly. You don't have to write kernel crypto code. Lucas: Exactly. Let's talk about one more practical example. I recently set up a CI pipeline where we needed to merge results from multiple build servers into a single directory tree. Each build server wrote JSON artifacts to its own S3 prefix. Instead of writing a custom sync script, I used s3fs on the aggregation server, mounted all the prefixes with different mount points, and then used unionfs-fuse to overlay them. Luna: Wait — unionfs-fuse is another FUSE filesystem that does what aufs or overlayfs does, but in user space? Lucas: Yes. So I mounted each build server's S3 bucket as /mnt/build1, /mnt/build2, etc., then mounted a union of them at /mnt/merged. The CI tool just read /mnt/merged and saw all artifacts as one directory. No kernel config changes, no custom kernel builds. Took maybe twenty minutes to set up. Luna: That's a neat hack. But there has to be a downside beyond latency. What about caching? If you read the same file twice, does it always go back to S3? Lucas: Most FUSE filesystems implement their own caching. s3fs defaults to a local disk cache in /tmp, with a configurable size and TTL. But it's up to the daemon. If the daemon doesn't cache, every read becomes a network call. Also, there's no page cache integration — the kernel can't cache FUSE data the way it caches ext4. The FUSE protocol does support a feature called 'kernel caching' where the module can cache attributes and dentries, but data caching is more limited. Luna: So if you have a hot file that thousands of processes read, you'd want to avoid FUSE for that. Lucas: Exactly. That's why you'd never run a web server's document root on FUSE. But for things like configuration management, backup retrieval, or even container image layers — tools like umoci use FUSE to mount OCI images without root — it's a great fit. Luna: Speaking of containers, have you played with LXC or Docker using FUSE for overlay filesystems? I've seen some projects that use fuse-overlayfs instead of the kernel overlay module. Lucas: Yes. fuse-overlayfs is useful in environments where you can't load kernel modules — like shared hosting or some CI runners. It's slower, but it works. The same is true for sshfs. You can mount a remote server's filesystem over SSH without any special kernel support on either side. Just SSH access. Luna: And that's why I think every sysadmin should understand FUSE even if they don't use it daily. It's a tool that gets you out of a corner when kernel modules aren't an option. Lucas: Right. And honestly, if today's episode gave you something usable — maybe you're thinking about that CI storage problem or an encryption use case — that's exactly the kind of practical value this show aims for. If it was worth a coffee to you, the link is buy me a coffee dot com slash fexingo. No pressure, just a way to keep the show ad-free and focused on content like this. Luna: It really does help. We don't run sponsors, so listener support is what keeps the server lights on, so to speak. Lucas: Exactly. So back to FUSE — one more thing I want to touch on is debugging. If you're writing your own FUSE filesystem, which I've done a few times, you can strace the daemon process directly. Luna: Oh, that's a big advantage over kernel modules. For a kernel filesystem, you'd need to use ftrace, printk, or crash dumps. Lucas: Right. With a FUSE daemon, you can literally strace -p <pid> and see every read, write, open call it receives. You can also use libfuse's debug mode to log all operations. The learning curve is much gentler. Luna: What libraries are people using to write FUSE filesystems these days? I know libfuse is the classic C library, but there are bindings for Python, Go, Rust... Lucas: libfuse version 3 is the current stable. For Python, there's fuse-python and pyfuse3. For Go, bazil.org/fuse is popular. Rust has fuser. They all wrap the same underlying protocol though — they open /dev/fuse and implement the callback interface. Luna: So if someone wanted to write a simple FUSE filesystem that, say, mirrors a REST API as a directory tree, how many lines of code would that take? Lucas: In Python with pyfuse3, maybe 150–200 lines for a read-only filesystem. You implement getattr, readdir, and open/read. That's it. The library handles the kernel communication. It's almost suspiciously easy. Luna: It feels like one of those 'too good to be true' things. But then you hit the performance wall. Lucas: Right. And that's the trade-off you need to evaluate per use case. For low-frequency access, or when the alternative is a complex kernel module you'd have to maintain across kernel versions, FUSE is a clear win. For high-performance storage, you stay in kernel space. Luna: So the mental model I take away is: FUSE is the duct tape of Linux filesystems. It's not pretty, it's not fast, but it lets you build custom storage behaviors quickly and safely. Lucas: I love that analogy. And like duct tape, you should know where it works and where it doesn't. Use it for the 'glue' problems — mounting cloud storage, overlaying directories, encrypting on the fly — and leave your production database on ext4 or XFS. Luna: One last thing — security. Since the FUSE daemon runs as a regular user, the access controls are based on that user's permissions. Is there any risk of privilege escalation? Lucas: Good question. By default, only the user who mounted the filesystem can access it — even root can't bypass that unless the mount was done with the 'allow_other' option. And if you use allow_other, you should also set 'default_permissions' so the kernel checks the daemon's uid/gid. Otherwise, the daemon could lie about permissions. Luna: So it's secure but you have to read the man page. As usual. Lucas: As usual. But the flexibility is unmatched. I think FUSE is one of those technologies that every sysadmin should have in their back pocket, even if they only reach for it once a year. Luna: Agreed. And now I'm thinking about that REST API filesystem idea again. Might prototype it this weekend in Rust. Lucas: Do it. And let us know how it goes — we might do a follow-up episode on writing a FUSE filesystem from scratch. Luna: That would be great. For now, I think we've given listeners a solid mental model of what FUSE is, when to use it, and when to steer clear. Lucas: Exactly. So that's FUSE — Filesystem in Userspace. A tool that turns your user-space code into a mountable filesystem, no kernel hacking required.