Latest / Linux Server Admin with Fexingo: Sysadmin, Bash, and Server Engineering / How to Use Linux Server Threat Detection with Falco
Transcript
- Lucas: If you manage Linux servers long enough, you eventually have that moment where you SSH in and something feels off. A process you don't recognize, a weird outbound connection, a cron job you didn't set up. Luna: Right, and by the time you notice, the attacker might have been there for days. What you need is something that alerts you in real time, not just when you review logs. Lucas: Exactly. And that's where Falco comes in. It's an open-source runtime security tool, originally created by Sysdig and now a CNCF graduated project. It monitors system calls and kernel events to detect suspicious behavior as it happens. Luna: I've seen it used mostly for containers, but it works on bare-metal and VMs too, right? Lucas: Absolutely. Falco hooks into the Linux kernel using eBPF — that's extended Berkeley Packet Filter — or a kernel module. It watches for things like file changes, network connections, process spawns, and privilege escalation. When something matches a rule, it generates an alert. Luna: So you get alerted instantly instead of digging through audit logs later. That's the dream. Lucas: Let's make it concrete. Imagine you run a web server that's been compromised. An attacker uploads a PHP shell and then tries to spawn a reverse shell back to their C2 server. If you have Falco running, it catches that execve syscall — the execution of a new binary — and flags it because it's coming from a web server process that shouldn't be spawning shells. Luna: Right, because normally Apache or Nginx shouldn't be running /bin/bash or /usr/bin/nc. That's a clear anomaly. Lucas: And Falco ships with a rich set of default rules covering the MITRE ATT&CK framework. Things like 'Launch Suspicious Network Tool in Container', 'Read Sensitive File Untrusted', 'Unexpected outbound connection from a privileged container'. You can use those out of the box. Luna: But you're probably going to want to write custom rules for your environment, right? Every setup is different. Lucas: That's the next step. Falco rules are written in YAML. They have a condition field that uses a simple syntax. For example, you might have a rule that alerts when any process reads /etc/shadow except the login process. The condition is 'open_read and fd.name=/etc/shadow and not proc.name in '. It's pretty readable. Luna: So you can tune it to reduce false positives. Because a default rule might fire on things you actually expect. Lucas: Right. And that's important — you don't want alert fatigue. Let me walk through installing Falco on a typical Ubuntu 24.04 server. First, you add the Falco repository, install the falco package, and it grabs the kernel module or eBPF probe automatically. Then you enable and start the service. Luna: And does it work with secure boot? Because that can be a pain with kernel modules. Lucas: Good question. With secure boot, you either need to sign the kernel module or use the eBPF probe instead, which is often easier. Falco can use eBPF if you set the environment variable FALCO_BPF_PROBE to a blank string. On newer kernels, eBPF is the recommended path anyway. Luna: Once it's running, how do you see alerts? Do you tail a log file? Lucas: By default, Falco outputs to syslog, which on most systems goes to /var/log/syslog. But you can also configure it to write to a file, send to stdout, or integrate with things like Slack, PagerDuty, or a SIEM via JSON output. I like to use the falcoctl tool to manage outputs. Luna: And for centralized logging, you could pipe it into your systemd journal, like we talked about in episode 64. Lucas: Exactly. In fact, I set up a rule that sends Falco alerts to the journal with a specific priority, and then I have rsyslog forward those to a central log server. That way all my servers' Falco alerts are in one place. Luna: Let's talk about a real rule example. You mentioned reverse shells. Can you write a rule that specifically catches a reverse shell attempt? Lucas: Sure. A reverse shell often uses /bin/bash with a redirect to a remote IP. A simple rule could be: condition: 'evt.type=execve and proc.name in and evt.arg.flags contains -i and fd.sockfamily=ip and fd.type=ipv4'. That catches an interactive shell that's connected to a network socket. But you'd want to exclude legitimate cases like SSH sessions. Luna: That seems doable. What about privilege escalation attempts? Like using sudo or su? Lucas: Falco has a default rule for that: 'Unexpected Privilege Escalation'. It triggers when a process tries to change its user or group to root, excluding expected paths like sudo or runuser. But you can refine it. For example, you might want to alert specifically if a web server process calls setuid. Luna: So it's a matter of understanding your baseline. What's normal for your environment, and what's not. Lucas: Right. And to help with that, Falco has a feature called 'Falco Event Generator' which you can run to simulate attacks and see if your rules fire. It's a good way to test your configuration before going live. Luna: I've also heard about Falco libraries — libsinsp and libscap — that let you embed Falco into your own tools. Is that something a sysadmin would use? Lucas: Probably not directly. Those are more for developers building security platforms. But as a sysadmin, you can use the falcoctl tool to manage Falco deployments across many servers. It helps with rule updates and configuration drift. Luna: One thing I wonder about is performance overhead. If you're monitoring every system call, doesn't that slow things down? Lucas: It's a concern, but eBPF is designed to be efficient. In practice, the overhead is usually less than 5 percent on CPU, and often lower. Sysdig did benchmarks showing around 2 to 6 percent overhead depending on the workload. For most production servers, that's acceptable given the security benefit. Luna: And you can also limit what Falco monitors using 'falcoctl rules' to only watch specific containers or processes. Lucas: Exactly. You can use 'macro' definitions to scope rules. For example, define a macro called 'trusted_images' that lists your approved container images, then exclude those from certain alerts. That's a best practice. Luna: Let's pivot to a real-world scenario. Say you have a honeypot container that's meant to attract attackers. You'd want Falco to alert loudly if anything touches it. Lucas: Absolutely. You can label that container with a specific tag and then write rules that apply only to that tag. For instance, a rule that triggers on any outbound connection from a container with label 'honeypot:true'. That way you get high-signal alerts. Luna: And with Falco, you can even set the output to include extra fields like the container name and image, so you know exactly what's happening. Lucas: Right. The output format is customizable. You can include %proc.name, %container.name, %fd.name, whatever you need. That makes the alerts actionable immediately. Luna: I want to talk about something that I think a lot of listeners might not realize — Falco isn't just for production. It's great for dev environments too. You can catch insecure practices before they go live. Lucas: That's a great point. You can run Falco in CI/CD pipelines. There's a tool called falco driver loader that can be used in containers, and you can have Falco rules as part of your Kubernetes admission controllers. It's a shift-left security tool. Luna: So if someone is new to Falco, what's the quickest way to get value? Install it, let it run with default rules, and then look at the alerts? Lucas: Exactly. Start with the defaults. They're pretty good. After a week, review the alerts and see what's noise. Then start writing exclusions or custom rules. The Falco community has a great rule library on GitHub. You'll find rules for specific applications like MySQL, Apache, Nginx. Luna: I also like that Falco integrates with Kubernetes. It can monitor pod lifecycle events and security contexts. That's huge for cluster admins. Lucas: Yeah, the K8s integration is deep. Falco can detect if a container is running with privileged mode or if a pod is mounting the host filesystem. Those are common misconfigurations that attackers exploit. Luna: I think we've covered a lot of ground. Let's wrap up with a practical piece of advice. What's one thing someone should do after installing Falco? Lucas: Set up a way to receive alerts in real time. Whether it's a Slack webhook, an email, or a syslog server. Don't just let them sit in a file. Because if no one sees the alert, it's as good as not having it. Luna: And test it. Use the event generator to simulate an attack and confirm you get notified. Lucas: Right. Falco gives you visibility into your runtime security that you can't get from traditional log analysis alone. If today's tech conversation gave you something usable, the way these episodes stay ad-free is listener support. If you find value in the show, you can buy me a coffee at buy me a coffee dot com slash fexingo. Luna: Yeah, it really helps keep the content coming without any ads. We appreciate every bit of support. Lucas: Back to Falco — one more thing I want to mention: the project has a great online simulator called 'Falco Playground' where you can write and test rules in your browser. No installation needed. Luna: That's perfect for learning. You can try out conditions and see what alerts they generate instantly. Lucas: So if you're serious about Linux server security, give Falco a try this week. Install it on a test server, run the event generator, and see what you catch. Luna: And then tune it for your production environment. You'll sleep better knowing you have real-time threat detection. Lucas: On that note, I'll leave you with this: the best security tool is the one you actually use. Falco is easy to install and its default rules are solid. Start there, and build on it.