Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Elasticsearch Powers Netflix's Search and Observe
Transcript
- Lucas: If these conversations are useful for what you're building or running, we're glad you're here. Today: how Netflix runs one of the largest Elasticsearch deployments on the planet — and why you should care even if you're not streaming video. Luna: When most people hear 'Netflix' and 'Elasticsearch', they think logs. And yes, Netflix ingests tens of billions of log events per day into Elasticsearch for observability. But it goes way deeper. Lucas: Right. Netflix actually runs two distinct Elasticsearch use cases in production. The first is the one everyone knows: the observability stack. That's logs, metrics, tracing data — all pumped into Elasticsearch clusters that feed their internal monitoring tools. Luna: And the second? Lucas: The second is search itself. Not the search you do on Netflix — that's a different system — but internal search. Think: engineers searching for a specific error across thousands of microservices, or a content manager finding a title by partial name. That's Elasticsearch under the hood. Luna: So Netflix is basically running Elasticsearch as both their logging backbone and their internal search engine. That's a lot of eggs in one basket. Lucas: It is, and they've learned some hard lessons. Let's start with the numbers. As of late 2025, Netflix ran over 150 Elasticsearch clusters totaling more than 6,000 data nodes. They store about 200 petabytes of data across all clusters. And they process roughly 10 billion indexing requests per day. Luna: Ten billion indexing requests per day. That's not a number you hear often. How do they even manage shard allocation at that scale? Lucas: That's where the architecture gets interesting. Netflix doesn't use Elasticsearch's default shard sizing. They have a very specific rule: each shard should be between 40 and 80 gigabytes. If a shard grows beyond 80 gigs, they split it. If it stays under 40 after a week, they merge it. Luna: That's a pretty tight band. Most teams I see just let Elasticsearch auto-allocate shards and hope for the best. Lucas: And that works fine at small scale. But when you're running tens of thousands of shards, uneven allocation kills performance. Netflix actually built a custom shard management tool called Elasticsearch Sizer. It runs daily and recommends rebalancing actions. Luna: So they're not just relying on Elasticsearch's built-in rebalancing. They have a separate system that tells them exactly which shards to move. Lucas: Exactly. And they couple that with index lifecycle management — ILM. For their observability data, they rotate indices every 24 hours. Older indices get moved to warm nodes, then cold nodes, then deleted after 30 days. That prevents the cluster from being overwhelmed by stale data. Luna: Thirty days retention for logs — that seems short. Don't they need historical data for trend analysis? Lucas: They keep longer-term aggregates in a separate data store — a time-series database. The raw logs only stay in Elasticsearch for 30 days. And they made that decision after a painful incident in 2022. Luna: What happened? Lucas: One of their largest Elasticsearch clusters — the one ingesting all CDN logs — hit a critical disk usage threshold during a traffic spike. The cluster started rejecting writes. That meant Netflix lost visibility into streaming quality for about 45 minutes during prime time. Luna: Losing observability during peak hours — that's a nightmare for any engineering team. Lucas: Postmortem revealed two root causes. First, they had no shard size limit at the time — some shards had grown to 200 gigabytes. Second, they had no ILM policy for that cluster. Logs were being retained indefinitely. Luna: So the fix was shard sizing plus ILM. That's textbook, but apparently easy to neglect when you're moving fast. Lucas: Right. And Netflix documented that incident publicly. It's a great case study for anyone running Elasticsearch in production. They also revamped their write path after that. They introduced a buffering layer using Kafka. Luna: So instead of writing directly to Elasticsearch, they write to Kafka first? Lucas: Yes. The applications emit logs to Kafka topics. Then a consumer — they call it the Elasticsearch Sink — reads from Kafka and writes to Elasticsearch in batches. That decouples the producers from the cluster and absorbs traffic spikes. Luna: That pattern is common in large-scale logging pipelines. But Netflix also uses Elasticsearch for their internal search — the one engineers use to find error messages, deployment metadata, that kind of thing. Lucas: That's a different cluster, with different indexing patterns. For the observability clusters, they optimize for write throughput. For the search clusters, they optimize for query latency. The search cluster indexes metadata — not raw logs — so the documents are smaller, and the queries are more complex. Luna: And they're searching across millions of documents in sub-second time. That puts pressure on the query layer. Lucas: Exactly. Netflix uses a feature called cross-cluster search to query multiple Elasticsearch clusters from a single endpoint. That way, an engineer can search across all microservice logs without knowing which cluster holds the data. Luna: Cross-cluster search is built into Elasticsearch, but it has latency implications. How does Netflix keep it fast? Lucas: They limit the number of clusters queried per search — usually to five or fewer. And they use index aliases to route queries to the right indices. They also cache frequently-run queries at the application layer. Luna: So it's not just throwing Elasticsearch at the problem. They have a whole orchestration layer on top. Lucas: And that orchestration layer is where most of the engineering effort goes. The Elasticsearch clusters themselves are relatively standard — they run on AWS EC2 instances with local SSDs. The magic is in how they manage them. Luna: One thing that surprised me when I read their tech blog: they don't use Elasticsearch's security features. They handle authentication and authorization at the proxy layer. Lucas: Right. They have a custom proxy called Elasticsearch Proxy — creative name — that sits in front of every cluster. It handles TLS termination, authentication, rate limiting, and query rewriting. That way, the Elasticsearch nodes themselves don't need to manage security. Luna: That also means they can upgrade Elasticsearch versions without worrying about security config changes. The proxy is the stable layer. Lucas: Exactly. And they've been aggressive about version upgrades. They try to stay within two minor versions of the latest Elasticsearch release. That gives them access to performance improvements and bug fixes without being on the bleeding edge. Luna: Honestly, the level of operational maturity here is impressive. Most orgs I talk to are still running Elasticsearch 6.x and hoping nothing breaks. Lucas: And that's a risk. Elasticsearch has had some serious bugs in older versions — like the 'split brain' issue in versions prior to 7.x, where a network partition could cause two clusters to believe they were the master. Netflix automated their upgrade process with rolling restarts and zero-downtime deployments. Luna: So what's the one takeaway for a CTO who's considering Elasticsearch for their own stack? Lucas: I'd say: don't treat Elasticsearch as a black box. It's a distributed system with sharp edges. You need to invest in shard management, index lifecycle policies, and a robust write path. If you're at Netflix's scale, you need to build your own tooling. But even at smaller scale, the principles apply. Luna: And if you're not at Netflix's scale, you can still learn from their mistakes. Set shard size limits early. Implement ILM from day one. Use a buffer like Kafka if your write volume is unpredictable. Lucas: And if today's episode helped you think through a problem in your own infrastructure — honestly, if it was worth a coffee to you, that's the link: buy me a coffee dot com slash fexingo. The smallest possible ask, and it keeps this show ad-free. Luna: Yeah, it really does. And we appreciate every single one. Lucas: So back to Netflix. The last thing I'll mention is that they're now experimenting with Elasticsearch's vector search capabilities for similarity search across internal documents. That's a whole new use case — one that might change how engineers find relevant code or incident reports. Luna: Vector search in Elasticsearch — that's a 2024 feature, right? It's still relatively new. Lucas: It is. But Netflix is already testing it in a small cluster. If it works, it could replace some of their existing similarity search tools. And that's the thing about Netflix — they're never done optimizing. They treat their infrastructure as a product. Luna: And that's a mindset that scales. Thanks, Lucas. Until next time. Lucas: Thanks, Luna. Take care.