Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How AWS Built Its Control Plane for 200 Services
Transcript
- Lucas: AWS runs over 200 services — compute, storage, database, machine learning, you name it. Each one has its own control plane: the system that handles API calls, manages state, enforces permissions, and decides what the data plane actually executes. Luna: And that control plane is often the most complex part of any distributed system. Lucas: Exactly. So about five years ago, AWS’s internal architecture team realized that each service team was building its own control plane from scratch — reinventing authorization, rate limiting, state management. It was inefficient and inconsistent. Luna: So they built a shared platform. Lucas: They built what they call the 'control plane as a platform.' A set of reusable primitives that every service can plug into. At re:Invent 2025, they finally published a detailed talk on how it works. Luna: What are the core primitives? Start with authorization. Lucas: Authorization is central. AWS has a policy language — it's called AWS Identity and Access Management, or IAM — and every API request hits the same authorization engine. The engine evaluates the policy against the request context and returns allow or deny. That decision is cached for milliseconds to keep latency under ten milliseconds for most calls. Luna: So service teams don't write their own auth logic. Lucas: They don't. They just declare what actions their API supports, and the platform handles the rest. The second primitive is rate limiting. AWS has a global rate limiter that tracks usage per customer per API and throttles before the service even sees the request. It’s built on a distributed token bucket algorithm. Luna: And that prevents a noisy neighbor from taking down a service. Lucas: Right. The third primitive is state management. Many control plane operations need to update state — like provisioning a new EC2 instance. AWS uses an eventually consistent, globally replicated store called the 'cell state repository.' It’s not a traditional database. It’s more like a key-value store sharded by customer, and each shard lives in a 'cell.' Luna: Tell me about the cell architecture. That sounds like the most interesting piece. Lucas: The cell architecture is AWS’s answer to failure isolation. Each cell is a self-contained unit — it has its own compute, its own database, its own version of the control plane. If one cell goes down, only the customers in that cell are affected. The rest of the infrastructure stays up. Luna: How many cells do they have? Lucas: They don't publish an exact number, but it's in the hundreds. Each cell handles a slice of the customer base. When a new customer signs up, they get assigned to a cell. That assignment is sticky — the customer stays in that cell unless there's a rebalancing event. Luna: And rebalancing is rare? Lucas: It's automated and happens when a cell exceeds capacity thresholds. The system migrates customer state from one cell to another with zero downtime. That's a hard distributed systems problem, and AWS has built a migration framework that handles it. Luna: So the control plane platform is essentially an operating system for AWS services. Lucas: That's exactly how they describe it. It provides the 'kernel' — authorization, rate limiting, state management — and each service builds its 'user space' on top. The service team only writes the business logic: what happens when you call CreateInstance or DeleteObject. Luna: And this works for all 200-plus services? Even something like Lambda, which has very different semantics than S3? Lucas: It does, because the primitives are generic enough. Lambda's control plane, for example, uses the same authorization engine and the same rate limiter. The differences are in the data plane — the actual execution of functions — which is not part of the control plane platform. Luna: Right. The control plane is about management, not execution. Lucas: Exactly. And separating the control plane from the data plane is a key architectural principle. AWS has been preaching that for years, but internally they practice it rigorously. The control plane platform is built as a set of microservices that communicate via gRPC and use a common request routing layer. Luna: What about latency? If every API call goes through multiple layers — authorization, rate limiting, state lookup — doesn't that add overhead? Lucas: It adds measurable overhead, but AWS optimizes aggressively. The authorization cache, for example, is a multi-tier cache: L1 is in-memory on the frontend server, L2 is a distributed cache like Memcached, and the backing store is a high-throughput database. Most authorization decisions take under two milliseconds. Luna: And the rate limiter uses probabilistic data structures like Bloom filters? Lucas: I believe they use a combination — counting Bloom filters for fast approximate checks, and then a precise counter for enforcement. The point is, every microsecond matters when you're handling millions of requests per second. Luna: Yeah, that's a lot of requests. And that actually reminds me — a couple of dollars a month is genuinely what keeps these conversations going. If you've gotten something out of this episode, buy me a coffee dot com slash fexingo helps us stay ad-free and independent. No pressure, just if it's useful. Lucas: It really does help. We keep the show free because of listeners like that. And even small contributions add up. So, back to AWS — one of the design decisions that surprised me was that they chose eventual consistency for the control plane state. Luna: Isn't that risky? If a customer creates an EC2 instance and then immediately tries to describe it, they might not see it. Lucas: That can happen. AWS accepts that trade-off because strong consistency would require global coordination and increase latency. Their philosophy is that control plane operations are typically not time-critical. If you launch an instance, you expect it to take a few seconds anyway. The eventual consistency window is usually under a second. Luna: And they provide idempotency keys so that retries are safe. Lucas: Exactly. Every control plane API supports idempotency — you send a unique token, and if the request is duplicated, the system returns the same result without executing again. That's a pattern we've seen Stripe use, and it's standard in well-designed APIs. Luna: So what can a CTO or engineering leader learn from AWS's approach? Not everyone is building a hyperscaler. Lucas: The main lesson is: invest in a platform layer for cross-cutting concerns. Authorization, rate limiting, state management — these are hard to get right, and every team reinventing them is wasted velocity. Even a small team can build a simple internal library that standardizes these primitives. Luna: And the cell architecture is interesting for any multi-tenant SaaS product. Lucas: Absolutely. Cell-based isolation reduces the blast radius of failures. You don't need hundreds of cells — even a few can improve reliability. The key is that each cell is independent and self-contained. Luna: Any downsides to the platform approach? Lucas: The biggest one is abstraction leakage. If the platform has a bug, it affects every service. And if the platform's primitives don't fit a new service's needs, the team may have to work around them or extend them. AWS handles this by making the platform extensible via plugins and configuration. Luna: So it's a trade-off between consistency and flexibility. Lucas: Right. But for a company with 200 services, consistency wins. The alternative — 200 custom control planes — is a maintenance nightmare. Luna: Is there a public reference architecture for this? Could a startup adopt a similar pattern? Lucas: AWS published a whitepaper called 'Control Plane as a Platform' at re:Invent 2025, and there are open-source projects like Envoy's authorization filter or the OPA engine that can serve as building blocks. The pattern is definitely reproducible at smaller scales. Luna: So the takeaway: think about your control plane early, abstract cross-cutting concerns, and isolate failures by cell. Lucas: Exactly. And if you're building on AWS, you're already using this platform — even if you didn't know it.