Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Anthropic Built Claude 3 for Enterprise Reliability
Transcript
- Lucas: Claude 3, Anthropic's latest large language model, hit general availability back in March, and what struck me wasn't just the benchmark scores — it was how they talked about reliability from day one. Luna: Yeah, I noticed that too. Most AI launches emphasize capability — 'look what it can do.' Anthropic led with uptime and consistency. Lucas: Exactly. And that's a fundamentally different engineering posture. So today I want to unpack one specific architectural decision: how Anthropic shards inference across GPU clusters to maintain sub-second response times even when individual nodes fail. Luna: Sharding at inference time — that's unusual. Most models run on a single GPU or a small tensor-parallel group. Why shard? Lucas: Right, so Claude 3 comes in three sizes — Haiku, Sonnet, and Opus. Opus is the largest, with something on the order of two trillion parameters, though they haven't confirmed the exact number. Running that on a single GPU is impossible, so they use a technique called 'pipeline parallelism' across dozens of GPUs, but they also add 'expert parallelism' — each request only activates a subset of the model. Luna: So it's a mixture of experts architecture under the hood? Lucas: That's the common assumption, yes. But the reliability angle is what I find interesting. They don't just distribute the computation; they also replicate critical routing layers. If one GPU in the pipeline stalls, the system can re-route through a hot spare within milliseconds. They've published that they target 99.95 percent uptime on the API — and that's not just infrastructure uptime, that's successful inference requests. Luna: That's a much higher bar. Most LLM APIs have degraded performance or timeouts under load. How do they handle the routing? Lucas: They built a custom 'request router' that sits in front of the model cluster. It doesn't just load-balance — it keeps a real-time map of which GPU shards are healthy, their current queue depths, and their recent error rates. A request comes in, the router decides which set of experts to activate and which physical GPUs to send each layer to. If a GPU starts returning errors above a threshold, the router blacklists it and shifts traffic. Luna: That sounds like a lot of state to maintain across a cluster that could be hundreds or thousands of GPUs. Lucas: They've mentioned using a distributed key-value store — something like etcd or Consul — to share health state across routers. The interesting trade-off is that the router itself becomes a potential single point of failure, so they run it as a replicated service with a leader election. They've said the failover happens in under a second. Luna: And all this is before you even get to the safety layer — the constitutional AI stuff. Lucas: Right, and that's the second architectural layer I want to explore. Constitutional AI isn't just a training technique — it's a runtime guardrail. Anthropic runs a separate 'critique' model that evaluates each response against a set of principles before it's returned to the user. That adds latency, and they had to optimize that too. Luna: So you have two models running in sequence for every request? That doubles inference cost. Lucas: It does, but they've been transparent about it. They say the critique model is much smaller — think Sonnet-sized — and they employ early-exit strategies. If the primary model's response is straightforward and low-risk, the critique model can skip full evaluation and just check a hash. They also batch critique evaluations across multiple requests to improve GPU utilization. Luna: That makes sense as an engineering trade-off. But I wonder how enterprise customers react to the added latency for safety. Lucas: From what I've seen, enterprises are willing to accept an extra 200 to 500 milliseconds if it means the model won't accidentally leak sensitive information or generate harmful content. In fact, some customers have asked for even more conservative guardrails — they want a higher bar for refusal than Anthropic's default. Luna: So Anthropic gives them knobs to tune the safety thresholds? Lucas: They do, through what they call 'system prompts' — controllable parameters that influence the critique model's strictness. But the engineering challenge is ensuring that loosening the guardrails doesn't introduce unpredictable behavior. They've invested heavily in regression testing suites that simulate millions of edge cases before any change goes live. Luna: That level of testing requires massive compute resources. How does their engineering org balance rapid iteration with that kind of safety rigor? Lucas: It's a deliberate tension. They've structured the org into two chains: a product engineering team that pushes new features and model updates, and a safety engineering team that acts as a gate. The safety team runs its own CI/CD pipeline that must pass before any deployment reaches production. They've said that about 30 percent of their total engineering capacity goes to safety and reliability infrastructure. Luna: That's a huge allocation. Most startups would consider that overhead. Lucas: But it's also their moat. Enterprise buyers in regulated industries — healthcare, finance, legal — won't touch a model that doesn't have documented reliability guarantees. Anthropic's pitch is that they've baked that into the architecture from the start, not bolted it on later. Luna: It also helps that they have a clear mission. The company's public focus on safety aligns the engineering incentives. If your CTO is constantly saying 'safety first,' then engineers don't push back when they have to write more tests. Lucas: And the numbers back them up. According to their published uptime reports, Claude 3's API has maintained over 99.9 percent availability since launch, with average response times under one second for the Haiku model and under three seconds for Opus. That's competitive with any major AI provider. Luna: I want to zoom in on one more thing: the sharding strategy for model updates. How do they deploy a new version without taking the whole system down? Lucas: Great question. They use a blue-green deployment pattern at the model level. They maintain two complete inference stacks — the current production version and the staging version running the new weights. Traffic is gradually shifted from blue to green, and if error rates spike, the router automatically rolls back. They've also invested in 'shadow testing' where a percentage of live requests are sent to both stacks, but only the production stack's response is returned to the user. The other response is logged for evaluation. Luna: That's an expensive way to test, but it catches regressions that unit tests never would. Lucas: Exactly. And it's a pattern that's becoming standard in high-stakes ML deployments. Look, I think there's a broader lesson here for CTOs. When you're building a product that involves AI, the tendency is to prioritize capability — making the model smarter. But Anthropic is proving that reliability can be a competitive advantage if you design for it from the architecture level. Luna: It's a different mindset. Instead of asking 'can we add this feature?' you ask 'can we guarantee this feature works under failure?' Lucas: Right. And we should mention that these conversations — the detailed architectural trade-offs, the org design — those are exactly the kinds of discussions we aim to bring you on this show. We deliberately don't run ads. If you find value in that, and you want to support the choice to keep this ad-free, you can find us at buy me a coffee dot com slash fexingo. No pressure, just a way to keep this going. Luna: And we appreciate it. It lets us dive deep without worrying about sponsor messaging. Lucas: So back to Anthropic. One area where I think they still have room to improve is observability for customers. Right now, the API gives you basic latency and error metrics, but enterprises want more granular insights — like which safety guardrails triggered a refusal, or why a particular request was slow. Luna: Do they expose any internal telemetry? Lucas: Not yet publicly, but I've heard they're working on a detailed logging layer that would let customers see the full inference trace — which experts were activated, how long each pipeline stage took, and whether the critique model intervened. That level of transparency would be a game-changer for compliance teams. Luna: It would also help CTOs justify the cost. If you can show your CFO exactly why a request took 800 milliseconds and that it passed three safety checks, it's easier to allocate budget. Lucas: Exactly. And that's the kind of product thinking that comes from having CTOs in the room during design conversations. Anthropic's engineering leadership includes people who have built infrastructure at Amazon and Google, so they understand enterprise procurement. Luna: Let's talk about the competition. How does Claude 3's architecture compare with GPT-4 or Gemini? Lucas: The biggest difference is the explicit safety layer. OpenAI and Google have safety systems too, but they're less transparent and more heuristic-based. Anthropic's constitutional AI is a trained model, which makes it more predictable and auditable. On the infrastructure side, OpenAI uses a mix of in-house and Azure GPUs, while Anthropic is primarily on Google Cloud TPUs. That affects their sharding strategies because TPUs have different networking characteristics. Luna: TPUs are more tightly coupled, right? So maybe they rely more on tensor parallelism than pipeline parallelism. Lucas: That's likely. And it means their failover handling has to account for different failure modes — TPU pods can have whole-group failures, not just individual chips. Anthropic's routing layer would need to handle that gracefully. Luna: Given all this, what's your take on the future of enterprise AI reliability? Are we going to see more companies adopt Anthropic's approach? Lucas: I think so. As regulation tightens — the EU AI Act, potential US frameworks — reliability and safety will become table stakes. The companies that have already invested in the architecture will have a head start. But it's not cheap. The compute cost alone for running a critique model on every request is significant. Enterprises will have to decide if the premium is worth it for their use case. Luna: And that's where the market segmentation comes in. For high-stakes applications like medical diagnosis or legal document analysis, the premium is trivial. For a chatbot answering customer support tickets, maybe not. Lucas: Exactly. And that's why Anthropic offers three tiers. Haiku is cheap and fast, Opus is expensive and thorough. They're letting the market self-select based on reliability needs. Luna: It's a smart strategy. And it forces the rest of the industry to compete on reliability, not just on benchmark scores. Lucas: Which, frankly, is better for everyone building on top of these models. A more reliable foundation means fewer late-night incidents for engineering teams. Luna: Alright, that's a good place to wrap. Next episode, we're going to look at how a different company handled reliability — maybe on the database side. Lucas: Sounds good. Until next time.