Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How NASA Rebuilt Mission Control for Artemis Lunar Ops
Transcript
- Lucas: A few weeks ago I was reading about the Artemis III milestones and something caught my eye — not the rocket or the lander, but the mission control center itself. NASA spent the last five years quietly replacing almost the entire software stack that flight directors use to talk to the spacecraft. Luna: The Apollo-era consoles? I remember visiting Johnson Space Center as a kid and seeing those grey boxes with the green text scrolling up. They looked like they belonged in a submarine. Lucas: That's exactly what they were, essentially. The core system — called the Mission Control Center, or MCC — had been running on IBM mainframes since the 1960s. Over decades, layers of emulation and wrappers were added, but the fundamental architecture was designed for a world where you controlled one spacecraft at a time with a handful of telemetry streams. Lucas: For Artemis, that world is gone. They're managing the Orion capsule, the Gateway station, landers, rovers, and a growing swarm of cubesats all at once. The old system could handle about 30 spacecraft simultaneously. Now they're planning for thousands of data sources per mission. Luna: So what did they build instead? I'm imagining a cloud-native ground station, but I can't see NASA running critical ops on a public cloud without some serious air-gapping. Lucas: You're on the right track. The new system is called MCC-21 — Mission Control Center for the 21st century. It's a hybrid architecture. The core command and telemetry processing runs on a private cloud inside Johnson Space Center, but they use commercial cloud services for data analytics, simulation, and backup. The interesting part is the data pipeline. Lucas: Traditionally, telemetry data came in packets every second and was stored in a flat file system that flight controllers would poll for updates. Polling meant there was always a lag — sometimes half a second, sometimes longer. MCC-21 flipped that to an event-driven model. Sensors push data into Kafka topics, and each console subscribes to exactly the streams it needs. Luna: So the flight dynamics officer doesn't get flooded with life-support data they don't care about. That's a big shift from the old 'everything to everyone' approach. Lucas: Exactly. The old philosophy was 'broadcast all telemetry to all consoles because you never know what might be useful.' That works when you have 30 channels. When you're looking at 10,000 concurrent sensor readings per vehicle, it becomes noise. The new system uses a metadata registry called the Telemetry Dictionary Service that lets each console subscribe with filters. Push instead of pull. Luna: And the latency improvement? Lucas: They measured a 40% reduction in end to end latency from the sensor to the display. For an alarm that needs a flight director to react within seconds, that's the difference between catching an anomaly and logging a failure. The mainframe cycle time was about 200 milliseconds just to process the packet. The new pipeline is under 50 milliseconds for most data. Lucas: But the hardware story is almost as important. The old consoles were custom-built CRT terminals that cost hundreds of thousands of dollars per unit. The new consoles are essentially COTS — commercial off-the-shelf — PCs running a web-based display layer called MCC Web. Each station has three monitors, but the software is identical across all of them. Luna: That must have been a cultural shock. I've read that veteran flight directors had muscle memory for those green-on-black screens. They could read a page of numbers in under a second. Lucas: Absolutely. And NASA did something smart — they didn't force the switch overnight. They ran the old and new systems in parallel for three years. There was what they called the 'glass cockpit' test, where during simulations, half the controllers used MCC-21 and half used the legacy system, and they compared response times and error rates. The new system won on almost every metric after an initial learning curve. Luna: What about reliability? The legacy system had a reputation for being bombproof. It ran on a triple-redundant mainframe setup, and I remember hearing it had never had a full outage during a mission. Lucas: That's the gold standard they had to match. The MCC-21 team targeted five nines — 99.999% uptime. They achieved it by designing for 'graceful degradation'. Each subsystem is deployed as a set of microservices that can fail independently. If the telemetry streaming service goes down, the command uplink service keeps running, and controllers get a degraded but still functional interface. The old mainframe was an all-or-nothing proposition. Luna: And the digital twin aspect — I saw a press release about them building a 'virtual Orion' for testing. Is that part of the new architecture? Lucas: It is, and it's one of the most radical changes. The old way was to run hardware-in-the-loop simulations — you'd have a physical mock-up of the spacecraft connected to the flight software, and you'd test commands against it. That's expensive and slow. MCC-21 uses a real-time digital twin that mirrors the actual spacecraft state. Before a critical command is sent to Orion, it can be tested against the twin in milliseconds. Lucas: This came into play during Artemis I, when there was an anomaly with the star tracker. Flight controllers used the twin to run a hundred different scenarios in under an hour, confident that the fix would work before they uplinked it. That kind of iteration was impossible with the old system. Luna: So the architecture fundamentally changed the speed of decision-making during a mission. That's a huge deal. Lucas: It is. And this is a pattern we're seeing across critical infrastructure — the old model of 'one big computer that does everything, tested against one big simulation' is giving way to distributed, event-driven systems with digital twins. NASA's case is instructive because the stakes are existential, yet they still managed a migration. Luna: On that note — and I think this ties in — it takes resources to produce deep dives like this. We keep the show ad-free and listener-supported, and a handful of listeners chip in monthly through Buy Me a Coffee. That's what funds our ability to research episodes like this one. Lucas: Yeah, and it's not about big pledges — it's just that listener support means we don't have to chase advertisers or clickbait. It keeps the focus on substance. Luna: Exactly. And if this kind of architecture conversation is useful for what you're building or running, that's the whole point. Lucas: So back to MCC-21 — one more interesting detail. The system is built on a Kubernetes cluster running in a private data center, but they use a commercial cloud provider for what they call 'edge analytics'. When a sensor stream shows an anomaly, the data is mirrored to a cloud instance for machine learning model inference, and the result is pushed back into the console display. That feedback loop runs in under 200 milliseconds. Luna: That's impressive for a government system. Security must have been a nightmare to get through the approval process. Lucas: It took three years just to get the security architecture certified. They had to build a 'data diode' — a hardware device that ensures data can only flow out of the control center, not in. The ML inference results come through an encrypted channel that's strictly one-way. No possibility of an injection attack from the cloud into the command network. Luna: So the cloud is essentially a read-only input for the mission control system. That's a smart way to get the benefits without the risk. Lucas: Exactly. And it's a model a lot of large organizations could learn from. You don't have to move everything to the cloud to benefit from cloud-native patterns. NASA kept the control loop local and used the cloud for computation that's hard to do on-premise. Luna: What about the human side? How do you retrain a flight director who's been reading green-on-black for two decades to use a web interface with real-time dashboards and drag and drop configuration? Lucas: They ran a year-long transition program. Every console had a 'buddy system' — an experienced flight controller paired with a younger engineer who was native to the new system. They also kept the old system running as a fallback for the first Artemis mission. But what surprised leadership was that after a few months, the veterans started finding new ways to use the data. For example, they created custom alert rules that combined multiple telemetry streams — something the old system couldn't do. Luna: So the architecture change actually unlocked new operational capabilities that the old system couldn't support. Lucas: Exactly. The legacy system was not just slow — it was rigid. You couldn't define a cross-sensor alert without a software change that took months. In MCC-21, a flight controller can configure a new alert rule in minutes using a web UI. That flexibility is going to be critical as Artemis missions get longer and more complex. Luna: What's next for the architecture? There's talk of a permanent lunar base and Mars missions. Does this system scale to those? Lucas: That's exactly the design target. The system is built to handle a 500-millisecond round-trip delay to the Moon and up to 20 minutes to Mars. For deep space, they're planning a 'store and forward' capability where the spacecraft carries a local digital twin and only sends critical telemetry back to Earth. The ground system needs to handle asynchronous data streams gracefully, and MCC-21's event-driven architecture is a good foundation for that. Luna: It's a fascinating case study in modernizing a legacy system where failure is not an option. And I think the lessons apply far beyond space. Lucas: Absolutely. Whether it's a financial exchange, a power grid, or a hospital system, the pattern is the same: you can preserve reliability while gaining agility, but you have to be thoughtful about the migration. NASA took a decade, ran parallel systems, invested in training, and designed for graceful degradation. That's the playbook. Luna: And they're now flying missions on it. Artemis II is scheduled for next year, and the flight directors will be using MCC-21 as the primary system for the first time with crew on board. Lucas: That's the real test. But from everything I've seen, the architecture is solid. The numbers — 40% latency reduction, five-nines uptime during simulations — speak for themselves. It's a rare example of a government-led digital transformation that actually delivered on its promises. Luna: It's a good reminder that the technology is only half the story. The culture and the migration strategy matter just as much. Lucas: Exactly. That's the part that often gets overlooked in architecture discussions. We'll keep an eye on how MCC-21 performs on Artemis II and report back.