Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How the UK Pensions Dashboard Project Reached Live Service
Transcript
- Lucas: So the UK's Pensions Dashboard went live in March 2026 — and for a lot of people outside of gov tech, this thing flew under the radar. But technically, it's one of the most ambitious federated data projects ever attempted in the public sector. Luna: Ambitious partly because the problem itself is huge — something like thirty million people in the UK have multiple pension pots, often scattered across providers that still run on mainframes from the eighties. Lucas: Right. And the dashboard doesn't centralise all that data. It's a federated model: you log in, the dashboard queries tens of thousands of provider endpoints through a standardised API, pulls back your pensions, and presents them in one view. No central database of pension records. Luna: That's a deliberate architectural choice. The government didn't want to build a honeypot of every citizen's retirement savings. So the data stays with the providers, and the dashboard just becomes the query layer. Lucas: Exactly. But making that work means solving two hard problems. First, identity matching — how do you take someone logging in through gov.uk Verify and reliably match them across fifty thousand separate systems that have never talked to each other? Luna: And the second is API compliance — because you're asking thousands of pension providers, many with minimal technical teams, to expose a consistent interface that meets security and latency requirements. Lucas: Let's start with identity, because I think it's the more interesting technical story. The dashboard team built a probabilistic matching engine — not a deterministic one. So it doesn't require a single national insurance number to match perfectly. Instead, it uses a combination of name, date of birth, address, and NI number, scored against confidence thresholds. Luna: Which matters because the data quality across providers is wildly inconsistent. One provider might have your middle name, another just initials, a third might still have your address from 1995. Lucas: Right. So the engine assigns a probability score to each potential match. If the score is above 98.5 percent — that was the target they set — it's considered a match and the dashboard automatically displays that pension. Below that threshold, it flags for manual review or asks the user to verify additional details. Luna: And that 98.5 percent — is that high enough? Because in a system with fifty thousand endpoints, even a one percent mismatch rate could mean hundreds of thousands of people not seeing a pension they're entitled to. Lucas: It's a real tension. In early pilots, they found that setting the threshold too high — say 99.9 percent — caused too many false negatives, especially for women who changed their surname after marriage. So 98.5 was a pragmatic compromise. They also built a user-initiated search feature: if your dashboard doesn't show a pension you think you have, you can trigger a deeper search that relaxes the matching rules. Luna: That's smart. It puts the user in the loop without overwhelming them with false positives upfront. Lucas: Now, the API side. The standard is something called the Pensions Dashboard API specification — it's built on RESTful principles with JSON payloads, but the really interesting part is the security model. Every provider endpoint has to be registered in a central directory, and each request carries a digital signature verified against a public key stored in that directory. Luna: So it's a kind of mutual TLS with an additional signing layer. Prevents a bad actor from impersonating either the dashboard or a provider. Lucas: Exactly. And the latency requirement is tight: the dashboard has to return results within fifteen seconds, even if it's querying dozens of providers for a single user. That forced some interesting design decisions around parallelisation and timeouts. They run queries in batches of ten, with a four-second timeout per batch. If a provider doesn't respond, it gets marked as unavailable and the user sees a note saying 'some pensions may be missing.' Luna: Which is honest, but also a bit of a letdown if you're a user hoping for a complete picture. Lucas: Better than holding up the entire response for one slow provider. And the dashboard team published that data — they track provider response times and uptime publicly, which creates a gentle pressure on laggards to improve. Luna: If today was actually useful to you, the way these stay ad-free is listener support — buy me a coffee dot com slash fexingo. Lucas: Yeah, it's a small thing that makes a big difference for keeping the conversation going. Luna: So back to the dashboard — the initial rollout in March only covered fourteen providers, but the plan is to scale to over fifty thousand by 2028. That's a massive onboarding pipeline. Lucas: It is. And the team designed a tiered compliance approach. Tier one providers — the big ones like Legal & General, Aviva — they had to meet the full spec from day one. Tier two and three get more time, with a simplified API for very small schemes that might have fewer than a hundred members. Luna: So there's an element of technical debt allowed for the long tail. A simplified API means less functionality — maybe no option to see contribution history, just the current pot value. Lucas: Correct. And the dashboard team built a test harness that providers can run against their API before going live. It simulates the full request-response cycle, including edge cases like missing NI numbers or duplicate records. They claim the median provider passes certification in about three weeks. Luna: Three weeks isn't bad, but for a small scheme run by a part-time administrator, that's still a significant lift. Lucas: Yeah, and that's where the government had to make a policy call: mandate compliance or make it voluntary. They went with a phased mandate — by 2028, all schemes must be connected. So the clock is ticking. Luna: Let's talk about scale. The dashboard is designed for thirty million users within five years. What does that mean for the infrastructure? Lucas: The core dashboard service — the one that handles authentication, session management, and the orchestration of provider queries — runs on AWS, containerised with Kubernetes. They did load testing for a peak scenario of ten thousand concurrent users, which they modelled based on retirement season: people turning sixty-five tend to check their pensions around their birthday. Luna: Interesting that they modelled the peak around a specific life event rather than just a random busy hour. Lucas: Yeah, they used ONS demographic data to simulate a cohort of people all hitting the dashboard within a few days of their birthday. The architecture handles it by scaling horizontally — each user session is stateless, so they can add pods as needed. The real bottleneck is provider endpoints, not the dashboard itself. Luna: Because each user might query dozens of providers, so the outbound connection pool is the constraint. Lucas: Precisely. They set a connection pool of five hundred concurrent outbound connections per dashboard instance, with a queue for overflow. If the queue grows beyond a certain size, they start shedding load by returning partial results. It's a graceful degradation pattern. Luna: I want to come back to the data model. Federated versus centralised — was this the right call? Because a centralised database would have made matching trivial and query performance predictable. Lucas: Politically, centralised was a non-starter. The data protection impact assessment flagged it as high risk. But technically, you're right — a federated model adds complexity, latency, and failure modes that a centralised system wouldn't have. The trade-off is privacy and security. Luna: And the fact that the data is updated by providers in real time — well, near real time. With a centralised store, you'd need a separate data pipeline to keep it current. Lucas: That's actually one of the clever parts of the federated approach: the dashboard always returns the live value from the provider. There's no stale data issue. But the cost is that every query is a live transaction, so you need the provider systems to be available. Luna: Which brings us to the uptime requirement. The dashboard availability service — the component that orchestrates all these queries — has a target of 99.99 percent uptime. That's less than an hour of downtime per year. Lucas: They achieved that by running active-active in two AWS regions — Ireland and London — with automatic failover. They also did something interesting with the provider directory: it's replicated in-memory on each dashboard instance, so even if the directory service goes down, existing sessions can still route queries. Luna: So the directory is essentially a distributed cache that gets refreshed periodically. That's a good pattern for a system where the provider list changes slowly — new registrations, but not every second. Lucas: Exactly. And the cache has a time to live of five minutes, so updates propagate within that window. During the March launch, they had zero unplanned downtime. That's a strong start. Luna: But the real test will be the ramp-up to fifty thousand providers. Each new provider introduces a new failure domain. What happens when one of them has a bad certificate or starts returning malformed JSON? Lucas: The dashboard has a circuit breaker pattern per provider. If a provider fails three consecutive requests, it's temporarily removed from the pool for that user session and flagged for investigation. The user sees a generic message: 'One of your providers is currently unavailable.' Luna: That feels like a sensible default. But doesn't it erode trust if users frequently see missing data? Lucas: It does. That's why the team publishes a provider health dashboard publicly. You can see which providers have been up or down over the last thirty days. It's a transparency play that also creates accountability. Luna: I like that. It's almost like a public service level agreement. If your provider is down a lot, you might consider switching providers — which is actually a market mechanism that improves the whole ecosystem. Lucas: Right. And that's the broader ambition here — not just a dashboard, but a nudge toward better pension data hygiene across the industry. The technical architecture is just the enabler. Luna: So where do you see this going next? I mean, after all fifty thousand providers are onboarded, what's the next technical challenge? Lucas: I think the next frontier is analytics. The dashboard currently shows you what you have, but it doesn't let you model what you'll need. The team has talked about adding a projection engine that uses your current pots and assumes growth rates to estimate retirement income. That would require storing user preferences — a step change in the data model. Luna: Which would push them closer to a centralised store, at least for user settings and projections. That's an interesting evolution from a purely federated system to a hybrid one. Lucas: Yeah. And it raises new privacy questions. But for now, the federated approach has proven it can work. The March launch was a genuine engineering milestone — one of the few times a large-scale government digital project actually shipped on time and within budget. Luna: And it gave us a real-world case study in how to design a federated identity and data system that balances privacy, performance, and scale. That's a playbook other countries are probably watching closely. Lucas: Absolutely. If you're building a national data-sharing infrastructure, the UK Pensions Dashboard is now one of the reference architectures. Luna: Alright, that's a good place to leave it. Thanks, Lucas. Lucas: Thanks, Luna.