Latest / Tech Leadership with Fexingo: Engineering Managers, CTOs, and Technical Leadership Conversations / How One Team Decided Their Database Was the Wrong Database
Transcript
- Lucas: So there's this moment every CTO dreads — you're three years into a product, the database has a million writes a day, and you realize the database you chose is wrong. Not just suboptimal — structurally wrong for the data you're actually storing. Luna: And it's never a clean realization either, right? It's usually a pager going off at 2 AM. Lucas: Exactly. In this case, the company was a B2B SaaS platform — we'll call them 'FlowMetrics' — and they'd built the whole thing on MongoDB. Made sense at the time. Schema-less, fast to iterate, great for a two-person founding team shipping fast. Luna: Right, the classic 'move fast' choice. Lucas: But by year three they had eighteen engineers, a relational data model — customers, invoices, subscription plans, usage records — and MongoDB was the source of a growing number of production bugs. The trigger was a cascading replica failure during a routine backup. A secondary went down, the primary took the full read load, and suddenly every query that depended on a join — which MongoDB doesn't really do — started timing out. Luna: If today's conversation gave you something usable — maybe a framework for thinking about your own infrastructure — that's the whole reason we do this show ad-free. If it was worth a coffee to you, buy me a coffee dot com slash fexingo. Seriously, that's it. No tiers, no perks. Just keeping the lights on. Lucas: Yeah, it's the smallest possible ask. And it means we never have to sell ad space to a cloud vendor. Back to FlowMetrics — the CTO, Sarah, told me the moment she knew they had to migrate was when she realized the team had written a custom mongodb to postgresql sync tool just to run internal analytics. They were already maintaining two databases. Luna: So they were paying the cost of a migration without getting the benefit. Lucas: Exactly. The sync tool was brittle, it broke every other week, and it only covered a subset of the data. That was the signal: the cognitive load of working around MongoDB's limitations had exceeded the cost of a full migration. Luna: What was the specific technical mismatch? I mean, MongoDB is great for certain things. Lucas: The core problem was that FlowMetrics's data was inherently relational. A customer has multiple subscriptions, each subscription has multiple invoices, each invoice has line items. Queries that needed to join across those collections were slow and error-prone. They'd tried embedding documents, but the documents grew too large. They'd tried manual reference-based lookups, but that led to N+1 query patterns. And consistency was a nightmare — MongoDB offers eventual consistency by default, and they needed strong consistency for billing. They'd added write concerns and read concerns, but it added complexity without fully solving it. Luna: So the straw that broke the camel's back was the replica failure? Lucas: That was the event. But the real decision happened in the post-mortem meeting. Sarah told me she asked a simple question: 'If we were starting today, would we pick MongoDB again?' And everyone said no. That's when they committed. Luna: So how did they actually do it? Big-bang cutover? Lucas: They went with a strangler fig pattern. They built a thin translation layer — a proxy that sat between the application and the databases. For any given request, the proxy could route reads and writes to either MongoDB or PostgreSQL. They started with a single entity: the 'customer' collection. They migrated customer data first, ran both databases in parallel for a week, compared query results, and only when they were confident did they flip the read path to PostgreSQL. Then they moved on to subscriptions, then invoices, then line items. The whole process took about three months. Luna: Three months isn't that long for a full database migration. Lucas: It's not. But they kept the scope tight. No schema redesign during the migration. They essentially mapped MongoDB collections to PostgreSQL tables with the same structure, then optimized later. The most painful part was the data validation — they wrote scripts that compared every row between the two systems, and those scripts caught edge cases they hadn't anticipated, like MongoDB's BSON timestamp format versus PostgreSQL's timestamptz. Luna: And the cutover itself — they didn't have downtime? Lucas: They had a planned two-hour maintenance window for the final flip. But because they'd already migrated entity by entity, most of the application was already running on PostgreSQL by then. The window was really just for the last few collections and to remove the MongoDB connection strings. Actual user-facing downtime was about 12 minutes. Luna: What was the metric that told them it worked? Lucas: p99 query latency dropped from 1.2 seconds to 40 milliseconds. That was the headline number. But the operational metric that mattered more was that the number of production incidents related to data inconsistency went from about three per week to zero. They stopped getting the 'invoice total doesn't match line items' bug reports. Luna: And the team's reaction? I imagine there was some sunk-cost attachment to MongoDB. Lucas: A bit. But Sarah framed it well — she said 'This is not a failure of MongoDB, it's a failure of our decision-making process. We chose the right tool for the wrong problem, and now we're fixing it.' And she made sure the engineers who had championed MongoDB originally were part of the migration design. Nobody was blamed. Luna: That's smart leadership. So what did they learn about making database decisions going forward? Lucas: They introduced a formal 'data model review' before any new service is built. Any team planning a new service has to write a one-page document describing their data access patterns — how often they read, how often they write, what joins they need, what consistency requirements they have. Then they bring it to an architecture review. The CTO has veto power, but she's only used it once. Luna: What was the one veto? Lucas: A team wanted to use a graph database for a document management system. The data was mostly simple CRUD with full-text search. Sarah asked them to model the data as relational first, then add a search index. They did, and it worked fine. The graph database would have been over-engineering. Luna: So the lesson isn't 'never use MongoDB' or 'always use PostgreSQL.' It's 'understand your data before you pick a database.' Lucas: Exactly. And if you're already in a mismatch, the cost of staying is often higher than the cost of leaving. FlowMetrics's migration cost them about 12 engineering-weeks over three months. But they estimate they were losing at least 5 engineering-weeks per quarter to MongoDB workarounds and bug fixes. Payback period was less than a year. Luna: And they got the latency improvement on top. Lucas: Right. The latency win was a bonus. The real win was that the team stopped fighting the database. They could focus on features again. Luna: I think a lot of CTOs listening are probably in a similar situation — they know their database is wrong, but they're afraid of the migration cost. Lucas: Yeah, and the fear is understandable. But Sarah's team proved that a strangler fig approach, combined with strict entity-by-entity migration, makes it manageable. And the alternative — staying with the wrong database — has its own hidden costs: developer productivity, incident response, data quality. Those add up. Luna: So the question for any CTO is: are you paying the migration cost already, just in a different form? Lucas: That's exactly the right framing. If you're writing sync tools, maintaining workarounds, or ignoring data inconsistency bugs, you're already in a migration — you just haven't admitted it to yourself.