Latest / The Operator's Manual with Fexingo: Daily Decisions, Tactical Wins, and Business Execution / How One Database Slowed Down an Entire Bank
Transcript
- Lucas: So there's this moment in every ops person's life where you're staring at a screen, and the number just doesn't move. You've checked the process, you've checked the people, you've checked the workflow — and it's still slow. Luna: Right. And usually the culprit is hiding somewhere you didn't think to look. Lucas: Exactly. Today I want to talk about a case that sounds almost too small to matter: one database index. One missing index on a single table in a major retail bank. And it was causing hours of delay per day across loan origination, risk reporting, and customer onboarding. Luna: Wait — a missing index? Like, a database thing? That can stall an entire bank? Lucas: It sounds implausible, but yes. This happened at a mid-tier US bank, not a tiny credit union. They had about 40 billion in assets. And their operations team had been blaming process inefficiency for months. Loan officers were complaining that the approval system took 15 minutes per application — and they process hundreds a day. Luna: Fifteen minutes doesn't sound terrible, until you multiply by 200 applications. Lucas: Exactly. That's 50 hours of wasted time daily. Plus the risk team couldn't generate end-of-day reports before midnight. They'd miss regulatory deadlines. So the COO brought in a consultant, and the consultant did something simple: asked the database administrator to run a query profiling the slowest operations. Luna: And that's when they found it. Lucas: One query — a lookup on the transactions table filtering by customer ID and date — was taking 8 seconds per execution. That query was called 3,000 times an hour. So that's about 6.7 hours of database time per day just from that one query. The table had 200 million rows, and the query was doing a full table scan because there was no index on. Luna: So the database was reading every single row to find the ones matching a customer? That's like flipping through every page of a phone book to find one name. Lucas: Perfect analogy. Adding a composite index took the DBA about 10 minutes — CREATE INDEX statement, ran during a maintenance window. After that, the same query executed in 50 milliseconds. That's 160 times faster. The loan approval screen, which had been taking 15 minutes, dropped to under 30 seconds. Luna: And that's just the direct impact. What about the downstream effects? Lucas: That's the part I find most interesting. The risk reports started finishing by 8 PM instead of 2 AM. The compliance team could submit their filings on time. Customer onboarding — which required a credit check that hit that same table — went from 45 minutes to 3 minutes. The bank estimated the index saved them about $2 million a year in productivity and avoided penalties. Luna: All from a 10-minute change that cost nothing except a DBA's time. Lucas: Right. And this is the point I want to drill into. Most operations leaders think about bottlenecks in terms of people, process, or equipment. But in any business that runs on software — which is almost every business now — the bottleneck can be as invisible as a missing index. And because it's invisible, teams waste months optimizing the wrong thing. Luna: So how do you catch it? Most ops managers aren't database experts. Lucas: You don't need to be. You need one thing: a simple dashboard that shows the slowest database queries by total time consumed. Every modern database — Postgres, MySQL, SQL Server — has a built-in view for that. The pg_stat_statements module in Postgres, for example. If you're an ops leader and you don't have that on your radar, you're flying blind. Luna: But isn't this just basic performance tuning? Why don't banks already have this in place? Lucas: Because it's not glamorous. Nobody gets promoted for adding an index. But people get fired when loan approvals take too long and customers leave. The funny thing is, the DBA at this bank had actually flagged the slow query six months earlier. But the operations team was so focused on retraining staff and redesigning workflows that they never prioritized the database ticket. Luna: Classic — the technical fix was sitting in a queue behind process improvements. Lucas: Exactly. And the DBA wasn't pushing hard because they assumed the ops team understood the impact. But the ops team didn't know how to read a query execution plan. So the gap was a communication failure, not a technical one. Luna: So what's the fix? Get ops and engineering speaking the same language? Lucas: At minimum, ops leaders should understand three concepts: full table scan, index, and query execution time. You don't need to write SQL, but you need to know that if a query takes longer than a second, it's worth investigating. And you need a regular review — say, monthly — where the DBA shows the top five slowest queries and what they cost in business terms. Luna: That's a great practice. And it's cheap. Lucas: Cheapest operational improvement I know. And the impact compounds. Once you fix one slow query, you free up database capacity, which makes other queries faster too. The bank in our story saw overall system response time improve by 40 percent after that single index. And they never hired more people or bought new servers. Luna: Alright, so I'm convinced every ops person needs a basic understanding of database performance. But let me push back a little — isn't this situation becoming less common with cloud databases that auto-index or use serverless? Lucas: Great question. Cloud databases like Amazon Aurora or Google Cloud SQL do have automated tuning features, but they're not magic. Auto-indexing can help, but it's reactive — it only creates indexes after it sees a pattern of slow queries. By then, you've already suffered the latency. And many organizations still run on-premise or have hybrid setups. Plus, even in cloud, misconfigured indexes are still a top cause of performance issues. Luna: So it's not going away. And the human element — ops not talking to DBAs — is still the root cause. Lucas: Exactly. And I think there's a broader lesson here: in any operation, the most impactful fix is often the one that's easiest to ignore because it doesn't look like a process problem. It looks like a technical detail. But the best operations leaders I've met are the ones who refuse to accept 'that's an IT issue' as a reason to stop digging. Luna: You know, speaking of digging into problems that others ignore — that's actually what keeps this show going. Lucas: Yeah. And a quick honest thing: a handful of listeners chip in monthly through buy me a coffee dot com slash fexingo, and that's literally what funds making this many episodes. We don't run ads, we're not backed by a big media company. It's just a small group of people who find these stories useful and want to keep them coming. Luna: And we genuinely appreciate that. It allows us to pick topics like a missing index at a bank, which isn't clickbait, but it's the kind of thing that can actually move the needle in someone's work. Lucas: Alright, back to the bank. So after they fixed that one index, they started a monthly 'query review' session. The DBA would bring the top five slowest queries, and the ops team would translate each one into a business impact. Like 'this query adds 30 seconds to every customer lookup, which costs us $X per month in call center time.' Luna: That's brilliant. It turns a technical metric into a dollar figure that executives care about. Lucas: And it builds a culture where people look for the invisible bottleneck. The next month, they found a report that was scheduled to run at peak hours, causing contention. They moved it to off-peak — zero cost, big improvement. Then they found a query that was pulling 10 times more columns than needed. Trimmed it down. Saved another 20 percent of database load. Luna: So the first fix created a habit of looking for more. Lucas: Exactly. And that's the real operational win — not the index itself, but the system for finding the next index. Because there's always a next index. The question is whether you have the visibility and the cross-functional communication to find it. Luna: So if someone listening is an ops leader at a company that runs on software, what's the one thing they should do tomorrow? Lucas: Call your DBA or your cloud ops person and ask: 'What's our single slowest query, and what's it costing us in business terms?' If they can't answer that, you've found your first bottleneck. Luna: Simple. And it might just save you millions. Lucas: Or at least a few frustrating hours. That's it for today. We'll be back tomorrow with another story from the operations manual.