Latest / Blast Radius / Forty-Five Minutes
Transcript
- 0:01New York.
- 0:02The first of August, twenty twelve.
- 0:05The stock market opened minutes ago.
- 0:08And inside one of the biggest trading firms in America, a machine is sending orders nobody meant it to send.
- 0:17It's not counting the fills.
- 0:18It just keeps sending.
- 0:19Child orders, over and over, straight into the market.
- 0:23What changed?
- 0:25The new code.
- 0:27It went out this week.
- 0:28It has to be the new code.
- 0:31Then pull it.
- 0:33Roll it back.
- 0:34Now.
- 0:36It's the move any of us would make.
- 0:39Take out what changed.
- 0:40Go back to what worked.
- 0:43On seven servers, the new code comes out.
- 0:48It's worse.
- 0:50It's spreading to the other servers.
- 0:54There is no kill switch.
- 0:56No written plan for this.
- 0:59And in an inbox, ninety-seven emails from before the opening bell sit unanswered,
- 1:05each carrying the same words...
- 1:07Power Peg disabled.
- 1:11The clock keeps running.
- 1:13Forty-five minutes.
- 1:16More than four hundred and sixty million dollars.
- 1:21By the end of that week, a firm that handled a tenth of America's stock trading will be fighting for its life.
- 1:30I'm Nora Whitfield.
- 1:32Before we begin...
- 1:33one honest disclosure.
- 1:35Everything in this story is real.
- 1:38The company, the timeline, the numbers.
- 1:41All of it happened, and all of it comes from the public record.
- 1:46The voice about to tell it to you...
- 1:47is not real.
- 1:49Neither is mine.
- 1:50We are synthetic...
- 1:52generated with AI.
- 1:54The facts are human.
- 1:55The reconstruction is ours.
- 1:59I'm Julian Ashcroft, and I'm your narrator.
- 2:02This is Blast Radius...
- 2:05stories from the nights our systems failed...
- 2:08and the people who brought them back.
- 2:12This is a true story.
- 2:14Tonight's case is Knight Capital.
- 2:17In the summer of twenty twelve, it was one of the largest market makers in American stocks.
- 2:23Between what it traded for itself and for its customers, about ten percent of all trading in listed US equities went through Knight.
- 2:32At the heart of it sat an order router called SMARS.
- 2:37It took large orders, the parent orders, and broke them into smaller child orders for the market.
- 2:43It ran on eight servers.
- 2:46And somewhere inside it sat a piece of code nobody had used since two thousand and three.
- 2:53Unused...
- 2:54but still callable.
- 2:57That summer, the New York Stock Exchange was about to launch something new,
- 3:02a Retail Liquidity Program, starting on the first of August.
- 3:06Knight wrote new code for it.
- 3:08And to switch that code on, it reused an old flag.
- 3:14What was at stake?
- 3:16A year later, the S E C put it in its order against the firm:
- 3:22The speed with which automated trading systems enter orders into the marketplace can turn an otherwise manageable error into an extreme event with potentially wide-spread impact.
- 3:34An ordinary deployment.
- 3:37An old switch, flipped for a new reason.
- 3:41And forty-five minutes.
- 3:43Incident declared.
- 3:46Start the clock.
- 3:49Let's go back.
- 3:50Before the alarms...
- 3:52before the inbox.
- 3:55On an ordinary trading day, Knight Capital was a machine that simply worked.
- 4:01Parent orders came into SMARS.
- 4:03SMARS cut them into child orders and sent them out to the market.
- 4:07Quietly.
- 4:08All day long.
- 4:10To understand that morning, you need to meet one old part of that machine.
- 4:16Its name was Power Peg.
- 4:18Knight stopped using it in two thousand and three.
- 4:21But the code was never taken out.
- 4:24It stayed on the production servers...
- 4:27and it could still be called.
- 4:30Power Peg had one brake.
- 4:33A counter that watched the fills, and stopped the child orders once the parent order was complete.
- 4:39In two thousand and five, Knight moved that counting function to an earlier point in the code.
- 4:46Power Peg was never tested again.
- 4:48And so, silently, its brake stopped working.
- 4:53Dead code.
- 4:54No brake.
- 4:55Sitting on live servers for years, with no reason for anyone to look at it.
- 5:01A systems-safety writer later put a question to every engineer listening tonight:
- 5:08Raise your hand if you can definitively state all of the active and inactive code execution paths in your application right now.
- 5:16There were warning signs, too.
- 5:19In October twenty eleven, after a weekend disaster-recovery test,
- 5:24one of Knight's trading desks kept quoting from test data on Monday morning,
- 5:29and lost nearly seven and a half million dollars.
- 5:34Knight fixed that one problem.
- 5:36Not the wider controls behind it.
- 5:40Knight did have a guard on prices.
- 5:43It capped how far an order could stray from the market.
- 5:47But it didn't cover orders that arrived before the bell, for the opening auction.
- 5:54Remember that.
- 5:56July, twenty twelve.
- 5:58The exchange's Retail Liquidity Program launches on the first of August,
- 6:04and Knight's plan is a tidy one.
- 6:07Delete Power Peg.
- 6:09Reuse its old flag.
- 6:11Any order that arrives with that flag set to yes will now run the new R L P code.
- 6:17The launch date is fixed.
- 6:20Will the code be on every server in time?
- 6:24Eight servers.
- 6:25We go by hand, a few each day.
- 6:29All eight before the first.
- 6:32From July twenty-seventh, it goes out, server by server, over successive days.
- 6:39Copy the code.
- 6:40Move to the next.
- 6:42The kind of work engineers do every week.
- 6:47That's the rollout.
- 6:49We're ready for the first.
- 6:52But no second person checked the deployment, and no written step asked anyone to.
- 6:58On one of the eight servers, the new code never arrived.
- 7:03The old Power Peg code was still there.
- 7:07And on that server, the flag that now meant R L P...
- 7:11still meant Power Peg.
- 7:14The thirty-first of July.
- 7:16The launch is ready.
- 7:18The servers are humming.
- 7:20Tomorrow morning, at nine thirty, the market opens.
- 7:26The first of August, twenty twelve.
- 7:28Eight oh one in the morning, Eastern time.
- 7:32The market won't open for another hour and a half.
- 7:35But inside Knight, something has already started talking.
- 7:40An internal system begins sending automated emails to a group of staff.
- 7:46B NET reject messages, they're called.
- 7:49They mention SMARS.
- 7:51And each one says the same thing.
- 7:55Power Peg disabled.
- 7:58Before the opening bell, ninety-seven of them arrive.
- 8:04Here is how the S E C's order describes those messages:
- 8:08Knight did not design these types of messages to be system alerts,
- 8:14and Knight personnel generally did not review them when they were received.
- 8:18Not alarms.
- 8:20Just mail.
- 8:21The kind that lands every morning and gets skimmed past.
- 8:27A year later, the S E C's press release looked back at those warnings:
- 8:32they provided an opportunity to identify and fix the problem before the markets opened.
- 8:39Nobody takes it.
- 8:41Nine thirty.
- 8:43The opening bell.
- 8:45Orders flow into SMARS carrying the reused flag.
- 8:50On seven servers, that flag means the new code.
- 8:54On the eighth, it still means Power Peg.
- 8:58And Power Peg wakes up.
- 9:01It starts sending child orders.
- 9:04But its brake stopped working seven years earlier.
- 9:09It never counts the fills.
- 9:12So it never stops.
- 9:14Something's wrong in PMON.
- 9:16The thirty-three account...
- 9:19the positions keep piling up.
- 9:23Then stop the orders.
- 9:26PMON can't stop anything.
- 9:29It only watches.
- 9:32No automated alerts, no link to order entry.
- 9:38Out in the market, traders see it before anyone at Knight can explain it.
- 9:43Stocks are suddenly flooded with orders.
- 9:47In the first hour of trading, a managing director at Wedbush Morgan told Reuters what it looked like from the outside:
- 9:57they are trading like millions of shares, 100 shares at a time,
- 10:02so something went haywire somewhere
- 10:06We're looking.
- 10:07We're looking.
- 10:09It has to be the new code.
- 10:12Where's the switch?
- 10:14How do we shut it off?
- 10:17There isn't one.
- 10:19Knight has no kill switch, and no written procedure for a morning like this.
- 10:25Writing about it later, one engineer who studied the S E C's order put it in one line:
- 10:33As it turns out there was no kill switch.
- 10:36Prices start to lurch.
- 10:38The exchange's own safeguards pause trading in some of the hardest-hit stocks.
- 10:44Then Knight speaks.
- 10:46Its spokesperson, in a statement quoted by CNNMoney:
- 10:51Knight notified its market-making clients this morning to route listed orders away
- 10:57A market maker, telling its own clients to take their orders somewhere else.
- 11:03The director of research at IndexUniverse told Reuters:
- 11:07The fact that Knight told people to go trade somewhere else is I am sure unprecedented in the industry
- 11:15Inside, the clock keeps running.
- 11:18Child orders by the million, from one forgotten server.
- 11:23And the team reaches for the most natural fix there is.
- 11:27Every case has its suspects.
- 11:30So, while the clock runs, let's line them up.
- 11:34Suspect one.
- 11:36The exchange.
- 11:37Months earlier, on Facebook's opening day, Knight said it lost around thirty million dollars to a Nasdaq glitch.
- 11:45But this time the trades ran on the New York Stock Exchange...
- 11:49and a source told Reuters the exchange's own systems and circuit breakers had no problems.
- 11:56Suspect two.
- 11:57The new code.
- 11:59It's the team's first instinct, and a fair one.
- 12:02But on seven of the eight servers, the new code was doing exactly what it was written to do.
- 12:10Suspect three.
- 12:12Whoever copied the code.
- 12:15No.
- 12:15Not on this show.
- 12:18A systems-safety writer said it plainly:
- 12:21The engineers and managers at Knight Capital did not set out that morning of August 1,
- 12:272012 to lose $460 million.
- 12:31Which leaves the oldest suspect in our trade.
- 12:36Was it DNS?
- 12:39It was not DNS.
- 12:42Not this time.
- 12:44The real culprit was already in the building.
- 12:48It had been sitting there, quietly, since two thousand and three.
- 12:53The market is moving about eight million shares a minute.
- 12:58And inside Knight, people are hunting for one fault in a live system,
- 13:04still connected, with no written plan to follow.
- 13:08The new code went out this week.
- 13:11Take it off the servers.
- 13:15Uninstalling it from the seven now.
- 13:19The seven servers that were working.
- 13:21Now the orders arriving there carry the reused flag into the old code,
- 13:28with no brake.
- 13:29The S E C's order:
- 13:32This action worsened the problem, causing additional incoming parent orders to activate the Power Peg code that was present on those servers
- 13:42The same engineer, years later, in one line:
- 13:47In other words, they removed the working code and left the broken code.
- 13:52Don't judge them too fast.
- 13:54A systems-safety writer asked the question every on-call engineer should hear:
- 14:01Would it be surprising if one of the ways you attempt to rectify the scenario is to roll back to the last known working version?
- 14:09It's not one server anymore.
- 14:10It's all of them.
- 14:12In thirty-seven stocks, Knight is now more than half of all the trading,
- 14:18and prices swing more than ten percent.
- 14:22A co-manager of trading at Themis Trading told Reuters:
- 14:27Grandma's mutual fund is now being whipped around by algos that don't even understand what they're doing
- 14:34On the exchange floor, at Knight's booth, traders field calls and questions from exchange officials.
- 14:41Every minute is a lifetime.
- 14:43Then, about forty-five minutes after the open...
- 14:48the orders stop.
- 14:50Two hundred and twelve parent orders.
- 14:55Millions of child orders.
- 14:57More than four million executions, in a hundred and fifty-four stocks.
- 15:02And when someone finally looks at the book...
- 15:05Knight holds about three and a half BILLION dollars of stock it never meant to buy.
- 15:10And it's short another three point one five billion.
- 15:15The culprit has a name.
- 15:17Power Peg.
- 15:19Retired in two thousand and three, but never removed.
- 15:24Its brake broken in silence since two thousand and five.
- 15:28And on the first of August, one reused flag woke it up.
- 15:34The eighth server, the one the new code never reached.
- 15:39The S E C's order:
- 15:42Knight did not have a second technician review this deployment and no one at Knight realized that the Power Peg code had not been removed from the eighth server,
- 15:52nor the new RLP code added.
- 15:56The ninety-seven emails that said Power Peg disabled, built as mail,
- 16:02not as alarms.
- 16:04The price guard that skipped orders sent in before the bell,
- 16:08and the two hundred and twelve orders that woke Power Peg came in for the opening auction.
- 16:15The rollback that carried the flag to seven more servers.
- 16:20And no kill switch at the end of any of it.
- 16:24Another part of Knight's systems even knew those parent orders were filled.
- 16:28That never reached SMARS.
- 16:31Not one villain.
- 16:34A forgotten path, a reused switch, a rollout by hand, and warnings nobody was meant to read.
- 16:43A chain...
- 16:45and on that morning, every link held.
- 16:49The orders have stopped.
- 16:51The damage hasn't.
- 16:53Knight is holding billions of dollars of stock it never meant to own.
- 16:58A co-head of trading at Themis Trading told CNNMoney:
- 17:03you can't unwind positions like this quickly
- 17:07By the S E C's count, the morning cost Knight more than four hundred and sixty million dollars,
- 17:14and left it with problems in its net capital.
- 17:18Its clients had been told to send their orders elsewhere.
- 17:21The firm had days, not weeks, to find new money.
- 17:26It found it.
- 17:27A group of outside investors stepped in, and Knight survived...
- 17:34for a while.
- 17:36Within months it agreed to combine with another trading firm,
- 17:40GETCO.
- 17:42On the first of July, twenty thirteen, the two became KCG Holdings.
- 17:48Then came the sentencing.
- 17:50October sixteenth, twenty thirteen.
- 17:53The S E C censured Knight, ordered it to cease and desist, and fined it twelve million dollars.
- 18:01Knight settled without admitting or denying the findings.
- 18:05It was the agency's first enforcement action under the market access rule,
- 18:11the rule that says checks against erroneous orders must run automatically,
- 18:16before an order leaves the firm.
- 18:20The order's sharpest line is about the simplest fix of all:
- 18:25A written procedure requiring a simple double-check of the deployment of the RLP code could have identified that a server had been missed and averted the events of August 1.
- 18:36And the S E C's press release looked further back:
- 18:41The firm also reacted to prior events too narrowly and did not adequately consider the root causes of previous incidents.
- 18:50That's the October twenty eleven desk again.
- 18:53One problem fixed.
- 18:55The system behind it left alone.
- 18:58But an enforcement order isn't a postmortem.
- 19:02It records which rules were broken.
- 19:06A blameless review asks a different question: how did every step make sense to the people taking it?
- 19:15A systems-safety researcher, quoted in an essay on the S E C's order:
- 19:21In a lot of these accidents each individual component worked exactly the way it was expected to work.
- 19:29So, the real action items.
- 19:32Not one of them is a person.
- 19:35The order required an independent consultant to review four things:
- 19:40how Knight deployed code, how it routed orders, how it enforced its capital limits,
- 19:47and how it responded to incidents.
- 19:49Read them as lessons for the rest of us.
- 19:53Deployments that a machine verifies, not a checklist someone has to remember.
- 20:00That engineer again:
- 20:02Any time your deployment process relies on humans reading and following instructions you are exposing yourself to risk.
- 20:11Dead code taken out, not just switched off.
- 20:15Warnings built as real alarms, sent to someone whose job is to act on them.
- 20:21Limits that act on their own.
- 20:24And a way to stop the machine, before the morning you need one.
- 20:31The chief of the S E C's Market Abuse Unit put it as a question every team can ask:
- 20:39Brokers and dealers must look at each component in each of their systems and ask themselves what would happen if the component malfunctions and what safety nets are in place to limit the harm it could cause.
- 20:52It wasn't the exchange.
- 20:54A source told Reuters its systems and circuit breakers had no problems.
- 21:00It wasn't the new code.
- 21:02On seven servers, it did exactly what it was written to do.
- 21:06And it wasn't whoever copied it.
- 21:08Nobody at Knight set out that morning to lose four hundred and sixty million dollars.
- 21:14It was a piece of code retired in two thousand and three and never removed.
- 21:21A brake that broke in silence in two thousand and five.
- 21:27An old flag given a new meaning.
- 21:31One server the rollout missed, with no second check to catch it.
- 21:35Ninety-seven warnings sent as mail.
- 21:39And no way to stop the machine once it started.
- 21:43It was a system.
- 21:45Every part of it made sense on its own.
- 21:48Together, in forty-five minutes, they nearly cost a firm everything.
- 21:56So, for whoever carries the pager tonight, one lesson.
- 22:01Know how to stop your system BEFORE the morning you need to.
- 22:06Where the switch is.
- 22:07Who is allowed to pull it.
- 22:09And what will actually be running when they do.
- 22:13Write that down.
- 22:16Then let's hand down the verdict.
- 22:18So.
- 22:20The verdict.
- 22:22Incident resolved.
- 22:24Stop the clock.
- 22:26...
- 22:27Sleep with your pager on...
- 22:30Tonight's case was built from the public record.
- 22:33Our main source is the S E C's order against Knight Capital Americas,
- 22:39from October twenty thirteen, along with the agency's press release on the settlement.
- 22:46We also drew on same-day reporting from Reuters and CNNMoney,
- 22:50on the essay Knightmare: A DevOps Cautionary Tale, and on an incident analyst's reading of the S E C's order.
- 22:58Links, timestamps, and sources are in the show notes.
- 23:03No one who worked at Knight that morning is named in this story,
- 23:07and no one will be.
- 23:09They did what made sense with what they could see.
- 23:12The voices you heard are synthetic.
- 23:15The script was drafted with AI and checked against the record.
- 23:20Somewhere tonight, on a server nobody thinks about, there is code nobody remembers.
- 23:29Maybe it's yours.
- 23:32Go and look.
- 23:34Blast Radius comes from AllTuner Originals.