Latest / Found in the Machine / Out of Sync: The Internet and Father Time
Transcript
- 0:34WWV Timekeeper: At the tone, three hours fifty-nine minutes, coordinated universal time.
- 0:48Daina Bouquin: There is a radio station that does nothing but say what time it is.
- 0:53Daina Bouquin: There's no music, no weather, no ads, no traffic updates, just a tone every second, and a voice on the minute announcing the time kept by the official atomic clocks of the US government.
- 1:11Daina Bouquin: It's called WWV.
- 1:14Daina Bouquin: It's been announcing the time since 1945.
- 1:18Daina Bouquin: From Colorado since 1966.
- 1:22Daina Bouquin: And you can call it on the phone.
- 1:25Daina Bouquin: A voice tells you the time, and then the seconds tick off until the line hangs up on you two minutes later.
- 1:32Daina Bouquin: And every one of those seconds that you hear is late.
- 1:38Daina Bouquin: The tick has to travel from Colorado to your ear.
- 1:42Daina Bouquin: And by the time you hear it, that moment is already over.
- 1:46Daina Bouquin: In 1981, there was a computer in Maryland receiving a related time signal from Colorado.
- 1:54Daina Bouquin: Other computers, some thousands of miles away, needed to know what time it was.
- 2:01Daina Bouquin: And the machine in Maryland could tell them.
- 2:04Daina Bouquin: Except its answers would be late too.
- 2:08Daina Bouquin: You see, the clock wasn't broken.
- 2:11Daina Bouquin: The radio was fine.
- 2:12Daina Bouquin: But the time it sent would already be old when it arrived.
- 2:17Daina Bouquin: The problem was you couldn't tell precisely how old.
- 2:24Daina Bouquin: I'm Daina Bouquin, and this is Found in the Machine.
- 2:31Daina Bouquin: I wear a mechanical watch.
- 2:39Daina Bouquin: It doesn't have a battery.
- 2:41Daina Bouquin: I have to wind it.
- 2:43Daina Bouquin: And I wind it most mornings.
- 2:45Daina Bouquin: But I do forget, and when I do, it stops.
- 2:49Daina Bouquin: So I pull out the crown and turn the hands until they agree with my phone.
- 2:55Daina Bouquin: It doesn't even have a second hand, and that's fine.
- 2:59Daina Bouquin: It tells me when I'm late for things, which is about all I ask of it.
- 3:03Daina Bouquin: But a computer has rather more to keep track of.
- 3:08Daina Bouquin: When a file changed, when a message arrives, whether one event happened before another.
- 3:16Daina Bouquin: And when computers work together, all of their clocks need to agree on when those things happen.
- 3:23Daina Bouquin: Inside a typical computer, a piece of quartz vibrates when electricity passes through it.
- 3:30Daina Bouquin: And the machine counts those vibrations and translates them into seconds.
- 3:35Daina Bouquin: It makes it so the computer can keep time even when it's turned off.
- 3:40Daina Bouquin: But quartz responds to temperature and it ages.
- 3:45Daina Bouquin: Its rhythm wanders.
- 3:48Daina Bouquin: Two computers that agree in the morning can disagree by sunset.
- 3:54Daina Bouquin: Not much, not in a way that you'd ever notice, but enough.
- 3:59Daina Bouquin: Now imagine those two computers are thousands of miles apart, exchanging messages across a network.
- 4:08Daina Bouquin: One can send a message at noon and the other records its arrival one second after noon.
- 4:16Daina Bouquin: But was the message traveling for one second?
- 4:20Daina Bouquin: Or was the receiving clock fast?
- 4:23Daina Bouquin: To measure the journey, you need clocks that agree.
- 4:28Daina Bouquin: To make the clocks agree, you need to measure the journey.
- 4:31Daina Bouquin: This was the problem that Dave Mills took up.
- 4:36Daina Bouquin: And he loved it.
- 4:38Daina Bouquin: Dave was born in Oakland in 1938 with glaucoma.
- 4:47Daina Bouquin: Surgery preserved some sight in his left eye, but for much of his life he worked with his face inches from whatever he was trying to see.
- 4:57Daina Bouquin: As a teenager, he became a ham radio operator, helping servicemen stationed at the South Pole communicate with their wives back home.
- 5:07Daina Bouquin: And he liked clocks too.
- 5:10Daina Bouquin: The machinery of them, the mathematics of them.
- 5:14Daina Bouquin: Eventually, Dave went to work at COMSAT Laboratories in Maryland in 1977, where he became involved with something called the ARPANET, one of the networks from which the internet would grow.
- 5:29Daina Bouquin: And the digital world was relatively small then.
- 5:33Daina Bouquin: Research computers in universities and laboratories connected through telephone circuits and satellites.
- 5:39Daina Bouquin: Messages could be delayed by busy equipment or take different routes across the network.
- 5:46Daina Bouquin: Dave wanted to measure the delays.
- 5:50Daina Bouquin: And he had considerable freedom to investigate.
- 5:54Daina Bouquin: He later described his research territory as his own little fief, a place where he could experiment and refine things largely on his own terms.
- 6:05Daina Bouquin: By 1981, Dave had a radio receiver at COMSAT listening to WWVB, another signal from Colorado.
- 6:16Daina Bouquin: It could distribute that time to machines on experimental networks extending into Europe.
- 6:22Daina Bouquin: To do it, Dave developed a method involving four timestamps.
- 6:28Daina Bouquin: One computer records when it asks for the time.
- 6:32Daina Bouquin: The other records when the request arrives and when its reply leaves.
- 6:37Daina Bouquin: The first records when the answer comes back.
- 6:41Daina Bouquin: Four moments from which a computer can estimate the delay and how far its clock is from the other.
- 6:51Daina Bouquin: Not perfectly, though, the outward journey might take longer than the return.
- 6:57Daina Bouquin: So the machines repeat the exchange, collecting measurements over and over, looking for the ones that are least disturbed by delays.
- 7:06Daina Bouquin: But then there is another problem.
- 7:09Daina Bouquin: What if the clock you're consulting is simply wrong?
- 7:13Daina Bouquin: Dave called clocks whose time estimates agreed closely enough with one another true chimers.
- 7:20Daina Bouquin: The ones that couldn't be trusted were false tickers.
- 7:24Daina Bouquin: And he devised elaborate mathematical procedures for comparing them, accounting for uncertainty and setting aside sources that disagreed too much.
- 7:35Daina Bouquin: But after a computer worked out what time it ought to be, there was the question of what to do about its own clock.
- 7:42Daina Bouquin: What if its clock was wrong?
- 7:44Daina Bouquin: With my watch, I can just move the hands.
- 7:47Daina Bouquin: But a computer may have been keeping records all along.
- 7:52Daina Bouquin: Turn its clock backward and events can appear to happen twice.
- 7:56Daina Bouquin: Move it forward and you can leave a gap.
- 8:00Daina Bouquin: So Dave developed ways to change the speed of the clock instead.
- 8:06Daina Bouquin: When a clock was determined to be behind, it could count seconds just a little faster.
- 8:13Daina Bouquin: One that was ahead could count them just a little slower.
- 8:17Daina Bouquin: He could make them drift on purpose, gradually coming into agreement.
- 8:23Daina Bouquin: He called his system the Network Time Protocol.
- 8:28Daina Bouquin: NTP.
- 8:30Daina Bouquin: An early specification appeared in 1985.
- 8:35Daina Bouquin: By 1988, Dave had refined NTP enough to bring clocks into agreement within tens of milliseconds.
- 8:44Daina Bouquin: A few thousandths of a second.
- 8:47Daina Bouquin: But still, he kept refining things.
- 8:52Daina Bouquin: Sometimes the measurement would suddenly jump away from the others.
- 8:57Daina Bouquin: A momentary disturbance that Dave called a popcorn spike.
- 9:03Daina Bouquin: He devised a popcorn spike suppressor to deal with it.
- 9:07Daina Bouquin: When network traffic made the delays unequal, there was a huff-n’-puff filter.
- 9:13Daina Bouquin: He was completely serious about the names and the mathematics behind them.
- 9:18Daina Bouquin: Dave wanted to know not merely that an algorithm worked, but why it worked and under what conditions it might fail.
- 9:27Daina Bouquin: But meanwhile, the internet was growing around him.
- 9:31Daina Bouquin: The computers moved beyond the laboratories into offices and home.
- 9:36Daina Bouquin: They began keeping track of financial transactions, recording events on distant servers, coordinating work across oceans.
- 9:45Daina Bouquin: Every machine still had its own imperfect clock.
- 9:49Daina Bouquin: Every message still took time to arrive.
- 9:53Daina Bouquin: But now there were millions of machines scattered across continents, and they were beginning to depend on one another.
- 10:01Daina Bouquin: They didn't each need a radio receiver or an atomic clock.
- 10:05Daina Bouquin: One computer could ask another for the time, which might ask another and another, all the way back through a hierarchy of machines to a reliable source like WWV.
- 10:18Daina Bouquin: The network had become enormous, and the discrepancies Dave was measuring were almost unimaginably small, and his little fief was everywhere.
- 10:31Daina Bouquin: This growth brought problems that had scarcely existed when Dave began.
- 10:38Daina Bouquin: In 2003, a University of Wisconsin Time server was overwhelmed by requests for the time from hundreds of thousands of consumer routers.
- 10:50Daina Bouquin: A mistake in their software had caused them to keep asking for the time again and again when they didn't receive an answer.
- 10:58Daina Bouquin: The server was trying to tell them what time it was, but they were making it difficult for the server to do anything at all.
- 11:05Daina Bouquin: NTP eventually gained a way for servers to tell clients to stop sending requests.
- 11:11Daina Bouquin: They called it the Kiss-o '-Death packet.
- 11:14Daina Bouquin: The internet was no longer a place where one engineer could oversee everything.
- 11:20Daina Bouquin: There were security vulnerabilities to address.
- 11:24Daina Bouquin: New applications and people who wanted to change the software.
- 11:28Daina Bouquin: Dave could be difficult to persuade.
- 11:31Daina Bouquin: He demanded mathematical proofs even when other programmers had tested the change and found that it worked.
- 11:39Daina Bouquin: His insistence on rigor had helped make NTP extraordinarily reliable.
- 11:45Daina Bouquin: It could also frustrate the people trying to maintain it.
- 11:49Daina Bouquin: One of them was Harlan Stenn, who began contributing to NTP in the early 1990s and eventually took responsibility for releasing its software.
- 11:59Daina Bouquin: He and Dave worked together for decades, but they never met in person.
- 12:04Daina Bouquin: They discussed the code over the telephone, trying to resolve problems in a system that had become increasingly difficult for anyone to understand completely.
- 12:14Daina Bouquin: Dave knew why he made his decisions, and he had spent years working through the mathematics, revising the algorithms and testing their behavior.
- 12:24Daina Bouquin: But he knew that he couldn't always explain his reasoning in terms his collaborators understood.
- 12:31Daina Bouquin: And as the years passed, communicating that knowledge became even harder.
- 12:37Daina Bouquin: Dave had spent much of his career with his face close to computer screens and circuit boards.
- 12:43Daina Bouquin: But eventually his remaining eyesight failed.
- 12:47Daina Bouquin: He continued working with software that read his computer screen aloud, but a mathematical equation is difficult to navigate by sound.
- 12:56Daina Bouquin: Symbols that occupy different positions on a page arrive one after another.
- 13:01Daina Bouquin: Finding your place can be painstaking.
- 13:05Daina Bouquin: And other people were now developing their own approaches to keeping internet time.
- 13:10Daina Bouquin: Dave continued working on NTP into his 80s.
- 13:14Daina Bouquin: In 2020, he submitted proposals for improvements to the people developing the next version.
- 13:20Daina Bouquin: But he struggled to get a real response.
- 13:24Daina Bouquin: He spent hours at his computer listening to his work read back to him, sometimes asking family members to help him check what he'd written.
- 13:33Daina Bouquin: And he knew that his proposals might not be adopted.
- 13:37Daina Bouquin: When journalist Nate Hopper asked why he kept working on them, Mills just said, "because it's there." "I like to improve what I do." Dave Mills died on January 17th, 2024.
- 13:56Daina Bouquin: He was 85.
- 13:59Daina Bouquin: In Colorado, the radio station is still broadcasting.
- 14:06Daina Bouquin: Every minute a voice announces the time.
- 14:09Daina Bouquin: Every second, there's another tone.
- 14:12Daina Bouquin: WWV was doing this when Mills was a child.
- 14:17Daina Bouquin: It was doing it when he began distributing time across a handful of research networks, and when those networks grew to encompass billions of devices. It was doing it on the day he died.
- 14:29Daina Bouquin: The signal still takes time to reach you and it always will.
- 14:34Daina Bouquin: And everywhere around you, computers are asking one another what time it is.
- 14:42Daina Bouquin: That answer too will always be a little late.
- 14:46Daina Bouquin: But the machines will figure out the delay.
- 14:51Daina Bouquin: They'll compare what they know.
- 14:53Daina Bouquin: They'll make corrections.
- 14:55Daina Bouquin: A little faster.
- 14:58Daina Bouquin: A little slower.
- 15:00Daina Bouquin: I'm Daina Bouquin and this is Found in the Machine.
- 15:08Daina Bouquin: If you like hearing my stories, please take a moment to leave a rating or a review wherever you're listening to this.
- 15:21Daina Bouquin: It helps other people find the show, and so it helps me make this more sustainable.
- 15:27Daina Bouquin: And if you want more content from me, you can go to foundinthemachine.com/ support.
- 15:36Daina Bouquin: You'll find a link to the show's Patreon.
- 15:39Daina Bouquin: Patreon members get short bonus episodes every other week.
- 15:43Daina Bouquin: You can also find other ways to support the show.
- 15:48Daina Bouquin: Thanks as always for listening, guys.