Latest / The DEX Show: A Show for IT Change Makers / 🗣️ ALL NEW FROM THE DEX SHOW Introducing Parallel: S1E1 - The Phantom Lag
Transcript
- 0:00Speaker 1: Change makers. Tom McGrath here introducing something a little bit
- 0:04Speaker 1: different for you all today, and that's the first installment
- 0:08Speaker 1: of a brand new podcast by the creators of The
- 0:11Speaker 1: Deck Show, which is called Parallel.
- 0:13Speaker 2: It's one it story.
- 0:16Speaker 3: Told week by week, very much inspired by the popular
- 0:19Speaker 3: Murder mystery series podcast serial. If you know that, but
- 0:24Speaker 3: if you don't, don't worry. I think you'll be able
- 0:26Speaker 3: to enjoy it.
- 0:27Speaker 2: Either way, very much the brainchild of the main man,
- 0:32Speaker 2: Tim Flower, who all listeners will know only too well
- 0:35Speaker 2: from The Deck Show and of course reality bites, and
- 0:37Speaker 2: indeed you should hear plenty of familiar voices pop up
- 0:42Speaker 2: here and there throughout the Parallel series. We're going to
- 0:46Speaker 2: post a couple of episodes here on the Deck Show feed,
- 0:50Speaker 2: which you can catch here of course, But if you're
- 0:52Speaker 2: enjoying it and you want to follow, and you want
- 0:54Speaker 2: to hear everything as it's released, check the show notes,
- 0:58Speaker 2: go over to the Parallel podcast feed where you can
- 1:03Speaker 2: give it a follow and keep fully up to date. Okay,
- 1:06Speaker 2: with that, I'm gonna hand it over to Parallel episode one.
- 1:11Speaker 1: Let us know what you think.
- 1:12Speaker 2: Whereny, Thank you so much everybody, Bye bye.
- 1:35Speaker 4: From next Think and the creators of the Deck show.
- 1:38Speaker 4: It's parallel one IT story told week by week because
- 1:43Speaker 4: technology problems don't happen in isolation or by themselves, they
- 1:47Speaker 4: happen in parallel. I'm Tim Flower. For the last ten years,
- 1:55Speaker 4: I've spent this part of my career helping IT teams
- 1:58Speaker 4: understand how product is is stolen both from IT and
- 2:03Speaker 4: from the employees we support. It's been a journey of
- 2:07Speaker 4: seeing workday has gone wrong, system's gone haywire, and the
- 2:11Speaker 4: lengths we in IT will go to for answers today,
- 2:16Speaker 4: we start with a story that began with a simple question,
- 2:20Speaker 4: what's been killing productivity at Zentech, a cutting edge company
- 2:25Speaker 4: known for its efficient operations and innovation, Or if you
- 2:29Speaker 4: want to get technical about it, and apparently I do,
- 2:33Speaker 4: where do employees lose twenty four minutes every day of
- 2:36Speaker 4: their working lives. Searching for answers can sometimes feel intrusive.
- 2:43Speaker 4: I have to dig into how people troubleshoot, their lack
- 2:46Speaker 4: of tools, the lack of visibility, even their lack of
- 2:50Speaker 4: ability sometimes and that can be risky, especially when people's
- 2:54Speaker 4: careers are at stake. What I've learned during the course
- 2:58Speaker 4: of my career is that every technician is different, unique,
- 3:03Speaker 4: complicated even but they all want the same thing to
- 3:07Speaker 4: solve the problem. But how do you solve a problem
- 3:11Speaker 4: you can't find? Or in zen Tech case, how do
- 3:14Speaker 4: you solve a problem that's actually bigger and wilder than
- 3:18Speaker 4: what you can even see? And yes, it seems like
- 3:21Speaker 4: every day in my career, I've been looking for solutions too,
- 3:25Speaker 4: So let's look into this one. For weeks, the employees
- 3:31Speaker 4: at zen Tech have been struggling software crashes, system slowdowns
- 3:36Speaker 4: and inexplicably long response times, projects delayed, deadlines missed, and
- 3:43Speaker 4: frustration building everywhere. And the problem wasn't isolated. I heard
- 3:48Speaker 4: through some trusted sources of app startup delays of ninety
- 3:52Speaker 4: minutes or more and it was systemic. Yet no one
- 3:56Speaker 4: knew why, or at least that's what they want wanted
- 4:00Speaker 4: you to believe. But let's start at the beginning. Zen
- 4:07Speaker 4: Tech has always been a high performing organization. Their employees
- 4:11Speaker 4: are smart, dedicated, They work long hours creating the kind
- 4:16Speaker 4: of tech that keeps the world connected. But this past
- 4:20Speaker 4: December everything changed. I decided to talk with a few employees. First.
- 4:26Speaker 5: I don't know, it started small, you know, things started
- 4:29Speaker 5: to hang here, crash there. We would reboot and move on.
- 4:32Speaker 5: I know we should have, but no one called the
- 4:35Speaker 5: help desk, but we had already been impacted by so much.
- 4:39Speaker 5: And it's pretty common knowledge that the help disc can't
- 4:42Speaker 5: actually help. In fact, they usually end up breaking something
- 4:46Speaker 5: else in the process. But then it got worse and
- 4:48Speaker 5: worse and worse, until we couldn't move anymore and we
- 4:51Speaker 5: had to.
- 4:52Speaker 4: Call Employee one didn't want to be identified, but like
- 4:56Speaker 4: so many at zen Tech, they found their daily routines
- 5:00Speaker 4: railed by an unseen force, A force that didn't just disrupt,
- 5:05Speaker 4: it dismantled, and for a company like Zentech, where time
- 5:09Speaker 4: equals money, this was no small disruption. And then there
- 5:14Speaker 4: was Susan, who voiced some pretty strong frustration.
- 5:20Speaker 6: You're staring at the screen waiting for something to load
- 5:23Speaker 6: five minutes fast and then ten, and the whole time
- 5:25Speaker 6: you're thinking this isn't normal, something's wrong, but no one
- 5:28Speaker 6: seems to know what it was. There's times when I
- 5:30Speaker 6: wonder if it really knows what they're doing.
- 5:34Speaker 4: So then I wanted to talk with the IT teams directly.
- 5:38Speaker 4: They'd been tasked with finding the cause of these issues,
- 5:42Speaker 4: but they seemed to be in over their head, and
- 5:44Speaker 4: what started as a simple troubleshooting mission quickly turned into
- 5:48Speaker 4: something much bigger. Early reports from traditional IT monitoring tools
- 5:55Speaker 4: suggested a hardware bottleneck, too many users with not enough capacity,
- 6:01Speaker 4: but when the data came back, they told a different story.
- 6:04Speaker 4: This wasn't just a case of too many cooks in
- 6:07Speaker 4: the kitchen. Something or someone seemed to be actively interfering.
- 6:14Speaker 4: So I decided to go right to the source and
- 6:17Speaker 4: talk with some of the engineers and technicians at zen
- 6:20Speaker 4: Tech to get their point of view. Terrence was the first,
- 6:24Speaker 4: and as smart as he is, he seemed baffled.
- 6:29Speaker 7: It didn't add up. The spikes and latency weren't random.
- 6:32Speaker 7: They seemed targeted, like specific users, specific times, and the patterns,
- 6:37Speaker 7: my goodness, they're not the kind of thing you'd expect
- 6:39Speaker 7: from normal wear and tear. As much tech as we
- 6:42Speaker 7: got in our command center, none of our teams can
- 6:44Speaker 7: figure it out, probably because we're all troubleshooting to prove
- 6:48Speaker 7: it wasn't our fault.
- 6:50Speaker 4: A pattern, but a pattern of what and who would
- 6:54Speaker 4: want to target zen Tech's employees anyway. It's initial theory
- 6:59Speaker 4: was a rogue script code left behind by a legacy
- 7:02Speaker 4: application long since removed, or perhaps someone with an agenda.
- 7:09Speaker 4: PAM is the IT manager, responsible for not only managing
- 7:12Speaker 4: the team, but for also making sure that things stay
- 7:16Speaker 4: running smoothly and issues solved quickly.
- 7:21Speaker 8: We honestly started to wonder if it was an inside job,
- 7:25Speaker 8: maybe a disgruntled employee or maybe somebody who recently left
- 7:28Speaker 8: the company.
- 7:29Speaker 9: But there just wasn't any evidence like these.
- 7:32Speaker 8: Just these phantom processes slowing everything down, and we were
- 7:35Speaker 8: getting more and more pressure to figure out what it was.
- 7:38Speaker 4: And while it searched for answers, employees were left to
- 7:42Speaker 4: grapple with the fallout. Jessica is a customer call center
- 7:46Speaker 4: rep who talks with Zentech customers all day long.
- 7:51Speaker 9: So here's the thing. You can only apologize to customers
- 7:55Speaker 9: so many times before they stopped listening. It felt like
- 7:58Speaker 9: every day something new was and we were the ones
- 8:01Speaker 9: left holding the bag. Our VP was asking everyone to
- 8:04Speaker 9: come in a half hour early, get ahead of any issues,
- 8:07Speaker 9: and be there for our customers, which we're happy to do.
- 8:11Speaker 9: But there's three thousand of us across the company in
- 8:14Speaker 9: this role. Do we all need to come in early
- 8:16Speaker 9: that's fifteen hundred hours of OT every day.
- 8:21Speaker 4: The company's productivity was plummeting, along with both its reputation
- 8:26Speaker 4: and employee morale. Zen Tech's leadership knew they needed a
- 8:30Speaker 4: breakthrough fast, and that breakthrough came not from traditional IT
- 8:36Speaker 4: but from an unexpected source, Zenech's digital employee experience or
- 8:41Speaker 4: De's team. Using advanced AI tools, they were able to
- 8:46Speaker 4: dig deeper than the command center IT teams ever could,
- 8:49Speaker 4: and what they found changed everything. Fred runs the DEXs team.
- 8:56Speaker 8: It wasn't just the one thing, right, it was a
- 8:59Speaker 8: company nation. You had devices that hadn't been rebooted in months.
- 9:04Speaker 8: He had employees struggling with apps they hadn't even been
- 9:07Speaker 8: trained to use properly. You got a rogue process then
- 9:11Speaker 8: installed on the devices by this roll up patch, but
- 9:14Speaker 8: nobody knew about. Unfortunately, we have this DEX platform now
- 9:18Speaker 8: just feeding us information like nothing else in the tech stack,
- 9:21Speaker 8: and we didn't even have to rely on the employees
- 9:24Speaker 8: for the data anymore. It was like peeling back layers
- 9:29Speaker 8: of an onion, you know, every single one revealed something new.
- 9:33Speaker 4: An onion or maybe a conspiracy because what the DEX
- 9:39Speaker 4: team uncovered wasn't just a series of unfortunate events. It
- 9:43Speaker 4: was becoming more and more clear that this wasn't some
- 9:47Speaker 4: normal run of the mill IT issue. It was sabotage.
- 9:52Speaker 4: But there was more work to do, and the Dex
- 9:54Speaker 4: team knew just where to turn to for answers. Next week,
- 10:00Speaker 4: we'll dive deeper into the sabotage, who planted the rogue
- 10:03Speaker 4: script and why, But more importantly, how did Zintec turn
- 10:07Speaker 4: things around. Stay tuned for the next episode of Parallel
- 10:12Speaker 4: Leave No Trace.