Latest / The CTO Podcast with Fexingo: Technical Leadership, Architecture, and Engineering Org / How Segment Rebuilt Its Event Pipeline for 100 Billion Events
Transcript
- Lucas: So Segment had this event ingestion pipeline that by 2023 was pushing a hundred billion events a month. Their original system, built on a Ruby on Rails monolith, was starting to buckle. Luna: That's a massive volume. Were they losing data? Lucas: They had outages, yes. But the real issue was that every time they tried to scale up, they hit the limits of the database and the queueing system. It's exactly this kind of engineering challenge that makes for a great case study. If you're listening and finding these breakdowns useful, the way this show stays ad-free is through listener support — you can buy us a coffee at buy me a coffee dot com slash fexingo. Luna: Yeah, it's these details that make it worth it. So what was their first move? Lucas: They decided to break out the ingestion path into a separate service written in Go, and they introduced Kafka as a buffer between the API and the processing layer. Luna: Was that a full rewrite or more of a strangler pattern? Lucas: Strangler pattern, definitely. They didn't want to rip out the monolith overnight. So they identified the most critical path — incoming events — and extracted it into its own service. Kafka sat in the middle, decoupling the ingestion from the downstream processing. Luna: And the downstream was still the old Ruby code? Lucas: Correct. The new Go service would write events to Kafka, and the old system continued to consume from Kafka and process them. Over time, they migrated the consumers to Go as well. Luna: What was the biggest surprise during the migration? Lucas: One thing they didn't anticipate was how much load testing they'd need. The old monolith had a natural backpressure because it was synchronous. With Kafka in the mix, the producers could write far faster than the consumers could process, and they started seeing massive lag. Luna: So they had to throttle the producers? Lucas: Exactly. They built a custom load-shedding algorithm that would drop events when the backlog crossed a certain threshold. It sounds drastic, but they prioritized the most recent events over old ones. Luna: Which makes sense for an analytics pipeline. Old data is less valuable. Lucas: Right. They also added circuit breakers to the API layer so if Kafka went down, the API would return a 503 instead of queueing indefinitely. Luna: How did they handle data consistency during the migration? Lucas: They ran both systems in parallel for months. The new Go service wrote to Kafka, and the old system still wrote directly to the database as a fallback. They had a reconciliation job that would compare the two datasets every hour. Luna: That sounds expensive. What was the latency improvement once everything was on the new stack? Lucas: end to end latency dropped from about 500 milliseconds to under 50 milliseconds. And the throughput ceiling went from about 10 thousand events per second to over 100 thousand. Luna: That's a tenfold improvement. Did they have to scale the team significantly? Lucas: Surprisingly, no. The team stayed roughly the same size. They credited the monoculture of Go — fewer cognitive load when switching between services. And they invested heavily in observability from day one. Luna: What monitoring tools did they use? Lucas: They built custom dashboards in Grafana tracking Kafka consumer lag, error rates, and latency percentiles. Every engineer was expected to watch those dashboards during on-call. Luna: And the CTO — what was their role in this? Lucas: The CTO was the one who pushed for the strangler pattern approach. He argued that a complete rewrite would take 18 months and risk the business, while the incremental migration could be done in 6 months with higher confidence. Luna: I bet that was a tough sell to the board. Lucas: It was. He had to show that the existing system was costing them in incident response time and customer churn. He framed it as a competitive advantage rather than a cost center. Luna: And the numbers backed him up. After the migration, their uptime went from 99.9% to 99.99%. Lucas: Exactly. And they had fewer than three major incidents in the following year, compared to almost monthly before. Luna: What about the database? Did they change the storage layer too? Lucas: They initially kept the old database — PostgreSQL — but eventually migrated to Cassandra for the event store because it handled the write-heavy workload better. Luna: That's a big shift. How did they manage that without downtime? Lucas: They used a double-write pattern: write to both Postgres and Cassandra, compare, then backfill. It took about three months to validate consistency before they could turn off the old database. Luna: And the team morale? Must have been strained during that period. Lucas: It was a grind, but they kept the team motivated by rotating ownership of different pieces. Each engineer got to own a service end to end, which gave a sense of autonomy. Luna: I love that. So what's the one lesson other CTOs should take from this? Lucas: I think it's that incremental rewrites, when paired with a strong load-shedding and observability strategy, can deliver massive scale improvements without the death march of a Big Bang rewrite. Luna: And the numbers don't lie — 10x throughput, 10x lower latency, and fewer outages. Lucas: Exactly. So if you're facing a similar growth curve, consider the strangler pattern and invest in Kafka early.