Time, State and Failure: Stream Processing with Kafka and Flink

Deep End Era. Engineering with Sankalp · Oct 28, 2026 · 52m

At 3:17 a.m. a pipeline leak alarm fires twice, then the leak detection goes silent for twenty minutes while the volume balance is still wrong. In episode one hundred and twelve, streaming engineer Mireille Dagenais and hydraulics engineer Anselm Varga find out why. We start with the humblest data structure in computing, the append-only log, and see how Kafka turned it into a company's nervous system. Then the three hard problems: what time is it (event time, processing time and watermarks), what do I remember (keyed state, windows and joins), and what happens when I crash (checkpoint…