Lay It Out by the Question: Columnar Layouts and Inverted Indexes

Deep End Era. Engineering with Sankalp · Oct 28, 2026 · 54m

In Dublin, a music-streaming start-up reads two terabytes to add up one number per country. In part nineteen of The Deep End's Data Structures series, data lead Siobhán Kelly and new engineer Arjun Pillai turn their tables on their side: Parquet's columns, encodings and footers, then the inverted index behind every search box, from Lucene's segments to skip lists and BM25. Two read-optimised structures that turn out to be close cousins. Siobhán and Arjun are fictional; the formats, papers and people are real.

Chapters

00:00:00 Cold open: two terabytes to add up one number

00:02:07 Rows and…