Why AI Is Replacing Tokens With Bytes

Not a ToE · Oct 29, 2026 · 52m

Today's language models never see raw text. Before learning begins, a hand-written pre-tokenizer and a fixed BPE vocabulary cut everything into tokens, like a paint-by-numbers grid laid over a masterpiece. This episode looks at what happens when we remove that layer. We cover Cartesia's hierarchical networks (H-Nets), which learn to chunk raw bytes dynamically inside the model; the "bitter lesson" argument for learning end-to-end from raw data; Jing Liang's claim that tokenization draws the boundary of machine cognition; recent work showing byte-level models start behind token models but…