Perplexity AI Open-Sources Unigram Tokenizer That Achieves 5x Lower p50 Latency Than Hugging Face — 2026-05-28

Impact Vector: AI Tools · May 28, 2026 · 4m

## Short Segments Perplexity AI's new Unigram tokenizer slashes latency by 5x, while Sakana AI's DiffusionBlocks offer a fresh take on neural network training. Later, we'll dive into how Perplexity's open-source release could reshape tokenization in AI workflows. First, let's explore Sakana AI's innovative approach to training deep networks. Sakana AI introduces DiffusionBlocks, a novel framework for training neural networks block by block. This approach significantly reduces memory requirements, addressing a major bottleneck in deep learning. Traditional end-to-end backpropagation demands…