【第175期】TensorLLM：使用多头自注意力提升模型能力

Seventy3 · Mar 24, 2025 · 13m

Seventy3: 用NotebookLM将论文生成播客，让大家跟着AI一起进步。

今天的主题是：

TensorLLM: Tensorising Multi-Head Attention for Enhanced Reasoning and Compression in LLMs

Summary

This research introduces TensorLLM, a novel framework for improving the reasoning abilities and compression of Large Language Models (LLMs) by focusing on the Multi-Head Attention (MHA) block. The method employs multi-head tensorisation and Tucker decomposition to denoise and compress MHA weights by enforcing a shared higher-dimensional subspace across multiple attention heads. Experiments demonstrate that TensorLLM enhances LLM reasoning…