Breaking

ICML Paper Estimates LLM Memory at About 3.6 Bits per Parameter

July 6, 2026 at 12:05 EDT

Research & Papers
Foundation Models
Security

A paper by researchers from Meta FAIR, Cornell, DeepMind and others estimates the memory capacity of GPT-style language models at about 3.6 bits per parameter. It introduces an information-theoretic framework that separates unintended memorization from generalization, and will be presented at ICML 2026.

ICML 2026 · Meta FAIR · Google DeepMind · Cornell · NVIDIA

Language Models Memorize ≈ 3.6 Bits Per Parameter

By training hundreds of models on random bitstrings, researchers isolated raw memory capacity from generalization — pinning GPT-style Transformers at a fixed, measurable limit.

3.6 bits

Per parameter, at bfloat16 precision

500K–1.5B

Parameter range · hundreds of models

675 MB

Theoretical capacity of a 1.5B model (~5.4B bits)

Higher precision adds almost nothing

Jumping from bfloat16 to float32 barely moves capacity — diminishing returns per parameter (bits/parameter).

3.6

bfloat16

~3.51–3.83

float32

What happens as data exceeds capacity

Fill capacity

Model memorizes specific data points

→

Plateau

Unintended memorization saturates

→

Grokking

Shift to generalization ("double descent")

Why it matters

A concrete, measurable per-bit capacity figure
Larger models on more data dilute per-point memorization — verbatim leaks and membership inference get statistically harder

The caveats

Empirical, limited to GPT-style Transformers
In real text, generalization is mixed in — per-sample memorization is messier
Rare, unique strings remain easy to memorize

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.