Breaking

Gemma 4 E2B Hits 255 Tokens Per Second in the Browser via WebGPU

July 1, 2026 at 16:27 EDT

Foundation Models
Open Source
AI Agents

Transformers.js creator Xenova used Anthropic's Fable 5 to generate custom WebGPU kernels that accelerated Google's open-weight gemma-4-E2B to 255 tokens per second for fully in-browser inference, and published a demo. It runs locally on an M4 Mac and can be tried by anyone in a browser.

On-device AI · WebGPU

An LLM Wrote the Kernels — and Hit 255 tok/s in the Browser

Using an agentic model to iteratively generate, verify and optimize custom WebGPU kernels, Gemma 4 E2B now decodes at 255 tokens/sec inside a browser on an Apple M4 Max — no installation, all inference on the local GPU.

255 tok/s

Decode speed after agentic optimization

128K

Context window (Gemma 4 E2B)

2.3B

Effective params (5.1B w/ embeddings)

Decode speed, same benchmark (M4 Max, tok/s)

Taller column = faster. All values share one unit: tokens/sec.

~76

LiteRT-LM
reference

~84

Before
optimization

255

Agentic
kernels

The optimized build runs roughly 3× faster than the earlier plateau — and well beyond the LiteRT-LM reference on the same chip.

The agentic optimization loop

Generate

Model writes WebGPU kernel code

→

Verify

Checks correctness & performance

→

Improve

Refines, then repeats the loop

The inference engine itself was written by an LLM — described as "the future of on-device inference."

Praise

Data stays in the browser — strong privacy
Instant to try, no installation needed
Concrete case of an LLM optimizing kernels itself

Caution

Benchmark tied to Apple M4 Max — hardware-dependent
Reproducibility harder to read
Stability on long/complex tasks still unverified

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.