Breaking

Karpathy's llm.c Trains LLMs in Pure C/CUDA, Runs ~7% Faster Than PyTorch

July 2, 2026 at 12:43 EDT

Open Source
Software Dev & Coding
Foundation Models

A GitHub repository led by AI researcher Andrej Karpathy, llm.c, is drawing attention as a project that implements LLM pretraining entirely in raw C and CUDA, with no dependency on PyTorch or cPython. Its latest CUDA implementation reportedly runs about 7% faster than PyTorch Nightly on an identical training run.

April 2024 · llm.c · MIT License

Training GPT in pure C & CUDA — no PyTorch, no Python

Andrej Karpathy's open-source project pretrains LLMs at the metal, stripping away framework abstractions — and clocks in roughly 7% faster than PyTorch Nightly.

~7%

faster than PyTorch Nightly

~1,000

lines of C in the CPU reference

1.6B

params, up to GPT-3 miniseries

vs PyTorch — reported gains

PyTorch

+19%

llm.c

Training speed

PyTorch

−29%

llm.c

Memory usage

Higher speed, lower memory footprint — reported against a PyTorch baseline.

GPT-2 · 124M

~90 min · ~$20

reproduced on a single node

GPT-2 · 1.6B

~24 hrs · ~$600–672

on a node with 8× H100

WHAT DEVELOPERS LIKE

Free of Python environment dependencies
Approachable for those with a C background
Excellent for learning low-level mechanics
Clear, step-by-step layernorm tutorial

THE CAVEATS

GPU-oriented; CPU version lags PyTorch
cuDNN lengthens compile times
CUDA knowledge needed for debugging
Large 500+ line optimization PRs often declined

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.