BREAKING
AirLLM runs 70B models on 4GB GPUs
One layer at a time
1
Load layer
↓
2
Run compute
↓
3
Release
↓
4
Next layer
0
GB
70B model
0
GB
405B model
0
GB
671B model
Memory bottleneck solved
FP16 70B
140
AirLLM
4
Reach vs speed trade-off
Upside
Reach
●
No mandatory quantization
●
GPU acceleration kept
●
pip install airllm
Downside
Speed
●
~0.2 tokens per second
●
I/O bound on disk
●
Best for offline batch jobs
Trade speed for reach
AI NEWS BLITZ
AirLLM lets massive language models run on a single 4GB consumer GPU.