Autonomous AI Experiments Deployed at Scale; Five Failure Modes Surface in Production

A major deployment of Andrej Karpathy's AutoResearch framework has revealed how autonomous experiments can run at production scale—and where the system breaks down. Researchers running the system for twelve weeks in a book recommendation pipeline executed 220+ experiments autonomously, achieving a 1.82× improvement in recall and a 2.1× lift in coherence over hand-tuned baselines. But operating at scale exposed five recurring failure modes absent from laboratory conditions: infrastructure fragility, agent memory decay, search-direction stagnation, iteration-cost asymmetry, and metric fixation.