Breaking

Azure, NVIDIA train Llama 3.1 405B in 7.07 minutes on 8,192 GPUs

June 16, 2026 at 20:39 EDT

Infra & Chips
Foundation Models

NVIDIA and Microsoft Azure said they trained Meta's 405-billion-parameter LLM "Llama 3.1 405B" to target quality in just 7.07 minutes, using NVIDIA Blackwell-generation GB200 NVL72 systems at a scale of 8,192 GPUs in the latest industry-standard benchmark MLPerf Training v6.0.

MLPerf Training v6.0 · Microsoft Azure + NVIDIA

Llama 3.1 405B pre-trained in 7.07 minutes

Azure and NVIDIA trained Meta's 405B-parameter model to reference quality across 8,192 Blackwell GPUs on GB200 NVL72 systems — the fastest time recorded for the benchmark and Azure's largest such cluster to date.

7.07 min

Time-to-train to reference quality

8,192

Blackwell GPUs across 128 racks

99.8%

Weak scaling efficiency at extreme scale

Near-perfect weak scaling

Adding 16 more racks barely moved step time — proof that scale-out overhead stayed contained.

1.2712s

7,168 GPUs
(112 racks)

→

1.2734s

8,192 GPUs
(128 racks)

Exposed cross-rack communication ≈ 20ms — about 1.6% of step time, overlapped with computation.

How Blackwell binds 72 GPUs per rack

Intra-rack

1,800 GB/s

per GPU via 5th-gen NVLink — critical-path communication runs here.

Inter-rack

MRC fabric

ConnectX-8 + Spectrum-X Ethernet — data-parallel traffic overlaps with compute.

Developer reaction

Cohere, Midjourney, Thinking Machines Lab, Nebius and Higgsfield cited 2–3× speedups on GB200/GB300 NVL72; GB300 with NVFP4 shows up to 1.6× over GB200 at comparable scale.

Worth noting

MLPerf is a benchmark-centric measure. Long-term evaluation in production environments — over runs lasting weeks to months — is still accumulating.

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.