Breaking

NVIDIA Explains Dense-vs-MoE Tradeoffs Using Nemotron 3.5 Lightning's 30B/3B Design

September 15, 2026 at 21:34 EDT

Foundation Models
AI Agents

On September 15, 2026, NVIDIA published a technical explainer that uses Nemotron 3.5 Lightning to show why total parameter count can mislead inference planning: the model has 30B parameters, but only 3B are active for each token. The company frames dense and Mixture-of-Experts architecture as a deployment choice that affects throughput, memory cost and serving complexity more than raw model size alone.

Sep. 15, 2026 · NVIDIA

The MoE promise is compute, not free memory

Nemotron 3.5 Lightning shows the split: 30B total parameters live in memory, while 3B are active for each token. That can lift throughput, but serving still has to carry the full model.

30B

total model capacity

active per token

context tokens

74%

routing cost reduction

Active parameters per token

Same unit, drawn proportionally: a dense 31B model uses about ten times the active parameters of Lightning.

31B

Dense model

Nemotron MoE

Takeaway: MoE lowers per-token compute, while memory still follows total parameters.

How a token moves

Dense

All parameters run for every token.

→

Router

Each layer selects a subset of experts.

→

MoE

Only selected experts compute, but all experts stay in memory.

Where the architecture lands

Positive read

High-volume agent execution can use a smaller active path for routine calls.

Critical read

Memory, router balance, quantization and judge latency still decide production value.

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.