NVIDIA Publishes Five Guidelines for Tuning LLM Speculative Decoding

NVIDIA has published five guidelines for tuning speculative decoding, a technique intended to make large-language-model inference faster without changing the target model’s accepted output sequence. The guidance treats draft length and draft architecture as a joint model, workload and hardware optimization problem.