Breaking

Open-Source 'LeetCUDA' Project Offers 200+ GPU Kernels and PyTorch-Based Learning Notes for CUDA Optimization

July 15, 2026 at 06:50 EDT

Open Source
Infra & Chips
Software Dev & Coding

An open-source repository called LeetCUDA is drawing attention among GPU engineers for bundling more than 200 hand-written CUDA kernels, modern learning notes, and PyTorch bindings into a single reference for high-performance NVIDIA GPU programming.

Open-Source · GPU Programming

LeetCUDA: 200+ Hand-Written Kernels to Demystify Tensor Core Programming

A single reference bundling runnable CUDA kernels, benchmarks, learning notes, and PyTorch bindings — centered on half-precision GEMM and a from-scratch FlashAttention-2.

200+

Hand-written CUDA kernels, graded Easy → Hard+

98–100%

of cuBLAS Tensor Core TFLOPS on HGEMM (L20, 4090, 3080)

1e-6

Max numerical error vs. cuBLAS references

Attention throughput vs. baselines

Measured TFLOPS — taller column is faster

FA-2 attention · D=64

RTX 3080 Laptop · ~1.5× vs FA-2

104

Large attention · D=512

L20 (ffpa-attn) · ~1.8× vs SDPA

Baseline (FA-2 / SDPA) LeetCUDA

What's inside

Tensor Cores · WMMA / MMA / CuTe FlashAttention-2 (pure MMA PTX) Block / warp / shmem swizzling Multi-stage pipelining Triton + CUTLASS variants PyTorch bindings + benchmarks

Where it shines

Tight PyTorch integration + detailed notes
Reproducible Tensor Core benchmarks
Strongest gains on smaller problem sizes and specific devices

The caveats

Advantages are not universal
Can trail FA-2 / SDPA on large-scale attention
MMA accumulation & softmax precision cited as factors

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.