speculative-decoding-lab

Speculative Decoding Lab is a hands-on implementation and benchmarking project for understanding speculative decoding in large language model inference. The repository explores speculative decoding at multiple levels of abstraction: a standard autoregressive baseline, a from-scratch implementation of draft-and-verify decoding, Hugging Face assisted generation, and vLLM speculative decoding. The goal is not only to demonstrate how speculative decoding works, but to measure when and why it improves inference performance using metrics such as latency, throughput, acceptance rate, tokens generated per verification step, and end-to-end speedup. Development is performed using WSL2 with NVIDIA GPU acceleration, with a primary environment of Python 3.13, PyTorch, Hugging Face Transformers, and CUDA. vLLM is maintained in a separate environment to preserve its validated PyTorch and CUDA dependency stack.