Inference Systems31 min
Making Each Token Cheaper
Bound decode time by bytes moved, then check what quantization, speculative decoding, and attention kernels actually save.
Huang Tzu Lin
TagBlog
#Quantization
1 post
1 post
Bound decode time by bytes moved, then check what quantization, speculative decoding, and attention kernels actually save.