Inference Infrastructure14 min
Continuous Batching
Explore token-budget scheduling, chunked prefill, preemption, and backpressure in a small runnable simulation.
Huang Tzu Lin
1 post
Explore token-budget scheduling, chunked prefill, preemption, and backpressure in a small runnable simulation.