Continuous batching from first principles
TL;DR: in this blog post, starting from attention mechanisms and KV caching, we derive continuous batching by optimizing for throughput. If you've ever used Qwen, Claude, or any other AI chatbot, you've probably noticed something: it takes...
Scan with your phone
Scan with your phone