{"id":180149,"date":"2026-06-09T18:06:37","date_gmt":"2026-06-09T18:06:37","guid":{"rendered":"https:\/\/ktromedia.com\/?p=180149"},"modified":"2026-06-09T18:06:37","modified_gmt":"2026-06-09T18:06:37","slug":"serving-multiple-users-at-once-how-continuous-batching-keeps-llm-inference-efficient","status":"publish","type":"post","link":"https:\/\/ktromedia.com\/?p=180149","title":{"rendered":"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient"},"content":{"rendered":"
\n

“”<\/span>“<\/span><\/p>\n

Continuous batching = iteration-level scheduling + ragged (packed) batching.<\/span><\/p>\n

\u00a0<\/p>\n

Two approaches are compared (both run BATCH_SIZE sequences concurrently, so the<\/span><\/p>\n

comparison is slot-for-slot fair):<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a01. Static batching (baseline):<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Prompts are processed BATCH_SIZE at a time.\u00a0\u00a0Each wave is padded to a<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 common length and run together until the LONGEST request in that wave<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 finishes; a hard “<\/span>batch <\/span>barrier<\/span>” then has to clear before the next wave<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 starts.\u00a0\u00a0Short requests sit idle behind the barrier.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a02. Continuous batching (production-aligned):<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Two ideas combine to keep the GPU busy.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 (a) Iteration-level scheduling: the moment a sequence finishes it frees<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 its slot, and the next queued prompt is admitted on the SAME step –<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 no waiting for the rest of the batch.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 (b) Ragged \/ packed batching – the part that makes it truly “<\/span>continuous<\/span>“:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 instead of padding every sequence into a rectangular [B, max_len]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 tensor, ALL in-flight tokens are concatenated into a single unpadded<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 [1, total_tokens] row and run in ONE forward pass.\u00a0\u00a0A block-diagonal<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 causal attention mask stops tokens from attending across sequence<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 boundaries, so packing is mathematically identical to running each<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 sequence on its own (verified: greedy output matches per-prompt<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 generation token-for-token).<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Because attention is governed entirely by the mask, a newly admitted<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 prompt’s multi-token PREFILL rides along in the same forward pass as<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 every other sequence’s single-token DECODE step.\u00a0\u00a0Prefill and decode are<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 fused: no padding, no separate prefill pass.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 KV cache: each sequence keeps its own DynamicCache; every step the caches<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 are concatenated along the time axis into one packed cache, and the newly<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 computed KV is scattered back per sequence.\u00a0\u00a0(Real engines store the<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 cache in fixed-size pages – “<\/span>paged <\/span>attention<\/span>” – to avoid this per-step<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 reassembly, but the attention\/masking logic is exactly what you see here.)<\/span><\/p>\n

“<\/span>“”<\/span><\/p>\n

\u00a0<\/p>\n

import <\/span>time<\/span><\/p>\n

\u00a0<\/p>\n

import <\/span>torch<\/span><\/p>\n

from <\/span>dataclasses <\/span>import <\/span>dataclass<\/span>,<\/span> <\/span>field<\/span><\/p>\n

from <\/span>typing <\/span>import <\/span>Optional<\/span><\/p>\n

from <\/span>transformers <\/span>import <\/span>AutoTokenizer<\/span>,<\/span> <\/span>AutoModelForCausalLM<\/span>,<\/span> <\/span>DynamicCache<\/span><\/p>\n

from <\/span>transformers<\/span>.<\/span>cache_utils <\/span>import <\/span>DynamicLayer<\/span><\/p>\n

\u00a0<\/p>\n

MODEL_ID<\/span> <\/span>=<\/span> <\/span>“openai-community\/gpt2”<\/span>\u00a0\u00a0 <\/span># swap for any causal LM<\/span><\/p>\n

BATCH_SIZE<\/span> <\/span>=<\/span> <\/span>3<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># max concurrent sequences (slots)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

def <\/span>_device_sync<\/span>(<\/span>model<\/span>)<\/span> <\/span>-><\/span> <\/span>None<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>“”<\/span>“Block until queued GPU work finishes, so timings are accurate.”<\/span>“”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>model<\/span>.<\/span>device<\/span>.<\/span>type<\/span> <\/span>==<\/span> <\/span>“cuda”<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>torch<\/span>.<\/span>cuda<\/span>.<\/span>synchronize<\/span>(<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>elif <\/span>model<\/span>.<\/span>device<\/span>.<\/span>type<\/span> <\/span>==<\/span> <\/span>“mps”<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>torch<\/span>.<\/span>mps<\/span>.<\/span>synchronize<\/span>(<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

def <\/span>static_batching<\/span>(<\/span>requests<\/span>:<\/span> <\/span>list<\/span>[<\/span>tuple<\/span>[<\/span>str<\/span>,<\/span> <\/span>int<\/span>]<\/span>]<\/span>,<\/span> <\/span>tokenizer<\/span>,<\/span> <\/span>model<\/span>)<\/span> <\/span>-><\/span> <\/span>list<\/span>[<\/span>str<\/span>]<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>“”<\/span>“Baseline. Process requests BATCH_SIZE at a time; each wave runs together<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0until its LONGEST request finishes, then a batch barrier clears before the<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0next wave starts.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0Downside: short requests in a wave idle until the wave’s longest is done –<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0and no slot can be refilled until the whole wave clears the barrier.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0“<\/span>“”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>not<\/span> <\/span>requests<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>return<\/span> <\/span>[<\/span>]<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>tokenizer<\/span>.<\/span>padding_side<\/span> <\/span>=<\/span> <\/span>“left”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>results<\/span>:<\/span> <\/span>dict<\/span>[<\/span>int<\/span>,<\/span> <\/span>str<\/span>]<\/span> <\/span>=<\/span> <\/span>{<\/span>}<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>indexed<\/span> <\/span>=<\/span> <\/span>list<\/span>(<\/span>enumerate<\/span>(<\/span>requests<\/span>)<\/span>)<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># (req_id, (prompt, cap))<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>wave_start <\/span>in<\/span> <\/span>range<\/span>(<\/span>0<\/span>,<\/span> <\/span>len<\/span>(<\/span>indexed<\/span>)<\/span>,<\/span> <\/span>BATCH_SIZE<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>wave<\/span> <\/span>=<\/span> <\/span>indexed<\/span>[<\/span>wave_start<\/span>:<\/span> <\/span>wave_start<\/span> <\/span>+<\/span> <\/span>BATCH_SIZE<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>wave_max<\/span> <\/span>=<\/span> <\/span>max<\/span>(<\/span>cap <\/span>for<\/span> <\/span>_<\/span>,<\/span> <\/span>(<\/span>_<\/span>,<\/span> <\/span>cap<\/span>)<\/span> <\/span>in<\/span> <\/span>wave<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># Show which request occupies each slot in this wave.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>slot<\/span>,<\/span> <\/span>(<\/span>req_id<\/span>,<\/span> <\/span>(<\/span>prompt<\/span>,<\/span> <\/span>cap<\/span>)<\/span>)<\/span> <\/span>in<\/span> <\/span>enumerate<\/span>(<\/span>wave<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>”\u00a0\u00a0++ slot {slot} ,<\/span> <\/span>flush<\/span>=<\/span>True<\/span>)<\/span><\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>prompts<\/span> <\/span>=<\/span> <\/span>[<\/span>p<\/span> <\/span>for<\/span> <\/span>_<\/span>,<\/span> <\/span>(<\/span>p<\/span>,<\/span> <\/span>_<\/span>)<\/span> <\/span>in<\/span> <\/span>wave<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>inputs<\/span> <\/span>=<\/span> <\/span>tokenizer<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>prompts<\/span>,<\/span> <\/span>return_tensors<\/span>=<\/span>“pt”<\/span>,<\/span> <\/span>padding<\/span>=<\/span>True<\/span>,<\/span> <\/span>truncation<\/span>=<\/span>True<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span>.<\/span>to<\/span>(<\/span>model<\/span>.<\/span>device<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>with <\/span>torch<\/span>.<\/span>no_grad<\/span>(<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>output_ids<\/span> <\/span>=<\/span> <\/span>model<\/span>.<\/span>generate<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>*<\/span>*<\/span>inputs<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>max_new_tokens<\/span>=<\/span>wave_max<\/span>,<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># whole wave decodes to the longest<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>pad_token_id<\/span>=<\/span>tokenizer<\/span>.<\/span>eos_token_id<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>do_sample<\/span>=<\/span>False<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>width<\/span> <\/span>=<\/span> <\/span>inputs<\/span>.<\/span>input_ids<\/span>.<\/span>shape<\/span>[<\/span>1<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>f<\/span>”\u00a0\u00a0*** batch barrier: all {len(wave)} slots wait for the longest “<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>f<\/span>“({wave_max} tokens) ***”<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>flush<\/span>=<\/span>True<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>slot<\/span>,<\/span> <\/span>(<\/span>(<\/span>req_id<\/span>,<\/span> <\/span>(<\/span>prompt<\/span>,<\/span> <\/span>cap<\/span>)<\/span>)<\/span>,<\/span> <\/span>row<\/span>)<\/span> <\/span>in<\/span> <\/span>enumerate<\/span>(<\/span>zip<\/span>(<\/span>wave<\/span>,<\/span> <\/span>output_ids<\/span>)<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>text<\/span> <\/span>=<\/span> <\/span>prompt<\/span> <\/span>+<\/span> <\/span>tokenizer<\/span>.<\/span>decode<\/span>(<\/span>row<\/span>[<\/span>width<\/span>:<\/span>width<\/span> <\/span>+<\/span> <\/span>cap<\/span>]<\/span>,<\/span> <\/span>skip_special_tokens<\/span>=<\/span>True<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>results<\/span>[<\/span>req_id<\/span>]<\/span> <\/span>=<\/span> <\/span>text<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>f<\/span>”\u00a0\u00a0— slot {slot} done\u00a0\u00a0req {req_id} ({cap}\/{wave_max} tokens): {text[:90]}”<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>flush<\/span>=<\/span>True<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>return<\/span> <\/span>[<\/span>results<\/span>[<\/span>k<\/span>]<\/span> <\/span>for<\/span> <\/span>k<\/span> <\/span>in<\/span> <\/span>sorted<\/span>(<\/span>results<\/span>)<\/span>]<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

@<\/span>dataclass<\/span><\/p>\n

class<\/span> <\/span>Sequence<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>“”<\/span>“State for a single in-flight sequence.”<\/span>“”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>req_id<\/span>:<\/span> <\/span>int<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># original request index (for ordering results)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>prompt<\/span>:<\/span> <\/span>str<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>max_new_tokens<\/span>:<\/span> <\/span>int<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># per-request cap so short requests finish early<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Tokens to feed on the NEXT step: the whole prompt right after admission<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># (prefill), then a single token per step (decode).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>pending_ids<\/span>:<\/span> <\/span>list<\/span>[<\/span>int<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Per-sequence KV-cache; None until this sequence has run once.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>kv_cache<\/span>:<\/span> <\/span>Optional<\/span>[<\/span>DynamicCache<\/span>]<\/span> <\/span>=<\/span> <\/span>None<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>kv_len<\/span>:<\/span> <\/span>int<\/span> <\/span>=<\/span> <\/span>0<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># number of cached tokens (prompt + generated)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>tokens_generated<\/span>:<\/span> <\/span>int<\/span> <\/span>=<\/span> <\/span>0<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>output_ids<\/span>:<\/span> <\/span>list<\/span>[<\/span>int<\/span>]<\/span> <\/span>=<\/span> <\/span>field<\/span>(<\/span>default_factory<\/span>=<\/span>list<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

def <\/span>_make_cache<\/span>(<\/span>layers_kv<\/span>:<\/span> <\/span>list<\/span>[<\/span>tuple<\/span>[<\/span>torch<\/span>.<\/span>Tensor<\/span>,<\/span> <\/span>torch<\/span>.<\/span>Tensor<\/span>]<\/span>]<\/span>)<\/span> <\/span>-><\/span> <\/span>DynamicCache<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>“”<\/span>“Build a DynamicCache from explicit per-layer (keys, values) tensors.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0We SET the tensors directly instead of calling DynamicLayer.update() (which<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0would append), because we are assembling caches from scratch each step.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0“<\/span>“”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>cache<\/span> <\/span>=<\/span> <\/span>DynamicCache<\/span>(<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>k<\/span>,<\/span> <\/span>v<\/span> <\/span>in<\/span> <\/span>layers_kv<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>layer<\/span> <\/span>=<\/span> <\/span>DynamicLayer<\/span>(<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>layer<\/span>.<\/span>lazy_initialization<\/span>(<\/span>k<\/span>,<\/span> <\/span>v<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>layer<\/span>.<\/span>keys<\/span> <\/span>=<\/span> <\/span>k<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>layer<\/span>.<\/span>values<\/span> <\/span>=<\/span> <\/span>v<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>cache<\/span>.<\/span>layers<\/span>.<\/span>append<\/span>(<\/span>layer<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>return<\/span> <\/span>cache<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

def <\/span>_ragged_step<\/span>(<\/span>seqs<\/span>:<\/span> <\/span>list<\/span>[<\/span>Sequence<\/span>]<\/span>,<\/span> <\/span>model<\/span>,<\/span> <\/span>device<\/span>,<\/span> <\/span>dtype<\/span>)<\/span> <\/span>-><\/span> <\/span>list<\/span>[<\/span>int<\/span>]<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>“”<\/span>“Run ONE packed forward pass over every active sequence.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0All sequences are flattened into a single row (batch dim = 1):<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0input_ids\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0[1, total_q]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0– every sequence’s pending tokens<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0position_ids\u00a0\u00a0 [1, total_q]\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0– each token’s position in ITS sequence<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0attention_mask [1, 1, total_q, total_kv + total_q]\u00a0\u00a0– block-diagonal causal<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0past_key_values\u00a0\u00a0packed cache [1, H, total_kv, D]<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0total_q\u00a0\u00a0= sum of pending tokens (1 per decoding seq, prompt_len per new seq)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0total_kv = sum of already-cached tokens across sequences<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0Returns the next greedy token for each sequence (same order as “seqs“).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0“<\/span>“”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>q_lens<\/span> <\/span>=<\/span> <\/span>[<\/span>len<\/span>(<\/span>s<\/span>.<\/span>pending_ids<\/span>)<\/span> <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>seqs<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>total_q<\/span> <\/span>=<\/span> <\/span>sum<\/span>(<\/span>q_lens<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>total_kv<\/span> <\/span>=<\/span> <\/span>sum<\/span>(<\/span>s<\/span>.<\/span>kv_len <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>seqs<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Packed inputs: concatenate every sequence’s pending tokens into one row.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>flat_ids<\/span> <\/span>=<\/span> <\/span>[<\/span>t<\/span> <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>seqs <\/span>for<\/span> <\/span>t<\/span> <\/span>in<\/span> <\/span>s<\/span>.<\/span>pending_ids<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>input_ids<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>tensor<\/span>(<\/span>[<\/span>flat_ids<\/span>]<\/span>,<\/span> <\/span>dtype<\/span>=<\/span>torch<\/span>.<\/span>long<\/span>,<\/span> <\/span>device<\/span>=<\/span>device<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Tag every KEY and every QUERY token with (sequence index, position-in-sequence).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Key space is laid out as [ cached tokens | this step’s new tokens ], matching<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># how the model appends new KV to the end of the packed cache.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>key_seq<\/span>,<\/span> <\/span>key_pos<\/span> <\/span>=<\/span> <\/span>[<\/span>]<\/span>,<\/span> <\/span>[<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>si<\/span>,<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>enumerate<\/span>(<\/span>seqs<\/span>)<\/span>:<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># cached block<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>p<\/span> <\/span>in<\/span> <\/span>range<\/span>(<\/span>s<\/span>.<\/span>kv_len<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>key_seq<\/span>.<\/span>append<\/span>(<\/span>si<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>key_pos<\/span>.<\/span>append<\/span>(<\/span>p<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>q_seq<\/span>,<\/span> <\/span>q_pos<\/span> <\/span>=<\/span> <\/span>[<\/span>]<\/span>,<\/span> <\/span>[<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>si<\/span>,<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>enumerate<\/span>(<\/span>seqs<\/span>)<\/span>:<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># new block (also queries)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>j<\/span> <\/span>in<\/span> <\/span>range<\/span>(<\/span>len<\/span>(<\/span>s<\/span>.<\/span>pending_ids<\/span>)<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>pos<\/span> <\/span>=<\/span> <\/span>s<\/span>.<\/span>kv_len<\/span> <\/span>+<\/span> <\/span>j<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>q_seq<\/span>.<\/span>append<\/span>(<\/span>si<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>q_pos<\/span>.<\/span>append<\/span>(<\/span>pos<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>key_seq<\/span>.<\/span>append<\/span>(<\/span>si<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>key_pos<\/span>.<\/span>append<\/span>(<\/span>pos<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>q_seq_t<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>tensor<\/span>(<\/span>q_seq<\/span>,<\/span> <\/span>device<\/span>=<\/span>device<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>q_pos_t<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>tensor<\/span>(<\/span>q_pos<\/span>,<\/span> <\/span>device<\/span>=<\/span>device<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>key_seq_t<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>tensor<\/span>(<\/span>key_seq<\/span>,<\/span> <\/span>device<\/span>=<\/span>device<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>key_pos_t<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>tensor<\/span>(<\/span>key_pos<\/span>,<\/span> <\/span>device<\/span>=<\/span>device<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Each token’s positional embedding uses its own sequence position, not its<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># offset in the packed row.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>position_ids<\/span> <\/span>=<\/span> <\/span>q_pos_t<\/span>.<\/span>unsqueeze<\/span>(<\/span>0<\/span>)<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># [1, total_q]<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Block-diagonal causal mask: a query may attend to a key only if they belong<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># to the SAME sequence (block-diagonal) and the key is not in the future<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># (causal).\u00a0\u00a0This is the whole trick – it makes packing equivalent to running<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># each sequence separately.\u00a0\u00a00.0 = attend, large-negative = blocked (additive).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>same<\/span> <\/span>=<\/span> <\/span>q_seq_t<\/span>[<\/span>:<\/span>,<\/span> <\/span>None<\/span>]<\/span> <\/span>==<\/span> <\/span>key_seq_t<\/span>[<\/span>None<\/span>,<\/span> <\/span>:<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>causal<\/span> <\/span>=<\/span> <\/span>key_pos_t<\/span>[<\/span>None<\/span>,<\/span> <\/span>:<\/span>]<\/span> <\/span> <\/span>q_pos_t<\/span>[<\/span>:<\/span>,<\/span> <\/span>None<\/span>]<\/span><\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>allowed<\/span> <\/span>=<\/span> <\/span>same<\/span> <\/span>&<\/span> <\/span>causal<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># [total_q, total_kv + total_q]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>attn_mask<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>zeros<\/span>(<\/span>1<\/span>,<\/span> <\/span>1<\/span>,<\/span> <\/span>total_q<\/span>,<\/span> <\/span>total_kv<\/span> <\/span>+<\/span> <\/span>total_q<\/span>,<\/span> <\/span>dtype<\/span>=<\/span>dtype<\/span>,<\/span> <\/span>device<\/span>=<\/span>device<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>attn_mask<\/span>.<\/span>masked_fill_<\/span>(<\/span>~<\/span>allowed<\/span>[<\/span>None<\/span>,<\/span> <\/span>None<\/span>]<\/span>,<\/span> <\/span>torch<\/span>.<\/span>finfo<\/span>(<\/span>dtype<\/span>)<\/span>.<\/span>min<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Packed KV-cache: concatenate each sequence’s cache along the time axis.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Freshly admitted sequences (kv_len == 0) contribute nothing here.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>cached<\/span> <\/span>=<\/span> <\/span>[<\/span>s<\/span> <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>seqs <\/span>if<\/span> <\/span>s<\/span>.<\/span>kv_len<\/span> <\/span>><\/span> <\/span>0<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>cached<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>num_layers<\/span> <\/span>=<\/span> <\/span>len<\/span>(<\/span>cached<\/span>[<\/span>0<\/span>]<\/span>.<\/span>kv_cache<\/span>.<\/span>layers<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>layers_kv<\/span> <\/span>=<\/span> <\/span>[<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>l<\/span> <\/span>in<\/span> <\/span>range<\/span>(<\/span>num_layers<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>ks<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>cat<\/span>(<\/span>[<\/span>s<\/span>.<\/span>kv_cache<\/span>.<\/span>layers<\/span>[<\/span>l<\/span>]<\/span>.<\/span>keys <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>cached<\/span>]<\/span>,<\/span> <\/span>dim<\/span>=<\/span>2<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>vs<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>cat<\/span>(<\/span>[<\/span>s<\/span>.<\/span>kv_cache<\/span>.<\/span>layers<\/span>[<\/span>l<\/span>]<\/span>.<\/span>values <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>cached<\/span>]<\/span>,<\/span> <\/span>dim<\/span>=<\/span>2<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>layers_kv<\/span>.<\/span>append<\/span>(<\/span>(<\/span>ks<\/span>,<\/span> <\/span>vs<\/span>)<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>past<\/span> <\/span>=<\/span> <\/span>_make_cache<\/span>(<\/span>layers_kv<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>else<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>past<\/span> <\/span>=<\/span> <\/span>DynamicCache<\/span>(<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>with <\/span>torch<\/span>.<\/span>no_grad<\/span>(<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>out<\/span> <\/span>=<\/span> <\/span>model<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>input_ids<\/span>=<\/span>input_ids<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>attention_mask<\/span>=<\/span>attn_mask<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>position_ids<\/span>=<\/span>position_ids<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>past_key_values<\/span>=<\/span>past<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>use_cache<\/span>=<\/span>True<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Greedy next token for each sequence: read the logits at its LAST pending<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># token (for a prefilling sequence that is the final prompt token).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>logits<\/span> <\/span>=<\/span> <\/span>out<\/span>.<\/span>logits<\/span>[<\/span>0<\/span>]<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># [total_q, vocab]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>offsets<\/span>,<\/span> <\/span>last_idx<\/span>,<\/span> <\/span>off<\/span> <\/span>=<\/span> <\/span>[<\/span>]<\/span>,<\/span> <\/span>[<\/span>]<\/span>,<\/span> <\/span>0<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>ql <\/span>in<\/span> <\/span>q_lens<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>offsets<\/span>.<\/span>append<\/span>(<\/span>off<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>last_idx<\/span>.<\/span>append<\/span>(<\/span>off<\/span> <\/span>+<\/span> <\/span>ql<\/span> <\/span>–<\/span> <\/span>1<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>off<\/span> <\/span>+=<\/span> <\/span>ql<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>next_tokens<\/span> <\/span>=<\/span> <\/span>[<\/span>int<\/span>(<\/span>logits<\/span>[<\/span>i<\/span>]<\/span>.<\/span>argmax<\/span>(<\/span>)<\/span>)<\/span> <\/span>for<\/span> <\/span>i<\/span> <\/span>in<\/span> <\/span>last_idx<\/span>]<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Scatter the newly computed KV back to each sequence.\u00a0\u00a0The output cache is<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># [ old packed block | new packed block ]; slice this step’s new block per<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># sequence and append it to that sequence’s own cache.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>out_kv<\/span> <\/span>=<\/span> <\/span>out<\/span>.<\/span>past_key_values<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>num_layers<\/span> <\/span>=<\/span> <\/span>len<\/span>(<\/span>out_kv<\/span>.<\/span>layers<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>si<\/span>,<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>enumerate<\/span>(<\/span>seqs<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>o<\/span>,<\/span> <\/span>ql<\/span> <\/span>=<\/span> <\/span>offsets<\/span>[<\/span>si<\/span>]<\/span>,<\/span> <\/span>q_lens<\/span>[<\/span>si<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>layers_kv<\/span> <\/span>=<\/span> <\/span>[<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>l<\/span> <\/span>in<\/span> <\/span>range<\/span>(<\/span>num_layers<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>k_new<\/span> <\/span>=<\/span> <\/span>out_kv<\/span>.<\/span>layers<\/span>[<\/span>l<\/span>]<\/span>.<\/span>keys<\/span>[<\/span>:<\/span>,<\/span> <\/span>:<\/span>,<\/span> <\/span>total_kv<\/span> <\/span>+<\/span> <\/span>o<\/span>:<\/span> <\/span>total_kv<\/span> <\/span>+<\/span> <\/span>o<\/span> <\/span>+<\/span> <\/span>ql<\/span>,<\/span> <\/span>:<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>v_new<\/span> <\/span>=<\/span> <\/span>out_kv<\/span>.<\/span>layers<\/span>[<\/span>l<\/span>]<\/span>.<\/span>values<\/span>[<\/span>:<\/span>,<\/span> <\/span>:<\/span>,<\/span> <\/span>total_kv<\/span> <\/span>+<\/span> <\/span>o<\/span>:<\/span> <\/span>total_kv<\/span> <\/span>+<\/span> <\/span>o<\/span> <\/span>+<\/span> <\/span>ql<\/span>,<\/span> <\/span>:<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>s<\/span>.<\/span>kv_cache <\/span>is<\/span> <\/span>None<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>layers_kv<\/span>.<\/span>append<\/span>(<\/span>(<\/span>k_new<\/span>,<\/span> <\/span>v_new<\/span>)<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>else<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>layers_kv<\/span>.<\/span>append<\/span>(<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>torch<\/span>.<\/span>cat<\/span>(<\/span>[<\/span>s<\/span>.<\/span>kv_cache<\/span>.<\/span>layers<\/span>[<\/span>l<\/span>]<\/span>.<\/span>keys<\/span>,<\/span> <\/span>k_new<\/span>]<\/span>,<\/span> <\/span>dim<\/span>=<\/span>2<\/span>)<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>torch<\/span>.<\/span>cat<\/span>(<\/span>[<\/span>s<\/span>.<\/span>kv_cache<\/span>.<\/span>layers<\/span>[<\/span>l<\/span>]<\/span>.<\/span>values<\/span>,<\/span> <\/span>v_new<\/span>]<\/span>,<\/span> <\/span>dim<\/span>=<\/span>2<\/span>)<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>s<\/span>.<\/span>kv_cache<\/span> <\/span>=<\/span> <\/span>_make_cache<\/span>(<\/span>layers_kv<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>s<\/span>.<\/span>kv_len<\/span> <\/span>+=<\/span> <\/span>ql<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>return<\/span> <\/span>next_tokens<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

def <\/span>visualize_ragged_step<\/span>(<\/span>seqs<\/span>:<\/span> <\/span>list<\/span>[<\/span>Sequence<\/span>]<\/span>,<\/span> <\/span>tokenizer<\/span>,<\/span> <\/span>title<\/span>:<\/span> <\/span>str<\/span>,<\/span> <\/span>slot_ids<\/span>:<\/span> <\/span>list<\/span>[<\/span>int<\/span>]<\/span>)<\/span> <\/span>-><\/span> <\/span>None<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>“”<\/span>“Illustrative print of ONE packed step: the concatenated input row and the<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0block-diagonal causal attention mask.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0This mirrors the masking logic in _ragged_step (recomputed here as a boolean<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0grid purely for display) so you can SEE that sequences are packed together<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0yet isolated by the mask.\u00a0\u00a0Each sequence gets a letter A, B, C, …<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0# = a query may attend to that key\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0. = blocked<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0“<\/span>“”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>labels<\/span> <\/span>=<\/span> <\/span>[<\/span>chr<\/span>(<\/span>ord<\/span>(<\/span>“A”<\/span>)<\/span> <\/span>+<\/span> <\/span>s<\/span>.<\/span>req_id<\/span>)<\/span> <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>seqs<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>q_lens<\/span> <\/span>=<\/span> <\/span>[<\/span>len<\/span>(<\/span>s<\/span>.<\/span>pending_ids<\/span>)<\/span> <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>seqs<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>total_q<\/span> <\/span>=<\/span> <\/span>sum<\/span>(<\/span>q_lens<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>total_kv<\/span> <\/span>=<\/span> <\/span>sum<\/span>(<\/span>s<\/span>.<\/span>kv_len <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>seqs<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>“\\n{‘=’ * 72}\\n\u00a0\u00a0{title}”<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>”\u00a0\u00a0total_q={total_q} tokens fed this step\u00a0\u00a0|\u00a0\u00a0total_kv={total_kv} cached”<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>”\u00a0\u00a0{len(seqs)} sequences packed into ONE unpadded row of shape [1, {total_q}]:\\n”<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># The concatenated tokens, grouped per sequence (this is the “ragged” row).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>i<\/span>,<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>enumerate<\/span>(<\/span>seqs<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>kind<\/span> <\/span>=<\/span> <\/span>f<\/span>“PREFILL({q_lens[i]})”<\/span> <\/span>if<\/span> <\/span>s<\/span>.<\/span>kv_len<\/span> <\/span>==<\/span> <\/span>0<\/span> <\/span>else<\/span> <\/span>f<\/span>“decode({q_lens[i]})”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>toks<\/span> <\/span>=<\/span> <\/span>” “<\/span>.<\/span>join<\/span>(<\/span>repr<\/span>(<\/span>tokenizer<\/span>.<\/span>decode<\/span>(<\/span>[<\/span>t<\/span>]<\/span>)<\/span>)<\/span> <\/span>for<\/span> <\/span>t<\/span> <\/span>in<\/span> <\/span>s<\/span>.<\/span>pending_ids<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>len<\/span>(<\/span>toks<\/span>)<\/span> <\/span>><\/span> <\/span>66<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>toks<\/span> <\/span>=<\/span> <\/span>toks<\/span>[<\/span>:<\/span>63<\/span>]<\/span> <\/span>+<\/span> <\/span>“…”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>”\u00a0\u00a0\u00a0\u00a0{labels[i]} = slot {slot_ids[i]}\u00a0\u00a0{kind:)<\/span><\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Rebuild the block-diagonal causal mask as a boolean grid for display.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>key_seq<\/span>,<\/span> <\/span>key_pos<\/span> <\/span>=<\/span> <\/span>[<\/span>]<\/span>,<\/span> <\/span>[<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>si<\/span>,<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>enumerate<\/span>(<\/span>seqs<\/span>)<\/span>:<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># cached keys<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>key_seq<\/span> <\/span>+=<\/span> <\/span>[<\/span>si<\/span>]<\/span> <\/span>*<\/span> <\/span>s<\/span>.<\/span>kv_len<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>key_pos<\/span> <\/span>+=<\/span> <\/span>list<\/span>(<\/span>range<\/span>(<\/span>s<\/span>.<\/span>kv_len<\/span>)<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>q_seq<\/span>,<\/span> <\/span>q_pos<\/span> <\/span>=<\/span> <\/span>[<\/span>]<\/span>,<\/span> <\/span>[<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>si<\/span>,<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>enumerate<\/span>(<\/span>seqs<\/span>)<\/span>:<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># new keys \/ queries<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>j<\/span> <\/span>in<\/span> <\/span>range<\/span>(<\/span>q_lens<\/span>[<\/span>si<\/span>]<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>q_seq<\/span>.<\/span>append<\/span>(<\/span>si<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>q_pos<\/span>.<\/span>append<\/span>(<\/span>s<\/span>.<\/span>kv_len<\/span> <\/span>+<\/span> <\/span>j<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>key_seq<\/span> <\/span>+=<\/span> <\/span>q_seq<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>key_pos<\/span> <\/span>+=<\/span> <\/span>q_pos<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>q_seq_t<\/span>,<\/span> <\/span>q_pos_t<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>tensor<\/span>(<\/span>q_seq<\/span>)<\/span>,<\/span> <\/span>torch<\/span>.<\/span>tensor<\/span>(<\/span>q_pos<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>key_seq_t<\/span>,<\/span> <\/span>key_pos_t<\/span> <\/span>=<\/span> <\/span>torch<\/span>.<\/span>tensor<\/span>(<\/span>key_seq<\/span>)<\/span>,<\/span> <\/span>torch<\/span>.<\/span>tensor<\/span>(<\/span>key_pos<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>allowed<\/span> <\/span>=<\/span> <\/span>(<\/span>q_seq_t<\/span>[<\/span>:<\/span>,<\/span> <\/span>None<\/span>]<\/span> <\/span>==<\/span> <\/span>key_seq_t<\/span>[<\/span>None<\/span>,<\/span> <\/span>:<\/span>]<\/span>)<\/span> <\/span>&<\/span> <\/span>(<\/span>key_pos_t<\/span>[<\/span>None<\/span>,<\/span> <\/span>:<\/span>]<\/span> <\/span> <\/span>q_pos_t<\/span>[<\/span>:<\/span>,<\/span> <\/span>None<\/span>]<\/span>)<\/span><\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>K<\/span> <\/span>=<\/span> <\/span>len<\/span>(<\/span>key_seq<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>def <\/span>row_str<\/span>(<\/span>cells<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># Space between sequence groups; ‘ | ‘ at the cached -> new-tokens split.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>out<\/span> <\/span>=<\/span> <\/span>[<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>ki <\/span>in<\/span> <\/span>range<\/span>(<\/span>K<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>total_kv<\/span> <\/span>><\/span> <\/span>0<\/span> <\/span>and<\/span> <\/span>ki<\/span> <\/span>==<\/span> <\/span>total_kv<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>out<\/span>.<\/span>append<\/span>(<\/span>” | “<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>elif <\/span>ki<\/span> <\/span>><\/span> <\/span>0<\/span> <\/span>and<\/span> <\/span>key_seq<\/span>[<\/span>ki<\/span>]<\/span> <\/span>!=<\/span> <\/span>key_seq<\/span>[<\/span>ki<\/span> <\/span>–<\/span> <\/span>1<\/span>]<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>out<\/span>.<\/span>append<\/span>(<\/span>” “<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>out<\/span>.<\/span>append<\/span>(<\/span>cells<\/span>[<\/span>ki<\/span>]<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>return<\/span> <\/span>“”<\/span>.<\/span>join<\/span>(<\/span>out<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>def <\/span>line<\/span>(<\/span>left<\/span>,<\/span> <\/span>cells<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>return<\/span> <\/span>f<\/span>“{left:>7} “<\/span> <\/span>+<\/span> <\/span>row_str<\/span>(<\/span>cells<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>“\\n\u00a0\u00a0block-diagonal causal mask\u00a0\u00a0(row = query, col = key)\u00a0\u00a0 # attend\u00a0\u00a0 . blocked”<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>total_kv<\/span> <\/span>><\/span> <\/span>0<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>”\u00a0\u00a0\u00a0\u00a0key layout:\u00a0\u00a0[ cached KV\u00a0\u00a0|\u00a0\u00a0this step’s new tokens ]”<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>line<\/span>(<\/span>“keys:”<\/span>,<\/span> <\/span>[<\/span>labels<\/span>[<\/span>key_seq<\/span>[<\/span>ki<\/span>]<\/span>]<\/span> <\/span>for<\/span> <\/span>ki <\/span>in<\/span> <\/span>range<\/span>(<\/span>K<\/span>)<\/span>]<\/span>)<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>qi <\/span>in<\/span> <\/span>range<\/span>(<\/span>total_q<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>cells<\/span> <\/span>=<\/span> <\/span>[<\/span>“#”<\/span> <\/span>if<\/span> <\/span>allowed<\/span>[<\/span>qi<\/span>,<\/span> <\/span>ki<\/span>]<\/span> <\/span>else<\/span> <\/span>“.”<\/span> <\/span>for<\/span> <\/span>ki <\/span>in<\/span> <\/span>range<\/span>(<\/span>K<\/span>)<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>line<\/span>(<\/span>f<\/span>“{labels[q_seq[qi]]} p{q_pos[qi]}”<\/span>,<\/span> <\/span>cells<\/span>)<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

def <\/span>continuous_batching<\/span>(<\/span>requests<\/span>:<\/span> <\/span>list<\/span>[<\/span>tuple<\/span>[<\/span>str<\/span>,<\/span> <\/span>int<\/span>]<\/span>]<\/span>,<\/span> <\/span>tokenizer<\/span>,<\/span> <\/span>model<\/span>)<\/span> <\/span>-><\/span> <\/span>list<\/span>[<\/span>str<\/span>]<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>“”<\/span>“Ragged continuous batching: dynamic scheduling + packed prefill\/decode.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0Scheduling policy:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0– Up to BATCH_SIZE sequences run concurrently.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0– A newly admitted sequence is queued with its full prompt as the next<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0tokens to feed; its prefill then happens packed into the next step<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0alongside everyone else’s decode.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0– Every step runs ONE packed forward pass across all active slots.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0– When a sequence finishes it is immediately replaced by the next prompt.<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0The admission log shows slots being reused (iteration-level scheduling).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0Two representative steps are visualized: the first step (all prompts being<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0prefilled at once) and the first step that fuses a new prompt’s prefill with<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0other sequences’ decode tokens.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0“<\/span>“”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>device<\/span> <\/span>=<\/span> <\/span>model<\/span>.<\/span>device<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>dtype<\/span> <\/span>=<\/span> <\/span>next<\/span>(<\/span>model<\/span>.<\/span>parameters<\/span>(<\/span>)<\/span>)<\/span>.<\/span>dtype<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>queue<\/span> <\/span>=<\/span> <\/span>list<\/span>(<\/span>enumerate<\/span>(<\/span>requests<\/span>)<\/span>)<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 <\/span># (req_id, (prompt, max_new_tokens))<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>slots<\/span>:<\/span> <\/span>list<\/span>[<\/span>Optional<\/span>[<\/span>Sequence<\/span>]<\/span>]<\/span> <\/span>=<\/span> <\/span>[<\/span>None<\/span>]<\/span> <\/span>*<\/span> <\/span>BATCH_SIZE<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>results<\/span>:<\/span> <\/span>dict<\/span>[<\/span>int<\/span>,<\/span> <\/span>str<\/span>]<\/span> <\/span>=<\/span> <\/span>{<\/span>}<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>def <\/span>_admit<\/span>(<\/span>slot_idx<\/span>:<\/span> <\/span>int<\/span>)<\/span> <\/span>-><\/span> <\/span>None<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>not<\/span> <\/span>queue<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>slots<\/span>[<\/span>slot_idx<\/span>]<\/span> <\/span>=<\/span> <\/span>None<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>return<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>req_id<\/span>,<\/span> <\/span>(<\/span>prompt<\/span>,<\/span> <\/span>max_new_tokens<\/span>)<\/span> <\/span>=<\/span> <\/span>queue<\/span>.<\/span>pop<\/span>(<\/span>0<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>prompt_ids<\/span> <\/span>=<\/span> <\/span>tokenizer<\/span>(<\/span>prompt<\/span>)<\/span>[<\/span>“input_ids”<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>slots<\/span>[<\/span>slot_idx<\/span>]<\/span> <\/span>=<\/span> <\/span>Sequence<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>req_id<\/span>=<\/span>req_id<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>prompt<\/span>=<\/span>prompt<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>max_new_tokens<\/span>=<\/span>max_new_tokens<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>pending_ids<\/span>=<\/span>list<\/span>(<\/span>prompt_ids<\/span>)<\/span>,<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># prefill rides the next step<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>f<\/span>”\u00a0\u00a0++ [step {step:3d}] slot {slot_idx} <\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>f<\/span>“({max_new_tokens} tok cap): {prompt!r}”<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>flush<\/span>=<\/span>True<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Fill the pool with the first batch of prompts (step 0 = before any decode).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>step<\/span> <\/span>=<\/span> <\/span>0<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>i<\/span> <\/span>in<\/span> <\/span>range<\/span>(<\/span>BATCH_SIZE<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>_admit<\/span>(<\/span>i<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>printed_mixed<\/span> <\/span>=<\/span> <\/span>False<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>while<\/span> <\/span>any<\/span>(<\/span>s<\/span> <\/span>is<\/span> <\/span>not<\/span> <\/span>None <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>slots<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>step<\/span> <\/span>+=<\/span> <\/span>1<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>active<\/span> <\/span>=<\/span> <\/span>[<\/span>(<\/span>i<\/span>,<\/span> <\/span>s<\/span>)<\/span> <\/span>for<\/span> <\/span>i<\/span>,<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>enumerate<\/span>(<\/span>slots<\/span>)<\/span> <\/span>if<\/span> <\/span>s<\/span> <\/span>is<\/span> <\/span>not<\/span> <\/span>None<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>seqs<\/span> <\/span>=<\/span> <\/span>[<\/span>s<\/span> <\/span>for<\/span> <\/span>_<\/span>,<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>active<\/span>]<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>slot_ids<\/span> <\/span>=<\/span> <\/span>[<\/span>i<\/span> <\/span>for<\/span> <\/span>i<\/span>,<\/span> <\/span>_<\/span> <\/span>in<\/span> <\/span>active<\/span>]<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># Visualize a couple of representative steps so the packing is visible<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># (printing every step would be far too much output).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>mixed<\/span> <\/span>=<\/span> <\/span>any<\/span>(<\/span>s<\/span>.<\/span>kv_len<\/span> <\/span>==<\/span> <\/span>0<\/span> <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>seqs<\/span>)<\/span> <\/span>and<\/span> <\/span>any<\/span>(<\/span>s<\/span>.<\/span>kv_len<\/span> <\/span>><\/span> <\/span>0<\/span> <\/span>for<\/span> <\/span>s<\/span> <\/span>in<\/span> <\/span>seqs<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>step<\/span> <\/span>==<\/span> <\/span>1<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>visualize_ragged_step<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>seqs<\/span>,<\/span> <\/span>tokenizer<\/span>,<\/span> <\/span>f<\/span>“STEP {step}\u00a0\u00a0–\u00a0\u00a0prompts packed together (all PREFILL)”<\/span>,<\/span> <\/span>slot_ids<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>elif <\/span>mixed <\/span>and<\/span> <\/span>not<\/span> <\/span>printed_mixed<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>visualize_ragged_step<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>seqs<\/span>,<\/span> <\/span>tokenizer<\/span>,<\/span> <\/span>f<\/span>“STEP {step}\u00a0\u00a0–\u00a0\u00a0PREFILL + DECODE fused in one pass”<\/span>,<\/span> <\/span>slot_ids<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>printed_mixed<\/span> <\/span>=<\/span> <\/span>True<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># ONE packed forward pass (prefill + decode fused, no padding).<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>next_tokens<\/span> <\/span>=<\/span> <\/span>_ragged_step<\/span>(<\/span>seqs<\/span>,<\/span> <\/span>model<\/span>,<\/span> <\/span>device<\/span>,<\/span> <\/span>dtype<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>for<\/span> <\/span>(<\/span>slot_idx<\/span>,<\/span> <\/span>seq<\/span>)<\/span>,<\/span> <\/span>tok <\/span>in<\/span> <\/span>zip<\/span>(<\/span>active<\/span>,<\/span> <\/span>next_tokens<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>seq<\/span>.<\/span>output_ids<\/span>.<\/span>append<\/span>(<\/span>tok<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>seq<\/span>.<\/span>tokens_generated<\/span> <\/span>+=<\/span> <\/span>1<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>seq<\/span>.<\/span>pending_ids<\/span> <\/span>=<\/span> <\/span>[<\/span>tok<\/span>]<\/span>\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span># next step: a single decode token<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>tok<\/span> <\/span>==<\/span> <\/span>tokenizer<\/span>.<\/span>eos_token_id <\/span>or<\/span> <\/span>seq<\/span>.<\/span>tokens_generated<\/span> <\/span>>=<\/span> <\/span>seq<\/span>.<\/span>max_new_tokens<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>result_text<\/span> <\/span>=<\/span> <\/span>seq<\/span>.<\/span>prompt<\/span> <\/span>+<\/span> <\/span>\\<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>tokenizer<\/span>.<\/span>decode<\/span>(<\/span>seq<\/span>.<\/span>output_ids<\/span>,<\/span> <\/span>skip_special_tokens<\/span>=<\/span>True<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>results<\/span>[<\/span>seq<\/span>.<\/span>req_id<\/span>]<\/span> <\/span>=<\/span> <\/span>result_text<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>f<\/span>”\u00a0\u00a0— step {step:3d}] slot {slot_idx} done\u00a0\u00a0req {seq.req_id} “<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>f<\/span>“({seq.tokens_generated}\/{seq.max_new_tokens} tokens): {result_text[:90]}”<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>flush<\/span>=<\/span>True<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>_admit<\/span>(<\/span>slot_idx<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>return<\/span> <\/span>[<\/span>results<\/span>[<\/span>k<\/span>]<\/span> <\/span>for<\/span> <\/span>k<\/span> <\/span>in<\/span> <\/span>sorted<\/span>(<\/span>results<\/span>)<\/span>]<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

def <\/span>main<\/span>(<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>“Loading {MODEL_ID}”<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>tokenizer<\/span> <\/span>=<\/span> <\/span>AutoTokenizer<\/span>.<\/span>from_pretrained<\/span>(<\/span>MODEL_ID<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>tokenizer<\/span>.<\/span>pad_token<\/span> <\/span>=<\/span> <\/span>tokenizer<\/span>.<\/span>eos<\/span>_<\/span>token<\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># Pick the fastest available device.\u00a0\u00a0On Apple Silicon (M1\/M2\/…) this is<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># the MPS GPU.\u00a0\u00a0We keep float32 on MPS on purpose: float16 there flips a few<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># greedy ties, which would break the “static == continuous, token-for-token”<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span># property this demo relies on.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>if<\/span> <\/span>torch<\/span>.<\/span>cuda<\/span>.<\/span>is_available<\/span>(<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>device<\/span>,<\/span> <\/span>dtype<\/span> <\/span>=<\/span> <\/span>“cuda”<\/span>,<\/span> <\/span>torch<\/span>.<\/span>float16<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>elif <\/span>torch<\/span>.<\/span>backends<\/span>.<\/span>mps<\/span>.<\/span>is_available<\/span>(<\/span>)<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>device<\/span>,<\/span> <\/span>dtype<\/span> <\/span>=<\/span> <\/span>“mps”<\/span>,<\/span> <\/span>torch<\/span>.<\/span>float32<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>else<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>device<\/span>,<\/span> <\/span>dtype<\/span> <\/span>=<\/span> <\/span>“cpu”<\/span>,<\/span> <\/span>torch<\/span>.<\/span>float32<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>model<\/span> <\/span>=<\/span> <\/span>AutoModelForCausalLM<\/span>.<\/span>from_pretrained<\/span>(<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>MODEL_ID<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>dtype<\/span>=<\/span>dtype<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>attn_implementation<\/span>=<\/span>“eager”<\/span>,<\/span>\u00a0\u00a0 <\/span># use our custom 4D mask directly<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>model<\/span>.<\/span>eval<\/span>(<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>model<\/span>.<\/span>to<\/span>(<\/span>device<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>“Running on {device} ({dtype})\\n”<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>requests<\/span> <\/span>=<\/span> <\/span>[<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>(<\/span>“The capital of France is”<\/span>,<\/span> <\/span>6<\/span>)<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>(<\/span>“Today’s weather is so”<\/span>,<\/span> <\/span>50<\/span>)<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>(<\/span>“In machine learning, a transformer is”<\/span>,<\/span> <\/span>300<\/span>)<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>(<\/span>“Once upon a time in a land far away,”<\/span>,<\/span> <\/span>30<\/span>)<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>(<\/span>“Quantum computing differs from classical computing because”<\/span>,<\/span> <\/span>180<\/span>)<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0<\/span>(<\/span>“The history of the Roman Empire began”<\/span>,<\/span> <\/span>45<\/span>)<\/span>,<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>]<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>“=== Static batching ===”<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>_device_sync<\/span>(<\/span>model<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>start<\/span> <\/span>=<\/span> <\/span>time<\/span>.<\/span>perf_counter<\/span>(<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>static_batching<\/span>(<\/span>requests<\/span>,<\/span> <\/span>tokenizer<\/span>,<\/span> <\/span>model<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>_device_sync<\/span>(<\/span>model<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>static_elapsed<\/span> <\/span>=<\/span> <\/span>time<\/span>.<\/span>perf_counter<\/span>(<\/span>)<\/span> <\/span>–<\/span> <\/span>start<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>“\\nStatic batching elapsed: {static_elapsed:.2f}s\\n”<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>“=== Continuous batching (ragged) ===”<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>_device_sync<\/span>(<\/span>model<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>start<\/span> <\/span>=<\/span> <\/span>time<\/span>.<\/span>perf_counter<\/span>(<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>continuous_batching<\/span>(<\/span>requests<\/span>,<\/span> <\/span>tokenizer<\/span>,<\/span> <\/span>model<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>_device_sync<\/span>(<\/span>model<\/span>)<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>continuous_elapsed<\/span> <\/span>=<\/span> <\/span>time<\/span>.<\/span>perf_counter<\/span>(<\/span>)<\/span> <\/span>–<\/span> <\/span>start<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>print<\/span>(<\/span>f<\/span>“\\nContinuous batching elapsed: {continuous_elapsed:.2f}s”<\/span>)<\/span><\/p>\n

\u00a0<\/p>\n

\u00a0<\/p>\n

if<\/span> <\/span>__name__<\/span> <\/span>==<\/span> <\/span>“__main__”<\/span>:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0<\/span>main<\/span>(<\/span>)<\/span><\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"

“”“ Continuous batching = iteration-level scheduling + ragged (packed) batching. \u00a0 Two approaches are compared (both run BATCH_SIZE sequences concurrently, so the comparison is slot-for-slot fair): \u00a0 \u00a0\u00a01. Static batching (baseline): \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Prompts are processed BATCH_SIZE at a time.\u00a0\u00a0Each wave is padded to a \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 common length and run together until the LONGEST request in<\/p>\n","protected":false},"author":1,"featured_media":180150,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[42],"tags":[],"class_list":{"0":"post-180149","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-ai"},"yoast_head":"\nServing Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient - Ktromedia<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/ktromedia.com\/?p=180149\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient - Ktromedia\" \/>\n<meta property=\"og:description\" content=\"“”“ Continuous batching = iteration-level scheduling + ragged (packed) batching. \u00a0 Two approaches are compared (both run BATCH_SIZE sequences concurrently, so the comparison is slot-for-slot fair): \u00a0 \u00a0\u00a01. Static batching (baseline): \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Prompts are processed BATCH_SIZE at a time.\u00a0\u00a0Each wave is padded to a \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 common length and run together until the LONGEST request in\" \/>\n<meta property=\"og:url\" content=\"https:\/\/ktromedia.com\/?p=180149\" \/>\n<meta property=\"og:site_name\" content=\"Ktromedia\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/KTROMedia\/\" \/>\n<meta property=\"article:published_time\" content=\"2026-06-09T18:06:37+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2560\" \/>\n\t<meta property=\"og:image:height\" content=\"1707\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"KTRO TEAM\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"KTRO TEAM\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"14 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/ktromedia.com\/?p=180149#article\",\"isPartOf\":{\"@id\":\"https:\/\/ktromedia.com\/?p=180149\"},\"author\":{\"name\":\"KTRO TEAM\",\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/person\/612bf2fbac107722ea365932cdd35f5b\"},\"headline\":\"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient\",\"datePublished\":\"2026-06-09T18:06:37+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/ktromedia.com\/?p=180149\"},\"wordCount\":2750,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/ktromedia.com\/#organization\"},\"image\":{\"@id\":\"https:\/\/ktromedia.com\/?p=180149#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg\",\"articleSection\":[\"\u4eba\u5de5\u667a\u80fd\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/ktromedia.com\/?p=180149#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/ktromedia.com\/?p=180149\",\"url\":\"https:\/\/ktromedia.com\/?p=180149\",\"name\":\"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient - Ktromedia\",\"isPartOf\":{\"@id\":\"https:\/\/ktromedia.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/ktromedia.com\/?p=180149#primaryimage\"},\"image\":{\"@id\":\"https:\/\/ktromedia.com\/?p=180149#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg\",\"datePublished\":\"2026-06-09T18:06:37+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/ktromedia.com\/?p=180149#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/ktromedia.com\/?p=180149\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ktromedia.com\/?p=180149#primaryimage\",\"url\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg\",\"contentUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg\",\"width\":2560,\"height\":1707},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/ktromedia.com\/?p=180149#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/ktromedia.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/ktromedia.com\/#website\",\"url\":\"https:\/\/ktromedia.com\/\",\"name\":\"Ktromedia\",\"description\":\"KTRO MEDIA Crypto News\",\"publisher\":{\"@id\":\"https:\/\/ktromedia.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/ktromedia.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/ktromedia.com\/#organization\",\"name\":\"Ktromedia\",\"url\":\"https:\/\/ktromedia.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/11\/ktroicon.png\",\"contentUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/11\/ktroicon.png\",\"width\":250,\"height\":250,\"caption\":\"Ktromedia\"},\"image\":{\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/KTROMedia\/\",\"https:\/\/www.linkedin.com\/company\/ktro-media\/\",\"https:\/\/t.me\/ktrogroup\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/person\/612bf2fbac107722ea365932cdd35f5b\",\"name\":\"KTRO TEAM\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/ktromedia.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/10\/cropped-Untitled-design-7-1-150x150.png\",\"contentUrl\":\"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/10\/cropped-Untitled-design-7-1-150x150.png\",\"caption\":\"KTRO TEAM\"},\"description\":\"KTRO MEDIA \u662f\u4e00\u5bb6\u5168\u7403\u6027\u7684\u534e\u6587WEB3\u5a92\u4f53\u516c\u53f8\u3002\u6211\u4eec\u81f4\u529b\u4e8e\u4e3a\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u9886\u57df\u63d0\u4f9b\u6700\u65b0\u7684\u65b0\u95fb\u3001\u89c1\u89e3\u548c\u8d8b\u52bf\u5206\u6790\u3002\u6211\u4eec\u7684\u5b97\u65e8\u662f\u4e3a\u5168\u7403\u7528\u6237\u63d0\u4f9b\u9ad8\u8d28\u91cf\u3001\u5168\u9762\u7684\u8d44\u8baf\u670d\u52a1\uff0c\u8ba9\u4ed6\u4eec\u66f4\u597d\u5730\u4e86\u89e3\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u7684\u6700\u65b0\u52a8\u6001\u3002\u6211\u4eec\u4e5f\u5e0c\u671b\u80fd\u5e2e\u5230\u66f4\u591a\u4f18\u79c0\u7684WEB3\u4ea7\u54c1\u627e\u5230\u66f4\u591a\u66f4\u597d\u7684\u8d44\u6e90\u597d\u8ba9\u8fd9\u9886\u57df\u53d8\u5f97\u66f4\u6210\u719f\u3002 \u6211\u4eec\u7684\u62a5\u9053\u8303\u56f4\u6db5\u76d6\u4e86\u533a\u5757\u94fe\u3001\u52a0\u5bc6\u8d27\u5e01\u3001\u667a\u80fd\u5408\u7ea6\u3001DeFi\u3001NFT \u548c Web3 \u751f\u6001\u7cfb\u7edf\u7b49\u9886\u57df\u3002\u6211\u4eec\u7684\u62a5\u9053\u4e0d\u4ec5\u6765\u81ea\u884c\u4e1a\u5185\u7684\u4e13\u5bb6\uff0c\u5148\u950b\u8005\u4e5f\u5305\u62ec\u4e86\u6211\u4eec\u81ea\u5df1\u7684\u5206\u6790\u548c\u89c2\u70b9\u3002\u6211\u4eec\u5728\u5404\u4e2a\u56fd\u5bb6\u548c\u5730\u533a\u90fd\u8bbe\u6709\u56e2\u961f\uff0c\u4e3a\u8bfb\u8005\u63d0\u4f9b\u672c\u5730\u5316\u7684\u62a5\u9053\u548c\u5206\u6790\u3002 \u9664\u4e86\u65b0\u95fb\u62a5\u9053\uff0c\u6211\u4eec\u8fd8\u63d0\u4f9b\u5e02\u573a\u7814\u7a76\u548c\u54a8\u8be2\u670d\u52a1\u3002\u6211\u4eec\u7684\u4e13\u4e1a\u56e2\u961f\u53ef\u4ee5\u4e3a\u60a8\u63d0\u4f9b\u6709\u5173\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u7684\u6df1\u5165\u5206\u6790\u548c\u5e02\u573a\u8d8b\u52bf\uff0c\u5e2e\u52a9\u60a8\u505a\u51fa\u66f4\u660e\u667a\u7684\u6295\u8d44\u51b3\u7b56\u3002 \u6211\u4eec\u7684\u4f7f\u547d\u662f\u6210\u4e3a\u5168\u7403\u534e\u6587\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u6700\u53d7\u4fe1\u8d56\u7684\u4fe1\u606f\u6765\u6e90\u4e4b\u4e00\u3002\u6211\u4eec\u5c06\u7ee7\u7eed\u4e0d\u65ad\u52aa\u529b\uff0c\u4e3a\u8bfb\u8005\u63d0\u4f9b\u6700\u65b0\u3001\u6700\u5168\u9762\u3001\u6700\u53ef\u9760\u7684\u4fe1\u606f\u670d\u52a1\u3002\",\"sameAs\":[\"https:\/\/ktromedia.com\"],\"url\":\"https:\/\/ktromedia.com\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient - Ktromedia","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/ktromedia.com\/?p=180149","og_locale":"en_US","og_type":"article","og_title":"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient - Ktromedia","og_description":"“”“ Continuous batching = iteration-level scheduling + ragged (packed) batching. \u00a0 Two approaches are compared (both run BATCH_SIZE sequences concurrently, so the comparison is slot-for-slot fair): \u00a0 \u00a0\u00a01. Static batching (baseline): \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 Prompts are processed BATCH_SIZE at a time.\u00a0\u00a0Each wave is padded to a \u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 common length and run together until the LONGEST request in","og_url":"https:\/\/ktromedia.com\/?p=180149","og_site_name":"Ktromedia","article_publisher":"https:\/\/www.facebook.com\/KTROMedia\/","article_published_time":"2026-06-09T18:06:37+00:00","og_image":[{"width":2560,"height":1707,"url":"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg","type":"image\/jpeg"}],"author":"KTRO TEAM","twitter_card":"summary_large_image","twitter_misc":{"Written by":"KTRO TEAM","Est. reading time":"14 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/ktromedia.com\/?p=180149#article","isPartOf":{"@id":"https:\/\/ktromedia.com\/?p=180149"},"author":{"name":"KTRO TEAM","@id":"https:\/\/ktromedia.com\/#\/schema\/person\/612bf2fbac107722ea365932cdd35f5b"},"headline":"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient","datePublished":"2026-06-09T18:06:37+00:00","mainEntityOfPage":{"@id":"https:\/\/ktromedia.com\/?p=180149"},"wordCount":2750,"commentCount":0,"publisher":{"@id":"https:\/\/ktromedia.com\/#organization"},"image":{"@id":"https:\/\/ktromedia.com\/?p=180149#primaryimage"},"thumbnailUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg","articleSection":["\u4eba\u5de5\u667a\u80fd"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/ktromedia.com\/?p=180149#respond"]}]},{"@type":"WebPage","@id":"https:\/\/ktromedia.com\/?p=180149","url":"https:\/\/ktromedia.com\/?p=180149","name":"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient - Ktromedia","isPartOf":{"@id":"https:\/\/ktromedia.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/ktromedia.com\/?p=180149#primaryimage"},"image":{"@id":"https:\/\/ktromedia.com\/?p=180149#primaryimage"},"thumbnailUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg","datePublished":"2026-06-09T18:06:37+00:00","breadcrumb":{"@id":"https:\/\/ktromedia.com\/?p=180149#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/ktromedia.com\/?p=180149"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ktromedia.com\/?p=180149#primaryimage","url":"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg","contentUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2026\/06\/Serving-Multiple-Users-at-Once-How-Continuous-Batching-Keeps-LLM.jpg","width":2560,"height":1707},{"@type":"BreadcrumbList","@id":"https:\/\/ktromedia.com\/?p=180149#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/ktromedia.com\/"},{"@type":"ListItem","position":2,"name":"Serving Multiple Users at Once: How Continuous Batching Keeps LLM Inference Efficient"}]},{"@type":"WebSite","@id":"https:\/\/ktromedia.com\/#website","url":"https:\/\/ktromedia.com\/","name":"Ktromedia","description":"KTRO MEDIA Crypto News","publisher":{"@id":"https:\/\/ktromedia.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/ktromedia.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/ktromedia.com\/#organization","name":"Ktromedia","url":"https:\/\/ktromedia.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ktromedia.com\/#\/schema\/logo\/image\/","url":"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/11\/ktroicon.png","contentUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/11\/ktroicon.png","width":250,"height":250,"caption":"Ktromedia"},"image":{"@id":"https:\/\/ktromedia.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/KTROMedia\/","https:\/\/www.linkedin.com\/company\/ktro-media\/","https:\/\/t.me\/ktrogroup"]},{"@type":"Person","@id":"https:\/\/ktromedia.com\/#\/schema\/person\/612bf2fbac107722ea365932cdd35f5b","name":"KTRO TEAM","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/ktromedia.com\/#\/schema\/person\/image\/","url":"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/10\/cropped-Untitled-design-7-1-150x150.png","contentUrl":"https:\/\/ktromedia.com\/wp-content\/uploads\/2025\/10\/cropped-Untitled-design-7-1-150x150.png","caption":"KTRO TEAM"},"description":"KTRO MEDIA \u662f\u4e00\u5bb6\u5168\u7403\u6027\u7684\u534e\u6587WEB3\u5a92\u4f53\u516c\u53f8\u3002\u6211\u4eec\u81f4\u529b\u4e8e\u4e3a\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u9886\u57df\u63d0\u4f9b\u6700\u65b0\u7684\u65b0\u95fb\u3001\u89c1\u89e3\u548c\u8d8b\u52bf\u5206\u6790\u3002\u6211\u4eec\u7684\u5b97\u65e8\u662f\u4e3a\u5168\u7403\u7528\u6237\u63d0\u4f9b\u9ad8\u8d28\u91cf\u3001\u5168\u9762\u7684\u8d44\u8baf\u670d\u52a1\uff0c\u8ba9\u4ed6\u4eec\u66f4\u597d\u5730\u4e86\u89e3\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u7684\u6700\u65b0\u52a8\u6001\u3002\u6211\u4eec\u4e5f\u5e0c\u671b\u80fd\u5e2e\u5230\u66f4\u591a\u4f18\u79c0\u7684WEB3\u4ea7\u54c1\u627e\u5230\u66f4\u591a\u66f4\u597d\u7684\u8d44\u6e90\u597d\u8ba9\u8fd9\u9886\u57df\u53d8\u5f97\u66f4\u6210\u719f\u3002 \u6211\u4eec\u7684\u62a5\u9053\u8303\u56f4\u6db5\u76d6\u4e86\u533a\u5757\u94fe\u3001\u52a0\u5bc6\u8d27\u5e01\u3001\u667a\u80fd\u5408\u7ea6\u3001DeFi\u3001NFT \u548c Web3 \u751f\u6001\u7cfb\u7edf\u7b49\u9886\u57df\u3002\u6211\u4eec\u7684\u62a5\u9053\u4e0d\u4ec5\u6765\u81ea\u884c\u4e1a\u5185\u7684\u4e13\u5bb6\uff0c\u5148\u950b\u8005\u4e5f\u5305\u62ec\u4e86\u6211\u4eec\u81ea\u5df1\u7684\u5206\u6790\u548c\u89c2\u70b9\u3002\u6211\u4eec\u5728\u5404\u4e2a\u56fd\u5bb6\u548c\u5730\u533a\u90fd\u8bbe\u6709\u56e2\u961f\uff0c\u4e3a\u8bfb\u8005\u63d0\u4f9b\u672c\u5730\u5316\u7684\u62a5\u9053\u548c\u5206\u6790\u3002 \u9664\u4e86\u65b0\u95fb\u62a5\u9053\uff0c\u6211\u4eec\u8fd8\u63d0\u4f9b\u5e02\u573a\u7814\u7a76\u548c\u54a8\u8be2\u670d\u52a1\u3002\u6211\u4eec\u7684\u4e13\u4e1a\u56e2\u961f\u53ef\u4ee5\u4e3a\u60a8\u63d0\u4f9b\u6709\u5173\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u7684\u6df1\u5165\u5206\u6790\u548c\u5e02\u573a\u8d8b\u52bf\uff0c\u5e2e\u52a9\u60a8\u505a\u51fa\u66f4\u660e\u667a\u7684\u6295\u8d44\u51b3\u7b56\u3002 \u6211\u4eec\u7684\u4f7f\u547d\u662f\u6210\u4e3a\u5168\u7403\u534e\u6587\u533a\u5757\u94fe\u548c\u91d1\u878d\u79d1\u6280\u884c\u4e1a\u6700\u53d7\u4fe1\u8d56\u7684\u4fe1\u606f\u6765\u6e90\u4e4b\u4e00\u3002\u6211\u4eec\u5c06\u7ee7\u7eed\u4e0d\u65ad\u52aa\u529b\uff0c\u4e3a\u8bfb\u8005\u63d0\u4f9b\u6700\u65b0\u3001\u6700\u5168\u9762\u3001\u6700\u53ef\u9760\u7684\u4fe1\u606f\u670d\u52a1\u3002","sameAs":["https:\/\/ktromedia.com"],"url":"https:\/\/ktromedia.com\/?author=1"}]}},"_links":{"self":[{"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/posts\/180149","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=180149"}],"version-history":[{"count":1,"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/posts\/180149\/revisions"}],"predecessor-version":[{"id":180151,"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/posts\/180149\/revisions\/180151"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=\/wp\/v2\/media\/180150"}],"wp:attachment":[{"href":"https:\/\/ktromedia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=180149"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=180149"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ktromedia.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=180149"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}