Pre-size the shared workspace for the decode split-KV attn logits.
Reserving at the worst case (max_model_len -> max num_kv_splits,
max_decode_rows) before warmup/cudagraph capture means the per-call
get_simultaneous in forward_mqa never has to grow the buffer at
runtime (which would raise once the workspace is locked).
NOTE: max_kv_len is not upstream. forward_mqa clamps its KV extent
to the layer's sliding window (min(max_seq_len, decode_sliding_window)),
so a windowed layer only ever uses _compute_num_kv_splits(window) splits.
Without this, a layer with a 2k window reserves for max_model_len's 256
splits instead of 4, i.e. ~17 GiB instead of ~256 MiB of scratch that the KV
cache then has to be sized around.
Source code in vllm/v1/attention/backends/mla/triton_mla.py
| def _reserve_attn_logits_workspace(
vllm_config: VllmConfig,
max_decode_rows: int,
q_num_heads: int,
kv_lora_rank: int,
max_kv_len: int | None = None,
caller: str = "metadata builder",
) -> None:
"""Pre-size the shared workspace for the decode split-KV attn logits.
Reserving at the worst case (max_model_len -> max num_kv_splits,
max_decode_rows) before warmup/cudagraph capture means the per-call
``get_simultaneous`` in ``forward_mqa`` never has to grow the buffer at
runtime (which would raise once the workspace is locked).
NOTE: ``max_kv_len`` is not upstream. ``forward_mqa`` clamps its KV extent
to the layer's sliding window (``min(max_seq_len, decode_sliding_window)``),
so a windowed layer only ever uses ``_compute_num_kv_splits(window)`` splits.
Without this, a layer with a 2k window reserves for max_model_len's 256
splits instead of 4, i.e. ~17 GiB instead of ~256 MiB of scratch that the KV
cache then has to be sized around.
"""
if not is_workspace_manager_initialized():
return
kv_len = (
vllm_config.model_config.max_model_len if max_kv_len is None else max_kv_len
)
max_splits = _compute_num_kv_splits(
kv_len,
current_platform.num_compute_units(),
)
shape = (max_decode_rows, q_num_heads, max_splits, kv_lora_rank + 1)
logger.info_once(
"TritonMLA %s reserves %.2f MiB of decode attn logits workspace "
"(rows=%d, heads=%d, splits=%d, max_kv_len=%d)",
caller,
math.prod(shape) * 4 / 2**20,
max_decode_rows,
q_num_heads,
max_splits,
kv_len,
)
current_workspace_manager().get_simultaneous((shape, torch.float32))
|