Skip to content

vllm.v1.attention.backends.mla.triton_mla

_reserve_attn_logits_workspace(vllm_config, max_decode_rows, q_num_heads, kv_lora_rank, max_kv_len=None, caller='metadata builder')

Pre-size the shared workspace for the decode split-KV attn logits.

Reserving at the worst case (max_model_len -> max num_kv_splits, max_decode_rows) before warmup/cudagraph capture means the per-call get_simultaneous in forward_mqa never has to grow the buffer at runtime (which would raise once the workspace is locked).

NOTE: max_kv_len is not upstream. forward_mqa clamps its KV extent to the layer's sliding window (min(max_seq_len, decode_sliding_window)), so a windowed layer only ever uses _compute_num_kv_splits(window) splits. Without this, a layer with a 2k window reserves for max_model_len's 256 splits instead of 4, i.e. ~17 GiB instead of ~256 MiB of scratch that the KV cache then has to be sized around.

Source code in vllm/v1/attention/backends/mla/triton_mla.py
def _reserve_attn_logits_workspace(
    vllm_config: VllmConfig,
    max_decode_rows: int,
    q_num_heads: int,
    kv_lora_rank: int,
    max_kv_len: int | None = None,
    caller: str = "metadata builder",
) -> None:
    """Pre-size the shared workspace for the decode split-KV attn logits.

    Reserving at the worst case (max_model_len -> max num_kv_splits,
    max_decode_rows) before warmup/cudagraph capture means the per-call
    ``get_simultaneous`` in ``forward_mqa`` never has to grow the buffer at
    runtime (which would raise once the workspace is locked).

    NOTE: ``max_kv_len`` is not upstream. ``forward_mqa`` clamps its KV extent
    to the layer's sliding window (``min(max_seq_len, decode_sliding_window)``),
    so a windowed layer only ever uses ``_compute_num_kv_splits(window)`` splits.
    Without this, a layer with a 2k window reserves for max_model_len's 256
    splits instead of 4, i.e. ~17 GiB instead of ~256 MiB of scratch that the KV
    cache then has to be sized around.
    """
    if not is_workspace_manager_initialized():
        return
    kv_len = (
        vllm_config.model_config.max_model_len if max_kv_len is None else max_kv_len
    )
    max_splits = _compute_num_kv_splits(
        kv_len,
        current_platform.num_compute_units(),
    )
    shape = (max_decode_rows, q_num_heads, max_splits, kv_lora_rank + 1)
    logger.info_once(
        "TritonMLA %s reserves %.2f MiB of decode attn logits workspace "
        "(rows=%d, heads=%d, splits=%d, max_kv_len=%d)",
        caller,
        math.prod(shape) * 4 / 2**20,
        max_decode_rows,
        q_num_heads,
        max_splits,
        kv_len,
    )
    current_workspace_manager().get_simultaneous((shape, torch.float32))