MODEL INFRA / LONG-CONTEXT CACHEOpen source

HeadWiseKV

Not every KV head should pay the same price for full context

HeadWiseKV is a static, head-wise KV-cache allocation method for long-context LLM inference. It configures cache windows per layer and per KV head, physically changing cache residency in the llama.cpp runtime.

View on GitHubCapacity figures come from the retained public 24 GB 4090D CUDA full-GPU-offload measurements.

01262,144

Confirmed context

UD-Q4_K_XL reaches the model’s full context in the public 4090D measurements.

02159,744

Q6_K + KV q8_0

The unmodified llama.cpp path reaches 53,248 under the same public setup.

0324 GB

Measured VRAM

The retained results use a single 4090D with full CUDA GPU offload.

METHOD / HEAD-WISE ALLOCATION

Replace one model-wide cache policy with a static configuration per layer and head

Identify cache demand

Separate how individual layers and KV heads depend on long-range context instead of assuming one uniform window.

Encode SWA / full windows

Assign sliding or full windows to individual heads in an inspectable static allocation profile.

Change physical residency

The patched llama.cpp runtime slices KV-cache residency rather than adding only a logical attention mask.

Preserve reproduction paths

The repository includes source snapshots, focused patches, default configs, launch scripts, and retained experiment artifacts.

PUBLIC RESULTS

The target scenario is local long-context inference on a single 24 GB consumer GPU

Qwen3.6-27BQ4 / Q5 UD quantizationQ6_K + KV q8_0RTX 3090 / 4090 classGSM8K aggregateLoCoMo-MC10 aggregate

RESULT BOUNDARY

Capacity numbers apply only to the published model, quantization, and hardware conditions

Exact capacity results are 4090D-specific and should not be read as macOS Metal claims or guarantees for every 3090/4090-class GPU. Current quality results lack statistical significance testing and do not include complete raw logs for every capacity probe.