/plugins/plugin-local-inference/native/configs/gpu/SPECS.md

# Per-GPU specs and autotune math — single-GPU llama.cpp tier

Source of truth for the per-GPU JSON configs in this directory
(`3090.json`, `4090.json`, `5090.json`, `h200.json`) and for the
`gpu-autotune.ts` helper in `packages/app-core/src/services/local-inference/`.

Scope: **one GPU per host**. No tensor parallelism, no NVLink splits,
no multi-tenant scheduling. The product target is "one conversation at
a time on a single GPU box."

Inference engine: **llama.cpp / llama-server** (the buun-llama-cpp fork
that ships the QJL + Polar KV quant kernels). This file does not cover
vLLM / SGLang — those have different memory and parallelism models.

> All `expected_metrics` in the JSON configs are extrapolated, not
> measured. The `_provenance: "extrapolated"` field marks that explicitly.
> A real benchmark on each card replaces these once a runner is wired.

## Spec table

| Card | Arch (CC) | VRAM | Mem-BW | FP16 TFLOPs | FP8 TFLOPs | FP4 TFLOPs | INT4 TFLOPs | Max ctx (rec.) | Max parallel | Target RTF (voice) |
|---|---|---|---|---|---|---|---|---|---|---|
| **RTX 3090** | Ampere `sm_86` | 24 GiB GDDR6X | 936 GB/s | 71 | — | — | 284 | 65 536 | 4 | 0.55 |
| **RTX 4090** | Ada Lovelace `sm_89` | 24 GiB GDDR6X | 1 008 GB/s | 165 | 660 (E4M3/E5M2) | — | 660 | 131 072 | 8 | 0.40 |
| **RTX 5090** | Blackwell `sm_120` | 32 GiB GDDR7 | 1 792 GB/s | 209 | 838 | 1 676 | 838 | 262 144 | 12 | 0.30 |
| **H200 SXM** | Hopper `sm_90` | 141 GiB HBM3e | 4 800 GB/s | 989 | 1 979 | — | 1 979 | 1 048 576 | 16 | 0.20 |

RTF = real-time factor; lower is better. For voice streaming we need
RTF < 1 for steady-state and < 0.5 to leave headroom for TTS + ASR.

### Citations

- **RTX 3090** — NVIDIA Ampere GA102 whitepaper (2020). 24 GB GDDR6X
  at 19.5 Gbps × 384-bit = 936 GB/s. No FP8 tensor cores. Compute
  capability sm_86. flash-attn 2 supported; flash-attn 3 is Hopper-only.
- **RTX 4090** — NVIDIA Ada Lovelace AD102 whitepaper (2022). 24 GB
  GDDR6X at 21 Gbps × 384-bit = 1 008 GB/s. FP8 E4M3/E5M2 tensor cores
  (4th gen). Compute capability sm_89. flash-attn 2; flash-attn 3 kernels
  upstreamed but Hopper-tuned.
- **RTX 5090** — NVIDIA Blackwell GB202 whitepaper / launch deck (2025).
  32 GB GDDR7 at 28 Gbps × 512-bit = 1 792 GB/s. 5th-gen tensor cores
  with FP8 + FP4 (E2M1). Compute capability sm_120. llama.cpp sm_120
  kernel coverage is incomplete in early Blackwell builds — buun-llama-cpp
  records this in `CAPABILITIES.json`; the runtime probes it before
  promising QJL/Polar.
- **H200 SXM** — NVIDIA H200 datasheet (2024). 141 GB HBM3e at 4.8 TB/s.
  FP8 transformer engine (4th gen). Compute capability sm_90. Flash-attn 3
  first-class.

llama.cpp issues:
- Blackwell support tracking: <https://github.com/ggml-org/llama.cpp/issues/11279>
- KV cache quantization Q8/Q4: <https://github.com/ggml-org/llama.cpp/pull/7527>
- flash-attn-3 for Hopper: <https://github.com/ggml-org/llama.cpp/pull/13306>

## Autotune math

Two budgets dominate every choice:

1. **VRAM budget** — model weights + per-slot KV must fit.
2. **Memory bandwidth budget** — steady-state decode throughput is
   (weights-per-token) / (mem-bw). RTF ≈ tokens-per-second-audio /
   tokens-per-second-decode.

### KV-cache cost per token

Transformer KV cost: `bytes/token = 2 × n_layers × n_kv_heads × head_dim × bytes_per_element`
(factor of 2 = K and V).

Legacy Eliza-1 KV baseline (retired Qwen-shaped tiers; active Gemma tiers use
manifest-declared stock Gemma KV plus MTP):

| Bundle | n_layers | n_kv_heads | head_dim | FP16 KiB/tok | Q8K/Q4V KiB/tok | QJL+Polar KiB/tok |
|---|---|---|---|---|---|---|
| 2B class | 28 | 8 | 128 | 112 | 88 | 28 |
| 4B | 36 | 8 | 128 | 144 | 113 | 36 |
| 9B | 48 | 8 | 128 | 192 | 150 | 48 |
| 27B | 62 | 8 | 128 | 248 | 194 | 62 |

### Per-slot KV at recommended context

| Bundle | Ctx | KV quant | KV per slot |
|---|---|---|---|
| 2B | 32k | Q8K/Q4V | 32 768 × 88 KiB = **2.75 GiB** |
| 2B | 32k | QJL+Polar | 32 768 × 28 KiB = **0.88 GiB** |
| 9B | 65k | QJL+Polar | 65 536 × 48 KiB = **3.0 GiB** |
| 27B | 32k | QJL+Polar | 32 768 × 62 KiB = **2.0 GiB** |
| 27B | 128k | QJL+Polar | 131 072 × 62 KiB = **8.0 GiB** |
| 27B | 256k | QJL+Polar | 262 144 × 62 KiB = **16.0 GiB** |

### Parallel slot derivation

VRAM available for KV ≈ `vram - model_weights - reserved_headroom`.
Reserved headroom (driver + activations + drafter): 3 GiB on 24 GB
cards, 4 GiB on 5090, 6 GiB on H200. See `reservedHeadroomGb()` in
`packages/shared/src/local-inference/gpu-profiles.ts`.

**RTX 3090 (24 GiB, no FP8)** — uses Q8K / Q4V KV (Ampere has no q4_polar
kernel on the Polar fork).

- 2B (1.5 GiB model): KV budget 19.5 GiB / 2.75 GiB-per-slot = **7 max
  parallel @ 32k**. Config caps at 8 to leave OS-window headroom.
- 9B (5.4 GiB): KV budget 15.6 GiB / (65 536 × 150 KiB = 9.4 GiB-per-slot
  @ 64k) = 1 parallel @ 64k; at 32k it's 4.7 GiB-per-slot → **3 parallel**.
  Config picks 4 with kvSpillToCpu opt-in at 64k.
- 27B (16.8 GiB): KV budget 4.2 GiB / 2 GiB-per-slot @ 32k = **2 parallel**.

**RTX 4090 (24 GiB, FP8)** — QJL + Polar KV available.

- 2B: KV budget 19.5 GiB / 0.88 GiB-per-slot @ 32k = **16 parallel**
  (we cap at 16 for practical session-count reasons).
- 9B: 18 GiB / 3 GiB-per-slot @ 64k = 6; spec picks **8 parallel @ 32k**
  (slot KV = 1.5 GiB) for voice; 4 @ 64k for chat.
- 27B: 4.2 GiB / 2 GiB-per-slot @ 32k = **2 parallel**.
- voice (omnivoice + small LLM): omnivoice runs on CPU/Metal in fused
  mode; KV is only the small text drafter. Cap **4 parallel** at 8k for
  the voice loop.

**RTX 5090 (32 GiB, FP8/FP4)** — same KV math, 8 GiB more headroom.

- 2B: KV budget 27.5 GiB / 0.88 GiB = 31 → **24 parallel** (we leave
  realistic session headroom).
- 9B: 26.6 GiB / 3 GiB @ 64k = 8 → **12 parallel @ 64k**.
- 27B: 12 GiB / 2 GiB @ 32k = 6 → **6 parallel @ 32k**; at 128k it's
  8 GiB per slot → 1.5 → **1 parallel @ 128k**.

**H200 (141 GiB)** — the marquee box.

- 27b: 8 GiB per slot @ 128k → **16 parallel** (capped).
- 9b: ~0.45 GiB per slot @ 8k → **64 parallel**.

### Batch / ubatch derivation

- `batch_size` = logical batch fed to the prefill kernel per server tick.
  Doubles with VRAM (more headroom for activations) but caps at 4096 —
  beyond that, llama.cpp scheduler overhead eats the win.
- `ubatch_size` = physical micro-batch the GPU launches. Ada / Blackwell
  / Hopper want `≥ 512` to keep tensor cores saturated; Ampere is
  happiest at 256-512.

| Card | batch | ubatch | Why |
|---|---|---|---|
| 3090 | 2048 | 512 | Ampere; mem-bw-bound past 512 ubatch |
| 4090 | 2048 | 512 | Same dies as 3090 family; FP8 helps prompt eval not decode |
| 5090 | 4096 | 1024 | More SMs + GDDR7 bw lets the bigger ubatch land |
| H200 | 4096 | 2048 | HBM3e + sm_90 tensor cores; bigger ubatch wins |

### `n_gpu_layers`

Always **999** (all layers on GPU). Single-GPU only — we never split
across cards in this tier. The literal `-1` works equally well in
llama.cpp but `999` is unambiguous and survives clamping in older builds.

### `split_mode` / `main_gpu`

Always `"none"` / `0`. We never multi-GPU.

### `cache_type_k` / `cache_type_v`

- **Ampere (3090)**: `q8_0` / `q4_polar`. The q4_polar Polar-quant V
  kernel exists for sm_86 but the qjl1_256 K kernel does not — fall back
  to Q8 K.
- **Ada / Blackwell / Hopper (4090 / 5090 / H200)**: `qjl1_256` / `q4_polar`.
  Both kernels are pre-built and exposed in `CAPABILITIES.json`.

### `ctx_checkpoints` / `ctx_checkpoint_interval`

Used by the voice optimistic-rollback path. Mid-prefill snapshots cost
~per-checkpoint = `slot_kv_at_checkpoint`. Defaults per
`ctxCheckpointsForTier()` in `packages/shared/src/local-inference/catalog.ts`:

| Bundle | ctx_checkpoints | interval |
|---|---|---|
| 2B | 4 | 4 096 |
| 4B / 9B | 8 | 8 192 |
| 27B (incl. 256k) | 16 | 8 192 |

### MTP draft range

Per-card, picked from `mtpDraftMin` / `mtpDraftMax` in `gpu-profiles.ts`:

| Card | min | max |
|---|---|---|
| 3090 | 4 | 16 |
| 4090 | 4 | 24 |
| 5090 | 4 | 24 |
| H200 | 8 | 32 |

Draft window scales with compute throughput, not memory. Bigger cards
can verify a longer drafter run per round without latency hit.

### `p_min` / `draft_p_min`

`0.5` everywhere — drafter token accepted only if `p ≥ 0.5`. This is a
conservative default for voice latency. Higher values mean fewer
accepted drafts; lower values raise rollback waste.

## Known limits

- **3090 has no FP8** — 27B quality drops slightly without FP8
  attention; we keep Q8K KV for safety. Don't promise FP8 on `sm_86`.
- **5090 sm_120 kernel coverage** — early Blackwell llama.cpp builds may
  not ship `qjl1_256` for sm_120. The runtime probes
  `CAPABILITIES.json`; missing → fall back to `q8_0`/`q4_0` and surface
  a structured warning rather than silently. Don't fix in the autotune;
  fix in the kernel build.
- **flash-attn-3** — Hopper only (sm_90). 4090 / 5090 use flash-attn-2.
- **24 GiB cards at 27B + ≥64k ctx** — fits only with QJL+Polar AND
  single slot AND `--mlock`. Beyond 64k, opt-in `kvSpillToCpu=true`.
- **H200 256k @ 6 parallel** — radix cache helps when sessions share a
  long system prefix; otherwise fresh conversations should be scheduled
  conservatively to avoid KV pressure.

## Override mechanism

The autotune helper merges in this order (later wins):

1. `gpu-profiles.ts` static profile defaults
2. `packages/inference/configs/gpu/<id>.json` (this directory)
3. Bundle-specific override block (`bundle_recommendations.<bundle>`)
4. Per-call `overrides` arg to `selectGpuConfig()` (used by the CLI)
5. Env vars: `ELIZA_LOCAL_*` (see `ffi-streaming-backend.ts` for the full list,
   e.g. `ELIZA_LOCAL_UBATCH_SIZE`, `ELIZA_LOCAL_N_PARALLEL`).

When `selectGpuConfig()` gets a GPU it doesn't recognize, it falls back
on a VRAM bucket:

| VRAM (GiB) | Bucket | Falls back to |
|---|---|---|
| < 12 | tiny | Returns `null` — use catalog defaults |
| 12 – 18 | small | RTX 3090 profile, parallel halved |
| 18 – 28 | mid | RTX 3090 |
| 28 – 40 | mid-plus | RTX 5090 (capped) |
| 40 – 80 | large | RTX 5090 |
| ≥ 80 | huge | H200 |

Bucket fallback is "best effort" — if the user has an unsupported card,
log the fallback choice loudly so they know they're not on a tuned
profile.