# Eliza-1 local inference — memory management + end-to-end latency review Status: living document. Grounded in a code audit (file:line cited) of the current `plugins/plugin-local-inference` + iOS bridge, not aspirational docs. Goal: minimize model evictions, maximize parallelism/co-residency, and drive voice-in → handled → voice-out to the lowest wall-clock on Mac and iOS across the 8 / 12 / 16 / 20 / 24 / 48 / 128 GB device classes — with per-model and overall telemetry and an on-device grind-all-models test that proves every Eliza-1 model (text, ASR, TTS, VAD, embedding, vision) works. --- ## 1. The end-to-end voice waterfall (what actually happens today) mic → VAD → ASR → (separate drafter ∥ target verifier) → phrase chunker → TTS → ring buffer → speaker. Source: `services/voice/pipeline.ts`, `scheduler.ts`, `phrase-chunker.ts`, `engine.ts`. Instrumented checkpoints already exist (`services/latency-trace.ts:72-86`, `voiceLatencyTracer`): `vad-trigger, vad-speech-start, prewarm-fired, asr-first-partial, asr-final, llm-first-token, llm-first-replytext-char, replyText-first-emotion-tag, phrase-1-to-tts, tts-first-audio-chunk, audio-first-played`. Derived headline metrics: **TTFT** (vad→first token), **TTFA** (vad→first audio chunk), **TTAP** (vad→audio played). Exposed at `GET /api/dev/voice-latency`. ### Measured-today path (Mac, fused harness) — where time goes Approx wall-clock for first audio (2B entry tier, from harness defaults + code): | Stage | Span | Typical | Blocking? | |---|---|---|---| | VAD endpoint | mic → asr start | ~32 ms hop | gates ASR | | **ASR** | `pipeline.ts:317-480` feed+flush | ~300 ms fused local ASR budget | **fully blocks** drafter | | drafter round-1 | `pipeline.ts:350-358` | ~50–100 ms | awaited | | verifier | `pipeline.ts:386` | ~50–100 ms | overlapped w/ next draft | | phrase boundary | `phrase-chunker.ts:39-46` | 0–**700 ms** (time-budget flush) | **gates first TTS** | | **TTS first chunk** | `scheduler.ts:588-700` | ~97 ms (Kokoro) / ~200 ms (OmniVoice) | streaming | | ring → sink | `scheduler.ts:714-754` | <5 ms | sync | **Headline: ~300–600 ms to first audio best-case; the two largest controllable costs are (a) ASR fully blocking generation and (b) the phrase-chunker time-budget flush (700 ms default).** --- ## 2. The biggest inefficiencies (ranked, with file:line) ### Voice-loop serialization 1. **ASR fully completes before the LM starts** (`pipeline.ts:317`, `transcriber.flush()` at `:475`). No streaming-ASR → incremental LM. Mitigated only by speculative-on-pause (`turn-controller.ts:276-284`) which runs off the *partial* transcript and is discarded if it changes. 2. **Phrase-chunker time-budget flush = up to 700 ms first-audio delay** on token streams without early punctuation (`phrase-chunker.ts:28-46`, `ELIZA_PHRASE_FLUSH_MS`; timeout-flush at `:131-135`). The 8-token cap helps but punctuation-sparse replies still wait. 3. **MTP reject → TTS rollback re-synthesis** (`pipeline.ts:398-410`, `scheduler.reject`). Speculatively-synthesized chunks are dropped when the verifier rejects the draft tail; the corrected tokens re-synthesize. 4. **Per-chunk `await` serializes token→TTS fanout** (`engine.ts:2277-2289`): each accepted token `await`s the previous scheduler push (a serial promise chain) before the next chunk is processed. ### Memory / eviction 5. **One model per role, unload-then-load on every modality swap** (`memory-arbiter.ts:505-526`, `engine.ts:515-527`). Text↔vision↔asr↔tts of the *same role* force a full unload+load (seconds), even when both fit RAM. Different roles can co-reside, but the conservative swap policy + `waitForRefcountZero` microtask poll (`memory-arbiter.ts:583-597`) stalls swaps. 6. **Vision is a separate arbiter role but has no standalone model** (`service.ts:505-554`, role registration `:146`): vision-describe reuses the text bundle's projector and is co-resident with text. The separate `vision` role exists only for memory tracking/priority — it cannot load standalone (evicting text unloads the projector, leaving vision with nothing to run). Correctness note (verified): this is a role abstraction, **not** an independent-load bug; the win is to keep vision pinned to the text bundle's lifecycle rather than tracking it as if separately loadable. 7. **RAM budget re-derived per load** (`ram-budget.ts:158-179`): manifest read + KV synthesis on every load decision, uncached. ### iOS-specific 8. **ASR PCM shipped as a JSON number array over stdio** (`bridge.ts` — type `:191-194`, ASR handler `:2582-2585`): 1 s @ 16 kHz = 16k floats ≈ **~100 KB+ JSON vs 64 KB binary**, plus parse cost. Should be base64 / binary. 9. **No token streaming across the JS↔native IPC** (`bridge.ts:1790-1819`): `llama_generate` returns the whole completion in one `host_result`, then passes it to onStreamChunk as a single chunk. TTS gets early-token fanout only on the desktop in-process path; iOS waits for the full text before the message handler acts. 10. **No KV-cache quantization default on iOS** (`LlamaBridgeImpl.swift:682-687`): f16 KV doubles per-model RAM vs q8_0, pushing the MTP/co-residency threshold up. 11. **No runtime memory arbiter on iOS** (`LlamaBridgeImpl.swift:1507-1565`, `hardwareInfo`/`availableMemoryGB`): a single ~3 GB free-RAM capability probe gates MTP; no per-request rebalance, no jetsam-warning handler. (The TS `MemoryArbiter` does not run in the iOS native bridge.) 12. **TTS round-trips through a temp WAV file** (`LlamaBridgeImpl.swift:1310-1315`). ### Streaming / engine 13. **No KV reuse across stateless turns** (`engine.ts:721-723`): the default session resets chat history each turn → full prompt re-prefill. Conversation- pinned `cacheKey` slots avoid this; voice partials do not. 14. **MTP is a runtime/engine concern, with catalog metadata limited to 2B+**: active 2B+ tiers speculate via a **separate drafter**: Gemma 4 ships an official drafter GGUF alongside the target, and the catalog advertises it as a distinct drafter component. Correctness note (verified): the "only the fused backend speculates" behavior lives in the engine/backend; the catalog only advertises the draft window for tiers that carry a usable head. --- ## 3. Per-RAM-tier model load + co-residency policy (target design) Today's tiers (`device-tier.ts:52-72`): MAX ≥24 GB eff, GOOD ≥12, OKAY ≥6, POOR <6. Effective-model-RAM (`device-tier.ts:109-115`): Apple Silicon = total; dGPU = max(vram, total·0.5); CPU = total·0.5. Reserve 1536 MB (`ram-budget.ts:45`). The user asked specifically about 8/12/16/20/24/48/128 GB. Proposed resident set (co-resident = loaded simultaneously, no swap) to **eliminate voice-loop evictions**: | Device RAM | Text tier | Voice resident set (no evict) | Notes | |---|---|---|---| | **8 GB** (iPhone/base Mac) | 2B Q4 + separate drafter | ASR(0.6B) **xor** TTS resident; VAD(2 MB) always; embedding via text `--pooling last` | Tight: keep VAD+text hot; ASR and TTS alternate but pre-warm the next. Gemma 4 E2B runs stock q8_0 KV (MQA, dual head dims 512 global / 256 SWA). **This is the eviction-sensitive tier.** | | **12 GB** | 2B | text + ASR + TTS(kokoro) + VAD all resident | GOOD: no per-turn swaps; vision projector co-resident with text. | | **16 GB** | 2B/4B | + vision mmproj + OmniVoice co-resident | full multimodal hot. | | **20–24 GB** | 4B (MAX) | + image-gen warm, larger KV / longer ctx | all models parallelized + resident. | | **48 GB** | 9B | everything resident + multi-context (parallel rooms) | run drafter+target+voice with headroom. | | **128 GB** | 27B / 27b-256k | full set + 256k ctx + concurrent agents | no memory pressure ever; maximize parallel slots. | **Policy changes to get there:** - Co-resident roles by RAM headroom, not the blanket one-per-role swap. The arbiter already supports multi-role residency; the fix is to **stop evicting on modality change when the resident-set fits the budget** (`memory-arbiter.ts:505-526`). - Keep VAD + the active text model **pinned** (never evictable) in voice mode. - **Pre-warm the next stage's model** during the current stage (ASR running → warm TTS; LM generating → keep ASR warm for barge-in re-ASR) instead of fire-and-forget evicting ASR (`pipeline.ts:326-328`). - Run stock q8_0 KV on the shipped Gemma 4 tiers (MQA, dual head dims 512 global / 256 SWA — the head_dim=128 QJL/Polar kernels don't apply); use smaller Eliza-1 tiers, context clamps, and resident-set preloading for memory-constrained devices. - Cache the resolved RAM budget per (modelKey, ctx) instead of re-deriving (`ram-budget.ts`). --- ## 4. Optimization plan (ranked by expected wall-clock / eviction win) High confidence (implement + verify): 1. **Co-residency instead of swap** when the resident set fits budget — kills the seconds-scale ASR↔TTS↔vision reload on GOOD+ tiers. (`memory-arbiter.ts`) 2. **iOS PCM IPC: base64/binary, not JSON array** — removes ~100 KB JSON encode/parse per ASR call. (`bridge.ts` ASR route + Swift `handleAsrTranscribe`) 3. **Memory-aware tier/context selection on iOS + 8/12 GB tiers** — preserve correctness with F16 KV while selecting the smaller Eliza-1 tier/context that fits. (`LlamaBridgeImpl.swift`, load-args resolution) 4. **Pin VAD + active text model in voice mode; pre-warm next stage** — removes cold-load stalls between ASR and TTS. (`pipeline.ts`, `memory-arbiter.ts`) 5. **Lower/condition the phrase-flush budget + flush on first clause** — cut up to ~700 ms off TTFA. (`phrase-chunker.ts`) — ✅ **done.** The first phrase of each reply now flushes on a shorter `firstPhraseMaxAccumulationMs` budget (derived = `min(350, maxAccumulationMs/2)`, env `ELIZA_PHRASE_FLUSH_FIRST_MS`, reset per reply); later phrases keep the full 700 ms so the bulk is not fragmented. Clause-boundary flush (`, : ;`) already shipped. The scheduler picks the shorter window up automatically via `msUntilTimeBudget()`. Medium: 6. KV reuse for voice partials (don't reset history when the prefix is stable). (`engine.ts:721`) 7. Verify-before-TTS-dispatch for the last K draft tokens to cut rollback re-synth. (`pipeline.ts`) 8. iOS token streaming over IPC (incremental `host_result` frames) so the message handler + TTS act on first tokens. (`bridge.ts`, `FullBunEngineHost.swift`) 9. Cache RAM-budget resolution. (`ram-budget.ts`) — ✅ **done.** `defaultManifestLoader` now memoizes the manifest read+parse+validate keyed on `modelId + manifestSha256`, so the recommender (scores the whole catalog per refresh) and the load gate stop re-hitting disk. The SHA is the validated manifest's content hash → a re-downloaded bundle self-invalidates (no stale budgets); the legacy no-SHA path reads through unchanged. Tests 4/4. --- ## 5. On-device grind-all-models telemetry test (the deliverable) A self-test that, on the phone, loads + exercises EVERY Eliza-1 model and emits per-model + overall timing/telemetry until we are confident they all work. Built on the existing hooks (`latency-trace.ts`, `e2e-harness.ts` scoring, the iOS bridge diagnostics) — see `docs/ondevice-model-grind.md` (companion) for the route + runner. Per-model metrics: load ms, first-token/first-audio/first-result ms, throughput (tok/s or RTF), peak RSS delta, WER (ASR round-trip), VAD boundary MAE, pass/fail. Overall: e2e voice loop TTAP, 30-turn endurance, peak RSS, eviction count. --- ## 6. Open builds blocking live measurement - **Mac fused loop** needs `libelizainference.dylib` for `darwin-arm64-metal-fused` (only the stub exists, `omnivoice-fuse/libelizainference_stub.dylib`; no darwin-fused target in `build-llama-cpp-mtp.mjs`). Until built, `voice-duet`/`voice-e2e-hardware` reject the silent stub. - **iOS** already has the fused lib (`ios-arm64-metal-fused`) — on-device grind is the faster route to real numbers. CoreML Kokoro (this branch) adds the ANE TTS path.