# @elizaos/plugin-aosp-local-inference

AOSP-only llama.cpp FFI bindings (via `bun:ffi`) and local-inference bootstrap for elizaOS mobile builds.

## What it does

This package enables on-device inference for Eliza agents running on Android (AOSP). It connects the Bun runtime to a native `libllama.so` + shim via `bun:ffi`, and registers the following model type handlers on `AgentRuntime`:

| Model type | Backend |
|---|---|
| `TEXT_SMALL` / `TEXT_LARGE` | llama.cpp FFI (`libllama.so` via `libeliza-llama-shim.so`) |
| `TEXT_EMBEDDING` | llama.cpp FFI (separate embedding context, disabled by default) |
| `TEXT_TO_SPEECH` | Fused Kokoro (`libelizainference.so`) — WAV output at 24 kHz |
| `TRANSCRIPTION` | ASR via `libelizainference.so` |

All handlers self-gate on `ELIZA_LOCAL_LLAMA=1` and are no-ops on non-AOSP platforms. The package is safe to import unconditionally from the mobile agent bundle.

## Capabilities

- **Local LLM inference** — in-process llama.cpp via `bun:ffi`; no HTTP server subprocess.
- **Speculative decoding (MTP)** — optional in-process MTP drafter via `libeliza-llama-speculative-shim.so`; auto-paired when a `mtp/` drafter GGUF is found in the active model bundle.
- **Custom KV-cache quantization** — supports `q8_0`, `tbq3_0`, `tbq4_0`, `qjl1_256`, `q4_polar` quant types from the elizaOS/llama.cpp fork (default: K=`q8_0`, V=`f16` for chat).
- **Text-to-speech** — Kokoro synthesis via `libelizainference.so` (`eliza_inference_kokoro_*`); pre-warm support to amortize first-request latency.
- **Transcription** — ASR via `libelizainference.so` from PCM WAV input or `{ pcm, sampleRateHz }` params.
- **Model auto-download** — if no bundled GGUF is found, downloads `elizaos/eliza-1` from HuggingFace (opt-out: `ELIZA_DISABLE_MODEL_AUTO_DOWNLOAD=1`).
- **Cloud fallback** — for `TEXT_SMALL` / `TEXT_LARGE`, a secondary handler at priority -1 forwards to Eliza Cloud when the local FFI path fails with a recoverable error.

## How to enable

This package is loaded by `@elizaos/agent`'s mobile entrypoint and by `@elizaos/plugin-local-inference`'s `ensure-local-inference-handler.ts`. It is not an elizaOS plugin with auto-discovery — activation requires calling `ensureAospLocalInferenceHandlers(runtime)` after the agent runtime starts.

Set the following environment variable before launching the bun process (done automatically by `ElizaAgentService.java` in AOSP builds):

```
ELIZA_LOCAL_LLAMA=1
```

The package also auto-activates on `process.arch === "riscv64"` without requiring the env var (unless `ELIZA_DISABLE_FFI_LLAMA=1` is set).

## Required native libraries

The following shared libraries must be present in `cwd/{abi}/` (where `{abi}` is `arm64-v8a`, `x86_64`, or `riscv64`). They are compiled by `packages/app-core/scripts/aosp/compile-libllama.mjs`:

- `libllama.so` — elizaOS/llama.cpp fork (`v0.1.0-eliza`, based on `apothic/llama.cpp-1bit-turboquant` @ `main-b8198-b2b5273`)
- `libeliza-llama-shim.so` — struct-by-value wrapper (NEEDED-links `libllama.so`)
- `libeliza-llama-speculative-shim.so` — MTP speculative decoding (optional)
- `libelizainference.so` — fused Kokoro TTS + ASR (optional, required for voice)

`ElizaAgentService.java` sets `LD_LIBRARY_PATH` to the ABI dir before spawning bun.

## Required model files

Models are resolved from `$ELIZA_STATE_DIR/local-inference/models/` in priority order:

1. `assignments.json` + `registry.json` (managed by the local-inference service)
2. `manifest.json` (written by `packages/app-core/scripts/aosp/stage-default-models.mjs`)
3. Glob fallback scan for `*.gguf` matching expected name patterns

Default models auto-downloaded from `elizaos/eliza-1` on HuggingFace when not staged:
- **Chat:** `bundles/2b/text/eliza-1-2b-128k.gguf`
- **Embedding:** `bundles/4b/embedding/eliza-1-embedding.gguf`

## Key environment variables

| Variable | Default | Purpose |
|---|---|---|
| `ELIZA_LOCAL_LLAMA` | — | Set `"1"` to activate |
| `ELIZA_DISABLE_FFI_LLAMA` | — | Set `"1"` to force opt-out |
| `ELIZA_LLAMA_THREADS` | `os.cpus().length` | CPU thread count |
| `ELIZA_LLAMA_N_CTX` | `4096` | Chat context window |
| `ELIZA_LOCAL_EMBEDDING_ENABLED` | `"0"` | Set `"1"` to load embedding GGUF |
| `ELIZA_LLAMA_KV_TYPE_K` / `_V` | `q8_0` / `f16` | KV-cache quant type |
| `ELIZA_MTP` | — | Set `"1"` for MTP speculative decoding |
| `ELIZA_AOSP_TTS_PREWARM` | — | Set `"true"` to pre-warm TTS at boot |
| `ELIZA_DISABLE_MODEL_AUTO_DOWNLOAD` | — | Set `"1"` to skip HuggingFace download |
| `ELIZA_AOSP_LLAMA_DEBUG_LOG` | — | Set `"1"` for line-delimited debug log |

See `CLAUDE.md` for the full variable reference.

## Package layout

```
src/
  index.ts                          Barrel (bundle-safety sink included)
  aosp-llama-adapter.ts             bun:ffi loader, AospLlamaAdapter class
  aosp-llama-streaming.ts           Streaming-LLM FFI bindings
  aosp-local-inference-bootstrap.ts Model handler registration, TTS, ASR
  aosp-debug-log.ts                 Append-only debug log utility
__tests__/                          bun test suites
```