/plugins/plugin-elevenlabs/AGENTS.md

@elizaos/plugin-elevenlabs

Provides ElevenLabs-powered text-to-speech (TTS) and speech-to-text (STT) model handlers for Eliza agents.

Purpose / role

This plugin registers ModelType.TEXT_TO_SPEECH and ModelType.TRANSCRIPTION handlers with the elizaOS runtime so any agent that loads it can call runtime.useModel(ModelType.TEXT_TO_SPEECH, ...) or runtime.useModel(ModelType.TRANSCRIPTION, ...) backed by the ElevenLabs API. It is opt-in: add @elizaos/plugin-elevenlabs to the agent's plugins list in character config. It has no actions, providers, evaluators, services, or routes — only model handlers.

Plugin surface

Kind	Name	What it does
Model handler	ModelType.TEXT_TO_SPEECH	Streams audio from ElevenLabs TTS API, returns Uint8Array
Model handler	ModelType.TRANSCRIPTION	Sends audio to ElevenLabs Scribe STT API, returns transcript string
Tests	inline plugin test suite	Validates API key, voice settings, STT config, and (if key present) live connectivity

No actions, providers, evaluators, services, routes, or events are registered.

Layout

plugins/plugin-elevenlabs/
  src/
    index.ts           Plugin definition, all model handler logic, settings helpers
    index.node.ts      Node entry — re-exports index.ts (used as module/cjs)
    index.browser.ts   Browser entry — re-exports index.ts (used as browser build)
  __tests__/
    streaming.test.ts  Functional tests for TTS streaming path (chunk draining, SDK params, error propagation)
  build.ts             Bun build script
  package.json

Everything lives in src/index.ts. There are no separate action or service files.

Commands

Scripts that exist in this package's package.json:

bun run --cwd plugins/plugin-elevenlabs build          # compile dist/
bun run --cwd plugins/plugin-elevenlabs dev            # watch build (--hot)
bun run --cwd plugins/plugin-elevenlabs test           # run __tests__
bun run --cwd plugins/plugin-elevenlabs clean          # rm -rf dist .turbo .turbo-tsconfig.json tsconfig.tsbuildinfo
bun run --cwd plugins/plugin-elevenlabs lint           # biome check --write --unsafe
bun run --cwd plugins/plugin-elevenlabs lint:check     # biome check (read-only)
bun run --cwd plugins/plugin-elevenlabs format         # biome format --write
bun run --cwd plugins/plugin-elevenlabs format:check   # biome format (read-only)
bun run --cwd plugins/plugin-elevenlabs typecheck      # tsgo --project tsconfig.json --noEmit

Config / env vars

Settings are resolved from runtime.getSetting(key) first, then process.env[key]. All are read at call time inside the model handler — no service startup required.

Var	Required	Default	Notes
ELEVENLABS_API_KEY	Yes	—	ElevenLabs API key
ELEVENLABS_VOICE_ID	No	EXAVITQu4vr4xnSDxMaL	Voice ID for TTS
ELEVENLABS_MODEL_ID	No	eleven_monolingual_v1	TTS model ID
ELEVENLABS_VOICE_STABILITY	No	0.5	0–1
ELEVENLABS_VOICE_SIMILARITY_BOOST	No	0.75	0–1
ELEVENLABS_VOICE_STYLE	No	0	0–1
ELEVENLABS_VOICE_USE_SPEAKER_BOOST	No	true	boolean string
ELEVENLABS_OPTIMIZE_STREAMING_LATENCY	No	0	0–4
ELEVENLABS_OUTPUT_FORMAT	No	mp3_44100_128	ElevenLabs format enum
ELEVENLABS_BROWSER_URL	No	—	Proxy base URL for browser builds; skips API key requirement
ELEVENLABS_STT_MODEL_ID	No	scribe_v1	STT model ID
ELEVENLABS_STT_LANGUAGE_CODE	No	—	Leave unset for auto-detect
ELEVENLABS_STT_TIMESTAMPS_GRANULARITY	No	word	none / word / character
ELEVENLABS_STT_DIARIZE	No	false	boolean string
ELEVENLABS_STT_NUM_SPEAKERS	No	—	1–32; only used when diarize=true
ELEVENLABS_STT_TAG_AUDIO_EVENTS	No	false	boolean string

How to extend

This plugin has no extensible action registry — it only registers model handlers. To add a new model type handler:

1. Open src/index.ts.
2. Add a new key under the models object on elevenLabsPlugin using a ModelType.* constant from @elizaos/core.
3. Implement the handler as async (runtime: IAgentRuntime, input: <InputType>) => <OutputType>.
4. Document the new env vars in package.json under agentConfig.pluginParameters.

To add an action, provider, or service (not currently present), add an actions, providers, or services array to the plugin object following the pattern in the root AGENTS.md and elizaOS core docs.

Conventions / gotchas

• Browser builds: ELEVENLABS_BROWSER_URL redirects requests to a server-side proxy that injects the API key. In browser mode, configure either ELEVENLABS_BROWSER_URL or a real ELEVENLABS_API_KEY; otherwise model calls fail before contacting the SDK.
• Output format: The default output format is mp3_44100_128 (browser-safe). Avoid pcm_* formats in browser contexts; PCM is fine for Node/server deployments. Pass an explicit format field in the useModel input object to override per-call.
• TTS input shape: runtime.useModel(ModelType.TEXT_TO_SPEECH, ...) accepts either a plain string or { text, model?, voiceId?, format?, instructions? }.
• STT input shape: Accepts a URL string, a Buffer, or { audioUrl: string; prompt?: string }.
• Format validation: parseTtsOutputFormat and parseSttModelId throw on unrecognized enum values — use values from the @elevenlabs/elevenlabs-js/api enums.
• No WAV support: WAV header utilities were removed; use mp3 or pcm formats directly.
• SDK dependency: Uses @elevenlabs/elevenlabs-js ^2.16.0 (official ElevenLabs SDK). All API calls go through ElevenLabsClient.

⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

The binding, repo-wide standard is AGENTS.md. Read it. Nothing in this package is done until it is proven done — a reviewer must confirm it works without reading the code, from the artifacts you attach. This applies to every feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

• Record AND read model trajectories. Capture the actual inputs and outputs of the model from a live LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then open the trajectory and review it by hand. A captured-but-unread trajectory is not evidence (packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>).
• Real, full-featured E2E — no larp. Every feature ships detailed end-to-end tests that drive the real path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test does not count. If the real model/device/chain/connector/account is hard to reach, make it reachable — that is the work, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change.
• Screenshots + logs at every phase, plus a complete walkthrough video/run-through of the entire feature or view, start to finish (bun run test:e2e:record).
• Manually review every artifact the change touches — never just the green check: client logs (console + network), server logs ([ClassName] …), the model trajectories in and out, before/after full-page screenshots, and the domain artifacts listed below for this package.
• No residuals. No shortcuts. The goal is not "done" — it is everything done. Clear every blocker by the hard path: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in <details>); attach each evidence type or explicitly mark it N/A with a reason — never leave it blank. If develop moved and changed behavior, re-capture evidence; stale proof is worse than none.

Capture & manually review for this package — voice / audio:

• Captured audio of the real round-trip (STT in, TTS out) plus the transcript, with a narrated walkthrough of what is happening.
• Latency, barge-in/interruption, and wake-word behavior measured on real audio — across platforms, not Linux-x64-synthetic only (see #9958).
• The model trajectory for any LLM turn inside the loop.
• Failure paths: no mic, silence, noise, overlapping speech, network drop mid-stream.