# @elizaos/plugin-openai

OpenAI model-provider plugin for elizaOS: text generation, embeddings, image generation/description, audio transcription, text-to-speech, and deep research via the OpenAI Responses API.

## Purpose / role

Registers model handlers on the elizaOS `AgentRuntime` so Eliza agents can call `runtime.useModel(ModelType.*, ...)` backed by OpenAI (or any OpenAI-compatible endpoint — Cerebras, EvoLink, OpenRouter, local servers). Auto-enables when `OPENAI_API_KEY`, `CEREBRAS_API_KEY`, or `EVOLINK_API_KEY` is present in the environment. No actions, providers, services, or evaluators — only model handlers.

## Plugin surface

This plugin registers **model handlers only** (no actions, providers, services, evaluators, routes, or events):

| `ModelType` | Default model | Handler |
|---|---|---|
| `TEXT_SMALL` | `gpt-5.4-mini` | `handleTextSmall` |
| `TEXT_NANO` | falls back to small | `handleTextNano` |
| `TEXT_MEDIUM` | falls back to small | `handleTextMedium` |
| `TEXT_LARGE` | `gpt-5` | `handleTextLarge` |
| `TEXT_MEGA` | falls back to large | `handleTextMega` |
| `RESPONSE_HANDLER` | falls back to small | `handleResponseHandler` |
| `ACTION_PLANNER` | falls back to medium | `handleActionPlanner` |
| `TEXT_EMBEDDING` | `text-embedding-3-small` | `handleTextEmbedding` |
| `TEXT_TOKENIZER_ENCODE` | js-tiktoken | `handleTokenizerEncode` |
| `TEXT_TOKENIZER_DECODE` | js-tiktoken | `handleTokenizerDecode` |
| `IMAGE` | `dall-e-3` | `handleImageGeneration` |
| `IMAGE_DESCRIPTION` | `gpt-5-mini` | `handleImageDescription` |
| `TRANSCRIPTION` | `gpt-5-mini-transcribe` | `handleTranscription` |
| `TEXT_TO_SPEECH` | `gpt-5-mini-tts` / voice `nova` | `handleTextToSpeech` |
| `RESEARCH` | `o3-deep-research` | `handleResearch` (Responses API) |

All text handlers support streaming (`params.stream = true`) and structured output (`params.responseSchema`).

## Layout

```
plugins/plugin-openai/
  index.ts               # Plugin object (openaiPlugin); registers all model handlers
  index.node.ts          # Node entrypoint
  index.browser.ts       # Browser entrypoint
  auto-enable.ts         # shouldEnable(): true when OPENAI_API_KEY, CEREBRAS_API_KEY, or EVOLINK_API_KEY set
  init.ts                # initializeOpenAI(): validates API key on startup; browser skips server-only validation
  build.ts               # Bun.build config (node ESM + browser ESM) + tsc declarations
  models/
    index.ts             # Re-exports all handlers
    text.ts              # handleTextSmall/Nano/Medium/Large/Mega/ResponseHandler/ActionPlanner
    embedding.ts         # handleTextEmbedding (deterministic local fallback in Cerebras mode)
    image.ts             # handleImageGeneration, handleImageDescription
    audio.ts             # handleTranscription, handleTextToSpeech
    tokenizer.ts         # handleTokenizerEncode, handleTokenizerDecode (js-tiktoken)
    research.ts          # handleResearch (OpenAI Responses API, o3/o4-mini deep research)
  providers/
    openai.ts            # createOpenAIClient(): @ai-sdk/openai factory (proxy-aware)
    index.ts
  utils/
    config.ts            # All getSetting/getModel/getBaseURL helpers; Cerebras/EvoLink detection
    events.ts            # emitModelUsageEvent: fires model-usage telemetry on runtime
    audio.ts             # detectAudioMimeType, getFilenameForMimeType
    tokenization.ts      # tiktoken helpers
    index.ts
  types/
    index.ts             # Plugin-local types: TTSVoice, ImageSize, TokenUsage, TextStreamResult,
                         #   OpenAIPluginConfig, API response shapes, etc.
  prompts/               # evaluators.json (empty manifest — plugin ships no evaluators)
  __tests__/             # Vitest unit tests
```

## Commands

```bash
bun run --cwd plugins/plugin-openai build          # Bun.build (node ESM + browser ESM) + tsc d.ts
bun run --cwd plugins/plugin-openai dev            # hot-reload build (bun --hot build.ts)
bun run --cwd plugins/plugin-openai test           # vitest unit suite
bun run --cwd plugins/plugin-openai typecheck      # tsgo --noEmit
bun run --cwd plugins/plugin-openai lint           # biome check --write --unsafe
bun run --cwd plugins/plugin-openai lint:check     # biome check (read-only)
bun run --cwd plugins/plugin-openai format         # biome format --write
bun run --cwd plugins/plugin-openai clean          # rm -rf dist .turbo
```

## Config / env vars

All settings are read via `getSetting(runtime, key)` (runtime config first, then `process.env`).

| Var | Required | Default | Purpose |
|---|---|---|---|
| `OPENAI_API_KEY` | one-of | — | Auth for all OpenAI endpoints |
| `CEREBRAS_API_KEY` | one-of | — | Auth when using Cerebras endpoint |
| `EVOLINK_API_KEY` | one-of | — | Auth when using EvoLink endpoint |
| `OPENAI_BASE_URL` | no | `https://api.openai.com/v1` | Override API endpoint |
| `OPENAI_SMALL_MODEL` / `SMALL_MODEL` | no | `gpt-5.4-mini` | TEXT_SMALL model |
| `OPENAI_NANO_MODEL` / `NANO_MODEL` | no | falls back to small | TEXT_NANO model |
| `OPENAI_MEDIUM_MODEL` / `MEDIUM_MODEL` | no | falls back to small | TEXT_MEDIUM model |
| `OPENAI_LARGE_MODEL` / `LARGE_MODEL` | no | `gpt-5` | TEXT_LARGE model |
| `OPENAI_MEGA_MODEL` / `MEGA_MODEL` | no | falls back to large | TEXT_MEGA model |
| `OPENAI_RESPONSE_HANDLER_MODEL` | no | falls back to small | RESPONSE_HANDLER model |
| `OPENAI_ACTION_PLANNER_MODEL` | no | falls back to medium | ACTION_PLANNER model |
| `OPENAI_EMBEDDING_MODEL` | no | `text-embedding-3-small` | Embedding model |
| `OPENAI_EMBEDDING_URL` | no | `OPENAI_BASE_URL` | Override embeddings endpoint |
| `OPENAI_EMBEDDING_API_KEY` | no | `OPENAI_API_KEY` | Separate embedding auth |
| `OPENAI_EMBEDDING_DIMENSIONS` | no | `1536` | Embedding vector dimensions |
| `OPENAI_IMAGE_DESCRIPTION_MODEL` | no | `gpt-5-mini` | Vision model |
| `OPENAI_IMAGE_DESCRIPTION_BASE_URL` | no | `OPENAI_BASE_URL` | Override vision endpoint |
| `OPENAI_IMAGE_DESCRIPTION_API_KEY` | no | `OPENAI_API_KEY` | Separate vision auth |
| `OPENAI_IMAGE_DESCRIPTION_MAX_TOKENS` | no | `8192` | Max tokens for vision output |
| `OPENAI_IMAGE_MODEL` | no | `dall-e-3` | Image generation model |
| `OPENAI_TTS_MODEL` | no | `gpt-5-mini-tts` | Text-to-speech model |
| `OPENAI_TTS_VOICE` | no | `nova` | TTS voice (alloy/echo/fable/onyx/nova/shimmer) |
| `OPENAI_TTS_INSTRUCTIONS` | no | — | Style instructions for TTS |
| `OPENAI_TRANSCRIPTION_MODEL` | no | `gpt-5-mini-transcribe` | Audio transcription model |
| `OPENAI_RESEARCH_MODEL` | no | `o3-deep-research` | Deep research model |
| `OPENAI_RESEARCH_TIMEOUT` | no | `3600000` (1 hr) | Timeout for research requests (ms) |
| `OPENAI_EXPERIMENTAL_TELEMETRY` | no | `false` | Enable AI SDK telemetry |
| `OPENAI_REASONING_EFFORT` | no | — | `minimal`/`low`/`medium`/`high` for o-series models |
| `OPENAI_BROWSER_BASE_URL` | no | — | Browser-only proxy URL (keeps key server-side) |
| `OPENAI_BROWSER_EMBEDDING_URL` | no | — | Browser-only embeddings proxy URL |
| `OPENAI_ALLOW_BROWSER_API_KEY` | no | `false` | Send auth header in browser (opt-in) |
| `ELIZA_PROVIDER` | no | — | Set to `cerebras` or `evolink` to force that provider mode |
| `CEREBRAS_BASE_URL` | no | `https://api.cerebras.ai/v1` | Cerebras API base |
| `CEREBRAS_MODEL` | no | — | Override model name in Cerebras mode |
| `EVOLINK_BASE_URL` | no | `https://direct.evolink.ai/v1` | EvoLink API base |
| `EVOLINK_MODEL` | no | `gpt-5.2` | Override model name in EvoLink mode |

*Either `OPENAI_API_KEY`, `CEREBRAS_API_KEY`, or `EVOLINK_API_KEY` is required; the plugin will not auto-enable without one.

## How to extend

**Add a new model handler:**

1. Create `models/<name>.ts` exporting an async handler function matching the relevant `@elizaos/core` params/return type.
2. Re-export from `models/index.ts`.
3. Add a new `models: { [ModelType.NEW_TYPE]: async (runtime, params) => handler(runtime, params) }` entry in `index.ts`.
4. Add config helpers for any new env vars to `utils/config.ts`.

**Add a new model size tier:**

Model tiers (nano/medium/mega/response-handler/action-planner) all call the shared `generateTextByModelType()` in `models/text.ts`. Add a getter to `utils/config.ts` following the `OPENAI_<TIER>_MODEL` / `<TIER>_MODEL` fallback pattern, then wire it in `index.ts`.

## Conventions / gotchas

- **Dual build (node + browser).** Exports differ: `dist/node/index.node.js` and `dist/browser/index.browser.js`. Browser build avoids sending `Authorization` headers by default; set `OPENAI_BROWSER_BASE_URL` to a server-side proxy.
- **Cerebras mode.** Detected automatically from `ELIZA_PROVIDER=cerebras`, `OPENAI_BASE_URL` matching `*.cerebras.ai`, or presence of `CEREBRAS_API_KEY` without `OPENAI_API_KEY`. In Cerebras mode: structured output via `response_format: json_object` (not `json_schema`); `reasoning_effort` defaults to `"low"` for reasoning-capable models; `promptCacheRetention` is stripped (Cerebras rejects it); embeddings fall back to a deterministic local hash when no explicit embedding URL is set.
- **Strict-schema stripping (unconditional, ALL providers).** `sanitizeJsonSchema` in `models/text.ts` is the single wire choke point for every `response_format` schema (`buildStructuredOutput`) and every tool schema (`normalizeNativeTools`). It strips the constraint keywords strict-grammar providers (Cerebras via Eliza Cloud, OpenAI strict) 400 on — `maxItems`, `minItems`, `maxLength`, `minLength`, `pattern`, `format`, `minProperties`, `maxProperties` — folding each into the node's `description` so the model keeps the intent, and recurses through `properties`/`items`/`anyOf`/`oneOf`/`allOf`/`$defs`/`patternProperties`/`contains`/`if`-`then`-`else`. Numeric bounds (`minimum`/`maximum`/`multipleOf`/`uniqueItems`) pass through untouched. This is **not** gated on Cerebras mode — `isCerebrasMode` is proxy-blind (an agent on `api.elizacloud.ai` with `OPENAI_API_KEY` looks like plain OpenAI, which is exactly where the 400s fired). Real bounds are still enforced app-side: `parseAndValidate` re-checks the caller's ORIGINAL schema. So do NOT add per-schema constraint-stripping — the choke point already does it (#11123 / #11153).
- **Free-form record/map tool args use the #13111 strict-safe transform.** `sanitizeJsonSchema` still forces `additionalProperties: false` on every object for strict-grammar providers, but `normalizeNativeTools` rewrites declared free-form record/map tool args (`additionalProperties: true` or a value schema, e.g. contact `customFields`) into a model-facing `__eliza_record_entries` key/value array. Returned tool calls are reverse-mapped back to the original object shape before runtime validation, so tool authors still receive the schema they declared without reopening #11123/#11156 strict-schema 400s. Scoped to **tool parameters only** — `response_format` has no returned tool args to reverse-map and still uses plain sanitization.
- **EvoLink mode.** Detected automatically from `ELIZA_PROVIDER=evolink`, `OPENAI_BASE_URL` matching `*.evolink.ai`, or presence of `EVOLINK_API_KEY` without a conflicting key. Uses `EVOLINK_BASE_URL` (default `https://direct.evolink.ai/v1`) and defaults to `gpt-5.2` as the model.
- **Per-call model override.** Text handlers honor `params.model` before slot-level model settings. Workflow generation uses this for isolated calls such as Cerebras `gpt-oss-120b` without changing every OpenAI text call.
- **Reasoning models.** Pass `OPENAI_REASONING_EFFORT=low|medium|high` to control o-series / gpt-oss reasoning budgets. Valid values: `minimal`, `low`, `medium`, `high`.
- **Prompt caching.** Pass `providerOptions: { openai: { promptCacheKey: "...", promptCacheRetention: "24h" } }` on any `GenerateTextParams` call to enable OpenAI prompt caching.
- **Deep research.** `ModelType.RESEARCH` uses the OpenAI Responses API (`POST /responses`), not Chat Completions. It defaults to `o3-deep-research` and can take minutes to hours; use `params.background = true` for long jobs.
- **Tokenizer.** Uses `js-tiktoken` (WASM, browser-safe). `TEXT_TOKENIZER_ENCODE/DECODE` do not hit the network.
- **All API calls go through `recordLlmCall()`** from `@elizaos/core` for trajectory logging. Audio/embedding handlers carry a `// @trajectory-allow` comment where appropriate.
- **No barrel re-export of internal utils.** Import from `"../utils/config"`, `"../utils/events"`, etc. directly within the plugin.
- See root `AGENTS.md` for repo-wide architecture rules, logger conventions, ESM requirements, and naming standards.

<!-- BEGIN: evidence-and-e2e-mandate (managed; canonical standard = repo-root AGENTS.md) -->
## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests

> The binding, repo-wide standard is **[AGENTS.md](../../AGENTS.md)**. Read it.
> Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it
> works **without reading the code**, from the artifacts you attach. This applies to **every**
> feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof.

- **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model
  from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the
  providers/context, the raw model output, every tool/action call, and the result. Then **open
  the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence
  (`packages/scenario-runner/bin/eliza-scenarios run <scenario> --report <out>`).
- **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that
  drive the *real* path end to end. Not the happy "front door" only: cover error paths,
  edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that
  asserts against a mock/stub/fixture standing in for the thing under test **does not count**.
  If the real model/device/chain/connector/account is hard to reach, **make it reachable — that
  is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing
  them is part of your change.
- **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of
  the entire feature or view, start to finish (`bun run test:e2e:record`).
- **Manually review every artifact the change touches** — never just the green check: client
  logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out,
  before/after full-page screenshots, **and the domain artifacts listed below for this package.**
- **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every
  blocker by the **hard path**: build the real architecture, stand up the real
  model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a
  "follow-up." When unsure, research thoroughly, weigh the options, and ship the best,
  highest-effort, production-ready version. Keep going until every possibility is exhausted.

Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `<details>`); attach each evidence type **or**
explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed
behavior, **re-capture** evidence; stale proof is worse than none.

**Capture & manually review for this package — model provider:**
- A trajectory from a **live** call to this provider (not the proxy, not a mock): full request, raw response, token usage, finish reason, and streamed chunks.
- Proof of tool/function-calling and structured-output parsing against the real model.
- The error paths exercised: bad key, model-not-found, oversized context, timeout, rate-limit, mid-stream disconnect — plus latency and cost from the real call.
- If no key is available in CI, attach the documented live-run transcript as evidence — never a mocked client passed off as a pass.
<!-- END: evidence-and-e2e-mandate -->