# @elizaos/plugin-music Music library, discovery, playlists, analytics, playback engine, queue, routing API, and streaming routes for Eliza agents. ## Purpose / Role Adds comprehensive music capability to an Eliza agent: playback of YouTube and direct-URL audio, queue management, playlist persistence, smart music queries, YouTube search, Suno-backed AI music generation, multi-zone audio routing, and a streaming HTTP API. Auto-enabled when any of `LASTFM_API_KEY`, `GENIUS_API_KEY`, `THEAUDIODB_API_KEY`, `SPOTIFY_CLIENT_ID`, or `SPOTIFY_CLIENT_SECRET` is set; can also be loaded explicitly via character config. ## Plugin Surface ### Actions - **`MUSIC`** (`src/actions/music.ts`) — Umbrella action. Dispatches all music operations via a verb-shaped `action` parameter: `play`, `pause`, `resume`, `skip`, `stop`, `queue_view`, `queue_add`, `queue_clear`, `playlist_play`, `playlist_save`, `playlist_delete`, `playlist_add`, `search`, `play_query`, `download`, `play_audio`, `set_routing`, `set_zone`, `generate`, `extend`, `custom_generate`. Legacy aliases are accepted (e.g. `playlist`, `search_youtube`, `routing`, `zones`). ### Services - **`MusicService`** (`src/service.ts`, serviceType: `"music"`) — Core playback engine. Manages per-guild queues, audio broadcasting, Discord voice wiring, audio routing, and the `AudioCacheService`. - **`MusicLibraryService`** (`src/services/musicLibraryService.ts`, serviceType: `"musicLibrary"`) — Library, playlists, preferences, analytics, repetition control, song memory, Spotify client. Aggregated service over all component modules. ### Providers - **`MUSIC_INFO`** (`src/providers/musicInfoProvider.ts`) — Injects track/artist/album metadata into agent state. Contexts: `media`, `knowledge`. - **`WIKIPEDIA_MUSIC`** (`src/providers/wikipediaProvider.ts`) — Extracts music context from Wikipedia via LLM parsing. Contexts: `media`, `knowledge`. - **`MUSIC_LIBRARY`** (`src/providers/musicLibraryProvider.ts`) — Library stats, recent/most-played songs. Contexts: `media`, `knowledge`. - **`musicPlaylists`** (`src/providers/musicPlaylistsProvider.ts`) — User playlists as JSON context. Contexts: `media`, `knowledge`. - **`musicQueue`** (`src/providers/musicQueueProvider.ts`) — Current queue and now-playing track. Contexts: `media`, `knowledge`. ### Routes (`src/routes.ts`) All paths are under the plugin's mount prefix (e.g. `/api//music-player/`): | Method | Path | Auth | Purpose | |--------|------|------|---------| | GET | `/stream` | public | Live audio stream (OGG/Shoutcast/Icecast) | | GET | `/stream/:guildId` | public | Same, guildId in path param | | GET | `/now-playing` | public | Now-playing metadata JSON | | GET | `/now-playing/:guildId` | public | Same, guildId in path param | | GET | `/queue` | public | Queue JSON | | GET | `/queue/:guildId` | public | Same, guildId in path param | | GET | `/status` | public | Playback status JSON | | POST | `/control/pause` | authenticated | Pause playback | | POST | `/control/resume` | authenticated | Resume playback | | POST | `/control/stop` | authenticated | Stop playback | | POST | `/control/skip` | authenticated | Skip current track | ### Search Categories Registered via `registerMusicLibrarySearchCategories` on `init`: - `youtube` — YouTube video/music search - `wikipedia_music` — Wikipedia music metadata lookup ## Layout ``` src/ index.ts Plugin entry; exports musicPlugin (Plugin object) service.ts MusicService (playback engine, queues, routing) routes.ts Streaming + control HTTP routes queue.ts MusicQueue and QueuedTrack types contracts.ts IAudioBroadcast, BroadcastState, BroadcastTrackMetadata search-category.ts Search category registrations (YouTube, Wikipedia) route-fallback.ts tryHandleMusicPlayerStatusFallback helper discordVoice.ts Discord voice type bridge actions/ music.ts MUSIC umbrella action (dispatcher) music-player-action-docs.ts Action parameter docs for the music-player surface musicLibrary.ts Library sub-handler (playlist, search, play_query, download) playAudio.ts play_audio sub-handler playbackOp.ts Transport controls (pause/resume/skip/stop/queue) manageRouting.ts set_routing sub-handler manageZones.ts set_zone sub-handler downloadMusic.ts Download logic searchYouTube.ts YouTube search logic playMusicQuery.ts Smart play_query logic playlistOp.ts Playlist save/load logic confirmation.ts Confirmation merge helpers providers/ musicInfoProvider.ts MUSIC_INFO provider wikipediaProvider.ts WIKIPEDIA_MUSIC provider musicLibraryProvider.ts MUSIC_LIBRARY provider musicPlaylistsProvider.ts musicPlaylists provider musicQueueProvider.ts musicQueue provider services/ musicLibraryService.ts MusicLibraryService (aggregates all components) musicInfoService.ts MusicInfoService + MusicInfoHelper youtubeSearch.ts YouTubeSearchService + YouTubeSearchHelper musicEntityDetectionService.ts Entity detection from conversation musicStorage.ts MusicStorageService (permanent archive storage) audioCache.ts AudioCacheService (yt-dlp download + Opus transcoding) smartMusicFetch.ts SmartMusicFetchService spotifyClient.ts SpotifyClient (recommendations) wikipediaClient.ts WikipediaService + WikipediaClient wikipediaExtractionService.ts LLM-based Wikipedia extraction geniusClient.ts Genius API (lyrics URLs) lastFmClient.ts Last.fm API (artist/track metadata) musicBrainzClient.ts MusicBrainz API (metadata, free tier) theAudioDbClient.ts TheAudioDB API (artwork) serviceStatus.ts ServiceHealth tracking types components/ musicLibrary.ts Track/album/artist database functions playlists.ts Playlist CRUD functions preferences.ts User preference tracking analytics.ts DJ analytics and play-tracking repetitionControl.ts Anti-repetition logic songMemory.ts Song memory and request history djGuildSettings.ts Per-guild DJ config djIntroOptions.ts DJ intro prompt options djTips.ts DJ tip tracking componentData.ts Shared component data helpers storageContext.ts Storage context helpers core/ broadcast.ts Broadcast (stream multiplexer) streamCore.ts Low-level stream helpers streamMultiplexer.ts Multi-subscriber stream fan-out index.ts Re-exports Broadcast router/ audioRouter.ts AudioRouter (routing mode management) zoneManager.ts ZoneManager (multi-zone audio) mixSessionManager.ts MixSessionManager (mix sessions) index.ts Re-exports router types types/ index.ts TrackInfo, ArtistInfo, AlbumInfo, MusicInfoResult audioFeatures.ts AudioFeatures, RecommendationRequest, TrackRecommendation utils/ ffmpegEnv.ts FFmpeg binary resolution ytdlpCheck.ts yt-dlp binary discovery ytdlpCli.ts yt-dlp CLI helpers ytdlpYoutube.ts YouTube-specific yt-dlp options ytdlpFallback.ts Fallback fetch logic musicDebug.ts Debug logging helpers playbackTransportIntent.ts Intent detection for transport controls resolveMusicGuildId.ts Guild ID resolution helpers json.ts JSON parse helpers opusBroadcastNormalize.ts Opus broadcast normalization progressiveMessage.ts Progressive message helpers retry.ts Retry utility smartFetchService.ts Smart fetch service streamFallback.ts Stream fallback logic ``` ## Commands ```bash bun run --cwd plugins/plugin-music build # tsup build to dist/ bun run --cwd plugins/plugin-music dev # tsup watch mode bun run --cwd plugins/plugin-music test # vitest run bun run --cwd plugins/plugin-music typecheck # tsgo --noEmit bun run --cwd plugins/plugin-music lint # biome check --write --unsafe bun run --cwd plugins/plugin-music lint:check # biome check (no write) bun run --cwd plugins/plugin-music format # biome format --write bun run --cwd plugins/plugin-music format:check # biome format (no write) bun run --cwd plugins/plugin-music clean # rm -rf dist .turbo .turbo-tsconfig.json tsconfig.tsbuildinfo bun run --cwd plugins/plugin-music test:e2e # live smoke (requires running agent) ``` ## Config / Env Vars ### Runtime settings (via `runtime.getSetting(...)`) | Setting | Required | Default | Purpose | |---------|----------|---------|---------| | `LASTFM_API_KEY` | No | — | Last.fm metadata; triggers auto-enable | | `GENIUS_API_KEY` | No | — | Genius lyrics URLs; triggers auto-enable | | `THEAUDIODB_API_KEY` | No | — | TheAudioDB artwork; triggers auto-enable | | `SPOTIFY_CLIENT_ID` | No | — | Spotify recommendations; triggers auto-enable | | `SPOTIFY_CLIENT_SECRET` | No | — | Spotify auth; triggers auto-enable | | `SUNO_API_KEY` | No | — | AI music generation (generate/extend/custom_generate subactions) | | `MUSICBRAINZ_USER_AGENT` | No | `ElizaOS-MusicInfo/1.0.0 (https://github.com/elizaos/eliza)` | Custom User-Agent for MusicBrainz API | | `MUSIC_QUALITY_PREFERENCE` | No | `mp3_320` | Download quality preference | | `AUDIO_CACHE_DIR` | No | `/cache/audio` | Directory for pre-transcoded audio cache | ### Process env vars | Variable | Purpose | |----------|---------| | `YOUTUBE_COOKIES` | Path to Netscape cookies file for age-restricted YouTube content | | `YTDLP_COOKIES` | Alternative cookies path for yt-dlp | | `AUDIO_CACHE_FORMAT` | Output audio format override | | `ELIZA_MUSIC_DEBUG` | Enable verbose music debug logging | | `ELIZA_MUSIC_BROADCAST_NORMALIZE` | Enable broadcast audio normalization | | `FFMPEG_PATH` / `FFMPEG_LOCATION` | Override FFmpeg binary path | | `FFPROBE_PATH` | Override ffprobe binary path | | `HTTP_PROXY` / `HTTPS_PROXY` | Proxy for outbound requests | | `YOUTUBE_PROXY` / `YTDLP_PROXY` | Proxy specifically for yt-dlp | | `YTDLP_JS_RUNTIMES` | JS runtime list for yt-dlp (e.g. `nodejs`) | | `YTDLP_YOUTUBE_EXTRACTOR_ARGS` | Extra args passed to the YouTube extractor | | `SERVER_URL` | Base URL for streaming route self-references | ## How to Extend ### Add an action sub-handler 1. Create `src/actions/.ts` exporting an `Action` object with `name`, `validate`, `handler`, and `examples`. 2. Add a new verb to `MUSIC_SUBACTIONS` in `src/actions/music.ts`. 3. Add an alias entry to `SUBACTION_ALIASES` if needed. 4. Add a `DispatchKind` branch in `dispatchKindFor()` and handle it in the `handler` switch. ### Add a provider 1. Create `src/providers/.ts` exporting a `Provider` object (name, description, contexts, get). 2. Import and add it to the `providers` array in `src/index.ts`. ### Add a service 1. Extend `MusicLibraryService` with new methods if the feature belongs in the library layer, or create a new `Service` subclass under `src/services/`. 2. Add the new service class to `services: [...]` in the `musicPlugin` object in `src/index.ts` (if standalone) and export from `src/index.ts`. ## Conventions / Gotchas - **yt-dlp required at runtime.** Audio download and caching depend on `yt-dlp`. Discovery order (`src/utils/ytdlpCheck.ts`): `YT_DLP_PATH` env → workspace `scripts/bin/yt-dlp` → common system paths (`/usr/local/bin`, `/usr/bin`, `/opt/homebrew/bin`). Install via `brew install yt-dlp`, `pipx install yt-dlp`, or download binary. - **ffmpeg/ffprobe required.** Bundled via `ffmpeg-static`/`ffprobe-static` deps; paths overridden by `FFMPEG_PATH`/`FFPROBE_PATH`. - **Discord wiring is deferred.** The plugin init waits for the `discord` service load promise before wiring the voice manager. Music works web-only if Discord is absent. - **`@elizaos/plugin-suno` is a hard dep.** Generation subactions (`generate`, `extend`, `custom_generate`) delegate to `sunoGenerateMusicHandler` from that package; they are skipped/unreachable if `SUNO_API_KEY` is absent. - **Confirmation required for destructive ops.** `skip`, `stop`, `queue_add`, `queue_clear`, `playlist_save`, `playlist_delete`, `playlist_add`, and `download` require confirmation through `requireMusicConfirmation` in `src/actions/confirmation.ts`. - **MusicBrainz is the zero-config metadata source.** All other metadata APIs (Last.fm, Genius, TheAudioDB) are optional enhancements. - **MusicStorageService is a standalone exported utility.** It is not registered as a plugin service or auto-wired into `MusicLibraryService`; it is exported from `src/index.ts` for callers that want a permanent high-quality archive. Storage dir (`/storage/music`) and quality mode are constructor arguments, not env vars. - **`@elizaos/plugin-sql` is expected for persistence.** Library, playlists, preferences, and analytics components write to the agent's database via runtime memory/cache APIs. See the repo-wide rules in the root `AGENTS.md`. ## ⛔ NON-NEGOTIABLE — evidence, trajectories & real end-to-end tests > The binding, repo-wide standard is **[AGENTS.md](../../AGENTS.md)**. Read it. > Nothing in this package is *done* until it is *proven* done — a reviewer must confirm it > works **without reading the code**, from the artifacts you attach. This applies to **every** > feature, fix, refactor, and chore here. "Tests pass" is not proof; "CI is green" is not proof. - **Record AND read model trajectories.** Capture the *actual* inputs and outputs of the model from a **live** LLM — not the deterministic proxy, not a mock: the prompt, the providers/context, the raw model output, every tool/action call, and the result. Then **open the trajectory and review it by hand.** A captured-but-unread trajectory is not evidence (`packages/scenario-runner/bin/eliza-scenarios run --report `). - **Real, full-featured E2E — no larp.** Every feature ships detailed end-to-end tests that drive the *real* path end to end. Not the happy "front door" only: cover error paths, edge/empty/invalid input, concurrency, roles/permissions, and adversarial input. A test that asserts against a mock/stub/fixture standing in for the thing under test **does not count**. If the real model/device/chain/connector/account is hard to reach, **make it reachable — that is the work**, not an excuse to mock. If the existing tests here are shallow or mocked, fixing them is part of your change. - **Screenshots + logs at every phase**, plus a **complete walkthrough video/run-through** of the entire feature or view, start to finish (`bun run test:e2e:record`). - **Manually review every artifact the change touches** — never just the green check: client logs (console + network), server logs (`[ClassName] …`), the model trajectories in and out, before/after full-page screenshots, **and the domain artifacts listed below for this package.** - **No residuals. No shortcuts.** The goal is not "done" — it is *everything* done. Clear every blocker by the **hard path**: build the real architecture, stand up the real model/device/service, actually test it. Never leave a TODO, a stub, a stepping-stone, or a "follow-up." When unsure, research thoroughly, weigh the options, and ship the best, highest-effort, production-ready version. Keep going until every possibility is exhausted. Artifacts → attached inline in the PR (MP4 video, JPG screenshots, logs in `
`); attach each evidence type **or** explicitly mark it N/A with a reason — never leave it blank. If `develop` moved and changed behavior, **re-capture** evidence; stale proof is worse than none. **Capture & manually review for this package — voice / audio:** - Captured **audio** of the real round-trip (STT in, TTS out) plus the transcript, with a narrated walkthrough of what is happening. - Latency, barge-in/interruption, and wake-word behavior measured on real audio — across platforms, not Linux-x64-synthetic only (see #9958). - The model trajectory for any LLM turn inside the loop. - Failure paths: no mic, silence, noise, overlapping speech, network drop mid-stream.