omnigent-ai--omnigent
633 行
34 KiB
Markdown
633 行
34 KiB
Markdown
# Omnigent Desktop (Electron)
|
|
|
|
A thin [Electron](https://www.electronjs.org) desktop shell around the
|
|
existing Omnigent web UI. It shows the **same** UI you get in a browser, but
|
|
adds native niceties:
|
|
|
|
- **OS-native desktop notifications** (via the main-process `Notification`
|
|
API) when an agent finishes a turn (`running` → `idle`/`failed`), raises a
|
|
new elicitation (asks for input), or a runner disconnects (`online` →
|
|
`offline`). A notification fires for any such event **except** the one
|
|
conversation you're actively viewing (window focused _and_ that chat
|
|
open). Sessions already settled at launch don't fire; only fresh
|
|
transitions this client observes do. On a turn-end the notification body
|
|
shows the **first few lines of the agent's final message** when they can be
|
|
fetched (one best-effort `GET /items` call), falling back to a generic
|
|
"Agent finished and is ready for your input." On macOS each notification can
|
|
also **play a sound** — a system sound you pick in the **Notifications** menu
|
|
(see below). It's **off by default (opt-in)**: a fresh install stays silent
|
|
until you turn it on, so the sound never surprises you.
|
|
- **A foreground attention cue.** macOS (and Windows) suppress the notification
|
|
_banner_ for the **frontmost** app — and macOS suppresses its **sound** too —
|
|
so the notification still lands in Notification Center, but no toast pops (and
|
|
on macOS no sound plays), which reads as "notifications only work when the app
|
|
is in the background." Because the web layer already only notifies for
|
|
sessions you are _not_ actively viewing, the shell adds OS-level cues the
|
|
frontmost app _can_ produce: it **bounces the macOS dock icon** (or flashes
|
|
the taskbar frame on Windows/Linux), and on macOS it **plays the chosen sound
|
|
itself** (via `afplay`) instead of the suppressed notification sound. Because
|
|
the shell plays it, the alert is audible **whether Omnigent is backgrounded or
|
|
in front** — and the toast's own sound is muted so the cue never doubles.
|
|
- **Multiple windows** (**Server → New Window**, `Cmd/Ctrl+N`). Each window is
|
|
an independent view, opening on the current window's URL so you can then
|
|
navigate it to a different conversation and watch two side by side. A
|
|
window can also be opened against a **different server** (see "Multiple
|
|
servers" below). Notifications and the dock badge are app-wide (one badge
|
|
for all windows); a notification click focuses the window that fired it.
|
|
- **A dock / taskbar badge showing the number of unread sessions** at all
|
|
times (macOS dock badge, Linux Unity launcher count, via
|
|
`app.setBadgeCount`). A session becomes "unread" when it finishes a turn
|
|
or asks for input while you're not actively viewing it, and is cleared the
|
|
moment you view it. Runner disconnects notify but do **not** count toward
|
|
the badge.
|
|
- **The standard native menu** (App / Edit / View / Window / Help) built from
|
|
Electron's menu roles, so the usual text-editing shortcuts — Cmd/Ctrl-A,
|
|
C, V, X, Z — work inside the webview's text fields. Our custom actions —
|
|
**New Window**, **New Window on Different Server…**, and
|
|
**Change Server…** — live in a dedicated **Server** submenu. On macOS a
|
|
**Notifications** submenu turns the notification sound on/off (**Play
|
|
Notification Sound**, **off by default** — the user opts in) and picks which
|
|
macOS system sound to play (**Sound ▸** — Glass, Ping, Hero, …); choosing one
|
|
previews it, and the choice persists in `settings.json` and applies to the
|
|
next notification.
|
|
- **Browser-style file drag-and-drop** works out of the box: Electron does
|
|
not intercept file drops the way Tauri does by default, so dropping an
|
|
image onto a text field reaches the web app's HTML5 drop handler with no
|
|
extra configuration.
|
|
- **Microphone permission for voice dictation.** The composer's dictation
|
|
button uses the Web Speech API plus a `getUserMedia` audio stream (the mic
|
|
level meter). Both go through Chromium's permission layer, which in Electron
|
|
asks the _embedder_ (us) rather than showing Chrome's prompt — with no
|
|
handler wired, Chromium denies by default, so `recognition.start()` fails
|
|
instantly with `not-allowed` and the button appears dead. The main process
|
|
now wires `setPermissionRequestHandler` / `setPermissionCheckHandler` to
|
|
grant the audio permissions, and on macOS calls
|
|
`systemPreferences.askForMediaAccess("microphone")` lazily — on the first
|
|
actual mic request (the user clicking dictate), not at app startup — so the
|
|
OS-level mic gate is open too (packaged builds ship
|
|
`NSMicrophoneUsageDescription`).
|
|
|
|
> **Caveat — Web Speech may still not transcribe in Electron.** Granting the
|
|
> mic clears the _permission_ gate, but `SpeechRecognition` also depends on
|
|
> Google's cloud speech backend keyed to official Google Chrome builds, which
|
|
> Electron's bundled Chromium does **not** ship. So recognition can still
|
|
> fail (typically a `network` error) even with the mic allowed. The web app
|
|
> degrades gracefully (the button shows "Dictation unavailable" rather than
|
|
> crashing). Fully reliable in-app dictation would require a MediaRecorder
|
|
> capture + a server-side transcription endpoint (e.g. Whisper) wired to the
|
|
> composer's existing `onAudioRecorded` fallback — not yet implemented.
|
|
|
|
## How it works (zero UI duplication)
|
|
|
|
The desktop app does **not** ship a copy of the web UI. It bundles only a tiny
|
|
"connect to server" page (`setup/index.html`). On launch:
|
|
|
|
1. If no server URL is saved yet, it shows the setup page (one input +
|
|
Connect). You enter your Omnigent server URL (default
|
|
`http://localhost:8000`).
|
|
2. It persists that URL to the per-user app data dir (`settings.json` under
|
|
Electron's `userData` path) and **loads the server's own origin**, where
|
|
the server serves the real SPA (the production `web` build, the same
|
|
bytes a browser would load).
|
|
3. On subsequent launches it skips the setup page and loads the saved server
|
|
directly.
|
|
|
|
If the saved server fails to load (server down, DNS failure, TLS error), the
|
|
window falls back to the setup page with the error shown and the failed URL
|
|
pre-filled — the saved URL is kept, so Connect simply retries it.
|
|
|
|
Entering a plain-`http://` URL for a **non-local** host shows a warning first
|
|
(anyone on the network path can act as that server); a second Connect click
|
|
proceeds. `http://localhost:8000` connects with no friction.
|
|
|
|
Change the server later via the **Server → Change Server…** menu item, which
|
|
clears the saved URL and returns the focused window to the setup page.
|
|
|
|
Open another view with **Server → New Window** (`Cmd/Ctrl+N`). It clones the
|
|
focused window's current URL onto a new window against the same server, so two
|
|
conversations can be watched at once.
|
|
|
|
The native enhancements live on the web side in
|
|
[`../src/lib/nativeBridge.ts`](../src/lib/nativeBridge.ts). It detects the
|
|
Electron shell at runtime (the preload exposes `window.omnigentDesktop`
|
|
with `kind: "electron"`) and routes notifications/badge through the IPC
|
|
bridge; in a plain browser it falls back to the Web Notifications path. So the
|
|
one `web` bundle works both in a browser and under Electron.
|
|
|
|
## Architecture
|
|
|
|
```
|
|
electron/
|
|
package.json # Electron + electron-builder deps and build config
|
|
src/main.js # main process: window, settings, menu, IPC, badge, notify
|
|
src/preload.js # contextBridge: window.omnigentDesktop + omnigentSetup
|
|
src/find_preload.js # contextBridge for the find bar: window.omnigentFind
|
|
src/browserViewRegistry.js # per-conversation WebContentsView registry (browser pane)
|
|
src/browserViewBounds.js # CSS-px → window-DIP bounds conversion (browser pane)
|
|
src/browserIpc.js # omnigent:browser-* IPC handlers (extracted from main.js)
|
|
setup/index.html # the bundled "connect to server" setup page
|
|
find/index.html # the bundled find-in-page bar (Cmd/Ctrl+F)
|
|
icons/ # app icons
|
|
```
|
|
|
|
Native niceties beyond notifications/badge: a right-click context menu
|
|
(cut/copy/paste, spelling suggestions + Add to Dictionary, Copy Link
|
|
Address), window size/position persistence across launches, and
|
|
find-in-page (**Edit → Find…**, `Cmd/Ctrl+F`) — a small bar anchored to the
|
|
window's top-right corner; Enter / Shift+Enter step through matches, Esc
|
|
dismisses.
|
|
|
|
- **Main process** (`src/main.js`) owns settings persistence, window
|
|
creation, the application menu, permission handling (microphone), and IPC
|
|
handlers for the badge and notifications (`normalize_url`, `change_server`,
|
|
navigate-to-server, New Window).
|
|
- **Preload** (`src/preload.js`) is the only bridge between the remote
|
|
(untrusted) SPA and the main process. It runs with `contextIsolation` and
|
|
exposes a tiny, serialization-safe API via `contextBridge` — never raw
|
|
`ipcRenderer` or Node.
|
|
- **Security posture**: `nodeIntegration: false`, `contextIsolation: true`.
|
|
`window.open` / `target=_blank` links are opened in the user's real
|
|
browser, not chromeless Electron windows. Non-web schemes (`vscode://`,
|
|
`ssh://`, …) launch an OS protocol handler with page-controlled
|
|
arguments, so they prompt for consent first — showing the requesting
|
|
origin and the full URL — with an optional persisted "always allow this
|
|
scheme from this server". Beyond that, each window is
|
|
**pinned to the one server origin the user explicitly connected it to**,
|
|
and that pin — not navigation — is the trust boundary:
|
|
- Navigation is deliberately _not_ restricted: servers may sit behind
|
|
auth that redirects through external identity providers, so a window
|
|
can legitimately visit foreign origins mid-login.
|
|
- Instead, every privileged IPC handler verifies its sender frame.
|
|
`notify` / `setBadgeCount` only work when both the calling frame _and_
|
|
the window's top-level page are on the pinned origin (so a pinned-origin
|
|
iframe embedded in a hostile page gets nothing); the setup bridge
|
|
(`omnigentSetup`) only works for the bundled setup page itself, so a
|
|
server page can never read or silently re-point the saved server URL.
|
|
Foreign pages get an inert bridge.
|
|
- The microphone permission grant is likewise scoped: only the audio set,
|
|
only for pages on an origin some window is pinned to, and only when the
|
|
requesting page is the top-level page — everything else is denied.
|
|
|
|
## Embedded browser pane
|
|
|
|
The desktop shell hosts an **embedded browser pane**: a real Chromium page the
|
|
user can drive (URL bar + toolbar) and point-and-prompt in design mode. This PR
|
|
covers that user-facing pane plus the Electron/renderer plumbing; the
|
|
agent-facing builtin `browser_*` tools (navigate / snapshot / click / type /
|
|
screenshot) that can also drive the pane land in a separate PR. A
|
|
webview/iframe can't provide screenshots, arbitrary in-page JS, or cross-origin
|
|
navigation, so each browser is a native Electron **`WebContentsView`**
|
|
positioned over a placeholder `<div>` the SPA measures — not an in-page element.
|
|
|
|
```mermaid
|
|
sequenceDiagram
|
|
participant A as Agent (runner — any host)
|
|
participant S as Omnigent server
|
|
participant R as Renderer / BrowserPane (this PR)
|
|
participant V as WebContentsView (local Chromium)
|
|
|
|
Note over A,S: browser_* tools ship in a separate PR
|
|
A->>S: browser_navigate / click / snapshot …
|
|
S->>R: browser.action_request (SSE, on the session stream)
|
|
R->>R: claim the action (single-winner token)
|
|
R->>V: drive via IPC (navigate / capture / …)
|
|
V-->>R: result (URL, snapshot, screenshot)
|
|
R-->>S: POST action result + claim token
|
|
S-->>A: result JSON (or clean timeout)
|
|
```
|
|
|
|
The browser runs on the user's machine (a native `WebContentsView`); the agent —
|
|
which may run on a different host — drives it purely by messages: an action
|
|
request fans out over the session stream, the renderer claims and executes it
|
|
against its local Chromium, and the result is posted back.
|
|
|
|
**Pieces:**
|
|
|
|
- `src/browserViewRegistry.js` — a per-**conversation** `Map` of
|
|
`WebContentsView`s (cap 10). `setActive` attaches one view to the host window
|
|
and **detaches (does not destroy)** the previous one, so a background
|
|
conversation's page keeps running when the user switches away; views are
|
|
destroyed only on explicit close or window teardown. Each child view keeps
|
|
`nodeIntegration:false, contextIsolation:true, sandbox:true`.
|
|
- `src/browserViewBounds.js` — converts the placeholder's renderer CSS pixels to
|
|
window device-independent pixels (they diverge after `Cmd+/Cmd-` zoom).
|
|
- `src/main.js` — instantiates one registry **per shell window** and injects it
|
|
(plus the `isPinnedOriginSender` trust gate) into `registerBrowserIpc(...)`.
|
|
- `src/browserIpc.js` — the whole `ipcMain.handle('omnigent:browser-*')` surface,
|
|
extracted out of `main.js` so that file stays bounded:
|
|
`open-or-navigate`, `set-active`, `resize`, `screenshot`
|
|
(`capturePage().toPNG()` → base64), `execute`, `has-view`, `close`, plus the
|
|
toolbar handlers `go-back`, `go-forward`, `reload`, and `open-devtools`
|
|
(toggle, docked bottom), plus the design-mode handlers
|
|
`enable-design-mode` / `disable-design-mode` / `signal-design-result`
|
|
(inject / tear down the in-page element picker and paint result feedback).
|
|
Every handler is gated on `isPinnedOriginSender` (only
|
|
the connected server's own page may drive the views) and resolves the _sender
|
|
window's own_ registry, so one window can never manipulate another's panes.
|
|
On view creation it also wires `did-navigate` / `did-navigate-in-page`
|
|
listeners that push `browser-url-changed` + `browser-nav-state` to the renderer
|
|
so the toolbar's URL bar live-tracks the real URL (redirects, in-page link
|
|
clicks, agent navigation) instead of going stale.
|
|
- `src/preload.js` — adds `browserOpenOrNavigate/SetActive/Resize/Screenshot/`
|
|
`Execute/Close` + `browserHasView`, the toolbar methods
|
|
`browserGoBack/GoForward/Reload` + `openBrowserDevTools`, the design-mode
|
|
methods `browserEnableDesignMode/DisableDesignMode/SignalDesignResult`, and the
|
|
subscriptions `onBrowserViewCreated` / `onBrowserHostActiveChanged` /
|
|
`onBrowserViewClosed` / `onBrowserUrlChanged` / `onBrowserNavState` +
|
|
`onBrowserElementSelected` / `onBrowserElementPromptSubmit` /
|
|
`onBrowserElementPromptDismiss` to `window.omnigentDesktop`, each a thin
|
|
`ipcRenderer.invoke` / `ipcRenderer.on`.
|
|
- Renderer side (in `web/src`): `hooks/useBrowserAgentRelay.ts` receives the
|
|
`browser.action_request` SSE event (emitted by the separate agent-tools PR),
|
|
**claims** the action on the server
|
|
(atomic check-and-set so two windows on one server can't double-execute),
|
|
runs it via the preload bridge, and POSTs the result back with its claim
|
|
token; `components/BrowserPane/BrowserPane.tsx` measures the placeholder and
|
|
keeps the native view positioned over it. Both self-gate on
|
|
`isElectronShell()`, so a plain browser tab is inert (the action times out on
|
|
the server with a clean "is the desktop app open?" error).
|
|
|
|
**First-navigate activation.** The first `browser_navigate` on a conversation
|
|
creates the view **detached** (nothing is active yet), so no
|
|
`browser-host-active-changed` fires. The registry therefore also emits a
|
|
`browser-view-created` event on create; `BrowserPane` listens for it (and
|
|
probes `browserHasView` on remount), mounts its measuring placeholder, and
|
|
calls `browserSetActive(conversationId)` — which attaches the view and starts
|
|
bounds sync. Without this signal the pane would gate itself off forever and the
|
|
embedded browser would stay invisible. (`browserViewRegistry.test.js` locks the
|
|
create-signal → setActive → attached transition.)
|
|
|
|
**Toolbar.** When a view is attached, `BrowserPane` renders a user-facing
|
|
toolbar above the page: back / forward / reload, a DevTools toggle, and an
|
|
editable URL bar (Enter navigates; the typed value is normalized to add a
|
|
scheme — a dotless host like `localhost` gets `http://`, everything else
|
|
`https://`). The bar reflects the _real_ URL via `onBrowserUrlChanged`, but
|
|
never overwrites what the user is actively typing. The pane is a flex **column**:
|
|
the toolbar is a fixed-height row _above_ the measured container, because the
|
|
native `WebContentsView` paints over that container's rect — a toolbar inside it
|
|
would be hidden by the overlay. The URL bar reuses the existing
|
|
`browserOpenOrNavigate(..., {force:true})` path (the same one the relay uses), so
|
|
no separate navigation IPC exists for manual entry.
|
|
|
|
**Design mode (point-and-prompt).** A toolbar toggle (next to DevTools) injects
|
|
an in-page element picker into the `WebContentsView` via `executeJavaScript`:
|
|
hovering highlights the element under the cursor (overlay + `<Component>`/tag
|
|
label); clicking opens a popup anchored to that element with an input + Send.
|
|
On Send the popup emits a `console.log` marker (the injected script can't
|
|
`require('electron')`), which the per-view console-message listener in
|
|
`browserIpc.js` forwards to the SPA as `browser-element-prompt-submit`
|
|
(carrying the element info; a cropped element screenshot arrives on the earlier
|
|
`browser-element-selected` event). **There is no backend
|
|
design-edit route** — `AppShell` (where the relay is hoisted, so it's listening
|
|
even when the Browser tab isn't mounted) builds a `[Design Mode — …]` prompt,
|
|
attaches the screenshot as a `File`, and sends it through the _normal_ chat
|
|
path (`chatStore.send`, targeting the conversation's own bound agent). It then
|
|
calls `browserSignalDesignResult` so the popup paints green/red feedback. The
|
|
picker markers are `__omni_element_select__` / `__omni_element_prompt_submit__`
|
|
/ `__omni_element_dismiss__`, and the per-view
|
|
console listener is stored on the registry entry
|
|
(`designModeListener` / `designModeWebContents`) so `browserViewRegistry`'s
|
|
`close()` detaches it on teardown. Electron-only (needs `executeJavaScript` +
|
|
the native view); no server flag.
|
|
|
|
**JS trust boundary (important):** `omnigent:browser-execute` runs
|
|
arbitrary JS in the child view via `executeJavaScript(js, true)`. It is exposed
|
|
to the SPA **only for the relay's own fixed templates** (the DOM-snapshot walk,
|
|
and the click / type element resolvers) — there is deliberately **no
|
|
agent-facing generic `evaluate`**. This keeps the _agent_ boundary: the agent
|
|
picks elements by `ref`/`selector` and supplies text, but never ships a raw JS
|
|
string that main will run. (It does not, and is not intended to, defend against
|
|
XSS _within_ the visited page — that page runs its own scripts in its own
|
|
sandboxed view regardless.) Preserve this when extending the bridge: add typed,
|
|
argument-shaped actions, not a passthrough JS channel.
|
|
|
|
**Availability.** The pane is always on in this build — this shell machinery
|
|
activates the moment a `browser.action_request` arrives (the agent-side
|
|
`browser_*` tools that emit it ship in the separate tools PR). No flag to enable
|
|
it. Outside the Electron shell (a plain browser tab)
|
|
the renderer half is inert, so the tools fail cleanly with a "is the desktop
|
|
app open?" error rather than hanging.
|
|
|
|
## Prerequisites
|
|
|
|
- **Node** 22.x + npm (already used by `web`).
|
|
- Electron ships its own Chromium/Node, so no system webview libs are needed
|
|
on Linux for _running_ the built app, though packaging tools may pull a few
|
|
build deps.
|
|
|
|
## Run it (development)
|
|
|
|
From the `web/electron/` directory:
|
|
|
|
```bash
|
|
npm install # installs electron + electron-builder
|
|
npm start # launches the Electron shell
|
|
```
|
|
|
|
The shell opens on the bundled setup page. Point it at a running Omnigent
|
|
server (see below), Connect, and you're in.
|
|
|
|
> Note: this loads the UI from whatever server URL you give it — it does
|
|
> **not** run the Vite dev server. To develop the web UI itself with hot
|
|
> reload, run `npm run dev` (plain Vite in a browser) from `web/` as usual.
|
|
|
|
## Build a distributable
|
|
|
|
From `web/electron/`:
|
|
|
|
```bash
|
|
npm run build # current platform
|
|
npm run build:mac # .dmg + .zip (signed if an identity is available, not notarized)
|
|
npm run build:mac:release # .dmg + .zip, signed + notarized (requires credentials, see below)
|
|
npm run build:linux # AppImage + .deb
|
|
npm run build:win # NSIS installer
|
|
```
|
|
|
|
Output lands in `electron/dist/` (the DMG is named
|
|
`Omnigent-<version>-<arch>.dmg`).
|
|
|
|
## macOS code signing & notarization
|
|
|
|
The mac build is configured for Apple's **hardened runtime** with the
|
|
entitlements Electron needs (`build/entitlements.mac.plist`: V8 JIT plus
|
|
microphone for dictation). Signing is driven entirely by what credentials
|
|
are present — there are no code changes between a dev build and a release
|
|
build:
|
|
|
|
| Credentials present | Result |
|
|
| ------------------------------------------------------------------ | -------------------------------------------------------------------- |
|
|
| none | ad-hoc–signed app; runs locally, other Macs see a Gatekeeper warning |
|
|
| Developer ID cert | signed app; downloads still warn until notarized |
|
|
| Developer ID cert + Apple notarization creds (`build:mac:release`) | signed + notarized; installs cleanly everywhere |
|
|
|
|
### 1. Get a signing certificate
|
|
|
|
You need a **Developer ID Application** certificate from an Apple Developer
|
|
Program account (the kind used for distribution _outside_ the App Store).
|
|
Create it at <https://developer.apple.com/account/resources/certificates>
|
|
(or via Xcode → Settings → Accounts → Manage Certificates), then either:
|
|
|
|
- **Keychain (local builds):** install the cert + private key into your
|
|
login keychain. electron-builder auto-discovers it — `npm run build:mac`
|
|
just works. Verify with
|
|
`security find-identity -v -p codesigning` (you should see
|
|
`Developer ID Application: <Your Name> (<TEAMID>)`).
|
|
- **Env vars (CI):** export the cert + key as a password-protected `.p12`
|
|
and set:
|
|
|
|
```bash
|
|
export CSC_LINK=/path/to/developer-id.p12 # or a base64 string / https URL
|
|
export CSC_KEY_PASSWORD='the p12 password'
|
|
```
|
|
|
|
To force an **unsigned** build even when a cert is present (faster dev
|
|
iteration): `CSC_IDENTITY_AUTO_DISCOVERY=false npm run build:mac`.
|
|
|
|
### 2. Notarize (release builds)
|
|
|
|
Notarization uploads the signed app to Apple for malware scanning;
|
|
without it, macOS warns on first launch of a downloaded app. It needs
|
|
network access and Apple credentials — either an App Store Connect API
|
|
key (preferred for CI):
|
|
|
|
```bash
|
|
export APPLE_API_KEY=/path/to/AuthKey_XXXXXXXXXX.p8
|
|
export APPLE_API_KEY_ID=XXXXXXXXXX
|
|
export APPLE_API_ISSUER=<issuer-uuid>
|
|
```
|
|
|
|
or your Apple ID with an [app-specific password](https://support.apple.com/102654):
|
|
|
|
```bash
|
|
export APPLE_ID=you@example.com
|
|
export APPLE_APP_SPECIFIC_PASSWORD=xxxx-xxxx-xxxx-xxxx
|
|
export APPLE_TEAM_ID=<TEAMID>
|
|
```
|
|
|
|
then:
|
|
|
|
```bash
|
|
npm run build:mac:release
|
|
```
|
|
|
|
This is the same build with `mac.notarize=true` switched on; expect the
|
|
notarization step to add a few minutes (Apple-side processing). Verify the
|
|
result with:
|
|
|
|
```bash
|
|
spctl -a -vv dist/mac-arm64/Omnigent.app # → "accepted, source=Notarized Developer ID"
|
|
```
|
|
|
|
`build:mac:release` **fails loudly** if signing or notarization
|
|
credentials are missing — that's intentional, so a release artifact can't
|
|
silently ship unsigned.
|
|
|
|
## Getting a server to point at
|
|
|
|
Any reachable Omnigent server works. For a quick local target, run the
|
|
server from this repo:
|
|
|
|
```bash
|
|
# from the repo root, with the project venv:
|
|
.venv/bin/python -m omnigent.server # serves on http://localhost:8000
|
|
```
|
|
|
|
Then enter `http://localhost:8000` in the setup page.
|
|
|
|
## Managing servers and hosting
|
|
|
|
Beyond pointing at an already-running server, the shell can drive the local
|
|
`omnigent` CLI to start a server and register this machine as a **host** (a
|
|
machine that runs the agent work a server dispatches). Two concepts stay
|
|
deliberately separate:
|
|
|
|
- **Server** — the backend the webview talks to (local or remote).
|
|
- **Host** — _this machine_ executing agent work for a server. Because hosting
|
|
runs agent code, it is **opt-in** and **explicit**: the shell never connects
|
|
this machine as a runner on its own — not on connect, not on launch. You
|
|
connect it from the **host selection menu** inside the app (when starting a
|
|
chat, pick this machine), which drives `controlHost` over the bridge. That
|
|
request alone isn't trusted to authorize hosting: the SPA is served by the
|
|
server, so `start`/`restart` additionally require a **native, main-process
|
|
confirmation** the page can't forge or auto-dismiss (persisted per server
|
|
origin, so a trusted server is asked only once).
|
|
|
|
### Detecting the CLI and customizing its path
|
|
|
|
The CLI ships under two names that resolve to the same entry point — `omnigent`
|
|
(canonical) and `omni` (short alias) — and the shell probes **both**:
|
|
`settings.omnigent_path` first, then `PATH` (`omnigent` then `omni`), then the
|
|
well-known install locations (`~/.local/bin`, `~/.cargo/bin`, Homebrew,
|
|
`/usr/local/bin`, each tried under both names). A GUI-launched app inherits a
|
|
minimal `PATH`, which is why the install locations are probed directly. The path
|
|
is resolved once at startup and cached in-memory for the session.
|
|
|
|
You can see and change which binary is used in two places:
|
|
|
|
- **Setup page** — hidden by default behind a **gear icon** (top-right) that
|
|
opens a small modal. The resolved/auto-detected path shows as the field's
|
|
**placeholder** (the value stays empty until you type an override); set it via
|
|
free-text or a native file picker. When nothing is found the gear gets an
|
|
accent dot and the modal shows the install one-liner
|
|
```bash
|
|
curl -fsSL https://raw.githubusercontent.com/omnigent-ai/omnigent/main/scripts/install_oss.sh | sh
|
|
```
|
|
- **In-app** — **Settings → Local CLI** (desktop only): shows the resolved path
|
|
and version, a **Change…** button (native file picker) and **Reset to
|
|
auto-detected**. For safety the in-app surface exposes **no free-text setter**
|
|
— a connected server must not be able to silently repoint the CLI at an
|
|
arbitrary binary, so changing it requires a user-driven OS dialog.
|
|
|
|
A configured path is saved to `settings.json` (`omnigent_path`) only once it
|
|
validates as a runnable CLI; clearing it reverts to auto-detection. Connecting
|
|
to a **remote** server never needs the CLI — only "Start locally" and hosting do.
|
|
|
|
### Start locally
|
|
|
|
**"Start a server on this machine"** runs `omnigent server start` (idempotent —
|
|
reuses a healthy one) and then connects this window to its
|
|
`http://127.0.0.1:<port>` URL through the normal connect flow. It does not
|
|
connect this machine as a runner — that stays an explicit step in the app.
|
|
|
|
### Connecting this machine as a runner
|
|
|
|
There is **no** connect-time toggle and no sidebar status row: the shell never
|
|
connects a runner automatically. Inside the connected app, the host selection
|
|
menu (when starting a chat) tags this machine and offers to connect it. Choosing
|
|
it calls `controlHost("start")` over the bridge. Because that call originates in
|
|
server-served code, the main process does not treat it as the user's consent: on
|
|
the first `start`/`restart` for a server origin it shows a **native confirmation
|
|
dialog** ("Allow _host_ to manage Omnigent on this machine?") with **Don't Allow**
|
|
(default) / **Allow Once** / **Always Allow**. Only after approval does it — once
|
|
the CLI is authenticated for the server (remote only; local needs none) — either
|
|
adopt a daemon already serving that server (one you started by hand) or spawn
|
|
`omnigent host --server <url>`. **Allow Once** connects this time and re-prompts
|
|
next time; **Always Allow** records the origin in `settings.json`
|
|
(`allowed_hosting_origins`) so later connects skip the prompt. `stop` is
|
|
fail-safe and needs no confirmation. The same bridge exposes `stop` / `restart`.
|
|
|
|
Status is read live (host connected = a live daemon process **and** an online
|
|
host tunnel; the shell never caches it). The host surface goes through the JS
|
|
bridge — `window.omnigentDesktop` → `getHostStatus` / `getHostIdentity` /
|
|
`onHostStatusChanged` (read + live) and `controlHost` (start/stop/restart),
|
|
typed in [`../src/lib/nativeBridge.ts`](../src/lib/nativeBridge.ts) and gated to
|
|
the window's **pinned origin** like the badge/notification bridge.
|
|
|
|
### Lifecycle
|
|
|
|
The desktop **owns the host processes it starts**: quitting the app SIGTERMs
|
|
them (and stops a local server it started), so closing the app disconnects this
|
|
machine. A daemon the shell merely _adopted_ (you started it in a terminal) is
|
|
left running on quit. Hosting is **not** restored on the next launch — you
|
|
reconnect this machine explicitly from the host menu when you want it.
|
|
|
|
## Passkeys (WebAuthn)
|
|
|
|
External security keys (e.g. a YubiKey) work out of the box: Chromium's
|
|
content layer speaks CTAP to the key directly. That's also why the flow is
|
|
_invisible_ — the passkey sheet you see in Chrome/Safari is browser chrome,
|
|
which Electron doesn't ship. Touching the key completes the ceremony with no
|
|
UI.
|
|
|
|
For a visual flow, the shell enables Electron's **Touch ID platform
|
|
authenticator** (`app.configureWebAuthn`, Electron ≥ 42, macOS only):
|
|
registering or signing in with a platform passkey then shows the native
|
|
macOS Touch ID / keychain dialog, and a native chooser appears when several
|
|
saved passkeys match. Three pieces must agree before this activates:
|
|
|
|
1. `WEBAUTHN_KEYCHAIN_ACCESS_GROUP` in `src/main.js` —
|
|
`"<TEAM_ID>.ai.omnigent.desktop"`.
|
|
2. The same string in the `keychain-access-groups` entitlement in
|
|
`signing/entitlements.mac.plist`.
|
|
3. An **embedded Developer ID provisioning profile**
|
|
(`signing/omnigent.provisionprofile`, wired via `provisioningProfile`
|
|
in `package.json`). `keychain-access-groups` is a _restricted_
|
|
entitlement: a Developer ID signature alone doesn't authorize it, and
|
|
AMFI SIGKILLs the app at launch ("Launchd job spawn failed", POSIX
|
|
error 163). Create the profile in the Apple Developer portal: an App ID
|
|
for `ai.omnigent.desktop` (no extra capabilities — every profile
|
|
automatically authorizes keychain groups under `<TEAM_ID>.*`), then
|
|
Profiles → Distribution → Developer ID for that App ID. Verify with
|
|
`security cms -D -i signing/omnigent.provisionprofile`.
|
|
|
|
The signing identity's team must match the group prefix —
|
|
`package.json` pins `"identity"` for this reason (with several certs in
|
|
the keychain, electron-builder's auto-discovery can pick the wrong one).
|
|
Helpers must NOT inherit the keychain entitlement
|
|
(`entitlementsInherit` points at the minimal
|
|
`signing/entitlements.mac.inherit.plist`; a restricted entitlement on a
|
|
helper shows up as a "GPU process exited unexpectedly" crash loop).
|
|
|
|
It only works in a **code-signed** build, on Macs with a Secure Enclave.
|
|
Until all three are set — and always in unsigned `npm start` dev runs —
|
|
the platform authenticator stays off and security keys remain the
|
|
(working, silent) path.
|
|
|
|
Caveats: these passkeys are device-bound in the app's own keychain access
|
|
group — they are **not** synced via iCloud Keychain, and passkeys you saved
|
|
in Safari/Chrome are not visible to the app (and vice versa). Showing the
|
|
full system passkey sheet (iCloud Keychain, cross-device QR) for arbitrary
|
|
user-chosen servers would require Apple's browser-only
|
|
`web-browser.public-key-credential` entitlement, or per-domain associated
|
|
domains — neither fits an app whose servers are user-deployed.
|
|
|
|
## Localhost access (auth flows)
|
|
|
|
Trusted pages may call services on the user's own machine
|
|
(`http://localhost:<port>`, `127.0.0.1`, `[::1]`) even when those
|
|
services don't send CORS headers — authentication flows use this to
|
|
reach local auth helpers/token brokers. The shell injects the CORS (and
|
|
preflight) response headers itself, scoped to requests _from_ a trusted
|
|
page origin _to_ a loopback host; see `src/localhost_cors.js`. Trusted
|
|
means:
|
|
|
|
- a window's **pinned server origin**, or
|
|
- the **current top-level page of a pinned window** — auth flows
|
|
redirect the main frame through SSO/IdP origins that can't be known in
|
|
advance (server → SSO domain → localhost helper probe), and those
|
|
pages get localhost access while the user is actually on them.
|
|
In-window navigation only starts from the pinned server (links/popups
|
|
open in the external browser), so this doesn't extend to arbitrary
|
|
sites; iframes never match (main-frame origin only).
|
|
|
|
Anything else stays blocked by normal CORS, and a localhost service that
|
|
sends its own `Access-Control-Allow-Origin` keeps enforcing its own
|
|
policy untouched.
|
|
|
|
If a page needs localhost while _not_ being the visible top-level page,
|
|
hand-add its origin to `settings.json`:
|
|
|
|
```json
|
|
{ "localhost_allowed_origins": ["https://login.example.com"] }
|
|
```
|
|
|
|
(`settings.json` lives in Electron's per-user `userData` dir — on macOS,
|
|
`~/Library/Application Support/Omnigent/settings.json`.)
|
|
|
|
## Multiple servers
|
|
|
|
One server URL is saved as the default, but extra windows can be opened
|
|
against _different_ servers via **Server → New Window on Different
|
|
Server…**. It opens a setup page in **per-window** mode: the URL you connect
|
|
applies to that window only and is never saved, so the default server is
|
|
untouched and the extra connection ends when the window closes. These
|
|
windows get the same per-window origin pinning as regular ones. With windows
|
|
on more than one server, the dock badge shows the sum of each server's unread
|
|
count and notification titles are prefixed with the firing server's hostname.
|
|
|
|
## Implementation notes
|
|
|
|
- **Runtime:** bundled Chromium (so the build is ~100+ MB, but the renderer
|
|
matches Chrome's behavior exactly — no OS-webview quirks).
|
|
- **Native bridge detection:** `window.omnigentDesktop` (`kind: "electron"`),
|
|
exposed by the preload. The web-side `nativeBridge.ts` routes the badge to
|
|
`app.setBadgeCount` and notifications to the main-process `Notification` API
|
|
via IPC; in a plain browser it falls back to the Web Notifications path.
|
|
- **File drag-drop** works by default (Electron doesn't intercept HTML5 file
|
|
drops).
|
|
- **Toolchain:** Node only — no Rust or platform webview libraries.
|
|
|
|
> Historical note: an earlier Tauri-based shell lived in `web/src-tauri`.
|
|
> It was removed in favor of shipping Electron only; `nativeBridge.ts` no
|
|
> longer carries a Tauri code path.
|