文件历史

提交图

37 次代码提交

作者 SHA1 备注 提交日期
Gregor Žunič e81172eeac docs: bound model-owned browser collection 2026-07-18 22:05:02 -07:00
Gregor Žunič 3d481967d4 feat: let agents design browser observations 2026-07-18 21:29:09 -07:00
Gregor Žunič e7a86fc654 feat: add cost-efficient browser primitives 2026-07-18 18:10:58 -07:00
Gregor Žunič 23f4c8801a Restore sticky video editing heuristics 2026-07-16 20:14:20 -07:00
Gregor Žunič 3220402522 Fold video workflow into browser harness 2026-07-16 19:33:05 -07:00
Gregor Žunič 2cecabb27f Simplify and harden video workflow 2026-07-16 18:21:13 -07:00
Gregor Žunič 0f78a99bc5 Make recording opt-in and automate video production 2026-07-16 17:08:38 -07:00
Gregor Žunič 611fc046a0 Stage inset + revert cards to grotesk
Camera output maps to a 94% stage so backdrop margin stays visible on
every side at any zoom. Card titles back to Helvetica Neue semibold —
the serif experiment read poorly at video sizes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 11:26:01 -07:00
Gregor Žunič e0174988da Cards: editorial serif titles (Tiempos-style stack, regular weight)
Title cards switch from 800-weight system sans to a serif at regular
weight — Tiempos Headline with Iowan Old Style/Georgia fallbacks —
over the paper backdrop, with the mono subline as a muted eyebrow.
Captions move to Helvetica Neue.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 11:20:53 -07:00
Gregor Žunič bd8dfdbc31 Bottom rail redesign: one ink/paper palette, aligned baseline, logo in badge
Chips drop the cyan-mono dev styling for a single adaptive palette
(ink on paper over light backdrops, paper on ink over dark). Step chip,
caption, and session badge share one baseline rail; the browser-use
logo moves inside the session badge. Captions grow to 44px. Wide shots
get Screen-Studio margins (PAD 108) and gentler camera moves.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 11:16:55 -07:00
Gregor Žunič 71589264d3 Backdrop: flat warm off-white default, configurable bg, adaptive ink
bg takes one color (flat, default #F0EEE6) or [c1, c2] (gradient).
Cards, logo, and progress bar switch between dark ink and white based
on backdrop luminance; the window shadow softens on light backgrounds.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 11:10:45 -07:00
Gregor Žunič 15964f2002 Polish video look: window-aware zooms, bottom-docked telemetry, subtle captions
Zoom targets snap to the window edge when a shot would slice just below
it, so the full Chrome frame (tab, traffic lights, omnibox) stays in
frame at any zoom instead of degrading into a cropped screenshot. Chips
shrink to 15px and dock in the bottom corners clear of the window;
captions drop from 72px billboards to 34px pills; the click-coordinate
debug text is gone; backdrop is a lit gradient instead of a dot grid.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 10:52:56 -07:00
Gregor Žunič ab9a9b440b Add secret redaction: scrub credential URLs in trace, pixelate on-screen regions
Recorder always strips credential-bearing query/fragment params (OAuth
codes, tokens, api keys) from URLs before they land in events.jsonl.
The video template takes redact: {"frame.jpg": [{x,y,w,h}]} and
pixelates those page-px regions wherever the frame is drawn, tracking
zooms and after-swaps. Auto-record is back on by default (BH_RECORD=0
opts out); making a video is a separate, discretionary step.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-15 10:40:41 -07:00
Gregor Žunič c9742a90fb Compress make-video.md to minimal form, same content
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 10:02:03 -07:00
Gregor Žunič c5de1e5417 Trim skill docs: dedupe gotchas, drop formula internals
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-15 10:00:27 -07:00
Gregor Žunič d0f2649d26 Video template polish: zoom diet, error treatment, reaction framing
- Oversized typing overlay when the camera is wide (typing readable
  without a punch-in); zooms become rare emphasis, not the default
- error:true beats: red vignette, shake, ERROR chip, red caption pill
- Hero captions (72px, shrink-to-fit), 22px telemetry chips
- Sequenced motion: camera settles -> cursor glides -> click -> result
  swaps in as a near-hard cut; hard cuts by default between beats
- Skill: readability formula, zoom rationing, "frame the reaction" rule

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 22:21:11 -07:00
Gregor Žunič e88f4e327a Add built-in session recording + video generation
Record a browser session as a folder of per-action screenshots + an
events.jsonl trace (start_recording/stop_recording helpers, marker file
spans CLI invocations, no daemon changes, no new deps). An agent then
edits the recording into a Screen-Studio-style clip via the make-video
skill: a self-contained HTML compositor renders the frames + a
composition on a canvas (realistic Chrome window, calm zoom camera,
synthetic cursor, telemetry overlay, error treatment) and exports webm
through the browser's own MediaRecorder — no ffmpeg.

- src/browser_harness/recorder.py: capture layer (+ 6 unit tests)
- src/browser_harness/{run,helpers}.py: observe() hook + helper exports
- interaction-skills/video-template.html: the compositor/renderer
- interaction-skills/make-video.md: the editor skill
- SKILL.md: recording section

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 22:14:19 -07:00
Sarath Suresh a9f7b1d547 Remove -c script execution 2026-05-12 15:35:12 +05:30
Alezander9 a2443d1d51 docs: standardize harness docs against canonical browser connection reference
Cross-checked AGENTS.md, README.md, SKILL.md, install.md, and profile-sync.md for accuracy. Tightened the sync_local_profile docstring and the chrome://inspect stderr message to match. Cloud-bootstrap test updated to set BU_AUTOSPAWN. All unit tests pass.
2026-05-02 18:02:36 -07:00
Alezander9 db41106662 docs: replace remaining heredoc examples with -c flag
PR #229 (src layout refactor) updated SKILL.md and run.py to the -c form
when the CLI dropped stdin support, but missed install.md (steps 2 and 7)
and interaction-skills/profile-sync.md. Sync them up.

Picks up the still-relevant hunks from #215 (closes #215, refs #213).

Co-authored-by: Andres Gonzalez <62394570+FVTVLIX@users.noreply.github.com>
2026-04-29 13:55:44 -07:00
Alezander9 71cf3a87a0 Merge remote-tracking branch 'origin/main' into fix/windows-ipc
# Conflicts:
#	SKILL.md
#	admin.py
#	helpers.py
2026-04-27 17:41:08 -07:00
Alezander9 991ab21f80 Windows support: route IPC through ipc.py (TCP on Windows, AF_UNIX on POSIX)
The harness was Linux/macOS-only because daemon IPC hardcoded AF_UNIX sockets
at /tmp/bu-*.sock paths and asyncio.start_unix_server, all of which are
unavailable or invalid on Windows. Worse, uv-managed Python on Windows
(python-build-standalone) ships without socket.AF_UNIX entirely (#124).

New ipc.py centralizes the platform fork:
  - POSIX: AF_UNIX socket at <tempdir>/bu-<NAME>.sock (chmod 0600), unchanged
    semantics from the prior /tmp-hardcoded path.
  - Windows: TCP loopback on 127.0.0.1:<ephemeral>, with the chosen port
    written to <tempdir>/bu-<NAME>.port so clients can find the daemon.
    Uses asyncio.start_server (stdlib, no obscure APIs, no third-party deps).

Path discipline: log/pid/port files all sit under tempfile.gettempdir() so
they land in /tmp on Linux, $TMPDIR on macOS, %TEMP% on Windows. helpers.py
screenshot() default also moves from /tmp/shot.png to tempfile.gettempdir().

subprocess detach uses start_new_session=True on POSIX and
DETACHED_PROCESS|CREATE_NEW_PROCESS_GROUP on Windows via ipc.spawn_kwargs().

run.py reconfigures stdout to UTF-8 on Windows so print(page_info()) doesn't
UnicodeEncodeError on the 🟢 marker that helpers prepend to tab titles
(#124 item 4). cp1252 (PowerShell default) can't encode it.

Verified end-to-end on Windows 11 with Chrome remote debugging:
  - daemon spawns, allocates port, writes .port file
  - goto + page_info + screenshot round-trip through TCP loopback
  - restart_daemon cleans up .port and .pid

POSIX path is logically equivalent to the prior code (same AF_UNIX call,
same socket-file semantics, same chmod 0600), routed through ipc.py.

Closes #124 items 1, 2, 4. Item 3 (Chrome 147 user-data-dir) is a separate
concern not addressed here.
2026-04-27 16:40:56 -07:00
Saurav Panda 71cb1f3581 feat(helpers): add max_dim to capture_screenshot
Long agent sessions on 2× displays bust the 2000px-per-side limit some
image-aware LLMs enforce — a 2296×1143 CSS viewport produces a 4592×2286
PNG. Passing max_dim=1800 downscales the file before save (only when the
image actually exceeds max_dim), keeping callers that don't pass it
unchanged.
2026-04-27 12:00:36 -07:00
Sarath Suresh fbd9146df5 refactor: rename goto/click/screenshot to avoid Playwright name overlap
- goto()       → goto_url()
- click()      → click_at_xy()
- screenshot() → capture_screenshot()

These three shared exact names with Playwright but different argument
shapes, causing agent confusion. Updated helpers.py and all markdown
skill files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-24 11:26:07 +05:30
Magnus Müller 82a1f2f540 remote: stop_remote_daemon helper, skill updates from sub-agent testing (#96)
* remote: stop_remote_daemon helper, skill updates from sub-agent testing

Three fresh sub-agents ran typical-user prompts against the skill
("start a remote browser with my logged-in data" / "Stripe only, no
Google" / "refresh my existing profile"). All three completed
end-to-end — cookies made the round trip, filters scoped cleanly,
refresh was idempotent. But they hit a few doc/code seams worth
closing:

admin.py:
- Add stop_remote_daemon(name="remote"). Sub-agents kept reaching
  for restart_daemon() because there was no obvious way to end a
  remote session; the new alias pairs symmetrically with
  start_remote_daemon() and makes the intent explicit. Same
  underlying implementation — it's just naming for callers.

run.py:
- Pre-import stop_remote_daemon so it's usable from
  browser-harness <<'PY' ... PY without an extra import line.

interaction-skills/profile-sync.md:
- Drop the "close Chrome before syncing" trap at the top of the
  Traps section. Obsolete on profile-use v1.0.5+ — the tool copies
  the profile dir to a temp and syncs from the copy. Two sub-agents
  verified sync works with Chrome open on v1.0.5. Kept a small
  note at the bottom for anyone on older versions.
- Document the ♻️ / 📝 reuse-vs-create signal in sync_local_profile
  output, so agents can confirm cloud_profile_id was accepted
  without counting profiles.
- Clarify the API path convention for _browser_use and the raw
  examples: paths are relative to BU_API, not absolute
  /api/v3/... Sub-agents were copy-pasting the old doc form and
  getting 404s.
- Add a one-liner for looking up an existing cloud profile's UUID.
- Surface stop_remote_daemon in the Python API overview.

Housekeeping:
- Stopped 5 sub-agent-leftover remote browsers from today's testing
  burst (not committed; just an operations note).

* skill: UUID lookup pattern must handle 0 and >1 matches (cubic review)

cubic flagged the one-liner as unsafe: 'next(p["id"] for p in … if …)'
raises StopIteration on no match and silently picks the first duplicate
when names repeat. Profile names genuinely aren't unique — sub-3 in the
same testing round surfaced a duplicate on this account — so use a list
comprehension and require exactly one match before using the UUID.
2026-04-19 00:23:28 -07:00
Magnus Müller 759f44ff13 remote: paginate list_cloud_profiles; expose profile-use v1.0.4 sync flags (#93)
Two fixes that fell out of testing PR #84 against a real account.

1) list_cloud_profiles was hitting /profiles?pageSize=200 and getting 422
   back — the API caps pageSize at 100. Anyone with more than 10 profiles
   already saw a silent truncation before this (the request returned 10
   items by default) and anyone bumping past 100 would hit the hard error.
   Now paginates with pageSize=100 until totalItems is reached. My account
   is at 18 and climbs every sync_local_profile() call, so this was going
   to bite shortly.

2) sync_local_profile only exposed profile_name + browser. profile-use
   v1.0.4 shipped three more flags that solve real pain:
     --cloud-profile-id <uuid>  → update an existing cloud profile
     --domain <d>               → only these domains (repeatable)
     --exclude-domain <d>       → drop these domains (repeatable)
   Wired up as cloud_profile_id / include_domains / exclude_domains kwargs.
   When cloud_profile_id is passed, profile-use prints "♻️ Using existing
   cloud profile" instead of "Profile created: <uuid>" — special-cased the
   regex path to just return the caller-supplied UUID.

Also drops the "Upstream limitations" section from profile-sync.md (both
limitations are fixed as of v1.0.4) and adds two worked examples:
  - refresh the same cloud profile (cloud_profile_id=)
  - push only Stripe cookies into it (include_domains=["stripe.com"])

Verified end-to-end:
  sync_local_profile("browser-use.com", include_domains=["stripe.com"])
  → "Domain filter: 43 → 1 cookies (include=[stripe.com])"
  → cloud profile cookieDomains == ["m.stripe.com"]
  → second call with cloud_profile_id=uuid printed "Using existing cloud
    profile" and returned the same UUID (idempotent).
2026-04-18 22:38:43 -07:00
Magnus Müller 96ccb7692b remote: Python API for remote browsers, profiles, and local-profile sync (#84)
* remote: Python API for remote browsers, profiles, and local-profile sync

No CLI, no new entrypoint. Every helper is a Python function callable from
inside a normal `browser-harness <<'PY'` block. run.py pre-imports them.

admin.py:
- start_remote_daemon(name, profileName=None, **create_kwargs)
  Now forwards every documented POST /browsers kwarg (profileId, profileName,
  proxyCountryCode, timeout, customProxy, browserScreenWidth/Height, ...).
  profileName is resolved client-side via list_cloud_profiles — no browser-use
  API change needed. Prints liveUrl and auto-opens it locally when a GUI is
  detected (macOS/Windows always; Linux needs $DISPLAY / $WAYLAND_DISPLAY);
  headless servers print only.
- list_cloud_profiles() — GET /api/v3/profiles + per-profile detail; returns
  [{id, name, cookieDomains, lastUsedAt, userId}]. Agents should report
  len(cookieDomains) not the full list — profiles can have 500 cookies across
  dozens of domains.
- list_local_profiles(), sync_local_profile(name) — shell out to `profile-use`.
  sync_local_profile returns the newly-created cloud UUID.

Profile-sync skill rewritten Python-first with the chat-driven flow (ask the
user which profile; summarize by domain count, never dump cookies) and calls
out the two upstream limitations (sync always creates a new cloud profile; no
per-domain filtering) that need a PR to browser-use/profile-use — they can't
be fixed in browser-harness because the Browser Use API has no cookie
upload/download endpoint.

SKILL.md remote-browsers section updated to match, leading with the parallel
sub-agent use case.

* remote: fix misleading 'no GUI' message when webbrowser.open raises

cubic flagged this on #84: if _has_local_gui() is True but webbrowser.open
raises (e.g. no default browser configured), the code fell through to the
final 'no local GUI — share the liveUrl' line, which is wrong on both counts.
Restructure so each branch produces exactly one accurate message.
2026-04-18 22:19:47 -07:00
Magnus Müller 5d06079d7f page_info: surface pending native dialog (#65)
Agents rarely know to check for open alert/confirm/prompt/beforeunload
dialogs — they freeze the JS thread, so page_info()'s Runtime.evaluate
returns a stale or misleading viewport dict while the page is blocked.

Track the latest unhandled Page.javascriptDialogOpening in the daemon
(cleared on Page.javascriptDialogClosed) and, when one is open, return
{dialog: {type, message, ...}} from page_info() instead of the usual
viewport dict. The dialogs skill now points agents at this signal.
2026-04-18 14:58:18 -07:00
Magnus Müller 23ce2ec464 Fix daemon attaching to invisible omnibox popup (#60)
* fix daemon attaching to invisible omnibox popup on fresh Chrome

When Chrome opens fresh, the only page targets are chrome://
internal pages and the omnibox popup (1px invisible viewport).
The daemon's attach_first_page() fell back to the popup, making
all subsequent work invisible to the user.

Fix: when no real pages exist, create an about:blank tab via
Target.createTarget instead of attaching to the omnibox popup.

Tested configurations:
- Fresh start with no real tabs → creates about:blank (1112x817)
- Navigate without AppleScript → works, tab visible
- Recovery from stale socket → auto-reconnects
- Chrome restart from scratch → creates about:blank

Also adds interaction-skills/connection.md documenting the
omnibox popup problem and startup sequence.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* add connection skill reference to main SKILL.md

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-18 11:07:43 -07:00
Magnus Müller acda8c72a0 Simplify helper surface (#23)
* Simplify helper surface

* Trim common module and restore dispatch key
2026-04-17 21:37:58 -07:00
MagMueller 9325a8e14f add misc interaction skill stubs 2026-04-17 17:13:16 -07:00
MagMueller ebb08d2d2d restore scaffold notes except uploads 2026-04-17 17:03:26 -07:00
MagMueller 674fd69716 tighten interaction skill scaffolds 2026-04-17 16:58:53 -07:00
MagMueller 7287837653 scaffold interaction skill notes 2026-04-17 16:53:31 -07:00
MagMueller 13eed1c64e default list_tabs to include chrome pages 2026-04-17 16:43:24 -07:00
MagMueller baeb317790 document tab control and visible ordering 2026-04-17 16:34:30 -07:00
MagMueller d6942b61fe add interaction-skills and domain-skills structure
Two skill categories, pure markdown, no Python files:

- interaction-skills/ — generic browser patterns (dialogs, inputs, etc.)
  Flat .md files. Agent reads the relevant one before a task.

- domain-skills/ — per-site playbooks (tiktok/, linkedin/, etc.)
  Flat .md files per action (upload, schedule, post).
  Subfolders for large domains when needed.

Starting with:
- interaction-skills/dialogs.md — CDP vs JS dialog handling
- domain-skills/tiktok/upload.md — full upload flow with gotchas
- Placeholder folders for linkedin, spreadshirt, salesforce

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 13:46:37 -07:00