Without this, a caller passing BH_TMP_DIR=<custom dir> to a directory that
doesn't exist yet causes the first write (sock/port/pid/log/screenshot) to
fail with FileNotFoundError. Default _TMP (/tmp on POSIX, gettempdir() on
Windows) always exists, so this is latent today; it bites the per-session
scratch-dir use case browsercode is about to introduce.
Single root cause -> single fix: cover screenshots, sock, port, pid, and
log paths uniformly.
DETACHED_PROCESS overrides CREATE_NO_WINDOW per Win32 docs, so combining
them caused Windows to allocate a fresh console for the daemon. Closing
that window killed the daemon and forced Chrome to re-prompt for remote
debugging permission. Drop DETACHED_PROCESS, keep CREATE_NEW_PROCESS_GROUP
for terminal-close survival.
Lets callers (e.g. browsercode per-session scratch) redirect all harness
file output via one env var. Default behavior unchanged when unset:
/tmp on POSIX, gettempdir() on Windows.
helpers.capture_screenshot and the debug-click overlay now route through
ipc._TMP so the same knob covers screenshots.
The stale-daemon probe in ensure_daemon used raw socket.AF_UNIX, which:
1. Doesn't exist on uv-bundled Python on Windows (AttributeError).
2. Would point at a TCP display string '127.0.0.1:<port>', not a socket
path, even if AF_UNIX existed.
Either way the probe always raised, was swallowed by 'except Exception',
and fell through to restart_daemon — killing and respawning the daemon
on every warm call. Symptom is most visible on Windows where the warm
path is the common case.
Fix: use ipc.connect(name) which already does the right thing per
platform (AF_UNIX on POSIX, TCP loopback on Windows). Same call
daemon_alive and restart_daemon already use.
Also drop the now-unused _paths() helper; restart_daemon only needed
the pid_path half, inline that.
When remote-debugging is not enabled, `run_setup` opens chrome://inspect
once, then retries `ensure_daemon` in a loop. But `ensure_daemon` itself
also calls `_open_chrome_inspect()` on failure — opening a new tab every
~7 seconds and flooding the browser.
Add `_open_inspect` parameter to `ensure_daemon` so the retry loop in
`run_setup` can suppress redundant tab opens after the first one.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Saurav Panda <sgp65@cornell.edu>
Add browser_connections() / active_browser_connections() to admin, which probe
each daemon socket via a new connection_status meta message. The daemon tracks
target_id (set on attach and on set_session) and returns the live page title/URL
via Target.getTargetInfo. switch_tab now forwards target_id in set_session so the
daemon stays in sync after tab switches. run_doctor shows the count, each daemon
name, and the truncated active-page title and URL.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace naive `"return " in expression` check with a character-level parser that
ignores strings, line comments, and block comments. Add `_decode_unserializable_js_value`
to handle NaN, ±Infinity, -0, and BigInt. Extract `_runtime_value` and
`_runtime_evaluate` to unify error handling across `js()` and `page_info()`.
Wrap Runtime.evaluate TimeoutErrors in RuntimeError with expression context.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously js() silently returned None when Chrome reported a syntax or
runtime error. Now it raises RuntimeError with the error description,
line/column location, and a snippet of the failing expression.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(tests): reorganize into tests/unit and tests/integration
Moves all root-level test_*.py files into a structured tests/ directory:
- tests/unit/ — admin, helpers (was test_screenshot), run
- tests/integration/ — js expression tests
- tests/conftest.py — shared fake_png pytest fixture, eliminating duplication
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor: move to src layout, agent-workspace, and fix SKILL.md invocation format
- Move package to src/browser_harness/ and domain-skills/interaction-skills to agent-workspace/
- Fix all browser-harness <<'PY' heredoc examples in SKILL.md and run.py HELP string to use the correct -c '...' flag format (heredoc was never supported by the CLI)
- Update SKILL.md path references from domain-skills/ to agent-workspace/domain-skills/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
tempfile.gettempdir() on macOS returns /var/folders/xx/yy.../T/ (~49 chars).
Combined with bu-{64-char-name}.sock that exceeds the 104-byte sun_path
limit on macOS (108 on Linux), causing daemon startup to fail.
Pre-PR upstream hardcoded /tmp; this restores that for POSIX. Windows is
unaffected (uses TCP, not AF_UNIX).
* fix(daemon): fire-and-forget mark-title eval so load events don't stall
* fix(daemon): bound mark-title eval to 2s so stalled V8 can't pile up tasks
Fire-and-forget kept the perf win, but dropped the 2s upper bound. On a
discarded renderer or hard-hung V8, the Runtime.evaluate task would
never resolve, leaking one task per navigation. Restore the timeout
inside the _silent wrapper.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Saurav Panda <sgp65@cornell.edu>
Cut ipc.py from 127 to 68 lines (-46%) by removing restated docstrings and
keeping only load-bearing inline comments (path-traversal guard, uv-Python
AF_UNIX gating, Windows .port-file role). Same logic, same call sites.
Rename ipc -> _ipc per Python convention for internal modules. The IPC
plumbing is only called by daemon/admin/helpers; agents reading helpers.py
should not be pulled into transport details. Callers do 'import _ipc as ipc'
so internal ipc.foo references stay unchanged.
- daemon.py:1, SKILL.md:151 — docstring/diagram said named pipe on Windows;
the implementation is TCP loopback. Updated.
- daemon.py:209 — if ipc.serve() crashes (e.g. bind failure), the prior
shutdown path could miss it and leave the daemon waiting forever on
d.stop without a listening endpoint. Now race serve_task and stop.wait()
via asyncio.wait(FIRST_COMPLETED): if serve finishes first it must have
raised, so await it to surface the exception. Cleanup cancels both tasks
unconditionally.
- ipc.py:45 — BU_NAME flowed straight into f-strings building filesystem
paths, allowing path traversal outside tempdir. Validate via
^[A-Za-z0-9_-]{1,64}$ in a single _check() helper that all path
builders (log_path/pid_path/port_path/_sock_path) call. Bad names raise
ValueError early with a clear message.
The harness was Linux/macOS-only because daemon IPC hardcoded AF_UNIX sockets
at /tmp/bu-*.sock paths and asyncio.start_unix_server, all of which are
unavailable or invalid on Windows. Worse, uv-managed Python on Windows
(python-build-standalone) ships without socket.AF_UNIX entirely (#124).
New ipc.py centralizes the platform fork:
- POSIX: AF_UNIX socket at <tempdir>/bu-<NAME>.sock (chmod 0600), unchanged
semantics from the prior /tmp-hardcoded path.
- Windows: TCP loopback on 127.0.0.1:<ephemeral>, with the chosen port
written to <tempdir>/bu-<NAME>.port so clients can find the daemon.
Uses asyncio.start_server (stdlib, no obscure APIs, no third-party deps).
Path discipline: log/pid/port files all sit under tempfile.gettempdir() so
they land in /tmp on Linux, $TMPDIR on macOS, %TEMP% on Windows. helpers.py
screenshot() default also moves from /tmp/shot.png to tempfile.gettempdir().
subprocess detach uses start_new_session=True on POSIX and
DETACHED_PROCESS|CREATE_NEW_PROCESS_GROUP on Windows via ipc.spawn_kwargs().
run.py reconfigures stdout to UTF-8 on Windows so print(page_info()) doesn't
UnicodeEncodeError on the 🟢 marker that helpers prepend to tab titles
(#124 item 4). cp1252 (PowerShell default) can't encode it.
Verified end-to-end on Windows 11 with Chrome remote debugging:
- daemon spawns, allocates port, writes .port file
- goto + page_info + screenshot round-trip through TCP loopback
- restart_daemon cleans up .port and .pid
POSIX path is logically equivalent to the prior code (same AF_UNIX call,
same socket-file semantics, same chmod 0600), routed through ipc.py.
Closes#124 items 1, 2, 4. Item 3 (Chrome 147 user-data-dir) is a separate
concern not addressed here.
Bot review caught two issues:
1. capture_screenshot's max_dim path imports PIL but Pillow wasn't in
pyproject.toml dependencies, so a clean install would ImportError as
soon as the option was used.
2. NamedTemporaryFile + writing to f.name while the handle is open is
not portable on Windows (file lock). Switched the screenshot tests to
tempfile.TemporaryDirectory and folded the three near-identical bodies
into a small _run() helper.
Long agent sessions on 2× displays bust the 2000px-per-side limit some
image-aware LLMs enforce — a 2296×1143 CSS viewport produces a 4592×2286
PNG. Passing max_dim=1800 downscales the file before save (only when the
image actually exceeds max_dim), keeping callers that don't pass it
unchanged.
Covers URL-based search (dates, destination, travellers), traveller widget
JS interaction, child age dropdowns, and price filter usage. Documents that
the date picker is unreliable with coordinate clicks and should be bypassed
via URL parameters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three places still described paywalled post body_html as null/None:
- TL;DR limitations list
- Approach 2 section header
- substack_get_post docstring
All updated to correctly state that body_html is a truncated HTML preview
(not null) for paywalled posts, consistent with the Gotchas section and
empirical verification.
Adds domain-skills/substack/scraping.md with four field-tested approaches
for extracting data from any Substack publication without authentication
or a browser session. All approaches verified live on 2026-04-27.
Approach 1 — /api/v1/posts: paginated post list with title, slug, audience,
wordcount, reactions, and post_id. Supports offset pagination.
Approach 2 — /api/v1/posts/{slug}: full post content. Returns complete
body_html (~40KB) for free posts; truncated HTML preview for paywalled posts.
Use audience == "everyone" as the reliable signal for full content.
Approach 3 — /api/v1/post/{id}/comments: comment list with author, body,
date, and reaction counts. Uses integer post_id (not slug).
Approach 4 — /feed: lightweight RSS metadata (title/link/pubDate/description)
without JSON parsing overhead.
Covers both URL formats (native subdomain and custom domain), pagination,
paywalled post handling, and gotchas: reactions is an emoji-keyed dict not
an integer, comments endpoint uses post_id not slug, body_html is a truncated
preview (not null) for paid posts, no unauthenticated cross-publication search.
Co-Authored-By: Tianye Song <songtianye1997@gmail.com>
BU_CDP_URL takes a Chrome DevTools HTTP endpoint (e.g.
http://127.0.0.1:9333) and resolves it to the WS URL via /json/version,
mirroring how start_remote_daemon already handles cloud browsers.
The motivating use case is running a dedicated automation Chrome on a
non-default --user-data-dir to avoid both the Chrome 136 default-profile
lockdown and the Chrome 144+ "Allow remote debugging" per-connection
consent dialog. Pointing BU_CDP_URL at that instance lets the harness
attach without prompting the user.
Falls back to the existing default-profile DevToolsActivePort discovery
when neither BU_CDP_WS nor BU_CDP_URL is set, so existing setups are
unaffected.
Running `browser-harness -c` without a code argument crashed with an
unhandled IndexError at `exec(args[1])`. Added a length check to
produce a proper usage message instead.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a domain skill for enumerating Loom library folders
(loom.com/looms/videos/<slug>-<id>) — the private workspace variant
that requires an authenticated session.
yt-dlp's existing LoomFolderIE covers public-shared folders
(loom.com/share/folder/<id>) but the underlying /v1/folders/<id>
endpoint returns Forbidden for library IDs even with cookies.
That makes browser-harness with the user's open Chrome tab the
only practical programmatic route for private workspace content.
Field-tested on a 78-video folder; documents the data-videoid
selector, the scrollIntoView mechanic that beats scrollTop's
silent cap, dead-end endpoints, and the pipe-to-yt-dlp setup.
current_tab() and list_tabs() both return dicts shaped {"targetId": ...,
"url": ..., "title": ...}, but switch_tab() only accepted the bare string.
This made the natural pattern
original = current_tab()
new_tab(...)
switch_tab(original)
raise InvalidParameters from the CDP layer ("string value expected at
position 17"). Callers had to remember to pull ["targetId"] out manually,
which contradicts the harness's "obvious shapes work" ergonomic.
Make switch_tab tolerant of either shape: pull targetId from the dict if
one is passed, otherwise treat the input as the id string.
* fix(js): don't double-wrap IIFEs that contain return
The substring check `"return " in expression` incorrectly matched
expressions that already contained an IIFE with an internal return,
causing double-wrapping and a silent None result. Guard with
`not expression.strip().startswith("(")` so pre-wrapped expressions
pass through unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(cli): add --reload flag to restart the daemon
Kills the running daemon so the next call picks up code changes,
replacing the manual pkill workflow after editing helpers.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The substring check `"return " in expression` incorrectly matched
expressions that already contained an IIFE with an internal return,
causing double-wrapping and a silent None result. Guard with
`not expression.strip().startswith("(")` so pre-wrapped expressions
pass through unchanged.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
When falling through to the browser path for watch-page DOM (instead of
the http_get + ytInitialPlayerResponse blob), wait_for_load() is not
enough. The load event fires before YouTube's Polymer components
hydrate — h1.ytd-watch-metadata yt-formatted-string,
ytd-video-owner-renderer #channel-name a, and ytd-watch-info-text all
return null for ~2s after load. A wait(3) after wait_for_load() is
required before querying any watch-page selector.
Field-tested 2026-04-24 on Brave; same behavior observed on
ungoogled-chromium. The HTTP path remains the recommended approach for
metadata; this note exists for the cases that genuinely need the
rendered DOM (live UI state, etc.).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Markdown-style domain skill for Polymarket scraping via Gamma API
(api-first per repo doctrine) with DOM leaf-div-disambiguation fallback
for CSS-module SPAs. Covers market outcomes, metadata, and comments.
Live-tested against gamma-api.polymarket.com and a live event page:
- 9 outcomes extracted (e.g. April 7: YES 99.95 / NO 0.05, vol $45.7M)
- Metadata: title, end_date, total_volume, category, market_count
- 38 comments fetched (40 raw, 2 deleted skipped)
Gotcha documented: Gamma API comment envelopes for deleted comments
preserve id/createdAt/profile/media/parentCommentID but drop the body
field entirely — naive dict access throws KeyError. Guard with
'if "body" not in c: continue'.
DOM fallback pattern documented (not primary path): Polymarket has
zero data-testid attributes and CSS-module-hashed classes. Leaf-div
disambiguation (children.length === 0 + nearest-common-ancestor
grouping) is the only robust approach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Alezander9