PR #229 (src layout refactor) updated SKILL.md and run.py to the -c form
when the CLI dropped stdin support, but missed install.md (steps 2 and 7)
and interaction-skills/profile-sync.md. Sync them up.
Picks up the still-relevant hunks from #215 (closes#215, refs #213).
Co-authored-by: Andres Gonzalez <62394570+FVTVLIX@users.noreply.github.com>
Rebased from #170 (authored by @mvanhorn) onto current main.
The original branch was based on a stale README; only the two
intentional lines from that PR are reapplied here.
When the browser dies but the daemon process keeps running, target_id and session stay cached on the Bridge. Pre-fix, connection_status swallowed the Target.getTargetInfo failure and returned the cached IDs with page=null, so admin counted the daemon as healthy. Now it returns {error: cdp_disconnected} and admin's existing error-check skips it. Also returns {error: not_attached} when target_id is unset.
Three patterns discovered while populating the Shopify Knowledge Base App
for JING (silent training football brand). Each pattern saved 30+ minutes
of rediscovery; documenting so future agents on any Shopify store benefit.
- README.md: when to use the harness for Shopify admin vs Storefront API
/ Admin API / Shopify CLI; auth notes; Polaris React vs Web Components
detection.
- embedded-apps.md: every Shopify app surfaces in an iframe at
*-app.shopify.prod.shopifyapps.com. Use iframe_target() and pass
target_id to js() calls. Known iframe slugs table (extend as
discovered).
- polaris-inputs.md: Polaris React text inputs reject the standard
React-friendly synthetic value setter pattern — fields fill but Save
button stays disabled. Working pattern is .focus() via JS + CDP-native
keystrokes via type_text(). Includes full add-FAQ implementation +
notes on Polaris's web-component migration (polaris-react archived
Jan 2026).
- knowledge-base.md: canonical automation for the Shopify Knowledge
Base App (Winter '26 Edition). FAQ-pair flow, batch loop pattern,
/pairs/<id> URL signal for save success, brand-voice guidance,
query-log review cadence. App is currently English-only with no
public API — browser automation is the only path until Shopify
exposes endpoints.
Tested with 19 successful FAQ saves on the JING store on 2026-04-29.
Co-authored-by: Claude (JING build agent) <build@playjing.local>
When BH_TMP_DIR is set, the caller owns the directory and is expected to
give each daemon its own — so the bu-<NAME> filename prefix is redundant.
Dropping it cuts ~30 chars off the AF_UNIX path on POSIX, which matters
because sun_path is 104 bytes on macOS / 108 on Linux and a long BU_NAME
(e.g. an opencode-style session id) can blow past it.
Concretely the failing path on macOS for a 30-char session id under
~/.local/share/bcode/sessions/<sessionID>/bu-<sessionID>.sock was 117
bytes; with this change it becomes ~83 bytes.
When BH_TMP_DIR is unset, _TMP is the shared default (/tmp on POSIX,
gettempdir() on Windows) and the bu-<NAME> prefix is kept to disambiguate
multiple daemons sharing the dir. Backward compatible by default.
admin._daemon_endpoint_names: when BH_TMP_DIR is set, the dir is per-daemon
by construction, so glob discovery doesn't apply — just check whether our
local endpoint exists. Without BH_TMP_DIR, fall back to the existing
shared-dir glob.
Validation (_check on BU_NAME) still runs in both modes to catch garbage
names early.
Without this, a caller passing BH_TMP_DIR=<custom dir> to a directory that
doesn't exist yet causes the first write (sock/port/pid/log/screenshot) to
fail with FileNotFoundError. Default _TMP (/tmp on POSIX, gettempdir() on
Windows) always exists, so this is latent today; it bites the per-session
scratch-dir use case browsercode is about to introduce.
Single root cause -> single fix: cover screenshots, sock, port, pid, and
log paths uniformly.
DETACHED_PROCESS overrides CREATE_NO_WINDOW per Win32 docs, so combining
them caused Windows to allocate a fresh console for the daemon. Closing
that window killed the daemon and forced Chrome to re-prompt for remote
debugging permission. Drop DETACHED_PROCESS, keep CREATE_NEW_PROCESS_GROUP
for terminal-close survival.
Lets callers (e.g. browsercode per-session scratch) redirect all harness
file output via one env var. Default behavior unchanged when unset:
/tmp on POSIX, gettempdir() on Windows.
helpers.capture_screenshot and the debug-click overlay now route through
ipc._TMP so the same knob covers screenshots.
On Windows, `os.kill(pid, 0)` does not behave like its POSIX
counterpart. Instead of returning silently when the process
exists or raising ProcessLookupError when it doesn't, CPython
on Windows raises:
SystemError: <built-in function kill> returned a result
with an exception set
This happens because the underlying Win32 TerminateProcess API
does not accept signal 0 as an "is alive?" probe — Python's C
implementation hits an internal error path that doesn't set a
proper exception, and the interpreter surfaces SystemError.
`restart_daemon()` already catches `(ProcessLookupError, OSError)`
around both `os.kill(pid, 0)` and `os.kill(pid, signal.SIGTERM)`,
but SystemError isn't a subclass of either, so the harness crashes
on every second invocation on Windows: the daemon's stale pid file
points at a no-longer-running pid, the probe raises SystemError,
and the whole `browser-harness -c '...'` call dies with a stack
trace.
Repro on Windows 11 (Python 3.12, browser-harness 0.1.0):
browser-harness --reload # ok
browser-harness -c 'print(page_info())' # ok
browser-harness -c 'print(page_info())' # CRASH
Fix: add SystemError to the existing except clauses in
restart_daemon(). One-word change in two places, matches the
intent of the existing handlers (treat any "couldn't probe/signal
the pid" failure as "process is gone, move on").
The stale-daemon probe in ensure_daemon used raw socket.AF_UNIX, which:
1. Doesn't exist on uv-bundled Python on Windows (AttributeError).
2. Would point at a TCP display string '127.0.0.1:<port>', not a socket
path, even if AF_UNIX existed.
Either way the probe always raised, was swallowed by 'except Exception',
and fell through to restart_daemon — killing and respawning the daemon
on every warm call. Symptom is most visible on Windows where the warm
path is the common case.
Fix: use ipc.connect(name) which already does the right thing per
platform (AF_UNIX on POSIX, TCP loopback on Windows). Same call
daemon_alive and restart_daemon already use.
Also drop the now-unused _paths() helper; restart_daemon only needed
the pid_path half, inline that.
When remote-debugging is not enabled, `run_setup` opens chrome://inspect
once, then retries `ensure_daemon` in a loop. But `ensure_daemon` itself
also calls `_open_chrome_inspect()` on failure — opening a new tab every
~7 seconds and flooding the browser.
Add `_open_inspect` parameter to `ensure_daemon` so the retry loop in
`run_setup` can suppress redundant tab opens after the first one.
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Saurav Panda <sgp65@cornell.edu>
Add browser_connections() / active_browser_connections() to admin, which probe
each daemon socket via a new connection_status meta message. The daemon tracks
target_id (set on attach and on set_session) and returns the live page title/URL
via Target.getTargetInfo. switch_tab now forwards target_id in set_session so the
daemon stays in sync after tab switches. run_doctor shows the count, each daemon
name, and the truncated active-page title and URL.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace naive `"return " in expression` check with a character-level parser that
ignores strings, line comments, and block comments. Add `_decode_unserializable_js_value`
to handle NaN, ±Infinity, -0, and BigInt. Extract `_runtime_value` and
`_runtime_evaluate` to unify error handling across `js()` and `page_info()`.
Wrap Runtime.evaluate TimeoutErrors in RuntimeError with expression context.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Previously js() silently returned None when Chrome reported a syntax or
runtime error. Now it raises RuntimeError with the error description,
line/column location, and a snippet of the failing expression.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(tests): reorganize into tests/unit and tests/integration
Moves all root-level test_*.py files into a structured tests/ directory:
- tests/unit/ — admin, helpers (was test_screenshot), run
- tests/integration/ — js expression tests
- tests/conftest.py — shared fake_png pytest fixture, eliminating duplication
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor: move to src layout, agent-workspace, and fix SKILL.md invocation format
- Move package to src/browser_harness/ and domain-skills/interaction-skills to agent-workspace/
- Fix all browser-harness <<'PY' heredoc examples in SKILL.md and run.py HELP string to use the correct -c '...' flag format (heredoc was never supported by the CLI)
- Update SKILL.md path references from domain-skills/ to agent-workspace/domain-skills/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
tempfile.gettempdir() on macOS returns /var/folders/xx/yy.../T/ (~49 chars).
Combined with bu-{64-char-name}.sock that exceeds the 104-byte sun_path
limit on macOS (108 on Linux), causing daemon startup to fail.
Pre-PR upstream hardcoded /tmp; this restores that for POSIX. Windows is
unaffected (uses TCP, not AF_UNIX).
* fix(daemon): fire-and-forget mark-title eval so load events don't stall
* fix(daemon): bound mark-title eval to 2s so stalled V8 can't pile up tasks
Fire-and-forget kept the perf win, but dropped the 2s upper bound. On a
discarded renderer or hard-hung V8, the Runtime.evaluate task would
never resolve, leaking one task per navigation. Restore the timeout
inside the _silent wrapper.
---------
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com>
Co-authored-by: Saurav Panda <sgp65@cornell.edu>
Cut ipc.py from 127 to 68 lines (-46%) by removing restated docstrings and
keeping only load-bearing inline comments (path-traversal guard, uv-Python
AF_UNIX gating, Windows .port-file role). Same logic, same call sites.
Rename ipc -> _ipc per Python convention for internal modules. The IPC
plumbing is only called by daemon/admin/helpers; agents reading helpers.py
should not be pulled into transport details. Callers do 'import _ipc as ipc'
so internal ipc.foo references stay unchanged.
- daemon.py:1, SKILL.md:151 — docstring/diagram said named pipe on Windows;
the implementation is TCP loopback. Updated.
- daemon.py:209 — if ipc.serve() crashes (e.g. bind failure), the prior
shutdown path could miss it and leave the daemon waiting forever on
d.stop without a listening endpoint. Now race serve_task and stop.wait()
via asyncio.wait(FIRST_COMPLETED): if serve finishes first it must have
raised, so await it to surface the exception. Cleanup cancels both tasks
unconditionally.
- ipc.py:45 — BU_NAME flowed straight into f-strings building filesystem
paths, allowing path traversal outside tempdir. Validate via
^[A-Za-z0-9_-]{1,64}$ in a single _check() helper that all path
builders (log_path/pid_path/port_path/_sock_path) call. Bad names raise
ValueError early with a clear message.
The harness was Linux/macOS-only because daemon IPC hardcoded AF_UNIX sockets
at /tmp/bu-*.sock paths and asyncio.start_unix_server, all of which are
unavailable or invalid on Windows. Worse, uv-managed Python on Windows
(python-build-standalone) ships without socket.AF_UNIX entirely (#124).
New ipc.py centralizes the platform fork:
- POSIX: AF_UNIX socket at <tempdir>/bu-<NAME>.sock (chmod 0600), unchanged
semantics from the prior /tmp-hardcoded path.
- Windows: TCP loopback on 127.0.0.1:<ephemeral>, with the chosen port
written to <tempdir>/bu-<NAME>.port so clients can find the daemon.
Uses asyncio.start_server (stdlib, no obscure APIs, no third-party deps).
Path discipline: log/pid/port files all sit under tempfile.gettempdir() so
they land in /tmp on Linux, $TMPDIR on macOS, %TEMP% on Windows. helpers.py
screenshot() default also moves from /tmp/shot.png to tempfile.gettempdir().
subprocess detach uses start_new_session=True on POSIX and
DETACHED_PROCESS|CREATE_NEW_PROCESS_GROUP on Windows via ipc.spawn_kwargs().
run.py reconfigures stdout to UTF-8 on Windows so print(page_info()) doesn't
UnicodeEncodeError on the 🟢 marker that helpers prepend to tab titles
(#124 item 4). cp1252 (PowerShell default) can't encode it.
Verified end-to-end on Windows 11 with Chrome remote debugging:
- daemon spawns, allocates port, writes .port file
- goto + page_info + screenshot round-trip through TCP loopback
- restart_daemon cleans up .port and .pid
POSIX path is logically equivalent to the prior code (same AF_UNIX call,
same socket-file semantics, same chmod 0600), routed through ipc.py.
Closes#124 items 1, 2, 4. Item 3 (Chrome 147 user-data-dir) is a separate
concern not addressed here.
Bot review caught two issues:
1. capture_screenshot's max_dim path imports PIL but Pillow wasn't in
pyproject.toml dependencies, so a clean install would ImportError as
soon as the option was used.
2. NamedTemporaryFile + writing to f.name while the handle is open is
not portable on Windows (file lock). Switched the screenshot tests to
tempfile.TemporaryDirectory and folded the three near-identical bodies
into a small _run() helper.
Long agent sessions on 2× displays bust the 2000px-per-side limit some
image-aware LLMs enforce — a 2296×1143 CSS viewport produces a 4592×2286
PNG. Passing max_dim=1800 downscales the file before save (only when the
image actually exceeds max_dim), keeping callers that don't pass it
unchanged.
Covers URL-based search (dates, destination, travellers), traveller widget
JS interaction, child age dropdowns, and price filter usage. Documents that
the date picker is unreliable with coordinate clicks and should be bypassed
via URL parameters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three places still described paywalled post body_html as null/None:
- TL;DR limitations list
- Approach 2 section header
- substack_get_post docstring
All updated to correctly state that body_html is a truncated HTML preview
(not null) for paywalled posts, consistent with the Gotchas section and
empirical verification.
Adds domain-skills/substack/scraping.md with four field-tested approaches
for extracting data from any Substack publication without authentication
or a browser session. All approaches verified live on 2026-04-27.
Approach 1 — /api/v1/posts: paginated post list with title, slug, audience,
wordcount, reactions, and post_id. Supports offset pagination.
Approach 2 — /api/v1/posts/{slug}: full post content. Returns complete
body_html (~40KB) for free posts; truncated HTML preview for paywalled posts.
Use audience == "everyone" as the reliable signal for full content.
Approach 3 — /api/v1/post/{id}/comments: comment list with author, body,
date, and reaction counts. Uses integer post_id (not slug).
Approach 4 — /feed: lightweight RSS metadata (title/link/pubDate/description)
without JSON parsing overhead.
Covers both URL formats (native subdomain and custom domain), pagination,
paywalled post handling, and gotchas: reactions is an emoji-keyed dict not
an integer, comments endpoint uses post_id not slug, body_html is a truncated
preview (not null) for paid posts, no unauthenticated cross-publication search.
Co-Authored-By: Tianye Song <songtianye1997@gmail.com>
BU_CDP_URL takes a Chrome DevTools HTTP endpoint (e.g.
http://127.0.0.1:9333) and resolves it to the WS URL via /json/version,
mirroring how start_remote_daemon already handles cloud browsers.
The motivating use case is running a dedicated automation Chrome on a
non-default --user-data-dir to avoid both the Chrome 136 default-profile
lockdown and the Chrome 144+ "Allow remote debugging" per-connection
consent dialog. Pointing BU_CDP_URL at that instance lets the harness
attach without prompting the user.
Falls back to the existing default-profile DevToolsActivePort discovery
when neither BU_CDP_WS nor BU_CDP_URL is set, so existing setups are
unaffected.
Alezander9