Bot review caught two issues:
1. capture_screenshot's max_dim path imports PIL but Pillow wasn't in
pyproject.toml dependencies, so a clean install would ImportError as
soon as the option was used.
2. NamedTemporaryFile + writing to f.name while the handle is open is
not portable on Windows (file lock). Switched the screenshot tests to
tempfile.TemporaryDirectory and folded the three near-identical bodies
into a small _run() helper.
Long agent sessions on 2× displays bust the 2000px-per-side limit some
image-aware LLMs enforce — a 2296×1143 CSS viewport produces a 4592×2286
PNG. Passing max_dim=1800 downscales the file before save (only when the
image actually exceeds max_dim), keeping callers that don't pass it
unchanged.
Covers URL-based search (dates, destination, travellers), traveller widget
JS interaction, child age dropdowns, and price filter usage. Documents that
the date picker is unreliable with coordinate clicks and should be bypassed
via URL parameters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three places still described paywalled post body_html as null/None:
- TL;DR limitations list
- Approach 2 section header
- substack_get_post docstring
All updated to correctly state that body_html is a truncated HTML preview
(not null) for paywalled posts, consistent with the Gotchas section and
empirical verification.
Adds domain-skills/substack/scraping.md with four field-tested approaches
for extracting data from any Substack publication without authentication
or a browser session. All approaches verified live on 2026-04-27.
Approach 1 — /api/v1/posts: paginated post list with title, slug, audience,
wordcount, reactions, and post_id. Supports offset pagination.
Approach 2 — /api/v1/posts/{slug}: full post content. Returns complete
body_html (~40KB) for free posts; truncated HTML preview for paywalled posts.
Use audience == "everyone" as the reliable signal for full content.
Approach 3 — /api/v1/post/{id}/comments: comment list with author, body,
date, and reaction counts. Uses integer post_id (not slug).
Approach 4 — /feed: lightweight RSS metadata (title/link/pubDate/description)
without JSON parsing overhead.
Covers both URL formats (native subdomain and custom domain), pagination,
paywalled post handling, and gotchas: reactions is an emoji-keyed dict not
an integer, comments endpoint uses post_id not slug, body_html is a truncated
preview (not null) for paid posts, no unauthenticated cross-publication search.
Co-Authored-By: Tianye Song <songtianye1997@gmail.com>
BU_CDP_URL takes a Chrome DevTools HTTP endpoint (e.g.
http://127.0.0.1:9333) and resolves it to the WS URL via /json/version,
mirroring how start_remote_daemon already handles cloud browsers.
The motivating use case is running a dedicated automation Chrome on a
non-default --user-data-dir to avoid both the Chrome 136 default-profile
lockdown and the Chrome 144+ "Allow remote debugging" per-connection
consent dialog. Pointing BU_CDP_URL at that instance lets the harness
attach without prompting the user.
Falls back to the existing default-profile DevToolsActivePort discovery
when neither BU_CDP_WS nor BU_CDP_URL is set, so existing setups are
unaffected.
Running `browser-harness -c` without a code argument crashed with an
unhandled IndexError at `exec(args[1])`. Added a length check to
produce a proper usage message instead.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a domain skill for enumerating Loom library folders
(loom.com/looms/videos/<slug>-<id>) — the private workspace variant
that requires an authenticated session.
yt-dlp's existing LoomFolderIE covers public-shared folders
(loom.com/share/folder/<id>) but the underlying /v1/folders/<id>
endpoint returns Forbidden for library IDs even with cookies.
That makes browser-harness with the user's open Chrome tab the
only practical programmatic route for private workspace content.
Field-tested on a 78-video folder; documents the data-videoid
selector, the scrollIntoView mechanic that beats scrollTop's
silent cap, dead-end endpoints, and the pipe-to-yt-dlp setup.
current_tab() and list_tabs() both return dicts shaped {"targetId": ...,
"url": ..., "title": ...}, but switch_tab() only accepted the bare string.
This made the natural pattern
original = current_tab()
new_tab(...)
switch_tab(original)
raise InvalidParameters from the CDP layer ("string value expected at
position 17"). Callers had to remember to pull ["targetId"] out manually,
which contradicts the harness's "obvious shapes work" ergonomic.
Make switch_tab tolerant of either shape: pull targetId from the dict if
one is passed, otherwise treat the input as the id string.
* fix(js): don't double-wrap IIFEs that contain return
The substring check `"return " in expression` incorrectly matched
expressions that already contained an IIFE with an internal return,
causing double-wrapping and a silent None result. Guard with
`not expression.strip().startswith("(")` so pre-wrapped expressions
pass through unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* feat(cli): add --reload flag to restart the daemon
Kills the running daemon so the next call picks up code changes,
replacing the manual pkill workflow after editing helpers.py.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
The substring check `"return " in expression` incorrectly matched
expressions that already contained an IIFE with an internal return,
causing double-wrapping and a silent None result. Guard with
`not expression.strip().startswith("(")` so pre-wrapped expressions
pass through unchanged.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
When falling through to the browser path for watch-page DOM (instead of
the http_get + ytInitialPlayerResponse blob), wait_for_load() is not
enough. The load event fires before YouTube's Polymer components
hydrate — h1.ytd-watch-metadata yt-formatted-string,
ytd-video-owner-renderer #channel-name a, and ytd-watch-info-text all
return null for ~2s after load. A wait(3) after wait_for_load() is
required before querying any watch-page selector.
Field-tested 2026-04-24 on Brave; same behavior observed on
ungoogled-chromium. The HTTP path remains the recommended approach for
metadata; this note exists for the cases that genuinely need the
rendered DOM (live UI state, etc.).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Markdown-style domain skill for Polymarket scraping via Gamma API
(api-first per repo doctrine) with DOM leaf-div-disambiguation fallback
for CSS-module SPAs. Covers market outcomes, metadata, and comments.
Live-tested against gamma-api.polymarket.com and a live event page:
- 9 outcomes extracted (e.g. April 7: YES 99.95 / NO 0.05, vol $45.7M)
- Metadata: title, end_date, total_volume, category, market_count
- 38 comments fetched (40 raw, 2 deleted skipped)
Gotcha documented: Gamma API comment envelopes for deleted comments
preserve id/createdAt/profile/media/parentCommentID but drop the body
field entirely — naive dict access throws KeyError. Guard with
'if "body" not in c: continue'.
DOM fallback pattern documented (not primary path): Polymarket has
zero data-testid attributes and CSS-module-hashed classes. Leaf-div
disambiguation (children.length === 0 + nearest-common-ancestor
grouping) is the only robust approach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- manus/tasks.md: ProseMirror composer can mount late; wrap the
getBoundingClientRect lookup in a bounded retry + assertion so the
helper doesn't null-deref and block submission.
- perplexity/computer.md: id extraction must parse the URL pathname
before slicing — raw-href slicing corrupts when the URL carries
?view=thread or a hash fragment.
- perplexity/computer.md: all_todo_done() returned false on hydrate
while the doc claimed None; unify on None so "can't tell" states
(closed panel, empty panel) are distinguishable from "done=false".
When BH_DEBUG_CLICKS=1 (or --debug-clicks CLI flag), click_at_xy
captures a screenshot before each click and draws a red crosshair
at the exact click location, scaled by window.devicePixelRatio so
the marker aligns correctly on HiDPI/Retina displays.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
* fix(js): auto-wrap top-level return expressions in an IIFE
Agents naturally write js() calls with top-level `return`, which is a
syntax error in Runtime.evaluate. js() now detects `return ` and wraps
the expression in an IIFE so both styles work without callers needing
to know the CDP constraint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor(run): replace heredoc stdin with -c flag
Drops the <<'PY' heredoc pattern entirely. Agents now pass code via
`browser-harness -c "..."`, which is simpler to construct programmatically
and avoids heredoc quoting pitfalls.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Agents naturally write js() calls with top-level `return`, which is a
syntax error in Runtime.evaluate. js() now detects `return ` and wraps
the expression in an IIFE so both styles work without callers needing
to know the CDP constraint.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
SKILL.md now covers day-to-day usage only. Maintenance commands
(--doctor, --setup, --update) and the architecture section move to
install.md where setup and break-fix content belongs.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
manus/tasks.md
- Correct the Connect RPC transport description: the JS client wraps the
JSON payload in a Uint8Array before fetch, so a naive fetch-hook logs
it as <<bytes:N>> — the wire content matches content-type (JSON both
ways), not a separate binary codec.
- Replace the raw click(750, 505) example with a computed rect from
div.ProseMirror[contenteditable="true"] (SKILL.md: no pixel coords).
- Drop the specific-task prompt example ('reply with just the word PONG'
→ 'Reply with PONG') for a generic description of the auto-summary.
- Remove a real taskId from the URL-pattern docstring.
perplexity/computer.md
- Scope Todo-panel queries to the Radix popover. The Todo button has
data-state / aria-expanded / aria-controls; the open panel is mounted
with id === button.aria-controls. A global 'svg use' query across the
whole document picks up tool-invocation icons, sidebar icons, and top-
bar icons — scope-less completion detection is unreliable.
- Rewrite all_todo_done() to return None when the panel is closed or
hydrating and True only when every row's icon is #pplx-icon-check.
- Remove a real slug and taskId from the URL-pattern docstring.
- goto() → goto_url()
- click() → click_at_xy()
- screenshot() → capture_screenshot()
These three shared exact names with Playwright but different argument
shapes, causing agent confusion. Updated helpers.py and all markdown
skill files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers open-compose shortcut, multiple-dialog stacking, the Tab-inserts-literal-tab trap, attachments via DOM.setFileInputFiles on the visible dialog's input, and stable selectors for To/Subject/Body/Send.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small fixes in daemon.py for macOS users running Brave (or any
Chromium variant launched with --remote-debugging-port):
1. PROFILES list was missing the macOS path for Brave
(~/Library/Application Support/BraveSoftware/Brave-Browser).
The Linux Flatpak path was the only Brave entry, so macOS Brave
users got "DevToolsActivePort not found" even after enabling
remote debugging.
2. When Chromium is launched with an explicit --remote-debugging-port
flag, the DevToolsActivePort file is not always written to the
profile dir. CDP is fully live, but get_ws_url() can't find it.
Added a final fallback that probes 127.0.0.1:9222 and :9223 via
/json/version and uses the returned webSocketDebuggerUrl. The
loop is gated to the standard debugging ports and only runs after
the existing profile-dir scan fails, so it doesn't interfere with
the normal sticky-checkbox flow.
Both changes are additive — no existing behavior is altered.
Two new domain skills for amazon.com derived from a logged-in session:
- cart.md: documents the empty-cart trap where `[data-asin]` selectors
match recommendation widgets even on an empty cart, plus the stable
subtotal selectors and the price-change banner format.
- orders.md: covers the order-list card structure, the order-search
quirk (the `?search=` URL param doesn't filter — must submit the
form or hit the post-submit `/your-orders/search/` URL), and the
tracking page (whose `data-test-id` selectors are all null — extract
from `body.innerText` anchored on the "Arriving" line).
* fix(run): pass globals to exec so comprehensions resolve free vars
exec(sys.stdin.read()) inside main() passes different dicts for globals
and locals. Python comprehensions and generator expressions compile to a
nested function whose free-variable lookups can only see the globals
dict, so code like
for it in items:
low = it['text'].lower()
hit = any(k in low for k in KEYS) # NameError: 'low'
fails at module level under exec. `low` is stored in the exec locals
dict which the generator's implicit function cannot see.
Passing globals() as the only extra arg makes exec use the same dict
for both globals and locals, and comprehensions resolve cleanly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* docs(skill): eyeball click coords from screenshots; no getBoundingClientRect
The previous Clicking bullet said "screenshot -> look -> click(x,y)" but
did not rule out roundtripping through js("...getBoundingClientRect()")
to compute coords. Agents coming from Playwright / Selenium habits tend
to locate-first-click-second even when the screenshot already shows the
target, which is slower and more brittle (hidden inputs placed at
x=-9999, CSS-transformed elements, pseudo-elements) than just reading
the pixel off the image.
Rewrite the bullet to explicitly suppress that reflex and scope the
DOM-fallback to elements with no visible geometry. Also replace the
loose "compositor level" phrasing with the actual mechanism: Chrome's
browser-process hit-testing, which is why clicks pass through iframes
/ shadow DOM / cross-origin without extra work.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reuses the existing `_open_chrome_inspect()` helper so a bare
`browser-harness <<PY ... PY` invocation recovers from the three
cold-start failures that previously forced a manual restart_daemon()
or a separate `browser-harness --setup` run:
1. Stale daemon — previous run's Chrome was killed but the daemon
still listens on the unix socket. Probe with a real CDP call
(`Target.getTargets`) and require "result" in the reply. A
`{"meta":"session"}` probe would not work because the daemon
answers meta requests from a cached Python dict even when its
CDP WebSocket to Chrome is dead.
2. Cold Chrome — log tail says "DevToolsActivePort not found" or
"not live yet". `_open_chrome_inspect()` uses AppleScript
`activate` which launches Chrome if absent, then opens the
inspect page.
3. Missing Allow on chrome://inspect — log tail says
"WS handshake failed ... 403". Same recovery: open the inspect
page, print one stderr hint ("click Allow..."), retry spawn once.
Net: +37/-22 lines, one function touched, no new helpers.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Saurav Panda