PR #247 added these under domain-skills/ at the repo root, but the
canonical location for domain skills is agent-workspace/domain-skills/
(referenced in README.md, SKILL.md, and helpers.py).
Three patterns discovered while populating the Shopify Knowledge Base App
for JING (silent training football brand). Each pattern saved 30+ minutes
of rediscovery; documenting so future agents on any Shopify store benefit.
- README.md: when to use the harness for Shopify admin vs Storefront API
/ Admin API / Shopify CLI; auth notes; Polaris React vs Web Components
detection.
- embedded-apps.md: every Shopify app surfaces in an iframe at
*-app.shopify.prod.shopifyapps.com. Use iframe_target() and pass
target_id to js() calls. Known iframe slugs table (extend as
discovered).
- polaris-inputs.md: Polaris React text inputs reject the standard
React-friendly synthetic value setter pattern — fields fill but Save
button stays disabled. Working pattern is .focus() via JS + CDP-native
keystrokes via type_text(). Includes full add-FAQ implementation +
notes on Polaris's web-component migration (polaris-react archived
Jan 2026).
- knowledge-base.md: canonical automation for the Shopify Knowledge
Base App (Winter '26 Edition). FAQ-pair flow, batch loop pattern,
/pairs/<id> URL signal for save success, brand-voice guidance,
query-log review cadence. App is currently English-only with no
public API — browser automation is the only path until Shopify
exposes endpoints.
Tested with 19 successful FAQ saves on the JING store on 2026-04-29.
Co-authored-by: Claude (JING build agent) <build@playjing.local>
* refactor(tests): reorganize into tests/unit and tests/integration
Moves all root-level test_*.py files into a structured tests/ directory:
- tests/unit/ — admin, helpers (was test_screenshot), run
- tests/integration/ — js expression tests
- tests/conftest.py — shared fake_png pytest fixture, eliminating duplication
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* refactor: move to src layout, agent-workspace, and fix SKILL.md invocation format
- Move package to src/browser_harness/ and domain-skills/interaction-skills to agent-workspace/
- Fix all browser-harness <<'PY' heredoc examples in SKILL.md and run.py HELP string to use the correct -c '...' flag format (heredoc was never supported by the CLI)
- Update SKILL.md path references from domain-skills/ to agent-workspace/domain-skills/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers URL-based search (dates, destination, travellers), traveller widget
JS interaction, child age dropdowns, and price filter usage. Documents that
the date picker is unreliable with coordinate clicks and should be bypassed
via URL parameters.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Three places still described paywalled post body_html as null/None:
- TL;DR limitations list
- Approach 2 section header
- substack_get_post docstring
All updated to correctly state that body_html is a truncated HTML preview
(not null) for paywalled posts, consistent with the Gotchas section and
empirical verification.
Adds domain-skills/substack/scraping.md with four field-tested approaches
for extracting data from any Substack publication without authentication
or a browser session. All approaches verified live on 2026-04-27.
Approach 1 — /api/v1/posts: paginated post list with title, slug, audience,
wordcount, reactions, and post_id. Supports offset pagination.
Approach 2 — /api/v1/posts/{slug}: full post content. Returns complete
body_html (~40KB) for free posts; truncated HTML preview for paywalled posts.
Use audience == "everyone" as the reliable signal for full content.
Approach 3 — /api/v1/post/{id}/comments: comment list with author, body,
date, and reaction counts. Uses integer post_id (not slug).
Approach 4 — /feed: lightweight RSS metadata (title/link/pubDate/description)
without JSON parsing overhead.
Covers both URL formats (native subdomain and custom domain), pagination,
paywalled post handling, and gotchas: reactions is an emoji-keyed dict not
an integer, comments endpoint uses post_id not slug, body_html is a truncated
preview (not null) for paid posts, no unauthenticated cross-publication search.
Co-Authored-By: Tianye Song <songtianye1997@gmail.com>
Adds a domain skill for enumerating Loom library folders
(loom.com/looms/videos/<slug>-<id>) — the private workspace variant
that requires an authenticated session.
yt-dlp's existing LoomFolderIE covers public-shared folders
(loom.com/share/folder/<id>) but the underlying /v1/folders/<id>
endpoint returns Forbidden for library IDs even with cookies.
That makes browser-harness with the user's open Chrome tab the
only practical programmatic route for private workspace content.
Field-tested on a 78-video folder; documents the data-videoid
selector, the scrollIntoView mechanic that beats scrollTop's
silent cap, dead-end endpoints, and the pipe-to-yt-dlp setup.
When falling through to the browser path for watch-page DOM (instead of
the http_get + ytInitialPlayerResponse blob), wait_for_load() is not
enough. The load event fires before YouTube's Polymer components
hydrate — h1.ytd-watch-metadata yt-formatted-string,
ytd-video-owner-renderer #channel-name a, and ytd-watch-info-text all
return null for ~2s after load. A wait(3) after wait_for_load() is
required before querying any watch-page selector.
Field-tested 2026-04-24 on Brave; same behavior observed on
ungoogled-chromium. The HTTP path remains the recommended approach for
metadata; this note exists for the cases that genuinely need the
rendered DOM (live UI state, etc.).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Markdown-style domain skill for Polymarket scraping via Gamma API
(api-first per repo doctrine) with DOM leaf-div-disambiguation fallback
for CSS-module SPAs. Covers market outcomes, metadata, and comments.
Live-tested against gamma-api.polymarket.com and a live event page:
- 9 outcomes extracted (e.g. April 7: YES 99.95 / NO 0.05, vol $45.7M)
- Metadata: title, end_date, total_volume, category, market_count
- 38 comments fetched (40 raw, 2 deleted skipped)
Gotcha documented: Gamma API comment envelopes for deleted comments
preserve id/createdAt/profile/media/parentCommentID but drop the body
field entirely — naive dict access throws KeyError. Guard with
'if "body" not in c: continue'.
DOM fallback pattern documented (not primary path): Polymarket has
zero data-testid attributes and CSS-module-hashed classes. Leaf-div
disambiguation (children.length === 0 + nearest-common-ancestor
grouping) is the only robust approach.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- goto() → goto_url()
- click() → click_at_xy()
- screenshot() → capture_screenshot()
These three shared exact names with Playwright but different argument
shapes, causing agent confusion. Updated helpers.py and all markdown
skill files.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Covers open-compose shortcut, multiple-dialog stacking, the Tab-inserts-literal-tab trap, attachments via DOM.setFileInputFiles on the visible dialog's input, and stable selectors for To/Subject/Body/Send.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two new domain skills for amazon.com derived from a logged-in session:
- cart.md: documents the empty-cart trap where `[data-asin]` selectors
match recommendation widgets even on an empty cart, plus the stable
subtotal selectors and the price-change banner format.
- orders.md: covers the order-list card structure, the order-search
quirk (the `?search=` URL param doesn't filter — must submit the
form or hit the post-submit `/your-orders/search/` URL), and the
tracking page (whose `data-test-id` selectors are all null — extract
from `body.innerText` anchored on the "Arriving" line).
Captures the durable shape of LinkedIn's received-invitations page:
- URL filter slugs (PEOPLE_WITH_MUTUAL_CONNECTION, PEOPLE_WITH_MUTUAL_SCHOOL)
and the chip counter as authoritative remaining-count.
- Accept/Ignore aria-label formats — they differ from each other, so you
cannot derive one from the other.
- The "follows you" trap: Accept renders as <a href=current-URL>, not
<button>; .click() follows href and no click path (MouseEvent, CDP
Input.dispatchMouseEvent) triggers the accept handler. Route these to
Ignore or skip.
- Pagination: list renders ~10 rows, replaced by "is now a connection"
acknowledgments after accepts; reload the URL to fetch the next slice,
scrolling does nothing.
- "Take care when connecting" safety modal appears intermittently.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the previous commit added early raises for LOGIN_REQUIRED and
ERROR in scrape_video(), two doc entries still described the old
behavior ("will succeed but status is LOGIN_REQUIRED"). Updated both
the Gotchas section and the 'What Requires a Browser' list to reflect
that scrape_video() now raises ValueError on age-restricted videos.
Two issues raised by automated review on PR #137:
1. max_results limit could be exceeded
The `break` in youtube_search() only exits the inner loop over
itemSectionRenderer contents. If YouTube returns multiple sections,
the outer loop would continue appending results beyond max_results.
Fixed by replacing `break` with `return results` to exit both loops
immediately once the limit is reached.
2. Regex match not null-checked before .group(1)
scrape_video() called m.group(1) directly after re.search(), which
raises AttributeError if the pattern is not found (e.g. private video,
deleted video, region-blocked content, or YouTube HTML structure change).
Fixed by adding an explicit None check with a descriptive ValueError,
and an early playabilityStatus check that surfaces LOGIN_REQUIRED and
ERROR states with clear messages before attempting to parse videoDetails.
Adds domain-skills/youtube/scraping.md with four verified approaches
for extracting YouTube data without a browser or API key.
## What's included
**Approach 1 — oEmbed API (fastest)**
- Single HTTP call, ~0.3s per video
- Returns title, author, channel URL, thumbnail, embed HTML
- Bulk fetching via ThreadPoolExecutor with real timing data
- Verified on multiple video IDs
**Approach 2 — Watch page ytInitialPlayerResponse**
- Full video metadata: title, author, channel_id, duration, view_count,
publish_date, upload_date, category, like_count, keywords, is_live,
is_private, is_unlisted, available_countries (249 codes), embed_url
- Correct regex pattern (non-greedy with lookahead) to parse the JSON blob
- Real output values verified and included as comments
- Accurate gotchas: viewCount/lengthSeconds are strings not ints,
likeCount lives in microformat not videoDetails
**Approach 3 — Search results (no API key)**
- Parses ytInitialData from /results?search_query= (server-side rendered)
- Returns up to ~14-20 results with videoId, title, channel, duration,
views, published, description snippet, thumbnail URL
- Verified: 15 results returned for "python tutorial"
**Approach 4 — Channel metadata**
- Handles both @handle and channel ID (UC...) URL formats
- Extracts channel_id, title, description, subscriber count, avatar,
banner from pageHeaderViewModel + channelMetadataRenderer
- Verified on @RickAstleyYT: "4.48m subscribers"
**Utilities**
- thumbnail_urls(): all 5 sizes with availability notes (maxres may 404)
- extract_video_id(): handles watch, youtu.be, /shorts/, /embed/ formats
**What requires a browser**
- Clear list of what http_get cannot access: trending, playlists,
comments, caption text, age-restricted videos
**URL patterns reference table**
**Gotchas (all verified)**
- ytInitialPlayerResponse regex non-greedy requirement
- viewCount/lengthSeconds are string types, not int
- likeCount location (microformat, not videoDetails)
- oEmbed 404 on private/deleted videos
- Caption baseUrl returns empty in all tested conditions
(plain http_get, XHR, and fetch with cookies) — not a session issue
- Search result count varies (~14-20), never assume fixed count
- Subscriber count is a rounded string, not an integer
Captures the site shape for centilebrain.org 'Generate Estimates' flow:
- Six Shiny iframes (modality x sex) at centilebrain-app.shinyapps.io
- Selectors for email, file input, compute, download
- Wait markers and three non-obvious traps (iframe target_id staleness,
MUI switch checkboxes, coordinate-vs-JS button click after scroll)
- End-to-end example using upload_file(..., target_id=...) with
iframe_target()
- Output zip schema
Requires the iframe target_id support added to upload_file().
Synthetic .click() on the visible Star button does not persist the
star — there's a hidden 0x0 fallback button that querySelector finds
first, and the visible React button swallows synthetic events. Submit
the form directly; CSRF is already embedded.
Field-tested while completing the install.md verification step.
Both skills share the post-article DOM surface (div[role="article"]
and the data-ad-*-preview message selectors), but differ in URL shape,
sort options, and rate-limit ceilings. Pages are public and tolerate a
higher rate; Groups gate content behind membership and are stricter.
Each file covers: URL patterns, DOM anchors with verification notes,
a collect-as-you-go scroll pattern (FB virtualizes the feed so
scroll-then-collect misses posts), the l.facebook.com/l.php redirector
decoder, a Firecrawl handoff example, rate-limit discipline, a
self-inspection JS block for detecting selector drift, and a full
end-to-end example that emits JSON on stdout for downstream tools.
The groups.md anchors were verified against a logged-in account on
2026-04-18. pages.md inherits the post-article anchors from groups.md
(shared React component) and adds Page-specific header/metadata
selectors; a gotchas log section invites confirmation on first live use.
Documents the Monaco + React-canvas seam that causes most 'automation
silently did nothing' failures in Framer:
- Double-click requires the full pointer+mouse event chain (detail:2 matters)
- Monaco paste must be clipboard + OS-level Cmd+A/Cmd+V, then wait before save
- Publish button only mounts when a page is selected in the Pages tab; it
rejects synthetic clicks and must be driven by screen-coord input
- Framer autolayout preempts programmatic Header position/left/right writes,
forcing nodes offscreen; delete-and-copy is the working workaround
Plus the list of canvas-level interactions that consistently reject
automation (drag-drop, variant switching, property binding, Page Settings)
so agents know to escalate to the human instead of retrying.
Stable data-testid selectors table for Pages/Layers/Assets tabs, and the
edge-cache + sitemap quirks on framer-hosted live domains.
Add routes, filter URL format, GraphQL bootstrap, and the 'credentials: include'
pattern for reusing the tab's NextAuth session cookie without JWE juggling.
Auth quirk documented: injecting only the JWE into a fresh Chrome profile
triggers a login loop; UI needs the full cookie set (persistent profile),
while backend GraphQL works with the JWE alone.
PubMed/NCBI (ESearch→ESummary/EFetch pipeline; count is string; ELink broken 2026; CollectiveName branch),
CrossRef (title/container-title always lists; abstract has JATS XML tags; type=proceedings-article not conference-paper),
OpenAlex (abstract_inverted_index reconstruction; cursor >10K pages; group_by not group-by; concepts deprecated use topics),
FRED (fredgraph.csv timeouts headlessly; API needs free key; BLS/WorldBank/AlphaVantage as keyless alternatives),
MusicBrainz (Mozilla/5.0 gets 403; recording length in ms; CAA front flag vs types array differ).
CoinMarketCap (internal data-api/v3 fully open, no auth; 25 calls no rate limit),
Quora (full Chrome UA required; push() payloads double-encoded JSON; 3 SSR answers only),
Itch.io (http_get works; game cards via CSS selectors; RSS feeds exist),
Steam (appdetails single appid only; price in cents; ISteamApps/GetAppList dead in 2026),
HowLongToBeat (two-step token flow /api/find/init then POST; comp_* in seconds not hours).
Letterboxd (http_get works on film pages; JSON-LD CDATA gotcha; API needs OAuth),
Gutenberg (Gutendex REST API; text via /cache/epub/; .opf is 404, use .rdf),
Metacritic (internal backend API key in HTML; Nuxt __NUXT_DATA__ not __NEXT_DATA__),
RAWG (API needs key; window.CLIENT_PARAMS in HTML has full game data without key),
OpenLibrary (full free API; missing cover = 43-byte GIF not 404; description dual type).
Glassdoor (Cloudflare managed challenge; browser only; __NEXT_DATA__ + DOM fallbacks),
Medium (?format=json strips XSSI prefix; GraphQL /_/graphql no auth; RSS 10-item cap),
SoundCloud (oEmbed no-auth; __sc_hydration apiClient.id as client_id; API v2 with pagination),
Genius (OS token in UA bypasses 403; internal /api/songs no auth; strip first lyrics div header),
Dev.to (public REST API; burst limit 6 req then 429/1s; listings empty without auth).
Coursera (public API no auth, q=search is POST-only/405 on GET),
DuckDuckGo (Instant Answer API, skip_disambig=1 essential, widget answers unusable),
SEC EDGAR (company UA required for www.sec.gov; 10 req/s; XBRL frames for cross-company),
TrustPilot (http_get works; __NEXT_DATA__ has reviews; 10-page cap per filter).
Eventbrite (JSON-LD ItemList, __NEXT_DATA__ for detail; public API needs auth),
Etsy (DataDome blocks http_get; browser CDP + official API v3 with free key),
CoinGecko (free API, sleep 5s between calls; /coins/list for IDs not symbols),
eBay (http_get works until rate limit; JSON-LD Product schema on detail pages).
Saurav Panda