Captures the durable shape of LinkedIn's received-invitations page:
- URL filter slugs (PEOPLE_WITH_MUTUAL_CONNECTION, PEOPLE_WITH_MUTUAL_SCHOOL)
and the chip counter as authoritative remaining-count.
- Accept/Ignore aria-label formats — they differ from each other, so you
cannot derive one from the other.
- The "follows you" trap: Accept renders as <a href=current-URL>, not
<button>; .click() follows href and no click path (MouseEvent, CDP
Input.dispatchMouseEvent) triggers the accept handler. Route these to
Ignore or skip.
- Pagination: list renders ~10 rows, replaced by "is now a connection"
acknowledgments after accepts; reload the URL to fetch the next slice,
scrolling does nothing.
- "Take care when connecting" safety modal appears intermittently.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
After the previous commit added early raises for LOGIN_REQUIRED and
ERROR in scrape_video(), two doc entries still described the old
behavior ("will succeed but status is LOGIN_REQUIRED"). Updated both
the Gotchas section and the 'What Requires a Browser' list to reflect
that scrape_video() now raises ValueError on age-restricted videos.
Two issues raised by automated review on PR #137:
1. max_results limit could be exceeded
The `break` in youtube_search() only exits the inner loop over
itemSectionRenderer contents. If YouTube returns multiple sections,
the outer loop would continue appending results beyond max_results.
Fixed by replacing `break` with `return results` to exit both loops
immediately once the limit is reached.
2. Regex match not null-checked before .group(1)
scrape_video() called m.group(1) directly after re.search(), which
raises AttributeError if the pattern is not found (e.g. private video,
deleted video, region-blocked content, or YouTube HTML structure change).
Fixed by adding an explicit None check with a descriptive ValueError,
and an early playabilityStatus check that surfaces LOGIN_REQUIRED and
ERROR states with clear messages before attempting to parse videoDetails.
Adds domain-skills/youtube/scraping.md with four verified approaches
for extracting YouTube data without a browser or API key.
## What's included
**Approach 1 — oEmbed API (fastest)**
- Single HTTP call, ~0.3s per video
- Returns title, author, channel URL, thumbnail, embed HTML
- Bulk fetching via ThreadPoolExecutor with real timing data
- Verified on multiple video IDs
**Approach 2 — Watch page ytInitialPlayerResponse**
- Full video metadata: title, author, channel_id, duration, view_count,
publish_date, upload_date, category, like_count, keywords, is_live,
is_private, is_unlisted, available_countries (249 codes), embed_url
- Correct regex pattern (non-greedy with lookahead) to parse the JSON blob
- Real output values verified and included as comments
- Accurate gotchas: viewCount/lengthSeconds are strings not ints,
likeCount lives in microformat not videoDetails
**Approach 3 — Search results (no API key)**
- Parses ytInitialData from /results?search_query= (server-side rendered)
- Returns up to ~14-20 results with videoId, title, channel, duration,
views, published, description snippet, thumbnail URL
- Verified: 15 results returned for "python tutorial"
**Approach 4 — Channel metadata**
- Handles both @handle and channel ID (UC...) URL formats
- Extracts channel_id, title, description, subscriber count, avatar,
banner from pageHeaderViewModel + channelMetadataRenderer
- Verified on @RickAstleyYT: "4.48m subscribers"
**Utilities**
- thumbnail_urls(): all 5 sizes with availability notes (maxres may 404)
- extract_video_id(): handles watch, youtu.be, /shorts/, /embed/ formats
**What requires a browser**
- Clear list of what http_get cannot access: trending, playlists,
comments, caption text, age-restricted videos
**URL patterns reference table**
**Gotchas (all verified)**
- ytInitialPlayerResponse regex non-greedy requirement
- viewCount/lengthSeconds are string types, not int
- likeCount location (microformat, not videoDetails)
- oEmbed 404 on private/deleted videos
- Caption baseUrl returns empty in all tested conditions
(plain http_get, XHR, and fetch with cookies) — not a session issue
- Search result count varies (~14-20), never assume fixed count
- Subscriber count is a rounded string, not an integer
Captures the site shape for centilebrain.org 'Generate Estimates' flow:
- Six Shiny iframes (modality x sex) at centilebrain-app.shinyapps.io
- Selectors for email, file input, compute, download
- Wait markers and three non-obvious traps (iframe target_id staleness,
MUI switch checkboxes, coordinate-vs-JS button click after scroll)
- End-to-end example using upload_file(..., target_id=...) with
iframe_target()
- Output zip schema
Requires the iframe target_id support added to upload_file().
Synthetic .click() on the visible Star button does not persist the
star — there's a hidden 0x0 fallback button that querySelector finds
first, and the visible React button swallows synthetic events. Submit
the form directly; CSRF is already embedded.
Field-tested while completing the install.md verification step.
Both skills share the post-article DOM surface (div[role="article"]
and the data-ad-*-preview message selectors), but differ in URL shape,
sort options, and rate-limit ceilings. Pages are public and tolerate a
higher rate; Groups gate content behind membership and are stricter.
Each file covers: URL patterns, DOM anchors with verification notes,
a collect-as-you-go scroll pattern (FB virtualizes the feed so
scroll-then-collect misses posts), the l.facebook.com/l.php redirector
decoder, a Firecrawl handoff example, rate-limit discipline, a
self-inspection JS block for detecting selector drift, and a full
end-to-end example that emits JSON on stdout for downstream tools.
The groups.md anchors were verified against a logged-in account on
2026-04-18. pages.md inherits the post-article anchors from groups.md
(shared React component) and adds Page-specific header/metadata
selectors; a gotchas log section invites confirmation on first live use.
Documents the Monaco + React-canvas seam that causes most 'automation
silently did nothing' failures in Framer:
- Double-click requires the full pointer+mouse event chain (detail:2 matters)
- Monaco paste must be clipboard + OS-level Cmd+A/Cmd+V, then wait before save
- Publish button only mounts when a page is selected in the Pages tab; it
rejects synthetic clicks and must be driven by screen-coord input
- Framer autolayout preempts programmatic Header position/left/right writes,
forcing nodes offscreen; delete-and-copy is the working workaround
Plus the list of canvas-level interactions that consistently reject
automation (drag-drop, variant switching, property binding, Page Settings)
so agents know to escalate to the human instead of retrying.
Stable data-testid selectors table for Pages/Layers/Assets tabs, and the
edge-cache + sitemap quirks on framer-hosted live domains.
Add routes, filter URL format, GraphQL bootstrap, and the 'credentials: include'
pattern for reusing the tab's NextAuth session cookie without JWE juggling.
Auth quirk documented: injecting only the JWE into a fresh Chrome profile
triggers a login loop; UI needs the full cookie set (persistent profile),
while backend GraphQL works with the JWE alone.
PubMed/NCBI (ESearch→ESummary/EFetch pipeline; count is string; ELink broken 2026; CollectiveName branch),
CrossRef (title/container-title always lists; abstract has JATS XML tags; type=proceedings-article not conference-paper),
OpenAlex (abstract_inverted_index reconstruction; cursor >10K pages; group_by not group-by; concepts deprecated use topics),
FRED (fredgraph.csv timeouts headlessly; API needs free key; BLS/WorldBank/AlphaVantage as keyless alternatives),
MusicBrainz (Mozilla/5.0 gets 403; recording length in ms; CAA front flag vs types array differ).
CoinMarketCap (internal data-api/v3 fully open, no auth; 25 calls no rate limit),
Quora (full Chrome UA required; push() payloads double-encoded JSON; 3 SSR answers only),
Itch.io (http_get works; game cards via CSS selectors; RSS feeds exist),
Steam (appdetails single appid only; price in cents; ISteamApps/GetAppList dead in 2026),
HowLongToBeat (two-step token flow /api/find/init then POST; comp_* in seconds not hours).
Letterboxd (http_get works on film pages; JSON-LD CDATA gotcha; API needs OAuth),
Gutenberg (Gutendex REST API; text via /cache/epub/; .opf is 404, use .rdf),
Metacritic (internal backend API key in HTML; Nuxt __NUXT_DATA__ not __NEXT_DATA__),
RAWG (API needs key; window.CLIENT_PARAMS in HTML has full game data without key),
OpenLibrary (full free API; missing cover = 43-byte GIF not 404; description dual type).
Glassdoor (Cloudflare managed challenge; browser only; __NEXT_DATA__ + DOM fallbacks),
Medium (?format=json strips XSSI prefix; GraphQL /_/graphql no auth; RSS 10-item cap),
SoundCloud (oEmbed no-auth; __sc_hydration apiClient.id as client_id; API v2 with pagination),
Genius (OS token in UA bypasses 403; internal /api/songs no auth; strip first lyrics div header),
Dev.to (public REST API; burst limit 6 req then 429/1s; listings empty without auth).
Coursera (public API no auth, q=search is POST-only/405 on GET),
DuckDuckGo (Instant Answer API, skip_disambig=1 essential, widget answers unusable),
SEC EDGAR (company UA required for www.sec.gov; 10 req/s; XBRL frames for cross-company),
TrustPilot (http_get works; __NEXT_DATA__ has reviews; 10-page cap per filter).
Eventbrite (JSON-LD ItemList, __NEXT_DATA__ for detail; public API needs auth),
Etsy (DataDome blocks http_get; browser CDP + official API v3 with free key),
CoinGecko (free API, sleep 5s between calls; /coins/list for IDs not symbols),
eBay (http_get works until rate limit; JSON-LD Product schema on detail pages).
Ran actual browser-harness sessions against each site and rewrote
the skill files from live test findings. Key corrections:
GitHub: wait(2) after wait_for_load() for React hydration; search API
separate 10 req/min limit; search/code needs auth (401 unauthed)
HackerNews: athing also matches comment rows (use 'athing submission');
job posts break naive score-zip; html.unescape() required for titles
Amazon: .zg-item-immersion gone from Best Sellers; #priceblock_ourprice
returns null (legacy); review count selector collides with cross-sell
widget — use [aria-label*='ratings'] instead
News: The Verge is Atom not RSS (namespace dict required); Reuters
hard-blocks http_get with 403 even with User-Agent; BBC shows no
consent banner from US IP; parallel fetch is 4.3x faster (0.16s vs 0.70s)
ProductHunt: goto() ERR_ABORTED — always use new_tab(); /posts/ URLs
don't exist (it's /products/); homepage has 30 fixed items no lazy load;
[data-test^='post-item-'] is the correct card selector
Captures three learnings from a real run that mis-reported $3,080 AUD for
a UACC-Rack-12U-Wall (real AU street ~$420-630): Tech Geeks is Shopify so
use /products/<handle>.js for canonical price and SKU; its .js `available`
flag is unreliable so cross-check the DOM for sold-out markers; and
sold-out pages there carry stale/junk prices that must never enter a
final table without a second-source sanity check.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two skill categories, pure markdown, no Python files:
- interaction-skills/ — generic browser patterns (dialogs, inputs, etc.)
Flat .md files. Agent reads the relevant one before a task.
- domain-skills/ — per-site playbooks (tiktok/, linkedin/, etc.)
Flat .md files per action (upload, schedule, post).
Subfolders for large domains when needed.
Starting with:
- interaction-skills/dialogs.md — CDP vs JS dialog handling
- domain-skills/tiktok/upload.md — full upload flow with gotchas
- Placeholder folders for linkedin, spreadshirt, salesforce
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Magnus Müller