项目文件夹

文件
Sarath S Menon fb1a51dd9b refactor: move to src layout, agent-workspace, and fix SKILL.md invocation format (#229)
* refactor(tests): reorganize into tests/unit and tests/integration

Moves all root-level test_*.py files into a structured tests/ directory:
- tests/unit/ — admin, helpers (was test_screenshot), run
- tests/integration/ — js expression tests
- tests/conftest.py — shared fake_png pytest fixture, eliminating duplication

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* refactor: move to src layout, agent-workspace, and fix SKILL.md invocation format

- Move package to src/browser_harness/ and domain-skills/interaction-skills to agent-workspace/
- Fix all browser-harness <<'PY' heredoc examples in SKILL.md and run.py HELP string to use the correct -c '...' flag format (heredoc was never supported by the CLI)
- Update SKILL.md path references from domain-skills/ to agent-workspace/domain-skills/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-04-28 15:38:11 +05:30

5.5 KiB

Gmail — Compose and send

URL: https://mail.google.com

Prerequisites

  • Logged into Gmail in the attached Chrome profile.
  • Keyboard shortcuts enabled (Gmail default for most accounts).

Open compose

press_key("c")   # Gmail shortcut — opens a new compose dialog with the "To" field focused
wait(1)

Multiple compose dialogs stack — pick the visible one

Gmail keeps minimized drafts as dialogs at the bottom of the page. document.querySelectorAll('div[role="dialog"]') returns all of them (minimized and open). The minimized ones have small bounding rects (~h ≤ 40) and their inner inputs report offsetParent === null.

Always pick the visible dialog by size, not index:

idx = js("""(() => {
  const ds = [...document.querySelectorAll('div[role="dialog"]')];
  return ds.findIndex(d => d.getBoundingClientRect().height > 200);
})()""")

…and scope every subsequent query to dialogs[idx]. Using index 1 blindly works sometimes but breaks the moment the user has a second minimized draft already sitting at the bottom.

Trap: Tab inserts a literal \t into the "To" field

After press_key("c"), focus is on [aria-label="To recipients"]. press_key("Tab") does not advance focus — it inserts a tab character into the input. Confirmed by reading back value and finding "\t".

Either click the next field directly, or commit the recipient as a chip first (e.g. by typing a valid address; Gmail chips it automatically once the input loses focus or you type a separator).

The recipient does become a chip once you click away. Read chips from [role="dialog"] [data-hovercard-id]not from the input's value.

Fill the fields

# After press_key("c"), "To" is focused
type_text("someone@example.com")

# Don't Tab — click subject directly
sub = js("""(() => {
  const d = [...document.querySelectorAll('div[role="dialog"]')].find(d => d.getBoundingClientRect().height > 200);
  const s = d.querySelector('input[name="subjectbox"]');
  const r = s.getBoundingClientRect();
  return {x: r.x + r.width/2, y: r.y + r.height/2};
})()""")
click(sub["x"], sub["y"])
type_text("Subject here")

body = js("""(() => {
  const d = [...document.querySelectorAll('div[role="dialog"]')].find(d => d.getBoundingClientRect().height > 200);
  const b = d.querySelector('div[aria-label="Message Body"], div[role="textbox"]');
  const r = b.getBoundingClientRect();
  return {x: r.x + 40, y: r.y + 30};
})()""")
click(body["x"], body["y"])
type_text("Body text goes here.")

Attachments — use DOM.setFileInputFiles on the visible compose's input

The paperclip button opens a native file picker that browser-harness can't drive. Instead, set files directly on Gmail's hidden file input.

Gotcha: there is one input[type="file"][name="Filedata"] per compose dialog. If you use upload_file('input[type="file"][name="Filedata"]', ...), the default DOM.querySelector returns the first match — usually belongs to a stale/minimized compose, and Gmail ignores it. Always target the input scoped to the visible compose:

doc = cdp("DOM.getDocument", depth=-1)
ids = cdp("DOM.querySelectorAll", nodeId=doc["root"]["nodeId"],
          selector='input[type="file"][name="Filedata"]')["nodeIds"]
# Pick the one whose ancestor dialog has height > 200
# (quickest: the last one is usually the newest compose)
cdp("DOM.setFileInputFiles", files=["/abs/path.png"], nodeId=ids[-1])
wait(3)

After upload, input.files reads back as empty — Gmail consumes the FileList immediately. Don't treat that as failure. Instead, verify by screenshot or by searching the compose for the filename chip:

ok = js("""(() => {
  const d = [...document.querySelectorAll('div[role="dialog"]')].find(d => d.getBoundingClientRect().height > 200);
  return [...d.querySelectorAll('*')].some(e => /\\.\\w+ \\(\\d+[KMG]?\\)/.test(e.textContent || ''));
})()""")

The attachment chip format is filename.ext (61K) — size appears only once Gmail has finished ingesting the file.

Send

send = js("""(() => {
  const d = [...document.querySelectorAll('div[role="dialog"]')].find(d => d.getBoundingClientRect().height > 200);
  const b = [...d.querySelectorAll('[role="button"]')].find(b => (b.getAttribute('aria-label')||'').startsWith('Send'));
  const r = b.getBoundingClientRect();
  return {x: r.x + r.width/2, y: r.y + r.height/2};
})()""")
click(send["x"], send["y"])
wait(2)

Verify by looking for the "Message sent" toast at the bottom-left, or by checking that the visible compose dialog's height has collapsed. ⌘+Enter also sends but requires keyboard-shortcut support in the current account.

Stable selectors

  • To field: [aria-label="To recipients"]
  • Subject: input[name="subjectbox"]
  • Body: div[aria-label="Message Body"] (also matches div[role="textbox"] inside the dialog)
  • Send button: [role="dialog"] [role="button"][aria-label^="Send"]
  • Attach file input: input[type="file"][name="Filedata"] (one per dialog)
  • Recipient chip: [data-hovercard-id] inside the dialog

Traps

  • Tab in the "To" field inserts \t — never Tab between fields, click them.
  • input.files is cleared by Gmail after setFileInputFiles — don't use it as a success check.
  • The first match of input[type="file"] can belong to a stale/minimized compose; pick by dialog, not by index.
  • press_key("c") only works if keyboard shortcuts are enabled in the account. If it no-ops, fall back to clicking the left-rail Compose pencil.