/docs/self-modifying-harness.md

Self-modifying Browser Harness

Browser Harness is intentionally small. The core helpers cover navigation, screenshots, raw CDP, JavaScript, coordinates, keyboard input, and files. When a site needs something more specific, the agent should extend the harness while it works instead of treating the site as an unsolved edge case.

The loop is:

1. Inspect the page with capture_screenshot() and page_info().
2. Try the smallest built-in primitive: click_at_xy, type_text, press_key, upload_file, js, or raw cdp.
3. If the missing operation is reusable, add a helper to agent-workspace/agent_helpers.py.
4. Call the helper from browser-harness -c.
5. Verify with another screenshot or direct page state check.
6. If the helper teaches a durable site-specific trick, save it as a domain skill under agent-workspace/domain-skills/<site>/.

This is the main difference from a fixed browser automation wrapper: the agent can patch the harness during the task, then immediately retry with the new primitive.

Example 1: file upload behind a styled button

Many sites hide the real <input type="file"> and expose a styled button. If a visual click opens the file picker, do not ask the user to handle it. Find the input and set the file through CDP.

# browser-harness -c '...'
from pathlib import Path

new_tab("https://example.com/profile")
wait_for_load()

avatar = Path("/tmp/avatar.png").resolve()
upload_file("input[type=file]", str(avatar))

print(js("""
const input = document.querySelector('input[type=file]');
return input && input.files.length;
"""))
capture_screenshot("/tmp/uploaded.png", max_dim=1800)

If the selector is unstable, add a task helper:

# agent-workspace/agent_helpers.py
def upload_first_file_input(path):
    inputs = js("""
    return [...document.querySelectorAll('input[type=file]')]
      .map((el, i) => ({ i, accept: el.accept, name: el.name, id: el.id }));
    """)
    if not inputs:
        raise RuntimeError("no file input on page")
    upload_file("input[type=file]", path)

The helper is intentionally narrow: it solves the current page shape without adding a framework around uploads.

Example 2: drag-and-drop that ignores simple clicks

For visual drag handles, use compositor-level mouse events first. These pass through shadow DOM, iframes, and framework wrappers because Chrome receives a real pointer sequence.

# browser-harness -c '...'
capture_screenshot("/tmp/before-drag.png", max_dim=1800)

cdp("Input.dispatchMouseEvent", type="mousePressed", x=180, y=420,
    button="left", clickCount=1)
cdp("Input.dispatchMouseEvent", type="mouseMoved", x=340, y=420,
    button="left")
cdp("Input.dispatchMouseEvent", type="mouseMoved", x=520, y=420,
    button="left")
cdp("Input.dispatchMouseEvent", type="mouseReleased", x=520, y=420,
    button="left", clickCount=1)

capture_screenshot("/tmp/after-drag.png", max_dim=1800)

If the app only responds to DOM dragover / drop, add a helper that performs that site's expected event sequence:

# agent-workspace/agent_helpers.py
def dom_drag_between(source_selector, target_selector):
    return js(f"""
    const source = document.querySelector({source_selector!r});
    const target = document.querySelector({target_selector!r});
    if (!source || !target) throw new Error("missing drag source or target");

    const data = new DataTransfer();
    for (const type of ["dragstart", "dragenter", "dragover", "drop", "dragend"]) {{
      const node = type === "dragstart" || type === "dragend" ? source : target;
      node.dispatchEvent(new DragEvent(type, {{
        bubbles: true,
        cancelable: true,
        dataTransfer: data
      }}));
    }}
    return true;
    """)

Use the DOM path only after the low-level pointer path fails; the pointer path is closer to what a user does.

Example 3: signature field or canvas drawing

Canvas widgets often have no useful DOM children. Treat them as coordinate surfaces. Locate the canvas, draw through CDP mouse events, and verify with a screenshot or by checking the backing canvas pixels.

# agent-workspace/agent_helpers.py
def sign_canvas(selector="canvas", points=None):
    points = points or [
        (0.15, 0.65), (0.30, 0.35), (0.45, 0.70),
        (0.62, 0.32), (0.80, 0.58),
    ]
    rect = js(f"""
    const el = document.querySelector({selector!r});
    if (!el) throw new Error("canvas not found");
    const r = el.getBoundingClientRect();
    return {{ x: r.x, y: r.y, w: r.width, h: r.height }};
    """)
    absolute = [
        (rect["x"] + x * rect["w"], rect["y"] + y * rect["h"])
        for x, y in points
    ]
    first = absolute[0]
    cdp("Input.dispatchMouseEvent", type="mousePressed", x=first[0], y=first[1],
        button="left", clickCount=1)
    for x, y in absolute[1:]:
        cdp("Input.dispatchMouseEvent", type="mouseMoved", x=x, y=y,
            button="left")
    last = absolute[-1]
    cdp("Input.dispatchMouseEvent", type="mouseReleased", x=last[0], y=last[1],
        button="left", clickCount=1)

Then call it:

# browser-harness -c '...'
sign_canvas("canvas.signature")
capture_screenshot("/tmp/signature.png", max_dim=1800)

The reusable part is not the exact signature shape. It is the conversion from a canvas selector to real pointer coordinates.

Example 4: coordinate-only controls

Some controls are visible but hostile to selectors: canvas maps, custom sliders, SVG editors, image crop boxes, or cross-origin iframe buttons. Use the screenshot as the source of truth, convert device pixels to CSS pixels when needed, and click the visible target.

# browser-harness -c '...'
path = capture_screenshot("/tmp/target.png", max_dim=1800)
info = page_info()
dpr = js("window.devicePixelRatio") or 1
print({"screenshot": path, "viewport": info, "devicePixelRatio": dpr})

# If the target was measured at image pixel (960, 540) on a 2x screenshot:
click_at_xy(960 / dpr, 540 / dpr)
capture_screenshot("/tmp/clicked.png", max_dim=1800)

If the same coordinate pattern repeats, add a helper with semantic names:

# agent-workspace/agent_helpers.py
def click_canvas_percent(selector, x_pct, y_pct):
    rect = js(f"""
    const el = document.querySelector({selector!r});
    if (!el) throw new Error("target not found");
    const r = el.getBoundingClientRect();
    return {{ x: r.x, y: r.y, w: r.width, h: r.height }};
    """)
    click_at_xy(rect["x"] + rect["w"] * x_pct, rect["y"] + rect["h"] * y_pct)

Now the task can say click_canvas_percent("canvas.map", 0.72, 0.41) instead of carrying brittle absolute coordinates through the rest of the run.

Edge-case benchmark

Use edge-case-benchmark.html to test the four patterns on a single page: hidden file upload, drag/drop event payloads, signature canvas, and a coordinate-only canvas target.

For a normal local browser session:

# browser-harness -c '...'
from pathlib import Path

new_tab(Path("docs/edge-case-benchmark.html").resolve().as_uri())
wait_for_load()
print(page_info())

On a remote Browser Use Cloud browser, local file:// URLs are not available. Load the HTML into about:blank instead:

# browser-harness -c '...'
from pathlib import Path

html = Path("docs/edge-case-benchmark.html").read_text()
new_tab("about:blank")
frame_id = cdp("Page.getFrameTree")["frameTree"]["frame"]["id"]
cdp("Page.setDocumentContent", frameId=frame_id, html=html)

The benchmark exposes window.bhBenchmarkResults() so an agent can verify completion without relying on a visual guess.

What to commit back

Commit helpers that are durable and general enough to help the next run:

• site-specific login, upload, checkout, export, or scraping flows go in agent-workspace/domain-skills/<site>/;
• reusable interaction mechanics go in interaction-skills/;
• one-off task glue can stay in agent-workspace/agent_helpers.py on the user's machine.

Do not commit secrets, user data, screenshots of private sessions, or fixed pixel coordinates from one viewport. Commit the durable map of how the site works, not the diary of one run.