/docs/self-modifying-harness.md

Self-modifying harness

Browser Harness works because the agent is not trapped behind a fixed browser API. When a page has a new edge case, the agent can inspect the page, add the missing helper in agent-workspace/agent_helpers.py or a domain skill, retry the task, and keep the working code for the next run.

The loop is:

1. Use capture_screenshot() and page_info() to understand the current state.
2. Try the smallest existing primitive: click_at_xy, type_text, press_key, upload_file, js, or raw cdp.
3. If the primitive is missing, add a helper in agent-workspace/agent_helpers.py or a site-specific note in agent-workspace/domain-skills/<site>/.
4. Run the helper, then verify with a screenshot or page state.
5. Commit only durable knowledge: selectors, DOM events, CDP calls, endpoint shape, or framework quirks. Do not commit secrets or one-off pixel positions.

That is why browser edge cases are not permanent. They are usually just missing local code.

Example 1: file upload

Start with the built-in helper when there is a real file input:

upload_file("input[type=file]", "/tmp/invoice.pdf")
js("""
const input = document.querySelector('input[type=file]');
input.dispatchEvent(new Event('input', {bubbles: true}));
input.dispatchEvent(new Event('change', {bubbles: true}));
""")

If the app hides the input behind a custom button, add a helper that finds the real input once and then reuses the same path:

# agent-workspace/agent_helpers.py
def upload_to_hidden_input(label_text, path):
    selector = js(f"""
    const labels = [...document.querySelectorAll('label,button,[role=button]')];
    const trigger = labels.find(e => e.innerText.trim().includes({label_text!r}));
    const root = trigger?.closest('form,section,div') || document;
    const input = root.querySelector('input[type=file]');
    if (!input) throw new Error('no file input near upload trigger');
    input.setAttribute('data-bh-upload-target', '1');
    return '[data-bh-upload-target="1"]';
    """)
    upload_file(selector, path)

Keep this in a domain skill if the selector is specific to one product.

Example 2: drag and drop

Try compositor-level input first when the page responds to real pointer motion:

cdp("Input.dispatchMouseEvent", type="mouseMoved", x=120, y=220)
cdp("Input.dispatchMouseEvent", type="mousePressed", x=120, y=220, button="left")
cdp("Input.dispatchMouseEvent", type="mouseMoved", x=500, y=260, button="left")
cdp("Input.dispatchMouseEvent", type="mouseReleased", x=500, y=260, button="left")

Some React or editor surfaces require DOM drag events with a DataTransfer payload. Put that site-specific sequence in a helper:

# agent-workspace/agent_helpers.py
def dom_drag_text(source_selector, target_selector, payload):
    js(f"""
    const source = document.querySelector({source_selector!r});
    const target = document.querySelector({target_selector!r});
    const dt = new DataTransfer();
    dt.setData('text/plain', {payload!r});
    for (const type of ['dragstart', 'dragenter', 'dragover', 'drop', 'dragend']) {{
      const node = type === 'dragstart' || type === 'dragend' ? source : target;
      node.dispatchEvent(new DragEvent(type, {{bubbles: true, cancelable: true, dataTransfer: dt}}));
    }}
    """)

The durable knowledge is not the one drag coordinate. It is whether that app wants real pointer events, DOM drag events, or a hidden file input.

Example 3: canvas signature

Canvas fields often have no useful DOM target inside the drawing area. Use the canvas bounds plus raw mouse events:

box = js("""
const r = document.querySelector('canvas.signature').getBoundingClientRect();
return {x: r.left, y: r.top, w: r.width, h: r.height};
""")

points = [
    (box["x"] + 20, box["y"] + box["h"] * 0.55),
    (box["x"] + 80, box["y"] + box["h"] * 0.35),
    (box["x"] + 150, box["y"] + box["h"] * 0.60),
    (box["x"] + 220, box["y"] + box["h"] * 0.40),
]
cdp("Input.dispatchMouseEvent", type="mousePressed", x=points[0][0], y=points[0][1], button="left")
for x, y in points[1:]:
    cdp("Input.dispatchMouseEvent", type="mouseMoved", x=x, y=y, button="left")
cdp("Input.dispatchMouseEvent", type="mouseReleased", x=points[-1][0], y=points[-1][1], button="left")

Verify by reading app state or screenshotting the canvas. If the page stores a hidden signature value, add a site helper that checks that field too.

Example 4: coordinate-only target

Some UI is only visible pixels: maps, whiteboards, games, image editors, or custom canvas widgets. Do not hunt for selectors that do not exist. Let the screenshot drive the next action:

shot = capture_screenshot("/tmp/target.png", max_dim=1800)
info = page_info()
print(info)
# read the target position from the screenshot, convert image pixels to CSS
# pixels if devicePixelRatio > 1, then click:
click_at_xy(412, 318)
capture_screenshot("/tmp/after.png", max_dim=1800)

If the target is stable enough to compute, write the helper:

# agent-workspace/agent_helpers.py
def click_canvas_target(canvas_selector, rel_x, rel_y):
    box = js(f"""
    const r = document.querySelector({canvas_selector!r}).getBoundingClientRect();
    return {{x: r.left, y: r.top, w: r.width, h: r.height}};
    """)
    click_at_xy(box["x"] + box["w"] * rel_x, box["y"] + box["h"] * rel_y)

The point is not that coordinates are magic. The point is that the agent can choose coordinates, DOM, CDP, or a new helper after it sees the actual page.

Benchmark page

docs/edge-case-benchmark.html contains a small local page with four tasks:

• file upload
• drag and drop
• canvas signature
• coordinate-only canvas click

Use it when changing helper behavior or when demonstrating the self-modifying loop. A run is successful when window.bhBenchmarkResults() returns all four checks as true.