项目文件夹

文件
Gelei Deng ab5fbb4d90 feat: ship the durable multi-model autonomous PentestGPT runtime (#493)
* first refactor

* feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark

Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for
BOTH Claude Code and Codex, and add a multi-model benchmark harness.

Backend (multi-model):
- Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/
  backend.py) wraps unified_agent's Codex backend and translates its events into
  AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex
  (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction,
  and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1).

Docker tool (tool-only image; the benchmark stays OUTSIDE the image):
- Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/
  pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore
  (keeps creds/benchmark/workspace out of the build context).
- Persistent dual login (the hard part) — asymmetric by token model:
  * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint
    exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS
    host creds live in the Keychain and can't be copied).
  * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens
    are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455
    OAuth callback is forwarded into the container via a socat hop (-p 1455:8455).
  * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s).
- docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking
  preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/
  run/shell/down/nuke).
- Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex
  logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an
  isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login.

Benchmark (multi-model, host-side):
- benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers
  the loopback port, runs the pipeline across the 4 model combos, judges by the baked
  FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates).
  Includes the partial pilot's results (results.jsonl + REPORT.md).

Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README
docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: fail controller on backend error messages

* fix: allow listing sessions without target

* docs: add docker xbow benchmark report

* fix: infer concrete backend constructor type

* docs: refresh docker benchmark documentation

* feat(benchmark): add pure single-agent baseline + pipeline comparison

Add a "pure single agent" benchmark variant -- one bare `claude -p` /
`codex exec` call per target (no pipeline) -- to quantify what the 3-stage
PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets.

- pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the
  pipeline's shared fragments collapsed into ONE turn, so prompt content is
  held constant and the only variable is the multi-stage decomposition.
- benchmark/pilot/run_docker_bench.py: docker-network runner
  (--variant single|pipeline). Brings the target up, discovers the container's
  internal IP+network (skips DB side-cars/ports), docker-runs the tool image on
  that network, and scores the ground-truth flag against the agent's *assistant
  text* only (parity with the pipeline's raw streaming). Reads stdout in chunks
  to handle >64KB JSON lines. Resumable; --dry-run supported.
- benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head
  pipeline-vs-single per model on the common non-infra set.
- tests/unit/test_single_agent_prompt.py: prompt-builder coverage.
- docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated.

Recorded result (10 medium/hard targets x 4 models, container-to-container,
same baseline image digest 0c4c0f3e..., commit dca0019 image):

  Model               Pipeline   Single
  Claude Opus           5/10      7/10   (single +2)
  Claude Sonnet         6/10      4/10   (pipeline +2)
  Codex gpt-5.5         7/10      7/10   (tie)
  Codex gpt-5.4-mini    3/10      4/10   (single +1)
  TOTAL                21/40     22/40

Single agent matches the pipeline on solve rate (55% vs 52%) while using
~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline
only clearly helps Claude Sonnet (which times out solo); Opus is better solo.
Full per-challenge grid in DOCKER_COMPARISON.md; raw records in
docker_single_results.jsonl.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(benchmark): add pentestgpt_agent docker harness

* bench: refresh pentestgpt_agent smoke result

* fix(benchmark): make repeat rows variant-aware

* fix(agent): fall back for semantic executor labels

* fix(agent): tolerate executor prose evidence

* fix(benchmark): score accepted framework findings

* bench: append partial framework repeat results

* bench: complete framework repeat sweep

* bench: expose framework executor concurrency

* bench: add extended parallel framework sweep

* checkpoint: preserve working agent and benchmark state

* feat: harden durable agent loop and xbow qualification

* fix: reserve an exploit result turn

* docs: record clean xbow qualification

* build: consume unified-agent from the git wrapper repo

Repoint pentestgpt_agent_new's unified-agent dependency from the local
editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned
git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock
and update test_dependency.py to assert the external package is installed
from that VCS URL (not the repo-root vendored copy) at version 0.2.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: make pentestgpt_agent_new the sole framework

Remove the retired ledger-based pentestgpt_agent package (instructor/executor/
judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new
project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent
from the git wrapper) is now the single maintained framework.

Repoint the top-level tooling to it:
- pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from
  the wheel packages.
- Makefile: lint/format target parent code only; typecheck/check/ci now run the
  nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new /
  check-agent-new, so `make check` finally covers it; `make run` delegates to the
  pentestgpt-agent-new CLI.
- Dockerfile: stop copying the removed package (kept the build working); note the
  framework is not baked into the image yet.
- docker container-health test: import the substrate packages that actually ship.
- CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and
  the deprioritized benchmark/Docker rewire.

The XBOW `--variant framework` path and docker-bench Makefile targets still point
at the old in-image framework and are left as a pending rewire (benchmarks
deprioritized); the naive `--variant single` path is unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: rename pentestgpt_agent_new -> pentestgpt_agent

The framework reclaims the clean name now that the old ledger-based package is
gone. Rename the nested project folder, its src package, the distribution
(pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in
the package, the umbrella Makefile, the Dockerfile, the docker health test, and
CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit;
the git-sourced unified-agent dependency is unchanged. `make check` is green
(108 nested tests). The two historical *_REPORT.md files keep the old name as
dated records.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: extract benchmark harness to sibling xbow-benchmark repo

Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this
project clean. The harness was decoupled from the framework code (it scores
container output, never imports pentestgpt_agent/unified_agent), so only
operational ties remain and they now live in the sibling repo.

- Remove benchmark/ and the 4 harness unit tests (relocated + repointed there).
- Strip the docker-bench-*/bench-* targets and their config vars from the
  Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a
  help pointer to `make -C ../xbow-benchmark help`.

The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and
runs the pentestgpt:latest image built here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: harden autonomous framework and runtime integration

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:49:08 +08:00

236 行
7.2 KiB
Python

import sys
import pytest
from claude_agent_sdk import (
AssistantMessage,
ResultMessage,
StreamEvent,
SystemMessage,
TextBlock,
ThinkingBlock,
ToolResultBlock,
ToolUseBlock,
UserMessage,
)
from unified_agent.backends.claude_code import (
ClaudeCodeBackend,
build_options,
normalize_message,
)
from unified_agent.events import (
AssistantText,
Reasoning,
SessionStarted,
TextDelta,
ToolCall,
ToolResult,
TurnCompleted,
)
from unified_agent.types import (
AgentAuthError,
RunOptions,
SandboxPolicy,
ToolServerError,
ToolServerSpec,
)
def opts(tmp_path, **kw) -> RunOptions:
return RunOptions(workspace=tmp_path, **kw)
TOOL_SERVER = ToolServerSpec(
server_name="unified",
command=[sys.executable, "-m", "unified_agent.tool_server", "tests.fixture_registry:REG"],
env={"PYTHONPATH": "/repo"},
)
# --- option mapping -------------------------------------------------------
def test_read_only_policy_maps_to_dontask_and_no_write_tools(tmp_path):
o = build_options(opts(tmp_path, sandbox=SandboxPolicy.READ_ONLY))
assert o.permission_mode == "dontAsk"
assert "Read" in o.allowed_tools and "Bash" not in o.allowed_tools
assert set(o.disallowed_tools) >= {"Write", "Edit", "Bash"}
assert o.cwd == str(tmp_path)
assert o.setting_sources == ["project"]
def test_workspace_write_policy_maps_to_acceptedits_with_bash(tmp_path):
o = build_options(opts(tmp_path, sandbox=SandboxPolicy.WORKSPACE_WRITE))
assert o.permission_mode == "acceptEdits"
assert {"Bash", "Write", "Edit", "Read"} <= set(o.allowed_tools)
assert o.disallowed_tools == []
def test_full_access_policy_maps_to_bypass(tmp_path):
o = build_options(opts(tmp_path, sandbox=SandboxPolicy.FULL_ACCESS))
assert o.permission_mode == "bypassPermissions"
def test_tool_server_becomes_stdio_mcp_config_and_allowlist(tmp_path):
o = build_options(opts(tmp_path, tool_server=TOOL_SERVER))
cfg = o.mcp_servers["unified"]
assert cfg["type"] == "stdio"
assert cfg["command"] == sys.executable
assert cfg["args"][0:2] == ["-m", "unified_agent.tool_server"]
assert cfg["env"] == {"PYTHONPATH": "/repo"}
assert "mcp__unified__*" in o.allowed_tools
def test_instructions_append_to_claude_code_preset(tmp_path):
o = build_options(opts(tmp_path, instructions="be terse"))
assert o.system_prompt == {
"type": "preset",
"preset": "claude_code",
"append": "be terse",
}
def test_effort_passthrough(tmp_path):
assert build_options(opts(tmp_path, effort="xhigh")).effort == "xhigh"
assert build_options(opts(tmp_path)).effort is None
def test_default_keeps_native_claude_code_system_prompt(tmp_path):
"""Regression: system_prompt=None strips Claude Code's system prompt entirely
(including working-directory context), making file ops land outside the
workspace. The agent unit must always run with the native preset."""
o = build_options(opts(tmp_path))
assert o.system_prompt == {"type": "preset", "preset": "claude_code"}
def test_output_schema_and_passthroughs(tmp_path):
schema = {"type": "object", "properties": {"n": {"type": "integer"}}}
o = build_options(
opts(
tmp_path,
output_schema=schema,
resume="sess-1",
max_turns=4,
model="claude-opus-4-8",
extra_env={"X": "1"},
stream_text=True,
)
)
assert o.output_format == {"type": "json_schema", "schema": schema}
assert o.resume == "sess-1"
assert o.max_turns == 4
assert o.model == "claude-opus-4-8"
assert o.env == {"X": "1"}
assert o.include_partial_messages is True
# --- message normalization ------------------------------------------------
def norm(msg, tmp_path, **kw):
return list(normalize_message(msg, opts(tmp_path, **kw)))
def test_init_system_message_yields_session_started(tmp_path):
msg = SystemMessage(
subtype="init",
data={"session_id": "s-9", "mcp_servers": [{"name": "unified", "status": "connected"}]},
)
events = norm(msg, tmp_path, tool_server=TOOL_SERVER)
assert events == [SessionStarted(session_id="s-9")]
def test_failed_tool_server_raises(tmp_path):
msg = SystemMessage(
subtype="init",
data={"session_id": "s", "mcp_servers": [{"name": "unified", "status": "failed"}]},
)
with pytest.raises(ToolServerError, match="unified"):
norm(msg, tmp_path, tool_server=TOOL_SERVER)
def test_assistant_blocks_normalize(tmp_path):
msg = AssistantMessage(
content=[
ThinkingBlock(thinking="hmm", signature="sig"),
TextBlock(text="hello"),
ToolUseBlock(id="t1", name="mcp__unified__add_numbers", input={"a": 1, "b": 2}),
],
model="claude-opus-4-8",
)
events = norm(msg, tmp_path)
assert events == [
Reasoning(text="hmm"),
AssistantText(text="hello"),
ToolCall(name="mcp__unified__add_numbers", input={"a": 1, "b": 2}, call_id="t1"),
]
def test_auth_error_raises(tmp_path):
msg = AssistantMessage(content=[], model="m", error="authentication_failed")
with pytest.raises(AgentAuthError):
norm(msg, tmp_path)
def test_tool_result_in_user_message(tmp_path):
msg = UserMessage(content=[ToolResultBlock(tool_use_id="t1", content="3.0", is_error=False)])
events = norm(msg, tmp_path)
assert events == [ToolResult(call_id="t1", output="3.0", is_error=False)]
def test_result_message_maps_to_turn_completed(tmp_path):
msg = ResultMessage(
subtype="success",
duration_ms=1500,
duration_api_ms=900,
is_error=False,
num_turns=3,
session_id="s-9",
total_cost_usd=0.0123,
usage={
"input_tokens": 100,
"cache_read_input_tokens": 40,
"output_tokens": 25,
},
result="done!",
structured_output={"n": 1},
)
[event] = norm(msg, tmp_path)
assert isinstance(event, TurnCompleted)
assert event.success is True
assert event.final_text == "done!"
assert event.usage.input_tokens == 100
assert event.usage.cached_input_tokens == 40
assert event.usage.output_tokens == 25
assert event.cost_usd == 0.0123
assert event.session_id == "s-9"
assert event.duration_ms == 1500
assert event.structured_output == {"n": 1}
assert event.stop_reason == "success"
def test_error_result_message(tmp_path):
msg = ResultMessage(
subtype="error_max_turns",
duration_ms=10,
duration_api_ms=5,
is_error=True,
num_turns=9,
session_id="s",
errors=["ran out of turns"],
)
[event] = norm(msg, tmp_path)
assert event.success is False
assert "ran out of turns" in event.error
def test_stream_event_text_delta_only_when_enabled(tmp_path):
ev = {"type": "content_block_delta", "delta": {"type": "text_delta", "text": "he"}}
msg = StreamEvent(uuid="u", session_id="s", event=ev, parent_tool_use_id=None)
assert norm(msg, tmp_path, stream_text=True) == [TextDelta(text="he")]
assert norm(msg, tmp_path, stream_text=False) == []
def test_backend_name():
assert ClaudeCodeBackend().name == "claude"