项目文件夹

文件
Gelei Deng ab5fbb4d90 feat: ship the durable multi-model autonomous PentestGPT runtime (#493)
* first refactor

* feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark

Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for
BOTH Claude Code and Codex, and add a multi-model benchmark harness.

Backend (multi-model):
- Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/
  backend.py) wraps unified_agent's Codex backend and translates its events into
  AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex
  (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction,
  and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1).

Docker tool (tool-only image; the benchmark stays OUTSIDE the image):
- Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/
  pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore
  (keeps creds/benchmark/workspace out of the build context).
- Persistent dual login (the hard part) — asymmetric by token model:
  * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint
    exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS
    host creds live in the Keychain and can't be copied).
  * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens
    are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455
    OAuth callback is forwarded into the container via a socat hop (-p 1455:8455).
  * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s).
- docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking
  preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/
  run/shell/down/nuke).
- Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex
  logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an
  isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login.

Benchmark (multi-model, host-side):
- benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers
  the loopback port, runs the pipeline across the 4 model combos, judges by the baked
  FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates).
  Includes the partial pilot's results (results.jsonl + REPORT.md).

Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README
docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: fail controller on backend error messages

* fix: allow listing sessions without target

* docs: add docker xbow benchmark report

* fix: infer concrete backend constructor type

* docs: refresh docker benchmark documentation

* feat(benchmark): add pure single-agent baseline + pipeline comparison

Add a "pure single agent" benchmark variant -- one bare `claude -p` /
`codex exec` call per target (no pipeline) -- to quantify what the 3-stage
PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets.

- pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the
  pipeline's shared fragments collapsed into ONE turn, so prompt content is
  held constant and the only variable is the multi-stage decomposition.
- benchmark/pilot/run_docker_bench.py: docker-network runner
  (--variant single|pipeline). Brings the target up, discovers the container's
  internal IP+network (skips DB side-cars/ports), docker-runs the tool image on
  that network, and scores the ground-truth flag against the agent's *assistant
  text* only (parity with the pipeline's raw streaming). Reads stdout in chunks
  to handle >64KB JSON lines. Resumable; --dry-run supported.
- benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head
  pipeline-vs-single per model on the common non-infra set.
- tests/unit/test_single_agent_prompt.py: prompt-builder coverage.
- docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated.

Recorded result (10 medium/hard targets x 4 models, container-to-container,
same baseline image digest 0c4c0f3e..., commit dca0019 image):

  Model               Pipeline   Single
  Claude Opus           5/10      7/10   (single +2)
  Claude Sonnet         6/10      4/10   (pipeline +2)
  Codex gpt-5.5         7/10      7/10   (tie)
  Codex gpt-5.4-mini    3/10      4/10   (single +1)
  TOTAL                21/40     22/40

Single agent matches the pipeline on solve rate (55% vs 52%) while using
~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline
only clearly helps Claude Sonnet (which times out solo); Opus is better solo.
Full per-challenge grid in DOCKER_COMPARISON.md; raw records in
docker_single_results.jsonl.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(benchmark): add pentestgpt_agent docker harness

* bench: refresh pentestgpt_agent smoke result

* fix(benchmark): make repeat rows variant-aware

* fix(agent): fall back for semantic executor labels

* fix(agent): tolerate executor prose evidence

* fix(benchmark): score accepted framework findings

* bench: append partial framework repeat results

* bench: complete framework repeat sweep

* bench: expose framework executor concurrency

* bench: add extended parallel framework sweep

* checkpoint: preserve working agent and benchmark state

* feat: harden durable agent loop and xbow qualification

* fix: reserve an exploit result turn

* docs: record clean xbow qualification

* build: consume unified-agent from the git wrapper repo

Repoint pentestgpt_agent_new's unified-agent dependency from the local
editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned
git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock
and update test_dependency.py to assert the external package is installed
from that VCS URL (not the repo-root vendored copy) at version 0.2.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: make pentestgpt_agent_new the sole framework

Remove the retired ledger-based pentestgpt_agent package (instructor/executor/
judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new
project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent
from the git wrapper) is now the single maintained framework.

Repoint the top-level tooling to it:
- pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from
  the wheel packages.
- Makefile: lint/format target parent code only; typecheck/check/ci now run the
  nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new /
  check-agent-new, so `make check` finally covers it; `make run` delegates to the
  pentestgpt-agent-new CLI.
- Dockerfile: stop copying the removed package (kept the build working); note the
  framework is not baked into the image yet.
- docker container-health test: import the substrate packages that actually ship.
- CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and
  the deprioritized benchmark/Docker rewire.

The XBOW `--variant framework` path and docker-bench Makefile targets still point
at the old in-image framework and are left as a pending rewire (benchmarks
deprioritized); the naive `--variant single` path is unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: rename pentestgpt_agent_new -> pentestgpt_agent

The framework reclaims the clean name now that the old ledger-based package is
gone. Rename the nested project folder, its src package, the distribution
(pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in
the package, the umbrella Makefile, the Dockerfile, the docker health test, and
CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit;
the git-sourced unified-agent dependency is unchanged. `make check` is green
(108 nested tests). The two historical *_REPORT.md files keep the old name as
dated records.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: extract benchmark harness to sibling xbow-benchmark repo

Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this
project clean. The harness was decoupled from the framework code (it scores
container output, never imports pentestgpt_agent/unified_agent), so only
operational ties remain and they now live in the sibling repo.

- Remove benchmark/ and the 4 harness unit tests (relocated + repointed there).
- Strip the docker-bench-*/bench-* targets and their config vars from the
  Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a
  help pointer to `make -C ../xbow-benchmark help`.

The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and
runs the pentestgpt:latest image built here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: harden autonomous framework and runtime integration

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:49:08 +08:00

235 行
8.5 KiB
Python

"""UnifiedAgent: one class that drives Claude Code or Codex interchangeably.
SuperAgent fans the same task out to several UnifiedAgents concurrently.
"""
from __future__ import annotations
import asyncio
from collections.abc import AsyncIterator, Iterable, Mapping
from pathlib import Path
from typing import Any
from .backends.base import AgentBackend
from .events import (
AgentEvent,
AssistantText,
FileChanged,
SessionStarted,
ToolCall,
TurnCompleted,
)
from .skills import install_skills
from .task import Task
from .tools import ToolRegistry, build_tool_server_spec
from .types import (
BackendUnavailableError,
RunOptions,
SandboxPolicy,
ToolServerSpec,
UnifiedResult,
UnifiedUsage,
)
BACKEND_NAMES = ("claude", "codex")
def _make_backend(backend: str | AgentBackend) -> AgentBackend:
if not isinstance(backend, str):
return backend
try:
if backend == "claude":
from .backends.claude_code import ClaudeCodeBackend
return ClaudeCodeBackend()
if backend == "codex":
from .backends.codex import CodexBackend
return CodexBackend()
except ImportError as e:
raise BackendUnavailableError(
f"backend {backend!r} is installed-time unavailable: {e}. "
f"Install the SDK ('pip install claude-agent-sdk' / 'pip install --pre openai-codex')."
) from e
raise BackendUnavailableError(
f"unknown backend {backend!r}; expected one of {BACKEND_NAMES} or an AgentBackend instance"
)
def collect(backend_name: str, events: list[AgentEvent], error: str | None = None) -> UnifiedResult:
"""Fold a normalized event stream into a UnifiedResult."""
turn = next((e for e in reversed(events) if isinstance(e, TurnCompleted)), None)
last_text = next((e.text for e in reversed(events) if isinstance(e, AssistantText)), None)
session_id = next((e.session_id for e in events if isinstance(e, SessionStarted)), None)
text = (turn.final_text if turn and turn.final_text else last_text) or ""
success = bool(turn and turn.success) and error is None
return UnifiedResult(
backend=backend_name,
success=success,
text=text,
structured_output=turn.structured_output if turn else None,
usage=turn.usage if turn else UnifiedUsage(),
cost_usd=turn.cost_usd if turn else None,
session_id=(turn.session_id if turn and turn.session_id else session_id),
duration_ms=turn.duration_ms if turn else None,
tool_calls=[e for e in events if isinstance(e, ToolCall)],
file_changes=[e for e in events if isinstance(e, FileChanged)],
events=list(events),
error=error or (turn.error if turn else None),
)
class UnifiedAgent:
"""One agent unit. Same task, tools, skills and events on either backend.
Args:
backend: "claude", "codex", or any AgentBackend instance.
workspace: directory the agent works in (created if missing; skills are
installed into its .claude/skills and .agents/skills).
model: backend-native model name (e.g. "claude-opus-4-8" / "gpt-5.5");
None uses each backend's default.
sandbox: unified write policy (see SandboxPolicy for the per-backend mapping).
tools: a ToolRegistry or "pkg.mod:REGISTRY" spec served to both backends
over stdio MCP as mcp__<server>__<tool>.
skills_dir: canonical Agent Skills source dir, installed into both hosts'
discovery dirs on first run.
instructions: appended to each backend's native system prompt
(Claude: claude_code preset append; Codex: developer_instructions).
effort: reasoning effort. Both accept "low"|"medium"|"high"|"xhigh";
Claude additionally "max", Codex "none"/"minimal".
extra_env: environment passed to the agent process and the tool server.
stream_text: emit TextDelta events for incremental assistant text.
"""
def __init__(
self,
backend: str | AgentBackend,
workspace: str | Path = ".",
*,
model: str | None = None,
sandbox: SandboxPolicy = SandboxPolicy.WORKSPACE_WRITE,
tools: ToolRegistry | str | None = None,
skills_dir: str | Path | None = None,
instructions: str | None = None,
effort: str | None = None,
extra_env: Mapping[str, str] | None = None,
stream_text: bool = False,
skills_install_mode: str = "symlink",
):
self.backend = _make_backend(backend)
self.workspace = Path(workspace).resolve()
self.model = model
self.sandbox = SandboxPolicy(sandbox)
self.tools = tools
self.skills_dir = Path(skills_dir).resolve() if skills_dir else None
self.instructions = instructions
self.effort = effort
self.extra_env = dict(extra_env or {})
self.stream_text = stream_text
self.skills_install_mode = skills_install_mode
self._tool_server: ToolServerSpec | None = None
self._prepared = False
@property
def name(self) -> str:
return self.backend.name
def _prepare(self) -> None:
if self._prepared:
return
self.workspace.mkdir(parents=True, exist_ok=True)
if self.skills_dir is not None:
install_skills(self.workspace, self.skills_dir, mode=self.skills_install_mode)
if self.tools is not None:
self._tool_server = build_tool_server_spec(self.tools, self.extra_env)
self._prepared = True
def _run_options(
self,
output_schema: dict[str, Any] | None,
resume: str | None,
max_turns: int | None,
) -> RunOptions:
return RunOptions(
workspace=self.workspace,
model=self.model,
sandbox=self.sandbox,
instructions=self.instructions,
effort=self.effort,
tool_server=self._tool_server,
output_schema=output_schema,
resume=resume,
max_turns=max_turns,
extra_env=dict(self.extra_env),
stream_text=self.stream_text,
)
async def stream(
self,
task: Task | str,
*,
output_schema: dict[str, Any] | None = None,
resume: str | None = None,
max_turns: int | None = None,
) -> AsyncIterator[AgentEvent]:
"""Yield normalized events for one run. Raises on backend failure."""
self._prepare()
prompt = Task.coerce(task).render(self.backend.name)
opts = self._run_options(output_schema, resume, max_turns)
async for event in self.backend.stream(prompt, opts):
yield event
async def run(
self,
task: Task | str,
*,
output_schema: dict[str, Any] | None = None,
resume: str | None = None,
max_turns: int | None = None,
) -> UnifiedResult:
"""Run one task to completion; failures come back as success=False results."""
events: list[AgentEvent] = []
try:
async for event in self.stream(
task, output_schema=output_schema, resume=resume, max_turns=max_turns
):
events.append(event)
except Exception as e:
return collect(self.backend.name, events, error=f"{type(e).__name__}: {e}")
return collect(self.backend.name, events)
def run_sync(self, task: Task | str, **kwargs: Any) -> UnifiedResult:
return asyncio.run(self.run(task, **kwargs))
class SuperAgent:
"""Controls several UnifiedAgents as one unit."""
def __init__(self, agents: Mapping[str, UnifiedAgent]):
if not agents:
raise ValueError("SuperAgent needs at least one agent")
self._agents = dict(agents)
@property
def agents(self) -> dict[str, UnifiedAgent]:
return dict(self._agents)
async def run(self, name: str, task: Task | str, **kwargs: Any) -> UnifiedResult:
return await self._agents[name].run(task, **kwargs)
async def run_all(
self,
task: Task | str,
*,
only: Iterable[str] | None = None,
**kwargs: Any,
) -> dict[str, UnifiedResult]:
"""Run the same task on every (selected) agent concurrently."""
names = list(only) if only is not None else list(self._agents)
results = await asyncio.gather(*(self._agents[n].run(task, **kwargs) for n in names))
return dict(zip(names, results, strict=True))
def run_all_sync(self, task: Task | str, **kwargs: Any) -> dict[str, UnifiedResult]:
return asyncio.run(self.run_all(task, **kwargs))