greydgl--pentestgpt
ab5fbb4d90
* first refactor
* feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark
Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for
BOTH Claude Code and Codex, and add a multi-model benchmark harness.
Backend (multi-model):
- Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/
backend.py) wraps unified_agent's Codex backend and translates its events into
AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex
(gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction,
and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1).
Docker tool (tool-only image; the benchmark stays OUTSIDE the image):
- Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/
pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore
(keeps creds/benchmark/workspace out of the build context).
- Persistent dual login (the hard part) — asymmetric by token model:
* Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint
exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS
host creds live in the Keychain and can't be copied).
* Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens
are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455
OAuth callback is forwarded into the container via a socat hop (-p 1455:8455).
* scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s).
- docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking
preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/
run/shell/down/nuke).
- Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex
logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an
isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login.
Benchmark (multi-model, host-side):
- benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers
the loopback port, runs the pipeline across the 4 model combos, judges by the baked
FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates).
Includes the partial pilot's results (results.jsonl + REPORT.md).
Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README
docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: fail controller on backend error messages
* fix: allow listing sessions without target
* docs: add docker xbow benchmark report
* fix: infer concrete backend constructor type
* docs: refresh docker benchmark documentation
* feat(benchmark): add pure single-agent baseline + pipeline comparison
Add a "pure single agent" benchmark variant -- one bare `claude -p` /
`codex exec` call per target (no pipeline) -- to quantify what the 3-stage
PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets.
- pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the
pipeline's shared fragments collapsed into ONE turn, so prompt content is
held constant and the only variable is the multi-stage decomposition.
- benchmark/pilot/run_docker_bench.py: docker-network runner
(--variant single|pipeline). Brings the target up, discovers the container's
internal IP+network (skips DB side-cars/ports), docker-runs the tool image on
that network, and scores the ground-truth flag against the agent's *assistant
text* only (parity with the pipeline's raw streaming). Reads stdout in chunks
to handle >64KB JSON lines. Resumable; --dry-run supported.
- benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head
pipeline-vs-single per model on the common non-infra set.
- tests/unit/test_single_agent_prompt.py: prompt-builder coverage.
- docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated.
Recorded result (10 medium/hard targets x 4 models, container-to-container,
same baseline image digest 0c4c0f3e..., commit dca0019 image):
Model Pipeline Single
Claude Opus 5/10 7/10 (single +2)
Claude Sonnet 6/10 4/10 (pipeline +2)
Codex gpt-5.5 7/10 7/10 (tie)
Codex gpt-5.4-mini 3/10 4/10 (single +1)
TOTAL 21/40 22/40
Single agent matches the pipeline on solve rate (55% vs 52%) while using
~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline
only clearly helps Claude Sonnet (which times out solo); Opus is better solo.
Full per-challenge grid in DOCKER_COMPARISON.md; raw records in
docker_single_results.jsonl.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(benchmark): add pentestgpt_agent docker harness
* bench: refresh pentestgpt_agent smoke result
* fix(benchmark): make repeat rows variant-aware
* fix(agent): fall back for semantic executor labels
* fix(agent): tolerate executor prose evidence
* fix(benchmark): score accepted framework findings
* bench: append partial framework repeat results
* bench: complete framework repeat sweep
* bench: expose framework executor concurrency
* bench: add extended parallel framework sweep
* checkpoint: preserve working agent and benchmark state
* feat: harden durable agent loop and xbow qualification
* fix: reserve an exploit result turn
* docs: record clean xbow qualification
* build: consume unified-agent from the git wrapper repo
Repoint pentestgpt_agent_new's unified-agent dependency from the local
editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned
git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock
and update test_dependency.py to assert the external package is installed
from that VCS URL (not the repo-root vendored copy) at version 0.2.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: make pentestgpt_agent_new the sole framework
Remove the retired ledger-based pentestgpt_agent package (instructor/executor/
judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new
project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent
from the git wrapper) is now the single maintained framework.
Repoint the top-level tooling to it:
- pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from
the wheel packages.
- Makefile: lint/format target parent code only; typecheck/check/ci now run the
nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new /
check-agent-new, so `make check` finally covers it; `make run` delegates to the
pentestgpt-agent-new CLI.
- Dockerfile: stop copying the removed package (kept the build working); note the
framework is not baked into the image yet.
- docker container-health test: import the substrate packages that actually ship.
- CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and
the deprioritized benchmark/Docker rewire.
The XBOW `--variant framework` path and docker-bench Makefile targets still point
at the old in-image framework and are left as a pending rewire (benchmarks
deprioritized); the naive `--variant single` path is unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: rename pentestgpt_agent_new -> pentestgpt_agent
The framework reclaims the clean name now that the old ledger-based package is
gone. Rename the nested project folder, its src package, the distribution
(pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in
the package, the umbrella Makefile, the Dockerfile, the docker health test, and
CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit;
the git-sourced unified-agent dependency is unchanged. `make check` is green
(108 nested tests). The two historical *_REPORT.md files keep the old name as
dated records.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: extract benchmark harness to sibling xbow-benchmark repo
Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this
project clean. The harness was decoupled from the framework code (it scores
container output, never imports pentestgpt_agent/unified_agent), so only
operational ties remain and they now live in the sibling repo.
- Remove benchmark/ and the 4 harness unit tests (relocated + repointed there).
- Strip the docker-bench-*/bench-* targets and their config vars from the
Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a
help pointer to `make -C ../xbow-benchmark help`.
The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and
runs the pentestgpt:latest image built here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: harden autonomous framework and runtime integration
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
232 行
8.4 KiB
Python
232 行
8.4 KiB
Python
"""Claude Code backend via claude-agent-sdk (bundles the Claude Code CLI).
|
|
|
|
Sandbox policy is mapped onto Claude Code's permission system. Claude Code has
|
|
no OS sandbox by default, so READ_ONLY removes write/shell tools entirely
|
|
(`dontAsk` denies anything not allow-listed), while Codex can still execute
|
|
commands inside its read-only OS sandbox — the honest asymmetry documented in
|
|
the README.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from collections.abc import AsyncIterator, Iterator
|
|
|
|
from claude_agent_sdk import (
|
|
AssistantMessage,
|
|
ClaudeAgentOptions,
|
|
CLINotFoundError,
|
|
ProcessError,
|
|
ResultMessage,
|
|
StreamEvent,
|
|
SystemMessage,
|
|
TextBlock,
|
|
ThinkingBlock,
|
|
ToolResultBlock,
|
|
ToolUseBlock,
|
|
UserMessage,
|
|
query,
|
|
)
|
|
|
|
from ..events import (
|
|
AgentEvent,
|
|
AssistantText,
|
|
RawEvent,
|
|
Reasoning,
|
|
SessionStarted,
|
|
TextDelta,
|
|
ToolCall,
|
|
ToolResult,
|
|
TurnCompleted,
|
|
)
|
|
from ..types import (
|
|
AgentAuthError,
|
|
AgentRunError,
|
|
BackendUnavailableError,
|
|
RunOptions,
|
|
SandboxPolicy,
|
|
ToolServerError,
|
|
UnifiedUsage,
|
|
)
|
|
|
|
READ_TOOLS = ["Read", "Glob", "Grep"]
|
|
WRITE_TOOLS = ["Write", "Edit", "NotebookEdit"]
|
|
EXEC_TOOLS = ["Bash"]
|
|
|
|
# statuses the init message may report for an MCP server that is not usable
|
|
_BAD_MCP_STATUSES = {"failed", "needs-auth", "needs_auth", "error"}
|
|
|
|
|
|
def build_options(opts: RunOptions) -> ClaudeAgentOptions:
|
|
allowed: list[str]
|
|
disallowed: list[str] = []
|
|
if opts.sandbox is SandboxPolicy.READ_ONLY:
|
|
permission_mode = "dontAsk" # deny anything not allow-listed, never prompt
|
|
allowed = list(READ_TOOLS)
|
|
disallowed = WRITE_TOOLS + EXEC_TOOLS
|
|
elif opts.sandbox is SandboxPolicy.WORKSPACE_WRITE:
|
|
permission_mode = "acceptEdits"
|
|
allowed = READ_TOOLS + WRITE_TOOLS + EXEC_TOOLS
|
|
else: # FULL_ACCESS
|
|
permission_mode = "bypassPermissions"
|
|
allowed = READ_TOOLS + WRITE_TOOLS + EXEC_TOOLS
|
|
|
|
mcp_servers: dict = {}
|
|
if opts.tool_server is not None:
|
|
spec = opts.tool_server
|
|
mcp_servers[spec.server_name] = {
|
|
"type": "stdio",
|
|
"command": spec.command[0],
|
|
"args": spec.command[1:],
|
|
"env": dict(spec.env),
|
|
}
|
|
allowed.append(f"mcp__{spec.server_name}__*")
|
|
|
|
return ClaudeAgentOptions(
|
|
cwd=str(opts.workspace),
|
|
model=opts.model,
|
|
permission_mode=permission_mode,
|
|
allowed_tools=allowed,
|
|
disallowed_tools=disallowed,
|
|
mcp_servers=mcp_servers,
|
|
# Hermetic: only project-level settings/skills/CLAUDE.md from the
|
|
# workspace; no user/local config bleeding into runs.
|
|
setting_sources=["project"],
|
|
# Always run with Claude Code's native system prompt: passing None
|
|
# strips it entirely — including the working-directory env context —
|
|
# which makes the model write files outside the workspace. This
|
|
# mirrors Codex, where developer_instructions append to (never
|
|
# replace) the base prompt.
|
|
system_prompt=(
|
|
{"type": "preset", "preset": "claude_code", "append": opts.instructions}
|
|
if opts.instructions
|
|
else {"type": "preset", "preset": "claude_code"}
|
|
),
|
|
output_format=(
|
|
{"type": "json_schema", "schema": opts.output_schema} if opts.output_schema else None
|
|
),
|
|
effort=opts.effort,
|
|
resume=opts.resume,
|
|
max_turns=opts.max_turns,
|
|
env=dict(opts.extra_env),
|
|
include_partial_messages=opts.stream_text,
|
|
)
|
|
|
|
|
|
def _stringify_block_content(content) -> str:
|
|
if content is None:
|
|
return ""
|
|
if isinstance(content, str):
|
|
return content
|
|
parts = []
|
|
for item in content:
|
|
if isinstance(item, dict):
|
|
parts.append(str(item.get("text", item)))
|
|
else:
|
|
parts.append(str(item))
|
|
return "\n".join(parts)
|
|
|
|
|
|
def normalize_message(message, opts: RunOptions) -> Iterator[AgentEvent]:
|
|
if isinstance(message, SystemMessage):
|
|
if message.subtype == "init":
|
|
data = message.data or {}
|
|
if opts.tool_server is not None:
|
|
for server in data.get("mcp_servers", []):
|
|
if (
|
|
server.get("name") == opts.tool_server.server_name
|
|
and server.get("status") in _BAD_MCP_STATUSES
|
|
):
|
|
raise ToolServerError(
|
|
f"MCP tool server '{server['name']}' failed to connect in "
|
|
f"Claude Code (status={server.get('status')!r})"
|
|
)
|
|
session_id = data.get("session_id")
|
|
if session_id:
|
|
yield SessionStarted(session_id=session_id)
|
|
else:
|
|
yield RawEvent(backend="claude", kind=f"system:{message.subtype}", data=message.data)
|
|
|
|
elif isinstance(message, AssistantMessage):
|
|
if message.error == "authentication_failed":
|
|
raise AgentAuthError(
|
|
"Claude Code is not authenticated: run `claude /login` or set ANTHROPIC_API_KEY"
|
|
)
|
|
if message.error:
|
|
yield RawEvent(
|
|
backend="claude", kind=f"assistant_error:{message.error}", data=message.error
|
|
)
|
|
for block in message.content:
|
|
if isinstance(block, TextBlock):
|
|
yield AssistantText(text=block.text)
|
|
elif isinstance(block, ThinkingBlock):
|
|
yield Reasoning(text=block.thinking)
|
|
elif isinstance(block, ToolUseBlock):
|
|
yield ToolCall(name=block.name, input=block.input, call_id=block.id)
|
|
elif isinstance(block, ToolResultBlock):
|
|
yield ToolResult(
|
|
call_id=block.tool_use_id,
|
|
output=_stringify_block_content(block.content),
|
|
is_error=bool(block.is_error),
|
|
)
|
|
|
|
elif isinstance(message, UserMessage):
|
|
content = message.content
|
|
if isinstance(content, list):
|
|
for block in content:
|
|
if isinstance(block, ToolResultBlock):
|
|
yield ToolResult(
|
|
call_id=block.tool_use_id,
|
|
output=_stringify_block_content(block.content),
|
|
is_error=bool(block.is_error),
|
|
)
|
|
|
|
elif isinstance(message, StreamEvent):
|
|
if opts.stream_text:
|
|
event = message.event or {}
|
|
delta = event.get("delta") or {}
|
|
if event.get("type") == "content_block_delta" and delta.get("type") == "text_delta":
|
|
yield TextDelta(text=delta.get("text", ""))
|
|
|
|
elif isinstance(message, ResultMessage):
|
|
usage = message.usage or {}
|
|
success = (not message.is_error) and message.subtype == "success"
|
|
errors = getattr(message, "errors", None) or []
|
|
yield TurnCompleted(
|
|
success=success,
|
|
final_text=message.result,
|
|
usage=UnifiedUsage(
|
|
input_tokens=usage.get("input_tokens", 0) or 0,
|
|
cached_input_tokens=usage.get("cache_read_input_tokens", 0) or 0,
|
|
output_tokens=usage.get("output_tokens", 0) or 0,
|
|
),
|
|
cost_usd=message.total_cost_usd,
|
|
session_id=message.session_id,
|
|
duration_ms=message.duration_ms,
|
|
structured_output=message.structured_output,
|
|
stop_reason=message.subtype,
|
|
error=None if success else ("; ".join(errors) or message.subtype),
|
|
)
|
|
|
|
else:
|
|
yield RawEvent(backend="claude", kind=type(message).__name__, data=message)
|
|
|
|
|
|
class ClaudeCodeBackend:
|
|
name = "claude"
|
|
|
|
async def stream(self, prompt: str, opts: RunOptions) -> AsyncIterator[AgentEvent]:
|
|
try:
|
|
async for message in query(prompt=prompt, options=build_options(opts)):
|
|
for event in normalize_message(message, opts):
|
|
yield event
|
|
except CLINotFoundError as e:
|
|
raise BackendUnavailableError(f"Claude Code CLI not found: {e}") from e
|
|
except ProcessError as e:
|
|
stderr = (e.stderr or "").strip()
|
|
lowered = stderr.lower()
|
|
if "auth" in lowered or "login" in lowered or "api key" in lowered:
|
|
raise AgentAuthError(f"Claude Code process failed (auth): {stderr}") from e
|
|
raise AgentRunError(
|
|
f"Claude Code process failed (exit {e.exit_code}): {stderr[-2000:]}"
|
|
) from e
|