项目文件夹

文件
Gelei Deng ab5fbb4d90 feat: ship the durable multi-model autonomous PentestGPT runtime (#493)
* first refactor

* feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark

Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for
BOTH Claude Code and Codex, and add a multi-model benchmark harness.

Backend (multi-model):
- Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/
  backend.py) wraps unified_agent's Codex backend and translates its events into
  AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex
  (gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction,
  and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1).

Docker tool (tool-only image; the benchmark stays OUTSIDE the image):
- Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/
  pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore
  (keeps creds/benchmark/workspace out of the build context).
- Persistent dual login (the hard part) — asymmetric by token model:
  * Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint
    exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS
    host creds live in the Keychain and can't be copied).
  * Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens
    are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455
    OAuth callback is forwarded into the container via a socat hop (-p 1455:8455).
  * scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s).
- docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking
  preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/
  run/shell/down/nuke).
- Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex
  logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an
  isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login.

Benchmark (multi-model, host-side):
- benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers
  the loopback port, runs the pipeline across the 4 model combos, judges by the baked
  FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates).
  Includes the partial pilot's results (results.jsonl + REPORT.md).

Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README
docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: fail controller on backend error messages

* fix: allow listing sessions without target

* docs: add docker xbow benchmark report

* fix: infer concrete backend constructor type

* docs: refresh docker benchmark documentation

* feat(benchmark): add pure single-agent baseline + pipeline comparison

Add a "pure single agent" benchmark variant -- one bare `claude -p` /
`codex exec` call per target (no pipeline) -- to quantify what the 3-stage
PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets.

- pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the
  pipeline's shared fragments collapsed into ONE turn, so prompt content is
  held constant and the only variable is the multi-stage decomposition.
- benchmark/pilot/run_docker_bench.py: docker-network runner
  (--variant single|pipeline). Brings the target up, discovers the container's
  internal IP+network (skips DB side-cars/ports), docker-runs the tool image on
  that network, and scores the ground-truth flag against the agent's *assistant
  text* only (parity with the pipeline's raw streaming). Reads stdout in chunks
  to handle >64KB JSON lines. Resumable; --dry-run supported.
- benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head
  pipeline-vs-single per model on the common non-infra set.
- tests/unit/test_single_agent_prompt.py: prompt-builder coverage.
- docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated.

Recorded result (10 medium/hard targets x 4 models, container-to-container,
same baseline image digest 0c4c0f3e..., commit dca0019 image):

  Model               Pipeline   Single
  Claude Opus           5/10      7/10   (single +2)
  Claude Sonnet         6/10      4/10   (pipeline +2)
  Codex gpt-5.5         7/10      7/10   (tie)
  Codex gpt-5.4-mini    3/10      4/10   (single +1)
  TOTAL                21/40     22/40

Single agent matches the pipeline on solve rate (55% vs 52%) while using
~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline
only clearly helps Claude Sonnet (which times out solo); Opus is better solo.
Full per-challenge grid in DOCKER_COMPARISON.md; raw records in
docker_single_results.jsonl.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* feat(benchmark): add pentestgpt_agent docker harness

* bench: refresh pentestgpt_agent smoke result

* fix(benchmark): make repeat rows variant-aware

* fix(agent): fall back for semantic executor labels

* fix(agent): tolerate executor prose evidence

* fix(benchmark): score accepted framework findings

* bench: append partial framework repeat results

* bench: complete framework repeat sweep

* bench: expose framework executor concurrency

* bench: add extended parallel framework sweep

* checkpoint: preserve working agent and benchmark state

* feat: harden durable agent loop and xbow qualification

* fix: reserve an exploit result turn

* docs: record clean xbow qualification

* build: consume unified-agent from the git wrapper repo

Repoint pentestgpt_agent_new's unified-agent dependency from the local
editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned
git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock
and update test_dependency.py to assert the external package is installed
from that VCS URL (not the repo-root vendored copy) at version 0.2.0.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: make pentestgpt_agent_new the sole framework

Remove the retired ledger-based pentestgpt_agent package (instructor/executor/
judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new
project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent
from the git wrapper) is now the single maintained framework.

Repoint the top-level tooling to it:
- pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from
  the wheel packages.
- Makefile: lint/format target parent code only; typecheck/check/ci now run the
  nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new /
  check-agent-new, so `make check` finally covers it; `make run` delegates to the
  pentestgpt-agent-new CLI.
- Dockerfile: stop copying the removed package (kept the build working); note the
  framework is not baked into the image yet.
- docker container-health test: import the substrate packages that actually ship.
- CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and
  the deprioritized benchmark/Docker rewire.

The XBOW `--variant framework` path and docker-bench Makefile targets still point
at the old in-image framework and are left as a pending rewire (benchmarks
deprioritized); the naive `--variant single` path is unaffected.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* refactor: rename pentestgpt_agent_new -> pentestgpt_agent

The framework reclaims the clean name now that the old ledger-based package is
gone. Rename the nested project folder, its src package, the distribution
(pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in
the package, the umbrella Makefile, the Dockerfile, the docker health test, and
CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit;
the git-sourced unified-agent dependency is unchanged. `make check` is green
(108 nested tests). The two historical *_REPORT.md files keep the old name as
dated records.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* chore: extract benchmark harness to sibling xbow-benchmark repo

Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this
project clean. The harness was decoupled from the framework code (it scores
container output, never imports pentestgpt_agent/unified_agent), so only
operational ties remain and they now live in the sibling repo.

- Remove benchmark/ and the 4 harness unit tests (relocated + repointed there).
- Strip the docker-bench-*/bench-* targets and their config vars from the
  Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a
  help pointer to `make -C ../xbow-benchmark help`.

The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and
runs the pentestgpt:latest image built here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* feat: harden autonomous framework and runtime integration

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-13 16:49:08 +08:00

173 行
6.6 KiB
Python

"""Tests for Docker container health.
Docker tests that verify the container starts correctly and has
the required tools installed.
"""
import subprocess
import time
from pathlib import Path
import pytest
# Project root directory
PROJECT_ROOT = Path(__file__).parent.parent.parent
@pytest.mark.docker
@pytest.mark.slow
class TestContainerHealth:
"""Tests for container health and required tools."""
@pytest.fixture(scope="class")
def running_container(self):
"""Start the container for testing and clean up after.
This fixture starts the container, yields the container name,
and ensures cleanup after all tests in the class complete.
"""
# Start the container in detached mode
result = subprocess.run(
["docker", "compose", "up", "-d"],
cwd=PROJECT_ROOT,
capture_output=True,
text=True,
timeout=120,
)
if result.returncode != 0:
pytest.skip(f"Could not start container: {result.stderr}")
# Wait for container to be ready
time.sleep(5)
container_name = "pentestgpt"
# Check if container is running
check_result = subprocess.run(
["docker", "ps", "-q", "-f", f"name={container_name}"],
capture_output=True,
text=True,
timeout=10,
)
if not check_result.stdout.strip():
pytest.skip("Container not running")
yield container_name
# Cleanup: stop and remove container
subprocess.run(
["docker", "compose", "down"],
cwd=PROJECT_ROOT,
capture_output=True,
timeout=60,
)
def _exec_in_container(self, container: str, command: str) -> subprocess.CompletedProcess:
"""Execute a command in the container."""
return subprocess.run(
["docker", "exec", container, "bash", "-c", command],
capture_output=True,
text=True,
timeout=30,
)
def test_container_starts(self, running_container: str):
"""Test that the container starts successfully."""
result = subprocess.run(
["docker", "ps", "-f", f"name={running_container}", "--format", "{{.Status}}"],
capture_output=True,
text=True,
timeout=10,
)
assert result.returncode == 0
assert "Up" in result.stdout, "Container should be running"
def test_workspace_exists(self, running_container: str):
"""Test that /workspace directory exists."""
result = self._exec_in_container(running_container, "test -d /workspace && echo ok")
assert "ok" in result.stdout, "/workspace directory should exist"
def test_python_installed(self, running_container: str):
"""Test that Python 3.12+ is installed."""
result = self._exec_in_container(running_container, "python3 --version")
assert result.returncode == 0
assert "Python 3.1" in result.stdout, "Python 3.12+ should be installed"
def test_uv_installed(self, running_container: str):
"""Test that the uv package manager is installed."""
result = self._exec_in_container(running_container, "uv --version")
assert result.returncode == 0
assert "uv" in result.stdout, "uv should be installed"
def test_nmap_installed(self, running_container: str):
"""Test that nmap is installed."""
result = self._exec_in_container(running_container, "which nmap")
assert result.returncode == 0
assert "nmap" in result.stdout, "nmap should be installed"
def test_curl_installed(self, running_container: str):
"""Test that curl is installed."""
result = self._exec_in_container(running_container, "which curl")
assert result.returncode == 0
assert "curl" in result.stdout, "curl should be installed"
def test_git_installed(self, running_container: str):
"""Test that git is installed."""
result = self._exec_in_container(running_container, "git --version")
assert result.returncode == 0
assert "git version" in result.stdout, "git should be installed"
def test_netcat_installed(self, running_container: str):
"""Test that netcat is installed."""
result = self._exec_in_container(
running_container, "which nc || which netcat || which ncat"
)
assert result.returncode == 0, "netcat should be installed"
def test_ripgrep_installed(self, running_container: str):
"""Test that ripgrep is installed."""
result = self._exec_in_container(running_container, "rg --version")
assert result.returncode == 0
assert "ripgrep" in result.stdout, "ripgrep should be installed"
def test_tmux_installed(self, running_container: str):
"""Test that tmux is installed."""
result = self._exec_in_container(running_container, "tmux -V")
assert result.returncode == 0
assert "tmux" in result.stdout, "tmux should be installed"
def test_pentester_user_exists(self, running_container: str):
"""Test that the pentester user exists."""
result = self._exec_in_container(running_container, "id pentester")
assert result.returncode == 0
assert "pentester" in result.stdout, "pentester user should exist"
def test_sudo_available(self, running_container: str):
"""Test that sudo is available."""
result = self._exec_in_container(running_container, "which sudo")
assert result.returncode == 0
assert "sudo" in result.stdout, "sudo should be installed"
def test_agent_packages_importable(self, running_container: str):
"""Test that the substrate packages baked into the image are importable.
The maintained framework (pentestgpt_agent) is not baked into this
image yet (see Dockerfile), so only the substrate + legacy CLI ship here.
"""
result = self._exec_in_container(
running_container,
"python3 -c 'import unified_agent, pentestgpt_legacy; print(\"ok\")'",
)
assert result.returncode == 0, f"Import failed: {result.stderr}"
assert "ok" in result.stdout, "substrate packages should be importable"
def test_claude_code_available(self, running_container: str):
"""Test that Claude Code CLI is available."""
result = self._exec_in_container(running_container, "which claude || echo 'not found'")
# Claude might not be installed in all environments, but check
if "not found" in result.stdout:
pytest.skip("Claude Code CLI not installed in container")
assert "claude" in result.stdout