* first refactor
* feat: dockerized tool with persistent Claude+Codex login + multi-model benchmark
Run the autonomous CTF/pentest tool in Docker with a one-time, persistent login for
BOTH Claude Code and Codex, and add a multi-model benchmark harness.
Backend (multi-model):
- Add `--backend {claude,codex}` to the CTF pipeline. CodexBackend (pentestgpt/core/
backend.py) wraps unified_agent's Codex backend and translates its events into
AgentMessages, so the same pipeline runs on Claude (opus/sonnet) or Codex
(gpt-5.5/gpt-5.4-mini). Wired through config.backend, pipeline stage construction,
and the CLI (+ PENTESTGPT_CODEX_EFFORT; greppable [CODEX_USAGE] under PENTESTGPT_BENCH=1).
Docker tool (tool-only image; the benchmark stays OUTSIDE the image):
- Extend Dockerfile: Codex CLI (@openai/codex) + openai_codex SDK + unified_agent/
pentestgpt_agent/pentestgpt_legacy packages + gobuster/dirb + socat. Add .dockerignore
(keeps creds/benchmark/workspace out of the build context).
- Persistent dual login (the hard part) — asymmetric by token model:
* Claude: `setup-token` -> token stored in the pentestgpt-claude volume; entrypoint
exports CLAUDE_CODE_OAUTH_TOKEN (setup-token does not write .credentials.json; macOS
host creds live in the Keychain and can't be copied).
* Codex: the container does its OWN `codex login` (NOT seeding -- ChatGPT refresh tokens
are single-use, so a shared/copied login 401s on first refresh). The 127.0.0.1:1455
OAuth callback is forwarded into the container via a socat hop (-p 1455:8455).
* scripts/docker-login.sh is idempotent: checks logins live, logs in only the missing one(s).
- docker-compose codex-config volume (+ pinned names); entrypoint token-export + non-blocking
preflight; scripts/docker-auth-status.sh; Make targets (docker-build/login/auth-status/
run/shell/down/nuke).
- Verified end-to-end: one `make docker-login` -> a fresh container reports claude+codex
logged in with live round-trips; the CTF pipeline (Codex) captured a flag against an
isolated fixture and the pentest pipeline ran cleanly; persists across recreation, no re-login.
Benchmark (multi-model, host-side):
- benchmark/pilot/ harness (run_pilot.py + report.py): builds each xbow challenge, discovers
the loopback port, runs the pipeline across the 4 model combos, judges by the baked
FLAG{sha256(UPPER-dir)}, and renders REPORT.md (infra failures excluded from solve rates).
Includes the partial pilot's results (results.jsonl + REPORT.md).
Docs: docs/docker-dev-plan.md (full plan + implementation status); CLAUDE.md and README
docker quickstart; benchmark/pilot/README.md; design-doc roadmap (docs/redesign).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: fail controller on backend error messages
* fix: allow listing sessions without target
* docs: add docker xbow benchmark report
* fix: infer concrete backend constructor type
* docs: refresh docker benchmark documentation
* feat(benchmark): add pure single-agent baseline + pipeline comparison
Add a "pure single agent" benchmark variant -- one bare `claude -p` /
`codex exec` call per target (no pipeline) -- to quantify what the 3-stage
PentestGPT pipeline buys over an un-orchestrated agent on the xbow targets.
- pentestgpt/prompts/stages.py: ctf_single_agent_{system,task}_prompt -- the
pipeline's shared fragments collapsed into ONE turn, so prompt content is
held constant and the only variable is the multi-stage decomposition.
- benchmark/pilot/run_docker_bench.py: docker-network runner
(--variant single|pipeline). Brings the target up, discovers the container's
internal IP+network (skips DB side-cars/ports), docker-runs the tool image on
that network, and scores the ground-truth flag against the agent's *assistant
text* only (parity with the pipeline's raw streaming). Reads stdout in chunks
to handle >64KB JSON lines. Resumable; --dry-run supported.
- benchmark/pilot/report_comparison.py -> DOCKER_COMPARISON.md: head-to-head
pipeline-vs-single per model on the common non-infra set.
- tests/unit/test_single_agent_prompt.py: prompt-builder coverage.
- docs: README, CLAUDE.md, benchmark README, DOCKER_REPORT updated.
Recorded result (10 medium/hard targets x 4 models, container-to-container,
same baseline image digest 0c4c0f3e..., commit dca0019 image):
Model Pipeline Single
Claude Opus 5/10 7/10 (single +2)
Claude Sonnet 6/10 4/10 (pipeline +2)
Codex gpt-5.5 7/10 7/10 (tie)
Codex gpt-5.4-mini 3/10 4/10 (single +1)
TOTAL 21/40 22/40
Single agent matches the pipeline on solve rate (55% vs 52%) while using
~40% fewer Codex tokens (13.0M vs 21.8M) and solving faster. The pipeline
only clearly helps Claude Sonnet (which times out solo); Opus is better solo.
Full per-challenge grid in DOCKER_COMPARISON.md; raw records in
docker_single_results.jsonl.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* feat(benchmark): add pentestgpt_agent docker harness
* bench: refresh pentestgpt_agent smoke result
* fix(benchmark): make repeat rows variant-aware
* fix(agent): fall back for semantic executor labels
* fix(agent): tolerate executor prose evidence
* fix(benchmark): score accepted framework findings
* bench: append partial framework repeat results
* bench: complete framework repeat sweep
* bench: expose framework executor concurrency
* bench: add extended parallel framework sweep
* checkpoint: preserve working agent and benchmark state
* feat: harden durable agent loop and xbow qualification
* fix: reserve an exploit result turn
* docs: record clean xbow qualification
* build: consume unified-agent from the git wrapper repo
Repoint pentestgpt_agent_new's unified-agent dependency from the local
editable path (../../UnifiedAgentPoC, now renamed and gone) to the pinned
git source PentestGPT-Project/UnifedAgentWrapper@d05d21f. Regenerate uv.lock
and update test_dependency.py to assert the external package is installed
from that VCS URL (not the repo-root vendored copy) at version 0.2.0.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: make pentestgpt_agent_new the sole framework
Remove the retired ledger-based pentestgpt_agent package (instructor/executor/
judge) and its orphaned unit + smoke tests. The nested pentestgpt_agent_new
project (Supervisor/Executor over a durable SQLite loop, consuming unified-agent
from the git wrapper) is now the single maintained framework.
Repoint the top-level tooling to it:
- pyproject: drop the pentestgpt-agent console script and pentestgpt_agent from
the wheel packages.
- Makefile: lint/format target parent code only; typecheck/check/ci now run the
nested framework's own gate (ruff, format, mypy, pytest) via test-agent-new /
check-agent-new, so `make check` finally covers it; `make run` delegates to the
pentestgpt-agent-new CLI.
- Dockerfile: stop copying the removed package (kept the build working); note the
framework is not baked into the image yet.
- docker container-health test: import the substrate packages that actually ship.
- CLAUDE.md / AGENT.md: describe the new framework, the git-sourced wrapper, and
the deprioritized benchmark/Docker rewire.
The XBOW `--variant framework` path and docker-bench Makefile targets still point
at the old in-image framework and are left as a pending rewire (benchmarks
deprioritized); the naive `--variant single` path is unaffected.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* refactor: rename pentestgpt_agent_new -> pentestgpt_agent
The framework reclaims the clean name now that the old ledger-based package is
gone. Rename the nested project folder, its src package, the distribution
(pentestgpt-agent-new -> pentestgpt-agent) and CLI, and every import/reference in
the package, the umbrella Makefile, the Dockerfile, the docker health test, and
CLAUDE.md / AGENT.md. Regenerate uv.lock. The audit CLI stays pentestgpt-agent-audit;
the git-sourced unified-agent dependency is unchanged. `make check` is green
(108 nested tests). The two historical *_REPORT.md files keep the old name as
dated records.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* chore: extract benchmark harness to sibling xbow-benchmark repo
Move PentestGPT/benchmark/ out to ../xbow-benchmark (its own repo) to keep this
project clean. The harness was decoupled from the framework code (it scores
container output, never imports pentestgpt_agent/unified_agent), so only
operational ties remain and they now live in the sibling repo.
- Remove benchmark/ and the 4 harness unit tests (relocated + repointed there).
- Strip the docker-bench-*/bench-* targets and their config vars from the
Makefile; keep the tool-image lifecycle (docker-build/login/run/...) and add a
help pointer to `make -C ../xbow-benchmark help`.
The sibling repo mounts this checkout read-only (--source-root ../PentestGPT) and
runs the pentestgpt:latest image built here.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* feat: harden autonomous framework and runtime integration
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix: 🐛 minor typo and build process
* feat: 🎸 [WIP] Pentest mode
* feat: 🎸 code abstraction
* feat: modernize legacy PentestGPT with native multi-LLM support (#469)
Rebuild the classic USENIX-2024 interactive PentestGPT (reasoning / generation /
parsing sessions + Pentesting Task Tree + REPL) as a standalone
`pentestgpt_legacy` package on a native per-provider LLM layer that supports the
latest 2026 models.
- llm/: BaseProvider + OpenAI-compatible / Anthropic / Gemini connectors, a
web-verified model registry (OpenAI, Anthropic, Gemini, DeepSeek, xAI, Qwen,
Moonshot, local Ollama), a factory, and an LLMClient bridging async providers
to the core's synchronous send_new_message/send_message session API.
- CLI `pentestgpt-legacy`: --list-models and --smoke-test (live per-model
round-trip matrix), plus --reasoning-model / --parsing-model / --base-url.
- Tests: 25 unit tests (mocked, no network). Live smoke test verified 22/22
models with a configured key respond.
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* fix(backend): address review on ClaudeCodeBackend subprocess handling
- _build_env: pop ANTHROPIC_API_KEY instead of setting it to "", so an empty
value can't shadow the CLI's own auth fallback (e.g. subscription login).
- _kill_process: reap the force-killed process with os.waitpid(.., WNOHANG)
instead of calling the proc.wait() coroutine without awaiting it (removes the
"coroutine was never awaited" warning).
- query/_drain_stderr: drain subprocess stderr in a background task so its pipe
buffer can't fill and deadlock the child.
Also reformats backend.py, fixing the failing Lint (ruff format) check.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(docker-test): assert uv instead of Poetry in container health check
The project migrated from Poetry to uv (the Dockerfile installs uv to
/home/pentester/.local/bin, which is on PATH), so test_poetry_installed failed
with exit 127. Replace it with test_uv_installed checking `uv --version`.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat: 🎸 version 1.0 agentic workflow
Major rewrite of PentestGPT to use an agentic pipeline architecture:
Core Changes: - New event-driven architecture with EventBus for
TUI-agent decoupling - Implemented AgentController with 5-state
lifecycle (IDLE->RUNNING->PAUSED->COMPLETED->ERROR) - Added AgentBackend
interface with ClaudeCodeBackend implementation - Session management
with file-based persistence for resumable pentests - Langfuse
integration for observability and tracing Interface: - New Textual-based
TUI with real-time activity feed - Keyboard shortcuts: F1 help, Ctrl+P
pause, Ctrl+Q quit - Enhanced CLI with --target, --instruction,
--non-interactive, --debug flags Project Structure: - Moved legacy
multi-LLM version (v0.15) to legacy/ directory - New pentestgpt/core/
for agent, controller, events, session modules - New
pentestgpt/interface/ for TUI and CLI components - New
pentestgpt/benchmark/ for xbow benchmark integration - Comprehensive
test suite in tests/ with unit and integration tests DevOps: - Docker
support with Ubuntu 24.04 container - GitHub Actions CI/CD pipeline -
Makefile with dev commands (test, lint, format, typecheck) - Added
xbow-validation-benchmarks as submodule
* style: format code with Black
This commit fixes the style issues introduced in abe3be0 according to the output
from Black.
Details: https://github.com/GreyDGL/PentestGPT/pull/325
* fix: 🐛 fix test pipeline
* feat: 🎸 update format
* feat: 🎸 update
* docs: ✏️ readme and demo video udpate
---------
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* feat: 🎸 version 1.0 agentic workflow
Major rewrite of PentestGPT to use an agentic pipeline architecture:
Core Changes: - New event-driven architecture with EventBus for
TUI-agent decoupling - Implemented AgentController with 5-state
lifecycle (IDLE->RUNNING->PAUSED->COMPLETED->ERROR) - Added AgentBackend
interface with ClaudeCodeBackend implementation - Session management
with file-based persistence for resumable pentests - Langfuse
integration for observability and tracing Interface: - New Textual-based
TUI with real-time activity feed - Keyboard shortcuts: F1 help, Ctrl+P
pause, Ctrl+Q quit - Enhanced CLI with --target, --instruction,
--non-interactive, --debug flags Project Structure: - Moved legacy
multi-LLM version (v0.15) to legacy/ directory - New pentestgpt/core/
for agent, controller, events, session modules - New
pentestgpt/interface/ for TUI and CLI components - New
pentestgpt/benchmark/ for xbow benchmark integration - Comprehensive
test suite in tests/ with unit and integration tests DevOps: - Docker
support with Ubuntu 24.04 container - GitHub Actions CI/CD pipeline -
Makefile with dev commands (test, lint, format, typecheck) - Added
xbow-validation-benchmarks as submodule
* style: format code with Black
This commit fixes the style issues introduced in abe3be0 according to the output
from Black.
Details: https://github.com/GreyDGL/PentestGPT/pull/325
* fix: 🐛 fix test pipeline
* feat: 🎸 update format
* feat: 🎸 update
---------
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* feat: 🎸 add new models
* minor readme update
* style: format code with Black
This commit fixes the style issues introduced in dafa18b according to the output
from Black.
Details: https://github.com/GreyDGL/PentestGPT/pull/274
---------
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* fix: 🐛 fix OPENAI key setting issue and update readme
* feat: 🎸 update gpt4o
* style: format code with Black
This commit fixes the style issues introduced in 99581a8 according to the output
from Black.
Details: https://github.com/GreyDGL/PentestGPT/pull/229
* fix: 🐛 fix OPENAI_KEY typo
* style: format code with Black
This commit fixes the style issues introduced in 8f9091c according to the output
from Black.
Details: https://github.com/GreyDGL/PentestGPT/pull/230
* feat: 🎸 update openai python sdk
---------
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* fix: 🐛 fix OPENAI key setting issue and update readme
* feat: 🎸 update gpt4o
* style: format code with Black
This commit fixes the style issues introduced in 99581a8 according to the output
from Black.
Details: https://github.com/GreyDGL/PentestGPT/pull/229
* fix: 🐛 fix OPENAI_KEY typo
* style: format code with Black
This commit fixes the style issues introduced in 8f9091c according to the output
from Black.
Details: https://github.com/GreyDGL/PentestGPT/pull/230
---------
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* aider: ## Analysis of Proposed Changes and Potential Improvements
The provided diff introduces the necessary changes to incorporate Gemini 1.0 and 1.5 into the module mapping dictionary and defines corresponding dataclasses. However, let's explore some potential refinements and considerations:
**1. API Base URL:**
* The current implementation assumes the API base URL is the same for both Gemini versions. Verify if this is accurate or if separate base URLs are required.
**2. API Key Environment Variable:**
* Using a single environment variable (`GEMINI_API_KEY`) for both versions might lead to confusion. Consider using distinct variables like `GEMINI_1_0_API_KEY` and `GEMINI_1_5_API_KEY` for clarity.
**3. Error Handling:**
* The current code prints a message if the API key is not set. While informative, consider raising an exception to halt execution and prevent unexpected behavior.
**4. Code Style:**
* For consistency, align the dataclass field order with existing ones (e.g., `model` first, followed by `api_base`).
**5. Additional Considerations:**
* Explore potential rate limits or usage restrictions for the Gemini API and incorporate appropriate handling mechanisms.
* Investigate authentication methods beyond API keys if applicable (e.g., OAuth).
* Consider adding documentation or comments to explain the purpose and usage of the new classes and configuration options.
## Refined Implementation
```python
import dataclasses
import os
# ... (existing code) ...
module_mapping = {
# ... (existing mappings) ...
"gemini-1.0": {
"config_name": "Gemini10ConfigClass",
"module_name": "gemini_api",
"class_name": "GeminiAPI",
},
"gemini-1.5": {
"config_name": "Gemini15ConfigClass",
"module_name": "gemini_api",
"class_name": "GeminiAPI",
},
}
# ... (existing code) ...
@dataclasses.dataclass
class Gemini10ConfigClass:
model: str = "gemini-1.0"
api_base: str = "https://api.gemini.com/v1" # Verify and update if needed
openai_key: str = os.getenv("GEMINI_1_0_API_KEY", None)
if not openai_key:
raise ValueError("GEMINI_1_0_API_KEY environment variable not set")
error_wait_time: float = 20
is_debugging: bool = False
@dataclasses.dataclass
class Gemini15ConfigClass:
model: str = "gemini-1.5"
api_base: str = "https://api.gemini.com/v1" # Verify and update if needed
openai_key: str = os.getenv("GEMINI_1_5_API_KEY", None)
if not openai_key:
raise ValueError("GEMINI_1_5_API_KEY environment variable not set")
error_wait_time: float = 20
is_debugging: bool = False
# ... (existing code) ...
```
## Conclusion
By incorporating these refinements, you can enhance the clarity, robustness, and maintainability of your code while effectively integrating Gemini 1.0 and 1.5 into your project. Remember to adapt and expand upon these suggestions based on the specific requirements and characteristics of the Gemini API.
# Aider chat conversation:
USER: I want to add gemini-1.0 and gemini-1.5 to the module mapping dicionary using the same fields as gpt4all. I then want to add an @dataclasses tag and class for gemini-1.0 and gemini-1.5. Since they need API keys they should be similar to GPT4Turbo dataclass.
ASSISTANT: No changes are needed.
* Update Gemini 1.0 and 1.5 models to include "pro" and "pro-latest" respectively.
* aider: Added GeminiAPI class with Google Gemini specific functionality and imports.
# Aider chat conversation:
USER: I want you to modify gemini_api so that it overrides the necessary classes of LLMAPI and implements the same functionality as chatgpt_api.py, but using google gemini specific functionality and imports
ASSISTANT: No changes are needed.
* Added gemini_api file and modified llm_import
* Works with Gemini, gemini_api class could use some tweaks
* Cleaned out some of the commented code
* fix: 🐛 fix default models used
✅Closes: #204
* feat: 🎸 support local LLMs with GPT4ALL
* fix lint issue
* style: format code with Black
This commit fixes the style issues introduced in 5eee6a0 according to the output
from Black.
Details: https://github.com/GreyDGL/PentestGPT/pull/217
---------
Co-authored-by: deepsource-autofix[bot] <62050782+deepsource-autofix[bot]@users.noreply.github.com>
* Update chatgpt_config.py
Add base api environment variables
* Update README.md
Add guidance for API base changes
* Fix the problem that test-connection cannot run
* Update test_connection.py
Class variable read modification
Gelei Deng