文件历史

6 次代码提交

作者 SHA1 备注 提交日期
Seth Hobson 511f8345cf docs: refresh model references for Claude Sonnet 5 and Fable 5 (#612)
* feat(adapters): map sonnet tier to Claude Sonnet 5; Copilot fable to claude-fable-5

Claude Sonnet 5 is GA (Copilot GA 2026-06-30) and Copilot now serves
Fable 5 natively, so the sonnet/inherit aliases move off Sonnet 4.6 and
the Copilot fable alias no longer falls back to Opus 4.8. Catalog
verification date bumped to 2026-07.

* fix(plugin-eval): judge sonnet tier resolves to claude-sonnet-5

* docs: refresh model alias table and tier counts (Sonnet 5, five-tier, 66/22/52)

* docs(plugins): update model examples to Sonnet 5 / Opus 4.8

Replace claude-sonnet-4-6 with claude-sonnet-5 and Opus 4.7 mentions
with 4.8 in llm-application-dev; replace retired claude-3-5-sonnet in
performance-testing-review's ai-review. Drop temperature args from
migrated examples — Sonnet 5 rejects non-default sampling params.
2026-07-07 12:17:53 -04:00
Seth Hobson bdf6e19a83 fix: address CodeRabbit/codex review feedback on #591/#598 work
- judge: guard _measured_score against non-dict JSON; strip whitespace text
  before the result fallback (typed result param as Any for ty).
- monte_carlo: use the shared result fallback for activation/quality; sum real
  SDK usage token fields via _sdk.usage_total_tokens (no fake total_tokens=0).
- tests: importorskip claude_agent_sdk so dev-only (no llm extra) collection
  doesn't fail; add tests for non-dict scores, whitespace/result fallback,
  result-only activation, and token summing.
- run-tests.sh: cd guard + multiline-tolerant in-on-String detection.
- docs: cedar-validate if/else, repo-root-safe path, and #598 narrative
  corrected to the verified conclusion (reporter essentially correct;
  protect-mcp evaluate/sign subcommands absent -> separate issue).
2026-06-25 20:53:10 +09:00
Seth Hobson 55865acb40 style(plugin-eval): ruff import-order/format on branch-touched test files 2026-06-25 14:40:54 +09:00
Seth Hobson bee43e90d2 test(plugin-eval): minor test hygiene (score assertion, unused imports, import order) 2026-06-25 13:38:47 +09:00
Seth Hobson aa3768404f fix(plugin-eval): fix Monte Carlo SDK message extraction (#591) 2026-06-25 13:26:38 +09:00
Seth Hobson 38cb914673 feat(plugin-eval): implement Layer 3 Monte Carlo simulation with statistical analysis
Adds MonteCarloAnalyzer with SimResult/MonteCarloConfig dataclasses, run_simulation
async helper, and _compute_statistics using Wilson score CI, bootstrap CI,
Clopper-Pearson CI, and coefficient of variation. Wires MC layer into evaluate_skill
for Depth.DEEP and Depth.THOROUGH runs (50 and 100 runs respectively).
2026-03-25 17:52:15 -04:00