文件历史

8 次代码提交

作者 SHA1 备注 提交日期
Seth Hobson 511f8345cf docs: refresh model references for Claude Sonnet 5 and Fable 5 (#612)
* feat(adapters): map sonnet tier to Claude Sonnet 5; Copilot fable to claude-fable-5

Claude Sonnet 5 is GA (Copilot GA 2026-06-30) and Copilot now serves
Fable 5 natively, so the sonnet/inherit aliases move off Sonnet 4.6 and
the Copilot fable alias no longer falls back to Opus 4.8. Catalog
verification date bumped to 2026-07.

* fix(plugin-eval): judge sonnet tier resolves to claude-sonnet-5

* docs: refresh model alias table and tier counts (Sonnet 5, five-tier, 66/22/52)

* docs(plugins): update model examples to Sonnet 5 / Opus 4.8

Replace claude-sonnet-4-6 with claude-sonnet-5 and Opus 4.7 mentions
with 4.8 in llm-application-dev; replace retired claude-3-5-sonnet in
performance-testing-review's ai-review. Drop temperature args from
migrated examples — Sonnet 5 rejects non-default sampling params.
2026-07-07 12:17:53 -04:00
Seth Hobson bdf6e19a83 fix: address CodeRabbit/codex review feedback on #591/#598 work
- judge: guard _measured_score against non-dict JSON; strip whitespace text
  before the result fallback (typed result param as Any for ty).
- monte_carlo: use the shared result fallback for activation/quality; sum real
  SDK usage token fields via _sdk.usage_total_tokens (no fake total_tokens=0).
- tests: importorskip claude_agent_sdk so dev-only (no llm extra) collection
  doesn't fail; add tests for non-dict scores, whitespace/result fallback,
  result-only activation, and token summing.
- run-tests.sh: cd guard + multiline-tolerant in-on-String detection.
- docs: cedar-validate if/else, repo-root-safe path, and #598 narrative
  corrected to the verified conclusion (reporter essentially correct;
  protect-mcp evaluate/sign subcommands absent -> separate issue).
2026-06-25 20:53:10 +09:00
Seth Hobson bee43e90d2 test(plugin-eval): minor test hygiene (score assertion, unused imports, import order) 2026-06-25 13:38:47 +09:00
Seth Hobson 5571d9a673 refactor(plugin-eval): remove dead auth/model_tier config (#591) 2026-06-25 13:30:28 +09:00
Seth Hobson 09dade9bcd fix(plugin-eval): omit unmeasured judge dimensions instead of scoring 0.5 (#591) 2026-06-25 13:22:01 +09:00
Seth Hobson bceb724a96 fix(plugin-eval): include partial raw text in errored judge marker (#591) 2026-06-25 13:19:21 +09:00
Seth Hobson 90ebe9dbc2 fix(plugin-eval): read judge LLM text from AssistantMessage/ResultMessage (#591)
ResultMessage has no .content attribute, so the judge always read empty
text and silently fell back to 0.5. Extract from AssistantMessage TextBlocks
with ResultMessage.result fallback; mark errored/empty/non-JSON as unmeasured.
2026-06-25 13:16:18 +09:00
Seth Hobson 5849ee3546 feat(plugin-eval): implement Layer 2 LLM judge with Agent SDK and model tiering 2026-03-25 17:48:53 -04:00