9a95fc7ba2
chore(deps): bump pytest from 9.0.2 to 9.0.3 in /plugins/plugin-eval
...
Bumps [pytest](https://github.com/pytest-dev/pytest ) from 9.0.2 to 9.0.3.
- [Release notes](https://github.com/pytest-dev/pytest/releases )
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pytest-dev/pytest/compare/9.0.2...9.0.3 )
---
updated-dependencies:
- dependency-name: pytest
dependency-version: 9.0.3
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-04-14 03:11:04 +00:00
696049247d
chore(deps): bump cryptography in /plugins/plugin-eval
...
Bumps [cryptography](https://github.com/pyca/cryptography ) from 46.0.6 to 46.0.7.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pyca/cryptography/compare/46.0.6...46.0.7 )
---
updated-dependencies:
- dependency-name: cryptography
dependency-version: 46.0.7
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-04-08 22:58:57 +00:00
1cb03edc32
Merge pull request #467 from wshobson/dependabot/uv/plugins/plugin-eval/cryptography-46.0.6
...
chore(deps): bump cryptography from 46.0.5 to 46.0.6 in /plugins/plugin-eval
2026-04-03 13:00:00 -04:00
7cfc81a6b3
Merge pull request #468 from wshobson/dependabot/uv/plugins/plugin-eval/pygments-2.20.0
...
chore(deps): bump pygments from 2.19.2 to 2.20.0 in /plugins/plugin-eval
2026-04-03 12:59:46 -04:00
4893f601f0
chore(deps): bump anthropic in /plugins/plugin-eval
...
Bumps [anthropic](https://github.com/anthropics/anthropic-sdk-python ) from 0.86.0 to 0.87.0.
- [Release notes](https://github.com/anthropics/anthropic-sdk-python/releases )
- [Changelog](https://github.com/anthropics/anthropic-sdk-python/blob/main/CHANGELOG.md )
- [Commits](https://github.com/anthropics/anthropic-sdk-python/compare/v0.86.0...v0.87.0 )
---
updated-dependencies:
- dependency-name: anthropic
dependency-version: 0.87.0
dependency-type: direct:production
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-04-01 21:21:29 +00:00
c3ba18110b
chore(deps): bump pygments from 2.19.2 to 2.20.0 in /plugins/plugin-eval
...
Bumps [pygments](https://github.com/pygments/pygments ) from 2.19.2 to 2.20.0.
- [Release notes](https://github.com/pygments/pygments/releases )
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES )
- [Commits](https://github.com/pygments/pygments/compare/2.19.2...2.20.0 )
---
updated-dependencies:
- dependency-name: pygments
dependency-version: 2.20.0
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-03-30 20:28:41 +00:00
584a79703c
chore(deps): bump cryptography in /plugins/plugin-eval
...
Bumps [cryptography](https://github.com/pyca/cryptography ) from 46.0.5 to 46.0.6.
- [Changelog](https://github.com/pyca/cryptography/blob/main/CHANGELOG.rst )
- [Commits](https://github.com/pyca/cryptography/compare/46.0.5...46.0.6 )
---
updated-dependencies:
- dependency-name: cryptography
dependency-version: 46.0.6
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com >
2026-03-29 04:29:22 +00:00
91fe43e152
docs(plugin-eval): add CLAUDE.md, full evaluation framework docs, and README updates
...
- Create CLAUDE.md with project conventions and PluginEval quick reference
- Add docs/plugin-eval.md with complete framework documentation (layers,
dimensions, scoring, anti-patterns, statistical methods, project structure)
- Add PluginEval section to main README under What's New
- Expand plugin-eval/README.md with quick start and command table
2026-03-26 17:40:56 -04:00
50a9730f05
fix(plugin-eval): address PR review comments
...
P1: Normalize Monte Carlo nested dict sub_scores to flat dimension→float
before blending. Prevents TypeError when deep/thorough runs include MC.
P2: Plugin-level eval always shows "Estimated" confidence since only
Layer 1 runs (judge/MC are per-skill only). Prevents misleading
"Certified" label on plugin-level certify runs.
2026-03-26 17:33:24 -04:00
500ccf1f51
improve(plugin-eval): tune evaluation-methodology skill for higher score
2026-03-26 13:05:51 -04:00
2e04276283
improve(plugin-eval): expand evaluation-methodology skill from stub to comprehensive reference
...
- Expanded SKILL.md to 480 lines covering all three evaluation layers,
composite scoring formula with blend weights, dimension grade interpretation,
all five anti-pattern flags with fix guidance, Elo ranking mechanics, CLI
reference with code examples, and a troubleshooting section
- Added references/rubrics.md (510 lines) with full anchored rubrics for all
four judge dimensions (triggering accuracy, orchestration fitness, output
quality, scope calibration) including per-level examples and calibration norms
- Updated description to include specific trigger contexts for autonomous invocation
2026-03-26 12:11:12 -04:00
b97707d7fc
improve(plugin-eval): add rubrics.md reference for evaluation-methodology skill
2026-03-26 12:08:30 -04:00
c3db0946ca
improve(plugin-eval): expand evaluation-methodology from stub to comprehensive reference
2026-03-26 12:07:01 -04:00
5e3e83d3c9
improve: enhance 5 bottom skills + add technical depth signal
...
Skills improved: saga-orchestration, architecture-patterns,
deployment-pipeline-design, kpi-dashboard-design, incident-runbook-templates
Algorithm: add multi-language depth bonus, table/decision matrix detection
to structural_completeness scoring. Addresses validator finding that
scoring penalizes expert-audience reference skills.
2026-03-26 12:05:49 -04:00
aaa3e07c0f
tune(plugin-eval): improve score differentiation in static analyzer
...
- Pushiness: reward specific use-case enumeration (not just trigger presence)
- Orchestration: raise baseline to 0.70 (pure workers are the norm)
- Coherence: raise baseline to 0.50 (standalone skills are fine)
- Result: Mean 72.1→73.6, Skills 75+: 21%→36%, better top/bottom spread
2026-03-26 00:56:56 -04:00
4ccf9424a4
fix(plugin-eval): address code review findings
...
- Fix _beta_cdf Simpson's rule lower bound (was 0.001, now 0.0 for a>=1)
- Fix parse_agent crash on YAML-list tools: field (list vs string)
- Replace hardcoded absolute paths with ${CLAUDE_PLUGIN_ROOT}
- Register plugin-eval in marketplace.json
2026-03-26 00:54:01 -04:00
60cb369070
refactor(plugin-eval): simplify — parse once, deduplicate composite, single event loop
2026-03-25 22:13:32 -04:00
082595b5da
fix(plugin-eval): fix MISSING_TRIGGER false positives
...
The regex only matched "use when" but not "use this skill when" —
a common pattern in the codebase. Updated to match all trigger
variants: "use when", "use this skill when", "use proactively",
"trigger when".
Eliminates 17 false-positive MISSING_TRIGGER flags.
2026-03-25 21:47:23 -04:00
111140fa83
fix(plugin-eval): calibrate static scoring against real plugin corpus
...
- Frontmatter: boost name/description scoring, reward longer descriptions
- Orchestration: raise baseline to 0.65 (most skills are workers), add code block bonus
- Progressive disclosure: skills under 600 lines no longer penalized for missing refs
- Pushiness: add "Use this skill when..." pattern, context-based triggers
- Engine: exclude unmeasured dimensions from composite (show "—" not fake "F")
- Engine: renormalize weights to measured dimensions only
Before: corpus range 52-71, After: corpus range 63-80
2026-03-25 21:30:48 -04:00
8fd490b7c8
fix(plugin-eval): restructure LLM judge as Claude Code native agent
...
Layer 2 now runs via eval-judge agent using Max plan session auth
instead of Agent SDK subprocess calls. No API key required.
Also fixes composite scoring to exclude unmeasured dimensions.
2026-03-25 20:57:01 -04:00
5aff99cf4f
test(plugin-eval): add end-to-end tests against real plugins
2026-03-25 17:54:24 -04:00
38cb914673
feat(plugin-eval): implement Layer 3 Monte Carlo simulation with statistical analysis
...
Adds MonteCarloAnalyzer with SimResult/MonteCarloConfig dataclasses, run_simulation
async helper, and _compute_statistics using Wilson score CI, bootstrap CI,
Clopper-Pearson CI, and coefficient of variation. Wires MC layer into evaluate_skill
for Depth.DEEP and Depth.THOROUGH runs (50 and 100 runs respectively).
2026-03-25 17:52:15 -04:00
b4afcc4063
feat(plugin-eval): add Claude Code plugin shell (commands, agents, skill)
...
Add plugin.json, three commands (eval/compare/certify), eval-orchestrator agent,
and evaluation-methodology skill to wire plugin-eval into the claude-agents ecosystem.
2026-03-25 17:51:28 -04:00
5849ee3546
feat(plugin-eval): implement Layer 2 LLM judge with Agent SDK and model tiering
2026-03-25 17:48:53 -04:00
5dc9e1fe58
feat(plugin-eval): implement Elo ranking system and corpus management
2026-03-25 17:48:18 -04:00
14bfce6e93
feat(plugin-eval): add reporter (JSON/MD/HTML) and Typer CLI with score/certify/compare/init commands
2026-03-25 17:45:34 -04:00
88061a68ce
feat(plugin-eval): implement eval engine with composite scoring and layer blending
...
Adds EvalEngine that coordinates static analysis, blends layer scores
across dimensions with renormalized weights, and produces PluginEvalResult
with composite score, badge, and per-dimension grades. Layer 2/3 stubs
ready for Tasks 8/9.
2026-03-25 17:42:11 -04:00
882f344786
feat(plugin-eval): implement Layer 1 static analysis with anti-pattern detection
2026-03-25 17:39:58 -04:00
118595e4e8
feat(plugin-eval): add markdown/frontmatter parser for skills, agents, plugins
2026-03-25 17:36:23 -04:00
b54562428d
feat(plugin-eval): add statistics module (bootstrap, Wilson, Clopper-Pearson, kappa)
2026-03-25 17:34:17 -04:00
2b270aedb2
feat(plugin-eval): add Pydantic data models for all eval layers
2026-03-25 17:33:32 -04:00
ef6fe431c3
feat(plugin-eval): scaffold project with uv, ruff, ty
2026-03-25 17:31:00 -04:00
dependabot[bot]