There was no way to measure whether skills trigger correctly, stay
distinct, or change agent behavior. This adds evals, aligned with what
the community has converged on, with a deterministic CI tier on top:
- evals/cases/<skill>.json for all 24 skills. The evals[] block uses
Anthropic skill-creator's evals.json schema verbatim (id, prompt,
expected_output, expectations[]) so its runner, benchmarks, and eval
viewer work against our files unmodified. A trigger block (this
repo's extension) adds positive/negative routing prompts per skill.
- scripts/run-evals.js, zero-dependency runner:
Tier 2 (CI): trigger evals via stemmed TF-IDF ranking over skill
descriptions (positive prompts must rank top-k, negative prompts
must not rank first), catalog collision detection between skill
descriptions, schema and coverage checks.
Tier 3 (opt-in): --behavioral <skill> executes each eval through
headless claude -p and grades the transcript against expectations[]
(superpowers-style); --dry-run previews without spending tokens.
- CI: run the deterministic tier in the validate-skills job.
- Docs: evals/README.md defines the framework and prior art;
CONTRIBUTING requires an eval file for new skills (warning-level in
the runner until in-flight skill PRs clear); CLAUDE.md pointers.
Current baseline: 120 checks pass, 85% trigger rank-1 rate across 72
positive prompts, zero catalog collisions.
The validate-skills job pinned actions/checkout@v4 while the other two
jobs in the same workflow (test-fresh-install, test-plugin-structure)
use @v6. Align all three on @v6 for consistency.
- scripts/validate-skills.js: zero-dependency Node.js validator that
checks every skill for valid frontmatter, description length (≤1024),
required sections (Overview, When to Use, Common Rationalizations,
Red Flags, Verification), and dead cross-skill references
- Skills with type:meta or exempt:sections in frontmatter skip section
checks; applied to using-agent-skills (meta) and idea-refine (legacy
structure predating the anatomy spec)
- CI: validate-skills job runs before plugin-manifest validation and
blocks merge on any error; uses Node 20, no npm install required
Runs on every push and PR:
1. Validates marketplace and plugin manifests
2. Tests marketplace add + plugin install end-to-end
No ANTHROPIC_API_KEY required — these are CLI/filesystem
operations, not LLM calls.
ZhiyaoWen999