addyosmani--agent-skills
Addy Osmani
45e1449138
There was no way to measure whether skills trigger correctly, stay distinct, or change agent behavior. This adds evals, aligned with what the community has converged on, with a deterministic CI tier on top: - evals/cases/<skill>.json for all 24 skills. The evals[] block uses Anthropic skill-creator's evals.json schema verbatim (id, prompt, expected_output, expectations[]) so its runner, benchmarks, and eval viewer work against our files unmodified. A trigger block (this repo's extension) adds positive/negative routing prompts per skill. - scripts/run-evals.js, zero-dependency runner: Tier 2 (CI): trigger evals via stemmed TF-IDF ranking over skill descriptions (positive prompts must rank top-k, negative prompts must not rank first), catalog collision detection between skill descriptions, schema and coverage checks. Tier 3 (opt-in): --behavioral <skill> executes each eval through headless claude -p and grades the transcript against expectations[] (superpowers-style); --dry-run previews without spending tokens. - CI: run the deterministic tier in the validate-skills job. - Docs: evals/README.md defines the framework and prior art; CONTRIBUTING requires an eval file for new skills (warning-level in the runner until in-flight skill PRs clear); CLAUDE.md pointers. Current baseline: 120 checks pass, 85% trigger rank-1 rate across 72 positive prompts, zero catalog collisions.
74 行
1.8 KiB
YAML
74 行
1.8 KiB
YAML
name: Test Plugin Installation
|
|
|
|
on:
|
|
push:
|
|
pull_request:
|
|
workflow_dispatch:
|
|
|
|
jobs:
|
|
validate-skills:
|
|
name: Validate skill content
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
|
|
- name: Set up Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '20'
|
|
|
|
- name: Validate all skills
|
|
run: node scripts/validate-skills.js
|
|
|
|
- name: Run skill evals (trigger + routing)
|
|
run: node scripts/run-evals.js
|
|
|
|
validate-commands:
|
|
name: Validate command parity and description sync
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
|
|
- name: Set up Node.js
|
|
uses: actions/setup-node@v4
|
|
with:
|
|
node-version: '20'
|
|
|
|
- name: Validate commands across all tool directories
|
|
run: node scripts/validate-commands.js
|
|
|
|
validate:
|
|
name: Validate plugin structure
|
|
needs: [validate-skills, validate-commands]
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
|
|
- name: Install Claude Code
|
|
run: npm install -g @anthropic-ai/claude-code
|
|
|
|
- name: Validate marketplace and plugin manifests
|
|
run: claude plugin validate .
|
|
|
|
test-install:
|
|
name: Test plugin installation
|
|
runs-on: ubuntu-latest
|
|
needs: validate
|
|
steps:
|
|
- uses: actions/checkout@v6
|
|
|
|
- name: Install Claude Code
|
|
run: npm install -g @anthropic-ai/claude-code
|
|
|
|
- name: Configure git to use HTTPS
|
|
run: git config --global url."https://github.com/".insteadOf "git@github.com:"
|
|
|
|
- name: Add marketplace
|
|
run: claude plugin marketplace add ./
|
|
|
|
- name: List marketplaces
|
|
run: claude plugin marketplace list
|
|
|
|
- name: Install plugin
|
|
run: claude plugin install agent-skills@addy-agent-skills --scope user
|