`plugin-eval certify <plugin-dir>` advertises a deep, three-layer
evaluation (static + judge + Monte Carlo) but `EvalEngine.evaluate_plugin`
only runs the static layer regardless of requested depth, since judge
and Monte Carlo are per-skill primitives. The docstring on that method
acknowledges this, but nothing surfaces it to the user:
- The CLI emits no warning.
- The markdown report prints `**Depth:** deep` even though only the
static layer ran.
- The user's only signal is a footnote ("No model usage") near the
bottom of the report and the `Confidence: Estimated` row — both
easy to miss when the requested depth said otherwise.
This change makes the downgrade impossible to miss without changing
the underlying eval behaviour (per-skill judge aggregation is a larger
feature, not a bug fix):
1. **CLI warning to stderr.** `_run_score` detects plugin-target runs
at non-quick depth and prints a yellow `warning:` line to stderr
naming the skipped layers and the workaround (run on a single skill
to get the deeper layers).
2. **In-band markdown callout.** `Reporter` now derives the *effective*
depth from the set of layers actually present in the result. When it
differs from the requested depth, the report header reads
`Depth: deep (requested) → quick (effective)` and a `> Note:` block
above the score table explains why and how to get the deeper layers.
3. **Effective-depth helper.** `_effective_depth(result)` maps the set
of layer names (`static` / `judge` / `monte_carlo`) back to a `Depth`
value, so the reporter never has to trust `result.config.depth` when
describing what actually ran.
Tests:
- `TestDepthDowngradeWarning` (4 tests): asserts the helper, the
"no warning when honored" path, and the warning content for both
deep and standard requests.
- `TestCLI` (2 new tests): asserts the stderr warning is emitted on
plugin-level certify and is *not* emitted at quick depth.
Full plugin-eval suite (75 tests) passes.
Seth Hobson