cline--cline
wehub-resource-sync
153 行
4.3 KiB
Markdown
153 行
4.3 KiB
Markdown
# Cline Evaluation Framework
|
|
|
|
A layered testing system for measuring Cline's performance at different levels.
|
|
|
|
> Note: Smoke tests (Layer 2) are partially disabled while the eval framework is repointed at the new SDK CLI. The scenarios under `evals/smoke-tests/` are preserved and `npm run eval:smoke:run` still works against whatever `cline` is on `$PATH` (install with `npm i -g cline`). The old build-and-link helpers and the auto-running `cline-evals-regression.yml` workflow are off until someone wires the build step at the new SDK CLI.
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
evals/
|
|
├── smoke-tests/ # Quick provider validation (minutes)
|
|
│ ├── run-smoke-tests.ts
|
|
│ └── scenarios/ # 5 curated test scenarios
|
|
│
|
|
├── e2e/ # Full E2E with cline-bench (hours)
|
|
│ └── run-cline-bench.ts
|
|
│
|
|
├── cline-bench/ # Real-world tasks (git submodule)
|
|
│ └── tasks/ # 12 production bug fixes
|
|
│
|
|
├── analysis/ # Metrics and reporting framework
|
|
│ ├── src/
|
|
│ │ ├── metrics.ts # pass@k, pass^k calculations
|
|
│ │ ├── classifier.ts # Failure pattern matching
|
|
│ │ └── reporters/ # Markdown, JSON output
|
|
│ └── patterns/
|
|
│ └── cline-failures.yaml
|
|
│
|
|
└── baselines/ # Performance baselines for regression detection
|
|
```
|
|
|
|
## Test Layers
|
|
|
|
### Layer 1: Contract Tests (Unit)
|
|
|
|
Location: `src/core/api/transform/__tests__/`
|
|
|
|
Tests API transform logic without LLM calls:
|
|
- Thinking trace preservation
|
|
- Tool call parsing (XML, native formats)
|
|
- Provider format conversions
|
|
|
|
```bash
|
|
npm run test:unit -- --grep "Thinking\|Tool Call"
|
|
```
|
|
|
|
### Layer 2: Smoke Tests (Minutes)
|
|
|
|
Location: `evals/smoke-tests/`
|
|
|
|
Quick validation across providers with real LLM calls:
|
|
- 5 curated scenarios
|
|
- 3 trials per test for pass@k metrics
|
|
- Runs the `cline` CLI with `--config`, `-y`, `-t`, and `-m`
|
|
|
|
```bash
|
|
# Set API key (Cline provider)
|
|
export CLINE_API_KEY=sk-...
|
|
|
|
# Run smoke tests
|
|
npm run eval:smoke:run
|
|
|
|
# Run specific scenario
|
|
npm run eval:smoke:run -- --scenario 01-create-file
|
|
|
|
# Run with specific model (overrides per-scenario models)
|
|
npm run eval:smoke:run -- --model anthropic/claude-sonnet-4.5
|
|
```
|
|
|
|
### Layer 3: E2E Tests (Hours)
|
|
|
|
Location: `evals/e2e/` + `evals/cline-bench/`
|
|
|
|
Full agent tests on production-grade tasks:
|
|
- 12 real-world coding problems
|
|
- Docker/Daytona execution via Harbor
|
|
- Nightly CI runs
|
|
|
|
```bash
|
|
# Prerequisites: Python 3.13, Harbor, Docker
|
|
npm run eval:e2e
|
|
|
|
# Specific task
|
|
npm run eval:e2e -- --tasks discord
|
|
|
|
# Different provider
|
|
npm run eval:e2e -- --provider openai --model gpt-4o
|
|
```
|
|
|
|
## Metrics
|
|
|
|
The framework calculates:
|
|
|
|
| Metric | Formula | Interpretation |
|
|
|--------|---------|----------------|
|
|
| **pass@k** | P(≥1 of k passes) | Solution finding capability |
|
|
| **pass^k** | P(all k pass) | Reliability |
|
|
| **Flakiness** | Entropy of pass rate | Consistency |
|
|
|
|
With 3 trials:
|
|
- All pass → `pass` (reliable)
|
|
- All fail → `fail` (broken)
|
|
- Mixed → `flaky` (needs investigation)
|
|
|
|
## CI Integration
|
|
|
|
- Current PR gate: contract tests only
|
|
- Smoke test CI: temporarily disabled while the workflow is repointed at the SDK CLI
|
|
- Nightly: E2E tests with cline-bench are not yet implemented, see TODO
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
# Run all fast tests
|
|
npm run test:unit
|
|
npm run eval:smoke:run
|
|
|
|
# Run E2E (requires setup)
|
|
cd evals/cline-bench
|
|
# Follow README.md for Harbor setup
|
|
npm run eval:e2e
|
|
```
|
|
|
|
## Adding Tests
|
|
|
|
### Smoke Test Scenario
|
|
|
|
1. Create `evals/smoke-tests/scenarios/<name>/config.json`
|
|
2. Add optional `template/` directory with starting files
|
|
3. Run to verify: `npm run eval:smoke:run -- --scenario <name>`
|
|
|
|
### Contract Test
|
|
|
|
1. Add to `src/core/api/transform/__tests__/`
|
|
2. Run: `npm run test:unit -- --grep "YourTest"`
|
|
|
|
### E2E Task
|
|
|
|
Contribute to [cline/cline-bench](https://github.com/cline/cline-bench)
|
|
|
|
## Resources
|
|
|
|
- [cline-bench tasks](evals/cline-bench/README.md)
|
|
- [Smoke test scenarios](evals/smoke-tests/README.md)
|
|
|
|
## TODO
|
|
|
|
- [ ] **Nightly E2E CI**: Add scheduled workflow for cline-bench tests
|
|
- Requires: Docker runner, Harbor setup, ~1-2 hour timeout
|
|
- Should run on schedule (e.g., nightly) not per-PR
|
|
- Separate secrets for E2E environment
|
|
- [ ] **Native tool calling smoke tests**: Add CLI support for `native_tool_call_enabled` setting to test Claude 4 with native tools
|