# Local Deep Research Benchmark Results

> ## 📦 This directory is being archived — new results go to a dedicated repo
>
> Community benchmark results have moved to a dedicated GitHub repository
> (source of truth) with auto-synced leaderboard CSVs on Hugging Face:
>
> 👉 **GitHub (submit PRs here): [LearningCircuit/ldr-benchmarks](https://github.com/LearningCircuit/ldr-benchmarks)**
>
> 👉 **Hugging Face (browse leaderboards): [local-deep-research/ldr-benchmarks](https://huggingface.co/datasets/local-deep-research/ldr-benchmarks)**
>
> The new setup offers:
> - CI validation of every submission (schema, sharing-policy, secrets scan)
> - Auto-generated leaderboard CSVs (per-benchmark and combined) synced to HF
> - Dataset Viewer on Hugging Face for browsing
> - One canonical place to compare runs across SimpleQA, BrowseComp, and xbench-DeepSearch
>
> **Where to submit new results:** open a Pull Request against the
> [GitHub repo](https://github.com/LearningCircuit/ldr-benchmarks). The
> same YAML export from the LDR web UI (`/benchmark` → YAML button) works
> unchanged — just drop it under `results/{dataset}/{strategy}/{search_engine}/`.
>
> **What stays here:** the existing `.yaml` result files in this folder
> and `benchmark_template.yaml` are kept as a historical archive and for
> reference. They are not being deleted. New submissions, however, should
> go to the new repo so results stay consolidated in one place.
>
> **Why the move:** keeping benchmark data in the code repo bloats git
> history on every clone, even though the data is static. A dedicated
> repo solves this cleanly and gives us a CI pipeline + viewer + leaderboards
> built for exactly this purpose.

---

## Historical archive (pre-migration)

This directory contains community-contributed benchmark results for various LLMs tested with Local Deep Research.

## Contributing Your Results

### Easy Method (v0.6.0+)
1. Run benchmarks using the LDR web interface at `/benchmark`
2. Go to Benchmark Results page
3. Click the green "YAML" button next to your completed benchmark
4. Review the downloaded file and add any missing info (hardware specs are optional)
5. Submit a PR to add your file to this directory

### Manual Method
1. Run benchmarks using the LDR web interface at `/benchmark`
2. Copy `benchmark_template.yaml` to a new file named: `[model_name]_[date].yaml`
   - Example: `llama3.3-70b-q4_2025-01-23.yaml`
   - Optional: Include your username: `johnsmith_llama3.3-70b-q4_2025-01-23.yaml`
3. Fill in your results manually
4. Submit a PR to add your file to this directory

## Important Guidelines

- **Test both strategies**: focused-iteration and source-based
- **Use consistent settings**: Start with 20-50 SimpleQA questions
- **Include all metadata**: Hardware specs, configuration, and versions are crucial
- **Be honest**: Negative results are as valuable as positive ones
- **Add notes**: Your observations help others understand the results
- **Review for PII**: If you include individual examples in your export, review the file for any personally identifiable information before submitting a PR

## Recommended Test Configuration

### For Large Models (70B+)
- Context Window: 32768+ tokens
- Focused-iteration: 8 iterations, 5 questions each
- Source-based: 5 iterations, 3 questions each

### For Smaller Models (<70B)
- Context Window: 16384+ tokens (adjust based on model)
- Focused-iteration: 5 iterations, 3 questions each
- Source-based: 3 iterations, 3 questions each

## Current Baseline

- **Model**: GPT-4.1-mini
- **Strategy**: focused-iteration (8 iterations, 5 questions)
- **Accuracy**: ~95% on SimpleQA (preliminary results from 20-100 question samples)
- **Search**: SearXNG
- **Verified by**: 2 independent testers

## Understanding the Results

### Accuracy Ranges
- **90%+**: Excellent - matches GPT-4 performance
- **80-90%**: Very good - suitable for most research tasks
- **70-80%**: Good - works well with human oversight
- **<70%**: Limited - may struggle with complex research

### Common Issues
- **Low accuracy**: Often due to insufficient context window
- **Timeouts**: Model too slow for iterative research
- **Memory errors**: Reduce context window or batch size
- **Rate limiting**: SearXNG may throttle excessive requests

## Viewing Results

Browse the YAML files in this directory to see how different models perform. Look for patterns like:
- Which quantization levels maintain accuracy
- Minimum viable model size for research tasks
- Best strategy for different model architectures
- Hardware requirements for acceptable performance

## Questions?

Join our [Discord](https://discord.gg/ttcqQeFcJ3) to discuss results and get help with benchmarking.