mirrors / vibrantlabsai--ragas

项目文件夹

vibrantlabsai--ragas
文件
main
T
wehub-resource-sync e4f55014ae chore: import upstream snapshot with attribution
2026-07-13 13:35:10 +08:00

12 KiB
原始文件 永久链接 Blame 文件历史

Development Guide for Ragas Monorepo

This comprehensive guide covers development workflows for the Ragas monorepo, designed for both human developers and AI agents.

Quick Start (for Developers)

# 1. Clone and enter the repository
git clone https://github.com/vibrantlabsai/ragas.git

# 2. Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh

# 3. Choose your installation type:

# RECOMMENDED: Minimal dev setup (fast)
make install-minimal

# FULL: Complete dev environment (comprehensive)
make install

# 4. Verify everything works
make check

# 5. Start developing!
make help  # See all available commands

Quick Start (for AI Agents)

AI agents working with this codebase should use these standardized commands:

# Essential commands for AI development
make help           # See all available targets
make install-minimal # Minimal dev setup (fast)
make install        # Full environment (modern uv sync)
make check          # Quick health check (format + type)
make test           # Run all tests
make run-ci         # Full CI pipeline locally

# Individual development tasks
make format         # Format and lint all code
make type           # Type check all code
make clean          # Clean generated files

Key Points for AI Agents:

  • Always use make commands rather than direct tool invocation
  • Use uv run prefix for any direct Python tool usage
  • Check make help for the complete command reference
  • The CI pipeline uses the same commands as local development

Monorepo Architecture

This repository is organized as a single project with integrated experimental features:

/                              # Main ragas project
├── src/ragas/                 # Main source code
│   └── experimental/          # Experimental features
├── tests/                     # Tests (unit, e2e, benchmarks)
│   └── experimental/          # Experimental tests
├── examples/                  # Example code
├── pyproject.toml             # Dependencies and configuration
├── docs/                      # Documentation
├── .github/workflows/         # CI/CD pipeline
├── Makefile                   # Build commands
└── CLAUDE.md                  # AI assistant instructions

Project Components

  • Ragas Core: The main evaluation toolkit for LLM applications (in src/ragas/)
  • Ragas Experimental: Advanced features integrated at src/ragas/experimental/
  • Infrastructure: Single CI/CD, documentation, and build system

Examples Package (ragas-examples)

  • Lives under examples/ as an installable package ragas-examples
  • Published independently to PyPI via GitHub Actions workflow publish-examples.yml
  • Versioning via Git tags with prefix examples-v (e.g., examples-v0.1.0)
  • Local development: uv pip install -e . -e ./examples
  • Run examples: python -m ragas_examples.benchmark_llm.prompt

Development Environment Setup

Prerequisites

  • Python 3.9+
  • uv (recommended) or pip
  • Git

Setup Process

Option 1: Using Make (Recommended)

# Recommended: Minimal dev setup
make install-minimal

# Full: Complete environment
make install

Option 2: Manual Setup

# Install uv if not available
curl -LsSf https://astral.sh/uv/install.sh | sh

# Minimal dev: Core + essential dev tools
uv pip install -e ".[dev-minimal]"

# Full dev: Everything (uses modern uv sync)
uv sync --group dev

Which Option to Choose?

Use make install-minimal if you're:

  • Contributing to ragas development
  • Need testing and linting tools
  • Want fast CI/CD builds
  • Working on code quality, docs, or basic features

Use make install if you're:

  • Working on ML features requiring the full stack
  • Need observability tools (Phoenix, MLflow)
  • Developing with notebooks and advanced integrations
  • Want the complete development environment

Installation Methods Explained

  • install-minimal: Uses uv pip install -e ".[dev-minimal]" for selective minimal dev dependencies
  • install: Uses uv sync --group dev for complete modern dependency management

Verification

make check  # Runs format + type checking
make test   # Runs all tests

Available Commands Reference

Run make help to see all targets. Here are the essential commands:

Setup & Installation

  • make install-minimal - Install minimal dev setup (recommended)
  • make install - Install full environment with uv sync (complete)

Code Quality

  • make format - Format and lint all code (includes unused import cleanup)
  • make type - Type check all code
  • make check - Quick health check (format + type, no tests)

Testing

  • make test - Run all unit tests
  • make test-e2e - Run end-to-end tests
  • make benchmarks - Run performance benchmarks
  • make benchmarks-docker - Run benchmarks in Docker

CI/Build

  • make run-ci - Run complete CI pipeline locally
  • make clean - Clean all generated files

Documentation

  • make build-docs - Build all documentation
  • make build-docs-pdf - Build documentation with PDF export (requires WeasyPrint)
  • make serve-docs - Serve documentation locally
  • See docs/community/pdf_export.md for PDF export details and limitations

Development Workflows

Daily Development

# 1. Start your work
git checkout -b feature/your-feature

# 2. Make changes to code

# 3. Check your work
make check           # Format and type check
make test            # Run tests

# 4. Commit and push
git add .
git commit -m "feat: your feature description"
git push origin feature/your-feature

Before Submitting PR

make run-ci          # Run full CI pipeline
# Ensure all checks pass before creating PR

Development Workflow

# Use the Makefile for all development
make help           # See available commands
make format         # Format all code (core + experimental)
make type           # Type check all code
make test           # Run all tests (core + experimental)
make check          # Quick format + type check
make run-ci         # Run full CI pipeline

# Or use direct commands for specific tasks
uv run pytest tests/unit          # Run core unit tests
uv run pytest tests/unit  # Run unit tests
uv run pyright src               # Type check source code

Testing Strategy

Test Types

  1. Unit Tests: Fast, isolated tests for individual components
  2. End-to-End Tests: Integration tests for complete workflows
  3. Benchmarks: Performance tests for evaluation metrics

Running Tests

# All tests
make test

# Specific test categories
uv run pytest tests/unit
uv run pytest tests/e2e

# With coverage or specific options
uv run pytest tests/unit -k "test_name"

Test Organization

  • Unit Tests: tests/unit/
  • End-to-End Tests: tests/e2e/
  • Benchmarks: tests/benchmarks/

Code Quality & CI/CD

Code Quality Pipeline

The make format command runs:

  1. isort: Import sorting
  2. ruff format: Code formatting
  3. ruff --fix-only: Auto-fix issues (including unused imports)
  4. ruff check: Final linting validation

Type Checking

make type  # Type check all code with pyright

CI/CD Pipeline

Our GitHub Actions CI runs:

  1. Dependency Installation: Using uv for consistent environments
  2. Code Quality Checks: Format and type validation
  3. Testing: Unit and integration tests across Python 3.9-3.12
  4. Multi-OS Testing: Ubuntu, macOS, Windows

Local CI Simulation

make run-ci  # Runs: format + type + test

Project Guidelines

Ragas Project

  • Language: Python with type hints
  • Testing: pytest with nbmake for notebook tests
  • Style: Google-style docstrings
  • Architecture: Modular metrics and evaluation framework with experimental features
  • Dependencies: All defined in pyproject.toml

Adding Dependencies

  • All features: Add to pyproject.toml
  • Always: Test with make install and make test

Troubleshooting

Common Issues

Import Errors

# Reinstall in development mode
make install

Test Failures

# Run specific failing test
uv run pytest tests/unit/test_specific.py -v

# Check experimental test dependencies
uv run pytest tests/unit --collect-only

Formatting Issues

# Fix formatting
make format

# Check specific files
uv run ruff check path/to/file.py --fix

CI Failures

# Run the same checks locally
make run-ci

# Individual checks
make format  # Must pass
make type    # Must pass  
make test    # Must pass

Development Environment Issues

uv Not Found

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# or use pip: pip install uv

Dependency Conflicts

# Clean install
make clean
make install

Getting Help

  • Documentation: Check CLAUDE.md for AI assistant guidance
  • Commands: Run make help for all available targets
  • Issues: Check existing GitHub issues or create a new one

Contributing Guidelines

Pull Request Process

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/amazing-feature
  3. Develop using the workflows above
  4. Test thoroughly: make run-ci
  5. Submit a pull request with clear description

Commit Message Format

feat: add new evaluation metric
fix: resolve import error in experimental
docs: update development guide
test: add unit tests for metric base

Code Review Checklist

  • All tests pass (make test)
  • Code is formatted (make format)
  • Type checking passes (make type)
  • Documentation is updated
  • Appropriate tests are included

AI Agent Best Practices

Recommended Workflow for AI Agents

  1. Understand the task: Read relevant documentation and code
  2. Plan the approach: Identify which project(s) need changes
  3. Use standardized commands: Always prefer make targets
  4. Test incrementally: Use make check frequently during development
  5. Validate thoroughly: Run make run-ci before completing

Command Patterns for AI Agents

# Always start with understanding the current state
make help
ls -la  # Check current directory structure

# For code changes
make format  # After making changes
make test    # Verify functionality

# For project-specific work
make help                       # See available commands

# For investigation
uv run pytest --collect-only  # See available tests
uv run ruff check --no-fix    # Check issues without fixing

File Modification Guidelines

  • Prefer editing existing files over creating new ones
  • Use project conventions (check similar files for patterns)
  • Update tests when modifying functionality
  • Follow existing code style (enforced by make format)

Python 3.13 on macOS ARM: NumPy fails to install (builds from source)

  • Symptom: make install attempts to build numpy==2.0.x from source on Python 3.13 (no prebuilt wheel), failing with C/C++ errors.
  • Status: Ragas CI supports Python 3.9–3.12. Python 3.13 is not officially supported yet.

Workarounds:

  1. Recommended: use Python 3.12
uv python install 3.12
rm -rf .venv
uv venv -p 3.12
make install
  1. Stay on 3.13 (best effort):
  • Install minimal first, then add extras as needed:
rm -rf .venv
uv venv -p 3.13
make install-minimal
uv pip install "ragas[tracing,gdrive,ai-frameworks]"
  • Or force a newer NumPy wheel:
uv pip install "numpy>=2.1" --only-binary=:all:

If conflicts pin NumPy to 2.0.x, temporarily set numpy>=2.1 in pyproject.toml and run uv sync --group dev.

Happy coding! 🚀

For additional context and instructions specific to AI assistants, see CLAUDE.md.

在新工单中引用 查看 Git Blame 复制永久链接