/DEVELOPMENT.md

Development

This document explains how to set up a development environment for contributing to cleanlab.

Setting up a virtual environment

While this is not required, we recommend that you do development and testing in a virtual environment. There are a number of tools to do this, including virtualenv, pipenv, and venv. You can compare the tools and choose what is right for you. Here, we'll explain how to get set up with venv, which is built in to Python 3.

$ python3 -m venv ./ENV  # create a new virtual environment in the directory ENV
$ source ./ENV/bin/activate  # switch to using the virtual environment

You only need to create the virtual environment once, but you will need to activate it every time you start a new shell. Once the virtual environment is activated, the pip install commands below will install dependencies into the virtual environment rather than your system Python installation.

Installing dependencies and cleanlab

Run the following commands in the repository's root directory.

1. Install development requirements with pip install -r requirements-dev.txt

2. Install cleanlab as an editable package with pip install -e .

For Macs with Apple silicon: replace tensorflow in requirements-dev.txt with: tensorflow-macos==2.9.2 and tensorflow-metal==0.5.1

Testing

Run all the tests:

$ pytest

Run a specific file or test:

$ pytest -k <filename or filter expression>

Run with verbose output:

$ pytest --verbose

Run with code coverage:

$ pytest --cov=cleanlab/ --cov-config .coveragerc --cov-report=html

The coverage report will be available in coverage_html_report/index.html, which you can open with your web browser.

Type checking

Cleanlab uses mypy typing. Type checking happens automatically during CI but can be run locally.

Check typing in all files:

$ mypy cleanlab

Note our CI adds a few additional flags to the mypy command it uses in the file: .github/workflows/ci.yml. If you want to exactly match the mypy command that is executed in CI, copy these flags, and also ensure your version of mypy and related packages like pandas-stubs match the latest released versions (used in our CI).

Examples

You can check that the examples still work with changes you make to cleanlab by manually running the notebooks. You can also run all example notebooks as follows:

git clone https://github.com/cleanlab/examples.git

Then specify your local version of cleanlab source in the first line of: examples/requirements.txt. E.g. you can edit this line to point to your local version of cleanlab as a relative path such as ../cleanlab if the cleanlab and examples repos are sibling directories on your computer.

Finally execute the bash script:

examples/run_all_notebooks.sh

How to style new code contributions

cleanlab follows the Black code style. This is enforced by CI, so please format your code by invoking black before submitting a pull request.

Generally aim to follow the PEP-8 coding style. Please do not use wildcard import * in any files, instead you should always import the specific functions that you need from a module.

Pre-commit hook

This repo uses the pre-commit framework to easily set up code style checks that run automatically whenever you make a commit. You can install the git hook scripts with:

$ pre-commit install

EditorConfig

This repo uses EditorConfig to keep code style consistent across editors and IDEs. You can install a plugin for your editor, and then your editor will automatically ensure that indentation and line endings match the project style.

Adding new modules into the source code

You should go through the following checklist if you intend to add new functionality to the package in a separate module.

• Add brief description of the module’s purpose in a comment at the top of file and docstrings for every function.
• Import the module my_module.py into main __init__.py
• Create detailed unit tests (typically in a new file tests/test_my_module.py)
• Add new module to docs index pages (docs/source/index.rst) and create .rst file in docs/source/cleanlab/ (so that module appears on docs.cleanlab.ai -- please verify its documentation also looks good there)
• Create a QuickStart (docs/source/tutorials) notebook that runs main module functionality in 5min or less and add it to index pages (docs/source/tutorials/index.rst, docs/source/index.rst). Clear cell output before pushing.
• Create an examples notebook that runs more advanced module functionality with a more real-world application (can have a longer run time). Push with printed cell output.

Documentation

You can build the docs from your local cleanlab version by following these instructions.

If editing existing docs or adding new tutorials, please first read through our guidelines.

Documentation style

cleanlab uses NumPy style docstrings (example).

Aspects that are not covered in the NumPy style or that are different from the NumPy style are documented below:

• Referring to the cleanlab package: we refer to cleanlab without any special formatting, so no cleanlab, just cleanlab.

• Cross-referencing: when mentioning functions/classes/methods, always cross-reference them to create a clickable link. Cross-referencing code from Jupyter notebooks is not currently supported.

• Variable, module, function, and class names: when not cross-references, should be written between single back-ticks, like `pred_probs`. Such names in Jupyter notebooks (Markdown) can be written between single back-ticks as well.

• Math: We support LaTeX math with the inline :math:`x+y` or the block:

  .. math::
  
     \sum_{0}^{n} 2n+1
  

• Pseudocode vs math: Prefer pseudocode in double backticks over LaTeX math.

• Bold vs italics: Use italics when defining a term, and use bold sparingly for extra emphasis.

• Shapes: Do not include shapes in the type of parameters, instead use np.array or array_like as the type and specify allowed shapes in the description. See, for example, the documentation for cleanlab.classification.CleanLearning.fit(). Format for 1D shape: (N,1)

• Optional arguments: for the most part, just put , optional in the type.

• Type unions: if a parameter or return type is something like "a numpy array or None", you can use "or" to separate types, e.g. np.array or None, and it'll be parsed correctly.

• Parameterized types: Use standard Python type hints for referring to parameters and parameterized types in docs, e.g. Iterable[int] or list[float].

Common variable names / terminology used throughout codebase

• N - the number of examples/datapoints in a dataset.
  • num_examples may also be used when additional clarity is needed.
• K - the number of classes (unique labels) for a dataset.
  • num_classes may also be used when additional clarity is needed.
• labels - a label for each example, length should be N (sample-size of dataset)
• classes - set of possible labels for any one example, length should be K (number of possible categories in classification problem)

Try to adhere to this standardized terminology unless you have good reason not to!

Relative Link Formatting Instructions

Use relative linking to connect information between docs and jupyter notebooks, and make sure links will remain valid in the future as new cleanlab versions are released! Sphinx/html works with relative paths so try to specify relative paths if necessary. For specific situations:

• Link another function from within a source code docstring: :py:func:`function_name <cleanlab.file.function_name>`
• Link another class from within a source code docstring: :py:class:`class_name <cleanlab.file.class_name>`
• Link a tutorial notebook from within a source code docstring: :ref:`notebook_name <notebook_name>`
• Link a function from within a tutorial notebook: [function_name](../cleanlab/file.rst#cleanlab.file.function_name)
• Link a specific section of a notebook from within the notebook: [section title](#section-title)
• Link a different tutorial notebook from within a tutorial notebook: [another notebook](another_notebook.html). (Note this only works when the other notebook is in same folder as this notebook, otherwise may need to try relative path)
• Link another specific section of different notebook from within a tutorial notebook: [another notebook section title](another_notebook.html#another-notebook-section-title)
• Linking examples notebooks from inside tutorial notebooks can be simply done by linking global url of the example notebook in master branch of github.com/cleanlab/examples/