* added LearningWithNoisyLabels.find_label_issues instance method
* LNL.find_label_issues no longer memoizes
* verbose unit test coverage
* Fixed all issues in PR. Fixed confident joint usage in LNL.find_label_issues. Fixed other minor bugs. Added tests
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com>
Co-authored-by: Anish Athalye <me@anishathalye.com>
This was used to skip e.g. importing PyTorch on some tests. However,
PyTorch supports all versions of Python that Cleanlab supports, so
there's no need to issue this warning. Furthermore, having this kind of
code in our codebase increases maintenance burden and contributes to
user frustration. If we mark a version of some library as unsupported,
but the library adds support for it in the future, there's no way users
can use the two packages together without waiting for us to push a new
version of cleanlab that marks that it's compatible and then switching
to the new version, which may be painful for unrelated reasons. It's
better to not have these kinds of checks; if users are using a version
of Python that's unsupported by some other library that they want to
use, that's not our problem, and they'll have issues installing or
importing it and figure it out themselves.
This patch also removes mentions of Python 2.
Preserving the output cells in Jupyter notebooks that are checked in to
this repo bloat the repo size, and keeping the output is not necessary
because the notebooks are run as part of the CI for producing docs.
For example, in https://github.com/cleanlab/cleanlab/pull/165, the
`audio.ipynb` is 112K (and now part of the repo history forever), while
clearing the output cells reduces that size down to 24K. I suspect that
Git's delta compression will also work better without the output cells
there as we make changes to the notebook. Also, the diffs will be more
readable.
* Move _process_label_issues_kwargs to fit method
* Add test for sklearn GridSearchCV with kwargs
* Add comment to test for sklearn GridSearchCV with kwargs
* Change cv=3 for GridSearchCV in test
* improved language in docs Quickstart
* LearningWithNoisyLabels refactor to improve API flexibility and UX
* fix LearningWithNoisy rare label handling
* additional arg-checking unit tests
LearningWithNoisyLabels used `sklearn.utils.check_X_y` to enforce that
`X` was 2D, to be in line with what sklearn's standard estimators
expect. However, LearningWithNoisyLabels is dataset-agnostic: it doesn't
look at the data points themselves. If the underlying classifier
supports data in a different shape, there's no reason
LearningWithNoisyLabels should prohibit it. Users have requested that we
relax this unnecessary restriction [1] so LearningWithNoisyLabels will
more naturally support e.g. image datasets and CNN models.
Thanks to @kothari1997narayan for suggesting this change.
[1]: https://github.com/cleanlab/cleanlab/issues/86
Inspired by [1] and similar examples. See also: the pull request that
switches from `copy.deepcopy` to `sklearn.base.clone` [2]. Even the
latter calls `sklearn.base.clone(..., safe=False)` on the values
returned by `get_params()`, so the user has to be sure that their model
is correctly clonable, otherwise they may get silently incorrect
behavior.
[1]: https://github.com/cleanlab/cleanlab/issues/87
[2]: https://github.com/cleanlab/cleanlab/pull/144
Arbitrary scikit-learn estimators are not safe to clone via
`copy.deepcopy()`, for example when they contain state that cannot be
pickled, such as a Keras model that contains a `_thread.RLock` [1]. The
correct way to clone an estimator is to use `sklearn.base.clone()` [2],
which constructs a new unfitted estimator with the same parameters as
the original by calling `get_params()` followed by creating a new object
of the class, passing the parameters to `__init__()`. This patch
switches from `copy.deepcopy` to `sklearn.base.clone`.
This patch also updates the MNIST PyTorch CNN example model to be
compatible with `sklearn.base.clone`. Writing a custom `get_params`
implementation usually wouldn't be necessary with a more standard
estimator that inherits from `BaseEstimator`, but it was necessary to
override the method in this case because of the nonstandard
implementation that supports multiple datasets and does data loading
within the estimator itself based on the `dataset` argument.
[1]: https://github.com/cleanlab/cleanlab/issues/87
[2]: https://scikit-learn.org/stable/modules/generated/sklearn.base.clone.html
* utilities -> internals
* docbuilding instructions improvement
* fixup formatting of contributing.md
* change contributor guidelines language to be optional
* line formatting
`np.long` was deprecated in NumPy 1.20, and this was making the test
output noisy with deprecation warnings. This patch replaces the type
with `np.int32` (the labels are 0--9).
* Add label quality scoring function for users to choose the scoring method.
Add confidence_weighted_entropy as another label quality score that is available for scoring and ranking.
Add option in scoring function to adjust predicted probabilities by subtracting the class thresholds.
Refactor order_label_issues to use the new label quality scoring function and accept keyword args.
* Cleanup docstring in label quality scoring functions
* Change rank_by_kwargs default to empty dict. Dict is used as keyword args for label quality scoring function.
* Cleanup docstrings. Add **rank_by_kwargs to signature of order_label_issues function.
* Add exception handler for invalid rank_by methods
* Add **rank_by_kwargs to allow keyword args in find_label_issues()
* Add test for confidence_weighted_entropy rank scoring function
* Add test for scoring function that accepts scoring method
* Cleanup comments
* Update test for scoring functions
* Update test for scoring functions
* Modify order_label_issues() function to run score_label_quality with (labels, pred_probs) and then filter with label_issues_mask. This is more robust to allow us to adjust the pred_probs (e.g. subtract confident class thresholds)
* Update test for label quality scoring
* Update test for label quality scoring
* Update exception handler for label scoring function
* Add test for subtracting confident class thresholds
* Update test for subtracting confident class thresholds
* Update test for subtracting confident class thresholds
* Add ensemble label quality scoring function
* Update test for ensemble label quality scoring function
* Update test for ensemble label quality scoring function
* Update docstrings for functions that accept adj_pred_probs with description of the adjustment to pred_probs
* Update find_label_issues to pass rank_by_kwargs as a dict
* Cleanup comments
* Refactor tests for scoring functions to avoid redundancies
* Refactor tests for scoring functions to avoid redundancies. Use parameterize instead of for-loop
* Parameterize scoring method to test ensemble scoring function
* Print scoring method when scoring function test fails
* Cleanup docstrings to avoid redundancies
* Move class confident threshold functions to new label quality utils module. Add weight_ensemble_members_by parameter to label quality ensemble scoring function allows users to choose weighting scheme (uniform, accuracy). Enhance tests for CI.
* Add test for CI. Test bad arg for weight_ensemble_members_by parameter.
* Add explicit error checking of pred_probs_list arg.
* Cleanup docstring and print statements
* Cleanup docstrings. Move get_entropy to utils. Refactor score_label_quality_ensemble to print accuracy weights
* Change default label quality scoring method to normalized_margin. Add to docstring explaining when to use normalized_margin vs self_confidence.
* Cleanup docstring
* Cleanup docstrings. Rename get_entropy() to get_normalized_entropy()
* Rename score_label_quality() to get_label_quality_scores(). Rename score_label_quality_ensemble() to get_label_quality_ensemble_scores().
* Remove extra indent in docstring
* Rename adj_pred_probs to adjust_pred_probs. Move get_confident_thresholds() to count.py module.
* Add comment to explain why we add a dimension for numpy broadcasting.
* Change renormalization logic when adjusting pred_probs. Raise ValueError when adjust_pred_probs is used with unsupported scoring method. Add test for ValueError.
* Update tests to account for unsupported method when running adjust_pred_probs.
* Enhance docstring tests
Anish Athalye