also re-order rank.py functions by moving specific types of label-quality scores to be last in file
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
* Added base structure outline for get_ood_scores()
Added base test for get_ood_scores()
* Added warning for illogical param combo
* Addressed PR comments
* Added better unit tests
* TODO: test for correctly identifying OOD example
* Moved logic from get_ood_scores to _subtract_confident_thresholds
* TODO: Is this a cleaner way of doing this? Minimal code repeat but strange change to _subtract function
* Addressed type issue
* Switched logic for getting confident thresholds
* Fixing typecheck issues wiht labels parameter being None
* Simplified helper function. Testing type
* Fixed mypy static typing issue
* Mypy typecheck logic test
* removed uncessesary imports in util file
* typechecker debugging (add assert)
* Fixed type logic and removed confident_thresholds=None return
* Added extra arg in helper func to end of func
* Added zero-index checking for label param
* Added ood examples to outlier score notebook
* Added skeleton file structure for implementing outliers
* Make adjust_pred_probs=True by default not false
* Added base Outlier class functionality
TODO:
* test_outlier.py
* Added logic tests for function
* Updated get_ood_scores to always return confident_thresholds
* Even if none were calculated (then return is None)
* Added warning for fit that doesn't calculate confident_thresholds
* Moved get_outlier_scores and get_ood_scores to outlier.py
* Changed param dicts to dicts
* Added docstring to outlier.py
* Added proper return types
* Fixed mypy typing issues
* Switched outliers -> features; ood -> predictions naming conv
* Switched docstring to stem from fit and score functions
* Changed return of helper functions
* Fixed tutorials notebook to use OutOfDistribution class
* Moved imports to top of file
* Added option for different knn objects, addressed pr comments
* Switched params arg to init only
* Addressed PR comment for notebook, cleared notebook
* Fixed PR comments, wording.
* Testing relative links on build
* Changed ood_pred_probs scores to reflect 0 = most ood 1 = least
* Added MLP for detection outliers with pred_probs
* Fixed typos
* Added MLP Import
* Improved warning when fit call unnecessary
* Changed referenced to params dict in warnings/errors
* Added bagging+MLP classifier into notebook
* Reverted tutorial wording
* OOD tutorial improvements
* cleanup OOD documentation
* adopting->using
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
* Changed all instances of np.array in docstring to np.ndarray
* np.array is NOT a proper class name it is just a function to
create np.ndarrays and therefore should not be parameter class
* Black formatting compliance
* update N in classification.py
* minor docstring changes on K-1 classes
* use K in shape
* minor grammar fixes
* docs language improvements
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
* Added support to build KNN graph with only training data
- Added default classifier for get_knn_distance_ood_scores()
* Added unit test for checking default KNN model is used when nbrs=None
* Changed default num neighbors from 10 to K.
* Changed unit test to check score sums
* Fixed k=15
* Added support to build KNN graph with only training data
- Added default classifier for get_knn_distance_ood_scores()
- Added unit test for checking default KNN model is used when nbrs=None
- Changed default num neighbors from 10 to K.
- Changed unit test to check score sums
* Improved code readibility, extended unit test to check for default k
* Changed naming convention from get_knn_distance_ood_scores to get_outlier_scores
- Changed nbrs to knn
- Improved unit test readability
* Made unit test more robust to check if user-set value is passed
* changed classifier -> estimator
* fixed test warning handle
* Updated function headers to better definition
* Improved header writing added functionality to avoid training/testing with identical features
* Updated docstring, updated handling of features=None and test for it
* Added ValueError for k>len(features) and test to catch ValueError
* Updated argument types to Optional, added runtime typecheck.
* Added unit test to check TypeError
* Changed Exception type thrown when knn=None, features=None
* TypeError to ValueError
* Reversed scoring for outlier severity
* Added test to check t parameter
* Added t parameter for global rescaler
* Improved function definition concerning 't' parameter
* Fixed tests
* Removed repeated calls
* Added assertion to make sure X_ood is always smallest outlier score
* Improved logic for checking X_ood has the smallest score
Our package doesn't have type annotations everywhere, so we can't use
mypy in strict mode just yet. Still, adding type checking in CI is
valuable, so we don't have unchecked annotations in our code.
This patch includes basic fixes to make type checking pass, including
switching the incorrect `np.array` type annotation for `np.ndarray` and
adding some assertions for flow-sensitive typing.
Add clarification of the labels format requirements for all major API functions.
* Fix broken link
* Clarify reqs for labels format. rank.py does not yet support multi_label
* add double ticks to code in docstrings
* clarify docstring
* further clarify multi_label vs single label labels reqs
* also add new docstring to confident joint
* Add KNN distance OOD scoring function and unit tests
* Update KNN distance OOD scoring function
* Change query_features to features in unit tests for KNN distance OOD scoring function
* Update KNN distance OOD scoring function
* Update tests for KNN distance OOD scoring function to use auto for algo
* Allow k=None for KNN Distance OOD score
* Add log_loss_search weighting method for ensemble label quality scoring function
* Update log_loss_search weighting method
* Add test for log_loss_search method
* Add parameter to Ensemble label quality scoring function for t values in log_loss_search method
* Update ensemble label quality scoring function docstring
* Update ensemble label quality scoring function comments
* Update docstring in ensemble label quality scoring function
* Update docstring in ensemble label quality scoring function
* Update docstring in ensemble label quality scoring function
* Modify verbose printout for log_loss_search
* Add clipping of pred_prob when calculating weights for log_loss_search
* Add clipping of pred_prob and renormalization when calculating weights for log_loss_search
* Add comments for log_loss_search weighting scheme
* Allow user to pass custom_weights to ensemble scoring method
* Add tests for ensemble scoring with custom_weights
* Add check to make sure length of custom_weights matches len(pred_probs_list)
* Update tests for usage of custom_weights in ensemble scoring
* df return type, need tests still
* Add pandas as a dependency
We already decided that pandas will be a dependency of cleanlab (also
used in the dataset module, see
https://github.com/cleanlab/cleanlab/pull/182).
* Tweak documentation
* addressed comments
* remove lazy import
* address 2nd round comments
* unit tests
* improve codecov
* Fix typo
* methods to save more space
* nocover statements for prints
* extra nocover
* nocover warnings
* test docstring formatting
* test docstring formatting2
* test docstring formatting2
* move compress to helper, find-label docs params
* readded stuff lost in merge conflict
* addressed remaining PR review comments
* docs formatting
* docs formatting2
* docs formatting3
* docs formatting4
* docs formatting5
* docs formatting5
* docs formatting6
* docs formatting7
* docs formatting8
* docs formatting9
* docs formatting19
* docs formatting20
* docs formatting20
* docs formatting21
* code formatting
* fix a bug where confident joint isnt computed
The confident joint wasn't getting computed if noise_matrix was passed in and pred_probs was not passed in. But that's bad because it stops workflows like:
```python
cl = CleanLearning()
cl.fit(data, labels, noise_matrix=noise_matrix)
cleanlab.dataset.health_summary(labels, confident_joint=cl.confident_joint)
```
* fixed bug from last commit. code in wrong place.
* print overwrite bugfix
Co-authored-by: Anish Athalye <me@anishathalye.com>
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com>
* Makes some methods private that are not intended to be user-facing.
* Adds experimental module with fasttext.py and coteaching.py
* Adds header descriptions to code files which will render in docs
* Many miscellaneous fixes
* added LearningWithNoisyLabels.find_label_issues instance method
* LNL.find_label_issues no longer memoizes
* verbose unit test coverage
* Fixed all issues in PR. Fixed confident joint usage in LNL.find_label_issues. Fixed other minor bugs. Added tests
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com>
Co-authored-by: Anish Athalye <me@anishathalye.com>
* improved language in docs Quickstart
* LearningWithNoisyLabels refactor to improve API flexibility and UX
* fix LearningWithNoisy rare label handling
* additional arg-checking unit tests
* utilities -> internals
* docbuilding instructions improvement
* fixup formatting of contributing.md
* change contributor guidelines language to be optional
* line formatting
* Add label quality scoring function for users to choose the scoring method.
Add confidence_weighted_entropy as another label quality score that is available for scoring and ranking.
Add option in scoring function to adjust predicted probabilities by subtracting the class thresholds.
Refactor order_label_issues to use the new label quality scoring function and accept keyword args.
* Cleanup docstring in label quality scoring functions
* Change rank_by_kwargs default to empty dict. Dict is used as keyword args for label quality scoring function.
* Cleanup docstrings. Add **rank_by_kwargs to signature of order_label_issues function.
* Add exception handler for invalid rank_by methods
* Add **rank_by_kwargs to allow keyword args in find_label_issues()
* Add test for confidence_weighted_entropy rank scoring function
* Add test for scoring function that accepts scoring method
* Cleanup comments
* Update test for scoring functions
* Update test for scoring functions
* Modify order_label_issues() function to run score_label_quality with (labels, pred_probs) and then filter with label_issues_mask. This is more robust to allow us to adjust the pred_probs (e.g. subtract confident class thresholds)
* Update test for label quality scoring
* Update test for label quality scoring
* Update exception handler for label scoring function
* Add test for subtracting confident class thresholds
* Update test for subtracting confident class thresholds
* Update test for subtracting confident class thresholds
* Add ensemble label quality scoring function
* Update test for ensemble label quality scoring function
* Update test for ensemble label quality scoring function
* Update docstrings for functions that accept adj_pred_probs with description of the adjustment to pred_probs
* Update find_label_issues to pass rank_by_kwargs as a dict
* Cleanup comments
* Refactor tests for scoring functions to avoid redundancies
* Refactor tests for scoring functions to avoid redundancies. Use parameterize instead of for-loop
* Parameterize scoring method to test ensemble scoring function
* Print scoring method when scoring function test fails
* Cleanup docstrings to avoid redundancies
* Move class confident threshold functions to new label quality utils module. Add weight_ensemble_members_by parameter to label quality ensemble scoring function allows users to choose weighting scheme (uniform, accuracy). Enhance tests for CI.
* Add test for CI. Test bad arg for weight_ensemble_members_by parameter.
* Add explicit error checking of pred_probs_list arg.
* Cleanup docstring and print statements
* Cleanup docstrings. Move get_entropy to utils. Refactor score_label_quality_ensemble to print accuracy weights
* Change default label quality scoring method to normalized_margin. Add to docstring explaining when to use normalized_margin vs self_confidence.
* Cleanup docstring
* Cleanup docstrings. Rename get_entropy() to get_normalized_entropy()
* Rename score_label_quality() to get_label_quality_scores(). Rename score_label_quality_ensemble() to get_label_quality_ensemble_scores().
* Remove extra indent in docstring
* Rename adj_pred_probs to adjust_pred_probs. Move get_confident_thresholds() to count.py module.
* Add comment to explain why we add a dimension for numpy broadcasting.
* Change renormalization logic when adjusting pred_probs. Raise ValueError when adjust_pred_probs is used with unsupported scoring method. Add test for ValueError.
* Update tests to account for unsupported method when running adjust_pred_probs.
* Enhance docstring tests
* Refactor modules pruning to filter and latent_estimation to count
* Remove polyplex (research) algorithms from cleanlab
* Create new module rank and move scoring functions to rank.
* Rename test to match new module names
* Fixed error in normalized margin. added ranking for arbitrary psx and labels.
* Remove unused tests and methods. add multi-label support for baseline.
* Move baseline methods to filter and delete baseline module.
* change filter.get_noise_indices to filter.find_label_issues
* Rename baseline methods. fill out docstrings.
* Only require 1 example to be left in each class after removing errors. (instead of 5)
* Remove K as a parameter to count.compute_confident_joint
* Add C_argmax and C_ij methods from CL paper to find_label_issues
* Add warnings for new prune methods and frac_noise. Fix tests.
* Add baseline tests to test_rank_filter and delete baseline test
* Remove inverse_noise_matrix parameter in classification call to find_label_issues
* add todo to update docstring with new ranking functions
* 100% tests pass. add multi-label support for prune_method
* Major NOT-backwards-compatible name changes to most components
* More Major NOT-backwards-compatible name changes
* fixed s -> label mistakes
* Several nomenclature updates from PR feedback. models renamed to example models.
* Remove python2 support across all modules.
* major api changes. psx -> pred_probs. prob_given_label -> self_confidence. testing added.
* enable python version 3.9 for pytorch model.
* ran spellcheck
* ran grammar check
* Update count.py
* Update filter.py
* Update setup.py and ci.yml to no longer support Python 2 and py3.4/5
* Increase test coverage and documentation of rank module methods.
* create utils submodule and move util and latent_algebra
* Rename y everywhere to true_labels, and p(true_label=..)
* Enforce positional arguments in methods. Fully remove py2 support.
Jonas Mueller