* Added base structure outline for get_ood_scores()
Added base test for get_ood_scores()
* Added warning for illogical param combo
* Addressed PR comments
* Added better unit tests
* TODO: test for correctly identifying OOD example
* Moved logic from get_ood_scores to _subtract_confident_thresholds
* TODO: Is this a cleaner way of doing this? Minimal code repeat but strange change to _subtract function
* Addressed type issue
* Switched logic for getting confident thresholds
* Fixing typecheck issues wiht labels parameter being None
* Simplified helper function. Testing type
* Fixed mypy static typing issue
* Mypy typecheck logic test
* removed uncessesary imports in util file
* typechecker debugging (add assert)
* Fixed type logic and removed confident_thresholds=None return
* Added extra arg in helper func to end of func
* Added zero-index checking for label param
* Added ood examples to outlier score notebook
* Added skeleton file structure for implementing outliers
* Make adjust_pred_probs=True by default not false
* Added base Outlier class functionality
TODO:
* test_outlier.py
* Added logic tests for function
* Updated get_ood_scores to always return confident_thresholds
* Even if none were calculated (then return is None)
* Added warning for fit that doesn't calculate confident_thresholds
* Moved get_outlier_scores and get_ood_scores to outlier.py
* Changed param dicts to dicts
* Added docstring to outlier.py
* Added proper return types
* Fixed mypy typing issues
* Switched outliers -> features; ood -> predictions naming conv
* Switched docstring to stem from fit and score functions
* Changed return of helper functions
* Fixed tutorials notebook to use OutOfDistribution class
* Moved imports to top of file
* Added option for different knn objects, addressed pr comments
* Switched params arg to init only
* Addressed PR comment for notebook, cleared notebook
* Fixed PR comments, wording.
* Testing relative links on build
* Changed ood_pred_probs scores to reflect 0 = most ood 1 = least
* Added MLP for detection outliers with pred_probs
* Fixed typos
* Added MLP Import
* Improved warning when fit call unnecessary
* Changed referenced to params dict in warnings/errors
* Added bagging+MLP classifier into notebook
* Reverted tutorial wording
* OOD tutorial improvements
* cleanup OOD documentation
* adopting->using
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
* Added support to build KNN graph with only training data
- Added default classifier for get_knn_distance_ood_scores()
* Added unit test for checking default KNN model is used when nbrs=None
* Changed default num neighbors from 10 to K.
* Changed unit test to check score sums
* Fixed k=15
* Added support to build KNN graph with only training data
- Added default classifier for get_knn_distance_ood_scores()
- Added unit test for checking default KNN model is used when nbrs=None
- Changed default num neighbors from 10 to K.
- Changed unit test to check score sums
* Improved code readibility, extended unit test to check for default k
* Changed naming convention from get_knn_distance_ood_scores to get_outlier_scores
- Changed nbrs to knn
- Improved unit test readability
* Made unit test more robust to check if user-set value is passed
* changed classifier -> estimator
* fixed test warning handle
* Updated function headers to better definition
* Improved header writing added functionality to avoid training/testing with identical features
* Updated docstring, updated handling of features=None and test for it
* Added ValueError for k>len(features) and test to catch ValueError
* Updated argument types to Optional, added runtime typecheck.
* Added unit test to check TypeError
* Changed Exception type thrown when knn=None, features=None
* TypeError to ValueError
* Reversed scoring for outlier severity
* Added test to check t parameter
* Added t parameter for global rescaler
* Improved function definition concerning 't' parameter
* Fixed tests
* Removed repeated calls
* Added assertion to make sure X_ood is always smallest outlier score
* Improved logic for checking X_ood has the smallest score
* Add KNN distance OOD scoring function and unit tests
* Update KNN distance OOD scoring function
* Change query_features to features in unit tests for KNN distance OOD scoring function
* Update KNN distance OOD scoring function
* Update tests for KNN distance OOD scoring function to use auto for algo
* Allow k=None for KNN Distance OOD score
* Add log_loss_search weighting method for ensemble label quality scoring function
* Update log_loss_search weighting method
* Add test for log_loss_search method
* Add parameter to Ensemble label quality scoring function for t values in log_loss_search method
* Update ensemble label quality scoring function docstring
* Update ensemble label quality scoring function comments
* Update docstring in ensemble label quality scoring function
* Update docstring in ensemble label quality scoring function
* Update docstring in ensemble label quality scoring function
* Modify verbose printout for log_loss_search
* Add clipping of pred_prob when calculating weights for log_loss_search
* Add clipping of pred_prob and renormalization when calculating weights for log_loss_search
* Add comments for log_loss_search weighting scheme
* Allow user to pass custom_weights to ensemble scoring method
* Add tests for ensemble scoring with custom_weights
* Add check to make sure length of custom_weights matches len(pred_probs_list)
* Update tests for usage of custom_weights in ensemble scoring
* utilities -> internals
* docbuilding instructions improvement
* fixup formatting of contributing.md
* change contributor guidelines language to be optional
* line formatting
* Add label quality scoring function for users to choose the scoring method.
Add confidence_weighted_entropy as another label quality score that is available for scoring and ranking.
Add option in scoring function to adjust predicted probabilities by subtracting the class thresholds.
Refactor order_label_issues to use the new label quality scoring function and accept keyword args.
* Cleanup docstring in label quality scoring functions
* Change rank_by_kwargs default to empty dict. Dict is used as keyword args for label quality scoring function.
* Cleanup docstrings. Add **rank_by_kwargs to signature of order_label_issues function.
* Add exception handler for invalid rank_by methods
* Add **rank_by_kwargs to allow keyword args in find_label_issues()
* Add test for confidence_weighted_entropy rank scoring function
* Add test for scoring function that accepts scoring method
* Cleanup comments
* Update test for scoring functions
* Update test for scoring functions
* Modify order_label_issues() function to run score_label_quality with (labels, pred_probs) and then filter with label_issues_mask. This is more robust to allow us to adjust the pred_probs (e.g. subtract confident class thresholds)
* Update test for label quality scoring
* Update test for label quality scoring
* Update exception handler for label scoring function
* Add test for subtracting confident class thresholds
* Update test for subtracting confident class thresholds
* Update test for subtracting confident class thresholds
* Add ensemble label quality scoring function
* Update test for ensemble label quality scoring function
* Update test for ensemble label quality scoring function
* Update docstrings for functions that accept adj_pred_probs with description of the adjustment to pred_probs
* Update find_label_issues to pass rank_by_kwargs as a dict
* Cleanup comments
* Refactor tests for scoring functions to avoid redundancies
* Refactor tests for scoring functions to avoid redundancies. Use parameterize instead of for-loop
* Parameterize scoring method to test ensemble scoring function
* Print scoring method when scoring function test fails
* Cleanup docstrings to avoid redundancies
* Move class confident threshold functions to new label quality utils module. Add weight_ensemble_members_by parameter to label quality ensemble scoring function allows users to choose weighting scheme (uniform, accuracy). Enhance tests for CI.
* Add test for CI. Test bad arg for weight_ensemble_members_by parameter.
* Add explicit error checking of pred_probs_list arg.
* Cleanup docstring and print statements
* Cleanup docstrings. Move get_entropy to utils. Refactor score_label_quality_ensemble to print accuracy weights
* Change default label quality scoring method to normalized_margin. Add to docstring explaining when to use normalized_margin vs self_confidence.
* Cleanup docstring
* Cleanup docstrings. Rename get_entropy() to get_normalized_entropy()
* Rename score_label_quality() to get_label_quality_scores(). Rename score_label_quality_ensemble() to get_label_quality_ensemble_scores().
* Remove extra indent in docstring
* Rename adj_pred_probs to adjust_pred_probs. Move get_confident_thresholds() to count.py module.
* Add comment to explain why we add a dimension for numpy broadcasting.
* Change renormalization logic when adjusting pred_probs. Raise ValueError when adjust_pred_probs is used with unsupported scoring method. Add test for ValueError.
* Update tests to account for unsupported method when running adjust_pred_probs.
* Enhance docstring tests
* Refactor modules pruning to filter and latent_estimation to count
* Remove polyplex (research) algorithms from cleanlab
* Create new module rank and move scoring functions to rank.
* Rename test to match new module names
* Fixed error in normalized margin. added ranking for arbitrary psx and labels.
* Remove unused tests and methods. add multi-label support for baseline.
* Move baseline methods to filter and delete baseline module.
* change filter.get_noise_indices to filter.find_label_issues
* Rename baseline methods. fill out docstrings.
* Only require 1 example to be left in each class after removing errors. (instead of 5)
* Remove K as a parameter to count.compute_confident_joint
* Add C_argmax and C_ij methods from CL paper to find_label_issues
* Add warnings for new prune methods and frac_noise. Fix tests.
* Add baseline tests to test_rank_filter and delete baseline test
* Remove inverse_noise_matrix parameter in classification call to find_label_issues
* add todo to update docstring with new ranking functions
* 100% tests pass. add multi-label support for prune_method
* Major NOT-backwards-compatible name changes to most components
* More Major NOT-backwards-compatible name changes
* fixed s -> label mistakes
* Several nomenclature updates from PR feedback. models renamed to example models.
* Remove python2 support across all modules.
* major api changes. psx -> pred_probs. prob_given_label -> self_confidence. testing added.
* enable python version 3.9 for pytorch model.
* ran spellcheck
* ran grammar check
* Update count.py
* Update filter.py
* Update setup.py and ci.yml to no longer support Python 2 and py3.4/5
* Increase test coverage and documentation of rank module methods.
* create utils submodule and move util and latent_algebra
* Rename y everywhere to true_labels, and p(true_label=..)
* Enforce positional arguments in methods. Fully remove py2 support.
Jonas Mueller