* Added base structure outline for get_ood_scores()
Added base test for get_ood_scores()
* Added warning for illogical param combo
* Addressed PR comments
* Added better unit tests
* TODO: test for correctly identifying OOD example
* Moved logic from get_ood_scores to _subtract_confident_thresholds
* TODO: Is this a cleaner way of doing this? Minimal code repeat but strange change to _subtract function
* Addressed type issue
* Switched logic for getting confident thresholds
* Fixing typecheck issues wiht labels parameter being None
* Simplified helper function. Testing type
* Fixed mypy static typing issue
* Mypy typecheck logic test
* removed uncessesary imports in util file
* typechecker debugging (add assert)
* Fixed type logic and removed confident_thresholds=None return
* Added extra arg in helper func to end of func
* Added zero-index checking for label param
* Added ood examples to outlier score notebook
* Added skeleton file structure for implementing outliers
* Make adjust_pred_probs=True by default not false
* Added base Outlier class functionality
TODO:
* test_outlier.py
* Added logic tests for function
* Updated get_ood_scores to always return confident_thresholds
* Even if none were calculated (then return is None)
* Added warning for fit that doesn't calculate confident_thresholds
* Moved get_outlier_scores and get_ood_scores to outlier.py
* Changed param dicts to dicts
* Added docstring to outlier.py
* Added proper return types
* Fixed mypy typing issues
* Switched outliers -> features; ood -> predictions naming conv
* Switched docstring to stem from fit and score functions
* Changed return of helper functions
* Fixed tutorials notebook to use OutOfDistribution class
* Moved imports to top of file
* Added option for different knn objects, addressed pr comments
* Switched params arg to init only
* Addressed PR comment for notebook, cleared notebook
* Fixed PR comments, wording.
* Testing relative links on build
* Changed ood_pred_probs scores to reflect 0 = most ood 1 = least
* Added MLP for detection outliers with pred_probs
* Fixed typos
* Added MLP Import
* Improved warning when fit call unnecessary
* Changed referenced to params dict in warnings/errors
* Added bagging+MLP classifier into notebook
* Reverted tutorial wording
* OOD tutorial improvements
* cleanup OOD documentation
* adopting->using
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
* 🎨 remove wildcard imports
* 🏷️ review type annotations in display_issues
Restrict nested lists with `List`. Keep unrestricted lists as `list`. Remove hard-coded tags from docstrings, should be auto-tagged in later PR.
* ♻️ use `isinstance` for type-check at runtime
* 🏷️ review type annotations in common_label_issues and filter_by_token
Remove hard-coded tags in docstrings. Should be auto-tagged in later PR.
* ✅ improve test coverage of display issues
No part of the function is easily testable except ensuring it completes exection. Some parts handle edge cases that were never reached during testing.
* ✅ parametrize tests for coverage on common_label_issues and filter_by_token
* ✨ search tokenized sentence for coloring
Searches through the list of tokens before trying to match substrings. Thanks for this code suggestion Eric!
* 🚧 fix signature in all calls to color_sentence
* ✅ update color_sentence test after changing its api
(sentence, word) -> (word, tokens)
* ⏪ use regex for coloring tokens in sentence
Find word boundaries with regex, use builtin replace() as fallback w/o boundaries.
Closes#288
* 🚑 invert fallback condition
Use replace if NO substitutions were made with regex.
* ✅ add test fixture for get_label_quality_scores
* ✅ test softmin_sentence_score
include test cases for temperature limits
* ♻️ cleanup softmin_sentence_score
Remove unused keyword-only args, Fix tag in docstring, Change format of nested functions.
* ⚡ specialize edge-case temperature=inf in softmin_sentence_score
* ♻️ simplify temperature lookup
* ♻️ cleanup get_label_quality_scores
Remove unused args/variables. Update parameter list in docstring. Rename parameter of inner function. Function always returns a tuple.
* 🏷️ tag token_scores as optional
* ✅ test raised error
* 🩹 skip untestable elif statement
the elif statement only gets partial coverage because it can't evaluate to False due to the `assert sentence_score_method` at the start of the function
* 🏷️ restrict parameters for list types
* 🐛 only process characters in input token
Example: process_token("Cleanlab", [("C", "a"), ("a", "C")]) should return "aleCnlCb", not "CleCnlCb".
* ✅ use all sentences in test_get_sentence
* ✅ add test cases to test_filter_sentence
* ✅ extent test cases in test_mapping
* 📝 clean up docstrings
Restrict arg types based on docstrings. Fix punctuation and typos. Add examples to docstring.
* ✅ split tests for filter_sentence
* ✅ extend test_merge_probs
* ✅ test merge_probs with ignored/normalized columns in probs
* ✅ extend test cases for get_sentences
* ⚰️ remove unused pandas import
* 🏷️ pass strict mypy check
We ignore np.max as it is untyped.
No issues found in token_classification_utils.py by running
```
mypy --install-types --non-interactive --strict cleanlab/internal/token_classification_utils.py
```
* 👷 add strict type-checking in CI
* 💚 use strict type-check for single file
* ⏪ remove strict type check in CI
* refactor: 🏷️ use np.ndarray type instead of npt.NDArray
* ⏪ go back to generic np.ndarray type
* ♻️ always return tuple in filter_sentence
Remove unused argument+docstring. Simplify relevant unit tests.
* 🔥 resolve comments on typing
Remove ignore-comments. Remove duplicate tag in docstring. Remove unused imports.
* 🔥 remove duplicate tag in docstring
* validation_func docstring
* torch,tf compatibility+tests
* keras test
* skip tests if python < 3.7
* pytorch numpy int bug on windows
* make tensorflow test work on windows
* move tf env variable setting
* pytorch test increase epochs
* install cpu-tensorflow on windows CI
* torch test optimizer to adam
* fix bugs in shuffled TF dataset
* dummy unit test for TF on windows
* dummy code for TF windows testing
* deal with np.int bug on windows
* remove windows debugging code
* docstrings for new functionality
* address merge conflicts
* reformat after merge
* addressed comments
* 🏷️ annotate function args and return values
Starting with the validation module:
- I think (X, y) might need some custom Union type to handle both numpy arrays and pandas dataframe, etc.
- All of the "assert" functions return None.
Ref #307
* refactor: 🏷️ swap npt.NDArray -> np.ndarray
np.ndarray seems more consistent with the rest of the repo.
Maybe it's necessary to go back to npt.NDArray when disallowing generics?
See numpy docs: https://numpy.org/devdocs/reference/typing.html#numpy.typing.NDArray
* refactor: 🔥 remove unused import
* 🏷️ unconstrain X input types
* 🐛 handle label type issues
- Have to restrict the output of labels_to_array to pass mypy checks.
- Returning the values of pd.Series isn't type-stable.
* 🏷️ include np.generic in arg-type union
* test: ✅ test labels_to_array
* 🏷️ add type aliases for X and y
* 🚨 ignore type-checks for pandas indexing assertions
CI typechecker runs on Python 3.10 which gives this error:
'cleanlab/internal/validation.py:125: error: No overload variant of "__getitem__" of "_iLocIndexerSeries" matches argument type "List[int]"'
It should be fine to let mypy ignore these expressions as they don't return anything.
* 🥅 specify errors to ignore
"type: ignore" doesn't pass strict mypy type-checks unless the specific errors are provided
* 🏷️ annotate label series to array
* 🗑️ change labels arg -> y in CleanLearning.fit()
Set `label` as an optional keyword-only argument.
Anyone still using it in this method should get a deprecation warning.
Fixes#281
* 📝 add note for y/labels in docstring
* 🥅 make y an optional positional arg.
Should now resolve deprecated signatures.
* 📝 labels -> y in module docstring
* ⏪ revert "label -> y deprecation"
This reverts commit ab319a0cca2cec715a84eb5f628bbab7706c5f9c.
This reverts commit 1b739002d848e1f0acb6390a666f6e695e25fcaa.
This reverts commit 88bb6c3bcca298dab414c3cb20101783d78d35d1.
This reverts commit d988e3c3932107e779598d02d8f16d7e6671e9e7.
* ✨ add y alias for labels
Resolves#281
* Added support to build KNN graph with only training data
- Added default classifier for get_knn_distance_ood_scores()
* Added unit test for checking default KNN model is used when nbrs=None
* Changed default num neighbors from 10 to K.
* Changed unit test to check score sums
* Fixed k=15
* Added support to build KNN graph with only training data
- Added default classifier for get_knn_distance_ood_scores()
- Added unit test for checking default KNN model is used when nbrs=None
- Changed default num neighbors from 10 to K.
- Changed unit test to check score sums
* Improved code readibility, extended unit test to check for default k
* Changed naming convention from get_knn_distance_ood_scores to get_outlier_scores
- Changed nbrs to knn
- Improved unit test readability
* Made unit test more robust to check if user-set value is passed
* changed classifier -> estimator
* fixed test warning handle
* Updated function headers to better definition
* Improved header writing added functionality to avoid training/testing with identical features
* Updated docstring, updated handling of features=None and test for it
* Added ValueError for k>len(features) and test to catch ValueError
* Updated argument types to Optional, added runtime typecheck.
* Added unit test to check TypeError
* Changed Exception type thrown when knn=None, features=None
* TypeError to ValueError
* Reversed scoring for outlier severity
* Added test to check t parameter
* Added t parameter for global rescaler
* Improved function definition concerning 't' parameter
* Fixed tests
* Removed repeated calls
* Added assertion to make sure X_ood is always smallest outlier score
* Improved logic for checking X_ood has the smallest score
* cleanlearning w dfs
* work for sparse matrix as well
* simplify logic of labels_to_array and extend types
* address pr feedback
* add unit test
* rare label dataframe
* modularize subsetting code
* series rarelabel test
* replace cal.com with slack/email
* Add general method to find num_classes from labels
* compute num_classes with pred_probs.shape[1]
* fix broken commits, address 2nd round of comments
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com>
* Add KNN distance OOD scoring function and unit tests
* Update KNN distance OOD scoring function
* Change query_features to features in unit tests for KNN distance OOD scoring function
* Update KNN distance OOD scoring function
* Update tests for KNN distance OOD scoring function to use auto for algo
* Allow k=None for KNN Distance OOD score
* Add log_loss_search weighting method for ensemble label quality scoring function
* Update log_loss_search weighting method
* Add test for log_loss_search method
* Add parameter to Ensemble label quality scoring function for t values in log_loss_search method
* Update ensemble label quality scoring function docstring
* Update ensemble label quality scoring function comments
* Update docstring in ensemble label quality scoring function
* Update docstring in ensemble label quality scoring function
* Update docstring in ensemble label quality scoring function
* Modify verbose printout for log_loss_search
* Add clipping of pred_prob when calculating weights for log_loss_search
* Add clipping of pred_prob and renormalization when calculating weights for log_loss_search
* Add comments for log_loss_search weighting scheme
* Allow user to pass custom_weights to ensemble scoring method
* Add tests for ensemble scoring with custom_weights
* Add check to make sure length of custom_weights matches len(pred_probs_list)
* Update tests for usage of custom_weights in ensemble scoring
* df return type, need tests still
* Add pandas as a dependency
We already decided that pandas will be a dependency of cleanlab (also
used in the dataset module, see
https://github.com/cleanlab/cleanlab/pull/182).
* Tweak documentation
* addressed comments
* remove lazy import
* address 2nd round comments
* unit tests
* improve codecov
* Fix typo
* methods to save more space
* nocover statements for prints
* extra nocover
* nocover warnings
* test docstring formatting
* test docstring formatting2
* test docstring formatting2
* move compress to helper, find-label docs params
* readded stuff lost in merge conflict
* addressed remaining PR review comments
* docs formatting
* docs formatting2
* docs formatting3
* docs formatting4
* docs formatting5
* docs formatting5
* docs formatting6
* docs formatting7
* docs formatting8
* docs formatting9
* docs formatting19
* docs formatting20
* docs formatting20
* docs formatting21
* code formatting
* fix a bug where confident joint isnt computed
The confident joint wasn't getting computed if noise_matrix was passed in and pred_probs was not passed in. But that's bad because it stops workflows like:
```python
cl = CleanLearning()
cl.fit(data, labels, noise_matrix=noise_matrix)
cleanlab.dataset.health_summary(labels, confident_joint=cl.confident_joint)
```
* fixed bug from last commit. code in wrong place.
* print overwrite bugfix
Co-authored-by: Anish Athalye <me@anishathalye.com>
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com>
This test had a typo in it ("pred_neq_given" rather than
"predicted_neq_given"), but the way the test was written, it didn't
fail, because GridSearchCV raises a warning if some fits fail. This
patch fixes the typo and adds the assertion that no warnings are raised
during the grid search.
It is annoying to re-enable this test because it's annoying to install
fastText / it doesn't run on all platforms we test in CI. Furthermore,
the data isn't available anymore (at least at the URL specified in
`get_cooking_stackexchange_data.sh`).
* Makes some methods private that are not intended to be user-facing.
* Adds experimental module with fasttext.py and coteaching.py
* Adds header descriptions to code files which will render in docs
* Many miscellaneous fixes
* CleanLearning = Machine Learning with cleaned data
* Replace lnl instance naming with cl everywhere (CleanLearning)
* Add support for multi-class as well
* Add test based on #158
Co-authored-by: Anish Athalye <me@anishathalye.com>
* added LearningWithNoisyLabels.find_label_issues instance method
* LNL.find_label_issues no longer memoizes
* verbose unit test coverage
* Fixed all issues in PR. Fixed confident joint usage in LNL.find_label_issues. Fixed other minor bugs. Added tests
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com>
Co-authored-by: Anish Athalye <me@anishathalye.com>
This was used to skip e.g. importing PyTorch on some tests. However,
PyTorch supports all versions of Python that Cleanlab supports, so
there's no need to issue this warning. Furthermore, having this kind of
code in our codebase increases maintenance burden and contributes to
user frustration. If we mark a version of some library as unsupported,
but the library adds support for it in the future, there's no way users
can use the two packages together without waiting for us to push a new
version of cleanlab that marks that it's compatible and then switching
to the new version, which may be painful for unrelated reasons. It's
better to not have these kinds of checks; if users are using a version
of Python that's unsupported by some other library that they want to
use, that's not our problem, and they'll have issues installing or
importing it and figure it out themselves.
This patch also removes mentions of Python 2.
* Move _process_label_issues_kwargs to fit method
* Add test for sklearn GridSearchCV with kwargs
* Add comment to test for sklearn GridSearchCV with kwargs
* Change cv=3 for GridSearchCV in test
* improved language in docs Quickstart
* LearningWithNoisyLabels refactor to improve API flexibility and UX
* fix LearningWithNoisy rare label handling
* additional arg-checking unit tests
Ulyana