* error handling for rare classes
* change subtract to symmetric difference
* remove extra np.unique
* add warning for all instances of getting consensus labels
* error checking edits
* Add typing
Co-authored-by: Elías Snorrason <eliassno@gmail.com>
* black formatting
* update typing - labels_multiannotator will always already be converted to pd.dataframe
* make pred_probs options in typing
* add =None
* docstring edits
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
* add basic docstring
* Changed to verbose
* Removed unessesary calculation out of get_labels_quality_m
* Update cleanlab/multiannotator.py
* comment for lost classes check so it can be grepped
* caution about setting verbose to false
* advise against verbose=false in docstring
Co-authored-by: Elías Snorrason <eliassno@gmail.com>
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
Co-authored-by: Ulyana <ulyana@cleanlab.ai>
* docs: 📝 add usage example for get_sentence
Also remove types in docstrings.
* docs: 📝 add usage examples for filter_sentence
* docs: 📝 add page for internal.token_classification_utils
* docs: 📝 remove name of return variable
auto-tagging won't work correctly for the return value if it's not a tuple
* 📝 add pages for modules in cleanlab.token_classification
* fix indentation
* 📝 add token classification api to toctree
* 🎨 format docstrings
* 📝 add usage examples
* update example comment
* 📝 add usage examples
* 📝 add return variable name in docstring
* 📝 remove some hard-coded params types in rank module
Keep more complex parameters hard-coded for now. E.g. "enum"-like parameters.
* 🎨 remove indentation of parameters in docstrings
* 🎨 fix indentation
* colored docstring typo + formatting
* typo: occurence
* probs_merged docstring typo + formatting
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
* header for multiannotator module
* header for outlier module
* header for token_classification.filter
* header for token_classification.rank
* header for token_classification.summary
* header for util
* header for latent_algebra
* header for label_quality_utils
* header for token_classification_utils
* header for huggingfacekerasclassifier
* add experimental modules' dependencies to readme
* Added base structure outline for get_ood_scores()
Added base test for get_ood_scores()
* Added warning for illogical param combo
* Addressed PR comments
* Added better unit tests
* TODO: test for correctly identifying OOD example
* Moved logic from get_ood_scores to _subtract_confident_thresholds
* TODO: Is this a cleaner way of doing this? Minimal code repeat but strange change to _subtract function
* Addressed type issue
* Switched logic for getting confident thresholds
* Fixing typecheck issues wiht labels parameter being None
* Simplified helper function. Testing type
* Fixed mypy static typing issue
* Mypy typecheck logic test
* removed uncessesary imports in util file
* typechecker debugging (add assert)
* Fixed type logic and removed confident_thresholds=None return
* Added extra arg in helper func to end of func
* Added zero-index checking for label param
* Added ood examples to outlier score notebook
* Added skeleton file structure for implementing outliers
* Make adjust_pred_probs=True by default not false
* Added base Outlier class functionality
TODO:
* test_outlier.py
* Added logic tests for function
* Updated get_ood_scores to always return confident_thresholds
* Even if none were calculated (then return is None)
* Added warning for fit that doesn't calculate confident_thresholds
* Moved get_outlier_scores and get_ood_scores to outlier.py
* Changed param dicts to dicts
* Added docstring to outlier.py
* Added proper return types
* Fixed mypy typing issues
* Switched outliers -> features; ood -> predictions naming conv
* Switched docstring to stem from fit and score functions
* Changed return of helper functions
* Fixed tutorials notebook to use OutOfDistribution class
* Moved imports to top of file
* Added option for different knn objects, addressed pr comments
* Switched params arg to init only
* Addressed PR comment for notebook, cleared notebook
* Fixed PR comments, wording.
* Testing relative links on build
* Changed ood_pred_probs scores to reflect 0 = most ood 1 = least
* Added MLP for detection outliers with pred_probs
* Fixed typos
* Added MLP Import
* Improved warning when fit call unnecessary
* Changed referenced to params dict in warnings/errors
* Added bagging+MLP classifier into notebook
* Reverted tutorial wording
* OOD tutorial improvements
* cleanup OOD documentation
* adopting->using
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com>
* ✨ search tokenized sentence for coloring
Searches through the list of tokens before trying to match substrings. Thanks for this code suggestion Eric!
* 🚧 fix signature in all calls to color_sentence
* ✅ update color_sentence test after changing its api
(sentence, word) -> (word, tokens)
* ⏪ use regex for coloring tokens in sentence
Find word boundaries with regex, use builtin replace() as fallback w/o boundaries.
Closes#288
* 🚑 invert fallback condition
Use replace if NO substitutions were made with regex.
* 🏷️ restrict parameters for list types
* 🐛 only process characters in input token
Example: process_token("Cleanlab", [("C", "a"), ("a", "C")]) should return "aleCnlCb", not "CleCnlCb".
* ✅ use all sentences in test_get_sentence
* ✅ add test cases to test_filter_sentence
* ✅ extent test cases in test_mapping
* 📝 clean up docstrings
Restrict arg types based on docstrings. Fix punctuation and typos. Add examples to docstring.
* ✅ split tests for filter_sentence
* ✅ extend test_merge_probs
* ✅ test merge_probs with ignored/normalized columns in probs
* ✅ extend test cases for get_sentences
* ⚰️ remove unused pandas import
* 🏷️ pass strict mypy check
We ignore np.max as it is untyped.
No issues found in token_classification_utils.py by running
```
mypy --install-types --non-interactive --strict cleanlab/internal/token_classification_utils.py
```
* 👷 add strict type-checking in CI
* 💚 use strict type-check for single file
* ⏪ remove strict type check in CI
* refactor: 🏷️ use np.ndarray type instead of npt.NDArray
* ⏪ go back to generic np.ndarray type
* ♻️ always return tuple in filter_sentence
Remove unused argument+docstring. Simplify relevant unit tests.
* 🔥 resolve comments on typing
Remove ignore-comments. Remove duplicate tag in docstring. Remove unused imports.
* 🔥 remove duplicate tag in docstring
* Changed all instances of np.array in docstring to np.ndarray
* np.array is NOT a proper class name it is just a function to
create np.ndarrays and therefore should not be parameter class
* Black formatting compliance
* validation_func docstring
* torch,tf compatibility+tests
* keras test
* skip tests if python < 3.7
* pytorch numpy int bug on windows
* make tensorflow test work on windows
* move tf env variable setting
* pytorch test increase epochs
* install cpu-tensorflow on windows CI
* torch test optimizer to adam
* fix bugs in shuffled TF dataset
* dummy unit test for TF on windows
* dummy code for TF windows testing
* deal with np.int bug on windows
* remove windows debugging code
* docstrings for new functionality
* address merge conflicts
* reformat after merge
* addressed comments
* 🏷️ annotate function args and return values
Starting with the validation module:
- I think (X, y) might need some custom Union type to handle both numpy arrays and pandas dataframe, etc.
- All of the "assert" functions return None.
Ref #307
* refactor: 🏷️ swap npt.NDArray -> np.ndarray
np.ndarray seems more consistent with the rest of the repo.
Maybe it's necessary to go back to npt.NDArray when disallowing generics?
See numpy docs: https://numpy.org/devdocs/reference/typing.html#numpy.typing.NDArray
* refactor: 🔥 remove unused import
* 🏷️ unconstrain X input types
* 🐛 handle label type issues
- Have to restrict the output of labels_to_array to pass mypy checks.
- Returning the values of pd.Series isn't type-stable.
* 🏷️ include np.generic in arg-type union
* test: ✅ test labels_to_array
* 🏷️ add type aliases for X and y
* 🚨 ignore type-checks for pandas indexing assertions
CI typechecker runs on Python 3.10 which gives this error:
'cleanlab/internal/validation.py:125: error: No overload variant of "__getitem__" of "_iLocIndexerSeries" matches argument type "List[int]"'
It should be fine to let mypy ignore these expressions as they don't return anything.
* 🥅 specify errors to ignore
"type: ignore" doesn't pass strict mypy type-checks unless the specific errors are provided
* 🏷️ annotate label series to array
Our package doesn't have type annotations everywhere, so we can't use
mypy in strict mode just yet. Still, adding type checking in CI is
valuable, so we don't have unchecked annotations in our code.
This patch includes basic fixes to make type checking pass, including
switching the incorrect `np.array` type annotation for `np.ndarray` and
adding some assertions for flow-sensitive typing.
* cleanlearning w dfs
* work for sparse matrix as well
* simplify logic of labels_to_array and extend types
* address pr feedback
* add unit test
* rare label dataframe
* modularize subsetting code
* series rarelabel test
* replace cal.com with slack/email
* Add general method to find num_classes from labels
* compute num_classes with pred_probs.shape[1]
* fix broken commits, address 2nd round of comments
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com>
* df return type, need tests still
* Add pandas as a dependency
We already decided that pandas will be a dependency of cleanlab (also
used in the dataset module, see
https://github.com/cleanlab/cleanlab/pull/182).
* Tweak documentation
* addressed comments
* remove lazy import
* address 2nd round comments
* unit tests
* improve codecov
* Fix typo
* methods to save more space
* nocover statements for prints
* extra nocover
* nocover warnings
* test docstring formatting
* test docstring formatting2
* test docstring formatting2
* move compress to helper, find-label docs params
* readded stuff lost in merge conflict
* addressed remaining PR review comments
* docs formatting
* docs formatting2
* docs formatting3
* docs formatting4
* docs formatting5
* docs formatting5
* docs formatting6
* docs formatting7
* docs formatting8
* docs formatting9
* docs formatting19
* docs formatting20
* docs formatting20
* docs formatting21
* code formatting
* fix a bug where confident joint isnt computed
The confident joint wasn't getting computed if noise_matrix was passed in and pred_probs was not passed in. But that's bad because it stops workflows like:
```python
cl = CleanLearning()
cl.fit(data, labels, noise_matrix=noise_matrix)
cleanlab.dataset.health_summary(labels, confident_joint=cl.confident_joint)
```
* fixed bug from last commit. code in wrong place.
* print overwrite bugfix
Co-authored-by: Anish Athalye <me@anishathalye.com>
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com>
* added LearningWithNoisyLabels.find_label_issues instance method
* LNL.find_label_issues no longer memoizes
* verbose unit test coverage
* Fixed all issues in PR. Fixed confident joint usage in LNL.find_label_issues. Fixed other minor bugs. Added tests
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com>
Co-authored-by: Anish Athalye <me@anishathalye.com>
This was used to skip e.g. importing PyTorch on some tests. However,
PyTorch supports all versions of Python that Cleanlab supports, so
there's no need to issue this warning. Furthermore, having this kind of
code in our codebase increases maintenance burden and contributes to
user frustration. If we mark a version of some library as unsupported,
but the library adds support for it in the future, there's no way users
can use the two packages together without waiting for us to push a new
version of cleanlab that marks that it's compatible and then switching
to the new version, which may be painful for unrelated reasons. It's
better to not have these kinds of checks; if users are using a version
of Python that's unsupported by some other library that they want to
use, that's not our problem, and they'll have issues installing or
importing it and figure it out themselves.
This patch also removes mentions of Python 2.
LearningWithNoisyLabels used `sklearn.utils.check_X_y` to enforce that
`X` was 2D, to be in line with what sklearn's standard estimators
expect. However, LearningWithNoisyLabels is dataset-agnostic: it doesn't
look at the data points themselves. If the underlying classifier
supports data in a different shape, there's no reason
LearningWithNoisyLabels should prohibit it. Users have requested that we
relax this unnecessary restriction [1] so LearningWithNoisyLabels will
more naturally support e.g. image datasets and CNN models.
Thanks to @kothari1997narayan for suggesting this change.
[1]: https://github.com/cleanlab/cleanlab/issues/86
* utilities -> internals
* docbuilding instructions improvement
* fixup formatting of contributing.md
* change contributor guidelines language to be optional
* line formatting
Elías Snorrason