7d9a98b0cf
black formatting
2022-11-28 01:09:52 -08:00
f0226d5f3e
increment version to 2.2
2022-11-28 01:04:55 -08:00
8fda383d2d
point to find_label_issues in multilabel_classification.py
2022-11-28 00:46:56 -08:00
7b87535e48
Remove unnecessary type: ignore annotations ( #549 )
2022-11-26 14:44:54 +00:00
91cb6a7de0
Fix passing of kwargs to get_label_quality_scores ( #547 )
...
The functions `_get_consensus_stats` and
`_get_annotator_label_quality_score` take an argument
`label_quality_score_kwargs`, a dictionary of keyword arguments to pass
to `get_label_quality_scores`. When passing a
`label_quality_score_kwargs` dictionary to these functions, using the
unpacking operator is incorrect: that would be an extra level of
unpacking. The _implementations_ of these functions will unpack the
`label_quality_score_kwargs` when calling `get_label_quality_scores`.
This patch fixes the issue and adds a basic regression test.
[skip ci]
2022-11-26 14:41:47 +00:00
d4fe568024
Fix types to work with latest pandas-stubs ( #546 )
...
The latest pandas-stubs (1.5.2.221124) changed the type of
`.apply(f, ...)` such that the function `f` must return either a
`Scalar` or a `Series`. This patch updates the functions to return a
`Series`. Note that this has minimal impact on runtime performance,
because by default, the pd.Series constructor does *not* copy the
underlying data.
[skip ci]
2022-11-26 14:39:33 +00:00
a222859569
Improve docs in label error detection methods for classification data + filter/count mypy type annotations ( #543 )
2022-11-23 23:01:36 -08:00
17a82c32fc
Public multilabel quality scores method + softmin aggregation + more tests ( #542 )
...
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2022-11-23 18:34:43 -08:00
7139a8fd99
Refactor MultilabelScorer helper methods and tests ( #540 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-11-20 15:43:23 -08:00
4d0ad6ed04
Fix multilabel_py dimensionality ( #535 )
...
* Go from a (2**K,) vector to a (K, 2) matrix of prior probabilities.
2022-11-14 15:19:17 +00:00
886e6aee6e
make mypy v0.990 happy ( #530 )
2022-11-08 09:56:51 +00:00
7b9c0cf54e
Remove strong worded requirement for out-of-sample pred probs ( #520 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-11-07 20:10:07 -08:00
25025d9275
Validate forgetting factor in EMA ( #527 )
...
- Added runtime check that `0 <= alpha <= 1` to ensure EMA uses proportional weighting.
- Moved all EMA tests to separate class.
2022-11-07 15:56:33 +00:00
f37ba9ff39
support missing classes in multilabel tasks ( #523 )
2022-11-04 15:00:43 -07:00
16530f40cf
fix mypy typing ( #524 )
2022-11-04 17:25:50 -04:00
1085faca4a
support missing classes in multiannotator functions ( #519 )
2022-11-03 23:25:03 -07:00
cbe3ce0f20
move multilabel functions to utils ( #515 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-11-03 10:16:28 -07:00
2bff4d5dd6
Better missing class support for label quality scoring ( #518 )
2022-11-02 18:14:16 -07:00
c2d319d0ed
Add support for missing classes ( #511 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-10-31 16:23:41 -07:00
8ff371c16c
separate word coloring from token-replacement ( #514 )
...
Fixes #513
2022-10-31 14:21:05 +00:00
b12d76b6d1
Multilabel code restructuring with aggregation/scorer functions ( #509 )
...
_find_label_issues_multilabel uses EMA instead of mean-pooling when computing label quality scores
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-10-28 16:26:39 -07:00
738acae810
Point to format label function for multiannotator ( #506 )
2022-10-25 10:52:21 -07:00
f8a831d239
Add internal function to properly format labels ( #504 )
2022-10-24 10:54:59 -07:00
7761aefa18
Trying to fix typings ( #502 )
...
* Add typing for filter.py
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2022-10-20 21:53:11 +00:00
00781ce984
update paper links ( #503 )
...
Co-authored-by: huiwengoh <45724323+huiwengoh@users.noreply.github.com >
2022-10-18 00:01:00 -07:00
e060f551fb
supporting multilabel via one-vs-rest reductions ( #483 )
...
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-10-17 12:09:23 -07:00
26fa264344
Extending label quality scores to multilabel data ( #499 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-10-13 09:56:43 -07:00
57bcfd0a99
Mypy typechecking fix for num_label_errors ( #500 )
2022-10-07 19:43:40 -07:00
2906bf6292
Update # issues returned by num_label_issues ( #485 )
...
* Added code and test for returning correct num label issues
* Fixed docstring
* Addressed mypy uncertain typing issue
* Added typing and typing checks
* Addressed PR comments
2022-10-05 20:55:49 -07:00
42a2f6851f
increment version beyond latest release ( #476 )
2022-10-05 17:43:00 -07:00
7bfdd9169d
Make type checking pass with mypy 0.981 ( #488 )
2022-10-05 17:42:11 -07:00
72c4297c16
Update __init__.py to include major files ( #490 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-10-05 17:41:13 -07:00
57adf33369
Adding pylint CI checks ( #465 )
...
* Update ci.yml
* remove unused import
* 💚 merge pylint jobs
* 🔧 add .pylintrc
* 🚨 handle pylint errors
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2022-09-23 16:10:03 +00:00
32f7319da9
fix return type of token classification get_label_quality_scores [include in 2.1 docs]
2022-09-22 14:40:18 -07:00
34acd0f19f
tf-io tutorial version pinned, docstring improvements [include in 2.1 docs] ( #466 )
2022-09-22 14:37:44 -07:00
1d33ea8b3c
let label_quality_utils docstrings type be autogenerated [include in 2.1 docs] ( #475 )
2022-09-22 14:34:27 -07:00
dd95a1c2bb
Add helper function to reduce code duplication ( #463 )
2022-09-16 20:38:25 -07:00
460e1fa0c9
update readme +version for v2.1 ( #457 )
...
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com >
2022-09-16 20:21:36 -07:00
5da046da9e
Added support for returning ranked issue idxs ( #459 )
...
* Added support for returning ranked issue idxs
- code
- tests
- docstring
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com >
2022-09-16 20:20:52 -07:00
06ed233a25
Error handling for rare classes ( #455 )
...
* error handling for rare classes
* change subtract to symmetric difference
* remove extra np.unique
* add warning for all instances of getting consensus labels
* error checking edits
* Add typing
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
* black formatting
* update typing - labels_multiannotator will always already be converted to pd.dataframe
* make pred_probs options in typing
* add =None
* docstring edits
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
* add basic docstring
* Changed to verbose
* Removed unessesary calculation out of get_labels_quality_m
* Update cleanlab/multiannotator.py
* comment for lost classes check so it can be grepped
* caution about setting verbose to false
* advise against verbose=false in docstring
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
Co-authored-by: Ulyana <ulyana@cleanlab.ai >
2022-09-16 19:26:29 -07:00
7256cd3546
More improvements to token classification code and documentation ( #452 )
...
* improved token docs/code
* format paragraphs in docstrings
* fix typo
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2022-09-16 14:04:45 +00:00
2959210c95
more links in keras wrapper docstrings
2022-09-15 20:20:31 -07:00
0657b14792
add links in keras wrapper docs
2022-09-15 19:20:37 -07:00
bc2dabb764
Change output of display_issues ( #450 )
2022-09-15 18:05:46 -07:00
95b0742342
Make softmin_sentence_score a private function ( #449 )
...
This scoring function is only used internally. Might as well be private.
2022-09-15 23:19:15 +00:00
1b813d9666
fix bug in hard-coded test. generalize the test ( #448 )
...
* fix bug in hard-coded test. generalize the test
* 🐛 cast rounded num_issues to int
np.rint outputs an array of the same shape and type as its input. num_label_issues is expected to return an integer.
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2022-09-15 18:16:56 -04:00
1b76f4ff1b
Improve docstrings for keras wrapper ( #447 )
...
* improved docstrings for keras wrapper
* hide inheritance from keras wrapper docstrings
* black formatting
2022-09-15 15:36:35 -04:00
3ff8fab2d9
make num_label_issues = cj calibrated offdiag sum ( #445 )
...
The discrepancy occurs because cj calibrated maintains a guarantee that it will perfectly count every example in the dataset (assumes no out of distribution examples).
You can see this when by following this change of methods:
* see `calibrate_confident_joint`, specifically this line here: https://github.com/cleanlab/cleanlab/blob/master/cleanlab/count.py#L172
* round here https://github.com/cleanlab/cleanlab/blob/9240a8c56d020abec8566a0a56d22dcb99d32528/cleanlab/internal/util.py#L209
* the actual rounding occurs here https://github.com/cleanlab/cleanlab/blob/9240a8c56d020abec8566a0a56d22dcb99d32528/cleanlab/internal/util.py#L174
the fix to make `num_label_issues` the same is just to round instead of flooring.
see the two GREEN columns in the attached image
> [nit] consider `np.rint()` instead of `np.round()` for clarity
good call. updated
2022-09-15 11:44:27 -07:00
a4ad6460a2
Format return docstrings and add typing ( #437 )
...
* typing and return docstrings
* address typing complaints
* allow redefinition
* typing complains for pandas
* .values -> to_numpy()
2022-09-15 11:46:14 +00:00
7ccdb5491f
Add keras wrapper to docs ( #443 )
...
* Delete huggingface_keras_classifier.py
* Update keras.py
* readme language, remove huggingfacekerasclassifier
* language improvements
* reorder modules
* make keras.py appear in docs
2022-09-15 00:47:31 -07:00
Jonas Mueller