f77ef9f065
increase hypothesis deadline to 500ms for test_find_overlapping_classes_with_confident_joint ( #695 )
...
This addresses the flaky test issue on macOS and Windows platforms in the CI, where the test occasionally exceeded the previous 200ms deadline.
2023-05-04 13:14:26 -04:00
a95a447010
Update tests for custom issue manager example ( #692 )
...
* issuemanager.get_summary->make_summary
* test(datalab): ♻️ move tests for issue managers into their respective test modules
* test(datalab): ✅ add tests for custom issue manager
- Validate scores provided to IssueManager.make_summary
- Move fixture for custom issue manager to conftest.py
- Test make_summary on a "custom" IssueManager
* test(datalab): add __init__.py for discoverability of issue manager tests
---------
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2023-05-03 00:40:27 +00:00
f8b11d37b6
refactor(datalab): ♻️ rename get_summary -> get_issue_summary ( #691 )
...
This only pertains to Datalab/DataIssues. IssueManager has a different method with the get_summary name that is used in a different way.
2023-05-02 19:15:31 +00:00
d5e81f0c6d
allow for kwargs in token find_label_issues ( #686 )
2023-04-28 10:28:39 -07:00
5650a87d85
Remove the "health_summary_kwargs" dictionary argument from LabelIssueManager ( #683 )
2023-04-28 00:01:05 +00:00
ff3bdd1e48
Add a less restrictive Reporter.report() method for Datalab ( #680 )
...
Add a new method `Reporter.report()` with fewer restrictions. The existing `get_report()` method is a pure function that returns a string as output. The new `report()` method can have side-effects, such as printing the report.
2023-04-27 18:23:01 +00:00
64edc95630
Introduce Datalab ( #614 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
Co-authored-by: Jesse Cummings <jecummin@mit.edu >
Co-authored-by: Jesse Cummings <jecummin@users.noreply.github.com >
Co-authored-by: Hui Wen <45724323+huiwengoh@users.noreply.github.com >
2023-04-25 09:26:43 -07:00
f8c1866c58
move methods to multilabel_classification module ( #657 )
...
adds new dataset summarization methods for multilabel settings
---------
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2023-04-13 21:50:11 -07:00
1e653def93
remove min batch size restriction in LabelInspector ( #665 )
2023-04-11 13:05:02 -07:00
64c6cfcdd5
Pass confident joint computed in CleanLearning to filter.find_label_issues ( #661 )
2023-04-05 21:21:17 -07:00
fa1db6e1d5
Bug fixes + improvements to multiannotator module ( #654 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2023-03-20 23:36:41 -07:00
00776646a7
Fix bug with call to find_overlapping_issues without specifying labels ( #652 )
...
* fix None labels issue
* build: ➕ add hypothesis to dev dependencies. Used for property-based testing
* test: ✅ test that find_overlapping_classes can run by only providing a confident joint
Resolves #651
---------
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2023-03-13 15:37:00 +00:00
6eb60198e6
update tests to be more stringent ( #635 )
2023-02-15 17:34:11 -08:00
3757637d8c
Fix batched multiprocessing being slower on tall matrices ( #634 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2023-02-15 13:14:39 -08:00
4850050803
label_issues_batched multiprocessing ( #630 )
...
4x speedup (48 cores, 10k classes x 1M examples)
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2023-02-12 20:25:02 -08:00
4d2753ae05
Fix KerasWrapper summary method ( #631 )
2023-02-12 18:40:52 -08:00
686cbf63f1
Method to estimate label issues with limited memory via mini-batches ( #615 )
...
Co-authored-by: clu0 <33559427+clu0@users.noreply.github.com >
2023-02-10 16:59:16 -08:00
d99788b9de
Fixed duplicate issue removal in find_label_issues ( #624 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2023-02-10 08:28:29 -08:00
6aaee8332b
Create new cleanlab.models module ( #601 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2023-02-07 10:03:32 -08:00
c191d8781d
Fix format compatibility with latest black==23. release ( #620 )
2023-02-07 10:01:45 -08:00
35d5323479
Fix dictionary type annotation for OutOfDistribution object ( #616 )
...
* Also improve params dict default to be None
2023-02-01 20:07:31 -08:00
0c02ec953b
Add two methods to filter.find_label_issues ( #595 )
...
"low_self_confidence" or "low_ normalized_margin"
Co-authored-by: huiwengoh <45724323+huiwengoh@users.noreply.github.com >
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2023-01-31 23:26:20 -08:00
888246e2d5
Improve num_label_issues usage of confident_joint to match find_label_issues ( #610 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2023-01-27 14:47:02 -08:00
2b6c56424d
10x speedup in find_label_issues on linux via better multiprocessing ( #596 )
...
and bug fixes related to partition vs sort
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2023-01-19 20:14:03 -05:00
7b589f6d63
updating copyright year to include 2023 ( #594 )
2023-01-06 21:25:57 -08:00
ebadffdebe
Use euclidean distance for identifying outliers for lower dimensional features ( #581 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2023-01-06 21:25:23 -08:00
248bb91378
Revamp text tutorial to use cleanlab Keras wrapper ( #584 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2023-01-06 08:53:02 -08:00
f61469420b
Multiannotator Active Learning Support ( #538 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-12-21 18:31:24 -08:00
91cb6a7de0
Fix passing of kwargs to get_label_quality_scores ( #547 )
...
The functions `_get_consensus_stats` and
`_get_annotator_label_quality_score` take an argument
`label_quality_score_kwargs`, a dictionary of keyword arguments to pass
to `get_label_quality_scores`. When passing a
`label_quality_score_kwargs` dictionary to these functions, using the
unpacking operator is incorrect: that would be an extra level of
unpacking. The _implementations_ of these functions will unpack the
`label_quality_score_kwargs` when calling `get_label_quality_scores`.
This patch fixes the issue and adds a basic regression test.
[skip ci]
2022-11-26 14:41:47 +00:00
17a82c32fc
Public multilabel quality scores method + softmin aggregation + more tests ( #542 )
...
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2022-11-23 18:34:43 -08:00
7139a8fd99
Refactor MultilabelScorer helper methods and tests ( #540 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-11-20 15:43:23 -08:00
4d0ad6ed04
Fix multilabel_py dimensionality ( #535 )
...
* Go from a (2**K,) vector to a (K, 2) matrix of prior probabilities.
2022-11-14 15:19:17 +00:00
25025d9275
Validate forgetting factor in EMA ( #527 )
...
- Added runtime check that `0 <= alpha <= 1` to ensure EMA uses proportional weighting.
- Moved all EMA tests to separate class.
2022-11-07 15:56:33 +00:00
f37ba9ff39
support missing classes in multilabel tasks ( #523 )
2022-11-04 15:00:43 -07:00
1085faca4a
support missing classes in multiannotator functions ( #519 )
2022-11-03 23:25:03 -07:00
cbe3ce0f20
move multilabel functions to utils ( #515 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-11-03 10:16:28 -07:00
2bff4d5dd6
Better missing class support for label quality scoring ( #518 )
2022-11-02 18:14:16 -07:00
c2d319d0ed
Add support for missing classes ( #511 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-10-31 16:23:41 -07:00
8ff371c16c
separate word coloring from token-replacement ( #514 )
...
Fixes #513
2022-10-31 14:21:05 +00:00
b12d76b6d1
Multilabel code restructuring with aggregation/scorer functions ( #509 )
...
_find_label_issues_multilabel uses EMA instead of mean-pooling when computing label quality scores
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-10-28 16:26:39 -07:00
f8a831d239
Add internal function to properly format labels ( #504 )
2022-10-24 10:54:59 -07:00
e060f551fb
supporting multilabel via one-vs-rest reductions ( #483 )
...
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-10-17 12:09:23 -07:00
26fa264344
Extending label quality scores to multilabel data ( #499 )
...
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
2022-10-13 09:56:43 -07:00
2906bf6292
Update # issues returned by num_label_issues ( #485 )
...
* Added code and test for returning correct num label issues
* Fixed docstring
* Addressed mypy uncertain typing issue
* Added typing and typing checks
* Addressed PR comments
2022-10-05 20:55:49 -07:00
57adf33369
Adding pylint CI checks ( #465 )
...
* Update ci.yml
* remove unused import
* 💚 merge pylint jobs
* 🔧 add .pylintrc
* 🚨 handle pylint errors
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2022-09-23 16:10:03 +00:00
5da046da9e
Added support for returning ranked issue idxs ( #459 )
...
* Added support for returning ranked issue idxs
- code
- tests
- docstring
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
Co-authored-by: Curtis G. Northcutt <curtis.northcutt@gmail.com >
2022-09-16 20:20:52 -07:00
06ed233a25
Error handling for rare classes ( #455 )
...
* error handling for rare classes
* change subtract to symmetric difference
* remove extra np.unique
* add warning for all instances of getting consensus labels
* error checking edits
* Add typing
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
* black formatting
* update typing - labels_multiannotator will always already be converted to pd.dataframe
* make pred_probs options in typing
* add =None
* docstring edits
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
* add basic docstring
* Changed to verbose
* Removed unessesary calculation out of get_labels_quality_m
* Update cleanlab/multiannotator.py
* comment for lost classes check so it can be grepped
* caution about setting verbose to false
* advise against verbose=false in docstring
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
Co-authored-by: Jonas Mueller <1390638+jwmueller@users.noreply.github.com >
Co-authored-by: Ulyana <ulyana@cleanlab.ai >
2022-09-16 19:26:29 -07:00
7256cd3546
More improvements to token classification code and documentation ( #452 )
...
* improved token docs/code
* format paragraphs in docstrings
* fix typo
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2022-09-16 14:04:45 +00:00
95b0742342
Make softmin_sentence_score a private function ( #449 )
...
This scoring function is only used internally. Might as well be private.
2022-09-15 23:19:15 +00:00
1b813d9666
fix bug in hard-coded test. generalize the test ( #448 )
...
* fix bug in hard-coded test. generalize the test
* 🐛 cast rounded num_issues to int
np.rint outputs an array of the same shape and type as its input. num_label_issues is expected to return an integer.
Co-authored-by: Elías Snorrason <eliassno@gmail.com >
2022-09-15 18:16:56 -04:00
Elías Snorrason