cleanlab--cleanlab
Jonas Mueller
626 行
28 KiB
Plaintext
626 行
28 KiB
Plaintext
{
|
|
"cells": [
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ffe0d62e",
|
|
"metadata": {},
|
|
"source": [
|
|
"# FAQ\n",
|
|
"\n",
|
|
"Answers to frequently asked questions about the [cleanlab](https://github.com/cleanlab/cleanlab) open-source package.\n",
|
|
"\n",
|
|
"The code snippets in this FAQ come from a fully executable notebook you can run via Colab or locally by downloading it [here](https://github.com/cleanlab/cleanlab/blob/master/docs/source/tutorials/faq.ipynb).\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "2a4efdde",
|
|
"metadata": {
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# This cell is hidden on docs.cleanlab.ai. Execute it to ensure all other cells below can be executed in your own notebook\n",
|
|
"\n",
|
|
"import os \n",
|
|
"import logging \n",
|
|
"import numpy as np \n",
|
|
"import sklearn \n",
|
|
"import cleanlab \n",
|
|
"\n",
|
|
"np.random.seed(123)\n",
|
|
"\n",
|
|
"# Toy dataset:\n",
|
|
"N = 50\n",
|
|
"K = 3\n",
|
|
"num_errors = 4\n",
|
|
"labels = np.random.randint(low=0, high=K, size=N)\n",
|
|
"pred_probs = np.random.random_sample(N*K).reshape((N,K))\n",
|
|
"pred_probs[np.arange(N),labels] += 4 # make pred_probs accurate\n",
|
|
"pred_probs = pred_probs/pred_probs.sum(axis=1)[:, np.newaxis]\n",
|
|
"data = np.array([[label+np.random.uniform(), label+np.random.uniform()] for label in labels])\n",
|
|
"# introduce label errors in last few examples:\n",
|
|
"og0_indices = labels[-num_errors:] == 0\n",
|
|
"labels[-num_errors:] = 0\n",
|
|
"labels[-num_errors:][og0_indices] = 1\n",
|
|
"\n",
|
|
"your_classifier=sklearn.linear_model.LogisticRegression() # toy classifier"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d504ec58",
|
|
"metadata": {},
|
|
"source": [
|
|
"### What data can cleanlab detect issues in?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "5e70efbc",
|
|
"metadata": {},
|
|
"source": [
|
|
"Currently, cleanlab can be used to detect label issues in any classification dataset, including those involving: multiple annotators per example (multi-annotator), or multiple labels per example (multi-label). This includes data from any modality such as: image, text, tabular, audio, etc. For text data, cleanlab also supports NLP tasks like entity recognition in which each word is individually labeled (token classification). We're [working to add support](https://github.com/orgs/cleanlab/projects/2) for all other common supervised learning tasks. If you have a particular task in mind, [let us know](https://github.com/cleanlab/cleanlab/issues?q=is%3Aissue)!"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "eca36874",
|
|
"metadata": {},
|
|
"source": [
|
|
"### How do I format classification labels for cleanlab?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d5d0fbb3",
|
|
"metadata": {},
|
|
"source": [
|
|
"cleanlab only works with integer-encoded labels in the range `{0,1, ... K-1}` where `K = number_of_classes`. The `labels` array should only contain integer values in the range `{0, K-1}` and be of shape `(N,)` where `N = total_number_of_data_points`.\n",
|
|
"Do not pass in `labels` where some classes are entirely missing or are extremely rare, as cleanlab may not perform as expected. It is better to remove such classes entirely from the dataset first (also dropping the corresponding dimensions from `pred_probs` and then renormalizing it).\n",
|
|
"\n",
|
|
"**Text or string labels** should to be mapped to integers for each possible value. For example if your original data labels look like this: `[\"dog\", \"dog\", \"cat\", \"mouse\", \"cat\"]`, you should feed them to cleanlab like this: `labels = [1,1,0,2,0]` and keep track of which integer uniquely represents which class (classes were ordered alphabetically in this example). \n",
|
|
"\n",
|
|
"**One-hot encoded labels** should be integer-encoded by finding the argmax along the one-hot encoded axis. An example of what this might look like is shown below."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "239d5ee7",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np \n",
|
|
"\n",
|
|
"# This example arr has 4 labels (one per data point) where \n",
|
|
"# each label can be one of 3 possible classes\n",
|
|
"\n",
|
|
"arr = np.array([[0,1,0],[1,0,0],[0,0,1],[1,0,0]])\n",
|
|
"labels_proper_format = np.argmax(arr, axis=1) # How labels should be formatted when passed into the model"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "4181cac7",
|
|
"metadata": {},
|
|
"source": [
|
|
"### How do I infer the correct labels for examples cleanlab has flagged?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d4db5e1",
|
|
"metadata": {},
|
|
"source": [
|
|
"If you have a classifier that is compatible with [CleanLearning](../cleanlab/classification.html) (i.e. follows the sklearn API), here's an easy way to see predicted labels alongside the label issues:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "28b324aa",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"cl = cleanlab.classification.CleanLearning(your_classifier)\n",
|
|
"issues_dataframe = cl.find_label_issues(data, labels)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "6d4db5e2",
|
|
"metadata": {},
|
|
"source": [
|
|
"Alternatively if you have already computed out-of-sample predicted probabilities (`pred_probs`) from a classifier:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "28b324ab",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"cl = cleanlab.classification.CleanLearning()\n",
|
|
"issues_dataframe = cl.find_label_issues(X=None, labels=labels, pred_probs=pred_probs)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b386dfc8",
|
|
"metadata": {},
|
|
"source": [
|
|
"Otherwise if you have already found issues via:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "90c10e18",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"issues = cleanlab.filter.find_label_issues(labels, pred_probs)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "ad9ca03e",
|
|
"metadata": {},
|
|
"source": [
|
|
"then you can see your trained classifier's class prediction for each flagged example like this: "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "88839519",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class_predicted_for_flagged_examples = pred_probs[issues].argmax(axis=1)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a668b74b",
|
|
"metadata": {},
|
|
"source": [
|
|
"Here you can see the classifier's class prediction for every example via:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "558490c2",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"class_predicted_for_all_examples = pred_probs.argmax(axis=1)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "f9450eed",
|
|
"metadata": {},
|
|
"source": [
|
|
"We caution against just blindly taking the predicted label for granted, many of these suggestions may be wrong! \n",
|
|
"You will be able to produce a much better version of your dataset interactively using [Cleanlab Studio](https://cleanlab.ai/studio/?utm_source=github&utm_medium=docs&utm_campaign=clostostudio), which helps you efficiently fix issues like this in large datasets."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "bcc97591",
|
|
"metadata": {},
|
|
"source": [
|
|
"### How should I handle label errors in train vs. test data?\n",
|
|
"\n",
|
|
"If you do not address label errors in your test data, you may not even know when you have produced a better ML model because the evaluation is too noisy. For the best-trained models and most reliable evaluation of them, you should fix label errors in both training and testing data.\n",
|
|
"\n",
|
|
"To do this efficiently, first use cleanlab to automatically find label issues in both sets. You can simply merge these two sets into one larger dataset and run cross-validation training + `find_label_issues()` on the merged datataset. Calling the [CleanLearning.find_label_issues()](../cleanlab/classification.html) method on your merged dataset does both these steps for you with any scikit-learn compatible classifier you choose.\n",
|
|
"\n",
|
|
"After finding label issues, be **wary** about auto-correcting the labels for test examples (as cautioned against above). Instead manually fix the labels for your test data via careful review of the flagged issues. You can use [Cleanlab Studio](https://cleanlab.ai/studio/) to fix labels efficiently.\n",
|
|
"\n",
|
|
"Auto-correcting labels for your training data is fair game, which should improve ML performance (if properly evaluated with clean test labels). You can boost ML performance further by manually fixing the training examples flagged with label issues, as demonstrated in this article:\n",
|
|
"\n",
|
|
"[**Handling Mislabeled Tabular Data to Improve Your XGBoost Model**](https://cleanlab.ai/blog/label-errors-tabular-datasets/)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "21f42f24",
|
|
"metadata": {},
|
|
"source": [
|
|
"### How can I find label issues in big datasets with limited memory? "
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "089f505e",
|
|
"metadata": {},
|
|
"source": [
|
|
"For a dataset with many rows and/or classes, there are more efficient methods in the `label_issues_batched` module. These methods read data in mini-batches and you can reduce the `batch_size` to control how much memory they require. Below is an example of how to use the `find_label_issues_batched()` method from this module, which can load mini-batches of data from `labels`, `pred_probs` saved as .npy files on disk. You can also run this method on Zarr arrays loaded from .zarr files. Try playing with the `n_jobs` argument for further multiprocessing speedups. If you need greater flexibility, check out the `LabelInspector` class from this module."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "41714b51",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"# We'll assume your big arrays of labels, pred_probs have been saved to file like this:\n",
|
|
"from tempfile import mkdtemp\n",
|
|
"import os.path as path\n",
|
|
"\n",
|
|
"labels_file = path.join(mkdtemp(), \"labels.npy\")\n",
|
|
"pred_probs_file = path.join(mkdtemp(), \"pred_probs.npy\")\n",
|
|
"np.save(labels_file, labels)\n",
|
|
"np.save(pred_probs_file, pred_probs)\n",
|
|
"\n",
|
|
"# Code to find label issues by loading data from file in batches:\n",
|
|
"from cleanlab.experimental.label_issues_batched import find_label_issues_batched\n",
|
|
"\n",
|
|
"batch_size = 10000 # for efficiency, set this to as large of a value as your memory can handle\n",
|
|
"\n",
|
|
"# Indices of examples with label issues, sorted by label quality score (most severe to least severe):\n",
|
|
"indices_of_examples_with_issues = find_label_issues_batched(\n",
|
|
" labels_file=labels_file, pred_probs_file=pred_probs_file, batch_size=batch_size\n",
|
|
")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "20476c70",
|
|
"metadata": {
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# This cell is hidden on docs.cleanlab.ai, and is only for internal testing. You can ignore it.\n",
|
|
"\n",
|
|
"issue_indices = cleanlab.filter.find_label_issues(labels, pred_probs, filter_by = \"low_self_confidence\", return_indices_ranked_by=\"self_confidence\")\n",
|
|
"assert np.abs(len(issue_indices) - len(indices_of_examples_with_issues)) < 2, \"num issues differ in batched mode\"\n",
|
|
"set1 = set(issue_indices)\n",
|
|
"set2 = set(indices_of_examples_with_issues)\n",
|
|
"intersection = len(list(set1.intersection(set2)))\n",
|
|
"union = len(set1) + len(set2) - intersection\n",
|
|
"assert float(intersection) / union > 0.95, \"issue indices differ in batched mode\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "438b424d",
|
|
"metadata": {},
|
|
"source": [
|
|
"**To use less memory and get results faster if your dataset has many classes:** Try merging the rare classes into a single \"Other\" class before you find label issues. The resulting issues won't be affected much since cleanlab anyway does not have enough data to accurately diagnose label errors in classes that are rarely seen. To do this, you should aggregate all the probability assigned to the rare classes in `pred_probs` into a single new dimension of `pred_probs_merged` (where this new array no longer has columns for the rare classes). Here is a function that does this for you, which you can also modify as needed:"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "6983cdad",
|
|
"metadata": {
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# This cell is hidden on docs.cleanlab.ai\n",
|
|
"# Add two rare additional classes to the dataset:\n",
|
|
"\n",
|
|
"num_rare_instances = 3\n",
|
|
"small_prob = 1e-4\n",
|
|
"pred_probs = np.hstack((pred_probs, np.ones((len(pred_probs),2))*small_prob))\n",
|
|
"pred_probs = pred_probs / np.sum(pred_probs, axis=1)[:, np.newaxis]\n",
|
|
"labels[:num_rare_instances] = 3\n",
|
|
"labels[num_rare_instances:(2*num_rare_instances)] = 4"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "9092b8a0",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from cleanlab.internal.util import value_counts # use this to count how often each class occurs in labels\n",
|
|
"\n",
|
|
"def merge_rare_classes(labels, pred_probs, count_threshold = 10):\n",
|
|
" \"\"\" \n",
|
|
" Returns: labels, pred_probs after we merge all rare classes into a single 'Other' class.\n",
|
|
" Merged pred_probs has less columns. Rare classes are any occuring less than `count_threshold` times.\n",
|
|
" Also returns: `class_mapping_orig2new`, a dict to map new classes in merged labels back to classes \n",
|
|
" in original labels, useful for interpreting outputs from `dataset.heath_summary()` or `count.confident_joint()`.\n",
|
|
" \"\"\"\n",
|
|
" num_classes = pred_probs.shape[1]\n",
|
|
" num_examples_per_class = value_counts(labels, num_classes=num_classes)\n",
|
|
" rare_classes = [c for c in range(num_classes) if num_examples_per_class[c] < count_threshold]\n",
|
|
" if len(rare_classes) < 1:\n",
|
|
" raise ValueError(\"No rare classes found at the given `count_threshold`, merging is unnecessary unless you increase it.\")\n",
|
|
"\n",
|
|
" num_classes_merged = num_classes - len(rare_classes) + 1 # one extra class for all the merged ones\n",
|
|
" other_class = num_classes_merged - 1\n",
|
|
" labels_merged = labels.copy()\n",
|
|
" class_mapping_orig2new = {} # key = original class in `labels`, value = new class in `labels_merged`\n",
|
|
" new_c = 0\n",
|
|
" for c in range(num_classes):\n",
|
|
" if c in rare_classes:\n",
|
|
" class_mapping_orig2new[c] = other_class\n",
|
|
" else:\n",
|
|
" class_mapping_orig2new[c] = new_c\n",
|
|
" new_c += 1\n",
|
|
" labels_merged[labels == c] = class_mapping_orig2new[c]\n",
|
|
"\n",
|
|
" merged_prob = np.sum(pred_probs[:, rare_classes], axis=1, keepdims=True) # total probability over all merged classes for each example\n",
|
|
" pred_probs_merged = np.hstack((np.delete(pred_probs, rare_classes, axis=1), merged_prob)) # assumes new_class is as close to original_class in sorted order as is possible after removing the merged original classes\n",
|
|
" # check a few rows of probabilities after merging to verify they still sum to 1:\n",
|
|
" num_check = 1000 # only check a few rows for efficiency\n",
|
|
" ones_array_ref = np.ones(min(num_check,len(pred_probs)))\n",
|
|
" if np.isclose(np.sum(pred_probs[:num_check], axis=1), ones_array_ref).all() and (not np.isclose(np.sum(pred_probs_merged[:num_check], axis=1), ones_array_ref).all()):\n",
|
|
" raise ValueError(\"merged pred_probs do not sum to 1 in each row, check that merging was correctly done.\")\n",
|
|
" \n",
|
|
" return (labels_merged, pred_probs_merged, class_mapping_orig2new)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "b0a01109",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from cleanlab.filter import find_label_issues # can alternatively use find_label_issues_batched() shown above\n",
|
|
"\n",
|
|
"labels_merged, pred_probs_merged, class_mapping_orig2new = merge_rare_classes(labels, pred_probs, count_threshold=5)\n",
|
|
"examples_w_issues = find_label_issues(labels_merged, pred_probs_merged, return_indices_ranked_by=\"self_confidence\")"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8b1da032",
|
|
"metadata": {
|
|
"nbsphinx": "hidden"
|
|
},
|
|
"outputs": [],
|
|
"source": [
|
|
"# This cell is hidden on docs.cleanlab.ai, and is only for internal testing. You can ignore it.\n",
|
|
"\n",
|
|
"rare_classes = [c for c in class_mapping_orig2new.keys() if class_mapping_orig2new[c] == pred_probs_merged.shape[1]-1]\n",
|
|
"og_examples_w_issues = find_label_issues(labels, pred_probs, return_indices_ranked_by=\"self_confidence\")\n",
|
|
"examples_of_interest = [x for x in examples_w_issues if labels[x] not in rare_classes]\n",
|
|
"og_examples_of_interest = [x for x in og_examples_w_issues if labels[x] not in rare_classes]\n",
|
|
"assert set(examples_of_interest) == set(og_examples_of_interest), \"merged label issues differ from non-merged label issues\""
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3868ee8b",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Why isn’t CleanLearning working for me?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "d13c9cd0",
|
|
"metadata": {},
|
|
"source": [
|
|
"At this time, CleanLearning only works with data formatted as numpy matrices or pd.DataFrames, \n",
|
|
"and with models that are compatible with the `sklearn` API \n",
|
|
"(check out [skorch](https://github.com/skorch-dev/skorch) for Pytorch compatibility and [scikeras](https://github.com/adriangb/scikeras) for Tensorflow/Keras compatibility). \n",
|
|
"You can still use cleanlab with other data formats though! Just separately obtain predicted probabilities (`pred_probs`) from your model via cross-validation and pass them as inputs. \n",
|
|
"\n",
|
|
"\n",
|
|
"If CleanLearning is running successfully but not improving predictive accuracy of your model, here are some tips:\n",
|
|
"\n",
|
|
"1. Use cleanlab to find label issues in your test data as well (we recommend pooling `labels` across both training and test data into one input for `find_label_issues()`). Then manually review and fix label issues identified in the test data to verify accuracy measurements are actually meaningful.\n",
|
|
"\n",
|
|
"2. Try different values for `filter_by`, `frac_noise`, and `min_examples_per_class` which can be set via the `find_label_issues_kwargs` argument in the initialization of `CleanLearning()`.\n",
|
|
"\n",
|
|
"3. Try to find a better model (eg. via hyperparameter tuning or changing to another classifier). `CleanLearning` can find better label issues by leveraging a better model, which allows it to produce better quality training data. This can form a virtuous cycle in which better models -> better issue detection -> better data -> even better models! \n",
|
|
"\n",
|
|
"4. Try jointly tuning both model hyperparameters and `find_label_issues_kwargs` values.\n",
|
|
"\n",
|
|
"5. Does your dataset have a *junk* (or *clutter*, *unknown*, *other*) class? If you have bad data, consider creating one (c.f. Caltech-256).\n",
|
|
"\n",
|
|
"6. Consider merging similar/overlapping classes found via ``cleanlab.dataset.find_overlapping_classes``.\n",
|
|
"\n",
|
|
"Other general tips to improve label error detection performance:\n",
|
|
"\n",
|
|
"1. Try creating more restrictive new filters by combining their intersections (e.g. `combined_boolean_mask = mask1 & mask2` where `mask1` and `mask2` are the boolean masks created by running `find_label_issues` with different values of the `filter_by` argument).\n",
|
|
"\n",
|
|
"2. If your `pred_probs` are obtained via a neural network, try averaging the `pred_probs` over the last K epochs of training instead of just using the final `pred_probs`. Similarly, you can try averaging `pred_probs` from several models (remember to re-normalize) or using ``cleanlab.rank.get_label_quality_ensemble_scores``.\n"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "9ae3899c",
|
|
"metadata": {},
|
|
"source": [
|
|
"### How can I use different models for data cleaning vs. final training in CleanLearning?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "a2ce1518",
|
|
"metadata": {},
|
|
"source": [
|
|
"The code below demonstrates CleanLearning with 2 different classifiers: `LogisticRegression()` and `GradientBoostingClassifier()`.\n",
|
|
"A `LogisticRegression` model is used to detect label issues (via cross-validation run inside CleanLearning) and a `GradientBoostingClassifier` model is finally trained on a clean subset of the data with issues removed.\n",
|
|
"This can be done with any two classifiers."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "4c9e9030",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"from cleanlab.classification import CleanLearning\n",
|
|
"import numpy as np\n",
|
|
"from sklearn.linear_model import LogisticRegression\n",
|
|
"from sklearn.ensemble import GradientBoostingClassifier\n",
|
|
"\n",
|
|
"# Make example data\n",
|
|
"data = np.vstack([np.random.random((100, 2)), np.random.random((100, 2)) + 10])\n",
|
|
"labels = np.array([0] * 100 + [1] * 100)\n",
|
|
"\n",
|
|
"# Introduce label errors\n",
|
|
"true_errors = [97, 98, 100, 101, 102, 104]\n",
|
|
"for idx in true_errors:\n",
|
|
" labels[idx] = 1 - labels[idx]\n",
|
|
"\n",
|
|
"# CleanLearning with 2 different classifiers: one classifier is used to detect label issues \n",
|
|
"# and a different classifier is subsequently trained on the clean subset of the data.\n",
|
|
"\n",
|
|
"model_to_find_errors = LogisticRegression() # this model will be trained many times via cross-validation\n",
|
|
"model_to_return = GradientBoostingClassifier() # this model will be trained once on clean subset of data\n",
|
|
"\n",
|
|
"cl0 = CleanLearning(model_to_find_errors)\n",
|
|
"issues = cl0.find_label_issues(data, labels)\n",
|
|
"\n",
|
|
"cl = CleanLearning(model_to_return).fit(data, labels, label_issues=issues)\n",
|
|
"pred_probs = cl.predict_proba(data) # predictions from GradientBoostingClassifier\n",
|
|
"\n",
|
|
"print(cl0.clf) # will be LogisticRegression()\n",
|
|
"print(cl.clf) # will be GradientBoostingClassifier()"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "b71fef02",
|
|
"metadata": {},
|
|
"source": [
|
|
"### How do I hyperparameter tune only the final model trained (and not the one finding label issues) in CleanLearning?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "e7ec1956",
|
|
"metadata": {},
|
|
"source": [
|
|
"The code below demonstrates CleanLearning using a `GradientBoostingClassifier()` with no hyperparameter-tuning to find label issues but with hyperparameter-tuning via `RandomizedSearchCV(...)` for the final training of this model on the clean subset of the data.\n",
|
|
"This is a useful trick to avoid expensive hyperparameter-tuning for every fold of cross-validation (which is needed to find label issues)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "code",
|
|
"execution_count": null,
|
|
"id": "8751619e",
|
|
"metadata": {},
|
|
"outputs": [],
|
|
"source": [
|
|
"import numpy as np\n",
|
|
"from cleanlab.classification import CleanLearning\n",
|
|
"from sklearn.ensemble import GradientBoostingClassifier\n",
|
|
"from sklearn.model_selection import RandomizedSearchCV\n",
|
|
"\n",
|
|
"# Make example data\n",
|
|
"data = np.vstack([np.random.random((100, 2)), np.random.random((100, 2)) + 10])\n",
|
|
"labels = np.array([0] * 100 + [1] * 100)\n",
|
|
"\n",
|
|
"# Introduce label errors\n",
|
|
"true_errors = [97, 98, 100, 101, 102, 104]\n",
|
|
"for idx in true_errors:\n",
|
|
" labels[idx] = 1 - labels[idx]\n",
|
|
"\n",
|
|
"# CleanLearning with no hyperparameter-tuning during expensive cross-validation to find label issues\n",
|
|
"# but hyperparameter-tuning for the final training of model on clean subset of the data:\n",
|
|
"\n",
|
|
"model_to_find_errors = GradientBoostingClassifier() # this model will be trained many times via cross-validation\n",
|
|
"model_to_return = RandomizedSearchCV(GradientBoostingClassifier(),\n",
|
|
" param_distributions = {\n",
|
|
" \"learning_rate\": [0.001, 0.05, 0.1, 0.2, 0.5],\n",
|
|
" \"max_depth\": [3, 5, 10],\n",
|
|
" }\n",
|
|
" ) # this model will be trained once on clean subset of data\n",
|
|
"\n",
|
|
"cl0 = CleanLearning(model_to_find_errors)\n",
|
|
"issues = cl0.find_label_issues(data, labels)\n",
|
|
"\n",
|
|
"cl = CleanLearning(model_to_return).fit(data, labels, label_issues=issues) # CleanLearning for hyperparameter final training\n",
|
|
"pred_probs = cl.predict_proba(data) # predictions from hyperparameter-tuned GradientBoostingClassifier\n",
|
|
"\n",
|
|
"print(cl0.clf) # will be GradientBoostingClassifier()\n",
|
|
"print(cl.clf) # will be RandomizedSearchCV(estimator=GradientBoostingClassifier(),...)"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3a28168h",
|
|
"metadata": {},
|
|
"source": [
|
|
"### What ML models should I run cleanlab with? How do I fix the issues cleanlab has identified?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a117547",
|
|
"metadata": {},
|
|
"source": [
|
|
"These questions are automatically handled for you in [Cleanlab Studio](https://cleanlab.ai/studio/?utm_source=github&utm_medium=docs&utm_campaign=clostostudio) -- our platform for no-code data improvement.\n",
|
|
"While this open-source library **finds** data issues, an interface is needed to efficiently **fix** these issues in your dataset. [Cleanlab Studio](https://cleanlab.ai/studio/?utm_source=github&utm_medium=docs&utm_campaign=clostostudio) is a no-code platform to find and fix problems in real-world ML datasets. Studio automatically runs optimized versions of the algorithms from this open-source library on top of AutoML models fit to your data, and presents detected issues in a smart data editing interface. Think of it like a data cleaning assistant that helps you quickly improve the quality of your data (via AI/automation + streamlined UX)."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "3a28168f",
|
|
"metadata": {},
|
|
"source": [
|
|
"### What license is cleanlab open-sourced under?"
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1a117546",
|
|
"metadata": {},
|
|
"source": [
|
|
"[AGPL-3.0 license](https://github.com/cleanlab/cleanlab/blob/master/LICENSE)\n",
|
|
"\n",
|
|
"**What does this mean?** If you're working at a company, you can use this open-source library to clean up your internal datasets. You can also use this open-source library to clean up a dataset used to train a model that is deployed in a commercial product.\n",
|
|
"For non-commercial purposes, feel free to release altered versions of the source code as long as you include the same license.\n",
|
|
"\n",
|
|
"Please email `info@cleanlab.ai` to discuss licensing needs if you would like to offer a commercial product that utilizes any cleanlab source code."
|
|
]
|
|
},
|
|
{
|
|
"cell_type": "markdown",
|
|
"id": "1520a93f",
|
|
"metadata": {},
|
|
"source": [
|
|
"### Can't find an answer to your question?\n",
|
|
"\n",
|
|
"If your question is not addressed in these tutorials, please refer to the: [Cleanlab Github issues](https://github.com/cleanlab/cleanlab/issues?q=is%3Aissue), [Cleanlab Code Examples](https://github.com/cleanlab/examples) or our [Slack Community](https://cleanlab.ai/slack).\n",
|
|
"\n",
|
|
"If your question is not addressed anywhere, please open a [new Github issue](https://github.com/cleanlab/cleanlab/issues/new/choose). Our developers may also provide personalized assistance in our [Slack Community](https://cleanlab.ai/slack)."
|
|
]
|
|
}
|
|
],
|
|
"metadata": {
|
|
"kernelspec": {
|
|
"display_name": "Python 3 (ipykernel)",
|
|
"language": "python",
|
|
"name": "python3"
|
|
},
|
|
"language_info": {
|
|
"codemirror_mode": {
|
|
"name": "ipython",
|
|
"version": 3
|
|
},
|
|
"file_extension": ".py",
|
|
"mimetype": "text/x-python",
|
|
"name": "python",
|
|
"nbconvert_exporter": "python",
|
|
"pygments_lexer": "ipython3",
|
|
"version": "3.9.12"
|
|
}
|
|
},
|
|
"nbformat": 4,
|
|
"nbformat_minor": 5
|
|
}
|