mirrors / cleanlab--cleanlab

项目文件夹

cleanlab--cleanlab
文件
jwmueller-quickstartdatalab
T
Sanjana cf5d2a02eb Added installation instructions for package extras (#697) 
Co-authored-by: Elías Snorrason <eliassno@gmail.com>
Co-authored-by: huiwengoh <45724323+huiwengoh@users.noreply.github.com>
2023-05-09 15:19:24 -04:00

552 行
16 KiB
Plaintext
原始文件 永久链接 Blame 文件历史

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Image Classification with PyTorch and Datalab\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This 5-minute quickstart tutorial demonstrates how to find potential label errors in image classification data. Here we use the MNIST dataset containing 70,000 images of handwritten digits from 0 to 9.\n",
"\n",
"**Overview of what we'll do in this tutorial:**\n",
"\n",
"- Build a simple [PyTorch](https://pytorch.org/) neural net and wrap it with [skorch](https://skorch.readthedocs.io/) to make it scikit-learn compatible.\n",
"\n",
"- Use this model to compute out-of-sample predicted probabilities, `pred_probs`, via cross-validation.\n",
"\n",
"- Use these predictions to estimate which images in the dataset are mislabeled via cleanlab's `Datalab` class.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
" \n",
"Already have a `model`? Run cross-validation to get out-of-sample `pred_probs` and then the code below to find any potential label errors in your dataset.\n",
"\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```python\n",
"from cleanlab import Datalab\n",
"\n",
"lab = Datalab(data=your_dataset, label_name=\"column_name_of_labels\")\n",
"lab.find_issues(pred_probs=your_pred_probs, issue_types={\"label\":{}})\n",
"\n",
"lab.get_issues(\"label\")\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install and import required dependencies\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"You can use `pip` to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install matplotlib torch torchvision skorch datasets\n",
"!pip install \"cleanlab[datalab]\"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Package installation (this cell is hidden from docs.cleanlab.ai).\n",
"# If running on Colab, may want to use GPU (select: Runtime > Change runtime type > Hardware accelerator > GPU)\n",
"# Package versions used: matplotlib==3.5.1 torch==1.11.0 skorch==0.11.0\n",
"\n",
"dependencies = [\"cleanlab\", \"matplotlib\", \"torch\", \"torchvision\", \"skorch\", \"datasets\"]\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" missing_dependencies = []\n",
" for dependency in dependencies:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")\n",
"\n",
"# Suppress benign warnings: \n",
"import warnings \n",
"warnings.filterwarnings(\"ignore\", \"Lazy modules are a new feature.*\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"from torch import nn\n",
"from sklearn.datasets import fetch_openml\n",
"from sklearn.model_selection import cross_val_predict\n",
"from sklearn.metrics import accuracy_score\n",
"from skorch import NeuralNetClassifier"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# This (optional) cell is hidden from docs.cleanlab.ai \n",
"\n",
"import numpy as np \n",
"\n",
"SEED = 123 # for reproducibility \n",
"np.random.seed(SEED)\n",
"torch.manual_seed(SEED)\n",
"torch.backends.cudnn.deterministic = True\n",
"torch.backends.cudnn.benchmark = False\n",
"torch.cuda.manual_seed_all(SEED)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Fetch and scale the MNIST dataset\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"mnist = fetch_openml(\"mnist_784\") # Fetch the MNIST dataset\n",
"\n",
"X = mnist.data.astype(\"float32\").to_numpy() # 2D array (images are flattened into 1D)\n",
"X /= 255.0 # Scale the features to the [0, 1] range\n",
"X = X.reshape(len(X), 1, 28, 28) # reshape into [N, C, H, W] for PyTorch\n",
"\n",
"labels = mnist.target.astype(\"int64\").to_numpy() # 1D array of given labels"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Bringing Your Own Data (BYOD)?\n",
"\n",
"Assign your data's features to variable `X` and its labels to variable `labels` instead, and continue with the rest of the tutorial.\n",
"\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Define a classification model\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, we define a simple neural network with PyTorch.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"class ClassifierModule(nn.Module):\n",
" def __init__(self):\n",
" super().__init__()\n",
"\n",
" self.cnn = nn.Sequential(\n",
" nn.Conv2d(1, 6, 3),\n",
" nn.ReLU(),\n",
" nn.BatchNorm2d(6),\n",
" nn.MaxPool2d(kernel_size=2, stride=2),\n",
" nn.Conv2d(6, 16, 3),\n",
" nn.ReLU(),\n",
" nn.BatchNorm2d(16),\n",
" nn.MaxPool2d(kernel_size=2, stride=2),\n",
" )\n",
" self.out = nn.Sequential(\n",
" nn.Flatten(),\n",
" nn.LazyLinear(128),\n",
" nn.ReLU(),\n",
" nn.Linear(128, 10),\n",
" nn.Softmax(dim=-1),\n",
" )\n",
"\n",
" def forward(self, X):\n",
" X = self.cnn(X)\n",
" X = self.out(X)\n",
" return X"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Ensure your classifier is scikit-learn compatible\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As some cleanlab features require scikit-learn compatibility, we adapt the above PyTorch neural net accordingly. [skorch](https://skorch.readthedocs.io) is a convenient package that helps with this. Alternatively, you can also easily wrap an arbitrary model to be scikit-learn compatible as demonstrated [here](https://github.com/cleanlab/cleanlab#use-cleanlab-with-any-model-for-most-ml-tasks)."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model_skorch = NeuralNetClassifier(ClassifierModule)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Compute out-of-sample predicted probabilities\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we'd like cleanlab to identify potential label errors in the whole dataset and not just the training set, we can consider using the entire dataset when computing the out-of-sample predicted probabilities, `pred_probs`, via cross-validation.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"num_crossval_folds = 3 # for efficiency; values like 5 or 10 will generally work better\n",
"pred_probs = cross_val_predict(\n",
" model_skorch,\n",
" X,\n",
" labels,\n",
" cv=num_crossval_folds,\n",
" method=\"predict_proba\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"An additional benefit of cross-validation is that it facilitates more reliable evaluation of our model than a single training/validation split."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"predicted_labels = pred_probs.argmax(axis=1)\n",
"acc = accuracy_score(labels, predicted_labels)\n",
"print(f\"Cross-validated estimate of accuracy on held-out data: {acc}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Use cleanlab to find label issues\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Based on the given labels and out-of-sample predicted probabilities, cleanlab can quickly help us identify label issues in our dataset. \n",
"\n",
"Here, we use cleanlab's `Datalab` to find potential label errors in our data. `Datalab` has several ways of loading the data. In this case, we’ll simply wrap the training features and noisy labels in a dictionary. We can instantiate our `Datalab` object with the dictionary created, and then pass in the model predicted probabilities we obtained above, and specify that we want to look for label errors by specifying that using the `issue_types` argument."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from cleanlab import Datalab\n",
"\n",
"data = {\"X\": X, \"y\": labels}\n",
"\n",
"lab = Datalab(data, label_name=\"y\")\n",
"lab.find_issues(pred_probs=pred_probs, issue_types={\"label\":{}})"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"After the audit is complete, we can view the results and information regarding the labels using Datalab's `get_issues` method."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"issue_results = lab.get_issues(\"label\")\n",
"issue_results.head()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The dataframe above contains a label quality score for each example. These numeric scores lie between 0 and 1, where lower scores indicate examples more likely to be mislabeled. It contains a boolean column specifying whether or not each example is identified to have a label issue (indicating it is likely mislabeled).\n",
"\n",
"We can sort the results obtained by label score to find the indices of the 15 most likely mislabeled examples in our dataset."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ranked_label_issues = issue_results.sort_values(\"label_score\").index\n",
"\n",
"print(f\"Top 15 most likely label errors: \\n {ranked_label_issues.values[:15]}\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"`ranked_label_issues` is a list of indices ranked by the label score of each example, the top indices in the list corresponding to examples that are worth inspecting more closely. To help visualize specific examples, we define a `plot_examples` function (can skip these details)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the implementation of `plot_examples` **(click to expand)**</summary>\n",
"\n",
"```python\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_examples(id_iter, nrows=1, ncols=1):\n",
" for count, id in enumerate(id_iter):\n",
" plt.subplot(nrows, ncols, count + 1)\n",
" plt.imshow(X[id].reshape(28, 28), cmap=\"gray\")\n",
" plt.title(f\"id: {id} \\n label: {y[id]}\")\n",
" plt.axis(\"off\")\n",
"\n",
" plt.tight_layout(h_pad=2.0)\n",
"```\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_examples(id_iter, nrows=1, ncols=1):\n",
" for count, id in enumerate(id_iter):\n",
" plt.subplot(nrows, ncols, count + 1)\n",
" plt.imshow(X[id].reshape(28, 28), cmap=\"gray\")\n",
" plt.title(f\"id: {id} \\n label: {labels[id]}\")\n",
" plt.axis(\"off\")\n",
"\n",
" plt.tight_layout(h_pad=2.0)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's look at the top 15 examples cleanlab thinks are most likely to be incorrectly labeled. We can see a few label errors and odd edge cases. Feel free to change the values below to display more/fewer examples."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_examples(ranked_label_issues[range(15)], 3, 5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Let's zoom into some specific examples from the above set:"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given label is **4** but looks more like a **7**:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_examples([59915])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given label is **4** but also looks like **9**:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_examples([24798])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"A very odd looking **5**:\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_examples([59701])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Given label is **3** but could be a **7**:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_examples([50340])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"cleanlab has shortlisted the most likely label errors to speed up your data cleaning process. With this list, you can decide whether to fix label issues or prune some of these examples from the dataset. \n",
"\n",
"You can see that even widely-used datasets like MNIST contain problematic labels. Never blindly trust your data! You should always check it for potential issues, many of which can be easily identified by cleanlab.\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"nbsphinx": "hidden"
},
"outputs": [],
"source": [
"# Note: This cell is only for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"highlighted_indices = [59915, 24798, 59701, 50340] # verify these examples were found by cleanlab\n",
"if not all(x in ranked_label_issues for x in highlighted_indices):\n",
" raise Exception(\"Some highlighted examples are missing from ranked_label_issues.\")"
]
}
],
"metadata": {
"interpreter": {
"hash": "ced20e3e49bb4fa4ce8ad38f8f2535b7fc4c39b2b89554502b5dbdad1ad67eda"
},
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.8"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
在新工单中引用 查看 Git Blame 复制永久链接