mirrors / cleanlab--cleanlab

项目文件夹

cleanlab--cleanlab
文件
macos-12-ci
T
ChG a0f24add61 Use existing Datalab to audit additional new data (#1049) 
Co-authored-by: Elías Snorrason <eliassno@gmail.com>
2024-04-06 02:51:59 +00:00

683 行
26 KiB
Plaintext
原始文件 永久链接 Blame 文件历史

此文件含有不可见的 Unicode 字符
此文件含有人类无法区分的不可见的 Unicode 字符，但可以由计算机进行不同的处理。 如果您是想特意这样的，可以安全地忽略该警告。 使用 Escape 按钮显示他们。
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# DataMonitor: Leverage statistics from Datalab to audit new data\n",
"\n",
"Once you've fitted your `Datalab` instance on some training data, it stores some statistics about the training data that may prove useful to monitor new data.\n",
"This notebook shows the process of applying Datalab to find issues in training data and then using the same statistics to monitor new data.\n",
"\n",
"This involves a new class called `DataMonitor` that takes a Datalab instance as input to, then run similar issue checks on new data in a more efficient way, especially for\n",
"smaller batches of data.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\">\n",
"Quickstart\n",
"<br/>\n",
"\n",
"Already ran `Datalab` on a dataset? Already have (out-of-sample) `pred_probs` from a model trained on an new set of labels? Run the code below to examine your dataset for label issues.\n",
"\n",
"<div class=markdown markdown=\"1\" style=\"background:white;margin:16px\"> \n",
" \n",
"```ipython3 \n",
"from cleanlab.experimental.datalab.data_monitor import DataMonitor\n",
"\n",
"monitor = DataMonitor(datalab=your_datalab)\n",
"\n",
"for batch in new_data_batches:\n",
" # Process data to get labels and predicted probabilities\n",
" your_labels = get_your_labels(batch)\n",
" your_pred_probs = get_pred_probs(batch)\n",
" \n",
" # Find issues in the batch\n",
" monitor.find_issues(labels=your_labels, pred_probs=your_pred_probs)\n",
"```\n",
" \n",
"</div>\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 1. Install and import required dependencies\n",
"\n",
"You can use pip to install all packages required for this tutorial as follows:\n",
"\n",
"```ipython3\n",
"!pip install matplotlib\n",
"!pip install \"cleanlab[datalab]\"\n",
"\n",
"# Make sure to install the version corresponding to this tutorial\n",
"# E.g. if viewing master branch documentation:\n",
"# !pip install git+https://github.com/cleanlab/cleanlab.git\n",
"```"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Package installation (hidden on docs website).\n",
"dependencies = [\"cleanlab\", \"matplotlib\", \"datasets\"] # TODO: make sure this list is updated\n",
"\n",
"if \"google.colab\" in str(get_ipython()): # Check if it's running in Google Colab\n",
" %pip install cleanlab # for colab\n",
" cmd = ' '.join([dep for dep in dependencies if dep != \"cleanlab\"])\n",
" %pip install $cmd\n",
"else:\n",
" dependencies_test = [dependency.split('>')[0] if '>' in dependency \n",
" else dependency.split('<')[0] if '<' in dependency \n",
" else dependency.split('=')[0] for dependency in dependencies]\n",
" missing_dependencies = []\n",
" for dependency in dependencies_test:\n",
" try:\n",
" __import__(dependency)\n",
" except ImportError:\n",
" missing_dependencies.append(dependency)\n",
"\n",
" if len(missing_dependencies) > 0:\n",
" print(\"Missing required dependencies:\")\n",
" print(*missing_dependencies, sep=\", \")\n",
" print(\"\\nPlease install them before running the rest of this notebook.\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import numpy as np\n",
"from sklearn.linear_model import LogisticRegression\n",
"from sklearn.model_selection import cross_val_predict\n",
"\n",
"from cleanlab import Datalab\n",
"from cleanlab.experimental.datalab.data_monitor import DataMonitor"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2. Create and load the data (can skip these details)\n",
"\n",
"For this tutorial, we'll re-use the toy classification dataset from the `Datalab` quickstart tutorial. The dataset has two numerical features and a label column with three possible classes. Each example is classified as either: *low*, *mid* or *high*.\n",
"\n",
"Here we show a workflow for finding label issues on data unseen by `Datalab` using the `DataMonitor` class."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code for data generation. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(800, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.1, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate\n",
"```\n",
"\n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from sklearn.model_selection import train_test_split\n",
"from cleanlab.benchmarking.noise_generation import (\n",
" generate_noise_matrix_from_trace,\n",
" generate_noisy_labels,\n",
")\n",
"\n",
"SEED = 123\n",
"np.random.seed(SEED)\n",
"\n",
"BINS = {\n",
" \"low\": [-np.inf, 3.3],\n",
" \"mid\": [3.3, 6.6],\n",
" \"high\": [6.6, +np.inf],\n",
"}\n",
"\n",
"BINS_MAP = {\n",
" \"low\": 0,\n",
" \"mid\": 1,\n",
" \"high\": 2,\n",
"}\n",
"\n",
"\n",
"def create_data():\n",
"\n",
" X = np.random.rand(800, 2) * 5\n",
" y = np.sum(X, axis=1)\n",
" # Map y to bins based on the BINS dict\n",
" y_bin = np.array([k for y_i in y for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_bin_idx = np.array([BINS_MAP[k] for k in y_bin])\n",
"\n",
" # Split into train and test\n",
" X_train, X_test, y_train, y_test, y_train_idx, y_test_idx = train_test_split(\n",
" X, y_bin, y_bin_idx, test_size=0.1, random_state=SEED\n",
" )\n",
"\n",
" # Add several (5) out-of-distribution points. Sliding them along the decision boundaries\n",
" # to make them look like they are out-of-frame\n",
" X_out = np.array(\n",
" [\n",
" [-1.5, 3.0],\n",
" [-1.75, 6.5],\n",
" [1.5, 7.2],\n",
" [2.5, -2.0],\n",
" [5.5, 7.0],\n",
" ]\n",
" )\n",
" # Add a near duplicate point to the last outlier, with some tiny noise added\n",
" near_duplicate = X_out[-1:] + np.random.rand(1, 2) * 1e-6\n",
" X_out = np.concatenate([X_out, near_duplicate])\n",
"\n",
" y_out = np.sum(X_out, axis=1)\n",
" y_out_bin = np.array([k for y_i in y_out for k, v in BINS.items() if v[0] <= y_i < v[1]])\n",
" y_out_bin_idx = np.array([BINS_MAP[k] for k in y_out_bin])\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_out])\n",
" y_train = np.concatenate([y_train, y_out])\n",
" y_train_idx = np.concatenate([y_train_idx, y_out_bin_idx])\n",
"\n",
" # Add an exact duplicate example to the training set\n",
" exact_duplicate_idx = np.random.randint(0, len(X_train))\n",
" X_duplicate = X_train[exact_duplicate_idx, None]\n",
" y_duplicate = y_train[exact_duplicate_idx, None]\n",
" y_duplicate_idx = y_train_idx[exact_duplicate_idx, None]\n",
"\n",
" # Add to train\n",
" X_train = np.concatenate([X_train, X_duplicate])\n",
" y_train = np.concatenate([y_train, y_duplicate])\n",
" y_train_idx = np.concatenate([y_train_idx, y_duplicate_idx])\n",
"\n",
" py = np.bincount(y_train_idx) / float(len(y_train_idx))\n",
" m = len(BINS)\n",
"\n",
" noise_matrix = generate_noise_matrix_from_trace(\n",
" m,\n",
" trace=0.9 * m,\n",
" py=py,\n",
" valid_noise_matrix=True,\n",
" seed=SEED,\n",
" )\n",
"\n",
" noisy_labels_idx = generate_noisy_labels(y_train_idx, noise_matrix)\n",
" noisy_labels = np.array([list(BINS_MAP.keys())[i] for i in noisy_labels_idx])\n",
"\n",
" return X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"X_train, y_train_idx, noisy_labels, noisy_labels_idx, X_out, X_duplicate = create_data()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"train_X, test_X, train_y_true, test_y_true, train_y, test_y, train_y_idx, test_y_idx = train_test_split(X_train, y_train_idx, noisy_labels, noisy_labels_idx, test_size=400, random_state=SEED)\n",
"data = {\"X\": train_X, \"y\": train_y}\n",
"test_data = {\"X\": test_X, \"y\": test_y}"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We make a scatter plot of the features, with a color corresponding to the observed labels. Incorrect given labels are highlighted in red if they do not match the true label, outliers highlighted with an a black cross, and duplicates highlighted with a cyan cross."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<details><summary>See the code to visualize the data. **(click to expand)**</summary>\n",
" \n",
"```ipython3\n",
"# Note: This pulldown content is for docs.cleanlab.ai, if running on local Jupyter or Colab, please ignore it.\n",
"\n",
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(6, 4))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-2.5, 8.5)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
" \n",
" title_fontproperties = {\"weight\":\"semibold\", \"size\": 8}\n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.76, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" second_legend = ax.legend(handles=[label_err], loc=[0.76, 0.46], title=\"Type of Issue\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()\n",
"```\n",
" \n",
"</details>"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import matplotlib.pyplot as plt\n",
"\n",
"def plot_data(X_train, y_train_idx, noisy_labels_idx):\n",
" # Plot data with clean labels and noisy labels, use BINS_MAP for the legend\n",
" fig, ax = plt.subplots(figsize=(6, 4))\n",
" \n",
" low = ax.scatter(X_train[noisy_labels_idx == 0, 0], X_train[noisy_labels_idx == 0, 1], label=\"low\")\n",
" mid = ax.scatter(X_train[noisy_labels_idx == 1, 0], X_train[noisy_labels_idx == 1, 1], label=\"mid\")\n",
" high = ax.scatter(X_train[noisy_labels_idx == 2, 0], X_train[noisy_labels_idx == 2, 1], label=\"high\")\n",
" \n",
" ax.set_title(\"Noisy labels\")\n",
" ax.set_xlabel(r\"$x_1$\", fontsize=16)\n",
" ax.set_ylabel(r\"$x_2$\", fontsize=16)\n",
"\n",
" # Plot true boundaries (x+y=3.3, x+y=6.6)\n",
" ax.set_xlim(-2.5, 8.5)\n",
" ax.set_ylim(-3.5, 9.0)\n",
" ax.plot([-0.7, 4.0], [4.0, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
" ax.plot([-0.7, 7.3], [7.3, -0.7], color=\"k\", linestyle=\"--\", alpha=0.5)\n",
"\n",
" # Draw red circles around the points that are misclassified (i.e. the points that are in the wrong bin)\n",
" for i, (X, y) in enumerate(zip([X_train, X_train], [y_train_idx, noisy_labels_idx])):\n",
" for j, (k, v) in enumerate(BINS_MAP.items()):\n",
" label_err = ax.scatter(\n",
" X[(y == v) & (y != y_train_idx), 0],\n",
" X[(y == v) & (y != y_train_idx), 1],\n",
" s=180,\n",
" marker=\"o\",\n",
" facecolor=\"none\",\n",
" edgecolors=\"red\",\n",
" linewidths=2.5,\n",
" alpha=0.5,\n",
" label=\"Label error\",\n",
" )\n",
" \n",
" title_fontproperties = {\"weight\":\"semibold\", \"size\": 8}\n",
" first_legend = ax.legend(handles=[low, mid, high], loc=[0.76, 0.7], title=\"Given Class Label\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" second_legend = ax.legend(handles=[label_err], loc=[0.76, 0.46], title=\"Type of Issue\", alignment=\"left\", title_fontproperties=title_fontproperties, fontsize=8, markerscale=0.5)\n",
" \n",
" ax = plt.gca().add_artist(first_legend)\n",
" ax = plt.gca().add_artist(second_legend)\n",
" plt.tight_layout()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"plot_data(train_X, train_y_true, train_y_idx)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3. Get out-of-sample predicted probabilities from a classifier"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"To detect certain types of issues in classification data (e.g. label errors), `Datalab` and `DataMonitor` rely on predicted class probabilities from a trained model. Ideally, the prediction for each example should be out-of-sample (to avoid overfitting), coming from a copy of the model that was not trained on this example. \n",
"\n",
"\n",
"Similar to what is shown in the `Datalab` quickstart tutorial, this tutorial uses a simple logistic regression model \n",
"and the `cross_val_predict()` function from scikit-learn to generate out-of-sample predicted class probabilities for every example in the training set. You can replace this with *any* other classifier model and train it with cross-validation to get out-of-sample predictions.\n",
"Make sure that the columns of your `pred_probs` are properly ordered with respect to the ordering of classes, which for Datalab is: lexicographically sorted by class name."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"model = LogisticRegression()\n",
"pred_probs = cross_val_predict(\n",
" estimator=model, X=train_X, y=train_y, cv=5, method=\"predict_proba\",\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4. Use Datalab to find issues in the dataset\n",
"\n",
"These steps are pretty much identical to the `Datalab` quickstart tutorial. We'll use the `Datalab` class to find issues in the training data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"lab = Datalab(data=data, label_name=\"y\", task=\"classification\")\n",
"\n",
"# For simplicity, let's leverage the cross-validated predicted probabilities to find possible label issues\n",
"lab.find_issues(pred_probs=pred_probs, issue_types={\"label\": {}})\n",
"\n",
"lab.report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Great! The `Datalab` instance has seen some training data and found some issues. This would be a good time to look at any major issues that may be easily resolved. For example, if there are many label errors of a certain class, you may want to investigate why this is happening and fix the issue at the source.\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5. Use DataMonitor to find issues in new data"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"Now, how do you monitor new data for the same issues? You pass the `Datalab` instance to the `DataMonitor` class, which can then be used to monitor new data for the same issues."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Set up the data monitor\n",
"monitor = DataMonitor(datalab=lab)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For new data, you may be running model predictions on-the-fly and want to monitor the predictions for issues. \n",
"This requires a slightly different approach than the one used for training data, when feeding the data in batches to the DataMonitor.\n",
"\n",
"Here, we'll simulate a stream of data points annotated with some given labels and some model predictions. We'll then use the `DataMonitor` class to monitor the data stream for issues.\n",
"\n",
"Generally, you would have a model already trained on the full training data and would be running predictions on new data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"from tqdm.auto import tqdm\n",
"from time import sleep\n",
"\n",
"# Fit a classification model on the full training set\n",
"model = LogisticRegression()\n",
"model.fit(train_X, lab.labels)\n",
"\n",
"\n",
"# Here, we simulate a streaming scenario by processing some of test data, 1 sample at a time\n",
"batch_size = 1\n",
"def generate_stream(data: dict, batch_size=1, sleep_time=0.1):\n",
" n = len(next(iter(data.values())))\n",
" for i in tqdm(range(0, n, batch_size), total=n // batch_size, desc=f\"Streaming data, {batch_size} sample(s) at a time\"):\n",
" batch = {k: v[i:i + batch_size] for k, v in data.items()}\n",
" \n",
" # Simulate some processing time\n",
" sleep(sleep_time)\n",
" \n",
" yield {\"labels\": batch[\"y\"], \"pred_probs\": model.predict_proba(batch[\"X\"])}\n",
"\n",
"singleton_stream = generate_stream({\"X\": test_X[:50], \"y\": test_y[:50]})\n",
"# TODO: Add seamless Singleton Support designed to intuitively\n",
"# handle single data points without requiring the user to wrap singletons in additional data structures\n",
"\n",
"batched_stream = generate_stream({\"X\": test_X[50:], \"y\": test_y[50:]}, batch_size=50, sleep_time=0.75)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Process a stream of data to provide the necessary arguments for the find_issues method\n",
"for processed_singleton in singleton_stream:\n",
" monitor.find_issues(**processed_singleton)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The same principle works for larger batches of data, but the main idea is to not exceed the memory limits of the system by loading the entire dataset at once."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"for processed_batch in batched_stream:\n",
" monitor.find_issues(**processed_batch)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `DataMonitor` keeps track of the issue masks and issue scores for each data point that is streamed through it. During the call to `DataMonitor.find_issues`, any time an issue is found, it prints out the troublesome data points in the batch, along with the issue type and score."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6. Learn more about the issues in the additional data\n",
"\n",
"TODO\n",
"\n",
"The data monitor has several properties that allow you to inspect the results of\n",
"the full monitoring process."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# View the full issues dataframe (analogous to the Datalab.issues DataFrame)\n",
"display(monitor.issues)\n",
"\n",
"# Look at particular issue types\n",
"# TODO\n",
"# monitor.get_issues(\"label\")\n",
"monitor.issues.sort_values(\"label_score\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Look at a summary of all the issue checks across the full monitoring process\n",
"monitor.issue_summary\n",
"\n",
"# TODO: Align the behavior of the DataMonitor.issue_summary with the Datalab.get_issue_summary method "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.7"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
在新工单中引用 查看 Git Blame 复制永久链接