.. _issue_manager_creating_your_own: Creating Your Own Issues Manager ================================ This guide walks through the process of creating your own :py:class:`IssueManager ` to detect a custom-defined type of issue alongside the pre-defined issue types in :py:class:`Datalab `. .. seealso:: - :py:meth:`register `: You can either use this function at runtime to register a new issue manager: .. code-block:: python from cleanlab.datalab.internal.issue_manager_factory import register register(MyIssueManager) # Defaults to task="classification" # register(MyIssueManagerForRegression, task="regression") # Alternative for regression tasks or add as a decorator to the class definition (currently only works for classification tasks): .. code-block:: python @register class MyIssueManager(IssueManager): ... Prerequisites ------------- As a starting point for this guide, we'll import the necessary things for the next section and create a dummy dataset. .. note:: .. include:: ../optional_dependencies.rst .. code-block:: python import numpy as np import pandas as pd from cleanlab import IssueManager # Create a dummy dataset N = 20 data = pd.DataFrame( { "text": [f"example {i}" for i in range(N)], "label": np.random.randint(0, 2, N), }, ) Implementing IssueManagers -------------------------- .. _basic_issue_manager: Basic Issue Check ~~~~~~~~~~~~~~~~~ To create a basic issue manager, inherit from the :py:class:`IssueManager ` class, assign a name to the class as the class-variable, `issue_name`, and implement the :py:meth:`find_issues ` method. The :py:meth:`find_issues ` method should mark each example in the dataset as an issue or not with a boolean array. It should also provide a score for each example in the dataset that quantifies the quality of the example with regards to the issue. .. code-block:: python class Basic(IssueManager): # Assign a name to the issue issue_name = "basic" def find_issues(self, **kwargs) -> None: # Compute scores for each example scores = np.random.rand(len(self.datalab.data)) # Construct a dataframe where examples are marked for issues # and the score for each example is included. self.issues = pd.DataFrame( { f"is_{self.issue_name}_issue" : scores < 0.1, self.issue_score_key : scores, }, ) # Score the dataset as a whole based on this issue type self.summary = self.make_summary(score = scores.mean()) .. _intermediate_issue_manager: Intermediate Issue Check ~~~~~~~~~~~~~~~~~~~~~~~~ To create an intermediate issue: - Perform the same steps as in the :ref:`basic issue check ` section. - Populate the `info` attribute with a dictionary of information about the identified issues. The information can be included in a report generated by :py:class:`Datalab `, if you add any of the keys to the `verbosity_levels` class-attribute. Optionally, you can also add a description of the type of issue this issue manager handles to the `description` class-attribute. .. code-block:: python class Intermediate(IssueManager): issue_name = "intermediate" # Add a dictionary of information to include in the report verbosity_levels = { 0: [], 1: ["std"], 2: ["raw_scores"], } # Add a description of the issue description = "Intermediate issues are a bit more involved than basic issues." def find_issues(self, *, intermediate_arg: int, **kwargs) -> None: N = len(self.datalab.data) raw_scores = np.random.rand(N) std = raw_scores.std() threshold = min(0, raw_scores.mean() - std) sin_filter = np.sin(intermediate_arg * np.arange(N) / N) kernel = sin_filter ** 2 scores = kernel * raw_scores self.issues = pd.DataFrame( { f"is_{self.issue_name}_issue" : scores < threshold, self.issue_score_key : scores, }, ) self.summary = self.make_summary(score = scores.mean()) # Useful information that will be available in the Datalab instance self.info = { "std": std, "raw_scores": raw_scores, "kernel": kernel, } Advanced Issue Check ~~~~~~~~~~~~~~~~~~~~ .. note:: WIP: This section is a work in progress. Use with Datalab ---------------- We can create a :py:class:`Datalab ` instance and run issue checks with the custom issue managers we created like so: .. code-block:: python from cleanlab.datalab.internal.issue_manager_factory import register from cleanlab import Datalab # Register the issue manager for issue_manager in [Basic, Intermediate]: register(issue_manager) # Instantiate a datalab instance datalab = Datalab(data, label_name="label") # Run the issue check issue_types = {"basic": {}, "intermediate": {"intermediate_arg": 2}} datalab.find_issues(issue_types=issue_types) # Print report datalab.report(verbosity=0) The report will look something like this: .. code-block:: text Here is a summary of the different kinds of issues found in the data: issue_type score num_issues basic 0.477762 2 intermediate 0.286455 0 (Note: A lower score indicates a more severe issue across all examples in the dataset.) ------------------------------------------- basic issues ------------------------------------------- Number of examples with this issue: 2 Overall dataset quality in terms of this issue: 0.4778 Examples representing most severe instances of this issue: is_basic_issue basic_score 13 True 0.003042 8 True 0.058117 11 False 0.121908 15 False 0.169312 17 False 0.229044 --------------------------------------- intermediate issues ---------------------------------------- About this issue: Intermediate issues are a bit more involved than basic issues. Number of examples with this issue: 0 Overall dataset quality in terms of this issue: 0.2865 Examples representing most severe instances of this issue: is_intermediate_issue intermediate_score kernel 0 False 0.000000 0.0 1 False 0.007059 0.009967 3 False 0.010995 0.087332 2 False 0.016296 0.03947 11 False 0.019459 0.794251