{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "title-cell",
   "metadata": {},
   "source": [
    "# Open-Set Recognition with MNIST: Reducing Network Agnostophobia\n",
    "\n",
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ludwig-ai/ludwig/blob/main/examples/open_set_recognition/open_set_mnist.ipynb)\n",
    "\n",
    "Standard image classifiers assign high-confidence predictions to *every* input — including images\n",
    "from classes never seen during training. This is called **network agnostophobia**: the model is\n",
    "incapable of saying \"I don't know.\"\n",
    "\n",
    "This notebook demonstrates three Ludwig models trained on MNIST digits **0–7** (known classes)\n",
    "with digits **8–9** acting as the unknown/background:\n",
    "\n",
    "| Model | Loss | Expected behaviour |\n",
    "|-------|------|--------------------|\n",
    "| CE Baseline | `softmax_cross_entropy` | High confidence even on unknown digits |\n",
    "| Entropic Open-Set | `entropic_open_set` | Pushes unknown confidence toward uniform |\n",
    "| Objectosphere | `objectosphere` | Creates a clear logit-norm gap between known and unknown |\n",
    "\n",
    "**Paper:** Dhamija, Günther, Boult — *Reducing Network Agnostophobia*, NeurIPS 2018.\n",
    "https://arxiv.org/abs/1811.04110"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "install-cell",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install ludwig torchvision --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "setup-header",
   "metadata": {},
   "source": [
    "## Setup and data preparation\n",
    "\n",
    "We download MNIST via `torchvision`, save each digit image as a PNG file, and build two CSVs:\n",
    "\n",
    "- **`train.csv`** — digits 0–7 with their true labels, plus digits 8–9 labelled as `\"background\"`\n",
    "- **`test.csv`** — digits 0–7 (known) and digits 8–9 (unknown) with their true labels\n",
    "\n",
    "The training set teaches the open-set models what \"background\" looks like. The test set lets us\n",
    "measure how confidently each model handles the unseen classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "data-prep",
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "from pathlib import Path\n",
    "\n",
    "import torch\n",
    "from PIL import Image\n",
    "from torchvision import datasets, transforms\n",
    "\n",
    "# ── configuration ──────────────────────────────────────────────────────────────\n",
    "DATA_DIR = Path(\"mnist_data\")  # raw MNIST download\n",
    "IMG_DIR = Path(\"mnist_images\")  # saved PNG files\n",
    "KNOWN_CLASSES = list(range(8))  # digits 0-7\n",
    "UNKNOWN_CLASSES = [8, 9]  # background / unknown\n",
    "\n",
    "# Limit samples per class so training is fast on CPU\n",
    "MAX_TRAIN_KNOWN = 500  # per known class\n",
    "MAX_TRAIN_UNKNOWN = 500  # per unknown class (background)\n",
    "MAX_TEST_KNOWN = 200  # per known class\n",
    "MAX_TEST_UNKNOWN = 200  # per unknown class\n",
    "\n",
    "IMG_DIR.mkdir(parents=True, exist_ok=True)\n",
    "\n",
    "# ── download ───────────────────────────────────────────────────────────────────\n",
    "mnist_train = datasets.MNIST(str(DATA_DIR), train=True, download=True, transform=transforms.ToTensor())\n",
    "mnist_test = datasets.MNIST(str(DATA_DIR), train=False, download=True, transform=transforms.ToTensor())\n",
    "\n",
    "print(f\"Downloaded: {len(mnist_train)} train / {len(mnist_test)} test samples\")\n",
    "\n",
    "\n",
    "# ── helper: save image and return path ────────────────────────────────────────\n",
    "def save_image(tensor: torch.Tensor, split: str, digit: int, idx: int) -> str:\n",
    "    \"\"\"Save a (1, H, W) float tensor as a grayscale PNG; return the file path.\"\"\"\n",
    "    folder = IMG_DIR / split / str(digit)\n",
    "    folder.mkdir(parents=True, exist_ok=True)\n",
    "    fpath = folder / f\"{idx:05d}.png\"\n",
    "    img = Image.fromarray((tensor.squeeze(0).numpy() * 255).astype(\"uint8\"), mode=\"L\")\n",
    "    img.save(fpath)\n",
    "    return str(fpath)\n",
    "\n",
    "\n",
    "# ── build train.csv ───────────────────────────────────────────────────────────\n",
    "# class counter for capping per-class samples\n",
    "from collections import defaultdict\n",
    "\n",
    "\n",
    "def build_csv(\n",
    "    dataset,\n",
    "    csv_path: str,\n",
    "    split: str,\n",
    "    known_classes,\n",
    "    unknown_classes,\n",
    "    max_known: int,\n",
    "    max_unknown: int,\n",
    "    label_unknown_as_background: bool,\n",
    "):\n",
    "    \"\"\"\n",
    "    Walk *dataset*, save PNGs, write *csv_path*.\n",
    "\n",
    "    label_unknown_as_background=True  → training split (unknown → \"background\")\n",
    "    label_unknown_as_background=False → test split (unknown keeps true digit string)\n",
    "    \"\"\"\n",
    "    counts_known = defaultdict(int)\n",
    "    counts_unknown = defaultdict(int)\n",
    "    rows = []\n",
    "\n",
    "    for global_idx, (img_tensor, digit) in enumerate(dataset):\n",
    "        digit = int(digit)\n",
    "        if digit in known_classes:\n",
    "            if counts_known[digit] >= max_known:\n",
    "                continue\n",
    "            path = save_image(img_tensor, split, digit, global_idx)\n",
    "            label = str(digit)\n",
    "            counts_known[digit] += 1\n",
    "        elif digit in unknown_classes:\n",
    "            if counts_unknown[digit] >= max_unknown:\n",
    "                continue\n",
    "            path = save_image(img_tensor, split, digit, global_idx)\n",
    "            label = \"background\" if label_unknown_as_background else str(digit)\n",
    "            counts_unknown[digit] += 1\n",
    "        else:\n",
    "            continue\n",
    "        rows.append({\"image_path\": path, \"label\": label})\n",
    "\n",
    "        # stop early once all caps are met\n",
    "        if all(counts_known[c] >= max_known for c in known_classes) and all(\n",
    "            counts_unknown[c] >= max_unknown for c in unknown_classes\n",
    "        ):\n",
    "            break\n",
    "\n",
    "    with open(csv_path, \"w\", newline=\"\") as f:\n",
    "        writer = csv.DictWriter(f, fieldnames=[\"image_path\", \"label\"])\n",
    "        writer.writeheader()\n",
    "        writer.writerows(rows)\n",
    "\n",
    "    known_total = sum(counts_known.values())\n",
    "    unknown_total = sum(counts_unknown.values())\n",
    "    print(f\"  {csv_path}: {known_total} known, {unknown_total} unknown/background\")\n",
    "    return rows\n",
    "\n",
    "\n",
    "print(\"Building train.csv ...\")\n",
    "train_rows = build_csv(\n",
    "    mnist_train,\n",
    "    \"train.csv\",\n",
    "    \"train\",\n",
    "    KNOWN_CLASSES,\n",
    "    UNKNOWN_CLASSES,\n",
    "    MAX_TRAIN_KNOWN,\n",
    "    MAX_TRAIN_UNKNOWN,\n",
    "    label_unknown_as_background=True,\n",
    ")\n",
    "\n",
    "print(\"Building test.csv ...\")\n",
    "test_rows = build_csv(\n",
    "    mnist_test,\n",
    "    \"test.csv\",\n",
    "    \"test\",\n",
    "    KNOWN_CLASSES,\n",
    "    UNKNOWN_CLASSES,\n",
    "    MAX_TEST_KNOWN,\n",
    "    MAX_TEST_UNKNOWN,\n",
    "    label_unknown_as_background=False,\n",
    ")\n",
    "\n",
    "print(\"Done.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "inspect-csv-header",
   "metadata": {},
   "source": [
    "Let's quickly inspect what our CSVs look like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "inspect-csv",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "train_df = pd.read_csv(\"train.csv\")\n",
    "test_df = pd.read_csv(\"test.csv\")\n",
    "\n",
    "print(\"train.csv label distribution:\")\n",
    "print(train_df[\"label\"].value_counts().sort_index())\n",
    "print()\n",
    "print(\"test.csv label distribution:\")\n",
    "print(test_df[\"label\"].value_counts().sort_index())\n",
    "print()\n",
    "print(train_df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "background-discovery-header",
   "metadata": {},
   "source": [
    "## Discover background class index\n",
    "\n",
    "Ludwig assigns integer indices to category labels based on frequency in the training data\n",
    "(most frequent first, with index 0 reserved for `<UNK>`).  Before using `entropic_open_set`\n",
    "or `objectosphere` loss we need to know the integer index that Ludwig assigns to the\n",
    "`\"background\"` label.\n",
    "\n",
    "The safest approach:\n",
    "1. Run a short training job with the baseline config (standard cross-entropy).\n",
    "2. Open `training_set_metadata.json` in the saved model directory.\n",
    "3. Look up `output_features -> label -> str2idx -> \"background\"`.\n",
    "\n",
    "We do exactly this below — first training the baseline, then extracting the index programmatically."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "baseline-header",
   "metadata": {},
   "source": [
    "## Baseline: softmax cross-entropy\n",
    "\n",
    "The baseline model is a standard image classifier trained with softmax cross-entropy.  It is\n",
    "trained **only on known classes** (digits 0–7); digits 8–9 are excluded from its training data.\n",
    "\n",
    "At test time the baseline will happily assign high confidence to unknown digits because it has\n",
    "no mechanism to express uncertainty."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baseline-config",
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "ENCODER = {\n",
    "    \"type\": \"stacked_cnn\",\n",
    "    \"conv_layers\": [\n",
    "        {\"num_filters\": 32, \"filter_size\": 3, \"pool_size\": 2, \"pool_stride\": 2},\n",
    "        {\"num_filters\": 64, \"filter_size\": 3, \"pool_size\": 2, \"pool_stride\": 2},\n",
    "    ],\n",
    "    \"fc_layers\": [{\"output_size\": 128, \"dropout\": 0.3}],\n",
    "}\n",
    "\n",
    "config_baseline = {\n",
    "    \"model_type\": \"ecd\",\n",
    "    \"input_features\": [{\"name\": \"image_path\", \"type\": \"image\", \"encoder\": ENCODER}],\n",
    "    \"output_features\": [{\"name\": \"label\", \"type\": \"category\", \"loss\": {\"type\": \"softmax_cross_entropy\"}}],\n",
    "    \"trainer\": {\"epochs\": 10, \"learning_rate\": 0.001, \"batch_size\": 128},\n",
    "}\n",
    "\n",
    "print(yaml.dump(config_baseline, default_flow_style=False))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "baseline-train",
   "metadata": {},
   "outputs": [],
   "source": [
    "from ludwig.api import LudwigModel\n",
    "\n",
    "# The baseline is trained only on known classes — filter out the background rows\n",
    "train_known_df = train_df[train_df[\"label\"] != \"background\"].copy()\n",
    "print(f\"Training baseline on {len(train_known_df)} samples (known classes only)\")\n",
    "\n",
    "model_baseline = LudwigModel(config=config_baseline, logging_level=\"WARNING\")\n",
    "_, _, output_dir_baseline = model_baseline.train(\n",
    "    dataset=train_known_df,\n",
    "    experiment_name=\"open_set_mnist\",\n",
    "    model_name=\"baseline\",\n",
    "    skip_save_processed_input=True,\n",
    ")\n",
    "print(f\"Baseline model saved to: {output_dir_baseline}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "background-index-discovery",
   "metadata": {},
   "source": [
    "### Find the background class index from `training_set_metadata.json`\n",
    "\n",
    "Now that we have a trained model we can inspect its vocabulary.  The agnostophobia models will\n",
    "be trained on the *full* training set (known + background), so the background label will appear\n",
    "in the vocabulary.  We run a quick vocabulary fit to discover its index before training those\n",
    "models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "find-bg-index",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "# The agnostophobia models are trained on the full dataset (including background).\n",
    "# We need to know what index Ludwig will assign to \"background\" in *that* vocabulary.\n",
    "# The simplest way is to train the entropic model first (or do a preprocessing run),\n",
    "# but we can also use Ludwig's preprocessing API directly.\n",
    "from ludwig.api import LudwigModel\n",
    "\n",
    "# Build a minimal config for preprocessing only\n",
    "config_for_vocab = {\n",
    "    \"model_type\": \"ecd\",\n",
    "    \"input_features\": [{\"name\": \"image_path\", \"type\": \"image\", \"encoder\": ENCODER}],\n",
    "    \"output_features\": [{\"name\": \"label\", \"type\": \"category\", \"loss\": {\"type\": \"softmax_cross_entropy\"}}],\n",
    "    \"trainer\": {\"epochs\": 1, \"batch_size\": 128},\n",
    "}\n",
    "\n",
    "vocab_model = LudwigModel(config=config_for_vocab, logging_level=\"WARNING\")\n",
    "_, _, output_dir_vocab = vocab_model.train(\n",
    "    dataset=train_df,  # full training set including background\n",
    "    experiment_name=\"open_set_mnist\",\n",
    "    model_name=\"vocab_probe\",\n",
    "    skip_save_processed_input=True,\n",
    ")\n",
    "\n",
    "metadata_path = Path(output_dir_vocab) / \"model\" / \"training_set_metadata.json\"\n",
    "with open(metadata_path) as f:\n",
    "    metadata = json.load(f)\n",
    "\n",
    "str2idx = metadata[\"label\"][\"str2idx\"]\n",
    "BACKGROUND_CLASS = str2idx[\"background\"]\n",
    "\n",
    "print(\"label str2idx:\")\n",
    "for k, v in sorted(str2idx.items(), key=lambda x: x[1]):\n",
    "    marker = \"  <-- background\" if k == \"background\" else \"\"\n",
    "    print(f\"  {v:3d} : {k!r}{marker}\")\n",
    "\n",
    "print(f\"\\nBACKGROUND_CLASS = {BACKGROUND_CLASS}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "entropic-header",
   "metadata": {},
   "source": [
    "## Entropic Open-Set loss\n",
    "\n",
    "The Entropic Open-Set model is trained on the full dataset — known digits plus digits 8–9\n",
    "labelled as `\"background\"`.  For known samples the loss is standard cross-entropy.  For\n",
    "background samples the loss *maximises* Shannon entropy, pushing the softmax output toward\n",
    "the uniform distribution.\n",
    "\n",
    "$$\n",
    "\\mathcal{L} =\n",
    "  \\underbrace{-\\log p_y}_{\\text{CE on known}}\n",
    "  \\;+\\;\n",
    "  \\underbrace{\\sum_k p_k \\log p_k}_{\\text{neg-entropy on background}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "entropic-train",
   "metadata": {},
   "outputs": [],
   "source": [
    "config_entropic = {\n",
    "    \"model_type\": \"ecd\",\n",
    "    \"input_features\": [{\"name\": \"image_path\", \"type\": \"image\", \"encoder\": ENCODER}],\n",
    "    \"output_features\": [\n",
    "        {\n",
    "            \"name\": \"label\",\n",
    "            \"type\": \"category\",\n",
    "            \"loss\": {\n",
    "                \"type\": \"entropic_open_set\",\n",
    "                \"background_class\": BACKGROUND_CLASS,\n",
    "            },\n",
    "        }\n",
    "    ],\n",
    "    \"trainer\": {\"epochs\": 10, \"learning_rate\": 0.001, \"batch_size\": 128},\n",
    "}\n",
    "\n",
    "print(f\"background_class = {BACKGROUND_CLASS}\")\n",
    "print(yaml.dump(config_entropic, default_flow_style=False))\n",
    "\n",
    "model_entropic = LudwigModel(config=config_entropic, logging_level=\"WARNING\")\n",
    "_, _, output_dir_entropic = model_entropic.train(\n",
    "    dataset=train_df,\n",
    "    experiment_name=\"open_set_mnist\",\n",
    "    model_name=\"entropic\",\n",
    "    skip_save_processed_input=True,\n",
    ")\n",
    "print(f\"Entropic model saved to: {output_dir_entropic}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "objectosphere-header",
   "metadata": {},
   "source": [
    "## Objectosphere loss\n",
    "\n",
    "The Objectosphere loss extends the Entropic Open-Set loss with a logit-norm objective:\n",
    "\n",
    "- **Known samples**: CE + hinge `max(0, ξ – ||z||)²` pushes logit L2 norms above ξ\n",
    "- **Background samples**: entropy maximisation + `ζ||z||²` suppresses logit norms toward zero\n",
    "\n",
    "This creates a clear norm gap between known and unknown samples, enabling a simple threshold\n",
    "detector at inference time.\n",
    "\n",
    "$$\n",
    "\\mathcal{L} =\n",
    "  \\underbrace{\\text{CE}(z_{\\text{known}}) + \\max(0,\\,\\xi - \\|z\\|)^2}_{\\text{known}}\n",
    "  \\;+\\;\n",
    "  \\underbrace{\\sum_k p_k \\log p_k + \\zeta\\,\\|z\\|^2}_{\\text{background}}\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "objectosphere-train",
   "metadata": {},
   "outputs": [],
   "source": [
    "config_objectosphere = {\n",
    "    \"model_type\": \"ecd\",\n",
    "    \"input_features\": [{\"name\": \"image_path\", \"type\": \"image\", \"encoder\": ENCODER}],\n",
    "    \"output_features\": [\n",
    "        {\n",
    "            \"name\": \"label\",\n",
    "            \"type\": \"category\",\n",
    "            \"loss\": {\n",
    "                \"type\": \"objectosphere\",\n",
    "                \"background_class\": BACKGROUND_CLASS,\n",
    "                \"xi\": 10.0,\n",
    "                \"zeta\": 0.1,\n",
    "            },\n",
    "        }\n",
    "    ],\n",
    "    \"trainer\": {\"epochs\": 10, \"learning_rate\": 0.001, \"batch_size\": 128},\n",
    "}\n",
    "\n",
    "model_objectosphere = LudwigModel(config=config_objectosphere, logging_level=\"WARNING\")\n",
    "_, _, output_dir_objectosphere = model_objectosphere.train(\n",
    "    dataset=train_df,\n",
    "    experiment_name=\"open_set_mnist\",\n",
    "    model_name=\"objectosphere\",\n",
    "    skip_save_processed_input=True,\n",
    ")\n",
    "print(f\"Objectosphere model saved to: {output_dir_objectosphere}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compare-header",
   "metadata": {},
   "source": [
    "## Compare: confidence on known vs unknown\n",
    "\n",
    "We now predict on the test set with each model and collect the **maximum softmax probability**\n",
    "for each sample.  A well-calibrated open-set model should:\n",
    "\n",
    "- Produce **high** max-prob on known digits (it is confident)\n",
    "- Produce **low** max-prob on unknown digits (8 and 9) — ideally near `1/num_classes`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "run-predictions",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Separate test rows into known and unknown\n",
    "test_known_df = test_df[~test_df[\"label\"].isin([\"8\", \"9\"])].copy()\n",
    "test_unknown_df = test_df[test_df[\"label\"].isin([\"8\", \"9\"])].copy()\n",
    "\n",
    "print(f\"Test known:   {len(test_known_df)} samples\")\n",
    "print(f\"Test unknown: {len(test_unknown_df)} samples\")\n",
    "\n",
    "\n",
    "def get_max_probs(model, df):\n",
    "    \"\"\"Return an array of max softmax probabilities for each row in df.\"\"\"\n",
    "    preds, _ = model.predict(dataset=df, skip_save_predictions=True)\n",
    "    return preds[\"label_probability\"].values\n",
    "\n",
    "\n",
    "print(\"Predicting with baseline ...\")\n",
    "probs_baseline_known = get_max_probs(model_baseline, test_known_df)\n",
    "probs_baseline_unknown = get_max_probs(model_baseline, test_unknown_df)\n",
    "\n",
    "print(\"Predicting with entropic ...\")\n",
    "probs_entropic_known = get_max_probs(model_entropic, test_known_df)\n",
    "probs_entropic_unknown = get_max_probs(model_entropic, test_unknown_df)\n",
    "\n",
    "print(\"Predicting with objectosphere ...\")\n",
    "probs_obj_known = get_max_probs(model_objectosphere, test_known_df)\n",
    "probs_obj_unknown = get_max_probs(model_objectosphere, test_unknown_df)\n",
    "\n",
    "print(\"Done.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "plot-histograms",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "fig, axes = plt.subplots(1, 3, figsize=(15, 4), sharey=True)\n",
    "bins = np.linspace(0, 1, 30)\n",
    "\n",
    "model_data = [\n",
    "    (\"CE Baseline\", probs_baseline_known, probs_baseline_unknown),\n",
    "    (\"Entropic Open-Set\", probs_entropic_known, probs_entropic_unknown),\n",
    "    (\"Objectosphere\", probs_obj_known, probs_obj_unknown),\n",
    "]\n",
    "\n",
    "for ax, (title, known, unknown) in zip(axes, model_data):\n",
    "    ax.hist(known, bins=bins, alpha=0.6, color=\"steelblue\", label=f\"Known (0-7)\\nmean={known.mean():.3f}\")\n",
    "    ax.hist(unknown, bins=bins, alpha=0.6, color=\"orangered\", label=f\"Unknown (8-9)\\nmean={unknown.mean():.3f}\")\n",
    "    ax.set_title(title, fontsize=13)\n",
    "    ax.set_xlabel(\"Max softmax probability\")\n",
    "    ax.legend(fontsize=9)\n",
    "\n",
    "axes[0].set_ylabel(\"Count\")\n",
    "fig.suptitle(\"Confidence on known vs unknown MNIST digits\", fontsize=14, y=1.02)\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"confidence_histograms.png\", dpi=120, bbox_inches=\"tight\")\n",
    "plt.show()\n",
    "print(\"Saved confidence_histograms.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "threshold-header",
   "metadata": {},
   "source": [
    "## Threshold-based detection\n",
    "\n",
    "We can turn max softmax probability into a binary known/unknown detector by choosing a threshold\n",
    "on a held-out validation set.  A sample with max-prob below the threshold is flagged as unknown.\n",
    "\n",
    "Below we sweep thresholds and plot the True Positive Rate (TPR — fraction of unknowns correctly\n",
    "flagged) against the False Positive Rate (FPR — fraction of known samples incorrectly flagged)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "threshold-detection",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score, roc_curve\n",
    "\n",
    "fig, axes = plt.subplots(1, 3, figsize=(15, 4), sharey=True)\n",
    "\n",
    "model_data_thresh = [\n",
    "    (\"CE Baseline\", probs_baseline_known, probs_baseline_unknown),\n",
    "    (\"Entropic Open-Set\", probs_entropic_known, probs_entropic_unknown),\n",
    "    (\"Objectosphere\", probs_obj_known, probs_obj_unknown),\n",
    "]\n",
    "\n",
    "for ax, (title, known, unknown) in zip(axes, model_data_thresh):\n",
    "    # Label: 0 = known, 1 = unknown; detector score = 1 - max_prob\n",
    "    y_true = np.concatenate([np.zeros(len(known)), np.ones(len(unknown))])\n",
    "    y_score = np.concatenate([1 - known, 1 - unknown])\n",
    "\n",
    "    fpr, tpr, thresholds = roc_curve(y_true, y_score)\n",
    "    auc = roc_auc_score(y_true, y_score)\n",
    "\n",
    "    ax.plot(fpr, tpr, lw=2, label=f\"AUC = {auc:.3f}\")\n",
    "    ax.plot([0, 1], [0, 1], \"--\", color=\"gray\", lw=1)\n",
    "    ax.set_title(title, fontsize=13)\n",
    "    ax.set_xlabel(\"False Positive Rate\")\n",
    "    ax.legend(fontsize=10)\n",
    "\n",
    "axes[0].set_ylabel(\"True Positive Rate\")\n",
    "fig.suptitle(\"ROC curve for unknown detection (1 - max_prob threshold)\", fontsize=14, y=1.02)\n",
    "plt.tight_layout()\n",
    "plt.savefig(\"roc_curves.png\", dpi=120, bbox_inches=\"tight\")\n",
    "plt.show()\n",
    "print(\"Saved roc_curves.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "summary-header",
   "metadata": {},
   "source": [
    "## Summary table\n",
    "\n",
    "The table below summarises the mean maximum softmax probability on known vs unknown test samples.\n",
    "A lower mean max-prob on unknowns indicates better open-set recognition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "summary-table",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "rows = [\n",
    "    (\"CE Baseline\", probs_baseline_known, probs_baseline_unknown),\n",
    "    (\"Entropic Open-Set\", probs_entropic_known, probs_entropic_unknown),\n",
    "    (\"Objectosphere\", probs_obj_known, probs_obj_unknown),\n",
    "]\n",
    "\n",
    "header = f\"{'Model':<22} | {'Mean max-prob (known)':>21} | {'Mean max-prob (unknown)':>23} | {'AUC (unknown det.)':>18}\"\n",
    "sep = \"-\" * len(header)\n",
    "print(sep)\n",
    "print(header)\n",
    "print(sep)\n",
    "\n",
    "for name, known, unknown in rows:\n",
    "    y_true = np.concatenate([np.zeros(len(known)), np.ones(len(unknown))])\n",
    "    y_score = np.concatenate([1 - known, 1 - unknown])\n",
    "    auc = roc_auc_score(y_true, y_score)\n",
    "    print(f\"{name:<22} | {known.mean():>21.4f} | {unknown.mean():>23.4f} | {auc:>18.4f}\")\n",
    "\n",
    "print(sep)\n",
    "print()\n",
    "print(\"Interpretation:\")\n",
    "print(\"  - CE Baseline:         high confidence on unknowns — the model cannot say 'I don't know'\")\n",
    "print(\"  - Entropic Open-Set:   lower mean max-prob on unknowns — closer to uniform distribution\")\n",
    "print(\"  - Objectosphere:       similar to entropic; also creates a logit-norm gap (not shown here)\")"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}