/notebooks/benchmarks/text/Machine Translation Benchmark Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text Data Explanation Benchmarking: Machine Translation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook demonstrates how to use the benchmark utility to benchmark the performance of  an explainer for text data. In this demo, we showcase explanation performance for partition explainer on a Machine Translation model. The metrics used to evaluate are \"keep positive\" and \"keep negative\". The masker used is Text Masker. \n",
    "\n",
    "The new benchmark utility uses the new API with MaskedModel as wrapper around user-imported model and evaluates masked values of inputs. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nlp\n",
    "from transformers import AutoModelForSeq2SeqLM, AutoTokenizer\n",
    "\n",
    "import shap\n",
    "import shap.benchmark as benchmark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data and Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\"Helsinki-NLP/opus-mt-en-es\")\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(\"Helsinki-NLP/opus-mt-en-es\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using custom data configuration default\n"
     ]
    }
   ],
   "source": [
    "dataset = nlp.load_dataset(\"xsum\", split=\"train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "s = [dataset[\"summary\"][i] for i in range(10)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Explainer Object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "explainers.Partition is still in an alpha state, so use with caution...\n"
     ]
    }
   ],
   "source": [
    "explainer = shap.Explainer(model, tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run SHAP Explanation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Partition explainer:  10%|███▎                             | 1/10 [00:00<?, ?it/s]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Partition explainer:  30%|███████▌                 | 3/10 [01:10<02:22, 20.37s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Partition explainer:  40%|██████████               | 4/10 [01:50<02:36, 26.09s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Partition explainer:  50%|████████████▌            | 5/10 [02:22<02:20, 28.00s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Partition explainer:  60%|███████████████          | 6/10 [02:41<01:40, 25.19s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Partition explainer:  70%|█████████████████▌       | 7/10 [03:17<01:25, 28.58s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Partition explainer:  80%|████████████████████     | 8/10 [03:45<00:56, 28.42s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Partition explainer:  90%|██████████████████████▌  | 9/10 [04:23<00:31, 31.24s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\r",
      "Partition explainer: 100%|████████████████████████| 10/10 [05:03<00:00, 33.67s/it]"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=48.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Partition explainer: 11it [05:38, 30.77s/it]                                      \n"
     ]
    }
   ],
   "source": [
    "shap_values = explainer(s)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Define Metrics (Sort Order & Perturbation Method)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "sort_order = \"positive\"\n",
    "perturbation = \"keep\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Benchmark Explainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=10.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYAAAAEGCAYAAABsLkJ6AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAo6klEQVR4nO3deXxV9Z3/8dcnOyGBhIQ9QFB2UKlEXH+K1bpUxbpWR9tptaPdnY7T6XSsbbW2M/OzrWPH+U3F1unUWtTWBdzqRsGlak3YF5XIGiAkIWFNQpKbz++Pe8EAIVxI7j25976fj8d95J5zzz3ncwI5n/Ndzvdr7o6IiKSetKADEBGRYCgBiIikKCUAEZEUpQQgIpKilABERFJURtABHI3i4mIvLS0NOgwRkYRSUVFR5+4DD16fUAmgtLSU8vLyoMMQEUkoZra+s/WqAhIRSVFKACIiKUoJQEQkRSVUG0BnWltbqaqqorm5OehQUkpOTg4lJSVkZmYGHYqIHKOETwBVVVXk5+dTWlqKmQUdTkpwd7Zt20ZVVRWjR48OOhwROUYJXwXU3NxMUVGRLv5xZGYUFRWp1CWS4BI+AQC6+AdAv3ORxJfwVUAiIkFyd0LtTlu70xpqp6Wtnea2dppbQzS3hmhqCdHUGqKx5ePl5tYQrSGntb2dtpCzb1R+p+P7/QcA4IqTSxhd3LdHY1cC6CHPPPMMV1xxBatWrWLChAkAzJ8/n5/+9Kc899xz+7f7whe+wKWXXsrVV19Na2srd955J08++ST5+flkZ2fz/e9/n4svvrjTY3zzm9/k4YcfZvfu3Yd81tLSwq233kp5eTlpaWncf//9zJgx44BtZs6cyZo1a1i+fDkAn/3sZ/nggw8A2L59OwUFBSxevJhHH32Ue++9d//3li5dysKFC5k6dWp3fkUiUWsLtbOnJcTuvW3s2dtGY8vHF869be2RC274Z2uonZaQ09LWvv8CHF7XTmtb+PO29nZa2sLfaWt3QqHw+9Z9P9vCF+Pw/py2UPv+C3qo3fe/2vb//Hh9exymVDGDk0cVJlcCMLOHgUuBGnefEmQs3TV79mzOOussZs+ezV133RXVd+688062bNnC8uXLyc7OZuvWrSxYsKDTbcvLy2loaDjsvh566CEAli1bRk1NDRdffDHvvfceaWnhWr6nnnqKvLy8A77z+OOP739/++23079/fwBuuOEGbrjhhv37+8xnPqOLvxzC3dnb1s7uvW3sbm5jV3Mbu/a20tQSYk9LiKaWNppb990Jt9Pc9vEFfN/Pva3hi/y+fezeG2L33laaW9u7FZsZZKWnkZmeRka6HfA+I83ISEsjPc3ITDcy0tPISDPysjNIj3y2b31mmoXXpRtpFvluZPv0tPC6tLR968M/s9LTyMlM3//KzUqnT1Y6fTI//pmTmU5muoVjSjPMDOsQe7yqWIMuAfwGeAD4bcBxdMvu3bt58803+fOf/8xll10WVQJobGzkoYceYu3atWRnZwMwePBgrr322kO2DYVCfPvb3+b3v/89Tz/9dKf7W7lyJZ/85CcBGDRoEAUFBZSXlzN9+nR2797Nz3/+c2bNmtXp/t2dJ554gnnz5h3y2ezZs7nuuuuOeD6SmBpb2qjdtZdte1rY3thC/Z5WdjS1srOplZ3NrexsamP33lZ2NYfvxPe0hGiMXLD3tIQIHcXtb2a6kZORTnZmGtmRnzkZ6eRlZzAoP4fRxRnk52SQl51B36wM+maHP+ubHX7fJzODPlnhC+e+C3hGmpGVEb64Z6WnRd6HL9JyZIEmAHd/3cxKe2p/dz27gpWbd/bU7gCYNKwfP7hscpfbzJkzh4suuohx48ZRVFRERUUF06ZN6/I7lZWVjBw5kn79+h0xhgceeICZM2cydOjQw25z0kknMXfuXK6//no2btxIRUUFGzduZPr06dx5553cfvvt5ObmdvrdN954g8GDBzN27NhDPnv88ceZM2fOEWOU3sfd2d7YSlVDExsbGtlQ38jG+vDPTQ1NVO9sprEldNjv52eHL8j5OZnk52RQkJtFSWEGuVnp+y/KfbPDF+zwhTszvC4r44C73pzMdLIz0nRR7oWCLgEckZndAtwCMHLkyICj6dzs2bO57bbbALjuuuuYPXs206ZNO2wx7miKd5s3b+YPf/gD8+fP73K7m266iVWrVlFWVsaoUaM444wzSE9PZ/HixXz00Ufcd999rFu37rDxX3/99Yesf/fdd8nNzWXKlISunUsqTS0htu5spmbXXmp37WV7UwvbG8N37NsbW9ne1EJDYys1O5up3tl8SFVKQW4mIwpzmTA0nxnjBzEwP5vivCyK8rIozA2/CnIzyc/JJD1NPb2SXa9PAO4+C5gFUFZW1mV580h36rFQX1/PvHnzWLZsGWZGKBTCzLj33nspKio6pN6+vr6e4uJixowZw4YNG9i5c2eXpYBFixZRWVnJmDFjgHDV0ZgxY6isrDxgu4yMDO677779y2eccQbjxo1jwYIFlJeXU1paSltbGzU1NcyYMWN/Qmlra+Opp56ioqLikGM/9thjnSYGiZ29bSE21jexoX5P5GcjVQ2NbNrexKaGJhoaWzv9XlZGGoW5mRT0yaJ/n0ymDO/PpyYNZnC/HEoKcxk5IJcRA/qQn6Mnt+VjvT4B9HZ//OMf+dznPseDDz64f90555zDG2+8wamnnsrmzZtZtWoVEydOZP369SxZsoSpU6eSm5vLzTffzG233caDDz5IVlYWtbW1zJ8/n2uuuWb/vi655BKqq6v3L+fl5R1y8YdwYnB3+vbtyyuvvEJGRgaTJk1i0qRJfOUrXwFg3bp1XHrppQeUJl599VUmTJhASUnJAftrb2/niSee4I033uipX1VKawu1s6u5jR1NrTQ0tlC7ay91u1uo3tkcvsA3NFHV0MTmHU37uwEC5GSmMaIwl+GFfTippIBhBX0Y3C+HQfnZDMzPZkDf8AU/JzM9uJOThKUE0E2zZ8/mO9/5zgHrrrrqKmbPns3ZZ5/N7373O774xS/S3NxMZmYmv/rVr/b3trnnnnv43ve+x6RJk8jJyaFv377cfffdUR977ty5lJeXc/fdd1NTU8OFF15IWloaw4cP55FHHolqH4e7y3/99dcZMWIExx13XNTxpCJ3p6GxlQ31jazftoeqhqb9d+t1u/eyvTHcqLp7b1un3zeDIf1yGF7Qh1NKCxlVVMKoolxGFeUyYkAuA/Oy9dCdxIy5x6ET6+EObjYbmAEUA1uBH7j7rw+3fVlZmR88Icy+u2uJv1T53be3O1t3Ne+vktlQ38i6uj2srdvDuro97Dro4l6Ym8nwwj4Mys+hIFIt069PBv37ZNK/TyYFuZkMzMuhOD+Lor7ZZGWocVRiy8wq3L3s4PVB9wJSBbMEzt2p3bWX9fWNbNjWyObt4aqYTdubqapvpKqhiZbQx42paQbDC/tQWtSXK04ezqiivowaEL5jLynsQ99sFawlMeh/qqQMd6eqoYkPqnfxwdZdVNbs5sOtu1hTu4em1gO7QxbnZTG0fx8mDM3nU5MGUzIgl1EDwo2pwwr66K5dkkJSJAB3Vz1pnAVZdRitzdub+MtH21ixeQcrNu1k1ZadB1TXDOufw5jB+Zw6uojS4n09ZXIZXtBHjaqSEhI+AeTk5LBt2zYNCR1H++YDyMnJCTqUQ6zftocXl1fz4vJqlmzcDkCfzHQmDs3n8k8MY+LQfkwYks+4wfnqEikpL+ETQElJCVVVVdTW1gYdSkrZNyNYb7CxvpFnl27mhWVbWL4p/CT4iSX9+faF4zl/4mDGDMrTQ00inUj4BJCZmalZqVLQjsZWnl+2hacXVfHeuvDDdlNHFHDHpydy8QlDKCnsfNgLEflYwicASR2NLW28uqqGuYs3seDDWlpDzvED+/LtC8cz86RhjBigi77I0VACkF6tuTXEgg9reW7pFl5duZWm1hBD+uXwt6eXcvnU4UwZ3k9tPyLHSAlAeqXKml38/t2NPLmwih1NrRTmZnLlycO57KRhTC8dQJrq9EW6TQlAeo1Qu/Paqq38+s21vLu2nsx044LJQ7hmWglnjikmU8MJi/QoJQAJ3M7mVp5euIn/eWst67Y1MrygD9+5aALXlJVQnJcddHgiSUsJQALR3u68t66ex8s38sKyLTS3tnPSiAIeuHA8F00eoslDROJACUDiqrJmN88s2sQzizdR1dBEXnYGV55cwnWnjOCE4f3VoCsSR0oAEnPt7c6892v49ZtreXvNNtIMzho7kH/41DgumjKE3Cz9NxQJgv7yJGaaWkL8cWEVD7+5lrV1exjaP4fvXDSBq6YNZ1B+7xtGQiTVKAFIj6vZ1cyj72zgkXfWU7+nhZNK+vOL6z/BxVOGqCePSC+iBCA9wt1ZuKGB//3Lel5cvoXWkHP+xEHccvbxnFJaqLp9kV5ICUC6ZdvuvTy9aBN/KK/ig627yM/J4MbTRvG500Zx3MC8oMMTkS4oAcgxWbShgf95a93+u/2TRhTwkytO4PKpwzQjlkiC0F+qRM3deW1VDf81v5JFG7aTn53BDaeO4vrpIxk/JD/o8ETkKCkBSFSWbNzOj19YxV/X1jNyQC4/vGwSV5eNIE93+yIJS3+90qWGPS3c8/wqnlxYRVHfLH50+WSumz5SvXlEkoASgHTK3Xl26RbumruCHU2tfGXG8Xx1xvGaRlEkiSgByCGqdzRzx9PLeO39Gk4q6c/vvnQqE4f2CzosEelhSgCyn7vzRPlG7nluFa3t7Xzvkol88czRmk9XJEkFmgDM7CLgfiAd+JW7/1uQ8aSync2t/P1ji5n3fg2njh7Av191IqXFfYMOS0RiKLAEYGbpwH8BnwKqgPfMbK67rwwqplS1tm4PN//ve2zY1sgPL5vE508v1YxbIikgyBLAdKDS3dcAmNljwOWAEkAcvbm6jq8+WkFGehqPfulUTj2uKOiQRCROguzLNxzY2GG5KrLuAGZ2i5mVm1l5bW1t3IJLdu7Or95Yw+cffpeh/fsw52tn6uIvkmJ6fSOwu88CZgGUlZV5wOEkhebWEP/y1DKeWrSJCycP5mfXTtUDXSIpKMi/+k3AiA7LJZF1EkNVDY189dGFLK3awbfOH8c3PjlG9f0iKSrIBPAeMNbMRhO+8F8H/E2A8SS9BR/WcttjiwiFnFmfm8YFk4cEHZKIBCiwBODubWb2deAlwt1AH3b3FUHFk8zcnQfmVfLzVz9k/OB8/vvGaYxWF0+RlBdoxa+7vwC8EGQMqeA/Xl3N/a+t5opPDOcnV5xAn6z0oEMSkV5ALX9J7uE313L/a6u5ZloJ//fqEzUzl4jspyEdk9iTFVXc/dxKLpw8mH+98gRd/EXkAEoASapifQP/9ORSzhxTxP3XfYIMDd8sIgfRVSEJ7Wxu5bbHFjG0fw7/feM0cjJV5y8ih1IbQBL6/jPL2bKjmSduPZ1+Gr9fRA5DJYAk8/SiKp5ZvJnbzhvLtFGFQYcjIr2YEkAS2bS9iTufWcEppYV87dwxQYcjIr2cEkAS+eHcFYTanZ9fO1WTuIjIESkBJImXV1Tzysqt3Hb+WEYMyA06HBFJAEoASaCxpY27nl3JuMF53HzW6KDDEZEEoV5ASeD+11azaXsTf/jy6WSqv7+IRElXiwT3QfUufv3GWq4tK+GU0gFBhyMiCUQJIIG1tzt3PL2M/JwM/vniiUGHIyIJRgkggf2hYiPl6xv47qcnMqBvVtDhiEiCUQJIUNt27+VfX3yf6aMHcM20kqDDEZEEpASQoH7ywvvsbm7jx5+ZolE+ReSYKAEkoLc/2saTC6u45ezjGDs4P+hwRCRBKQEkmObWEP/y9DJGDsjlG58cG3Q4IpLA9BxAgrn/tdWsrdvDo186VVM7iki3qASQQFZs3sGs19dwzbQSzhxTHHQ4IpLglAASRFuonX9+chmFuZnccYn6/ItI96kKKEH85i/rWLZpBw/8zScoyFWffxHpPpUAEkD1jmbue+VDZowfyCUnDA06HBFJEkoACeCe51fS2u7cNXOy+vyLSI85YgIws2uiWSex8VZlHc8t3cLXZoxhVFHfoMMRkSQSTQngu1Gui5qZXWNmK8ys3czKurOvZLa3LcSdc5YzqiiXW885LuhwRCTJHLYR2MwuBj4NDDezX3T4qB/Q1s3jLgeuBB7s5n6S2u/e2cCa2j385ounkJOpPv8i0rO66gW0GSgHZgIVHdbvAr7VnYO6+ypA9dldaG93fvv2Ok4pLWTG+EFBhyMiSeiwCcDdlwBLzOxRd+/uHf8xM7NbgFsARo4cGVQYcfdGZR3rtzVy+wXjgw5FRJJUNM8BrDYzP3ilu3dZKW1mrwJDOvnoDnefE2V8uPssYBZAWVnZIXEkq0feXk9xXhYXTe7sVygi0n3RJICOjbQ5wDXAEecedPfzjzWoVFfV0Mi897fy1RljyMpQT10RiY0jXl3cfVuH1yZ3/w/gktiHlrpm/3UDANefmjpVXiISf0csAZjZyR0W0wiXCLo1hISZXQH8JzAQeN7MFrv7hd3ZZ7LY2xbi8fc2ct7EwQwv6BN0OCKSxKK5kP+sw/s2YB1wbXcO6u5PA093Zx/J6k/Lq6nb3cLnThsVdCgikuSOmADc/dx4BCJhzy7ZwrD+OZyl4Z5FJMaiGQqiyMx+YWYLzazCzO43s6J4BJdqmltDvFVZx3kTB5OWpmckRCS2ouli8hhQC1wFXB15/3gsg0pV766tp6k1xCcn6MEvEYm9aNoAhrr7jzos32Nmn41VQKls3qqt5GSmcfrxKmCJSOxFUwJ42cyuM7O0yOta4KVYB5Zq3J15H9Rw1phijfsjInERTQL4O+D3wN7I6zHgVjPbZWY7YxlcKqms2c3G+ibOVfWPiMRJNL2A8uMRSKqb934NgOr/RSRuoukF9Fo066R7Xnu/holD+zG0vx7+EpH4OGwCMLMcMxsAFJtZoZkNiLxKgeFxizAF7GhspWJ9A+fp7l9E4qirKqBbgb8HhgELO6zfCTwQw5hSzoLVtYTaXfX/IhJXXc0HcD9wv5l9w93/M44xpZz579dQmJvJ1BEFQYciIikkmucAdpjZ5w9e6e6/jUE8KcfdebOyjrPGDiRdT/+KSBxFkwBO6fA+BziPcJWQEkAPqKzZTc2uvZw1Rg9/iUh8RdMN9Bsdl82sgPCzANID3qysA+BMDf4mInF2LNNN7QFG93QgqeqtyjpGFeVSUpgbdCgikmKimRDmWWDfXLxpwCTgiVgGlSraQu28s6aemVOHBR2KiKSgaNoAftrhfRuw3t2rYhRPSllStYPde9s09r+IBCKaKqANQH7ktUUX/57zVmUdZnD6cWoAFpH4O2wJwMz6Ab8CpgFLIqunmlkFcLO7ayC4bnqzso4pw/pT2Dcr6FBEJAV1VQL4BbASGOvuV7r7lcDxwDL0JHC37dnbxqINDer9IyKB6aoN4Ex3/0LHFe7uwN1mtjqmUaWAv66rpzXkqv8XkcAcSzdQAD2y2k1vra4jKyONstLCoEMRkRTVVQL4i5l938wOuNib2Z3A27ENK/m9vrqWslGFmv1LRALTVRXQN4BfA5VmtjiybiqwCLg5tmElty07mvhw626unlYSdCgiksK6Gg10J3CNmR1P+OEvgJXu/lF3D2pm9wKXAS3AR8AX3X17d/ebKF7/sBaAs8cNDDgSEUllR2wDcPeP3P3ZyKvbF/+IV4Ap7n4i8CHw3R7ab0JY8GEtg/tlM36wZtsUkeAcayNwt7j7y+7eFll8B0iZupC2UDtvrq7jnHEDOah5RUQkrgJJAAe5CXjxcB+a2S1mVm5m5bW1tXEMKzaWVG1nZ3Obqn9EJHBdPQk8oKsvunt9V5+b2avAkE4+usPd50S2uYPw+EKPdnGcWcAsgLKyMj/cdoliwQe1pBnq/y8igeuqF1AF4VFAO6uncOC4rnbs7ud39bmZfQG4FDgv8oBZSliwuo6pIwooyNXwDyISrK56AcVszH8zuwj4J+Acd2+M1XF6m/o9LSyt2s5t540NOhQRkSO3AVjYjZEHwDCzkWY2vZvHfYDw6KKvmNliM/tlN/eXEN5YXYs7nKP6fxHpBaKZD+D/Ae3AJ4EfAbuAJzlwruCj4u5jjvW7iez1D+soyM3kxJKCoEMREYkqAZzq7ieb2SIAd28wM1VgH6W2UDt//qCGs8cOJD1N3T9FJHjRdANtNbN0ItNCmtlAwiUCOQoV6xuo39PCBZMHBx2KiAgQXQL4BfA0MMjMfgy8CfwkplEloZdWbCUrI40Z4wcFHYqICBBFFZC7PxqZBew8wl1CP+Puq2IeWRJxd15eWc1ZY4rJy46m1k1EJPaifRCsBpjd8bMjPQgmH1u5ZSdVDU18/dyUbPsWkV4q2gfBRgINkfcFhCeKj9lzAsnm5RVbMYPzJ6n+X0R6j8O2Abj7aHc/DngVuMzdi929iPDTuy/HK8Bk8NKKaspGFVKclx10KCIi+0XTCHyau7+wb8HdXwTOiF1IyWXDtkber97FhZM7GxZJRCQ40bRIbjaz7wG/iyzfAGyOXUjJ5eWV1QBcMEkJQER6l2hKANcDAwl3BX0aGBRZJ1F4aUU1E4bkM7IoN+hQREQOEE030HrgNjPLDy/67tiHlRxqdjZTvr6Bvz9vXNChiIgcIprB4E6IDAOxHFhhZhVmNiX2oSW+F5dX4w6XnKjqHxHpfaKpAnoQ+Ad3H+Xuo4DbiUzQIl17fukWxg3OY8wgzf0rIr1PNAmgr7v/ed+Cu88H+sYsoiRRs7OZ99bXc8kJw4IORUSkU9H0AloTmQvgkcjyjcCa2IWUHFT9IyK9XTQlgJsI9wJ6KvIaGFknXVD1j4j0dtH0AmoAvhmHWJLGvuof9f4Rkd6sq8Hg5nb1RXef2fPhJAdV/4hIIuiqBHA6sJHwKKDvEh4ITqKg6h8RSQRdtQEMAf4FmALcD3wKqHP3Be6+IB7BJaLN25vU+0dEEkJXo4GG3P1P7v63wGlAJTDfzL4et+gS0LNLNuMOl09VAhCR3q3LRmAzywYuITz2TykfTw8phzFn8WZOGlFAabEelRCR3q2rRuDfEq7+eQG4y92Xxy2qBLV66y5WbtnJDy6bFHQoIiJH1FUJ4EZgD3Ab8E2z/W3ARnhQuH4xji3hzFm8mTSDS09U9Y+I9H6HTQDuHs1DYhLh7sxZsokzxxQzMF8zf4lI7xfIRd7MfmRmS81ssZm9bGYJf8u8cMN2NtY38Zmpw4MORUQkKkHd5d/r7ie6+1TgOeD7AcXRY+Ys3kR2RhoXTNbE7yKSGAJJAO6+s8NiX8CDiKOntIbaeX7pFs6fNJj8nMygwxERiUo0o4HGhJn9GPg8sAM4t4vtbgFuARg5cmR8gjtKb1bWsW1PCzNPSviaLBFJITErAZjZq2a2vJPX5QDufoe7jwAeBQ77cJm7z3L3MncvGzhwYKzC7Za5izfTLyeDGeN7Z3wiIp2JWQnA3c+PctNHCT9r8INYxRJLTS0hXlpRzeVTh5GdkR50OCIiUQuqF9DYDouXA+8HEUdPeHXVVhpbQsw8Sb1/RCSxBNUG8G9mNh5oB9YDXw4ojm6bs3gTQ/rlMH30gKBDERE5KoEkAHe/Kojj9rSGPS3M/6CWL55ZSnqaRssWkcSip3274cXl1bS1O5fr4S8RSUBKAN0wZ/Emjh/Yl8nDNCySiCQeJYBjtHl7E39dV8/Mk4bTYaA8EZGEoQRwjDTxi4gkOiWAY/TM4s1M1cQvIpLAlACOwYdbd7Fqy04+o7t/EUlgSgDH4JlFm0hPMy7RxC8iksCUAI6SuzNn8WZN/CIiCU8J4ChVrG9g0/YmVf+ISMJTAjhKzyzeRE5mGhdMHhJ0KCIi3aIEcBT2T/wycTB52YFNpSAi0iOUAI7C2x9to6GxVRO/iEhSUAI4Cn9aUU1uVjpnj9PELyKS+JQAohRqd15esZVzxw8iJ1MTv4hI4lMCiNKiDQ3U7d7LBZMHBx2KiEiPUAKI0ksrqslKT+OTEwYFHYqISI9QAoiCu/OnFdWcMaaI/JzMoMMREekRSgBRWLVlFxvrm7hIff9FJIkoAUThTyuqSTM4f5Lq/0UkeSgBROHlFdWUlQ6gOE9j/4hI8lACOIJ1dXt4v3oXF6r6R0SSjBLAEby0ohqAC1T9IyJJRgngCF5aUc2U4f0YMSA36FBERHqUEkAXanY2s3DDdi6cpOofEUk+gSYAM7vdzNzMioOM43BeWrkVgAunKAGISPIJLAGY2QjgAmBDUDEcycsrqhld3Jexg/KCDkVEpMcFWQK4D/gnwAOM4bB2NLby9kfbuGDyYMws6HBERHpcIAnAzC4HNrn7kiCOH415H2ylrd3V/VNEklbMprUys1eBzq6edwD/Qrj6J5r93ALcAjBy5Mgei+9IXlq+lcH9splaUhC3Y4qIxFPMEoC7n9/ZejM7ARgNLIlUrZQAC81surtXd7KfWcAsgLKysrhUFzW1hFjwYS1XTyshLU3VPyKSnOI+sa27LwP2j6lsZuuAMnevi3csh/Pa+1tpag2p+kdEkpqeA+jEo+9sYHhBH04/vijoUEREYibwBODupb3p7r+yZjdvr9nG35w6knRV/4hIEgs8AfQ2v393A5npxrVlI4IORUQkppQAOmhqCfHHio1cOHkIA/M19LOIJDclgA6eXbqZnc1t3HjaqKBDERGJOSWADh59Zz1jB+Vx6ugBQYciIhJzSgARS6u2s6RqBzecOlJDP4hISlACANydn7ywioLcTK6cVhJ0OCIicaEEALywrJp31tTzjxeMp19OZtDhiIjERcongKaWED9+fiUTh/bj+unxG2tIRCRoKZ8A/nvBR2ze0cxdMyfrwS8RSSkpnQA21jfyywUfMfOkYUxXzx8RSTEpnQB+8sIq0s347qcnBB2KiEjcpWwC+MtHdby4vJqvnXs8Q/v3CTocEZG4S8kE0BZq5+5nV1JS2Icv/Z/jgg5HRCQQKZkAZr+3kferd3HHpyeSk5kedDgiIoFIuQSwvbGFn7/8AacdN4CLpmjCFxFJXSmXAP7j1dXsaGrl+5dO1pAPIpLSUioBrN66i0feWc/100cyaVi/oMMREQlUyiQAd+fu51bSNyud2y8YH3Q4IiKBS5kE8NqqGt5YXce3PjWOAX2zgg5HRCRwKZEA9raFuOf5lYwZlKfJXkREIlIiAfzmrXWs29bInZdOIjM9JU5ZROSIUuJqWJyXzTXTSjhn3MCgQxER6TUygg4gHq6aVsJVmuhFROQAKVECEBGRQykBiIikqEASgJn90Mw2mdniyOvTQcQhIpLKgmwDuM/dfxrg8UVEUpqqgEREUlSQCeDrZrbUzB42s8LDbWRmt5hZuZmV19bWxjM+EZGkZu4emx2bvQp0Nt7yHcA7QB3gwI+Aoe5+05H2WVZW5uXl5T0ap4hIsjOzCncvO3h9zNoA3P38aLYzs4eA52IVh4iIdC6QRmAzG+ruWyKLVwDLo/leRUVFnZmtP8bDFhMudaQSnXNq0Dmnhu6cc6eDoMWsCqgrZvYIMJVwFdA64NYOCSFWxyzvrAiUzHTOqUHnnBpicc6BlADc/XNBHFdERD6mbqAiIikqlRLArKADCIDOOTXonFNDj59zIG0AIiISvFQqAYiISAdKACIiKSrpEoCZXWRmH5hZpZn9cyefZ5vZ45HP3zWz0gDC7FFRnPM/mNnKyNAbr5lZwk+MfKRz7rDdVWbmZpbQXQajOV8zuzby77zCzH4f7xh7WhT/r0ea2Z/NbFHk/3bCjyocGRqnxsw6fTbKwn4R+Z0sNbOTu3VAd0+aF5AOfAQcB2QBS4BJB23zVeCXkffXAY8HHXcczvlcIDfy/iupcM6R7fKB1wkPPVIWdNwx/jceCywCCiPLg4KOOw7nPAv4SuT9JGBd0HH3wHmfDZwMLD/M558GXgQMOA14tzvHS7YSwHSg0t3XuHsL8Bhw+UHbXA78b+T9H4HzzMziGGNPO+I5u/uf3b0xsvgOkOjzY0bz7wzhcab+HWiOZ3AxEM35/h3wX+7eAODuNXGOsadFc84O9Iu87w9sjmN8MeHurwP1XWxyOfBbD3sHKDCzocd6vGRLAMOBjR2WqyLrOt3G3duAHUBRXKKLjWjOuaObCd9BJLIjnnOkaDzC3Z+PZ2AxEs2/8ThgnJm9ZWbvmNlFcYsuNqI55x8CN5pZFfAC8I34hBaoo/1771JKTAovYWZ2I1AGnBN0LLFkZmnAz4EvBBxKPGUQrgaaQbiE97qZneDu24MMKsauB37j7j8zs9OBR8xsiru3Bx1Yoki2EsAmYESH5ZLIuk63MbMMwkXHbXGJLjaiOWfM7HzCQ3HPdPe9cYotVo50zvnAFGC+ma0jXFc6N4EbgqP5N64C5rp7q7uvBT4knBASVTTnfDPwBIC7vw3kEB4wLZlF9fcerWRLAO8BY81stJllEW7knXvQNnOBv428vxqY55HWlQR1xHM2s08ADxK++Cd63TAc4ZzdfYe7F7t7qbuXEm73mOnuiTqZRDT/r58hfPePmRUTrhJaE8cYe1o057wBOA/AzCYSTgDJPmvUXODzkd5ApwE7vBsDaSZVFZC7t5nZ14GXCPcieNjdV5jZ3UC5u88Ffk24qFhJuLHluuAi7r4oz/leIA/4Q6S9e4O7zwws6G6K8pyTRpTn+xJwgZmtBELAt909YUu2UZ7z7cBDZvYtwg3CX0jwmznMbDbhRF4cadv4AZAJ4O6/JNzW8WmgEmgEvtit4yX470tERI5RslUBiYhIlJQARERSlBKAiEiKUgIQEUlRSgAiIilKCUB6HTMLmdliM1tuZn8ws9wAYphhZmcc5rMfmtk/HrRuXaT/fVyZ2fxjfcDNzH5jZlf3dEySOJQApDdqcvep7j4FaAG+HM2XIk9295QZQKcJQCRZKAFIb/cGMMbM+kbGSv9rZPz3ywHM7AtmNtfM5gGvmVmemf2PmS2LjJd+VWS7C8zsbTNbGClV5EXWrzOzuyLrl5nZBAvPEfFl4FuRksj/iTZYMys1s1Vm9lBkXP6XzaxP5LP5ZnafmZVHtjnFzJ4ys9Vmdk+HfTxjZhWR798SWZceuWNfHonzWwcdNy3y+T2Rbe81s/civ4NbI9uYmT1g4TH2XwUGHfs/iySDpHoSWJJL5I7+YuBPhMcxmufuN5lZAfDXyEUMwuOnn+ju9Wb274Qfjz8hso/CSNXM94Dz3X2PmX0H+Afg7sj369z9ZDP7KvCP7v4lM/slsNvdf3oMoY8Frnf3vzOzJ4CrgN9FPmtx9zIzuw2YA0wj/ET6R2Z2X+Tp3Zsi59IHeM/MngRKgeGRUhGR38E+GcCjhMeQ/3Ekaexw91PMLBt4y8xeBj4BjCc8dv5gYCXw8DGcnyQJJQDpjfqY2eLI+zcID9/xF2Bmh7r3HGBk5P0r7r5vDPXz6TC8h7s3mNmlhC96b0WGwsgC3u5wvKciPyuAK6OI73CPz+9bv9bd98VfQfjivc++YSqWASv2jeNiZmsID/K1DfimmV0R2W4E4YTyAXCcmf0n8Dzwcod9Pgg84e4/jixfAJzYoX6/f2QfZwOz3T0EbI6UmiSFKQFIb9Tk7lM7rrDwlfsqd//goPWnAnuOsD8jnCSuP8zn+0ZHDRHd38Q24OBJOPKB7ZGfHUdbDQF9OjlW+0HbtQMZZjaDcBI73d0bzWw+kBNJZCcBFxKunroWuCny3b8A55rZz9y9mfD5fsPdX+oYoCXBlInSs9QGIIniJeAbkUSwb4TTzrwCfG3fgpkVEh4N9EwzGxNZ19fMxh3heLsIX8w78zrh0kh+ZH9XAksid9bd1R9oiFz8JxAeynrfCJ9p7v4k4eqsjnPB/prwIGFPRKrNXgK+YmaZke+OM7O+kbg/G2kjGEp4qlBJYUoAkih+RHhUxKVmtiKy3Jl7gMJIY+kS4Fx3ryU8OcxsM1tKuPpnwhGO9yxwRWeNwO6+FHgAeDNSVfVl4EvHdlqH+BPhksAq4N8IJy8Iz/o0P3K83wHfPSimnxOeE/gR4FeE6/cXWnhy8QcJl2yeBlZHPvstB1aDSQrSaKAiIilKJQARkRSlBCAikqKUAEREUpQSgIhIilICEBFJUUoAIiIpSglARCRF/X/iy7Zlkd55ogAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sp = benchmark.perturbation.SequentialPerturbation(explainer.model, explainer.masker, sort_order, perturbation)\n",
    "xs, ys, auc = sp.model_score(shap_values, s)\n",
    "sp.plot(xs, ys, auc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "sort_order = \"negative\"\n",
    "perturbation = \"keep\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=10.0), HTML(value='')))"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYoAAAEGCAYAAAB7DNKzAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/d3fzzAAAACXBIWXMAAAsTAAALEwEAmpwYAAAscElEQVR4nO3deXxV1bn/8c+TmQAJEBKmAGEKKEIQoiIO4NBWbdU6Vr21WrXUqbXe2lZ/rbdW23vtSO1wW6W016lYtRbROhdBFKcEmVFBxjCGeQgQkjy/P86GRkhODiQnO8n5vl+v/eKcvdfe+9kn5DxZa+29lrk7IiIi9UkKOwAREWnZlChERCQqJQoREYlKiUJERKJSohARkahSwg4gHrp27eoFBQVhhyEi0mqUlpZucvfcura1yURRUFBASUlJ2GGIiLQaZrayvm1qehIRkaiUKEREJColChERiapN9lGISMu3f/9+ysrK2Lt3b9ihJJSMjAzy8/NJTU2NeR8lChEJRVlZGR07dqSgoAAzCzuchODubN68mbKyMvr16xfzfmp6EpFQ7N27l5ycHCWJZmRm5OTkHHEtTolCREKjJNH8juYzV6II7N1fzYMzPuHNJZvCDkVEpEUJPVGY2bfNzM2saz3bf2ZmC81ssZn9xuL0J0hachIPvbGMp0tXx+PwItJCTZkyBTPjww8/PLhu+vTpfOELX/hUuWuvvZann34aiHTE33nnnQwaNIiRI0dy8skn8+KLLx527OXLl3PSSScxcOBAvvSlL1FZWXlYmcrKSr761a8ybNgwioqKmD59+sFtpaWlDBs2jIEDB/LNb36TA/MHzZkzh9GjRzNixAiKi4t57733AHj88ccZPnw4w4YNY8yYMcydO7fRnw+EnCjMrDfwWWBVPdvHAKcAw4HjgBOAsfGIJSnJOL0wlzeWbKKmRpM5iSSKyZMnc+qppzJ58uSY97n77rtZt24dCxYsYPbs2UyZMoWdO3ceVu573/set99+O0uXLqVz585MmjTpsDITJ04EYP78+bz66qt8+9vfpqamBoCbbrqJiRMnsmTJEpYsWcJLL70EwHe/+11++MMfMmfOHO69916++93vAtCvXz9mzJjB/Pnzufvuuxk/fvwRfx51CbtGMQH4LlDfN7MDGUAakA6kAhviFczYwly27K5kwdrt8TqFiLQgu3bt4s0332TSpEk88cQTMe1TUVHBxIkT+e1vf0t6ejoA3bp14/LLL/9UOXdn2rRpXHrppQBcc801TJky5bDjLVq0iDPPPBOAvLw8OnXqRElJCevWrWPHjh2MHj0aM+MrX/nKwf3NjB07dgCwfft2evbsCcCYMWPo3LkzAKNHj6asrOzIPpB6hHZ7rJldCKxx97n1tSa5+9tm9jqwDjDgd+6+uJ7jjQfGA/Tp0+eoYjptUFfMYMZH5QzP73RUxxCRI/ej5xayaO2OJj3msT2z+OH5Q6OWefbZZznnnHMoLCwkJyeH0tJSRo0aFXWfpUuX0qdPH7KysqKW27x5M506dSIlJfI1m5+fz5o1aw4rV1RUxNSpU7nyyitZvXo1paWlrF69mqSkJPLz8w+Wq73/r3/9az73uc9xxx13UFNTw6xZsw477qRJkzj33HOjxhiruNYozOw1M1tQx3Ih8P+A/2pg/4HAMUA+0As408xOq6usuz/k7sXuXpybW+cAiA3K6ZDOsF7ZzPi4/Kj2F5HWZfLkyVxxxRUAXHHFFQebn+r74zUeXaTXXXcd+fn5FBcX861vfYsxY8aQnJwcdZ8//OEPTJgwgdWrVzNhwgSuv/76T21//fXXmTRpEj/96U+bJMa41ijc/ey61pvZMKAfcKA2kQ/MNrMT3X19raIXAe+4+65gvxeBk4GZ8Yp5bGEu/zv9E7bv2U92u9ifXBSRo9fQX/7xsGXLFqZNm8b8+fMxM6qrqzEzfv7zn5OTk8PWrVsPK9+1a1cGDhzIqlWr2LFjR9RaRU5ODtu2baOqqoqUlBTKysro1avXYeVSUlKYMGHCwfdjxoyhsLCQzp07f6rpqPb+Dz/8MA888AAAl112GTfccMPBcvPmzeOGG27gxRdfJCcn5+g+nEOE0kfh7vPdPc/dC9y9ACgDRh6SJCDSyT3WzFLMLJVIR3adTU9NZWxhLtU1zqyluk1WpC17+umnufrqq1m5ciUrVqxg9erV9OvXj5kzZzJo0CDWrl3L4sWRr5uVK1cyd+5cRowYQWZmJtdffz233XbbwbuYysvLeeqppz51fDPjjDPOOHin1MMPP8yFF154WBwVFRXs3r0bgFdffZWUlBSOPfZYevToQVZWFu+88w7uziOPPHJw/549ezJjxgwApk2bxqBBgwBYtWoVF198MY8++iiFhYVN92G5e+gLsALoGrwuBv4UvE4GHiSSHBYBv4rleKNGjfKjtb+q2o/74Uv+vafnHvUxRKRhixYtCvX848aN8xdffPFT6x544AG/8cYb3d39zTff9JNOOsmLioq8uLjYX3nllYPl9u3b59/5znd8wIABPnToUD/xxBP9pZdeOuwcn3zyiZ9wwgk+YMAAv/TSS33v3r3u7v7ss8/63Xff7e7uy5cv98LCQh8yZIifddZZvmLFioP7v//++z506FDv37+/33LLLV5TU+Pu7jNnzvSRI0f68OHD/cQTT/SSkhJ3d7/++uu9U6dOXlRU5EVFRV7fd2Fdnz1Q4vV8p5p727sVtLi42BszcdHNj5fywaptzLrzTD05KhInixcv5phjjgk7jIRU12dvZqXuXlxX+bBvj22Rxhbmsm77XpZs3BV2KCIioVOiqMPphZG7pmZ8pLufRESUKOrQI7sdhd068MYSJQqReGqLTd8t3dF85koU9RhbmMu7y7ZQUVkVdigibVJGRgabN29WsmhGHsxHkZGRcUT7aeKieowtzGPizOW8u2wLZwzJCzsckTYnPz+fsrIyystVc29OB2a4OxJKFPUoLuhMu9RkZnxcrkQhEgepqalHNMuahEdNT/XISE3m5AE5Gs5DRBKeEkUUpw/qyvJNu1m5eXfYoYiIhEaJIoqxgyNNTm+oViEiCUyJIoqCnEz6dMlU85OIJDQliijMjLGFucz6ZDP7qqrDDkdEJBRKFA04vTCXispqSldsbbiwiEgbpETRgJMH5JCabGp+EpGEpUTRgA7pKRT37aJEISIJS4kiBuMG5/Lh+p2s274n7FBERJqdEkUMxgW3yWo0WRFJREoUMSjs1oEe2RlMV6IQkQSkRBEDM2Pc4FzeWrqJ/dU1YYcjItKslChiNLYwj537qihdqdtkRSSxKFHE6JSBOaQkmZqfRCThKFHEqGNGKsUFnZn+0cawQxERaVZKFEdg3OA8Ply/k/Xb94YdiohIs1GiOALjBucCMONj1SpEJHEoURyBwd060j0rg2kfKlGISOJQojgCZsbnh/dg2ocb2bhDzU8ikhiUKI7QV07uS1WN89i7q8IORUSkWShRHKG+Oe05a0gef313peaoEJGEEEqiMLN7zGyNmc0JlvPqKXeOmX1kZkvN7M7mjrM+147px6ZdlTw/d13YoYiIxF2YNYoJ7j4iWF44dKOZJQO/B84FjgWuNLNjmzvIupwyMIdBeR34y6zluHvY4YiIxFVLbno6EVjq7svcvRJ4Argw5JiASKf2tacUsGDNDg3pISJtXpiJ4lYzm2dmfzazznVs7wWsrvW+LFhXJzMbb2YlZlZSXh7/YTYuOr4XWRkp/OWtFXE/l4hImOKWKMzsNTNbUMdyIfAHYAAwAlgH/LKx53P3h9y92N2Lc3NzG3u4BmWmpXDlSX14ccE6Vm+piPv5RETCErdE4e5nu/txdSzPuvsGd6929xpgIpFmpkOtAXrXep8frGsxvjqmH8lJxp9mLgs7FBGRuAnrrqcetd5eBCyoo9j7wCAz62dmacAVwNTmiC9W3bMzuHBEL54sKWPr7sqwwxERiYuw+ih+ZmbzzWwecAZwO4CZ9TSzFwDcvQq4FXgZWAw86e4LQ4q3XuNP78+e/dU8+s7KsEMREYmLlDBO6u5X17N+LXBerfcvAIfdOtuSFHbryJlD8nh41grGn96fjNTksEMSEWlSLfn22FZj/On92by7kqdLy8IORUSkySlRNIGT+nWhKD+bP81cRnWNHsATkbZFiaIJmBnjTx/Ais0VvLZ4Q9jhiIg0KSWKJvK5od3o1akdk2YuDzsUEZEmpUTRRFKSk7ju1H68t2ILc1dvCzscEZEmo0TRhC4vzqdjegoT9QCeiLQhShRNqGNGajCsx3rKtmpYDxFpG5Qomtg1YwoA+D8NFigibYQSRRPr1akdnx/WgyfeX01FZVXY4YiINJoSRRx8eXRfdu2r4oX568MORUSk0ZQo4uCEgs4U5GTyZMnqhguLiLRwShRxYGZcVtyb95ZvYfmm3WGHIyLSKEoUcXLJyHySDJ4uVa1CRFo3JYo46Z6dwdjCXJ4uLdP4TyLSqilRxNHlxb3ZsGMfbyyJ/xzeIiLxokQRR2cd040u7dN4Sp3aItKKKVHEUVpKEl8c0YtXF21g5WZ1aotI66REEWfjT+9Pekoy3//HAtzVVyEirY8SRZx1z87ge+cM5s2lm3hm9pqwwxEROWJKFM3gP07qy6i+nfnxPxexede+sMMRETkiShTNICnJuP/iYezaV8V9zy8KOxwRkSOiRNFMBnXryE3jBjJlzlqmf7Qx7HBERGLWYKIws8tiWScNu+WMAfTPbc8PpizQyLIi0mrEUqO4K8Z10oD0lGTuv3g4ZVv38OvXloQdjohITFLq22Bm5wLnAb3M7De1NmUB+nP4KJ3YrwtXntiHP81cxgVFPTmuV3bYIYmIRBWtRrEWKAH2AqW1lqnA5+IfWtt157lDyOmQzp3PzKOquibscEREoqq3RuHuc4G5Zva4u6sG0YSy26Vyz/lDueWvs3lm9houP6F32CGJiNQrlj6KJWa27NAl7pG1cecN605R70488K8l7KuqDjscEZF6xZIoioETguU04DfAY405qZndY2ZrzGxOsJxXR5neZva6mS0ys4VmdltjztnSmBl3fLaQNdv28Lf3NWigiLRcDSYKd99ca1nj7r8GPt8E557g7iOC5YU6tlcB33b3Y4HRwC1mdmwTnLfFOHVgV07s14XfTlvKnkrVKkSkZYrlOYqRtZZiM7uRKH0bTcXd17n77OD1TmAx0Cve521OkVrFYMp37uPRd1aEHY6ISJ1i+cL/Za3XVcAK4PImOPetZvYVIndWfdvdt9ZX0MwKgOOBd6OUGQ+MB+jTp08ThNc8TuzXhdMLc/nD9E+48sQ+dMxIDTskEZFPsXgNfW1mrwHd69j0feAdYBPgwH1AD3e/rp7jdABmAD9x92diOXdxcbGXlJQcVdxh+GDVVi7631n85KLj+I+T+oYdjogkIDMrdffiurY1WKMwsxzgh8CpRL7Y3wTudffN0fZz97NjDG4i8Hw921KBvwOPx5okWqMRvTsxILc9z81dq0QhIi1OLHc9PQGUA5cAlwav/9aYk5pZj1pvLwIW1FHGgEnAYnf/VWPO19KZGecX9eTd5VvYsGNv2OGIiHxKLImih7vf5+7Lg+XHQLdGnvdnZjbfzOYBZwC3A5hZTzM7cAfUKcDVwJnRbqNtK84v6ok7/HPeurBDERH5lFg6s18xsyuAJ4P3lwIvN+ak7n51PevXEhlfCnd/E7DGnKc1GZDbgaE9s3hu3lquO7Vf2OGIiBwUS43ia8BfgX3B8gTwdTPbaWY74hlcojm/qCcfrNrG6i0VYYciInJQLA/cdXT3JHdPDZakYF1Hd89qjiATxeeHRbpunpu3NuRIRET+LZYH7v4VyzppvN5dMhnZpxPPzVU/hYi0HPUmCjPLMLMuQFcz62xmXYKlgDb2hHRLckFRTxav28HSjTvDDkVEBIheo/g6kfknhgCz+fd8FM8Cv4t/aInpvKD56eWFG0KOREQkot5E4e4PuHs/4A5371drKXJ3JYo4ycvKoCg/m38tVqIQkZYhlttjtwdjMn2Kuz8Sh3gEOHNIN379r4/ZvGsfOR3Sww5HRBJcLLfHnsCn56O4B7ggjjElvLOOycMdpn9UHnYoIiIN1yjc/Ru135tZJyLPUkicDO2ZRbesdKZ9uJFLRuWHHY6IJLhYahSH2g3o0eE4MjPOHJLHGx+Xs7+6JuxwRCTBxfIcxXNmNjVYngc+Av4R/9AS2xmD89i5r4r3V2wJOxQRSXCxdGb/otbrKmClu5fFKR4JnDKwK2kpSUxbvJExA7qGHY6IJLBYmp5WAR2DZZ2SRPNon57Cyf1zmPbhxrBDEZEEF+3J7CwzexJ4DbguWF4zs6fMTGM8NYOzjslj2abdLCvfFXYoIpLAotUofgMsAga5+8XufjEwAJiPnsxuFmcOyQPg1UV6+E5EwhMtUZzi7ve4+8HbbjziXuDk+Icm+Z0zKcrP1miyIhKqo7k9FhJoQqGwnV/UkwVrdvCJmp9EJCTREsUsM/uvYO7qg8zsbuDt+IYlB5xf1BMzmDpHtQoRCUe0RPENYBiw1Mz+HiyfAEXArc0SndAtK4PR/XJ4bu5a3D3scEQkAdX7HIW77wAuM7MBwLHB6kXu/kmzRCYHXTCiJ3c9M58Fa3YwLD877HBEJMHEMhXqJ+7+XLAoSYTg3OO6k5psTJ27JuxQRCQBHW1ntjSjTplpjC3M5bm566ipUfOTiDQvJYpW4oIRvVi/Yy/vaewnEWlm9fZRBPNl18vd9Y3VjM4+Jo/2ack8WbKa0f1zwg5HRBJItEEBSwGn7mcmHOgfl4ikTplpKVwyKp8n3lvNXeceQ25HzXwnIs0j2pzZ/dy9/yHzZR9YlCRCcM2YAiqra5j83qqwQxGRBBLLfBRmZl8OHrTDzPqY2YnxD00ONSC3A2MLc3nsnZVUVmlCIxFpHrF0Zv8vkbGdrgre7wR+35iTmtk9ZrbGzOYEy3lRyiab2QfBpEkJ79pTCti4cx8vLlgXdigikiBiSRQnufstwF4Ad98KpDXBuSe4+4hgeSFKuduAxU1wvjZh7KBc+ndtz1/eWhF2KCKSIGJJFPvNLJlIBzZmlgs0S7uHmeUDnwf+1Bznaw2SkoxrxhQwZ/U2Pli1NexwRCQBxJIofkNkjuw8M/sJ8Cbw301w7lvNbJ6Z/dnMOtdT5tfAd4khMZnZeDMrMbOS8vLyJgiv5bpkVD4d01P41asfa/wnEYm7WIbweJzIl/X/AOuAL7r7Uw3tZ2avmdmCOpYLgT8QmQRpRHDMX9ax/xeAje5eGsuFuPtD7l7s7sW5ubmx7NJqdUhP4Y7PDWbmkk38fbaG9RCR+Ir1gbuNwOTa2xp64M7dz44lADObCNTVUX0KcEHQ0Z0BZJnZY+7+5ViO29ZdPbovz89by33PL+L0wq7kdcwIOyQRaaOi1ShKgZLg33LgY2BJ8Dqmv/LrY2Y9ar29CFhwaBl3v8vd8929ALgCmKYk8W9JScb9lwxnz/5q7p6yQE1QIhI3DT5wB7wGnO/uXd09B/gC8Eojz/szM5tvZvOAM4DbAcysp5lFuwNKahmQ24H//EwhLy/cwHPzdLusiMSHNfSXqJnNd/dhDa1rSYqLi72kpCTsMJpFVXUNlz34NgvX7uChq0cxbnBe2CGJSCtkZqXuXlzXtljuelprZj8ws4Jg+T6geTlbiJTkJP5y7QkMyuvA+EdKef3DjWGHJCJtTCyJ4kogl8gtsv8A8oJ10kJ0ykzj8RtOorB7B77+aCnTPtwQdkgi0obEcnvsFne/DTgdOM3db9MQ4y1Pp8w0Hrv+JAZ378j4R0qZOleVPhFpGrEMCjjMzD4gcmfSQjMrNbPj4h+aHKlOmWk8/rWTGNm3M7c98QGPvbMy7JBEpA2IpenpQeA/3b2vu/cFvg08FN+w5GhlZaTyyHUncubgPH4wZQF/nKFpzkWkcWJJFO3d/fUDb9x9OtA+bhFJo2WkJvPHq0dxflFP7n/xQ54qWR12SCLSikWb4e6AZcFcFI8G778MLItfSNIUUpOT+OVlRWzdXcldz8yne3YGpw1q20ObiEh8xFKjuI7IXU/PBEtusE5auLSUJP73yyMZmNeBmx6bzeJ1O8IOSURaoVjuetrq7t9095HBclswJ4W0AlkZqfzlqyfQIT2FGx4uoaKyKuyQRKSViTYo4NRoO7r7BU0fjsRDj+x2/O6q47n0j2/zwL+WcNe5x4Qdkoi0ItH6KE4GVhMZNfZdwJolIomL4oIuXF6cz6SZy7n4+HwGd+8Ydkgi0kpEa3rqDvw/4DjgAeAzwCZ3n+HuM5ojOGlad557DB0yUjTarIgckWijx1a7+0vufg0wGlgKTDezW5stOmlSXdqncde5Q3hvxRaeLi0LOxwRaSWidmabWbqZXQw8BtzCv6dFlVbqslG9GdW3M//9wmI279oXdjgi0grUmyjM7BHgbWAk8CN3P8Hd73N3zb3ZiiUlGf9z8TB27avinucWhR2OiLQC0WoUXwYGAbcBs8xsR7DsNDPdkN+KFXbryDfPHMRzc9fy8sL1YYcjIi1ctD6KJHfvGCxZtZaO7p7VnEFK07tx3ACO7ZHFD6YsYFtFZdjhiEgLFsuT2dIGpSYn8fPLhrN1dyX3Pq8mKBGpnxJFAhvaM5ubxg3gmdlrmLV0U9jhiEgLpUSR4G45YyB9czL5wbMLqKyqCTscEWmBlCgSXEZqMvdcMJRl5buZOFODAovI4ZQohDMG53HO0O78dtoSyrZWhB2OiLQwShQCwH+dfyyGcc9UdWyLyKcpUQgAPTu141tnD+K1xRt48n3NiCci/6ZEIQfdcFp/ThmYw93PLmDh2u1hhyMiLYQShRyUnGQ8cMXxdMpM5ebHZ7Nj7/6wQxKRFkCJQj6la4d0fn/VSMq27uGOJ+dqOHIRUaKQwxUXdOGuc4fwyqINTHpzedjhiEjIQkkUZnaPma0xsznBcl495TqZ2dNm9qGZLTazk5s71kR1/an9+Myx3bj/xQ/5YJWmSBdJZGHWKCa4+4hgeaGeMg8AL7n7EKAIWNx84SU2M+MXlxbRLSuDW//6gQYOFElgLbbpycyygdOBSQDuXunu20INKsFkZ6by+/8Yycade7njKfVXiCSqMBPFrWY2z8z+bGad69jeDygH/mJmH5jZn8ysfX0HM7PxZlZiZiXl5eVxCzrRjOjdiTvPPYbXFm/kwTc0xIdIIopbojCz18xsQR3LhcAfgAHACGAd8Ms6DpFCZHa9P7j78cBu4M76zufuD7l7sbsX5+bmNvn1JLLrTingvGHd+dlLHzLrE40yK5JoUuJ1YHc/O5ZyZjYReL6OTWVAmbu/G7x/miiJQuLHzPjZpUV8tH4n35z8Ac9/4zS6Z2eEHZaINJOw7nrqUevtRcCCQ8u4+3pgtZkNDladBWggopB0SE/hwatHUVFZzc2Pl2pIcpEEElYfxc/MbL6ZzQPOAG4HMLOeZlb7DqhvAI8H5UYA/93skcpBA/M68rNLhzN71Tbuf/HDsMMRkWYSt6anaNz96nrWrwXOq/V+DlDcTGFJDL4wvCclK7by57eWc2K/LpxzXPewQxKROGuxt8dKy3XXeUMoys/mO0/PZdVmzV8h0tYpUcgRS09J5ndXjcSAW/46m31V1WGHJCJxpEQhR6V3l0x+cVkR89ds577ndY+BSFumRCFH7bNDuzP+9P489s4qpnywJuxwRCROlCikUb77ucGcWNCFu56Zz8cbdoYdjojEgRKFNEpKchK/u+p42qencONjpezaVxV2SCLSxJQopNHysjL47ZXHs2LTbn7yT/VXiLQ1ShTSJE4ekMP1p/Zj8nurKVmxJexwRKQJKVFIk/nW2YX0zM7g+/9YwP5qDfEh0lYoUUiTaZ+ewr0XHsdHG3YycaaGJBdpK5QopEmdfWw3Pje0Gw+8tkRPbYu0EUoU0uTuuWAoqclJ3PR4KTv37g87HBFpJCUKaXI9stvxu6uO56P1Oxn/SKmG+BBp5ZQoJC7GDc7j55cN5+1lm7n9b3OortF82yKtVSjDjEtiuOj4fDbvquTH/1xMkn3A/ZcMp0O6/suJtDb6rZW4uuG0/lRW1/CLlz9i4dod/O6q4xnaMzvssETkCKjpSeLu5nED+evXRlNRWcVFv5/FT/65iDc+LqeiUsN9iLQG5t722o6Li4u9pKQk7DDkEFt2V/KDKfN5ZeEGqmqc1GSjKL8TJ/Xvwuj+OYzq25nMNFVyRcJgZqXuXueMokoU0uwqKqsoWbGVWZ9s5p1lm5m/ZjvVNU5mWjLnHteDS0flc1K/LiQlGQDuTvnOfSzduIvlm3ezfvte1m3fy5791XzmmG58dmg3JRiRRlKikBZt174qSlZs4aUF63l+3jp27asiMy2Z9JQkUpKT2FtZzc5ao9ImGXTLyqDGnQ079tE+LZnzhvXgulP7cUyPrBCvRKT1UqKQVmNPZTUvL1zP3LJtVFX7wSaq/l3bMzCvI/1z25PXMZ2U5CRqapz3VmzhmdllPD9vHRWV1Zw1JI+bzxjAqL5dwr4UkVZFiULavG0VlTw8ayX/N2s5Wyv2M6pvZ752Wj8+c2x3koMmLBGpnxKFJIyKyir+9v5q/vzWclZv2UPfnEyuObmAy4rz6ZiRGnZ4Ii2WEoUknOoa55WF6/nTm8spXbmVDukpXDoqn6+P7U+P7HZhhyfS4ihRSEKbu3obf3lrOf+cv47kJOPrpw/g62P7604pkVqUKESA1VsquP+lD/nnvHV0z8rgx188jrOP7RZ2WCItQrREoSezJWH07pLJ768ayVM3nkzn9mnc8EgJ//PCYs3GJ9KAUBKFmd1jZmvMbE6wnFdPudvNbKGZLTCzyWaW0dyxSttzQkEX/nHzGK4e3ZcH31jGFQ+9w5pte8IOS6TFCrNGMcHdRwTLC4duNLNewDeBYnc/DkgGrmjuIKVtykhN5r4vHsdvrjyeD9ft4LO/msGkN5dTpdqFyGFaetNTCtDOzFKATGBtyPFIG3NBUU9e+tbpnNCvC/c9v4gLf/8W76/YEnZYIi1KmIniVjObZ2Z/NrPOh2509zXAL4BVwDpgu7u/Ut/BzGy8mZWYWUl5eXn8opY2p3eXTP5y7Qn8/qqRlO/cx2V/fJsvPfg2Mz4upy3e7CFypOJ215OZvQZ0r2PT94F3gE2AA/cBPdz9ukP27wz8HfgSsA14Cnja3R9r6Ny660mOVkVlFZPfW83EN5axfsdehvbM4vpT+/GF4T1JS2npFXCRo9eib481swLg+aAfovb6y4Bz3P364P1XgNHufnNDx1SikMbaV1XNlA/WMHHmcpZu3EVex3T+46S+XDyyF727ZIYdnkiTi5YoQnniyMx6uPu64O1FwII6iq0CRptZJrAHOAvQt780i/SUZL50Qh8uL+7NjI/LmfTmcia89jETXvuY0f27cMUJffjC8B6kJKuWIW1fKDUKM3sUGEGk6WkF8HV3X2dmPYE/uft5QbkfEWl6qgI+AG5w930NHV81ComH1VsqmPLBGv4+u4wVmyvo17U93zhzIBcU9VTCkFavRTc9xYMShcSTu/PKog38+rUlLF63g95d2nHx8fl88fhe9OvaPuzwRI6KEoVIHNTUOK8sWs8jb6/k7WWbcYdhvbI5c0geZwzJY3iv7IOz9B1qX1U12/fsZ+feKjqkp9ApM5X0lORmvgKRf1OiEImz9dv3MnXuGl6YH5l0yR0y05Jpl5pMSrKRbEZltbO/uoZ9VdXs3X/4g33t05LJy8ogr2M63bIy6NEpg/xO7ejZqR05HdLJbpdKVkYK2e1S1dQlTU6JQqQZbdldycwl5cxZvY3Kqhqqaw7M1JdEekoSaSlJkS/8zDQ6pqewu7KKrbsr2bJ7Pxt37mXjjn2s37GXddv3sL+67t/PjhmRWki3jhnkd25HfudMenZqR4/sSILp1amd5t+QI6JEIdIK1dQ45bv2sWbbHrburmTH3v1sr9jP9j1VbK2oZFtFJRt27KNsWwVrt+2luubTv8tdO6TTP7c9/bu2D/7twIC8DvTu3E41EjlMi7s9VkQalpRkdMvKoFtWw2NhVlXXUL5rH2u37WX99r2s2lLB8k27WFa+m1cWbWDL7sqDZdOSk+if255+XduT2zGdrh3SyemQRpfMNLq0TyM7M5Wqaqeyuoaq6sic5RmpyWSkJpPdLpXsdqmaXjbBKFGItAEpyUn0yG5X7+x92yoqWbZpN59s3MXS8l0s3bCLjzbsZNYnm9m+Z/8RncsMsjJS6ZGdQe8umfTunEmfLu3o3SWTPl0y6dI+jfbpKaSnJGGmhNIWKFGIJIBOmWmM7JPGyD6HDatGZVUNW3ZXsmV3JVsrKtm+Zz/JSUZaShKpSUnsr65h7/5q9uyP3Km1tWI/W3dXsnbbHlZu3s2bSzaxZ3/1YcdNskgCMyLJxfh00qidQyJlghIGSWYkBf+aGclJB9bZwX0P7G9YcHwOJqYDx/nUcRNA58w0nrzx5CY/rhKFSIJLS0mie3YG3bOPbroXd2fz7kpWbalg9ZYKtlXsZ3dlFRX7qtlfUwMeebL20H3+/frf22vcObCpusZxnOqaSH+N49R4pMyBHTw4lgfHOXRd5Nxtrx+2PllxuoFBiUJEGsXM6Noh0tdRV41FWj/d+iAiIlEpUYiISFRKFCIiEpUShYiIRKVEISIiUSlRiIhIVEoUIiISlRKFiIhE1SZHjzWzcmDlUe7eFdjUhOG0Brrmti/Rrhd0zUeqr7vn1rWhTSaKxjCzkvqG2m2rdM1tX6JdL+iam5KankREJColChERiUqJ4nAPhR1ACHTNbV+iXS/ompuM+ihERCQq1ShERCQqJQoREYkqYROFmZ1jZh+Z2VIzu7OO7elm9rdg+7tmVhBCmE0mhuv9TzNbZGbzzOxfZtY3jDibUkPXXKvcJWbmZtbqb6WM5ZrN7PLgZ73QzP7a3DE2tRj+b/cxs9fN7IPg//d5YcTZVMzsz2a20cwW1LPdzOw3wecxz8xGNvqk7p5wC5AMfAL0B9KAucCxh5S5Gfhj8PoK4G9hxx3n6z0DyAxe39SarzfWaw7KdQTeAN4BisOOuxl+zoOAD4DOwfu8sONuhmt+CLgpeH0ssCLsuBt5zacDI4EF9Ww/D3iRyJTho4F3G3vORK1RnAgsdfdl7l4JPAFceEiZC4GHg9dPA2eZWWudo73B63X31929Inj7DpDfzDE2tVh+xgD3AT8F9jZncHESyzV/Dfi9u28FcPeNzRxjU4vlmh3ICl5nA2ubMb4m5+5vAFuiFLkQeMQj3gE6mVmPxpwzURNFL2B1rfdlwbo6y7h7FbAdyGmW6JpeLNdb2/VE/iJpzRq85qBK3tvd/9mcgcVRLD/nQqDQzN4ys3fM7Jxmiy4+Yrnme4Avm1kZ8ALwjeYJLTRH+vveoJRGhSNtjpl9GSgGxoYdSzyZWRLwK+DakENpbilEmp/GEak1vmFmw9x9W5hBxdmVwP+5+y/N7GTgUTM7zt1rwg6stUjUGsUaoHet9/nBujrLmFkKkSrr5maJrunFcr2Y2dnA94EL3H1fM8UWLw1dc0fgOGC6ma0g0pY7tZV3aMfycy4Dprr7fndfDnxMJHG0VrFc8/XAkwDu/jaQQWTwvLYqpt/3I5GoieJ9YJCZ9TOzNCKd1VMPKTMVuCZ4fSkwzYOeolaowes1s+OBB4kkidbebg0NXLO7b3f3ru5e4O4FRPplLnD3knDCbRKx/L+eQqQ2gZl1JdIUtawZY2xqsVzzKuAsADM7hkiiKG/WKJvXVOArwd1Po4Ht7r6uMQdMyKYnd68ys1uBl4ncNfFnd19oZvcCJe4+FZhEpIq6lEjH0RXhRdw4MV7vz4EOwFNBn/0qd78gtKAbKcZrblNivOaXgc+a2SKgGviOu7fWmnKs1/xtYKKZ3U6kY/vaVvxHH2Y2mUiy7xr0u/wQSAVw9z8S6Yc5D1gKVABfbfQ5W/HnJSIizSBRm55ERCRGShQiIhKVEoWIiESlRCEiIlEpUYiISFRKFNIqmVm1mc0xswVm9pSZZYYQwzgzG1PPtnvM7I5D1q0Inl1oVmY2/WgfJDSz/zOzS5s6JmldlCiktdrj7iPc/TigErgxlp2Cp+ybyjigzkQh0pYoUUhbMBMYaGbtg7H63wvmHrgQwMyuNbOpZjYN+JeZdTCzv5jZ/GC8/kuCcp81s7fNbHZQS+kQrF9hZj8K1s83syEWmZ/kRuD2oGZzWqzBmlmBmS02s4nBnBCvmFm7YNt0M5tgZiVBmRPM7BkzW2JmP651jClmVhrsPz5YlxzUABYEcd5+yHmTgu0/Dsr+3MzeDz6DrwdlzMx+Z5H5HV4D8o7+xyJtRUI+mS1tR1BDOBd4icg4VdPc/Toz6wS8F3zZQWT8/uHuvsXMfkpkWINhwTE6B01CPwDOdvfdZvY94D+Be4P9N7n7SDO7GbjD3W8wsz8Cu9z9F0cR+iDgSnf/mpk9CVwCPBZsq3T3YjO7DXgWGEVkdIBPzGxC8CT1dcG1tAPeN7O/AwVAr6CWRfAZHJACPE5kDoOfBMllu7ufYGbpwFtm9gpwPDCYyLwN3YBFwJ+P4vqkDVGikNaqnZnNCV7PJDLkyizgglp9AxlAn+D1q+5+YAz/s6k1JIu7bzWzLxD5cnwrGMIkDXi71vmeCf4tBS6OIb76hjw4sH65ux+Iv5TIl/wBB4YXmQ8sPDBOj5ktIzLY22bgm2Z2UVCuN5HE8xHQ38x+C/wTeKXWMR8EnnT3nwTvPwsMr9X/kB0c43RgsrtXA2uDWpgkOCUKaa32uPuI2iss8g1/ibt/dMj6k4DdDRzPiCSTK+vZfmA03Wpi+73ZDBw6WUxHYFvwb+3ReauBdnWcq+aQcjVAipmNI5LsTnb3CjObDmQECa8I+ByRZrHLgeuCfWcBZ5jZL919L5Hr/Ya7v1w7QGvl04RKfKiPQtqSl4FvBAnjwIi4dXkVuOXAGzPrTGT02FPMbGCwrr2ZFTZwvp1EvvTr8gaR2k3H4HgXA3ODv9QbKxvYGiSJIUSGSD8wGmySu/+dSDNa7bmSJxEZLO7JoLnuZeAmM0sN9i00s/ZB3F8K+jB6EJkiVxKcEoW0JfcRGUVznpktDN7X5cdA56DTdy5whruXE5nEaLKZzSPS7DSkgfM9B1xUV2e2u88Dfge8GTSR3QjccHSXdZiXiNQsFgP3E0lyEJnFbHpwvseAuw6J6VdE5st+FPgTkf6H2Wa2gEjTVArwD2BJsO0RPt38JglKo8eKiEhUqlGIiEhUShQiIhKVEoWIiESlRCEiIlEpUYiISFRKFCIiEpUShYiIRPX/AVz9xHdanMR1AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "sp = benchmark.perturbation.SequentialPerturbation(explainer.model, explainer.masker, sort_order, perturbation)\n",
    "xs, ys, auc = sp.model_score(shap_values, s)\n",
    "sp.plot(xs, ys, auc)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}