mirrors / rasbt--deeplearning-models

项目文件夹

rasbt--deeplearning-models
文件
test
T
Sebastian Raschka 3a1265474c lower refresh rate to avoid crashing notebooks
2022-02-18 15:38:01 -08:00

1184 行
80 KiB
Plaintext
原始文件 永久链接 Blame 文件历史

{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"id": "dd556e23",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"torch : 1.10.1\n",
"pytorch_lightning: 1.6.0.dev0\n",
"torchmetrics : 0.6.2\n",
"matplotlib : 3.3.4\n",
"\n"
]
}
],
"source": [
"%load_ext watermark\n",
"%watermark -p torch,pytorch_lightning,torchmetrics,matplotlib"
]
},
{
"cell_type": "code",
"execution_count": 2,
"id": "f6f554b7",
"metadata": {},
"outputs": [],
"source": [
"%load_ext pycodestyle_magic\n",
"%flake8_on --ignore W291,W293,E703"
]
},
{
"cell_type": "markdown",
"id": "aa977d84",
"metadata": {},
"source": [
"<a href=\"https://pytorch.org\"><img src=\"https://raw.githubusercontent.com/pytorch/pytorch/master/docs/source/_static/img/pytorch-logo-dark.svg\" width=\"90\"/></a> &nbsp; &nbsp;&nbsp;&nbsp;<a href=\"https://www.pytorchlightning.ai\"><img src=\"https://raw.githubusercontent.com/PyTorchLightning/pytorch-lightning/master/docs/source/_static/images/logo.svg\" width=\"150\"/></a>\n",
"\n",
"# Model Zoo -- Basic Convolutional Network With Custom Weight Initialization (He) Trained on MNIST"
]
},
{
"cell_type": "markdown",
"id": "7ac40ea0",
"metadata": {},
"source": [
"## General settings and hyperparameters"
]
},
{
"cell_type": "markdown",
"id": "482e3be0",
"metadata": {},
"source": [
"- Here, we specify some general hyperparameter values and general settings\n",
"- Note that for small datatsets, it is not necessary and better not to use multiple workers as it can sometimes cause issues with too many open files in PyTorch. So, if you have problems with the data loader later, try setting `NUM_WORKERS = 0` instead."
]
},
{
"cell_type": "code",
"execution_count": 3,
"id": "0f7e034b",
"metadata": {},
"outputs": [],
"source": [
"BATCH_SIZE = 128\n",
"NUM_EPOCHS = 20\n",
"LEARNING_RATE = 0.005\n",
"NUM_WORKERS = 4"
]
},
{
"cell_type": "markdown",
"id": "12555a5d",
"metadata": {},
"source": [
"## Implementing a Neural Network using PyTorch Lightning's `LightningModule`"
]
},
{
"cell_type": "markdown",
"id": "e4a5614b",
"metadata": {},
"source": [
"- In this section, we set up the main model architecture using the `LightningModule` from PyTorch Lightning.\n",
"- We start with defining our neural network model in pure PyTorch, and then we use it in the `LightningModule` to get all the extra benefits that PyTorch Lightning provides."
]
},
{
"cell_type": "code",
"execution_count": 4,
"id": "9b73b854",
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"import torch.nn.functional as F\n",
"\n",
"\n",
"# Regular PyTorch Module\n",
"class PyTorchCNN(torch.nn.Module):\n",
" def __init__(self, num_classes):\n",
" super().__init__()\n",
"\n",
" # Calculate \"same\" padding:\n",
" # (w - k + 2*p)/s + 1 = o\n",
" # => p = (s(o-1) - w + k)/2\n",
" \n",
" self.features = torch.nn.Sequential(\n",
" # 28x28x1 => 28x28x8\n",
" torch.nn.Conv2d(\n",
" in_channels=1,\n",
" out_channels=8,\n",
" kernel_size=(3, 3),\n",
" stride=(1, 1),\n",
" padding=1), # (1(28-1) - 28 + 3) / 2 = 1\n",
" \n",
" # 28x28x8 => 14x14x8\n",
" torch.nn.MaxPool2d(\n",
" kernel_size=(2, 2),\n",
" stride=(2, 2),\n",
" padding=0), # (2(14-1) - 28 + 2) = 0\n",
" \n",
" # 14x14x8 => 14x14x16\n",
" torch.nn.Conv2d(\n",
" in_channels=8,\n",
" out_channels=16,\n",
" kernel_size=(3, 3),\n",
" stride=(1, 1),\n",
" padding=1), # (1(14-1) - 14 + 3) / 2 = 1 \n",
" \n",
" # 14x14x16 => 7x7x16 \n",
" torch.nn.MaxPool2d(\n",
" kernel_size=(2, 2),\n",
" stride=(2, 2),\n",
" padding=0) # (2(7-1) - 14 + 2) = 0\n",
" )\n",
"\n",
" self.output_layer = torch.nn.Linear(7*7*16, num_classes)\n",
" \n",
" ###############################################\n",
" # Reinitialize weights using He initialization\n",
" ###############################################\n",
" for m in self.modules():\n",
" if isinstance(m, torch.nn.Conv2d):\n",
" torch.nn.init.kaiming_normal_(m.weight.detach())\n",
" m.bias.detach().zero_()\n",
" elif isinstance(m, torch.nn.Linear):\n",
" torch.nn.init.kaiming_normal_(m.weight.detach())\n",
" m.bias.detach().zero_()\n",
"\n",
" def forward(self, x):\n",
" x = self.features(x)\n",
" x = torch.flatten(x, start_dim=1)\n",
" x = self.output_layer(x)\n",
" return x # x are the model's logits"
]
},
{
"cell_type": "code",
"execution_count": 5,
"id": "87f85fa6",
"metadata": {},
"outputs": [],
"source": [
"import pytorch_lightning as pl\n",
"import torchmetrics\n",
"\n",
"\n",
"# LightningModule that receives a PyTorch model as input\n",
"class LightningCNN(pl.LightningModule):\n",
" def __init__(self, model, learning_rate):\n",
" super().__init__()\n",
"\n",
" self.learning_rate = learning_rate\n",
" # The inherited PyTorch module\n",
" self.model = model\n",
"\n",
" # Save settings and hyperparameters to the log directory\n",
" # but skip the model parameters\n",
" self.save_hyperparameters(ignore=['model'])\n",
"\n",
" # Set up attributes for computing the accuracy\n",
" self.train_acc = torchmetrics.Accuracy()\n",
" self.valid_acc = torchmetrics.Accuracy()\n",
" self.test_acc = torchmetrics.Accuracy()\n",
" \n",
" # Defining the forward method is only necessary \n",
" # if you want to use a Trainer's .predict() method (optional)\n",
" def forward(self, x):\n",
" return self.model(x)\n",
" \n",
" # A common forward step to compute the loss and labels\n",
" # this is used for training, validation, and testing below\n",
" def _shared_step(self, batch):\n",
" features, true_labels = batch\n",
" logits = self(features)\n",
" loss = torch.nn.functional.cross_entropy(logits, true_labels)\n",
" predicted_labels = torch.argmax(logits, dim=1)\n",
"\n",
" return loss, true_labels, predicted_labels\n",
"\n",
" def training_step(self, batch, batch_idx):\n",
" loss, true_labels, predicted_labels = self._shared_step(batch)\n",
" self.log(\"train_loss\", loss)\n",
" self.train_acc(predicted_labels, true_labels)\n",
" self.log(\"train_acc\", self.train_acc, on_epoch=True, on_step=False)\n",
" return loss # this is passed to the optimzer for training\n",
"\n",
" def validation_step(self, batch, batch_idx):\n",
" loss, true_labels, predicted_labels = self._shared_step(batch)\n",
" self.log(\"valid_loss\", loss)\n",
" self.valid_acc(predicted_labels, true_labels)\n",
" self.log(\"valid_acc\", self.valid_acc,\n",
" on_epoch=True, on_step=False, prog_bar=True)\n",
"\n",
" def test_step(self, batch, batch_idx):\n",
" loss, true_labels, predicted_labels = self._shared_step(batch)\n",
" self.test_acc(predicted_labels, true_labels)\n",
" self.log(\"test_acc\", self.test_acc, on_epoch=True, on_step=False)\n",
"\n",
" def configure_optimizers(self):\n",
" optimizer = torch.optim.Adam(self.parameters(), lr=self.learning_rate)\n",
" return optimizer"
]
},
{
"cell_type": "markdown",
"id": "0cdfc165",
"metadata": {},
"source": [
"## Setting up the dataset"
]
},
{
"cell_type": "markdown",
"id": "381f2443",
"metadata": {},
"source": [
"- In this section, we are going to set up our dataset."
]
},
{
"cell_type": "markdown",
"id": "d9b43c84",
"metadata": {},
"source": [
"### Inspecting the dataset"
]
},
{
"cell_type": "code",
"execution_count": 6,
"id": "f50a28e3",
"metadata": {},
"outputs": [],
"source": [
"import torch\n",
"\n",
"from torchvision import datasets\n",
"from torchvision import transforms\n",
"from torch.utils.data import DataLoader\n",
"\n",
"\n",
"train_dataset = datasets.MNIST(root='./data', \n",
" train=True, \n",
" transform=transforms.ToTensor(),\n",
" download=True)\n",
"\n",
"train_loader = DataLoader(dataset=train_dataset, \n",
" batch_size=BATCH_SIZE, \n",
" num_workers=NUM_WORKERS,\n",
" drop_last=True,\n",
" shuffle=True)\n",
"\n",
"test_dataset = datasets.MNIST(root='./data', \n",
" train=False,\n",
" transform=transforms.ToTensor())\n",
"\n",
"test_loader = DataLoader(dataset=test_dataset, \n",
" batch_size=BATCH_SIZE,\n",
" num_workers=NUM_WORKERS,\n",
" drop_last=False,\n",
" shuffle=False)\n",
"\n",
"# Checking the dataset\n",
"all_train_labels = []\n",
"all_test_labels = []\n",
"\n",
"for images, labels in train_loader: \n",
" all_train_labels.append(labels)\n",
"all_train_labels = torch.cat(all_train_labels)\n",
" \n",
"for images, labels in test_loader: \n",
" all_test_labels.append(labels)\n",
"all_test_labels = torch.cat(all_test_labels)"
]
},
{
"cell_type": "code",
"execution_count": 7,
"id": "361b33ae",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
"Training label distribution: tensor([5919, 6733, 5949, 6119, 5833, 5414, 5907, 6250, 5842, 5938])\n",
"\n",
"Test labels: tensor([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])\n",
"Test label distribution: tensor([ 980, 1135, 1032, 1010, 982, 892, 958, 1028, 974, 1009])\n"
]
}
],
"source": [
"print('Training labels:', torch.unique(all_train_labels))\n",
"print('Training label distribution:', torch.bincount(all_train_labels))\n",
"\n",
"print('\\nTest labels:', torch.unique(all_test_labels))\n",
"print('Test label distribution:', torch.bincount(all_test_labels))"
]
},
{
"cell_type": "markdown",
"id": "31a45979",
"metadata": {},
"source": [
"### Performance baseline"
]
},
{
"cell_type": "markdown",
"id": "a1f29a77",
"metadata": {},
"source": [
"- Especially for imbalanced datasets, it's quite useful to compute a performance baseline.\n",
"- In classification contexts, a useful baseline is to compute the accuracy for a scenario where the model always predicts the majority class -- you want your model to be better than that!"
]
},
{
"cell_type": "code",
"execution_count": 8,
"id": "5ec92aba",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Baseline ACC: 11.35%\n"
]
}
],
"source": [
"majority_prediction = torch.argmax(torch.bincount(all_test_labels))\n",
"baseline_acc = torch.mean((all_test_labels == majority_prediction).float())\n",
"print(f'Baseline ACC: {baseline_acc*100:.2f}%')"
]
},
{
"cell_type": "markdown",
"id": "063ad85a",
"metadata": {},
"source": [
"### Setting up a `DataModule`"
]
},
{
"cell_type": "markdown",
"id": "42626ebd",
"metadata": {},
"source": [
"- There are three main ways we can prepare the dataset for Lightning. We can\n",
" 1. make the dataset part of the model;\n",
" 2. set up the data loaders as usual and feed them to the fit method of a Lightning Trainer -- the Trainer is introduced in the next subsection;\n",
" 3. create a LightningDataModule.\n",
"- Here, we are going to use approach 3, which is the most organized approach. The `LightningDataModule` consists of several self-explanatory methods as we can see below:\n"
]
},
{
"cell_type": "code",
"execution_count": 9,
"id": "c068349e",
"metadata": {},
"outputs": [],
"source": [
"import os\n",
"\n",
"from torch.utils.data.dataset import random_split\n",
"from torch.utils.data import DataLoader\n",
"\n",
"\n",
"class DataModule(pl.LightningDataModule):\n",
" def __init__(self, data_path='./'):\n",
" super().__init__()\n",
" self.data_path = data_path\n",
" \n",
" def prepare_data(self):\n",
" datasets.MNIST(root=self.data_path,\n",
" download=True)\n",
" return\n",
"\n",
" def setup(self, stage=None):\n",
" # Note transforms.ToTensor() scales input images\n",
" # to 0-1 range\n",
" train = datasets.MNIST(root=self.data_path, \n",
" train=True, \n",
" transform=transforms.ToTensor(),\n",
" download=False)\n",
"\n",
" self.test = datasets.MNIST(root=self.data_path, \n",
" train=False, \n",
" transform=transforms.ToTensor(),\n",
" download=False)\n",
"\n",
" self.train, self.valid = random_split(train, lengths=[55000, 5000])\n",
"\n",
" def train_dataloader(self):\n",
" train_loader = DataLoader(dataset=self.train, \n",
" batch_size=BATCH_SIZE, \n",
" drop_last=True,\n",
" shuffle=True,\n",
" num_workers=NUM_WORKERS)\n",
" return train_loader\n",
"\n",
" def val_dataloader(self):\n",
" valid_loader = DataLoader(dataset=self.valid, \n",
" batch_size=BATCH_SIZE, \n",
" drop_last=False,\n",
" shuffle=False,\n",
" num_workers=NUM_WORKERS)\n",
" return valid_loader\n",
"\n",
" def test_dataloader(self):\n",
" test_loader = DataLoader(dataset=self.test, \n",
" batch_size=BATCH_SIZE, \n",
" drop_last=False,\n",
" shuffle=False,\n",
" num_workers=NUM_WORKERS)\n",
" return test_loader"
]
},
{
"cell_type": "markdown",
"id": "4b8b5862",
"metadata": {},
"source": [
"- Note that the `prepare_data` method is usually used for steps that only need to be executed once, for example, downloading the dataset; the `setup` method defines the the dataset loading -- if you run your code in a distributed setting, this will be called on each node / GPU. \n",
"- Next, lets initialize the `DataModule`; we use a random seed for reproducibility (so that the data set is shuffled the same way when we re-execute this code):"
]
},
{
"cell_type": "code",
"execution_count": 10,
"id": "5cf11291",
"metadata": {},
"outputs": [],
"source": [
"torch.manual_seed(1) \n",
"data_module = DataModule(data_path='./data')"
]
},
{
"cell_type": "markdown",
"id": "6c2f8da1",
"metadata": {},
"source": [
"## Training the model using the PyTorch Lightning Trainer class"
]
},
{
"cell_type": "markdown",
"id": "e45c3c4a",
"metadata": {},
"source": [
"- Next, we initialize our model.\n",
"- Also, we define a call back so that we can obtain the model with the best validation set performance after training.\n",
"- PyTorch Lightning offers [many advanced logging services](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) like Weights & Biases. Here, we will keep things simple and use the `CSVLogger`:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"id": "1ba71aa8",
"metadata": {},
"outputs": [],
"source": [
"from pytorch_lightning.callbacks import ModelCheckpoint\n",
"from pytorch_lightning.loggers import CSVLogger\n",
"\n",
"\n",
"pytorch_model = PyTorchCNN(num_classes=10)\n",
"\n",
"lightning_model = LightningCNN(\n",
" pytorch_model, learning_rate=LEARNING_RATE)\n",
"\n",
"callbacks = [ModelCheckpoint(\n",
" save_top_k=1, mode='max', monitor=\"valid_acc\")] # save top 1 model \n",
"logger = CSVLogger(save_dir=\"logs/\", name=\"my-model\")"
]
},
{
"cell_type": "markdown",
"id": "59445fc4",
"metadata": {},
"source": [
"- Now it's time to train our model:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"id": "1d7904fb",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"GPU available: True, used: True\n",
"TPU available: False, using: 0 TPU cores\n",
"IPU available: False, using: 0 IPUs\n",
"LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
"\n",
" | Name | Type | Params\n",
"-----------------------------------------\n",
"0 | model | PyTorchCNN | 9.1 K \n",
"1 | train_acc | Accuracy | 0 \n",
"2 | valid_acc | Accuracy | 0 \n",
"3 | test_acc | Accuracy | 0 \n",
"-----------------------------------------\n",
"9.1 K Trainable params\n",
"0 Non-trainable params\n",
"9.1 K Total params\n",
"0.036 Total estimated model params size (MB)\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validation sanity check: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c681a1170051487b8975985933d57f9a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Training: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Validating: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Training took 1.89 min in total.\n"
]
}
],
"source": [
"import time\n",
"\n",
"\n",
"trainer = pl.Trainer(\n",
" max_epochs=NUM_EPOCHS,\n",
" callbacks=callbacks,\n",
" progress_bar_refresh_rate=50, # recommended for notebooks\n",
" accelerator=\"auto\", # Uses GPUs or TPUs if available\n",
" devices=\"auto\", # Uses all available GPUs/TPUs if applicable\n",
" logger=logger,\n",
" deterministic=True,\n",
" log_every_n_steps=100)\n",
"\n",
"start_time = time.time()\n",
"trainer.fit(model=lightning_model, datamodule=data_module)\n",
"\n",
"runtime = (time.time() - start_time)/60\n",
"print(f\"Training took {runtime:.2f} min in total.\")"
]
},
{
"cell_type": "markdown",
"id": "2d25c4ce",
"metadata": {},
"source": [
"## Evaluating the model"
]
},
{
"cell_type": "markdown",
"id": "54e93b6c",
"metadata": {},
"source": [
"- After training, let's plot our training ACC and validation ACC using pandas, which, in turn, uses matplotlib for plotting (you may want to consider a [more advanced logger](https://pytorch-lightning.readthedocs.io/en/latest/extensions/logging.html) that does that for you):"
]
},
{
"cell_type": "code",
"execution_count": 13,
"id": "cff635af",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"<AxesSubplot:xlabel='Epoch', ylabel='ACC'>"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAABMQElEQVR4nO3dd3hUVfrA8e9JJx0ICSEJLdQQQicUQRAVBATsqGBfVtfGuvjTVddVtqir61pXFuvaFrHQQVRIBJQaINQEQmiBQEILCenJ+f1xJxjiJEwyc2cm4f08zzzM3DL3zSWZd+6557xHaa0RQgghavJwdQBCCCHckyQIIYQQVkmCEEIIYZUkCCGEEFZJghBCCGGVl6sDcKSwsDDdvn37Bu177tw5AgICHBuQA0l89pH47CPx2ced40tJSTmhtW5ldaXWusk8+vXrpxsqKSmpwfs6g8RnH4nPPhKffdw5PmCTruUzVZqYhBBCWCUJQgghhFWSIIQQQlglCUIIIYRVkiCEEEJYJQlCCCGEVZIghBBCWCUJQgghGrND6+Gn1015a0kQQgjRGFWUwcq/wYdjYNOHUFLg8EM0qVIbQghxSTi5D76ZBkc2Qa/b4JqXwDfQ4YeRBCGEEI2F1rDlU1j2BHh6wY0fQvz1ph1OEoQQQjQGhadg0SOwexG0HwbXzYKQaFMPKQlCCCHc3b4kmP8AnDsBV82EwQ+Dh/m3kCVBCCGEuyovgRUzYe1bENYFbvsCIns57fCSIIQQwh3l7Iav74PjO2DAfXDVX8DH36khSIIQQgh3ojVsmA3f/Ql8g+DWL6DrGJeEIglCCCHcRf5xWPA7yPgBOl8NE9+GwHCXhWPqXQ6l1BilVLpSKkMp9aSV9d2UUmuVUiVKqRlW1nsqpbYopRabGacQQrhc2lJ4ZzAcWANjX4Hb5ro0OYCJVxBKKU/gbeAqIAvYqJRaqLXeVW2zU8AjwKRa3uZRYDcQbFacQgjhUqXnYPnTkPIhtO4J178H4d1cHRVg7hXEQCBDa52ptS4F5gATq2+gtc7RWm8EymrurJSKBsYB75kYoxBCuNY30yDlIxjyCNy3wm2SA4Ay5qw24Y2VuhEYo7W+z/J6KpCotX7IyrbPAQVa61eqLfsKeAEIAmZorcfXcpxpwDSAiIiIfnPmzGlQvAUFBQQGOn6ouqNIfPaR+Owj8dmntvian9pCr23PkdlhCofa3eT8wICRI0emaK37W1tn5k1qZWWZTdlIKTUeyNFapyilRtS1rdZ6NjAboH///nrEiDo3r1VycjIN3dcZJD77SHz2kfjsYzW+8lJ4Zwa06EjH21+lo5evS2Kri5kJIguIqfY6Gjhq475DgQlKqbGAHxCslPpUaz3FwTEKIYRrbPgPnNxr3Ix2w+QA5t6D2Ah0Vkp1UEr5AJOBhbbsqLX+o9Y6Wmvd3rLfSrOSQ3lFJVPfX8/KQ7+6DSKEEObIPwbJLxldWbuMdnU0tTLtCkJrXa6UeghYDngCH2itdyql7resn6WUag1swuilVKmUmg7Eaa3PmhVXTV6eHuw9XkBFYKWzDimEuNT98DyUF8OYF10dSZ1MHSintV4KLK2xbFa158cwmp7qeo9kINmE8M6LDQ/gaO5pMw8hhBCGwxsg9XMYOh1axro6mjrJjHJAp1aBZBdUYlaPLiGEAKCyEpY+DkGRMPxxV0dzUZIggNjwQIor4PjZEleHIoRoyrZ8AtlbjZLdJswA52iSIDCuIAD25Tp+TlchhACg6AyseB7aDoaerhnzUF+SIDCuIAAyciRBCCFMkvwCFJ2Ga/4BytowMfcj1VyB8CBf/DzlCkIIYY6AgoOQ8i70uwsiE1wdjs3kCgJQStEm0EOuIIQQjqc1nTLeNeZ2uOJPro6mXiRBWEQGeMgVhBDC8XYtoPmZ7XDFM+DfwtXR1IskCIvIQMXxsyXkF8uIaiGEg5QWwnfPUBDQAfrf4+po6k0ShEVkgHEq9uWec3EkQogm46fXIO8wezv/Bjw8XR1NvUmCsDifIOQ+hBDCEU4fgDWvQfyN5IX2cHU0DSIJwiLcX+HlociQ+xBCCEdY/rRx1XDVTFdH0mCSICy8PBTtWvrLFYQQwn77VkLaYhg+A0KiXB1Ng0mCqKZTeKD0ZBJC2Ke8FJY9Ac07wOBfTaDZqEiCqCa2VSAHTxZSViGlv4UQDbRhNpzYY5TydtOJgGwlCaKa2FaBlFdqDp4sdHUoQojGKP84JL9oTATUdYyro7GbJIhqOklNJiGEPVZYJgIa/YKrI3EISRDVdGwVAEhNJiFEAxzeCFs/g8EPQlgnV0fjEJIgqgny86Z1sJ8kCCFE/VRWwrLHIbC10XOpiZBqrjXEhgdIV1chLhVaG1OA5qZBYLjlEQEB4eDlY/v7bP0Ujm6B6y1F+ZoIUxOEUmoM8DrgCbyntX6xxvpuwIdAX+BprfUrluUxwMdAa6ASmK21ft3MWKvEtgrkm81H0FqjGknNdiFEPZWXwq75sO7fxge7Nc2aG8miKmlc8LzaMuUBPzwPMYMazURAtjItQSilPIG3gauALGCjUmqh1npXtc1OAY8Ak2rsXg78QWu9WSkVBKQopb6vsa8pOoUHUlBSzvGzJbQO8TP7cEIIZyo8BSkfwoZ3IT8bWnaGca9Cp1FQeBIKcqDgeLV/Lc+zNho9lMqLanljBVO/aTQTAdnKzCuIgUCG1joTQCk1B5gInP+Q11rnADlKqXHVd9RaZwPZluf5SqndQFT1fc0SW236UUkQQjQRuemw7h1InWN8yHccCRPehNhR4GG5Fdu8fd3voTWUFvw6eRQch5adILKX6T+GsymttTlvrNSNwBit9X2W11OBRK31r4YWKqWeAwqqmphqrGsPrALitdZnrayfBkwDiIiI6DdnzpwGxVtQUEBgYCCniyv5fXIRU7r7cGU77wa9lxmq4nNXEp99JD77WI1Pa5qf3kJ01iJantpMpfLmWOsRHIm6lnOB7Vwfn5sYOXJkita6v7V1Zl5BWLvWqlc2UkoFAl8D060lBwCt9WxgNkD//v31iBEj6hmmITk5mREjRqC15k9rv8MjNJIRI+Ib9F5mqIrPXUl89pH47HNBfGVFxpXC+lmWm88RMPIZPPrfTZuAMNq4Or5GxMwEkQXEVHsdDRy1dWellDdGcvhMa/2Ng2Or67jEtgqQrq5CNDZns2Hje7DpAyg6Ba0T4Lr/QI/rGn3JC1cxM0FsBDorpToAR4DJwG227KiM7kPvA7u11q+aF6J1seGB/Jxx0tmHFUI0xIkMuu3+F6z6CSrLods4GPQAtBva5G4aO5tpCUJrXa6UeghYjtHN9QOt9U6l1P2W9bOUUq2BTUAwUKmUmg7EAQnAVGC7Umqr5S2f0lovNSve6qq6uuYXlxHk5z73IYQQNZzKhA/HEFacDwPuhcTfQouOro6qyTB1HITlA31pjWWzqj0/htH0VNMarN/DcIqqnkyZuefoFRPqqjCEEHU5dwI+vQEqy9nc958MvGaKqyNqcqTUhhVStE8IN1d6Dj6/Gc4ehVu/oDDA2vdMYS9JEFa0a+mPl4eSG9VCuKOKcvjqHmME9A3vQ9tEV0fUZEktJiu8PT2M6UclQQjhXrSGJY/Bnm+NEdDdx7s6oiZNriBqEdsqUJqYhHA3P/4DNv8Xhs0wbkoLU0mCqEVsuEw/KoRb2fwxJP8det0GVzzj6mguCZIgatFJph8Vwn3sWQ6Lphu1kya8IeMbnEQSRC1iw38p2ieEcKGsFPjyLmjdE27+GDxlbJKzSIKoRaxMPyqE653cB5/fBAGt4PYvwdc9C941VZIgahHk501EsK/cqBZNX1mxMZ6goszVkVyoINcYCKc1TPnGmKRHOJV0c61DbKtA9uWec3UYQphDa9jxNSx/ypjTAMC/pdWZ0yKOnYR9+pdZ1Jo1/2UeBTOUFBhXDvnH4M5FENbJvGOJWkmCqEOncJl+VDRRuemw5A9wYDW06QPDH7fMqFY1CU4OHFpnvC4vpjtA2r9+2d/Dy5i3uSqRtOoKcRMhqp/9N5Aryox7DtmpMPlziBlg3/uJBpMEUYfYVsb0ozn5JUQEy+xyogkoPQerXoGf3wQff2OwWb+7wMPT+vZaQ0k+61cuIjGufY3pOHPgXI4xdee+FfDzGxAcbSSKuIkQPaD+Vxlaw+LpkPE9jH8Nul5j388r7CIJog5VNZn25RRIghCNX9oSWPYE5B02xhJcNRMCW9W9j1LgF0yRfxS0H1r7dkWnIX0Z7FoAG9+FdW9DUCR0nwA9JkFMYu1JqLrkF2DLp3D5E9D/7nr9eMLxJEHUoaqqa0ZuAUM6hbk4GiEa6PQBIzHs+RbC4+DuZdBuiGOP0aw59L7NeBTnGeMWdi2AlI9gw3+MZqju1xpXFm2HgKeVj55NH8KPL0GfKTDij46NTzSIJAiAxY8RdcYD8jpByC9VISOCfQn09WKf9GQSjVF5idHss+oVUJ5w9V8h8X7zxxH4hUDCzcajJB/2fmckiy2fGTO++YcZNZTiJkH7YUaySF9m1FjqfLXRtCT3/NyCJIjSQjj4E51z0+Bf70Jkb+OXt9u1qFZdiW0VQIaMhRCNzb4kWDoDTmYY39pHvwAhUc6PwzcI4m8wHqXnYO/3RrLY9qVxddGshZEUdi0w/vZu+kgGwrkRSRA+/vDgetYv/ZTE4FzYvRhW/tV4tOzEIwzk82MJUDnQ3G59QjjC2Wyj2+rOb4yZ1aZ8DZ2udHVUBp8A435Ej0lQVgQZK4zEkLYEgiPhtrnGNsJtSIKwKPKPhsumwGW/NwYNpS2BtMWM2P8Fo/TnVL76Gh7dxxvz3bYfJt9yhHupKDfa+pNegIpSGPEUDH0UvN20c4V3M+NKvft4KC8FXem+sV7CTP1KrJQao5RKV0plKKWetLK+m1JqrVKqRCk1oz77miq4DQz8DdyxgKRr1zK99HecDesNWz+HT66Dl2Phm2mwa6Fx2SyEKx1YA7MvN64c2ibCg+tgxBON5wPXy6fxxHqJMe0KQinlCbwNXAVkARuVUgu11ruqbXYKeASY1IB9naJ9dBvmV17G8IQHuf62FpCZZDRD7VkG274ALz+jwuTIp6B1vLPDE5eyA2sg+UVjsFtwFNz8idFTSG7wCgcxs4lpIJChtc4EUErNASYC5z/ktdY5QI5Salx993WWdi0D8PJQRk0mn2ijianbOOOS/uBPRlPU9i+NkZ8P/Gx8GxLCTPtXG91BD6w2RjOP/jv0u9u4nyaEA5mZIKKAw9VeZwG2Th5rz74O5e3pQVtr0496ekHHy41HpyuNujHr/g2XTXdFmKKp09pICMkvGl9MAiOMnkn97pLEIExjZoKwdp2rHb2vUmoaMA0gIiKC5ORkGw9xoYKCglr3DVHFbD9QWMd7+xDfcgDNV77A+nMxlPq2bFAMDY3PHUh89qk1Pq0JPbOd9gf+R2jeLkp8mnOo031kR15NZYkv/LzBtfG5CYnPHGYmiCwgptrraOCoo/fVWs8GZgP0799fjxgxot6BAiQnJ1PbvuuL03h3VSZDhw3H27OW+/oJ7eDtRIbkL4HRHzQohobG5w4kPvv8Kj6tITPZaEo6tNYoW3HNP/Dteyedvf3o7Or43IzEZw4zezFtBDorpToopXyAycBCJ+zrcLGW6UcPnapj+tEWHYwusju+hv2rnBecaFq0hn0r4YPR8MkkOH0Qxr4Cj2yFxN9Kbx/hVKZdQWity5VSDwHLAU/gA631TqXU/Zb1s5RSrYFNQDBQqZSaDsRprc9a29esWC+metG+qvpMVl02HVI/h6X/B/evlrESwnZaQ8YPkPwSZG0weiWN+yf0mQpevq6OTlyiTB0op7VeCiytsWxWtefHMJqPbNrXVTpaph/NyC3g6ro29G4GY16EObfBhtkw+EGnxCcasYpy2PsdfbY8Bz+mG+Wyx71qFKyTxCBcTEZS2yDYz5vwIF/25dgwKK7rWKNXU9ILEH8jBEWYH6Cwz57lxtwG3caBfwvnHPPkPqOs9dbPoeAYvr6tjCJ1vW+XrtLCbUiCsFGn8EDbivYpBdf8A/49CL5/Fq7/j/nBiYaprDD+j9a+Zbxe9KjRbTlukjHgzNHJoqwIdi+CzR8bXVaVh1Goru8drD/qy+X93aRmkhAWkiBsFNsqkPlbbJx+tGUsDH4I1lhm62o32CkxinooyYev7zPmSBg4DXpNNkqn7JoPix6Bxb93XLLI3gZbPjFG3hfnQfP2cMWfjLkTgtsAoI8lO+CHEsKxJEHYqFN4IPkl5eTmlxBuy+xyw2fAtrlGyeVpP1qfIEW4xplD8L9bIWe30UNo4G+M5VH94MrnjLmQd82HnfManiyK84wR9ps/geyt4OkLcROg7x3Q7jKpDCwaBfnUstH52eVyCmxLED4BMPpv8OWdsOkDSJxmcoTCJoc3wpxbjQqiU76C2CsuXK8UtOltPEb9uVqymP9LsugwHHpcB93GQ0C1QZFaG2MWNn9sbF9eBBHxRpNjz5ucd39DCAeRBGGj2HCjJ9O++kw/GjcROlwOSX81PlAuNv+vMNe2L2HBg0azzl1zoVWXurevmSyObTOuKn6VLCZB0RmjGelkBvgEGU1Wfe+ANn2keJ5otCRB2Kh1sB8BPp5G0T5bKQVjX4Z3hsCK52Di26bFJ+pQWUn7/Z9B8lxoNxRu+bT+3+aVgshexuN8sphvuWfxqLFN28Ew7A/GFwOZ+EY0AZIgbKSUIjY8kH259Zz/oVVXGPQA/Pwm9L0LYgaYEp+oRWkhzH+A9gfnG2MLxv3L/m6kFySLZyFnl1H2vWWsQ0IWwl3InbJ66NQq8NdVXW1x+RNGLZ2lM4yulcI5zmbDR2Nh1wL2dbwLJrzl+DEGSkFED0kOokmSBFEPseGBZOcVU1BSXr8dfYPg6r8avVk2/9eU2EQNR7fCu1dA7h6Y/DmH214n9wKEqCdJEPUQaym5kdmQq4j4G4zujStmQuEpB0cmLrBrIXx4jTEQ7d7l0G2sqyMSolGSBFEPVUX76nWjuopSMPYfUHzWSBLC8bSG1f+EuVMhPA5+sxJa93R1VEI0WpIg6qFtiwA8PVTD7kOA0VY9cBqkfARHNjs0tkteeQnMf8BIvvE3wl2LpQ6WEHaSBFEPPl4etGvhb1vRvtqM/CMEtIKlj0NlpeOCu5SdOwH/nQCp/4MRT8EN7xmVdYUQdpFurvUUa2vRvtr4hcBVM2H+/bD1M+g71XHBXUqKTsPe7yFtMez9AXQF3PghxF/v6siEaDIkQdRTbKtAktNzKKuorH360YtJuAVSPoQfnoPu46FZc4fG2GTlZUH6MiMpHFgDleUQGAEJN8OAe+V+gxAOJgminjqFB1JWYUw/WufscnXx8DCKxM2+HJL+boy2ro/KSji1z6gTlL3VKDrXcaQxQVFT6sqptfGzpS2B9CVwdIuxPKwLDHnYqIXUpq8UvhPCJJIg6qmqq+tFpx+9mMgE6H8PbHzPmFYyMsH6dhVlkJtO6+wVsHSpkRSObYcyy30QTx9jesqMH4wP0IlvNe7298oKOLzBuEpIWwKn9xvLowcYlVa7jrt4DSUhhEPYlCCUUgFAkda6UinVBegGLNNal5kanRuKrZqfur4lN6y54hmj+NvSx+Geb6G8GI7vMq4Kjm0zksHxXVBRQjcA7wCjGaXPlF9KPbTqCh5extwTK2YaVxaTPz8/z0CjUFYEmT8aSSF9GRSeMBJfh+Ew9BFjlr6g1q6OUohLjq1XEKuAYUqp5sAKYBNwC3B7XTsppcYArwOewHta6xdrrFeW9WOBQuAurfVmy7rfA/cBGtgO3K21LrYxXtNUTT/aoLEQNTVrbnwrXvgwvNHHmKdAW0px+IUaCSBxGkT2Zv3hYhLH3Aoentbfa9gfoFU3+GYazB5pJInofvbHaIayYsjaaNxHOLDGeF5RAr7Bxgxr3cYZ07b6Bbs6UiEuabYmCKW1LlRK3Qu8qbX+h1JqS507KOUJvA1cBWQBG5VSC7XWu6ptdg3Q2fJIBN4BEpVSUcAjQJzWukgpNReYDHxUj5/NNLENrclkTe8pxodk0WljtHXVlUFo2wvuJxSdTK49OVTpNg7u/Q7+N9kYSTzxbUi4yTFx2qO2hKA8oHWCMWFP7EhoP1zmYxbCjdicIJRSgzGuGO61cd+BQIbWOtPyBnOAiUD1BDER+FhrrYF1SqlQpVRktfdvppQqA/yBozbGarpO4YHM32rj9KMX4+EB1892TGBgDMb7TRLMvQO+uQ9ydsIVzzr3Rm5ZkZWEUHphQmg/DNoOgmahzotLCFEvtiaI6cAfgXla651KqY5A0kX2iQIOV3udhXGVcLFtorTWm5RSrwCHgCLgO631d9YOopSaBkwDiIiIIDk52aYfqKaCggKb963MKyO/uJwFy5MI9XPOB2994gNQ7X5P59IA2qz5Fyd2/8Tu7r+nwsvflNg8KkrxPb6FAx9+TuiZHQSfTcdDl6PxID+oI2fajOVMaE/yQrpT4WWZJyEb416Lk9T3/DmbxGcfic8kWut6PTBGXwfbsN1NGPcdql5PxWieqr7NEuCyaq9XAP2A5sBKoBXgDcwHplzsmP369dMNlZSUZPO2q/bk6HZPLNY/ZeQ2+Hj1VZ/4zqus1HrdLK2fa67124O0PrXfsUEd26n1khla/z1a6z8Ha/1cqNb/GaH18qe1Tv9W66Izjj2eHRp0/pxI4rOPxNdwwCZdy2eqrb2YPgfuByqAFCBEKfWq1rquDvxZQEy119H8upmotm2uBPZrrXMtx/8GGAJ8aku8Zqsq2rcvp4AhsTZOP+oKSkHibyGsM3x5l3Hz+pZPoP1lDX/PsmLYvdCYZ/vQWvD0hR6T2Ka7kDDuN8ZIcSFEk2Br+0ic1vosMAlYCrTFuCKoy0ags1Kqg1LKB+Mm88Ia2ywE7lCGQUCe1jobo2lpkFLK39LTaRSw28ZYTVc1/ahDuro6Q+wVcN9K8G8JH0+ETR/W/z1O7oPvnoFXu8M3v4GC48YcF4/thutnc6plf0kOQjQxtt6D8FZKeWMkiLe01mVKKV3XDlrrcqXUQ8ByjG6uH2jj/sX9lvWzMJLNWCADo5vr3ZZ165VSXwGbgXJgC+DAO7n2+WX6UQf1ZHKGsE5w3w/w9b2weLoxTeboF8Czjl+BijJIX2pcLWQmG+Mtuo0zBvi1Hy4jmIVo4mxNEP8BDgCpwCqlVDvg7MV20lovxUgC1ZfNqvZcAw/Wsu+fgT/bGJ/TxbYKZF3mSVeHUT/NQuG2ufD9s7D2LchNh5s+Av8WF2535rAx893mj40rhZAYY1Bfn6kyYE2IS4hNCUJr/QbwRrVFB5VSI80JqXGIbRXAvC1HKCgpJ9C3EVUs8fCE0X+D8O6waDq8Nwpu/cKYUzljhXG1sHe5UQepy2jjaqHTlRcfgyGEaHJsvUkdgvFtfrhl0Y/ATCDPpLjcXtWN6szcAhKiQ10bTEP0mQItO8EXU4wk4RcKeYeM6qjD/gB97zAG6wkhLlm2fvX9ANgB3Gx5PRX4ELhki+9XFerb11gTBBgD1X6TZJT6ABj9V6Pukae3a+MSQrgFWxNErNb6hmqvn1dKbTUhnkajXUvL9KP2zC7nDkJj4I75ro5CCOGGbO2GUqSUOt95Xik1FGOE8yWravpRhxTtE0IIN2TrFcT9wMeWexEAp4E7zQmp8ejoyKJ9QgjhZmy6gtBap2qtewEJQILWug9whamRNQKdwgM5cPIc5RWVrg5FCCEcrl4jnbTWZy0jqgEeMyGeRiW2VcD56UeFEKKpsWcobBOa/LhhHDq7nBBCuBl7EkSdpTYuBVVdXeVGtRCiKarzJrVSKh/riUABzUyJqBEJaeZNqyBfuVEthGiS6kwQWusgZwXSWHVqFShXEEKIJknKcdopNjyAfbkFVRMeCSFEkyEJwk6xrQLJLy4nt6DE1aEIIYRDSYKwU1XRPmlmEkI0NZIg7PRL0T7p6iqEaFokQdgpMsSPID8vfkzPcXUoQgjhUJIg7KSU4v7LY/lhdw4rdh93dThCCOEwkiAc4DfDOtIpPJBnF+ykqLTC1eEIIYRDmJoglFJjlFLpSqkMpdSTVtYrpdQblvXblFJ9q60LVUp9pZRKU0rtVkoNNjNWe/h4efDXSfEcOVPEGyv3ujocIYRwCNMShFLKE3gbuAaIA25VSsXV2OwaoLPlMQ14p9q614FvtdbdgF7AbrNidYRBHVtyQ99o3l2VyZ7j+a4ORwgh7GbmFcRAIENrnam1LgXmABNrbDMR+Fgb1gGhSqlIpVQwxvzX7wNorUu11mdMjNUhnhrbjQBfL56Zv0MGzgkhGj1l1geZUupGYIzW+j7L66lAotb6oWrbLAZe1FqvsbxeATwBlAOzgV0YVw8pwKNa61/1JVVKTcO4+iAiIqLfnDlzGhRvQUEBgYGBDdq3uuTDZXy0s5T7evpwWZTj5nZ2VHxmkfjsI/HZR+JruJEjR6ZorftbXam1NuUB3AS8V+31VODNGtssAS6r9noF0A/oj5EkEi3LXwf+crFj9uvXTzdUUlJSg/etrqKiUl/39hrdZ+Z3+lRBiUPeU2vHxWcWic8+Ep99JL6GAzbpWj5TzWxiygJiqr2OBo7auE0WkKW1Xm9Z/hXQl0bAw0Pxt+t6kldUxj+Wp7k6HCGEaDAzE8RGoLNSqoNSygeYDCyssc1C4A5Lb6ZBQJ7WOltrfQw4rJTqatluFEZzU6PQPTKYe4a2538bDpNy8JSrwxFCiAYxLUForcuBh4DlGD2Q5mqtdyql7ldK3W/ZbCmQCWQA7wK/q/YWDwOfKaW2Ab2Bv5sVqxmmX9mFyBA/np63gzKZs1oI0QjVOR+EvbTWSzGSQPVls6o918CDtey7FeNeRKMU4OvFn6/twf2fpvDRTwf4zfCOrg7J7RSWlrN4WzbfbM5iVLcIOUdCuBlTE8SlbnSPCK7oFs6/ftjDuIRI2oRe8pPwAbDjSB5zNh5iwZaj5JeU4+vlwc4jZ7ktsS0BvvIrKYS7kFIbJlJK8fyEHlRqzfOLdro6HJcqKCnn8/WHmPDWGsa/uYa5m7K4Mi6Cub8dzGf3JZJfUs78rUdcHaYQohr5umaymBb+PDKqM//4Np0Vu48zqnuEq0NyGq0127Ly+N+GQyxMPUphaQVdI4L487VxXNcnilB/n/Pb9WgTzMc/H+S2gW1RSrk4ciEESIJwivsu68i8zUd4dsFOhsSG0czH02HvXVxWwafrDhLk58XE3lH4eTvuvRsqr6iMBVuP8L8Nh9mdfZZm3p6MT4jk1sS29IkJ/VUCUEpx5+D2/N/X21i//xSDOrZ0UeRCiOokQThBVTG/W2av442Ve3liTDeHvO+6zJM8+fU2DpwsBODl5Xu4e2h7piS2I8TfcaO4baG1JuXgKT5ff5gl249SXFZJXGQwf5kUz8TebQj2qzuea3u14W9Ld/Px2gOSIJxsYepRCkvKmTywratDEW5GEoSTJFYr5nddnyi6RAQ1+L3yi8t4cVkan60/REyLZnx2XyIK+M+qTF5ens7bSRlMHtCWey5rT3Rzf8f9EFYcOVPE0m3ZfPhTEUeXryXAx5Pr+kRz68AYekaF2Nxc1MzHk1sGxPD+mv0cyyumdYifqXELQ3lFJTMX7eTUuVK6tA6ib9vmrg5JuBFJEE701Nhu/LD7OM/M38EX0wY1qK09KS2Hp+dtJ/tsMfde1oE/XN0Ffx/jv3FIpzB2Z5/l3VWZfLz2AP9de4DxCZFMG96RHm1CHPZzHDpZyLId2SzdcYzUw2cA6BDiwYvXx3NtrzYN7ok0JbEd767O5PP1B3ns6q4X30HYbV3mKU4UlOLj6cGML1NZ+sgwt2imFO5BejE5UctAX/54TTc27D/F15vr12Mnv1Tz+y+2cvdHGwnw9eLrB4bwp/Fx55NDle6Rwbx6S29W/d9I7h7Snh92HWfcG2uY+v56Vu/NbXCV2czcAt5OymD8m6sZ/nISLyxLQ2vN/43pStKMEfx5cDMmD7Svm2rblv5c0TWczzccoqRcJl5yhkWpRwnw8eTt2/uSmXuOf/2wx9UhCTciVxBOdnP/GOZuOszfl+5mVLdwmgf41Lm91pol27N5ak0hReVFPDKqMw+OjMXXq+5veW1Cm/HM+DgeHtWZz9cf4oOf9jP1/Q3ERQbz28s7Mq5nJF6edX8/yMjJZ+n2Yyzdnk3aMWOOiz5tQ3l6bHfGxLcmpsUvzVcHbfz5L2bq4Has+DCHb3ccY2LvKAe9q7CmtLySZTuyubpHa66Ki+DWgTG8uyqTMT1a00eamgSSIJyuqpjf+DfX8NK3abx4Q0Kt2x4/W8wz83fw/a7jdAj24N93D6V7ZHC9jhfSzJsHRsRyz2XtWbDlKLNXZ/LonK3849t07r2sA7cMiDn/rV9rTfpxIyks257N3pwClIL+7Zrz7Pg4xsS3Nn2w3/DOrWjf0p+P1x6UBGGy1XtzOVtczoRebQB4amx3fkzPZcaXqSyRpiaBJAiXqCrm9+7q/dzUP5p+7VpcsF5rzdxNh/nrkt2Ullfy1NhuxJYfqndyqM7Xy5ObB8RwY79oVqblMHtVJjMX7+L1FXuZMsjovbJs+zEyT5zDQ8HADi2YOrgHo3u0JiLYeTeMPTwUUwe35y+Ld7HjSB7xUY67dyIutCj1KKH+3gztFAZAkJ83L9yQwJ0fbOC1H/by5DWO6W0nGi9JEC4y/couLN6WzdPzdrDo4cvwtjT3HDpZyB/nbeOnjJMkdmjBSzck0D4sgOTkww45roeH4sq4CK6Mi2DzodPM/jGTfyfvw0MpBndsyb3DOnB1XGtaBfk65HgNcWO/aF5Zns7Haw/wjxt7uSyOpqyotILvdh1nYu82+Hj90tR4eZdWTB4Qw+xV+xgT35reMaGuC1K4nCQIFwnw9eK5CT347SdGMb97LuvARz8f4JXl6Xh6KP52XTy3DmiLh4d5o4r7tm3OrKn9OJZXjI+XBy0ucj/EWUKaeTOpTxTfbM7ij9d0v+h9GlF/K9NyKCyt4NqENr9a99S47vy4J5fHv0xl0cOXSVPTJUx6MbnQ1XERjLIU87vhnZ/5y+JdDOrYgu9+P5zbE9uZmhyqax3i5zbJocodg9tRUl7JlymOuXISF1qUepRWQb4kWhmUGOznzQvX92RvTgFvrNjrgugalx1H8li+85irwzCFJAgXUkrx3IQeaA0HT57jtVt688FdA6TqK8Z9moEdWvDJuoNUVJozb/qlKr+4jJXpOYzrGYlnLV9CRnQN5+b+0cz6cd/5sS7CupmLd/Hw/7ZwprDU1aE4nCQIF4tp4c+304eRNGMEk/pESaG6au4c3J7Dp4pITs8x9ThbDp1mz/F8U4/hTr7fdZzS8kqu7fXr5qXqnhkfR0SwHzO+TKVMkrRVeYVlpBw8TWl5JQu21pxRufGTBOEG2rUMOF/ZVPzi6h4RRAT78t+1jhpl8Wtpx84yefY6bpq1lsOnCk07jjtZmHqUqNBm9G0bWud2wX7e/N3S1LQgo8w5wTUyq/bmUlGpCfbzYu6mptccKglCuC1vTw9uG9iOVXty2X/inMPfv7C0nIc+30KQnzeVWvPAZykUlzXtEdynzpWyZu8JxveKtOlqdWTXcG7qF83S/WVsyzpjfoCNTFJaDqH+3ky/sgs7j55lx5E8V4fkUKYmCKXUGKVUulIqQyn1pJX1Sin1hmX9NqVU3xrrPZVSW5RSi82MU7ivWxNj8PZUfGLCVcSfF+xkX24Br0/uzWu39GbHkbM8u2CHw4/jTr7dcYzySn1+cJwtnhkfR4iPYsaXqVICpZqKSk3ynlwu79KK6/tG4ePlwZdN7CrCtAShlPIE3gauAeKAW5VScTU2uwbobHlMA96psf5RYLdZMQr3Fx7kxzXxkXyZcphzJeUOe995W7L4MiWLh0d2YminMEZ1j+DhKzoxd1MWczYccthx3M2i1KN0bBVAXD0GXYY08+aueB/2HC/gzRUZJkbXuKRmneHUuVKu6BZOqL8Po3u0Zv7Wo03qKtTMK4iBQIbWOlNrXQrMASbW2GYi8LE2rANClVKRAEqpaGAc8J6JMYpG4I7B7cgvdtyUpJm5BTw9bwcD27fgkVGdzy+ffmUXhnUO49kFO5tkc0rO2WLW7T/JtQlt6t0ZolcrL27sF807P+5je1bTakZpqOS0HDyUMbgQ4Ob+0eQVlfHdruMujsxxzBwoFwVUv97KAhJt2CYKyAZeA/4PqHPiBKXUNIyrDyIiIkhOTm5QsAUFBQ3e1xku5fi01rQN8uCd73fSpjCzQT29quIrrdD8dV0xHrqSye2LWLN61QXb3Ryj2XVYc/d7P/PckGYE+TinV5kz/n+/O1CG1hBRkkVycv163BQUFDAyVPODNzzw0U88N6QZXk4ap2MLV/x9LNhURGyIB1s3/AxApda09FP857tUgk9fWBXX3f9+a6W1NuUB3AS8V+31VODNGtssAS6r9noF0A8YD/zbsmwEsNiWY/br1083VFJSUoP3dYZLPb45Gw7qdk8s1uv2nWjQ/lXx/Wn+dt3uicV6xe5jtW6bevi07vzUUj3lvXW6vKKyQcdraHxmmvT2Gj3mtVUN2rcqvhW7j+l2TyzWryxPc2Bk9nP238fxvCLd7onF+q2Vey9Y/up36br9k4v14VPnLljuzn+/wCZdy2eqmU1MWUBMtdfRQM2vLbVtMxSYoJQ6gNE0dYVS6lPzQhXubkKvKEKaefOxHTerl23P5uO1B/nNsA5c0S2i1u0SokN5fmIPVu89wetNZH6Ew6cK2XLoTL1uTltzRbcIbugbzb+T9zW5Hjv1kWQZm3NFt/ALlt/UPxqAr1KynB6TGcxMEBuBzkqpDkopH2AysLDGNguBOyy9mQYBeVrrbK31H7XW0Vrr9pb9Vmqtp5gYq3BzzXw8ubl/NN/uPMaxvOJ6759bWMn/fb2NXjGhPD764lVKJw+I4eb+0byxMoMVuxt/m/LibdkAjE+ItPu9nh0fR1igDzO+TKW0vNLu92uMVqblEBniR7fWF7aARzf3Z2hsGF9uyqKyCQwuNC1BaK3LgYeA5Rg9keZqrXcqpe5XSt1v2WwpkAlkAO8CvzMrHtH4TRnUjkqt+byevYzKKip5J7UEgLdu7XNB9dLaKKWYOTGeHm2C+f0XWzl40vHjMJxpUepR+rQNvWCSp4YK8TdqNaUdy+etpEuvV1NJeQVr9p5gZLdwq/fDbh4Qw5EzRfy876QLonMsU8dBaK2Xaq27aK1jtdZ/syybpbWeZXmutdYPWtb31FpvsvIeyVrr8WbGKRqHdi0DGNk1nM/XH6rXN9dXlqeTmVfJSzck1OsD0s/bk1lT+qGU4v5PN1NU2ji7L2bkFLAr+6zVyq0NdUW3CK7vG8W/kzIuuaamjftPc660giu6hltdf3VcRJMZWS0jqUWjcsfgdpwoKGHZjmybtk9Kz+E/qzK5IsaLsT3r37wS08Kf1yb3Ju3YWZ6ev73Bc3q70qLUoygF4xzQvFTdn8f3oEXApdfUtDItBx8vD4Z0+nUlXDC+WEzqE8W3O4+RV2h+iRIzfyclQYhGpfqUpBdzLK+YP8xNpVvrICZ3a3itq5Fdw3l0VGe+2XyEz9Y3rkF0WmsWbTvKoA4tHT4zYIi/N3+/zmhqevsSampKSs9hcMeW+PvUPkrg5v4xRgG/VMeM3anLJ+sO8sCnKaZc4UqCEI2Kh4diyqB2pBw8XWfTRkWl5tE5Wyguq+Ct2/ri42lfn/1HrujMyK6teH7RTrYcOm3XeznTruyzZOaeu2jl1oa6Mi6CSb3b8E7yPjJzC0w5hjvZf+Ic+0+c+1XvpZrio0KIiwzmi43mNjOdLCjhleXp5BeX4+ft+I9zSRCi0bmpXwzNvD3rrM/0xoq9rN9/ir9MjKdTeKDdx/TwUPzrlt5EBPvxu882c7KgxO73dIZFqdl4eSjGxLc27RhPj4vD19uDPy/c2Sib4OpjZZr17q3W3DIgxvQCfi99m0ZhaQXPTehhylQBkiBEoxPib0xJOn/rEauTtPy87wRvrNzLDX2juaFftMOOG+rvw6wp/Th1rpRH5mxx+4mMtNYsSj3KZZ3DTJ0xsFWQLzOu7srqvSdYtqNpzqxWJSkth07hgTZ1dqia79usAn5bDp1m7qYs7r2sg0O+BFkjCUI0SlVTktbsKXKioITpc7bSISyAmRN7OPy48VEh/HVSPD9lnOSf36U7/P0dafOhMxw5U+TQ3ku1uT2xLXGRwcxctMuhRRXdSUFJOev3n7Tp6gG4oIBfaYVjv0xUVGqeXbCTiGBfHq5WT8zRJEGIRql7ZDAD2184JWllpeaxuamcKSrj7dv6EuBrTqmxm/rHcOvAtvw7eR/fufFcxItSj+Lj5cHVPWofNe4oXp4e/GVSPMfOFvPGyqY5j/WavScoq9CMrKV7qzVVBfy25Dj2BvKcjYfYfiSPp8Z2J9Ck33OQBCEasTuGtOPwqSJ+3GO0C89encmqPbk8Oz6O7vUoZ90Qz02Io1d0CH+Ym2rKZEb2qqjULNmezRVdwwny83bKMfu1a87N/aN5f/V+9jbBKVyT0nII8vOif/vmNu8zNDaMqNBmrMpyXHfX0+dKeXl5OokdWthdOuViJEGIRmt0j9aEB/ny358PknLwNC8vT2dcz0huT2xr+rF9vTz595R+eHkq7v8khcJS92pWWb//JLn5Jab1XqrNE2O6EeDrxbMLmtYNa601Sek5DO/cCm9P2z82PTwUN/aLZtfJSrJOO2ZK25e/M3otzZwYb/oc9pIgRKPl7enBbYlt+XFPLg98mkKbUD9euKGn6X80VaJCm/HGrX3Yk5PPbz9J4YQb9WxalJpNgI+nze3ljtIy0JfHR3dlbeZJFqbWr6S4O9t59Cw5+SWMbMD5vLGf4wr4bcs6w/82HOLOwe3p2rrOmRAcQhKEaNRuG9gWLw/F6cJS3rq1L8FOak6pMqxzK/5+XU/W7z/FmNdWsTLN9YX9SssrWbYjm6viImjm4+n04986sC0J0SH8bclu8ovNH0nsDFXdW6smB6qPmBb+xLX0sLuAX6XlxnTLAF+mX2XejenqJEGIRi082I+/TornzVv70Csm1CUx3DqwLQsfGkpYoC/3fLSJp+Ztd2mT008ZJzhTWOb05qUqnh6Kv0yMJ7eghNd+aBo3rFem5dArOoRWQb4N2n9YlLfdBfy+Ssli6+EzPDW2m9O+CJk5o5xbKCsrIysri+LiuktEh4SEsHu3+05/7Q7x+fn5ER0djbe3c7+lX8zkgebfc7iYbq2DWfDQUP753R7eXZ3J2n0nee2W3i5JWotSjxLs58WwzvX/tusovWJCuXVgWz76+QA39Y+mW2tzOw2Y6WRBCalZZ3jUju6kfSM8zxfwu6xzWL33zyss48Vv0+jfrjnX9YlqcBz11eQTRFZWFkFBQbRv377Otun8/HyCgsxv02soV8entebkyZNkZWXRoUMHl8Xhzny9PHlqbHdGdG3FjLmpXP/Ozzw6qjO/GxGLVz1ubNqjuKyC73YdZ1zPSJvKmpvp8au7smx7Ns/O38kXvx3ktHtDjpacnovWto2ero2Pp2JSnyjmbDxMXmEZIf71+5L16vfpnCksZebERKeexybfxFRcXEzLli0b7S+nu1BK0bJly4teiQkYEhvGsunDGZ8Qyavf7+Gm/6x12nwSyek5FJSUu6x5qbrmAT48eU03Nhw4xbwt5hetM8vK9BzCAn2JbxNi1/s0tIDfzqN5fLLuIFMHtSOujXOvxJp8ggAkOTiInEfbhTTz5vXJfXh9cm8ycgq45vXVzNlwyPSunwtTjxIW6MvgWOulqJ3tpn4x9Gkbyt+X7iavqPHdsC6vqGTVnlxGdm2Fh4d9v/9VBfzqM0+E1po/L9hJc38fHruqq13Hb4hLIkEI4SoTe0exfPpwekWH8uQ325n2SYpphf4KSspZsTuHcT1b42nnh5mjeFhuWJ86V8q/vnf8/N6nz5Xyu89SeGNzsSlTfKYcPE1+cbnDugvf3D+aHUfOsvOobQX85m05wqaDp3liTLd6N0s5giQIIUzWJrQZn92XyDPjuvNjei6jX1tNkqXbpCP9sOs4JeWVbtG8VF18VAhTBrXj47UHHFrZdOOBU4x9YzXf7jjG5pwKPqvnVLS2WJmeg7enatCNZWsm9YnCx9Po8noxZ4vL+PvSNHrHhJ4fS+FskiBMdubMGf7973/Xe7+xY8dy5syZeu9311138dVXX9V7P2EuDw/FfcM6suChoYQF+nD3Rxt5Zv52h07ysij1KG1C/Ojb1vZSEM7yh6u70tzfhz8t2GH3N/3KSs3bSRlMnr0OHy8PFjx4GT1aevDSsjSO5Tn2HllSWg4D2rdwWLmSUH8fru4RwbwtRyguq/v//vUf9nLyXAkzJ/awu3mroUztxaSUGgO8DngC72mtX6yxXlnWjwUKgbu01puVUjHAx0BroBKYrbV+3d54nl+0k11Hz1pdV1FRgadn/QcVxbUJ5s/X1l41tCpB/O53v6vX8ZYuXVrvWIT76x4ZzPwHh/LK8nTeW7OfnzNO8q9betv9vmcKS1m1N5e7h3Zw2YdJXUKaefPHsd2Z8WUqX6VkcfOAmAa9T25+CY/N3crqvScYnxDJC9f3JMjPmzt7+PLs2hL+tGAHs6f2c8j9sqzThew5XsDN/RsWa21uGRDD4m3ZfL/reK1Xe+nH8vno5wOWQYehDj1+fZh2BaGU8gTeBq4B4oBblVJxNTa7BuhseUwD3rEsLwf+oLXuDgwCHrSyb6Pw5JNPsm/fPnr37s2AAQMYOXIkt912Gz179gRg0qRJ9OvXjx49ejB79uzz+7Vv354TJ05w4MABunfvzsMPP0yPHj24+uqrKSoqsunYK1asoE+fPvTs2ZN77rmHkpKS8zHFxcWRkJDAjBkzAPjyyy+Jj4+nV69eDB8+3MFnQVTn5+3JM+Pj+Py+RIrKKrjhnZ/5YEcJX6dkkZlb0KAb2ct3HqOsQptevM0eN/SNYkD75rz4bZrVeTwu5ueME4x9YzUb9p/ihet78uatfc5/sw/39+Cxq7rw/a7jfOugOSmqmgEbUl6jLlUF/Gq7Wa215tkFOwjy8+Lxq51/Y7o6M68gBgIZWutMAKXUHGAisKvaNhOBj7XxF7FOKRWqlIrUWmcD2QBa63yl1G4gqsa+9VbXN32zxhm8+OKL7Nixg61bt5KcnMy4cePYsWPH+bEEH3zwAS1atKCoqIgBAwZwww030LLlhT1Q9u7dy3vvvcdHH33EzTffzNdff82UKVPqPG5xcTF33XUXK1asoEuXLtxxxx2888473HHHHcybN4+0tDSUUuebsWbOnMny5cuJiopqUNOWqL8hncL49tHh/GXJLhZvzWLVl6kAhPp70zsmlD4xzenTNpReMaGENKu7iWNh6lE6hAXQw8ndIOtDKcXMifGMf3MNLy9P52/X9bRpv4pKzesr9vLmyr10DAvgk3sHWh14d8/QDixMPcqzC3cypFPYRc/ZxaxMy6FdS386hgXY9T41VRXwe2PlXrJOFxLd/MLJhxZty2b9/lP87bp4mps40ZMtzEwQUUD1FJkFJNqwTRSW5ACglGoP9AHWWzuIUmoaxtUHERERJCcnX7A+JCSE/PyLlx6uqKiwabv6KigooLKykvz8fAoLC+nXrx9hYWHnj/Xyyy+zePFiAA4fPszWrVsZOHAgWmsKCgooKCigXbt29OjRg/z8fOLj40lPT6811rKyMoqKiti8eTNt27YlMjKS/Px8brrpJt59913uvPNOfHx8uPPOOxk9ejRjxowhPz+fgQMHMnXqVK677jquvfbaWpu/iouLf3WOq35Oa8vdhTvHN74VDB+kycefjLwK9p2pJOPICX5Mz6XqWqJNgCI21JPYUA9iQz2JClR4WJpRzpRU8nNGEdfGevPjjz+aEqMjz9+oGE8+X3+IWI8cOobU3ax7uriSWaklpJ+u5LIoL6Z21xxL28yxtF/Ht2b1Km5sW8HMdSU88v5K7o5vWFkMgJIKzZq9hYyI8XLIOa15/qLLKtEa/vn1GiZ2+iUJFJVrnl1dRPtgDyILM0lO3m/3se1hZoKw1ghY89q5zm2UUoHA18B0rbXVmwda69nAbID+/fvrESNGXLB+9+7dNl0ZmHUFERgYiIeHB0FBQfj7+xMcHHz+OMnJyaxevZr169fj7+/PiBEj8PT0JCgoCKUUgYHGNILNmjU7v9zf35+CgoJaY/X29qZZs2b4+/uf3wfA398fLy8vmjdvzqZNm1ixYgVz5szh/fffZ+XKlbz//vusX7+eJUuWMGzYMLZu3fqrKxkwym306dPnV8uTk5Opee7dSWOIb0KN+PKLy9iWlceWQ6fZcugMWw6fYfURo2kmwMeThOhQ+rQNJa+oDM0hHp4wmM4R5oy2d+T56zuojFH//JH5h32Zd+3QWrvkJqXn8Je5qRSXKf55U686p4+tHl+2927+syqTB8b2Z1DHho0HWZl2nLLKTUwd1ZfhDSjQV1d8VeYfXcfGk4X8857Lz983emHpbs6UZPLBvYPdorOBmQkiC6h+dycaqFn/t9ZtlFLeGMnhM631NybGaaqgoKBav+3n5eXRvHlz/P39SUtLY926dQ47brdu3Thw4AAZGRl06tSJTz75hMsvv5yCggIKCwsZO3YsgwYNolOnTgDs27ePxMREEhMTWbRoEYcPH7aaIITzBPl5M7RTGEM7GV0stdYcOlVoJItDp9ly+AyzV2VSXqnpHhlsWnJwtGA/b54Z151H52xlzsZD3J7Y7oL1ZRWVvPJdOv/5MZNurYN467a+9ZpzefqVXVi24xhPfbOdpY8Ow8+7/p1PVqbl4O/jSWLHFvXe11Y394/h0TlbWZt5kqGdwsjIKeD9Nfu5uX+0WyQHMDdBbAQ6K6U6AEeAycBtNbZZCDxkuT+RCORprbMtvZveB3ZrrV81MUbTtWzZkqFDhxIfH0+zZs2IiPhl+scxY8Ywa9YsEhIS6Nq1K4MGDXLYcf38/Pjwww+56aabKC8vZ8CAAdx///2cOnWKiRMnUlxcjNaaf/3rXwA8/vjj7N27F601o0aNolevXg6LRTiGUop2LQNo1zKASZaCbcVlFew8mkfrkGYujq5+JvRqw/82HOIf36ZzTXwkLSxt7VmnC3nkf1vYfOgMtyW25dnxcfX+gG/m48nfrotn6vsbeGtlBjNG1+9Gr9aapLRchnYKw9fLvHLpo3u0JtjPiy82HmZIbEueW7gTfx9PnhjTzbRj1pdpCUJrXa6UeghYjtHN9QOt9U6l1P2W9bOApRhdXDMwurnebdl9KDAV2K6U2mpZ9pTWulH2/fz888+tLvf19WXZsmVW1x04cACAsLAwduzYcf4qpKrXUW0++uij889HjRrFli1bLlgfGRnJhg0bfrXfN9802ou0S5qftyf92pn3LdcsVTesx76+mpeWpfHSjQl8t/MYj3+1jYpKzZu39rFrwN+wzq24oW80s37cx/hekfWqJrvneAFHzhTx0BWdGnx8W/h5e54v4Ddn42HWZJzg+Qk9aBnY8HsnjmbqOAjLB/rSGstmVXuugQet7LcG6/cnhBBNRJeIIO65rAOzV2VyrrScxduy6RkVwlu39aFdS/t7Dj0zrjvJ6Tk8+fV2vn5giM3lR6omBxrZ1fzZ+G7uH8PHaw/y9LztdI8Mdsp0ufUhI6kbqQcffJDevXtf8Pjwww9dHZYQ9fLIqM60DvZj8bZs7hnaga8eGOyQ5ABGNdlnr41j6+EzfLz2gM37JaXlEBcZTOsQP4fEUZeqAn6VGv4ysYfTysLbqsnPB9FUvf32264OQQi7Bfp68fG9Azl9rpTEBvY4qsuEXm2Yt+UILy9P5+oerYkKrfteTV5hGSmHTnP/5R0dHkttnpvQg705+fRv735Nhe6VroQQl5wuEUGmJAcw7nX8dVI8AM/M237RUeo/7s2lolI7rHqrLQZ2aPGrnlzuQhKEEKJJi27uzx+u7kpSei6LtmXXuW1SWg7N/b3pHeMe3UxdTRKEEKLJu2tIe3pFhzBz0c5a60BVVGqS03O4vEsrt5lPw9UkQQghmjxPD8UL1ydwprCMvy3ZbXWbrYfPcLqwzOHF+RozSRBupqq8xtGjR7nxxhutbjNixAg2bdpU63tUVYIVQvwirk0w04Z35MuULH7K+PXfR1JaDh4KLndAaY2m4tLqxbTsSTi23eqqZhXl4NmA09G6J1zz4sW3q6c2bdrIxD9CONgjozqzdHs2T83bzvLpwy8Ypb0yLYd+7ZoT6u/aCqruRK4gTPbEE09cMKPcc889x/PPP8+oUaPo27cvPXv2ZMGCBb/a78CBA8THG70vioqKuOuuu0hISOCWW26xeT4IgFdffZX4+Hji4+N57bXXADh37hzjxo2jV69exMfH88UXXwDW54kQoinx8/bk79f35ODJQl77Ye/55cfyitmVfVaal2q4tK4g6vimX2RSNdfJkyczffr08zPKzZ07l2+//Zbf//73BAcHc+LECQYNGsSECRNqnQXrnXfewd/fn23btrFt2zb69u1r07FTUlL48MMPWb9+PVprEhMTufzyy8nMzKRNmzYsWbIEMIoGnjp1yuo8EUI0NUNiw7ilfwzvrs5kfEIk8VEhJKUbo6ed2b21MZArCJP16dOHnJwcjh49SmpqKs2bNycyMpKnnnqKhIQErrzySo4cOcLx48drfY9Vq1Zxyy23AJCQkEBCQoJNx16zZg3XXXcdAQEBBAYGcv3117N69Wp69uzJDz/8wBNPPMHq1asJCQkhODgYPz8/7rvvPr755hv8/f0vfgAhGqmnxnanub8Pf/xmO+UVlSSl5dAmxI+ujaQirrNIgnCCG2+8ka+++oovvviCyZMn89lnn5Gbm0tKSgpbt24lIiKC4uK6J1tvyBy7tQ0K6tKlCykpKfTs2ZM//vGPzJw5Ey8vLzZs2MANN9zA/PnzGTNmTL2PJ0RjEeLvzfMTerD9SB7/WZXJmowTjOwW7pC5rJsSSRBOMHnyZObMmcNXX33FjTfeSF5eHuHh4Xh7e5OUlMTBgwfr3H/48OHMnTsXgB07drBt2zabjjt8+HDmz59PYWEh586dY968eQwbNoyjR4/i7+/PlClTmDFjBps3b6agoIC8vDzGjh3La6+9xtatW+39sYVwa2N7tubK7uG8vDydwtIKaV6y4tK6B+EiVdOFRkVFERkZye233861115L//796d27N9261V3//YEHHmDKlCkkJCTQu3dvBg4caNNx+/bty1133XV++/vuu48+ffqwfPlyHn/8cTw8PPD29uadd94hPz/f6jwRQjRVVSXH1+77kfJKzZDYMFeH5HYkQTjJ9u2/dK8NCwtj7dq1VrcrKCgAjLEMO3bsAIwpRz/66CObb6JXzSUB8Nhjj/HYY49dsH706NGMHj36V/tZmydCiKasTWgzXpvch2Nni2nmY97kQI2VJAghxCXtqriIi290iZIE0YglJiZSUlJywbJPPvmEnj17uigiIURTckkkCK11k+ydsH79eqce72KlkoUQTYupvZiUUmOUUulKqQyl1JNW1iul1BuW9duUUn1t3ddWfn5+nDx5Uj7c7KS15uTJk/j5mT/LlhDCPZh2BaGU8gTeBq4CsoCNSqmFWutd1Ta7BuhseSQC7wCJNu5rk+joaLKyssjNza1zu+LiYrf+8HOH+Pz8/IiOjnZpDEII5zGziWkgkKG1zgRQSs0BJgLVP+QnAh9r4+v9OqVUqFIqEmhvw7428fb2pkOHDhfdLjk5mT59+tT37Z3G3eMTQjQ9ZjYxRQGHq73OsiyzZRtb9hVCCGEiM68grN0VrnkjoLZtbNnXeAOlpgHTACIiIkhOTq5HiL8oKCho8L7OIPHZR+Kzj8RnH3ePrzZmJogsIKba62jgqI3b+NiwLwBa69nAbID+/fvrESNGNCjY5ORkGrqvM0h89pH47CPx2cfd46uNmQliI9BZKdUBOAJMBm6rsc1C4CHLPYZEIE9rna2UyrVh319JSUk5oZSqu7BR7cIAd56GTeKzj8RnH4nPPu4cX7vaVpiWILTW5Uqph4DlgCfwgdZ6p1Lqfsv6WcBSYCyQARQCd9e1rw3HbPBcgUqpTVrr/g3d32wSn30kPvtIfPZx9/hqY+pAOa31UowkUH3ZrGrPNfCgrfsKIYRwHin3LYQQwipJEL+Y7eoALkLis4/EZx+Jzz7uHp9VSkpQCCGEsEauIIQQQlglCUIIIYRVl1SCsKe6rJPii1FKJSmldiuldiqlHrWyzQilVJ5Saqvl8ayTYzyglNpuOfYmK+tddg6VUl2rnZetSqmzSqnpNbZx6vlTSn2glMpRSu2otqyFUup7pdRey7/Na9nXIRWNGxDfy0qpNMv/3zylVGgt+9b5u2BifM8ppY5U+z8cW8u+rjp/X1SL7YBSamst+5p+/uymtb4kHhjjKfYBHTFGaqcCcTW2GQsswyj1MQhY7+QYI4G+ludBwB4rMY4AFrvwPB4AwupY79JzWOP/+xjQzpXnDxgO9AV2VFv2D+BJy/MngZdqib/O31cT47sa8LI8f8lafLb8LpgY33PADBv+/11y/mqs/yfwrKvOn72PS+kK4nx1Wa11KVBVIba689VltdbrgKrqsk6htc7WWm+2PM8HdtP4ihS69BxWMwrYp7Vu6Mh6h9BarwJO1Vg8Efiv5fl/gUlWdrXl99WU+LTW32mtyy0v12GUunGJWs6fLVx2/qoopRRwM/A/Rx/XWS6lBGFPdVmnU0q1B/oA1qaNG6yUSlVKLVNK9XBuZGjgO6VUiqVQYk3ucg4nU/sfpivPH0CE1jobjC8FQLiVbdzlPN6DcUVozcV+F8z0kKUJ7INamujc4fwNA45rrffWst6V588ml1KCsKe6rFMppQKBr4HpWuuzNVZvxmg26QW8Ccx3cnhDtdZ9MSZ7elApNbzGepefQ6WUDzAB+NLKalefP1u5w3l8GigHPqtlk4v9LpjlHSAW6A1kYzTj1OTy8wfcSt1XD646fza7lBKEPdVlnUYp5Y2RHD7TWn9Tc73W+qzWusDyfCngrZQKc1Z8Wuujln9zgHkYl/LVufwcYvzBbdZaH6+5wtXnz+J4VbOb5d8cK9u49Dwqpe4ExgO3a0uDeU02/C6YQmt9XGtdobWuBN6t5biuPn9ewPXAF7Vt46rzVx+XUoI4X13W8g1zMkY12eoWAndYeuIMwlJd1lkBWtos3wd2a61frWWb1pbtUEoNxPg/POmk+AKUUkFVzzFuZu6osZlLz6FFrd/cXHn+qlkI3Gl5fiewwMo2tvy+mkIpNQZ4ApigtS6sZRtbfhfMiq/6Pa3rajmuy86fxZVAmtY6y9pKV56/enH1XXJnPjB62OzB6N3wtGXZ/cD9lucKYy7sfcB2oL+T47sM4zJ4G7DV8hhbI8aHgJ0YvTLWAUOcGF9Hy3FTLTG44zn0x/jAD6m2zGXnDyNRZQNlGN9q7wVaAiuAvZZ/W1i2bQMsrev31UnxZWC031f9Ds6qGV9tvwtOiu8Ty+/WNowP/Uh3On+W5R9V/c5V29bp58/eh5TaEEIIYdWl1MQkhBCiHiRBCCGEsEoShBBCCKskQQghhLBKEoQQQgirJEEIUQ9KqQp1YcVYh1UJVUq1r14VVAhX83J1AEI0MkVa696uDkIIZ5ArCCEcwFLb/yWl1AbLo5NleTul1ApLYbkVSqm2luURlrkWUi2PIZa38lRKvauM+UC+U0o1c9kPJS55kiCEqJ9mNZqYbqm27qzWeiDwFvCaZdlbGOXPEzCK3r1hWf4G8KM2igb2xRhNC9AZeFtr3QM4A9xg6k8jRB1kJLUQ9aCUKtBaB1pZfgC4QmudaSm4eExr3VIpdQKjFESZZXm21jpMKZULRGutS6q9R3vge611Z8vrJwBvrfVfnfCjCfErcgUhhOPoWp7Xto01JdWeVyD3CYULSYIQwnFuqfbvWsvznzEqiQLcDqyxPF8BPACglPJUSgU7K0ghbCXfToSon2Y1JqH/Vmtd1dXVVym1HuOL162WZY8AHyilHgdygbstyx8FZiul7sW4UngAoyqoEG5D7kEI4QCWexD9tdYnXB2LEI4iTUxCCCGskisIIYQQVskVhBBCCKskQQghhLBKEoQQQgirJEEIIYSwShKEEEIIq/4fFl9n78VEGIoAAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
},
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEGCAYAAAB/+QKOAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuNCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8QVMy6AAAACXBIWXMAAAsTAAALEwEAmpwYAAA6bUlEQVR4nO3deXhU5dn48e+dfQ9ZICTsS5BNdgIiKogLVeuKilWrFqTY4tbl1aq/2k1f21rfamuhuBaKoqKotbhSKCrIKrIrYYeEsGbPZH1+fzwTHMIkMGFOJjPcn+uaKzPnPGfOnZPJuec551nEGINSSinVUFigA1BKKdU6aYJQSinllSYIpZRSXmmCUEop5ZUmCKWUUl5FBDoAf0pPTzddu3Zt1rZlZWXEx8f7NyA/0Lh8o3H5RuPyTSjGtXr16kPGmLZeVxpjQuYxdOhQ01yLFi1q9rZO0rh8o3H5RuPyTSjGBawyjZxT9RKTUkoprzRBKKWU8koThFJKKa9C6ia1N9XV1ezduxeXy9VkueTkZDZv3txCUZ261hBXTEwMHTt2JDIyMqBxKKVaVsgniL1795KYmEjXrl0RkUbLlZSUkJiY2IKRnZpAx2WM4fDhw+zdu5du3boFLA6lVMsL+UtMLpeLtLS0JpODapyIkJaWdtIamFIq9IR8ggA0OZwmPX5KnZlC/hKTUkq1VtW1dWzKK2btnkKiIsLI6ZZK9/T4VvOlTBOEUqpVOVpWxa7iWg6XVpIaH9VqTpb+cLCkkjW7j7Jm91G+3FXIV3sLqaypO65MekIUOd1SyemaSk63NM5qn0h4WGCOgSYIhxUWFvLKK6/wox/9yKftLrvsMl555RXCw8Mdikyp1uVIWRV/X7KNWUt3UVFdy6NLPyE6IoysNrFkJseQmRxLVpuYY6/rfybGtM7WdTW1dWzZX2ITwq6jrNldyO4j5QBEhgv9spK5ZWQXhnROYUiXNlRU1bJixxFW7DzCih1HWLB+PwCJMREM75pqk0a3VM7ukExkeMvcHdAE4bDCwkL+9re/nZAgamtrmzz5L1iwALCtmJQKZYXlVTz/6Q5e+nwH5dW1XDkwiw4cJr1jD/KLKsgrcpFfWMHnuYc4UOKirsEkmInRETZZtLFJpFNqLP2ykjm7QzKp8VEt9nscLq1k7Z5C1uw+yupdR1m3t4jyqloA2iZGM7RzCreM7MzQLin0y0omJvLE///ubROYmNMZgH2FFazccYTlO46wYsdh/rPlAACxkeEM6dLmWNIY3CnFsd/pjEoQv/7XRjblFXtdd7ITdmP6ZiXx6Hf7Nbr+wQcfZNu2bQwaNIjIyEgSEhLIzMxk7dq1bNq0iauvvpo9e/bgcrm49957mTJlCgBdu3Zl1apV7N+/n+uvv57Ro0ezdOlSOnTowDvvvENsbKzX/T333HPMnDmTqqoqevbsyezZs4mLi6OgoICpU6eyfft2AKZPn86oUaOYNWsWTz75JCLCgAEDmD17ts/HQKnmKHZV8+JnO3jh0x2UVNZw+YBM7huXTXZGIosXL2bM6BObVdfU1lFQUkl+oU0ceYUVx57nF1Wwfm8Rh8uqjpXPSo6hfwebLPq7H20To08r7tLKGr4pKOGb/SV8XVDCNwUlfL2/lEOllQCEhwl9M5O4YVgnBnduw5DOKXRMifX5UlmHNrF0GNyBqwd3AOBQaSWrdtYnjCM8vXArxtjaSPck4bzzjd8vRZ1RCSIQnnjiCTZs2MDatWtZvHgxl19+ORs2bDjWp+DFF18kNTWViooKhg8fznXXXUdaWtpx77F161ZeffVVnnvuOW644QbefPNNbrnlFq/7u/baa7nzzjsBeOSRR3jhhRe4++67ueeee7jggguYP38+tbW1lJaWsnHjRh577DE+//xz0tPTOXLkiLMHQynsCfblz3cwc8l2il01XNovg/su6kWfzKSTbhsRHmZPnG28f0ECKKqoZmNeERv2FbFhXzEb9hXx0aaCY+szkqK/TRhZyZzdMZmMpJgT3sdVXUvugVKbDArsz692lnP4gw+PlYmNDKdXRgJjzmrLWRmJDOiYzICObYiN8v+l4fSEaMb3z2R8/8xjv+eaXUdZvuMIm7btcuQ+xRmVIJr6pt9SHdJycnKO63D2zDPPMH/+fAD27NnD1q1bT0gQ3bp1Y9CgQQAMHTqUnTt3Nvr+GzZs4JFHHqGwsJDS0lIuvfRSAP7zn/8wa9YsAMLDw0lOTmbWrFlMmDCB9PR0AFJTU/31ayp1gvKqGmYt28Xf/7uNo+XVjOvdjvsv7kX/Dsl+3U9ybCSjeqQzqkf6sWUlrmo25RWzfl8RG90/F245gHFfrmqbGE3/rCS6t01g39EKvikoYefhsmOXsyLDhR5tE8hOCeMHA3rSKyORszIS6ZgSS1iAbiAnx0Yytnc7xvZux+LF+x3ZxxmVIFoDzzHbFy9ezCeffMKyZcuIi4tjzJgxXjukRUd/WyUODw+noqKi0fe//fbbefvttxk4cCAvv/wyixcvbrSsMSakWoio1slVXcs/v9jFjP9u41BpFRf0asv9F/diUKc2LRZDYkwkI7qnMaL7t1++yipr2Jxvaxjr3TWNz3MP0zE1ll4ZiVwxMIuzMhI5q30CXdLiiQwPs5e+xvRssbgDTROEwxITExu90VxUVERKSgpxcXFs2bKFL7744rT3V1JSQmZmJtXV1cyZM4cOHez1y3HjxjF9+nTuu+8+amtrKSsrY9y4cVxzzTXcf//9pKWlceTIEa1FKL+prKll7oo9PLsolwMllZzbM40ZF/ViWNfW8RmLj45gWNfUVhNPa6QJwmFpaWmce+659O/fn9jYWDIyMo6tGz9+PDNmzGDAgAGcddZZjBw58rT399vf/pYRI0bQpUsXzj777GPJ6emnn2bKlCm88MILhIeHM336dM455xwefvhhLrjgAsLDwxk8eDAvv/zyacegglNtnWHv0XK2HSwl90ApOw+XU1NbR5gIYWFCmGCfiyAC4e7lcmy5XSYi1NTV8daafeQXucjpmsozNw1mZPe0kwehWhVNEC3glVde8bo8Ojqa999/3+u6+vsM0dHRbNiw4djyn/3sZ03u66677uKuu+46YXlGRgbvvPPOCctvu+02brvttibfU4UWV3Ut2w6Wsu1gGQu3VvFG3hq2HShl+6Eyqjw6baXERRITGU5tnaHO2EuSdcZQW2cwBuqMXV7rnn2szr2s/rr+0C4p/HHCQM7tqWOhBStHE4SIjAeeBsKB540xTzRYnwK8CPQAXMAPjDEb3OvuBe4EBHjOGPNnJ2NVKpQYYzhYWsnOQ9/WCOp/7iusOHYSF6BzWhE92yZwfq+29GybQI928fRom0CbuOb1IahPFoHq/av8x7EEISLhwLPAxcBeYKWIvGuM2eRR7CFgrTHmGhHp7S4/TkT6Y5NDDlAFfCAi/zbGbHUq3mDz4x//mM8///y4Zffeey933HFHgCJSDbmqa/lw435eW7mH3QXlZO9YQVabWDqk2GaaWe7mmu0So4loRs/YujpDQYmLnYfK2XW4jJ2Hj/9Z30kLICYyjB5tExjSOYUbhnWihzsR7N64mkvGjfXnr42IEK65ISQ4WYPIAXKNMdsBRGQucBXgmSD6Av8LYIzZIiJdRSQD6AN8YYwpd2/7X+Aa4A8OxhtUnn322UCHoBqRe6CUuSt28+aavRwtr6ZzahzpsWEUFFfy5Z5CCsurjysfHia0T4pxJ40YOqTY5JHVJpaObWKJjghn1xH3if/Qtwlg95Hy48bxiQoPo1NqLF3T4jmnexpd0+PokhZPj7bxZCV7b465f4ueyVXjxBhz8lLNeWORCcB4Y8xk9+tbgRHGmGkeZR4HYowxPxGRHGApMAIoB94BzgEqgIXAKmPM3V72MwWYApCRkTF07ty5x61PTk6mZ8+TN0trbk9qp7WWuHJzcykqKjr2urS0lISEhABG5F2g4qqqNawuqGXxnmq+PlpHuMDgduGM7RRJn7QwysvKjsVVUWM44jIcrqjjcIXhsMtw2OV+XmE4WmlOGE6iXmQYZMQJ7eLCaBcXRkackBEfRrs4ITXG3kD2hf4dfROKcY0dO3a1MWaYt3VO1iC8fVIbfuyfAJ4WkbXAeuBLoMYYs1lEfg98DJQCXwE13nZijJkJzAQYNmyYGTNmzHHrN2/efEod4AI9c1tjWktcMTExDB48+Nhr2x58TOACakRLx5V7oJRX3bWFQndt4YHxnZkwtONxQzr4EldtnaGg2A4jsa+wgsrqOjqnxdE1LZ52idF+7Zilf0ffnGlxOZkg9gKdPF53BPI8CxhjioE7AMQ2c9jhfmCMeQF4wb3ucff7KeWTujrDhrwiPt16iC93HyU5NopOqbF0SomjY0osnVLjyEiK8emGqqu6lg827OeVFbtZseMIEWHCpf3ac1NOZ0b1SDvtE3h4mBy7xOT1a51SLcTJBLESyBaRbsA+YCLwPc8CItIGKDfGVAGTgSXupIGItDPGHBCRzsC12MtNSp1UQbGLT7ceYsk3B/ks9xBH3IO39WyXQKmrmIISF55XViPD7QnZM2l0TImlY0ocnVJjaZsQjYiQe6CEV1fsOVZb6JIWxwPje59QW1AqVDiWIIwxNSIyDfgQ28z1RWPMRhGZ6l4/A3szepaI1GJvXk/yeIs3RSQNqAZ+bIw56lSsrUlCQgKlpaXk5eVxzz338NJLL51QZsyYMTz55JMMG6bfL8F+o1+18yhLth5kwZpy9n6wELCDm43p1Zbze7VldHY66Qn2JF5ZU8u+oxXsPVrBnqPl9ueRcvYcreCTzQUcKq067v2jI8JomxjN3qMVRIYLl/Rrz/dyOnNO99OvLSjVmjnaD8IYswBY0GDZDI/ny4DsRrY9z8nYWrusrCzmzZun80F4YYxh28FS/vuNrSUs33EYV3UdUeFh9EgWHvzOWZyf3Zbe7RO9nsCjI8Lp3jaB7m2939Qrr6phX4PkkVfk4uYRXbh+WMdjiUapUHdm9aR+/0HYv97rqtjaGghvxuFofzZ854lGVz/wwAN06dLl2IRBv/rVrxARlixZwtGjR6muruZ3v/sdV1111XHb7dy5kyuuuIJly5ZRUVHBHXfcwaZNm+jTp0+Tg/WB7U29cuVKKioqmDBhAr/+9a8BWLlyJffeey9lZWVER0ezcOFC4uLieOCBB/jwww8REe68807uvvuExmKtwqa8YmYt28mSbw6SV2QHNezeNp6Jwztzfq90RnZPY8XSzxhzQY/T2k9cVATZGYlkZwS+cYBSgXRmJYgAmDhxIvfdd9+xBPH666/zwQcfcP/995OUlMShQ4cYOXIkV155ZaPDEUyfPp24uDjWrVvHunXrGDJkSJP7fOyxx0hNTaW2tpZx48axbt06evfuzY033shrr73G8OHDKS4uJjY2lpkzZ7Jjxw6+/PJLIiIiWuWcEAdLKnnq46+Zu3IP8VERjO6ZzrQL23JedjqdUuMCHZ5SIevMShBNfNOvcKg56eDBgzlw4AB5eXkcPHiQlJQUMjMzuf/++1myZAlhYWHs27ePgoIC2rdv7/U9lixZwj333APAgAEDGDBgQJP7fP3115k5cyY1NTXk5+ezadMmRITMzEyGDx8OQFKSnZzlk08+YerUqURE2I9CaxrNtbKmlpc+38lf/5OLq7qWSed24+5x2STHts45iJUKNWdWggiQCRMmMG/ePPbv38/EiROZM2cOBw8eZPXq1URGRtK1a1ev80B4OtXBznbs2MGTTz7JypUrSUlJ4fbbb8flcjU690NrnBPCGMOHGwt4fMFmdh8p56I+7Xjosj6N3jNQSjnD9wFglM8mTpzI3LlzmTdvHhMmTKCoqIh27doRGRnJokWL2LVrV5Pbn3/++cyZMwewM8atW7eu0bLFxcXEx8eTnJxMQUHBsdFie/fuTV5eHitXrgRsB7yamhouueQSZsyYQU2N7YcY6EtMG/OKuOm5L5j6z9XERoYze1IOz982XJODUgGgNYgW0K9fP0pKSujQoQOZmZncfPPNfPe732XYsGEMGjSI3r17N7n9XXfdxR133MGAAQMYNGgQOTk5jZYdOHAggwcPpl+/fnTv3p1zzz0XgKioKF577TXuvvtuKioqiI2N5ZNPPmHy5Ml88803DBgwgMjISO68806mTZvW6Ps75WBJJX/66GteW7WHlLgofnd1fyYO79SsQeyUUv6hCaKFrF//beup9PR0li1b5rVcaWkpAF27dmXDhg2UlJQQGxtLwzGmmtLYpD/Dhw/3OmvdU089xVNPPXXK7+9Prmp7n+HZRblU1tQyeXQ3pl2o9xmUag00QaiAsPcZ9vPYgs3sOVLBRX0yePjyPnRLjz/5xkqpFqEJIoiNGDGCysrK45bNnj2bs88+27F97ius4Edz1rA1v4y2KxfRJi6KlLhIUuKi3I9I2sR/u6yN+2dqfBQxkXZU2g37ivjte5tYvuMIZ2Uk8s9JIxidne5YzEqp5jkjEkRrbKnjD8uXL2+R/dQPCb8xr4g7XlpJRXUtIzMjSEhtw9HyKg6XVpF7oJTC8mpKK70OugvYSWtS4qLYX+wiJS6Kx67pz43D9D6DUq1VyCeImJgYDh8+TFqazovbHMYYDh8+TFlNGLfNWEZybCTzpo4if8tqxowZfEL5qpo6CsurOFpezdHyqgbPqzlaVkVmcgyTzuuu9xmUauVCPkF07NiRvXv3cvDgwSbLuVwuYmJiWiiqU9ca4jriMkx7dxed0xJ46fbhtE+OIX+L97JREWG0S4qhXVLrO5ZKKd+EfIKIjIykW7duJy23ePHi4ybEaS0CGZcxhj9/spWnF27lvOx0/nbzEBJj9Fu/UmeKkE8Qqnmqa+t46K31vLF6L9cP7cjj155NpN4rUOqMoglCnaDEVc2P5qzh062HuO+ibO4dl633b5Q6A2mCUMfZX+TijpdXsrWghD9MGMANwzqdfCOlVEjSBKGO+Xp/CXe8tIKiimpevH045/dqG+iQlFIBpAlCAbB02yF+ONsOkPf61HPol5Uc6JCUUgGmCULx9pf7+Pm8r+iWHs9Ld+TQoU1soENSSrUCmiDOYMYY/rZ4G3/88GtGdk/l77cO085rSqljNEGcoWpq6/h/72zk1RW7uXpQFr+fMIDoiPBAh6WUakU0QZxhCsurWLrtMK8s381nuYf40Zge/OySswgL02asSqnjaYIIcZU1tazeeZTPcg/xWe4h1u8rwhhIiI7gsWv6c/OILoEOUSnVSmmCCDF1dYYt+0v4LPcgn+UeZsWOw7iq6wgPEwZ3asM9F2ZzXnY6Azu10Z7RSqkmaYIIAXmFFXy21dYQPs89xOGyKgB6tI1n4vDOjO6ZzojuqTqOklLKJ5ogglRheRV//mQrS7YeZPvBMgDSE6I5Lzudc3umMzo7ncxkba6qlGo+TRBBqKqmjimzV7Nm11HO7ZnO93I6Mzo7nbMyEnXMJKWU3ziaIERkPPA0EA48b4x5osH6FOBFoAfgAn5gjNngXnc/MBkwwHrgDmOMy8l4g4Exhofnr2fFjiM8PXEQVw3qEOiQlFIhyrG7lCISDjwLfAfoC9wkIn0bFHsIWGuMGQB8H5tMEJEOwD3AMGNMf2yCmehUrMHk70u288bqvdwzLluTg1LKUU42Y8kBco0x240xVcBc4KoGZfoCCwGMMVuAriKS4V4XAcSKSAQQB+Q5GGtQ+GDDfn7/wRauGJDJ/RdlBzocpVSIk/oJ6f3+xiITgPHGmMnu17cCI4wx0zzKPA7EGGN+IiI5wFJ3mdUici/wGFABfGSMubmR/UwBpgBkZGQMnTt3brPiLS0tJSEhoVnbOqk+rp1FtTy+wkXHhDAezIkhKjyw9xpa+/FqbTQu32hcvjmduMaOHbvaGDPM60pjjCMP4HrsfYf617cCf2lQJgl4CVgLzAZWAgOBFOA/QFsgEngbuOVk+xw6dKhprkWLFjV7WyctWrTI7C+qMDmPfWzOefwTU1BcEeiQjDGt+3i1RhqXb1o8rpoqY1Y8b8wL443Z92WjxULxeAGrTCPnVCdvUu8FPGeb6UiDy0TGmGLgDgCxzW92uB+XAjuMMQfd694CRgH/dDDeVqmy1jD5H6soddUw765RtEuMCXRISoWOujrY+Bb853dwdAeER8OcCTDpI0jtHujoAs7JexArgWwR6SYiUdibzO96FhCRNu51YFssLXEnjd3ASBGJcyeOccBmB2NtlerqDM+tq2RDXhHP3DSYPplJgQ5JqdBgDGz9GP5+Prw5CaLi4XtvwNTPoK4G/nkdlB4MdJQB51iCMMbUANOAD7En99eNMRtFZKqITHUX6wNsFJEt2NZO97q3XQ7MA9Zgm7iGATOdirW1+tPHX7OqoJaHL+vDuD4ZJ99AqVBQWUJEdbFz77/7C3jpMltTqCqBa5+HH34KvS6Btr1soijOt+srS5yLIwg42g/CGLMAWNBg2QyP58sAr81xjDGPAo86GV9r9ubqvTy7aBtjOkYwaXS3QIejlLMqS+CbD2HjfNj6MefWVsGOwdDrUsi+GDIHQ9hpfp8t2AgLfwvfvA8JGXD5n2Dw9yEi6vhynYbDDf+AV2+C126F771+YpkzhPakboVW7DjCg2+tY1SPNG7pUeFc7+i6WqgqhaoyqCz99rnnz0rPZfXPy+hzpBhcH0FiBiS0h4R2kNje/uPFpoD26FYnU1XmkRQ+ghoXJGbCsB+ws6CQbjW5sPgJWPy/EJduE0X2xdDjQvsZO1VHd8Kix2Hd6xCdBON+CSOm2stKjel1KVz5DLzzY3jnR3DNzNNPUEFIE0Qg1VRB2QFI7nhs0a7DZfxw9io6pcQx/eahfLnic//vd8sC+PdPocSHriURsfYfKjoBIuNJKj4Mq1ZCTcWJZcOjbKJIyHAnjXY2iSS6l6X2gPTsMyeJFOcRXlMW6Chah+oKe+1/41s2OVSX28/EkO9Dv2ug00gIC2PX4sV0GzMGyg7DtoU2gXzzAXz1KkgYdBrhThiXQEZ/75+l0gOw5I+w6iUIC4dz74Fz74O41FOLdfAtUFoAC39jY7z0MX8eiaCgCSKQ/v0TWDsHRt0DY35BUU04k/6xijoDL9w+nOQ4P4++Wl0BH/0/WPkctB8AQ2+3J/yoeIhKcD/i3Ykg8dvnUQn2H8zD8sWLGXPBBfbSQGmBfZTs93heAKX74ch22LUUKo4cH0tcOnQ5BzqPgi6joP3ZJ+wjqBXthY1v22/H+1YxMiIROv4del/ecjEU58GWf8OAGyAmueX221C1y57kN7xlT/JVpfbvP/AmmxS6jGr8bx+fZuMfcIOt8e5bbZPF1o/siXvhbyAx69tk0f0CMHXw+TPwxXRbKxnyfbjgfyApy/fYR//EfpaX/dV+2eHs0zoUwUYTRKAUbIQv/wlpPeHzP2O+/oAnw6ex81AasyeNoFt6E9Xf5jiwGeb9AA5sgnOm2Wp2RPTpvacIxCTZR/pJenbX15ZKCmwMu5bC7qWw+V92fXSS/VbY5Rzoci5kDT79+FpacT5sesd+O96z3C5rPwDGPoJr5atEzv0eDJ8Ml/wOIh0cadcY+GouvP8AVBbBp3+C7/we+lzZcrW2mkrYtsgmyK8XQGUxxKZC/+ug/7XQZTSE+3j6CQuHTjn2ceEj9njnfmKTxYa3YM0/ICzSHtvKYuh3LYx9GNJ7Nv/3EIHxT9jP7kePkNH7fmBM898vyGiCCJSPH7Un1kkfYfauouSNu3i06h4m9r2Tfl0u8t9+jIFVL8CHD9tawc1vQrYf3/9URUTZS2nJHaHjUBhyq11etA92L4Ndn8OuZfYbIUBEDHQYZr9ddhllTwpNXTMOlJIC2PyuPUHtXgYYe8njwkfsCSqtBwBragdxQfVi+01011K47gXIaDg0mZ/iee8+e1LufA6Muttex3/9+9DrO3DZH6FNp5O+TbPVVMHql+2lnbIDtubS90pbU+h2AYT7sVaclGk/R0Nutfvds9wmi9ICGPkjyBrkn/2EhcE1f4eyQ5z19TOQOxp6BuB/KAA0QQTC9v9C7sdw8W8hLpV/HMzmqZLHmdNhPmfn/h2eXwpXz4D2/U9vP+VH4N27Yct79gN99XR7P6A1Se4AZ0+wD7DXnHcv+zZpfPokLKmDsAjIHAgdhkJMm0YuhSUSX7oLju76dnl4lP+/NZcdctcU5tsYTR207QNjfmFPhG17nbCJCYu017B7jIX5d8FzY21NYvhk/8RnDGx4Exb8zF5KvPRxeyM2LByyL4XlM2DRY/DsCBj7kF3n6zf4ptTVwvp5dh+Fu2wt8Kq/QvexLdMCKCIKup1nH468fzRMnEP5Xy4g4bXvw+3/sp/FEKcJoqXV1cHHv4TkTpAzhUVfH+A3721iXN8e9L3lVfhmAfzrXpg5BsY8gNQNad5+dnwKb02BsoPuk8VdwdEKIz4N+lxhH2DvcexZbmsXu5baSyeVJdhR4E80HGCVx4KwiOPvr5zSPReP59HucuGR9phunA87loCphbRsOP/nNim063Nqv1/Pi+Cuz+Htu+zJfNsieyI91Run3pQdsvezNr1ja11XTz8+SYVHwKhp9pv8gp/DRw/Dutfgu083f5/1jLE3mxf+Bg5stJfUbn4Teo4LvUYIMcmsG/BLRm1+FOZcD5M+PlZDDFWaIFraxrcgfy1c83dyj1Zz9ytf0rt9En++cRDhYWJvYnYaCQt+Cv/5HYMTe0L/OdCu96m9f221bRb46VP2w3vTJ/6ragdCdKI9qXpW6evqbOupE5rilrHxy+X0y+7idR1VJd8+L9xzfHPe6vJTiye1O4y+3yaFjH7NOwkmtLOdsZZPt5cap4+Ca2dCt/N9f6/N/4J/3WevuY971DZ4aKxm0KYz3DTXXhJb8D/w/Dh6Zl0O5wy1x9lXu5bCJ7+GPV/Y4zLhReh7TXB8EWmmquhUuGU+vHgJzL7GJonE0O3EqgmiJdVUwsJfQ8bZcPYNzF2wharaOl64fRjx0R5/ivg0uP5l6HMlsW/fY4cDuPBhe3O5qZY+R3fCm5Nh70rbRG/87+034FATFvbtN3yOv2R2cF8EDB7j+3vW1dokcSyheCSTyhK7LnOg/Ybsj2/GYWFwzo+h62iYNwn+cSWc9xN7mepUrtOXH7E3ode/bmO65l+ndk9DBPpeBd3HwMLf0GHlC/DsGntv4lRbWO1fb2sMWz+yzZev+D8YfKt/7y+0Zuk94eY34OXvwpzr4PYF9n5iCNIE0ZJWvgCFu+GWtyAsjLyiCjqmxDY+d3T/a1mRF8a5R96wl6W2/NtePvBWrV0/D967HxCY8JJtKaJOXVi4/RbdnG/SpyNzIPzwv/Zk/+mf7P2p656H1CZ6z3/zIbx7D5QfsgnlvJ/6fnKOSYbL/8SXNb0YkjcL5n4Pel8B3/mDvS/kzZHttsPZ+jfs9hf9GnKmQFScb/sOBR2Gwg2z4NUb4bWb4eZ5wdfq7hSEbl2wtakohCV/sDfteo4DIK/QRVZjycGtOqoN3PhP25Pz4BaYfq5t311XZwtUltibnm9OstfBp36qySHYRMXb+xATXoJDW2HGebDujRPLuYrg7R/DKzfYexaTF8KYB0/rm3tx8lkwZbE92ecuhGdz4IsZtkZVr2Q/vPcT+Otw2Pye7Rtw71cw+r4zMznUy74IrnrW3pOa/8Nv/ydDiNYgWspn/2eTxMW/PrYov6iC87PbnnxbERh4o22h8e498MGD9h/1nB/BR4/YS0sXPADn/49/W6aoltX/Wug4zF4mfGuy7Vx22R9trSZ3oW2RVpJvT9BjHvTfN9bwSHuy73e17WH/wQOwbi5c8pjtZ/DFdKirhiG32Q5nie39s99QMHCibVb78S9tb+vxT4TUzXk9m7SEor22meGAG+0lBaC6to4DJZVktvGhw1RSlr32+eVs+OAhe1kgqQPc9h50Pdeh4FWLatPZXtNe8gfbl2DPctufYe0c22pq0sc2iTghpau9VLLxLXj/QXj5Mrv87Ott01idH8G7UffY/idfPAur/9H8BNHrUnvvsRXRBNESFj1u28pf+PCxRQXFLoyBrGQfJwASsUMHdB9r270P+f7pNZFUrU94hD0hdx8Db94Ja1+xDRQufMTZHthgP1/9r7MD4q17AzqPhMwBzu4z2InYPi2p3WwfkOZqe4pNpVuQJgin7d9g/8FHTbPfDt3yi1wAvtUgPLXpZC8LqNDVZRT8aKn9duql852jYlNgxJSW3WcwCwuDnDsDHYXfaYJw2ie/si0+zvvpcYvzCu0oqD7XINSZJSY5sAPtqTOatmJy0vbFdkiN8356wvj19TWI9poglFKtlCYIpzQYUqOh/MIKEqMjSIw5QzoXKaWCjl5icsrGtyD/K9t/IfLEWkJekYvMNlp7UEq1XlqDcEL9kBrtz7ZNBL3IL6povAe1Ukq1ApognLDyeTukxsW/aXTgsv1FLrK0BqGUasU0QfhbRaHt4NTjQvvworKmlkOlVVqDUEq1apog/K1+SI2Lft1okf31fSC0BZNSqhXTBOFPRXvtuDUDbmyy92leoU0QWc3tJKeUUi1AEwTAp0+ReniVndf2dCx63P70GFLDm/wi20lOaxBKqdZMm7lWlcHSZxhQcRS2/sVOddmcCdaPDalx93FDanhzbJgNvQehlGrFNEFExcNPv2b9289wdthW2PgOfPlPiE2FPt+1QzB3GX3yYbQ/edQ9pMZPTrrLvMIKUuIiiY1qYnY4pZQKME0QABHRHE4fDmN+DtUuOw7/hrfsLG1r/gFx6XbC937X2gHUGk77uX2xHTf/kt+dMKSGN/lFLq09KKVaPUcThIiMB54GwoHnjTFPNFifArwI9ABcwA+MMRtE5CzgNY+i3YFfGmP+7GS8gO313Pty+6iusPPubpwPX82FVS/aSUH6XmUvQ3Uaabf5+JeQ3BmGn9pojnmFdqpRpZRqzRxLECISDjwLXAzsBVaKyLvGmE0exR4C1hpjrhGR3u7y44wxXwODPN5nHzDfqVgbFRlrk0Hfq+y9im8+tENorJkFK2ZCYiZkDbFDalz7nNchNbzJL3IxvKvO4aCUat2crEHkALnGmO0AIjIXuArwTBB9gf8FMMZsEZGuIpJhjCnwKDMO2GaMOY2ZOPwgKt7ej+h/rZ0H+usPbM0i92ObJPpPOKW3Ka+qoaiiWsdhUkq1emKMceaNRSYA440xk92vbwVGGGOmeZR5HIgxxvxERHKApe4yqz3KvAisMcb8tZH9TAGmAGRkZAydO3dus+ItLS0lISHB5+3CayowEkZd+KnND5xXWsdDn1UwZUA0o7JOnp+bG5fTNC7faFy+0bh8czpxjR07drUxxvs8tsYYRx7A9dj7DvWvbwX+0qBMEvASsBaYDawEBnqsjwIOARmnss+hQ4ea5lq0aFGzt/XFkm8OmC4PvGeWbTt0SuVbKi5faVy+0bh8o3H55nTiAlaZRs6pTl5i2gt08njdEcjzLGCMKQbuABARAXa4H/W+g609eF5yCmr59b2otRWTUqqVc7In9UogW0S6iUgUMBF417OAiLRxrwOYDCxxJ416NwGvOhhji6vvJJeRfGqXpJRSKlAcq0EYY2pEZBrwIbaZ64vGmI0iMtW9fgbQB5glIrXYm9eT6rcXkThsC6gfOhVjIOQXVZCeEE10hHaSU0q1bo72gzDGLAAWNFg2w+P5MiC7kW3LgTQn4wuEPJ0HQikVJBq9xCQif6j/tt9g+f0i8ntnwwpd+YUVOkifUiooNHUP4gpgppflTwOXOxNO6NNhNpRSwaKpBGGMMXVeFtYB4lxIoavYVU1pZY1eYlJKBYWmEkS5iJxwf8C9rMK5kEJXfRNXrUEopYJBUzepfwm8LyK/A+p7Ng8DfgHc53BcISnPPVGQ1iCUUsGg0QRhjHlfRK4Gfg7c7V68AbjOGLO+BWILOVqDUEoFk0YThIjEAAXGmNsaLG8nIjHGGJfj0YWY/KIKwgTaJWonOaVU69fUPYhngPO8LL8Y+D9nwglteYUuMpJiiAjXqcCVUq1fU2eq0caYtxouNMbMAc53LqTQlV+kfSCUUsGjqQTRVFNW/QrcDNoHQikVTJo60R9wz9FwHPeyg86FFJqMMeRpL2qlVBBpqpnrz4HXReRljm/m+n3syKzKB4Xl1VTW1JHZRmsQSqng0GgNwhizAhiBvdR0O1Dfmuk2bJJQPjjWB0JrEEqpINHkaK7uiXoeFZHB2LkZbsPeoH6zBWILKcf6QGgNQikVJJrqB9ELeynpJuAw8Bp2DuuxLRRbSMnXGoRSKsg0VYPYAnwKfNcYkwt2qO8WiSoE5RW5iAwX0hO0k5xSKjg01YrpOmA/sEhEnhORcegors2WX1hBRlIMYWF6CJVSwaGpm9TzjTE3Ar2BxcD9QIaITBeRS1oovpCRV+QiS/tAKKWCyEk7vBljyowxc4wxVwAdgbXAg04HFmryiyrI1FFclVJBxKce0caYI8aYvxtjLnQqoFBUV2fYr72olVJBRofMaAGHyiqprjU6D4RSKqhogmgBOg+EUioYaYJoAfV9IHQcJqVUMNEE0QLyjtUgNEEopYKHJogWkF9UQXREGKnxUYEORSmlTpkmiBZg54GIQUQ7ySmlgocmiBagEwUppYKRJogWkF+oneSUUsHH0QQhIuNF5GsRyRWRE3pfi0iKiMwXkXUiskJE+nusayMi80Rki4hsFpFznIzVKbV1hoKSSh1mQykVdBxLECISDjwLfAfoC9wkIn0bFHsIWGuMGYCdhOhpj3VPAx8YY3oDA4HNTsXqpAMlLmrrjNYglFJBx8kaRA6Qa4zZboypAuYCVzUo0xdYCGCM2QJ0FZEMEUnCTkz0gntdlTGm0MFYHVPfxFVrEEqpYCPGGGfeWGQCMN4YM9n9+lZghDFmmkeZx4EYY8xPRCQHWIqd5rQWmAlswtYeVgP3GmPKvOxnCjAFICMjY+jcuXObFW9paSkJCQnN2rYpK/Jr+NtXlfz23Fg6Jfqej52K63RpXL7RuHyjcfnmdOIaO3bsamPMMK8rjTGOPIDrgec9Xt8K/KVBmSTgJewIsbOBldiEMAyowSYUsJebfnuyfQ4dOtQ016JFi5q9bVNm/neb6fLAe6awvKpZ2zsV1+nSuHyjcflG4/LN6cQFrDKNnFObnJP6NO0FOnm87gjkeRYwxhQDdwCI7SSww/2IA/YaY5a7i84jSIcYzyuqID4qnKQYJw+1Ukr5n5P3IFYC2SLSTUSisPNbv+tZwN1Sqb578WRgiTGm2BizH9gjIme5143DXm4KOvmFLjLbxGonOaVU0HHsa60xpkZEpgEfAuHAi8aYjSIy1b1+BtAHmCUitdgEMMnjLe4G5rgTyHbcNY1gk19UoWMwKaWCkqPXPYwxC4AFDZbN8Hi+DMhuZNu12HsRQS2vyEXv9kmBDkMppXymPakdVFVTx6HSSu0DoZQKSpogHFRQ7MIYHeZbKRWcNEE4KL9IZ5JTSgUvTRAOqp9JTueiVkoFI00QDsrTuaiVUkFME4SD8osqSIqJID5aO8kppYKPJggH5RW6yGqjtQelVHDSBOEg7SSnlApmmiAclF9kh9lQSqlgpAnCIa7qWo6UVZGlNQilVJDSBOEQ7QOhlAp2miAckl9o+0DoMBtKqWClCcIheUU61ahSKrhpgnBIfQ2ivd6DUEoFKU0QDskrcpEWH0VMZHigQ1FKqWbRBOGQ/KIKvf+glApqmiAcsr/IRfskvf+glApemiAckldYoaO4KqWCmiYIB5RV1lDsqtE+EEqpoKYJwgE6D4RSKhRognCAzgOhlAoFmiAcUF+D0JFclVLBTBOEA/IKXYhoJzmlVHDTBOGA/KIK2iZEExmuh1cpFbz0DOYAnQdCKRUKNEE4IK+wQueBUEoFPU0QfmaMsTUIbcGklApyjiYIERkvIl+LSK6IPOhlfYqIzBeRdSKyQkT6e6zbKSLrRWStiKxyMk5/Kq6oobyqVvtAKKWCXoRTbywi4cCzwMXAXmCliLxrjNnkUewhYK0x5hoR6e0uP85j/VhjzCGnYnRC3rEmrlqDUEoFNydrEDlArjFmuzGmCpgLXNWgTF9gIYAxZgvQVUQyHIzJccf6QGgNQikV5JxMEB2APR6v97qXefoKuBZARHKALkBH9zoDfCQiq0VkioNx+lW+ziSnlAoRYoxx5o1FrgcuNcZMdr++FcgxxtztUSYJeBoYDKwHegOTjTFfiUiWMSZPRNoBHwN3G2OWeNnPFGAKQEZGxtC5c+c2K97S0lISEhKata2nN7+p4t87qnn+kjjCRE77/fwVl79pXL7RuHyjcfnmdOIaO3bsamPMMK8rjTGOPIBzgA89Xv8C+EUT5QXYCSR5Wfcr4Gcn2+fQoUNNcy1atKjZ23q6/7UvzTmPf+KX9zLGf3H5m8blG43LNxqXb04nLmCVaeSc6uQlppVAtoh0E5EoYCLwrmcBEWnjXgcwGVhijCkWkXgRSXSXiQcuATY4GKvf5BdqJzmlVGhwrBWTMaZGRKYBHwLhwIvGmI0iMtW9fgbQB5glIrXAJmCSe/MMYL7YSzQRwCvGmA+citWf8osq6N8hOdBhKKXUaXMsQQAYYxYACxosm+HxfBmQ7WW77cBAJ2NzgnF3krukX/tAh6KUUqdNe1L70ZGyKipr6nSYb6VUSNAE4Uf1TVy1k5xSKhRogvCjvEKdalQpFTo0QfiR1iCUUqFEE4Qf5RVVEBUeRlp81MkLK6VUK6cJwo/yC120T44hLOz0e1ArpVSgaYLwo/yiCm3BpJQKGZog/Civ0EWW9qJWSoUITRB+UldnKCh2aQ1CKRUyNEH4yaHSSmrqjI7DpJQKGZog/CSvvolrktYglFKhQROEn+QX6kxySqnQognCT/J0JjmlVIjRBOEn+YUVxESG0SYuMtChKKWUX2iC8JP8IhdZybGIH6YZVUqp1kAThJ/kFVXo/QelVEjRBOEn+YUuHaRPKRVSNEH4QU1tHQdKXGRpJzmlVAjRBOEHBSWV1Bm0k5xSKqRogvCDY30gtAahlAohmiD84FgfCK1BKKVCiCYIP9AahFIqFGmC8IP8IheJ0REkxmgnOaVU6NAE4Qf52gdCKRWCNEH4QX6R9oFQSoUeTRB+kFeoEwUppUKPJojTVFlTy6HSSq1BKKVCjiaI01RQVAnoPBBKqdDjaIIQkfEi8rWI5IrIg17Wp4jIfBFZJyIrRKR/g/XhIvKliLznZJynI6/INnHVeSCUUqHGsQQhIuHAs8B3gL7ATSLSt0Gxh4C1xpgBwPeBpxusvxfY7FSM/pBfpDPJKaVCk5M1iBwg1xiz3RhTBcwFrmpQpi+wEMAYswXoKiIZACLSEbgceN7BGE9bXqHOJKeUCk1ijHHmjUUmAOONMZPdr28FRhhjpnmUeRyIMcb8RERygKXuMqtFZB7wv0Ai8DNjzBWN7GcKMAUgIyNj6Ny5c5sVb2lpKQkJCT5vN2tjJcv31/DsuPhm7fdkmhuX0zQu32hcvtG4fHM6cY0dO3a1MWaY15XGGEcewPXA8x6vbwX+0qBMEvASsBaYDawEBgJXAH9zlxkDvHcq+xw6dKhprkWLFjVrux+8tMKM//OSZu/3ZJobl9M0Lt9oXL7RuHxzOnEBq0wj59SIZqWcU7MX6OTxuiOQ51nAGFMM3AEgdq7OHe7HROBKEbkMiAGSROSfxphbHIy3WQZ0bEO/rLpAh6GUUn7nZIJYCWSLSDdgH/ak/z3PAiLSBig39h7FZGCJO2n8wv1ARMZgLzG1uuQAcO9F2YEOQSmlHOFYgjDG1IjINOBDIBx40RizUUSmutfPAPoAs0SkFtgETHIqHqWUUr5xsgaBMWYBsKDBshkez5cBTX4FN8YsBhY7EJ5SSqkmaE9qpZRSXmmCUEop5ZUmCKWUUl5pglBKKeWVJgillFJeaYJQSinllWNjMQWCiBwEdjVz83TgkB/D8ReNyzcal280Lt+EYlxdjDFtva0IqQRxOkRklWlswKoA0rh8o3H5RuPyzZkWl15iUkop5ZUmCKWUUl5pgvjWzEAH0AiNyzcal280Lt+cUXHpPQillFJeaQ1CKaWUV5oglFJKeXVGJQgRGS8iX4tIrog86GW9iMgz7vXrRGRIC8XVSUQWichmEdkoIvd6KTNGRIpEZK378csWim2niKx373OVl/UtfsxE5CyP47BWRIpF5L4GZVrkeInIiyJyQEQ2eCxLFZGPRWSr+2dKI9s2+Xl0IK4/isgW999pvnvCLm/bNvk3dyCuX4nIPo+/1WWNbNvSx+s1j5h2isjaRrZ18nh5PTe02GessblIQ+2BnbRoG9AdiAK+Avo2KHMZ8D4gwEhgeQvFlgkMcT9PBL7xEtsYTnFubj/HthNIb2J9QI5Zg7/rfmxnnxY/XsD5wBBgg8eyPwAPup8/CPy+OZ9HB+K6BIhwP/+9t7hO5W/uQFy/ws4aebK/c4serwbr/wT8MgDHy+u5oaU+Y2dSDSIHyDXGbDd2itO5wFUNylwFzDLWF0AbEcl0OjBjTL4xZo37eQmwGejg9H79JCDHzMM4YJsxprk96E+LMWYJcKTB4quAf7if/wO42sump/J59GtcxpiPjDE17pdfYOeJb1GNHK9T0eLHq56ICHAD8Kq/9neqmjg3tMhn7ExKEB2APR6v93LiSfhUyjhKRLoCg4HlXlafIyJficj7ItKvhUIywEcislpEpnhZH+hjNpHG/3EDcbwAMowx+WD/wYF2XsoE+rj9AFvz8+Zkf3MnTHNf+nqxkcslgTxe5wEFxpitjaxvkePV4NzQIp+xMylBiJdlDdv4nkoZx4hIAvAmcJ8xprjB6jXYyygDgb8Ab7dQWOcaY4YA3wF+LCLnN1gfsGMmIlHAlcAbXlYH6nidqkAet4eBGmBOI0VO9jf3t+lAD2AQkI+9nNNQIP83b6Lp2oPjx+sk54ZGN/OyzKdjdiYliL1AJ4/XHYG8ZpRxhIhEYj8Ac4wxbzVcb4wpNsaUup8vACJFJN3puIwxee6fB4D52Gqrp4AdM+w/5BpjTEHDFYE6Xm4F9ZfZ3D8PeCkTkOMmIrcBVwA3G/eF6oZO4W/uV8aYAmNMrTGmDniukf0F6nhFANcCrzVWxunj1ci5oUU+Y2dSglgJZItIN/c3z4nAuw3KvAt8390yZyRQVF+Nc5L7GucLwGZjzFONlGnvLoeI5GD/docdjiteRBLrn2Nvcm5oUCwgx8yt0W92gTheHt4FbnM/vw14x0uZU/k8+pWIjAceAK40xpQ3UuZU/ub+jsvzntU1jeyvxY+X20XAFmPMXm8rnT5eTZwbWuYz5sSd99b6wLa4+QZ7Z/9h97KpwFT3cwGeda9fDwxrobhGY6t+64C17sdlDWKbBmzEtkT4AhjVAnF1d+/vK/e+W9Mxi8Oe8JM9lrX48cImqHygGvuNbRKQBiwEtrp/prrLZgELmvo8OhxXLvaadP1nbEbDuBr7mzsc12z3Z2cd9gSW2RqOl3v5y/WfKY+yLXm8Gjs3tMhnTIfaUEop5dWZdIlJKaWUDzRBKKWU8koThFJKKa80QSillPJKE4RSSimvNEEo5QMRqZXjR5L126iiItLVczRRpQItItABKBVkKowxgwIdhFItQWsQSvmBe06A34vICvejp3t5FxFZ6B6IbqGIdHYvzxA7J8NX7sco91uFi8hz7rH/PxKR2ID9UuqMpwlCKd/ENrjEdKPHumJjTA7wV+DP7mV/xQ6HPgA7ON4z7uXPAP81djDBIdheuADZwLPGmH5AIXCdo7+NUk3QntRK+UBESo0xCV6W7wQuNMZsdw+utt8YkyYih7BDR1S7l+cbY9JF5CDQ0RhT6fEeXYGPjTHZ7tcPAJHGmN+1wK+m1Am0BqGU/5hGnjdWxptKj+e16H1CFUCaIJTynxs9fi5zP1+KHUUT4GbgM/fzhcBdACISLiJJLRWkUqdKv50o5ZtYOX7y+g+MMfVNXaNFZDn2i9dN7mX3AC+KyM+Bg8Ad7uX3AjNFZBK2pnAXdjRRpVoNvQehlB+470EMM8YcCnQsSvmLXmJSSinlldYglFJKeaU1CKWUUl5pglBKKeWVJgillFJeaYJQSinllSYIpZRSXv1/MCXeRJthYt4AAAAASUVORK5CYII=\n",
"text/plain": [
"<Figure size 432x288 with 1 Axes>"
]
},
"metadata": {
"needs_background": "light"
},
"output_type": "display_data"
}
],
"source": [
"import pandas as pd\n",
"\n",
"\n",
"metrics = pd.read_csv(f\"{trainer.logger.log_dir}/metrics.csv\")\n",
"\n",
"aggreg_metrics = []\n",
"agg_col = \"epoch\"\n",
"for i, dfg in metrics.groupby(agg_col):\n",
" agg = dict(dfg.mean())\n",
" agg[agg_col] = i\n",
" aggreg_metrics.append(agg)\n",
"\n",
"df_metrics = pd.DataFrame(aggreg_metrics)\n",
"df_metrics[[\"train_loss\", \"valid_loss\"]].plot(\n",
" grid=True, legend=True, xlabel='Epoch', ylabel='Loss')\n",
"df_metrics[[\"train_acc\", \"valid_acc\"]].plot(\n",
" grid=True, legend=True, xlabel='Epoch', ylabel='ACC')"
]
},
{
"cell_type": "markdown",
"id": "e1cdb768",
"metadata": {},
"source": [
"- The `trainer` automatically saves the model with the best validation accuracy automatically for us, we which we can load from the checkpoint via the `ckpt_path='best'` argument; below we use the `trainer` instance to evaluate the best model on the test set:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"id": "71cba03c",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"Restoring states from the checkpoint path at logs/my-model/version_14/checkpoints/epoch=17-step=7721.ckpt\n",
"LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0]\n",
"Loaded model weights from checkpoint at logs/my-model/version_14/checkpoints/epoch=17-step=7721.ckpt\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a851c4422bec4dd388a0a4d586fc1c2a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"Testing: 0it [00:00, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/html": [
"<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃<span style=\"font-weight: bold\"> Test metric </span>┃<span style=\"font-weight: bold\"> DataLoader 0 </span>┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│<span style=\"color: #008080; text-decoration-color: #008080\"> test_acc </span>│<span style=\"color: #800080; text-decoration-color: #800080\"> 0.9797000288963318 </span>│\n",
"└───────────────────────────┴───────────────────────────┘\n",
"</pre>\n"
],
"text/plain": [
"┏━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━┓\n",
"┃\u001b[1m \u001b[0m\u001b[1m Test metric \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m DataLoader 0 \u001b[0m\u001b[1m \u001b[0m┃\n",
"┡━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━┩\n",
"│\u001b[36m \u001b[0m\u001b[36m test_acc \u001b[0m\u001b[36m \u001b[0m│\u001b[35m \u001b[0m\u001b[35m 0.9797000288963318 \u001b[0m\u001b[35m \u001b[0m│\n",
"└───────────────────────────┴───────────────────────────┘\n"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/plain": [
"[{'test_acc': 0.9797000288963318}]"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"trainer.test(model=lightning_model, datamodule=data_module, ckpt_path='best')"
]
},
{
"cell_type": "markdown",
"id": "048fb8ed",
"metadata": {},
"source": [
"## Predicting labels of new data"
]
},
{
"cell_type": "markdown",
"id": "0a469bbd",
"metadata": {},
"source": [
"- You can use the `trainer.predict` method on a new `DataLoader` or `DataModule` to apply the model to new data.\n",
"- Alternatively, you can also manually load the best model from a checkpoint as shown below:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"id": "4493c25d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"logs/my-model/version_14/checkpoints/epoch=17-step=7721.ckpt\n"
]
}
],
"source": [
"path = trainer.checkpoint_callback.best_model_path\n",
"print(path)"
]
},
{
"cell_type": "code",
"execution_count": 16,
"id": "687cd03a",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"3:23: E703 statement ends with a semicolon\n"
]
}
],
"source": [
"pytorch_model = PyTorchCNN(num_classes=10)\n",
"lightning_model = LightningCNN.load_from_checkpoint(path, model=pytorch_model)\n",
"lightning_model.eval();"
]
},
{
"cell_type": "markdown",
"id": "b180bb27",
"metadata": {},
"source": [
"- Note that our PyTorch model, which is passed to the Lightning model requires input arguments. However, this is automatically being taken care of since we used `self.save_hyperparameters()` in our PyTorch model's `__init__` method.\n",
"- Now, below is an example applying the model manually. Here, pretend that the `test_dataloader` is a new data loader."
]
},
{
"cell_type": "code",
"execution_count": 17,
"id": "74c6c831",
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tensor([7, 2, 1, 0, 4])"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"test_dataloader = data_module.test_dataloader()\n",
"\n",
"all_true_labels = []\n",
"all_predicted_labels = []\n",
"for batch in test_dataloader:\n",
" features, labels = batch\n",
" \n",
" with torch.no_grad(): # since we don't need to backprop\n",
" logits = lightning_model(features)\n",
" \n",
" predicted_labels = torch.argmax(logits, dim=1)\n",
" all_predicted_labels.append(predicted_labels)\n",
" all_true_labels.append(labels)\n",
" \n",
"all_predicted_labels = torch.cat(all_predicted_labels)\n",
"all_true_labels = torch.cat(all_true_labels)\n",
"all_predicted_labels[:5]"
]
},
{
"cell_type": "markdown",
"id": "324643e8",
"metadata": {},
"source": [
"Just as an internal check, if the model was loaded correctly, the test accuracy below should be identical to the test accuracy we saw earlier in the previous section."
]
},
{
"cell_type": "code",
"execution_count": 18,
"id": "ed5e1d1d",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Test accuracy: 0.9797 (97.97%)\n"
]
}
],
"source": [
"test_acc = torch.mean((all_predicted_labels == all_true_labels).float())\n",
"print(f'Test accuracy: {test_acc:.4f} ({test_acc*100:.2f}%)')"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 5
}
在新工单中引用 查看 Git Blame 复制永久链接