lectures/07_lecture-naive-Bayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# DSCI 571: Supervised Machine Learning I \n",
    "\n",
    "## Lecture 7: Naive Bayes\n",
    "\n",
    "UBC Master of Data Science program, 2020-21\n",
    "\n",
    "Instructor: Varada Kolhatkar\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Populating the interactive namespace from numpy and matplotlib\n"
     ]
    }
   ],
   "source": [
    "# And import the libraries\n",
    "import IPython\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import HTML\n",
    "\n",
    "%pylab inline\n",
    "# pip install git+git://github.com/mgelbart/plot-classifier.git\n",
    "from plot_classifier import plot_classifier\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.feature_extraction.text import (\n",
    "    CountVectorizer,\n",
    "    TfidfTransformer,\n",
    "    TfidfVectorizer,\n",
    ")\n",
    "\n",
    "# train test split and cross validation\n",
    "from sklearn.model_selection import (\n",
    "    GridSearchCV,\n",
    "    cross_val_score,\n",
    "    cross_validate,\n",
    "    train_test_split,\n",
    ")\n",
    "\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "\n",
    "%matplotlib inline\n",
    "from sklearn.naive_bayes import MultinomialNB, BernoulliNB, GaussianNB\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", 200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Lecture learning objectives\n",
    "\n",
    "From this lecture, you will be able to \n",
    "\n",
    "- Explain the naive assumption of naive Bayes. \n",
    "- Predict targets by hand on toy examples using naive Bayes.\n",
    "- Use `scikit-learn`'s `MultiNomialNB`, `BernoulliNB`, and `GaussianNB`. \n",
    "- Use `predict_proba` for different classifiers and explain its usefulness. \n",
    "- Explain the need of smoothing in naive Bayes.\n",
    "- Explain how `alpha` controls the fundamental tradeoff. \n",
    "- Use naive Bayes for multi-class classification. \n",
    "- Name advantages and disadvantages of naive Bayes. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 1. Motivating example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "### Example: spam/non spam\n",
    "Last week in your lab you worked on spam classification using `SVC` classifier. \n",
    "\n",
    "$X = \\begin{bmatrix}\\text{\"URGENT!! You have been selected to receive a £900 prize reward!\",}\\\\ \\text{\"Lol your always so convincing.\"}\\\\ \\text{\"Congrats! 1 year special cinema pass for 2 is yours. call 09061209465 now!\"}\\\\ \\end{bmatrix}$ and $y = \\begin{bmatrix}\\text{spam} \\\\ \\text{non spam} \\\\ \\text{spam} \\end{bmatrix}$\n",
    "\n",
    "\n",
    "ML algorithms we have seen so far prefer fixed length numeric input that looks like this: \n",
    "\n",
    "$X = \\begin{bmatrix}1.0 & 4.0 & \\ldots & & 3.0\\\\ 0.0 & 2.0 & \\ldots & & 6.0\\\\ 1.0 & 0.0 & \\ldots & & 0.0\\\\ \\end{bmatrix}$ and $y = \\begin{bmatrix}\\text{spam} \\\\ \\text{non spam} \\\\ \\text{spam} \\end{bmatrix}$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### `CountVectorizer` to get bag-of-words (BOW) representation\n",
    "\n",
    "- So we used `CountVectorizer` to convert text data into feature vectors where\n",
    "    - each feature is a unique word in the text  \n",
    "    - each feature value represents the frequency or presence/absence of the word in the given message         \n",
    "    \n",
    "<img src='./images/bag-of-words.png' width=\"800\">\n",
    "\n",
    "[Source](https://web.stanford.edu/~jurafsky/slp3/4.pdf)       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Naive Bayes\n",
    "\n",
    "- For years, best spam filtering methods used naive Bayes.\n",
    "- Our first probabilistic classifier where we think of learning as a problem of statistical inference."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. Naive Bayes classifier\n",
    "\n",
    "\n",
    "Before understanding the theory, let's try `scikit-learn`'s implementation of Naive Bayes on Kaggle's [SMS Spam Collection Dataset](https://www.kaggle.com/uciml/sms-spam-collection-dataset)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "sms_df = pd.read_csv(\"data/spam.csv\", encoding=\"latin-1\")\n",
    "sms_df = sms_df.drop([\"Unnamed: 2\", \"Unnamed: 3\", \"Unnamed: 4\"], axis=1)\n",
    "sms_df = sms_df.rename(columns={\"v1\": \"target\", \"v2\": \"sms\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>target</th>\n",
       "      <th>sms</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>385</th>\n",
       "      <td>ham</td>\n",
       "      <td>It took Mr owl 3 licks</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4003</th>\n",
       "      <td>ham</td>\n",
       "      <td>Well there's a pattern emerging of my friends telling me to drive up and come smoke with them and then telling me that I'm a weed fiend/make them smoke too much/impede their doing other things so ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1283</th>\n",
       "      <td>ham</td>\n",
       "      <td>Yes i thought so. Thanks.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2327</th>\n",
       "      <td>spam</td>\n",
       "      <td>URGENT! Your mobile number *************** WON a å£2000 Bonus Caller prize on 10/06/03! This is the 2nd attempt to reach you! Call 09066368753 ASAP! Box 97N7QP, 150ppm</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1103</th>\n",
       "      <td>ham</td>\n",
       "      <td>Aiyah sorry lor... I watch tv watch until i forgot 2 check my phone.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     target  \\\n",
       "385     ham   \n",
       "4003    ham   \n",
       "1283    ham   \n",
       "2327   spam   \n",
       "1103    ham   \n",
       "\n",
       "                                                                                                                                                                                                          sms  \n",
       "385                                                                                                                                                                                    It took Mr owl 3 licks  \n",
       "4003  Well there's a pattern emerging of my friends telling me to drive up and come smoke with them and then telling me that I'm a weed fiend/make them smoke too much/impede their doing other things so ...  \n",
       "1283                                                                                                                                                                                Yes i thought so. Thanks.  \n",
       "2327                                  URGENT! Your mobile number *************** WON a å£2000 Bonus Caller prize on 10/06/03! This is the 2nd attempt to reach you! Call 09066368753 ASAP! Box 97N7QP, 150ppm  \n",
       "1103                                                                                                                                     Aiyah sorry lor... I watch tv watch until i forgot 2 check my phone.  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df, test_df = train_test_split(sms_df, test_size=0.2, random_state=123)\n",
    "X_train, y_train = train_df[\"sms\"], train_df[\"target\"]\n",
    "X_test, y_test = test_df[\"sms\"], test_df[\"target\"]\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def store_cross_val_results(model_name, scores, results_dict):\n",
    "    \"\"\"\n",
    "    Stores mean scores from cross_validate in results_dict for\n",
    "    the given model model_name.\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model_name :\n",
    "        scikit-learn classification model\n",
    "    scores : dict\n",
    "        object return by `cross_validate`\n",
    "    results_dict: dict\n",
    "        dictionary to store results\n",
    "\n",
    "    Returns\n",
    "    ----------\n",
    "        None\n",
    "\n",
    "    \"\"\"\n",
    "    results_dict[model_name] = {\n",
    "        \"mean_train_accuracy\": \"{:0.4f}\".format(np.mean(scores[\"train_score\"])),\n",
    "        \"mean_valid_accuracy\": \"{:0.4f}\".format(np.mean(scores[\"test_score\"])),\n",
    "        \"mean_fit_time (s)\": \"{:0.4f}\".format(np.mean(scores[\"fit_time\"])),\n",
    "        \"mean_score_time (s)\": \"{:0.4f}\".format(np.mean(scores[\"score_time\"])),\n",
    "        \"std_train_score\": \"{:0.4f}\".format(scores[\"train_score\"].std()),\n",
    "        \"std_valid_score\": \"{:0.4f}\".format(scores[\"test_score\"].std()),\n",
    "    }"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### SVC classifier for spam detection \n",
    "\n",
    "- Let's try SVC classifier on our dataset. (This is what you did in the lab last week.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_dict = {}\n",
    "pipe_svc= make_pipeline(CountVectorizer(), SVC())\n",
    "scores = cross_validate(pipe_svc, X_train, y_train, return_train_score=True)\n",
    "store_cross_val_results('SVC', scores, results_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_fit_time (s)</th>\n",
       "      <th>mean_score_time (s)</th>\n",
       "      <th>mean_train_accuracy</th>\n",
       "      <th>mean_valid_accuracy</th>\n",
       "      <th>std_train_score</th>\n",
       "      <th>std_valid_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SVC</th>\n",
       "      <td>0.5260</td>\n",
       "      <td>0.1126</td>\n",
       "      <td>0.9950</td>\n",
       "      <td>0.9785</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>0.0048</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    mean_fit_time (s) mean_score_time (s) mean_train_accuracy  \\\n",
       "SVC            0.5260              0.1126              0.9950   \n",
       "\n",
       "    mean_valid_accuracy std_train_score std_valid_score  \n",
       "SVC              0.9785          0.0005          0.0048  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `sklearn` naive Bayes classifier \n",
    "\n",
    "- Let's try Naive Bayes on this problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.naive_bayes import MultinomialNB\n",
    "\n",
    "pipe_nb = make_pipeline(CountVectorizer(), MultinomialNB())\n",
    "scores = cross_validate(pipe_nb, X_train, y_train, return_train_score=True)\n",
    "store_cross_val_results('Naive Bayes', scores, results_dict)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>mean_train_accuracy</th>\n",
       "      <th>mean_valid_accuracy</th>\n",
       "      <th>mean_fit_time (s)</th>\n",
       "      <th>mean_score_time (s)</th>\n",
       "      <th>std_train_score</th>\n",
       "      <th>std_valid_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SVC</th>\n",
       "      <td>0.9950</td>\n",
       "      <td>0.9785</td>\n",
       "      <td>0.5260</td>\n",
       "      <td>0.1126</td>\n",
       "      <td>0.0005</td>\n",
       "      <td>0.0048</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Naive Bayes</th>\n",
       "      <td>0.9935</td>\n",
       "      <td>0.9859</td>\n",
       "      <td>0.0422</td>\n",
       "      <td>0.0089</td>\n",
       "      <td>0.0007</td>\n",
       "      <td>0.0026</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            mean_train_accuracy mean_valid_accuracy mean_fit_time (s)  \\\n",
       "SVC                      0.9950              0.9785            0.5260   \n",
       "Naive Bayes              0.9935              0.9859            0.0422   \n",
       "\n",
       "            mean_score_time (s) std_train_score std_valid_score  \n",
       "SVC                      0.1126          0.0005          0.0048  \n",
       "Naive Bayes              0.0089          0.0007          0.0026  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The validation scores are a bit better on this particular dataset.\n",
    "- Way more faster than SVC classifier!! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Naive Bayes `predict`\n",
    "\n",
    "- Given a new message, we want to predict whether it's spam or non spam (ham).\n",
    "- Example: Predict whether the following message is spam or non spam (ham). \n",
    "> \"URGENT! Free!!\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['spam', 'ham'], dtype='<U4')"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_nb = make_pipeline(CountVectorizer(), MultinomialNB())\n",
    "pipe_nb.fit(X_train, y_train)\n",
    "deploy_test = [\"URGENT! Free!!\", \"Let's enjoy the last week of block 2!\"]\n",
    "pipe_nb.predict(deploy_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Probabilistic classifiers: `predict` by hand \n",
    "\n",
    "- What's it's doing under the hood? \n",
    "- Let's look at an example with a toy dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "X = [\n",
    "    \"URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!\",\n",
    "    \"Lol you are always so convincing.\",\n",
    "    \"Block 2 has interesting courses.\",\n",
    "    \"URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!\",\n",
    "    \"Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!\",\n",
    "    \"Block 2 has been interesting so far.\",\n",
    "]\n",
    "y = [\"spam\", \"non spam\", \"non spam\", \"spam\", \"spam\", \"non spam\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe_nb_toy = make_pipeline(CountVectorizer(max_features = 4, stop_words='english'), MultinomialNB())\n",
    "pipe_nb_toy.fit(X, y);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>block</th>\n",
       "      <th>free</th>\n",
       "      <th>prize</th>\n",
       "      <th>urgent</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has interesting courses.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has been interesting so far.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                              block  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      0   \n",
       "Lol you are always so convincing.                                                                                 0   \n",
       "Block 2 has interesting courses.                                                                                  1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      0   \n",
       "Block 2 has been interesting so far.                                                                              1   \n",
       "\n",
       "                                                                                                              free  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                     0   \n",
       "Lol you are always so convincing.                                                                                0   \n",
       "Block 2 has interesting courses.                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!     1   \n",
       "Block 2 has been interesting so far.                                                                             0   \n",
       "\n",
       "                                                                                                              prize  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      1   \n",
       "Lol you are always so convincing.                                                                                 0   \n",
       "Block 2 has interesting courses.                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      0   \n",
       "Block 2 has been interesting so far.                                                                              0   \n",
       "\n",
       "                                                                                                              urgent  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                       1   \n",
       "Lol you are always so convincing.                                                                                  0   \n",
       "Block 2 has interesting courses.                                                                                   0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                        1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!       0   \n",
       "Block 2 has been interesting so far.                                                                               0   \n",
       "\n",
       "                                                                                                                target  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      spam  \n",
       "Lol you are always so convincing.                                                                             non spam  \n",
       "Block 2 has interesting courses.                                                                              non spam  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       spam  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      spam  \n",
       "Block 2 has been interesting so far.                                                                          non spam  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pipe_nb_toy['countvectorizer'].transform(X)\n",
    "train_bow_df = pd.DataFrame(data.toarray(), columns=pipe_nb_toy['countvectorizer'].get_feature_names(), index=X)\n",
    "train_bow_df['target'] = y\n",
    "train_bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Suppose we are given text messages in `deploy_test` and we want to find the targets for these examples, how do we do it using naive Bayes?\n",
    "\n",
    "First, let's get numeric representation of our text messages. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>block</th>\n",
       "      <th>free</th>\n",
       "      <th>prize</th>\n",
       "      <th>urgent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT! Free!!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>I like block 2.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                 block  free  prize  urgent\n",
       "URGENT! Free!!       0     1      0       1\n",
       "I like block 2.      1     0      0       0"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deploy_test = [\"URGENT! Free!!\", \"I like block 2.\"]\n",
    "data = pipe_nb_toy['countvectorizer'].transform(deploy_test).toarray()\n",
    "bow_df = pd.DataFrame(data, columns=pipe_nb_toy['countvectorizer'].get_feature_names(), index=deploy_test)\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Naive Bayes prediction idea\n",
    "\n",
    "Suppose we want to predict whether the following message is \"spam\" or \"non spam\".\n",
    "> \"URGENT! Free!!\"\n",
    "\n",
    "Representation of the message: `[0, 1, 0, 1]`\n",
    "\n",
    "To predict the correct class, naive Bayes calculates the following probability scores. \n",
    "\n",
    "- $P(\\text{spam} \\mid \\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent=1})$ \n",
    "- $P(\\text{non spam} \\mid  \\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent=1})$\n",
    "- Picks the label with higher probability scores. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Applying Bayes' theorem \n",
    "\n",
    "Uses Bayes' theorem to calculate probabilities:\n",
    "\n",
    "$$P(A \\mid B) = \\frac{P(B \\mid A) \\times P(A)}{P(B)}$$\n",
    "\n",
    "$$P(\\text{spam} \\mid \\text{message})= \\frac{P(\\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent} = 1 \\mid \\text{spam}) \\times P(\\text{spam})}{P(\\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent=1})}$$\n",
    "\n",
    "$$P(\\text{non spam} \\mid \\text{message}) = \\frac{P(\\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent} = 1 \\mid \\text{non spam}) \\times P( \\text{non spam})}{P(\\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent=1})}$$\n",
    "\n",
    "- $P(\\text{message})$: marginal probability that a message has the given set of words \n",
    "    - Hard to calculate but can be ignored in our scenario as it occurs in the denominator for both $P(\\text{spam} \\mid \\text{message})$ and $P(\\text{non spam} \\mid \\text{message})$.\n",
    "    - So we ignore the denominator in both cases. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Let's focus on $P(\\text{spam} \\mid \\text{message})$\n",
    "\n",
    "- After ignoring the denominator: \n",
    "$$P(\\text{spam} \\mid \\text{message}) \\propto P(\\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent} = 1 \\mid \\text{spam}) \\times P(\\text{spam})$$\n",
    "\n",
    "- To calculate $P(\\text{spam} \\mid \\text{message})$, we need:  \n",
    "    - $P(\\text{spam})$: marginal probability that a message is spam\n",
    "    - $P(\\text{message}\\mid\\text{spam})$: conditional probability that message has words $w_1, w_2, \\dots, w_d$, given that it is spam.\n",
    "        - Hard to calculate because it would require huge numbers of parameters and impossibly large training sets. But we need it. \n",
    "        - with $d$ binary features, how many possible \"text messages\" are there?\n",
    "        - we cannot possibly have access to all the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Naive Bayes' approximation to calculate $P(\\text{message}|\\text{spam})$\n",
    "\n",
    "- A common assmption is **naive Bayes** assumption, which states that **features are independent, conditioned on the target**. \n",
    "    - Example: In our spam classification example, **once you know that a message is spam**, the probability that the word \"urgent\" appears is independent of whether \"free\" also appeared. \n",
    "    \n",
    "- We can write this mathematically as \n",
    "\n",
    "$$\\begin{equation}\n",
    "\\begin{split}\n",
    "& P(\\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent} = 1 \\mid \\text{spam}) \\\\\n",
    "&\\approx P(\\text{block} = 0 \\mid \\text{spam}) \\times P(\\text{free} = 1 \\mid \\text{spam}) \\times P(\\text{prize} = 0 \\mid \\text{spam}) \\times P(\\text{urgent} = 1 \\mid \\text{spam})\n",
    "\\end{split}\n",
    "\\end{equation}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Naive Bayes' approximation\n",
    "\n",
    "- In general, \n",
    "$$P(\\text{message} \\mid \\text{spam}) = P(w_1, w_2, . . . , w_d \\mid \\text{spam}) \\approx \\prod_{i=1}^{d}P(w_i \\mid \\text{spam})$$\n",
    "\n",
    "$$P(\\text{message} \\mid \\text{non spam}) = P(w_1, w_2, . . . , w_d \\mid \\text{non spam}) \\approx \\prod_{i=1}^{d}P(w_i \\mid \\text{non spam})$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Going back to estimating $P(\\text{spam} \\mid \\text{message})$\n",
    "\n",
    "With naive Bayes' assumption, to calculate $P(\\text{spam} \\mid \\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent} = 1)$, we need the following:  \n",
    "1. Prior probability: $P(\\text{spam})$ \n",
    "2. Conditional probabilities: \n",
    "    1. $P(\\text{block} = 0 \\mid \\text{spam})$\n",
    "    2. $P(\\text{free} = 1 \\mid \\text{spam})$\n",
    "    3. $P(\\text{prize} = 0 \\mid \\text{spam})$\n",
    "    4. $P(\\text{urgent} = 1 \\mid \\text{spam})$\n",
    "\n",
    "We use our training data to calculate these probabilities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>block</th>\n",
       "      <th>free</th>\n",
       "      <th>prize</th>\n",
       "      <th>urgent</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has interesting courses.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has been interesting so far.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                              block  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      0   \n",
       "Lol you are always so convincing.                                                                                 0   \n",
       "Block 2 has interesting courses.                                                                                  1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      0   \n",
       "Block 2 has been interesting so far.                                                                              1   \n",
       "\n",
       "                                                                                                              free  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                     0   \n",
       "Lol you are always so convincing.                                                                                0   \n",
       "Block 2 has interesting courses.                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!     1   \n",
       "Block 2 has been interesting so far.                                                                             0   \n",
       "\n",
       "                                                                                                              prize  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      1   \n",
       "Lol you are always so convincing.                                                                                 0   \n",
       "Block 2 has interesting courses.                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      0   \n",
       "Block 2 has been interesting so far.                                                                              0   \n",
       "\n",
       "                                                                                                              urgent  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                       1   \n",
       "Lol you are always so convincing.                                                                                  0   \n",
       "Block 2 has interesting courses.                                                                                   0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                        1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!       0   \n",
       "Block 2 has been interesting so far.                                                                               0   \n",
       "\n",
       "                                                                                                                target  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      spam  \n",
       "Lol you are always so convincing.                                                                             non spam  \n",
       "Block 2 has interesting courses.                                                                              non spam  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       spam  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      spam  \n",
       "Block 2 has been interesting so far.                                                                          non spam  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Prior probability\n",
    "    - $P(\\text{spam}) = 3/6$\n",
    "    \n",
    "- Conditional probabilities\n",
    "    - What is $P(\\text{block} = 0 \\mid \\text{spam})$? \n",
    "        - Given target is spam, how often \"block\" = 0? $3/3$\n",
    "    - $P(\\text{free} = 1 \\mid \\text{spam}) = 2/3$ \n",
    "    - $P(\\text{prize} = 0 \\mid \\text{spam}) = 1/3$\n",
    "    - $P(\\text{urgent} = 1 \\mid \\text{spam}) = 2/3$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Estimating $P(\\text{spam} \\mid \\text{message})$\n",
    "\n",
    "$$\\begin{equation}\n",
    "\\begin{split}\n",
    "P(\\text{spam} \\mid \\text{message}) &\\propto P(\\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent} = 1 \\mid \\text{spam}) \\times P(\\text{spam})\\\\\n",
    "&\\propto P(\\text{block} = 0 \\mid \\text{spam}) \\times P(\\text{free} = 1 \\mid \\text{spam}) \\\\\n",
    "& \\times P(\\text{prize} = 0 \\mid \\text{spam}) \\times P(\\text{urgent} = 1 \\mid \\text{spam}) \\times P(\\text{spam})\\\\\n",
    "&\\propto 3/3 \\times 2/3 \\times 1/3 \\times 2/3 \\times 3/6\\\\\n",
    "\\end{split}\n",
    "\\end{equation}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.07407407407407407"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spam_prior = 3/6\n",
    "block0_spam = 3/3\n",
    "free1_spam = 2/3\n",
    "prize0_spam = 1/3\n",
    "urgent1_spam = 2/3\n",
    "spam_prior * block0_spam * free1_spam * prize0_spam * urgent1_spam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Let's estimate $P(\\text{non spam} \\mid \\text{message})$\n",
    "\n",
    "With naive Bayes' assumption, to calculate $P(\\text{non spam} \\mid \\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent} = 1)$, we need the following:  \n",
    "1. Prior probability: $P(\\text{non spam})$ \n",
    "2. Conditional probabilities: \n",
    "    1. $P(\\text{block} = 0 \\mid \\text{non spam})$\n",
    "    2. $P(\\text{free} = 1 \\mid \\text{non spam})$\n",
    "    3. $P(\\text{prize} = 0 \\mid \\text{non spam})$\n",
    "    4. $P(\\text{urgent} = 1 \\mid \\text{non spam})$\n",
    "\n",
    "Again we use the data to calculate these probabilities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>block</th>\n",
       "      <th>free</th>\n",
       "      <th>prize</th>\n",
       "      <th>urgent</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has interesting courses.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has been interesting so far.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                              block  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      0   \n",
       "Lol you are always so convincing.                                                                                 0   \n",
       "Block 2 has interesting courses.                                                                                  1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      0   \n",
       "Block 2 has been interesting so far.                                                                              1   \n",
       "\n",
       "                                                                                                              free  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                     0   \n",
       "Lol you are always so convincing.                                                                                0   \n",
       "Block 2 has interesting courses.                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!     1   \n",
       "Block 2 has been interesting so far.                                                                             0   \n",
       "\n",
       "                                                                                                              prize  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      1   \n",
       "Lol you are always so convincing.                                                                                 0   \n",
       "Block 2 has interesting courses.                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      0   \n",
       "Block 2 has been interesting so far.                                                                              0   \n",
       "\n",
       "                                                                                                              urgent  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                       1   \n",
       "Lol you are always so convincing.                                                                                  0   \n",
       "Block 2 has interesting courses.                                                                                   0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                        1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!       0   \n",
       "Block 2 has been interesting so far.                                                                               0   \n",
       "\n",
       "                                                                                                                target  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      spam  \n",
       "Lol you are always so convincing.                                                                             non spam  \n",
       "Block 2 has interesting courses.                                                                              non spam  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       spam  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      spam  \n",
       "Block 2 has been interesting so far.                                                                          non spam  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Prior probability \n",
    "    - $P(\\text{non spam}) = 3/6$\n",
    "\n",
    "- Conditional probabilities \n",
    "    - What is $P(\\text{block} = 0 \\mid \\text{non spam})$? \n",
    "        - Given target is non spam, how often \"block\" = 0? $1/3$\n",
    "    - $P(\\text{free} = 1 \\mid \\text{non spam}) = 0/3$ \n",
    "    - $P(\\text{prize} = 0 \\mid \\text{non spam}) = 3/3$\n",
    "    - $P(\\text{urgent} = 1 \\mid \\text{non spam}) = 0/3$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Estimating $P(\\text{non spam} \\mid \\text{message})$\n",
    "\n",
    "$$\\begin{equation}\n",
    "\\begin{split}\n",
    "P(\\text{non spam} \\mid \\text{message}) &\\propto P(\\text{block} = 0, \\text{free} = 1, \\text{prize} = 0, \\text{urgent} = 1 \\mid \\text{non spam}) \\times P(\\text{non spam})\\\\\n",
    "&\\propto P(\\text{block} = 0 \\mid \\text{non spam}) \\times P(\\text{free} = 1 \\mid \\text{non spam}) \\\\\n",
    "& \\times P(\\text{prize} = 0 \\mid \\text{non spam}) \\times P(\\text{urgent} = 1 \\mid \\text{non spam}) \\times P(\\text{non spam})\\\\\n",
    "&\\propto 1/3 \\times 0 \\times 3/3 \\times 0 \\times 1/3\\\\\n",
    "\\end{split}\n",
    "\\end{equation}$$\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "non_spam_prior = 3/6\n",
    "block0_non_spam = 0/3\n",
    "free1_non_spam = 1/3\n",
    "prize0_non_spam = 1/3\n",
    "urgent1_non_spam = 2/3\n",
    "non_spam_prior * block0_non_spam * free1_non_spam * prize0_non_spam * urgent1_non_spam"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Bayes prediction\n",
    "\n",
    "Since $(\\text{spam} \\mid \\text{message})$ (0.074) is proportional to a larger number compared to $(\\text{non spam} \\mid \\text{message})$ (0), we predict $spam$! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Exercise for you\n",
    "\n",
    "- Predict for the second example in our `deploy_test`: \"I like block 2.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Summary naive Bayes `predict`\n",
    "- Use Bayes rule to calculate the conditional probability of spam or non spam given the message. \n",
    "\n",
    "$$ P(\\text{spam} \\mid \\text{message}) = \\frac{P(\\text{message} \\mid \\text{spam}) \\times P(\\text{spam})}{P(\\text{message})} $$\n",
    "\n",
    "$$ P(\\text{non spam} \\mid \\text{message}) = \\frac{P(\\text{message} \\mid \\text{non spam}) \\times P(\\text{non spam})}{P(\\text{message})} $$\n",
    "\n",
    "\n",
    "- $P(\\text{message})$: marginal probability that a message has the given set of words \n",
    "    - Hard to calculate but can be ignored in our scenario.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Summary naive Bayes `predict`\n",
    "$$ P(\\text{spam} \\mid \\text{message}) \\propto P(\\text{message} \\mid\\text{spam}) \\times P(\\text{spam}) =  P(w_1, w_2, \\dots, w_d \\mid \\text{spam}) \\times P(\\text{spam})$$\n",
    "\n",
    "- We need the following terms:  \n",
    "    - $P(\\text{spam})$: marginal probability that a message is spam\n",
    "    - $P(\\text{message}\\mid\\text{spam})$: conditional probability that message has words $w_1, w_2, \\dots, w_d$, given that it is spam.\n",
    "        - Hard to calculate but we need it. We would require huge numbers of parameters and impossibly large training sets. \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Summary naive Bayes `predict`\n",
    "$$ P(\\text{non spam} \\mid \\text{message}) \\propto P(\\text{message} \\mid\\text{non spam}) \\times P(\\text{non spam}) =  P(w_1, w_2, \\dots, w_d \\mid \\text{non spam}) \\times P(\\text{non spam})$$\n",
    "\n",
    "- We need the following terms:  \n",
    "    - $P(\\text{non spam})$: marginal probability that a message is non spam\n",
    "    - $P(\\text{message}\\mid\\text{non spam})$: conditional probability that message has words $w_1, w_2, \\dots, w_d$, given that it is non spam.\n",
    "        - Hard to calculate but we need it. We would require huge numbers of parameters and impossibly large training sets. \n",
    "        "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Summary naive Bayes `predict`\n",
    "\n",
    "- Estimate the following probabilities using naive Bayes' assumption. \n",
    "\n",
    "$$ P(\\text{spam} \\mid \\text{message}) \\propto P(\\text{spam}) \\times  \\prod_{i=1}^{d}P(w_i \\mid \\text{spam})$$\n",
    "\n",
    "$$ P(\\text{spam} \\mid \\text{message}) \\propto P(\\text{non spam}) \\times  \\prod_{i=1}^{d}P(w_i \\mid \\text{non spam})$$\n",
    "\n",
    "- Predict the class where $P(\\text{class} \\mid \\text{message})$ is proportional to a bigger value. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Questions\n",
    "\n",
    "- When we ignore the marginal probability P(message) when calculating $P(\\text{spam} | \\text{message})$ or $P(\\text{non spam} | \\text{message})$, are these going to be well-defined probabilites? Does it matter?  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 2. `predict_proba`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is our toy pipeline's prediction? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['spam'], dtype='<U8')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "deploy_test = [\"URGENT! Free!!\"]\n",
    "pipe_nb_toy.predict(deploy_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Naive Bayes classifier `predict_proba`\n",
    "- So far we have been looking into binary predictions but often a more granular information is useful. \n",
    "- Naive Bayes classifier gives you probability estimates for each class and we can get this information using `predict_proba` method of the classifier.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.23584906, 0.76415094]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_nb_toy.predict_proba(deploy_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['non spam', 'spam'], dtype='<U8')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_nb_toy.classes_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above: The classifier is \"76% confident\" that the class is spam! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Predicting probabilities\n",
    "\n",
    "- We have a new and useful method, `predict_proba`.\n",
    "- `predict` returns the class with the highest probability.\n",
    "- `predict_proba` gives us the actual probability scores. \n",
    "- Looking at the probabilities can help us understand the model.\n",
    "- We can find the spam messages where our classifier is most confident and least confident. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Although decision tree, kNN, and RBF SVM are not set in a probabilistic framework, they still have `predict_proba` method to get the probability estimates.  \n",
    "\n",
    "- [`DecisionTreeClasifier.predict_proba`](https://scikit-learn.org/stable/modules/generated/sklearn.tree.DecisionTreeClassifier.html#sklearn.tree.DecisionTreeClassifier.predict_proba)\n",
    "  - Computed as fraction of that class in leaf node.\n",
    "- [`KNeighborsClassifier.predict_proba`](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier.predict_proba)\n",
    "  - Computed as fraction of that class in $k$ neighbours.\n",
    "- [`MultinomialNB.predict_proba`](https://scikit-learn.org/stable/modules/generated/sklearn.naive_bayes.BernoulliNB.html#sklearn.naive_bayes.BernoulliNB.predict_proba)\n",
    "  - Already computed, just need to normalize them (so they sum to 1)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Naive Bayes classifier `fit`\n",
    "\n",
    "- Calculate prior probabilities and conditional probabilities for each feature given each class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "Note that when we estimated probabilities in our toy example (e.g., $P(\\text{word} \\mid spam)$), we happened to have each feature value as either 0 or 1, i.e., just the existence of a word in the document's bag of words. We computed $P(\\text{word} \\mid spam)$ as a fraction of times the word appears among all words in all messages of the spam class. If we want to work with frequencies instead of existence, we first concatenate all documents with that class (e.g., spam class) into one big \"class c\" text. Then we use the frequency of the word (e.g., _urgent_ below) in this concatenated document to give a (maximum likelihood) estimate of the probability:\n",
    "\n",
    "$$P(\\text{urgent} \\mid \\text{spam}) = \\frac{Count(\\text{urgent}, \\text{spam})}{\\sum_{w \\in vocabulary} Count(w, \\text{spam})}$$ \n",
    "\n",
    "$$P(\\text{urgent} \\mid \\text{spam}) = \\frac{\\text{how often _urgent_ occurs with spam}}{\\text{total number of tokens (all occurrences of all words) in spam}}$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Recall that when we worked through a toy example by hand, we estimated\n",
    "    - $P(\\text{non spam} \\mid \\text{message}) \\propto 0$\n",
    "    - $P(\\text{spam} \\mid \\text{message}) \\propto 0.074$\n",
    "- Why don't `predict_proba` scores match with the probability scores we calculated before? \n",
    "- The scores we computed are not normalized. Remember that we ignored the denominator.\n",
    "- These ones are normalized so that they sum to 1.\n",
    "- The model is using something called \"smoothing\" to avoid the problem of zero probabilities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.23584906, 0.76415094]])"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_nb_toy.predict_proba(deploy_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 3. Laplace smoothing\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>block</th>\n",
       "      <th>free</th>\n",
       "      <th>prize</th>\n",
       "      <th>urgent</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has interesting courses.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has been interesting so far.</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                              block  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      0   \n",
       "Lol you are always so convincing.                                                                                 0   \n",
       "Block 2 has interesting courses.                                                                                  1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      0   \n",
       "Block 2 has been interesting so far.                                                                              1   \n",
       "\n",
       "                                                                                                              free  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                     0   \n",
       "Lol you are always so convincing.                                                                                0   \n",
       "Block 2 has interesting courses.                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!     1   \n",
       "Block 2 has been interesting so far.                                                                             0   \n",
       "\n",
       "                                                                                                              prize  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      1   \n",
       "Lol you are always so convincing.                                                                                 0   \n",
       "Block 2 has interesting courses.                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      0   \n",
       "Block 2 has been interesting so far.                                                                              0   \n",
       "\n",
       "                                                                                                              urgent  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                       1   \n",
       "Lol you are always so convincing.                                                                                  0   \n",
       "Block 2 has interesting courses.                                                                                   0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                        1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!       0   \n",
       "Block 2 has been interesting so far.                                                                               0   \n",
       "\n",
       "                                                                                                                target  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      spam  \n",
       "Lol you are always so convincing.                                                                             non spam  \n",
       "Block 2 has interesting courses.                                                                              non spam  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       spam  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      spam  \n",
       "Block 2 has been interesting so far.                                                                          non spam  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Remember when we calculated $P(\\text{non spam} \\mid \\text{message})$, some of our conditional probabilities were zero. \n",
    "    - $P(\\text{free} = 1 \\mid \\text{non spam}) = 0/3$ \n",
    "    - $P(\\text{urgent} = 1 \\mid \\text{non spam}) = 0/3$\n",
    "\n",
    "- Naive Bayes naively multiplies all the feature likelihoods together, and if any of the terms is zero, it's going to void all other evidence and the probability of the class is going to be zero. \n",
    "- Sounds worrisome! \n",
    "- We have limited data and if we do not see a feature occurring with a class, it doesn't mean it would never occur with that class. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### A simplest solution: Laplace smoothing\n",
    "\n",
    "- The simplest way to avoid zero probabilities is to add one to all the counts.\n",
    "- All the counts that used to be zero will now have a count of 1, the counts of 1 will be 2, and so on. \n",
    "- In `scikit-learn` we control it using hyperparameter `alpha` (by default `alpha=1.0`). \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So our previous bag of words representation becomes like this: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>block</th>\n",
       "      <th>free</th>\n",
       "      <th>prize</th>\n",
       "      <th>urgent</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has interesting courses.</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!</th>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Block 2 has been interesting so far.</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                              block  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      1   \n",
       "Lol you are always so convincing.                                                                                 1   \n",
       "Block 2 has interesting courses.                                                                                  2   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      1   \n",
       "Block 2 has been interesting so far.                                                                              2   \n",
       "\n",
       "                                                                                                              free  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                     1   \n",
       "Lol you are always so convincing.                                                                                1   \n",
       "Block 2 has interesting courses.                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                      2   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!     2   \n",
       "Block 2 has been interesting so far.                                                                             1   \n",
       "\n",
       "                                                                                                              prize  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      2   \n",
       "Lol you are always so convincing.                                                                                 1   \n",
       "Block 2 has interesting courses.                                                                                  1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       2   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      1   \n",
       "Block 2 has been interesting so far.                                                                              1   \n",
       "\n",
       "                                                                                                              urgent  \\\n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                       2   \n",
       "Lol you are always so convincing.                                                                                  1   \n",
       "Block 2 has interesting courses.                                                                                   1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                        2   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!       1   \n",
       "Block 2 has been interesting so far.                                                                               1   \n",
       "\n",
       "                                                                                                                target  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                      spam  \n",
       "Lol you are always so convincing.                                                                             non spam  \n",
       "Block 2 has interesting courses.                                                                              non spam  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                       spam  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free!      spam  \n",
       "Block 2 has been interesting so far.                                                                          non spam  "
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pipe_nb_toy['countvectorizer'].transform(X)\n",
    "train_bow_df = pd.DataFrame(data.toarray() + 1, columns=pipe_nb_toy['countvectorizer'].get_feature_names(), index=X)\n",
    "train_bow_df['target'] = y\n",
    "train_bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Adjusting the counts \n",
    "\n",
    "Note that the following calculations would change now with updated counts now: \n",
    "\n",
    "$$P(\\text{word} \\mid \\text{spam}) = \\frac{Count(\\text{word}, \\text{spam}) + 1}{\\sum_{w \\in vocabulary} Count(w, \\text{spam}) + |vocabulary|}$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('countvectorizer', CountVectorizer()),\n",
       "                ('multinomialnb', MultinomialNB())])"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_nb = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0))\n",
    "pipe_nb.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `alpha` hyperparameter and the fundamental tradeoff \n",
    "\n",
    "- High alpha $\\rightarrow$ underfitting\n",
    "    - means we are adding large counts to everything and so we are diluting the data\n",
    "- Low alpha $\\rightarrow$ overfitting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>alpha</th>\n",
       "      <th>mean_train_score</th>\n",
       "      <th>mean_cv_score</th>\n",
       "      <th>std_cv_score</th>\n",
       "      <th>std_train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.000001</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.945</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.000010</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.950</td>\n",
       "      <td>0.044721</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.000100</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.955</td>\n",
       "      <td>0.036742</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.001000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.955</td>\n",
       "      <td>0.036742</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.010000</td>\n",
       "      <td>1.00000</td>\n",
       "      <td>0.955</td>\n",
       "      <td>0.036742</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>0.100000</td>\n",
       "      <td>0.99875</td>\n",
       "      <td>0.950</td>\n",
       "      <td>0.035355</td>\n",
       "      <td>0.002500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.99500</td>\n",
       "      <td>0.960</td>\n",
       "      <td>0.040620</td>\n",
       "      <td>0.002500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>10.000000</td>\n",
       "      <td>0.93625</td>\n",
       "      <td>0.870</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.009186</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>100.000000</td>\n",
       "      <td>0.87000</td>\n",
       "      <td>0.870</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.002500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1000.000000</td>\n",
       "      <td>0.87000</td>\n",
       "      <td>0.870</td>\n",
       "      <td>0.010000</td>\n",
       "      <td>0.002500</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         alpha  mean_train_score  mean_cv_score  std_cv_score  std_train_score\n",
       "0     0.000001           1.00000          0.945      0.040000         0.000000\n",
       "1     0.000010           1.00000          0.950      0.044721         0.000000\n",
       "2     0.000100           1.00000          0.955      0.036742         0.000000\n",
       "3     0.001000           1.00000          0.955      0.036742         0.000000\n",
       "4     0.010000           1.00000          0.955      0.036742         0.000000\n",
       "5     0.100000           0.99875          0.950      0.035355         0.002500\n",
       "6     1.000000           0.99500          0.960      0.040620         0.002500\n",
       "7    10.000000           0.93625          0.870      0.010000         0.009186\n",
       "8   100.000000           0.87000          0.870      0.010000         0.002500\n",
       "9  1000.000000           0.87000          0.870      0.010000         0.002500"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_dict = {\"alpha\": [], \"mean_train_score\": [], \"mean_cv_score\": [], \"std_cv_score\" : [], \"std_train_score\":[]}\n",
    "param_grid = {\"alpha\": 10.0**np.arange(-6,4)}\n",
    "\n",
    "for alpha in param_grid[\"alpha\"]:\n",
    "    pipe_nb = make_pipeline(CountVectorizer(), MultinomialNB(alpha = alpha))\n",
    "    scores = cross_validate(pipe_nb, X_train[:200], y_train[:200], return_train_score=True)\n",
    "    results_dict[\"alpha\"].append(alpha)\n",
    "    results_dict[\"mean_cv_score\"].append(np.mean(scores[\"test_score\"]))\n",
    "    results_dict[\"mean_train_score\"].append(np.mean(scores[\"train_score\"]))\n",
    "    results_dict[\"std_cv_score\"].append(scores[\"test_score\"].std())\n",
    "    results_dict[\"std_train_score\"].append(scores[\"train_score\"].std())\n",
    "\n",
    "results_df = pd.DataFrame(results_dict)\n",
    "results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXhU15ng/++rfUUISQhtqITNDgKDWGzhxLGdxCzxCrGdTuy4k3YTx24n6emOk+5Jfpnf9LRnuqenszih7cTJpBPbMdhubyRO2wnxAjYIEItYzI7EKjYtgEDLO3/cKyjJJVSCunVL0vt5Hj2qu563Lqjeuuece46oKsYYY0x3cX4HYIwxJjZZgjDGGBOSJQhjjDEhWYIwxhgTkiUIY4wxIVmCMMYYE1KC3wFEUm5urgYCAb/DMMaYfmPt2rXHVDUv1LYBlSACgQBVVVV+h2GMMf2GiOzraZtVMRljjAnJEoQxxpiQLEEYY4wJaUC1QRhjTF+1trZSV1dHS0uL36F4KiUlheLiYhITE8M+xrMEISJPAwuAo6o6KcR2Ab4PzAPOAF9U1XXutlvcbfHAT1X1ca/iNMYMbnV1dWRmZhIIBHA+lgYeVeX48ePU1dVRVlYW9nFeVjH9ArjlEtvnAqPdnweBnwCISDzwhLt9AnCviEzwME5jzCDW0tJCTk7OgE0OACJCTk5On++SPLuDUNW3RSRwiV1uA36pznjj74vIUBEpAALATlXdDSAiz7n7bvEq1j9sO0J7h1dnN/1dQrwwJCWBjOREMlISyExJID0pgfi4gfuBMtgM5OTQ6XLeo59tEEVAbdBynbsu1PpZPZ1ERB7EuQNh5MiRlxXIV3+9nrOt7Zd1rBm80pPiyUy5mDQykp3fmW4iubDsJpfMlAQyUhK6JJv0pPhB8eFkenbq1CmeeeYZHnrooT4dN2/ePJ555hmGDh3qUWT+JohQfxV6ifUhqeqTwJMAFRUVlzX70bKvXIvNm2R6cq6tg+ZzbTS3tNF8rpWmljaaWtpoPtdGU0ur+9v5OdTQ4qxraeP0+d6/dIjgJJLkhAvJpmticdcnJ1A4NJWxIzIZOSzN7l4GkFOnTvHjH//4Iwmivb2d+Pj4Ho9bvny516H5miDqgJKg5WLgIJDUw3rPTCzM8vL0ZpBq71AnsXQmkpY2mtxkcqlkc/LMeWpPnHH3baWltWv9Z3JCHFflZTAmP4PR+ZmMyc9kbH4mxdmpxFni6Hcee+wxdu3axdSpU0lMTCQjI4OCggKqq6vZsmULt99+O7W1tbS0tPDoo4/y4IMPAhdHjmhubmbu3LnMmTOHlStXUlRUxMsvv0xqauoVx+ZngngFeNhtY5gFNKjqIRGpB0aLSBlwALgH+JyPcRpzWeLjhKzURLJSE4HL/2Ntbe+guaWN/SfO8OGRJnYcbWb74SZW7znBf1Rf/O6UmhjP1cMzGJ2fwZj8TMa4v4uGplo1Vpi+92oNWw42RvScEwqH8N3PTOxx++OPP87mzZuprq5mxYoVzJ8/n82bN1/obfT0008zbNgwzp49y4wZM7jrrrvIycnpco4dO3bw7LPP8tRTT/HZz36WF154gc9//vNXHLuX3VyfBW4AckWkDvgukAigqkuA5ThdXHfidHN9wN3WJiIPA2/gdHN9WlVrvIrTmFiXGB9HdnoS2elJTCnpWt/c1NLKjqPN7DjSxPbDzew42sR7O4/x4roDF/ZJT4rn6vxMxgx3Esbo/AzGjshkxJAUSxwxaObMmV26ov7gBz/gpZdeAqC2tpYdO3Z8JEGUlZUxdepUAKZPn87evXsjEouXvZju7WW7Al/tYdtynARijLmEzJREpo3MZtrI7C7rG860suNoEx8eaebDI018eKSJP26vZ+nauovHJidcuNsY7d5xjM3PJC8zedAmjkt904+W9PT0C69XrFjBm2++yapVq0hLS+OGG24I2VU1OTn5wuv4+HjOnj0bkVjsSWpjBqCstEQqAsOoCAzrsv7k6fNOwjjazIeHncTx+y1HeG7NxY6DWamJF9s33LuOMSMyyc1I7l6MiYDMzEyamppCbmtoaCA7O5u0tDS2bdvG+++/H9XYLEEYM4hkpycxa1QOs0Z1raI41nzOSRyHneSx40gTr288xDNnWy/sMyw9iXEjMvnerRMZnZ8Z7dAHrJycHCorK5k0aRKpqank5+df2HbLLbewZMkSysvLGTt2LLNnz45qbKIDqH9nRUWF2nwQxkSGqlLfdI7tR5yqqs6kUXl1Lku+MN3v8CJm69atjB8/3u8woiLUexWRtapaEWp/u4MwxoQkIgwfksLwISlcP9qZcCw9OYFfrtrLidPnGZae5G+AxnM23LcxJmyLKoppbVderj7Q+86m37MEYYwJ27gRQ5hclMXSqrredzb9niUIY0yfLKooZsuhRmoONvgdivGYJQhjTJ/cOqWQpPg4u4sYBCxBGGP6ZGhaEp+cmM/L1Qc432bj5A9kliCMMX22aHoxJ8+08tbWI36HYjxkCcIY02fXj84jf0hyl6E7zMBjCcIY02fxccKd04pZsf0oRxv7No2lCe2Xv/wl5eXlTJkyhTvuuINAIEBHh1OFd+bMGUpKSmhtbe3lLJFlD8oZYy7LounF/GTFLl5cf4DFH7/K73Ai47ePweFNkT3niMkw9/FL7lJTU8M//MM/8N5775Gbm8uJEyd44IEH+NOf/sQnPvEJXn31VT796U+TmJgY2dh6YXcQxpjLMiovg+ml2SytqmUgDdnjhz/84Q8sXLiQ3NxcAIYNG8bdd9/Nb37zGwCee+457r777qjHZXcQxpjLtmh6MY+9uIn1tac+MuR4v9TLN32vqOpHhli/9dZb+da3vsWJEydYu3YtN954Y9TjsjsIY8xlm19eQEpiHMussfqK3HTTTTz//PMcP34cgBMnTpCRkcHMmTN59NFHWbBgwSXnp/aKJQhjzGXLTElk3qQCXt1wkJbWdr/D6bcmTpzI3/3d3/Hxj3+cKVOm8I1vfAOAu+++m1/96le+VC+BVTEZY67QwopiXlx/gDdqDnPb1CK/w+m37r//fu6///4u6xYuXOhr+47dQRhjrsjsshyKs1Nt6I0ByBKEMeaKxMUJC6cX896uYxw4FZm5kE1ssARhjLlid00rRhVesMbqAcXTBCEit4jIdhHZKSKPhdieLSIvichGEVktIpOCtn1dRGpEZLOIPCsiKV7Gaoy5fCXD0rjuqhyWra2jo6P/PRMxGJ7juJz36FmCEJF44AlgLjABuFdEJnTb7dtAtaqWA/cB33ePLQL+CqhQ1UlAPHCPV7EaY67cwunF7D9xhtV7T/gdSp+kpKRw/PjxAZ0kVJXjx4+TktK379le9mKaCexU1d0AIvIccBuwJWifCcA/AqjqNhEJiEh+UGypItIKpAEHPYzVGHOF5k4q4Dsv17C0qo7Zo3L8DidsxcXF1NXVUV9f73conkpJSaG4uLhPx3iZIIqA2qDlOmBWt302AHcC74rITKAUKFbVtSLyz8B+4Czwe1X9vYexGmOuUGpSPAvKC3i5+iDfu20iGcn9oxd9YmIiZWVlfocRk7xsg5AQ67rfwz0OZItINfAIsB5oE5FsnLuNMqAQSBeRz4csRORBEakSkaqB/g3AmFi3qKKYs63tLN94yO9QTAR4mSDqgJKg5WK6VROpaqOqPqCqU3HaIPKAPcDNwB5VrVfVVuBF4LpQhajqk6paoaoVeXl5XrwPY0yYpo3MZlReOkvX1va+s4l5XiaINcBoESkTkSScRuZXgncQkaHuNoAvA2+raiNO1dJsEUkTZwSrm4CtHsZqjIkAEeeZiDV7T7L32Gm/wzFXyLMEoaptwMPAGzgf7s+rao2ILBaRxe5u44EaEdmG09vpUffYD4BlwDpgkxvnk17FaoyJnLumFRMn2AB+A4AMpK5dFRUVWlVV5XcYxgx6X/z5arYfbuLdb95IfFyo5kgTK0RkrapWhNpmT1IbYyJu0fQSDjW08N7OY36HYq6AJQhjTMTdPGE4Q9MSWWrVTP2aJQhjTMQlJ8Rz25RC3qg5TMOZVr/DMZfJEoQxxhMLp5dwvq2DVzbaIAj9lSUIY4wnJhUNYdyITJZV2TMR/ZUlCGOMJzqfidhQ18CHR5r8DsdcBksQxhjP3HFNEQlxwlK7i+iXLEEYYzyTk5HMjeOG89L6A7S2d/gdjukjSxDGGE8tqijhWPN5Vmy3wTT7G0sQxhhP3TA2j9yMJJbZAH79jiUIY4ynEuPjuOOaIt7aepTjzef8Dsf0gSUIY4znFlWU0Nah/Ee1PRPRn1iCMMZ4bkx+JlOKs1haVTug534eaCxBGGOiYmFFCdsON1FzsNHvUEyYLEEYY6Li1vJCkhLi7JmIfsQShDEmKrLSEvnUhHxe3nCQc23tfodjwmAJwhgTNYsqSjh1ppU3txz1OxQTBksQxpiomXN1LgVZKSy1ZyL6BUsQxpioiY8T7pxWxNsf1nO4ocXvcEwvLEEYY6Jq4fQSOhReXG+zzcU6SxDGmKgqy01nRiCbZWvr7JmIGGcJwhgTdYuml7C7/jTr9p/yOxRzCZ4mCBG5RUS2i8hOEXksxPZsEXlJRDaKyGoRmRS0baiILBORbSKyVUSu9TJWY0z0zCsvIDUx3gbwi3GeJQgRiQeeAOYCE4B7RWRCt92+DVSrajlwH/D9oG3fB36nquOAKcBWr2I1xkRXRnIC8yYX8OqGQ5w9b89ExCov7yBmAjtVdbeqngeeA27rts8E4C0AVd0GBEQkX0SGAB8DfuZuO6+qdi9qjJ/OnoR1/w6nj0XkdIsqimk+18bvag5F5Hwm8rxMEEVA8P1jnbsu2AbgTgARmQmUAsXAKKAe+LmIrBeRn4pIeqhCRORBEakSkar6epuQxJiIO7EHlv8t/MtEeOVhWPnDiJx2ZmAYJcNSWVplvZlilZcJQkKs695l4XEgW0SqgUeA9UAbkABMA36iqtcAp4GPtGEAqOqTqlqhqhV5eXkRC96YQa92NfzmC/DDaVD1NEy4FYZPgH3vReT0cXHCwmklrNx1nNoTZyJyThNZXiaIOqAkaLkY6DIYvKo2quoDqjoVpw0iD9jjHlunqh+4uy7DSRjGGC91tMOWl+Gnn4SffRL2/AkqH4WvbYQ7lsDYuXBgHZxrjkhxd00vQgReWGd3EbHIywSxBhgtImUikgTcA7wSvIPbUynJXfwy8LabNA4DtSIy1t12E7DFw1iNGdzOn4YPnnTuFp6/D5qPwC3/E76+BW7+/2BIobNfaSVoO9R+cKmzha04O43rrsph2do6OjrsmYhYk+DViVW1TUQeBt4A4oGnVbVGRBa725cA44Ffikg7TgL4UtApHgF+7SaQ3cADXsVqzKDVeAhWP+lUIbWcguIZcPP3YPxnIC7+o/uXzAKJd6qZrr4pIiEsml7C135Tzft7jnPdVbkROaeJDM8SBICqLgeWd1u3JOj1KmB0D8dWAxVexmfMoHV4M6x6AjYthY42GL8Arn0ERs669HHJGVA0DfZGph0C4NMTR5CZnMCyqjpLEDHG0wRhjIkhqrDrLVj5I9j9R0hMg4oHYPZXYNio8M9TWukkl/NnICntisNKTYpnwZRCXlpfx/dum0hmSuIVn9NEhiUI4/yhH1zn9FqpXQ1Ha6Cjw++oYkd6jlP1UjLL+Z0dAAnVSS9GtZ1z7hRWPQFHt0BGPtz0HZj+AKQN6/v5AnPgvX+FutUw6oaIhLioophnV+9n+aZD3D1jZETOaa6cJYjBRhUaai8mg9oP4Mhmp5oBIGc0FM+EhBR/44wljXWw4TlY81NnOX04lMx0f2ZBwVRIjMHrdeaE07aw+kmn0Xn4RLjtxzB5ISQkX/55S2aBxDnVTKNuiEio15QM5aq8dJZW1VmCiCGWIAa6tnNwaMPFZFC7GpoPO9sS06BoutONsfPb8eV8oxwMOtqdb9+dibVuNWx7zdkWlwgFUy4mjeKZkNX9mdAoOr4L3v8JVP8aWs/AVTfC7T9xfkfizidliJMUI/Q8BICIsKiihMd/u43d9c2MysuI2LnN5bMEMdA0Hb6YCGpXw6FqaD/vbBtaCmUfu/hBNnwixNt/gbDExcOIyc7PDLezXXM91K25eL2rnob3f+xsG1IcdJcxE0aUQ7yHdeuqThwrfwjbXoe4BCj/LFz7VcifGPnyApXwwb9B61lITI3IKe+8poj/9bttLFtbx9/eMi4i5zRXxj4d+rP2Vqd66EJ10Wpo2O9si0+Gwmtg1l+6dwczITPf33gHmow8GDfP+QFoOw9HNkGtmzTq1kDNi862hBQonAYlMy7+e2RE4Mn/9jbY9qrT8HygClKGwvXfgJkPQuaIKz9/T0rnOMmorgrKro/IKYcPSeHjY/J4cd0B/vpTY4mP60ftPAOUJYj+5PRxp2qjMxkcWAttZ51tmQXOB8/sxc7vEZOvrJ7Z9F1CklNlVzTd+XcAaDjg/pu5SWPVj+E9d9Di7LKubRnDJ4R+9iCUc02w/ldOVdKpfU7D+bx/hqmfg6SQw5ZF1sjZgDjVTBFKEACLKkp46NfreGdHPTeMHR6x85rLYwkiVnW0Q/02t/rC/XA5scvZFpfgVFlMvz+ozru4f/WsGSyyiiDrDph4h7Pc2uJU+3W2Ce36I2z8jbMtKcNJLp0Jo7gCUrO7nq/xoFO1U/VzONfg7Pep/w7j5oefXCIhdSgUlMPedyN62pvGD2doWiJL19ZZgogBliBixdlTThXBheqJKjjf5GxLy3U+NKZ9wUkGhddEpP+58UFiivPte+RsZ1nVuQMI7lX2zr84w1kA5I51qqWKpsP+D2DzMtAO50nnax9xtvmldA5U/czpCBGhu9XkhHhun1rEMx/s59SZ8wxNS+r9IOMZSxB+UIXjO7s2JtdvA9TpPjh8IpQvutizaNgouzsYqESc6qHsgNOoDM5AeAfXXbx73Pa6U52UmA4zvgyzFsOwMj+jdgQq4f0nnKrO0usidtqF04v5xcq9vLLhIPddG4jYeU3fWYKIhgt/8EFdJM+edLalZDlJYNKdzu+i6U43QjN4JWc4vc3KPuYsq8KJ3ZCe6/x/iRUjrwXEeR4iggliUlEW4wuGsLSqzhKEzyxBRNqFKoPO7o8fwJGarlUG4+Zf7MmSOwbiPJ0a3PR3IpBzld9RfFTaMMifBHvfgY//TURPvWh6Mf/ttS1sO9zIuBH2hckvvSYIEVkALFdVG3shlO6NjnVrnKdWwW10nOZ0OyyZ5dwd2INoZiAJVMLa/+t08U2IXHvB7dcU8Y+/3cqyqjr+fkH3qexNtIRzB3EP8H0ReQH4uapu9Tim2NZ4sGuD4qEN0NHqbMsuc4Ye6OxZNHyCPYhmBrbSSvhgCRxc3/tIsH0wLD2Jm8bl8x/VB/jm3HEkxttdth96/fRS1c+LyBDgXpw5ohX4OfCsqjZ5HaCv2lvh8KaudwcN7jTbnQ8+XfvQxcbkDOuWZwaZ0krn9753I5ogwBnA73c1h/njtqN8aqKHD/2ZHoX19VZVG907iFTga8AdwN+IyA9UNTIzmMeC08e6JoMD6y4+iDak2OlSeO1XnTuE/MkRvaU2pl9Kz3HulPe+C9f/dURP/fExeeRmJLN0bZ0lCJ+E0wbxGeDPgauAfwdmqupREUkDtgL9O0G0nYNXH3UfRNvtrOscfK3igdgYfM2YWFZaCdXPOHfcERxvKiE+jjunFfH0u3s41nyO3AwbGSDawrmDWAT8H1V9O3ilqp4RkT/3JqwoSkiG+u3Ot6DpX3SHb54SsQHIjBnwApWw5imnPa44spNALppezJNv7+Y/1h/gy9f3YVIjExHhJIjvAoc6F0QkFchX1b2q+pZnkUXTg3/0OwJj+q/Odoi970Y8QYzOz2RKyVCWVtXxpTlliD0wGlXhdA1YCgR3cW131xljjNM5I3dsxMdl6rRoejHbjzSx6UCDJ+c3PQsnQSSo6vnOBfe1tc4aYy4KVML+953hxyPsM1MKSU6IY2lVXcTPbS4tnARRLyK3di6IyG3AsXBOLiK3iMh2EdkpIo+F2J4tIi+JyEYRWS0ik7ptjxeR9SLyWjjlGWN8UlrpDC55eGPET52VmsinJ47g5eoDtLS2R/z8pmfhJIjFwLdFZL+I1ALfBP6yt4NEJB54ApgLTADuFZHuj0R+G6hW1XLgPuD73bY/itNTyhgTywJznN9eVTNVFNPY0sZ/bjniyflNaL0mCFXdpaqzcT7kJ6jqdaq6M4xzzwR2quput1rqOeC2bvtMAN5yy9kGBEQkH0BEioH5wE/DfjfGGH9kjoCcqyM6T3Ww667KpTArhaVrrZopmsJ6fl1E5gMPAV8Xke+IyHfCOKwIqA1arnPXBdsA3OmWMRMoBYrdbf8K/C1dG8iNMbGqtBL2rXImu4qw+DjhrunFvLujnsMNLRE/vwmt1wQhIkuAu4FHAMF5LqI0jHOH6o+m3ZYfB7JFpNo9/3qgzR0g8Kiqrg0jvgdFpEpEqurr68MIyxjjicAcZ5a7I5s9Of3C6cV0KLywzu4ioiWcO4jrVPU+4KSqfg+4FigJ47i6bvsVAweDd1DVRlV9QFWn4rRB5AF7gErgVhHZi1M1daOI/CpUIar6pKpWqGpFXl4EJoE3xlye4OchvDh9Tjozy4axbG0dqt2/axovhJMgOu/nzohIIdAKhDOd1RpgtIiUiUgSzqiwrwTvICJD3W0AXwbedpPGt1S1WFUD7nF/UNXPh1GmMcYvWUXOiMZ7vWmHAOcuYs+x06zdd9KzMsxF4SSIV0VkKPBPwDpgL/BsbwepahvwMPAGTk+k51W1RkQWi8hid7fxQI2IbMPp7fRo39+CMSZmBCph/0ro8KbpcP7kAtKS4u2ZiCi55FAbIhIHvKWqp4AX3OcRUlQ1rEcaVXU5sLzbuiVBr1cBo3s5xwpgRTjlGWN8VjrHmT/76BYYMan3/fsoPTmBeZMLeG3jQb576wTSkmy+FS9d8g7CnUXufwctnws3ORhjBqGAt+0Q4Ay9cfp8uz0TEQXhVDH9XkTuEhslyxjTm6EjnZ993iWIGYFh5A9J5vWNh3rf2VyRcBLEN3AG5zsnIo0i0iQijR7HZYzpr0rnwL6V4FFPo7g4Yd7kAlZ8WE9TS6snZRhHOE9SZ6pqnKomqeoQd3lINIIzxvRDgUo4cxzqt3lWxILyAs63dfDW1qOelWHCm1HuY6HWd59AyBhjgK7jMg0f70kR15RkU5CVwmsbD3H7NTbbo1fC6QLwN0GvU3DGWFoL3OhJRMaY/m1oqTOH+953YeZfeFJEZzXTv6/aR2NLK0NSIjfVqbkonCqmzwT9fBKYBFj3AWNMaCJONdO+9zxrhwCYX17A+fYO3rTeTJ4Ja7C+bupwkoQxxoRWWgmn6+HYh54VcU3JUIqGplpvJg+F0wbxQy4OshcHTMUZhdUYY0ILbofIG+tJESLCvMkj+MXKvTScbSUr1aqZIi2cO4gqnDaHtcAq4Js2LpIx5pKGjYLMAs/mh+g0v7yQ1na1h+Y8Ek4j9TKgRVXb4cI0oGmqesbb0Iwx/ZaIU820912nHcKj52ynFGe51UwHWTi9uPcDTJ+EcwfxFpAatJwKvOlNOMaYASNQCc2H4fguz4oQEeaXF/DOjmM0nLGH5iItnASRoqrNnQvu6zTvQjLGDAiB653fHg67Ac4Ir20dyhtbDntazmAUToI4LSLTOhdEZDpw1ruQjDEDQs7VkD7c0/khAMqLsyjOtt5MXginDeJrwFIR6ZwNrgBnClJjjOlZ9+chPGqH6Kxm+tk7ezh5+jzZ6Um9H2TCEs6DcmuAccBXgIeA8eHMFW2MMQTmQOMBOLnH02IWTC6krUP5vVUzRVSvCUJEvgqkq+pmVd0EZIjIQ96HZozp90o7n4fwtpppUtEQRg5L4zWrZoqocNog/sKdUQ4AVT0JeDPAijFmYMkbC2m5nj8P0VnNtHLXcU6ePu9pWYNJOAkiLniyIBGJB6ySzxjTOxEovc7zOwhwejO1dyhv1Fg1U6SEkyDeAJ4XkZtE5EbgWeC33oZljBkwAtdDw344uc/TYiYWDiGQk8brm6yaKVLCSRDfxHlY7ivAV4GNdH1wzhhjetY5T3UUq5mON5/ztKzBIpxeTB3A+8BuoAK4CdjqcVzGmIEibzykZkepmqnQrWaysZkioccEISJjROQ7IrIV+BFQC6Cqn1DVH4VzchG5RUS2i8hOEXksxPZsEXlJRDaKyGoRmeSuLxGRP4rIVhGpEZFHL+/tGWN8Fxfnjsv0judFjS/IZFRuOq9vOtj7zqZXl7qD2IZzt/AZVZ2jqj8E2sM9sduY/QQwF5gA3CsiE7rt9m2gWlXLgfuA77vr24C/VtXxwGzgqyGONcb0F4E5cGofNNR5WkxnNdOqXcc5ZtVMV+xSCeIu4DDwRxF5SkRuAvryKORMYKeq7lbV88BzwG3d9pmA076Bqm4DAiKSr6qHVHWdu74Jp0rLJp41pr8qddsholHNVF5Ah8LvNltvpivVY4JQ1ZdU9W6cp6hXAF8H8kXkJyLyqTDOXYRbLeWq46Mf8huAOwFEZCZQCnQZs1dEAsA1wAehChGRB0WkSkSq6uvrwwjLGBN1+RMhJcvzgfsAxuZnclVeuo3NFAHhNFKfVtVfq+oCnA/vauAj7QkhhLrb6D5B7eNAtohUA48A63Gql5wTiGQALwBfU9XGHuJ7UlUrVLUiLy8vjLCMMVEXFw8jr3Pmh/CYU81UyAd7jnO0qcXz8gayPs1JraonVPXfVPXGMHavA0qClouBLi1Hqtqoqg+o6lScNog8YA+AiCTiJIdfq+qLfYnTGBODAnPgxG5o9P6b/QK3mukNq2a6In1KEH20BhgtImUikgTcA7wSvIOIDHW3AXwZeFtVG90nt38GbFXVf/EwRmNMtETpeQiAMfmZjB6eYWMzXSHPEoSqtgEP4zyJvRV4XlVrRGSxiCx2dxsP1IjINpzeTp3dWSuBLwA3iki1+zPPq1iNMVEwohySh0SlmgmcxurVe09wtNGqmS5XOPNBXDZVXQ4s77ZuSdDrVcDoEMe9S996TBljYl1cPIycHb0EMbmAf3CKgSAAABHtSURBVH1zB7/dfJj7rwtEpcyBxssqJmOM6SowB47vgCbvn3QenZ/JmPwM6810BSxBGGOip3N+iCi0Q4Az9MaafSc43GDVTJfDEoQxJnoKpkBSRvQSRPkIVOG3m+0u4nJYgjDGRE98ApTMilo7xNXDMxk3ItOqmS6TJQhjTHQF5kD9Njh9LCrFzZ9cQNW+kxxqOBuV8gYSSxDGmOgKRLcdYl55AQC/3WQPzfWVJQhjTHQVXgOJaVEZuA/gqrwMxhcMsZnmLoMlCGNMdMUnRrUdApyhN9buO8nBU1bN1BeWIIwx0ReohKM1cOZEVIqbN9mpZlpudxF9YgnCGBN9F56HWBmV4spy05lYaNVMfWUJwhgTfUXTICElqtVM88sLWL//FHUnz0StzP7OEoQxJvoSkqFkZlQmEOo0f7L1ZuorSxDGGH+UzoHDm+HsyegUl5PO5KIsXrNqprBZgjDG+CNQCSjsfz9qRc4vL2BD7SlqT1g1UzgsQRhj/FFUAfHJ0W2HsN5MfWIJwhjjj8QUKJ4R1QRRMiyNKcVZ1pspTJYgjDH+CVTC4Y3Q0hC1IueXF7CxroH9x62aqTeWIIwx/imtBO2A/R9ErcjOh+bsLqJ3liCMMf4pngFxibD3negVmZ3G1JKhvL7pYNTK7K8sQRhj/JOUBsUVURvZtdOC8gI2H2hk77HTUS23v7EEYYzxV2klHKyGc01RK3KuVTOFxdMEISK3iMh2EdkpIo+F2J4tIi+JyEYRWS0ik8I91hgzQAQqQduhNnrtEEVDU7lm5FCbaa4XniUIEYkHngDmAhOAe0VkQrfdvg1Uq2o5cB/w/T4ca4wZCEpmQVxCVLu7gvNMxJZDjeyxaqYeeXkHMRPYqaq7VfU88BxwW7d9JgBvAajqNiAgIvlhHmuMGQiS0qFwWtQmEOpkQ4D3zssEUQTUBi3XueuCbQDuBBCRmUApUBzmsbjHPSgiVSJSVV9fH6HQjTFRFaiEg+vgfPS+zRcOTWV6aTavWTVTj7xMEBJinXZbfhzIFpFq4BFgPdAW5rHOStUnVbVCVSvy8vKuJF5jjF9K50BHW1TbIcCpZtp6qJFd9c1RLbe/8DJB1AElQcvFQJeOx6raqKoPqOpUnDaIPGBPOMcaYwaQkbNA4v2rZrK7iJC8TBBrgNEiUiYiScA9wCvBO4jIUHcbwJeBt1W1MZxjjTEDSHImFE6N+vMQI7JSmBHItu6uPfAsQahqG/Aw8AawFXheVWtEZLGILHZ3Gw/UiMg2nB5Lj17qWK9iNcbEgNJKOLAWzkd3jKT5kwvYdriJnUej9xxGf+HpcxCqulxVx6jqVar6D+66Jaq6xH29SlVHq+o4Vb1TVU9e6lhjzAAWmAPt56FuTVSLnTu5ABF4faPNNNedPUltjIkNI2eDxEW9mil/SAozAsNsbKYQLEEYY2JDShaMKI96QzU4YzN9eKSZD49YNVMwSxDGmNgRmONUMbW2RLXYWyaNcKuZrLE6mCUIY0zsKK2E9nNwoCqqxQ7PTGFW2TBe33QI1ZCPXA1KliCMMbGj9FpAfKlmml9eyM6jzXx4xB6a62QJwhgTO1KzYcQk2BfdgfsAbpk4gjiB1zdaY3UnSxDGmNhSOgdq10DbuagWm5eZzOxRObxm1UwXWIIwxsSWQCW0nYUD66Je9PzyAnbXn2bbYevNBJYgjDGxprTS+e1rNZP1ZgJLEMaYWJM2DIZP9KWhOicjmWuvyrHeTC5LEMaY2BOodIb+bm+NetHzJxey59hpthxqjHrZscYShDEm9pRWQusZOLg+6kV/emI+8XFiM81hCcIYE4s62yGiPE81ONVM112Vw+sbrZrJEoQxJvZk5EHeuKgP3Ndp/uQC9h4/Q83BwV3NZAnCGBObSith//vQ3hb1oj89cQTxcTLoJxKyBGGMiU2BSjjfDIc2RL3o7PQkKq/OHfTVTJYgjDGxqXSO89uH5yEAFkwuYP+JM2w+MHirmSxBGGNiU2Y+5Iz25XkIgE9NzCchTnhtEE8kZAnCGBO7ApWwfxV0tEe96KFpScwZPbirmSxBGGNiV+B6ONcIhzf6Uvz8yQXUnTzLxroGX8r3myUIY0zsuvA8hE/VTBNGkBg/eHszWYIwxsSuIQUwbJRvz0NkpSVy/ei8QVvN5GmCEJFbRGS7iOwUkcdCbM8SkVdFZIOI1IjIA0Hbvu6u2ywiz4pIipexGmNiVGmlkyB8aIcAp5rpwKmzVNee8qV8P3mWIEQkHngCmAtMAO4VkQnddvsqsEVVpwA3AP9bRJJEpAj4K6BCVScB8cA9XsVqjIlhgeuhpQGO1PhS/M0T8kmKjxuUQ4B7eQcxE9ipqrtV9TzwHHBbt30UyBQRATKAE0DnY5MJQKqIJABpwODta2bMYBbonB/Cp2qm1EQ+NiaX5ZsO0dExuKqZvEwQRUBt0HKduy7Yj4DxOB/+m4BHVbVDVQ8A/wzsBw4BDar6+1CFiMiDIlIlIlX19fWRfg/GGL9lFcPQUl8G7us0v7yAgw0trB9k1UxeJggJsa57+v00UA0UAlOBH4nIEBHJxrnbKHO3pYvI50MVoqpPqmqFqlbk5eVFLnpjTOwIzHHbITp8Kf7m8fkkJQy+aiYvE0QdUBK0XMxHq4keAF5Ux05gDzAOuBnYo6r1qtoKvAhc52GsxphYFpgDZ09C/VZfis9MSeTjY/IGXTWTlwliDTBaRMpEJAmnkfmVbvvsB24CEJF8YCyw210/W0TS3PaJmwB//mcYY/zn8/MQAAvKCzjc2MK6/Sd9iyHaPEsQqtoGPAy8gfPh/ryq1ojIYhFZ7O72/wPXicgm4C3gm6p6TFU/AJYB63DaJuKAJ72K1RgT47JLIasE9r7jWwg3dVYzDaKH5hK8PLmqLgeWd1u3JOj1QeBTPRz7XeC7XsZnjOlHSith55ugChKqidNbGckJ3OBWM/3X+ROIi4t+DNFmT1IbY/qHwBw4cwzqt/sWwvzyAo40nmPtIKlmsgRhjOkfLjwP4V9315vG55M8iHozWYIwxvQP2WWQWejr8xAZyQl8Yuxwlm86RPsg6M1kCcIY0z+IOHcRe99z2iF8Mr+8gKNN56jae8K3GKLFEoQxpv8IzIHTR+H4Tt9CuHHccFISB0dvJksQxpj+o3Oeah+rmdKTE7hx3HCWbzo84KuZLEEYY/qPnKsgI9/XBAEwf3Ihx5rPsXrPwK5msgRhjOk/RC7OD+FjO8QnxuWRmhjP65sG9iDTnj4oZ4wxEReYAzUvwrI/h/gkX0JIA34x9DhHq8+xZo//c5l1JGYy6+GnI35eSxDGmP5l7FxY8zM4sNbXMKZoByfkPDT6GgYAzfFZnpzXEoQxpn8ZUggPrfQ7ClJw5iIYyKwNwhhjTEiWIIwxxoRkCcIYY0xIliCMMcaEZAnCGGNMSJYgjDHGhGQJwhhjTEiWIIwxxoQk6uN4JpEmIvXAPr/juEK5wDG/g4gRdi26suvRlV2Pi67kWpSqal6oDQMqQQwEIlKlqhV+xxEL7Fp0ZdejK7seF3l1LayKyRhjTEiWIIwxxoRkCSL2POl3ADHErkVXdj26sutxkSfXwtogjDHGhGR3EMYYY0KyBGGMMSYkSxDGGGNCsgTRT4jIDSLyjogsEZEb/I7HbyIy3r0Wy0TkK37H4zcRGSUiPxORZX7H4ofB/v67i9TfhyWIKBCRp0XkqIhs7rb+FhHZLiI7ReSxXk6jQDPOTId1XsUaDZG4Hqq6VVUXA58F+vXDUhG6HrtV9UveRhpdfbkuA/H9d9fH6xGZvw9VtR+Pf4CPAdOAzUHr4oFdwCggCdgATAAmA691+xkOxLnH5QO/9vs9+X093GNuBVYCn/P7PcXC9XCPW+b3+/HjugzE93+l1yMSfx8JYWcSc9lU9W0RCXRbPRPYqaq7AUTkOeA2Vf1HYMElTncSSPYizmiJ1PVQ1VeAV0TkdeAZ7yL2VoT/fwwYfbkuwJboRhd9fb0ekfj7sCom/xQBtUHLde66kETkThH5N+DfgR95HJsf+no9bhCRH7jXZLnXwfmgr9cjR0SWANeIyLe8Ds5HIa/LIHr/3fV0PSLy92F3EP6REOt6fGpRVV8EXvQuHN/19XqsAFZ4FUwM6Ov1OA4s9i6cmBHyugyi999dT9djBRH4+7A7CP/UASVBy8XAQZ9iiQV2Pbqy6xGaXZeuPL0eliD8swYYLSJlIpIE3AO84nNMfrLr0ZVdj9DsunTl6fWwBBEFIvIssAoYKyJ1IvIlVW0DHgbeALYCz6tqjZ9xRotdj67seoRm16UrP66HDdZnjDEmJLuDMMYYE5IlCGOMMSFZgjDGGBOSJQhjjDEhWYIwxhgTkiUIY4wxIVmCMFEhIs1XcOwKEenXQ3oDiEhARD4XtPxFEQk5rpaILBeRoREqt0BEXgsjts1Xuk+IYx4WkQf6coyJHZYgjImeAPC53nYCUNV5qnoqQuV+A3gqQufqq6eBv/KpbHOFLEGYqBKRDBF5S0TWicgmEbnNXR8QkW0i8n9FZKM7E1ZaiON/IiJVIlIjIt8LWj9DRFaKyAYRWS0imSISLyL/JCJr3HP+ZYjzpYvI6+5xm0Xkbnf9XhH5HyKyyi1vmoi8ISK7RGSxu4+459/svpe7L7UeeBy4XkSqReTr7rpCEfmdiOwQkf8VFNdeEcl1r8tWEXnKfc+/F5HUoPe80Y3xny7x7f4u4HdB1/kd9/qvE5HrQlyTL4rIy25c20Xku0Gb43uI5S/c67xBRF7o/LdT1TPAXhGZ2UNsJpb5PQmG/QyOH6DZ/Z0ADHFf5wI7cUakDOCMVlrpbnsa+C/u6xVAhft6mPs73l1fjjNRym5ghrttiFvOg8Dfu+uSgSqgrFtcdwFPBS1nub/3Al9xX/8fYCOQCeQBR4OO/U83lnxgP1BwifU3AK8FlfVFN+4snJkC9wElQeXnutelDZjqrn8e+Lz7ejNwnfv6cYImkgkqowxYG7ScBqS4r0cDVe7rQOfxblyHgBwg1S2nopdYcoLK+O/AI0HLfwf8td//B+2n7z92B2GiTYD/ISIbgTdxxrPPd7fVqup77utfAXNCHP9ZEVkHrAcm4syyNhY4pKprAFS1UZ0xaj4F3Cci1cAHOB94o7udbxNws4j8TxG5XlUbgra9ErTPB6rapKr1QIvbPjAHeFZV21X1CPAnYMYl1ofylqo2qGoLzqQ3pSH22aOq1e7rtUDALT9TVVe663uaEKYAqA9aTgSeEpFNwFKc6xfKf6rqcVU9izPMfOe/xUdicV9Pcu9MNgF/hvNv0+koUNhDOSaG2XwQJtr+DOdb+HRVbRWRvTjfnuGj8x10WRaRMuC/4NwpnBSRX7jHSohjcdc/oqpv9BSMqn4oItOBecA/isjvVfW/uZvPub87gl53LicQeix+LrE+lODzthP6b7L7Pql9KOMsF68vwNeBI8AUnCrmlh6O6+nfIlQsAL8AblfVDSLyRZy7pU4pbhymn7E7CBNtWThVNK0i8gm6fmMeKSLXuq/vBd7tduwQ4DTQICL5wFx3/TacuvwZAG77QwLOCJdfEZFEd/0YEUkPPqGIFAJnVPVXwD/jzPkbrreBu922jjycOYNXX2J9E0411RVT1ZNAk4jMdlfd08OuH3LxWz441/+QqnYAX8CpBgvlkyIyzG1juB14r4f9OmUCh9xr/Wfdto3BqaYy/YzdQZho+zXwqohUAdU4H+6dtgL3izNN4g7gJ8EHut9O1wM1OHX377nrz7sNwT90P9DOAjcDP8X5cFwnIoJT1XJ7t3gmA/8kIh1AK/CVPryXl4BrcSaKV+BvVfWwiPS0/jjQJiIbcL5xn+xDWaF8Cae66DROe0xD9x1U9bTbsH61qu4Efgy8ICKLgD/iJNxQ3sWZ3vZq4BlVrZKPzocc7L/iVOPtw6mSC06ElcD3Qh1kYpsN921igvvh85qqTvI5lH5DRDJUtdl9/RhQoKqPhtjvDpwqvb8P87xfxOkU8HAEYrwG+IaqfuFKz2Wiz+4gjOm/5ovIt3D+jvfh9D76CFV9SURyohlYkFycuwvTD9kdhDHGmJCskdoYY0xIliCMMcaEZAnCGGNMSJYgjDHGhGQJwhhjTEiWIIwxxoT0/wCVmAS83wLLeQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.semilogx(param_grid[\"alpha\"],results_dict[\"mean_train_score\"],label=\"train\");\n",
    "plt.semilogx(param_grid[\"alpha\"],results_dict[\"mean_cv_score\"],label=\"cv\");\n",
    "plt.legend();\n",
    "plt.xlabel('laplace smoothing (alpha)');\n",
    "plt.ylabel('Accuracy');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "#### Questions: True or False \n",
    "- Laplace smoothing helps us deal with unknown words, the words that do not occur in the vocabulary of the training set. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 4. Naive Bayes with continuous features\n",
    "\n",
    "- We can use Gaussian Naive Bayes if you have continuous features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>country</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>160</th>\n",
       "      <td>-76.4813</td>\n",
       "      <td>44.2307</td>\n",
       "      <td>Canada</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>127</th>\n",
       "      <td>-81.2496</td>\n",
       "      <td>42.9837</td>\n",
       "      <td>Canada</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>169</th>\n",
       "      <td>-66.0580</td>\n",
       "      <td>45.2788</td>\n",
       "      <td>Canada</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>188</th>\n",
       "      <td>-73.2533</td>\n",
       "      <td>45.3057</td>\n",
       "      <td>Canada</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>187</th>\n",
       "      <td>-67.9245</td>\n",
       "      <td>47.1652</td>\n",
       "      <td>Canada</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     longitude  latitude country\n",
       "160   -76.4813   44.2307  Canada\n",
       "127   -81.2496   42.9837  Canada\n",
       "169   -66.0580   45.2788  Canada\n",
       "188   -73.2533   45.3057  Canada\n",
       "187   -67.9245   47.1652  Canada"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/canada_usa_cities.csv\")\n",
    "train_df, test_df = train_test_split(df, test_size = 0.2, random_state = 123) # 80%-20% train test split on df\n",
    "X_train, y_train = train_df.drop(columns=[\"country\"]), train_df[\"country\"] \n",
    "X_test, y_test = test_df.drop(columns=[\"country\"]), test_df[\"country\"]\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAEGCAYAAABlxeIAAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3de3zcdZ3v8dcn16ZNk16ShlpoG6uglEuRMHKpF1zkYFcU13rbo8b6OIu69cBxKyhntavV3SNgjysPeziLLt1ZXV2UXVdl0cVFLo8qMKTcSg8UxRSkLWl6S5o0zfV7/vj+hkzTSTJJ5jfzm5n38/H4PWbmN5ffJ5Pk9/l97+acQ0REpCzfAYiISDQoIYiICKCEICIiASUEEREBlBBERCRQke8AMtHQ0OCWL1+e7zBERArK9u3bDzjnGjN9fUEkhOXLl9PW1pbvMERECoqZvTCV16vKSEREACUEEREJKCGIiAighCAiIgElBBERAQqkl9F0JBIQj0N7OzQ3Q2srxGL5jkpEJLqKsoSQSMDGjXDgACxZ4m83bvT7RUQkvaJMCPE41NfDvHlQVuZv6+v9fhERSa8oE0J7O9TVnbivrs7vFxGR9IoyITQ3Q3f3ifu6u/1+ERFJrygTQmsrdHXBkSMwMuJvu7r8fhERSa8oE0IsBps2QUMD7NnjbzdtUi8jEZGJFG2301hMCUBEZCqKsoQgIiJTp4QgIiKAEoKIiASUEEREBFBCEBGRgBKCiIgASggiIhIo2nEIM6Xps0Wk1KiEkMYr02fvOsiS3ds48K8PsHHtThJbd+Y7NBGR0CghpBGPQ/3QQebteoSy/uN++mzXRfyvfq9FFUSkaCkhpNHeDnV7n4GqKr9h1M0Zon1wiRZVEJGipYSQRnMzdB8ehqrKV/Z1D9TQPL9LiyqISNFSQkijtRW6Khs40lPJiIMj/TV0DdbQ+qpfalEFESlaSghpxGKw6cvQYAfZ01VLw6weNp3xfWIVj2lRBREpWqF2OzWz3cBRYBgYcs61mNmXgD8DOoOX/U/n3N1hxjEdsXUria3shfjfp/Q91aIKIlK8cjEO4VLn3IEx+77hnPt6Do49M1pUQURKiKqMREQECD8hOOAeM9tuZlen7P+0mT1lZreb2fx0bzSzq82szczaOjs7071ERESyKOyEcIlz7g3AO4D1ZvZm4FZgBbAK2AdsTvdG59xtzrkW51xLY2NjyGGKiEioCcE5tze43Q/8GIg55zqcc8POuRHg24Aq6UVEIiC0hGBmc8xsbvI+cDnwtJktTnnZe4Cnw4pBREQyF2Yvoybgx2aWPM73nXO/MLPvmtkqfPvCbuATIcYgIiIZCi0hOOd+D5ybZv9HwjqmiIhMn7qdyokSCVi7FpYt89vatZrhVaREKCHIqEQCrrkGtm2D8nK/bdsG116rpCBSApQQZFQ8DgcOwOzZUF3ttzlzoLNT036LlAAlBBnV3g79/cEaEIGqKr9P036LFD0lBBnV3OxLBQMDo/sGBvw+TfstUvSUEGRUays0NMCxY75U0N8Pvb3Q2Khpv0VKgBKCjIrF4JZbYPVqGB722+rV8M1vatZXkRKQi+mvpZDEYnDnnfmOQkTyQCUEEREBVEKQVImE7176ygpxreFVFeXyWCKSEZUQxEskYONGPw5hyRJ/u3FjOAPScnksEcmYEoJ48TjU18O8eVBW5m/r68MZkJbLY4lIxpQQxGtvh7q6E/fV1YUzIC2XxxKRjCkhiNfcDN3dJ+7r7g5nQFoujyUiGVNCEK+1Fbq64MgRGBnxt11d4QxIy+WxRCRjSgjixWKwaZMfqbxnj7/dtCmcnj+5PJaIZEzdTsXLdTfQWEwJQCRiVEIQdQMVEUAJQUDdQEUEUJVRYQmrWqe93ZcMUqkbqEjJUQmhUIRZrRO1bqCJBKxfD2vW+FtVXYnkhBJCoZioWmemJ9AodQNVe4ZI3ighFIrxRvc+/vjMT6BR6gaq9gyRvAm1DcHMdgNHgWFgyDnXYmYLgDuA5cBu4P3OucNhxlEUmpv9yX7evNF93d1+O+200f3J23h8aif06XQDDaNNQ+0ZInmTixLCpc65Vc65luDx54F7nXOvBe4NHstkxlbrPP88PPSQv6LfsQP27x99bS5OoGFV7eSjPUNtFiJAfqqM3g0ky/9x4Ko8xFB4Uqt1du6E556D00+HxYvh6FFoaxtNCrloEA6raifX7RlqsxB5RdgJwQH3mNl2M7s62NfknNsHENwuSvdGM7vazNrMrK2zszPkMAtELAZbtsB558FFF8GKFXDGGaPP79qVuwbhsGYszXV7htosRF4R9jiES5xze81sEfBLM3s20zc6524DbgNoaWlxYQVYkFLr2RctgpYWnww6O/0JdMOG8BuEx2vTyEbJJJfTWqjNQuQVoZYQnHN7g9v9wI+BGNBhZosBgtv9439CCUgk4PzzoaICzPzt+edPXGUxtp590SI4+2x4//t9CSIXJ9ModVWdiaiNwRDJo9ASgpnNMbO5yfvA5cDTwE+B5FmjFfhJWDFEXiLhGzIfewyGh/2+4WH/+JJL4ItfTP++KJyMo9RVdSai8F2KRIQ5F05tjJm9Gl8qAF819X3n3F+b2ULgh8BS4EXgfc65QxN9VktLi2trawslzry69FIS9/cQ52M8ziq6qaOObs7jCVr5B2LVO+DWW2HdupPfm+PZSRNbdxLffID2jtk0Nx2jdUMDsXUrQzteLmNJfPFnxLf00N7TSHNtJ63ra4l95cosRiySH2a2PaWH5+SvDyshZFOxJoRE9ZvYOPCXDFHGs7weYwRHGa/jGSoYYVPFV4id0QVPP53fOLfuZON1fdTXDFA3Z5Du3kq6+qrYdHNNzpNCtmPZ+sXn+aubaxmkkvk1fbyq8iAVIwN5+dlEsm2qCUEjlfMoPvgh6jnCPpZQTT+19FJFP/t4FfUcIV72MejoyHeYxDcfoL5mgHm1Q5SZMa92iPqaAeKbDxR0LIkE/NU35uPKyqif1c/x4Up2HTuVobKqtJ+X2LqT9Wc9wJrGR1l/1gMkvvgzjV+QoqKEkEfttoI6ujnKXKoYBKCKAY4GVUftNENTU56jhPaO2dTNGTxhX92cQdo7Zhd0LPE4DA46aqsGMYPq8mEqy4bYO7jwpM9LlkwOdFWyZOExDnSOsPHG2T4HaPyCFAklhDxqXnCEbuqYy1EGqARggCrm0k13+QKa7QXfhTTPmpuO0d1becK+7t5KmpuOFXQs7e0wf3Y/A0Oj/wZVZcMc7qs56fNOKpkMdFJf0Uv8+UumN35Bo6NPtHUrnHUWNDb6261b8x1RSVJCyJdEgtam/6CrfCGL2UM/1fQwhwGqWczLdM06hdbrFqVvUM6x1g0NdPVVcaSnghHnONJTQVdfFa0bGgo6luZmeNVrZzMwVM7AoIFz9A5UUMngSZ93UslkcJC6yuO0950yui/T8QsaHX2irVvhuut8766FC/3tddcpKeSBEkK+xOPEVvay6ZJ7OGP+AZZWdVBfeZyl83s5488vY9OvVkemp0ts3Uo23VxDQ/0gew7OpqF+MG+NrtmMpbUVKhbM44xzKqmuHOFIXzVWVs6Xr+s56fNOKplUVtI9OIvmmpdH92U6fkGjo0+0eTPU1EBtrR+LU1vrH2/enO/ISo56GeXLmjX+6rAsJSePjPg+/Xffnb+4SkymvXdP6t10eISuwyNsOvdfia0a8MmgqyuzsRj63Z+osdGXDMxG9zkHBw/60fcybVPtZaQlNPMlzKkfJGOZzpIRW7eSTewkvrn3lfEPG67uIXYIaN/jf2+ZThmi3/2Jmpp8Mq2tHd3X2xuJDhWlRgkhX1pbfb0x+Lrn7m4SLzQRH/5L2tfkZKyZTFFs3UpiJzXpTKNaL83vnq6uSHQgyIsNG3ybAcCcOT4Z9PX50lYpy/HgU1CVUX6l/MITs9/Cxj2foH7pvBPOEYU4G4RkIA//7JG2datvM+jo8CWDDRte6VBxwsj0uoO0Lr2PWM2O4v7ekh0P6uuZyQlBI5UL1Pr1J9ciHDnipwjasiV/cYnkU2rbTX/vEG2HX81BFlBnR3nTwme5fsW/ELvlw8WXFLJ0QtBI5QIV1vICIoUsOf5jYKiMhw6fwSEWUM4IPW4Odx9o4fJHvszaS/b5UePFJE8nBCWEiNAszCInS47/eO7QQo4ziwqGAMdxZlHGCCOU8cTQSj9qvJiSQp5OCEoIEaFZmKVkJRKwdi0sW+a3tWtfGaSXHP9xdHg2w5RTzggDVFHOMBUMMUIZAzbLjxrf0pPnHySL8nRCUEKIiGJZXkBkShIJuOYa2LYNysv9tm0bXHutH80fjEyvsgHKGGGQCoYpp4p+RiijnBHmlvf6UeM9jfn+abInTycEdTuNkFyuHClFZIIeOpEXj/vG09mzobra7zPzA9LicWJbtrCJndz0hX385975DFFONccBY4hy5lovp8960Y8ary2yQWx5OCGohCBSyLI5D1CuJtxLPc5dd8HRo1BVNfp8VRX097/SgBpbt5I791zMPbfvYc2rdzG3rI9hyllgh7lo9hNUDffRNTSH1vW14xxQMqWEIFLIsjUPUK4m3Bt7nMpKnxB6Uur/BwZ8aWFMA2ps3UrufP4NdAw38sAX7uXdtfcx0DdCw+DLbDr7TmJXamTzTKnKSKSQdXT4kkGqOXOmvrBS6oR7MHobj2e32mLscc4+Gw4d8luylHDs2Oigs3HErmwi9uiWEwdubdyohrcZUglBpJA1NfmpHlJNZx6gXPV7H3ucRYvgTW+CuXNheNhvq1fDN7858Yk93YyxQ0Pw8Y9rjYkZUEIQKWQbNvh5f3p6/AyhPT3+8VTnRQqj33uyreDii/2iNxdfDC++eHKSmTXLdzV94QW/3Xnn5Ff5YxPL/v3w7LN+hlStMTFtqjKKCE1tI9OS7E2U2sto06ap9zLK9oR7ybaCgwfhmWf81Tv49o3nnvMn7/PP98lgOscJZoxNvLyU+I7zaO9aSHPZO2ht+HdiyRIDZL/Kq8hpLqMIyNI8ViIzM92rknTdXtvaYNcu+M1vfIkllZmv5ikr82shLFsG5503taugRILEulvZ+LuPUF/ZS13ffrqpo6t8IZsu+jmx13WX9hoTAU1uV4A0sZ0UrK1b4TOf8SUA5/zJvqICTjvNV1/t3p3+fWVlvodRWRlcfrnvVTTFq6D1K37BgY4h5tEFg4NQUcERq6eh7AhbTvuab6heuBBuv71kr6xCmdzOzE43s3vN7Ong8Tlm9oXpBikn0sR2UrC++lU4fvzEE/zx476tYKKeTiMjoyvG/fa301pGtL17If21C9k2GOPnQ5exre98+gfMj1ju7vajnk85RW0JU5Bpo/K3gRuAQQDn3FPABzN5o5mVm9njZnZX8PhLZrbHzJ4ItjXTCbyYaGK7wpSrcVyRtnevTwTl5b50kLzt6fGDyybS1+eTx65dcM89JwxGy8TsqkEe3v9qjgzVcpS5tLtl3DP4Vo67Kn9FdcEFsGJFaa9XPUWZJoTZzrmxf+5DGb73WuCZMfu+4ZxbFWylW8EX0MR2hSdX47gir6LCVxUlDQ76gWVmvktpJsrKfGniwQf9FBZTMOgqODAyn56RGgaopJ9ZPME5JBrWjB5fxe2MZZoQDpjZCsABmNlaYN9kbzKzU4E/Br4z7QhLgCa2KzzpusGX5IVoSwsMDpLoXcn67r9hTe8PWT9yC4k5l8KFF05+gi8r8+0HyaqmKTg2UElVlWOYchxGOcNUc5xearkp8VbfFRXSF7e3bvVdYRsb/e10pvooQpl2O10P3Aa8zsz2AO3AhzN4398C1wNzx+z/tJl9FGgDNjjnDo99o5ldDVwNsHTp0gzDLFya2K6wtLf7kkGqkrwQvfFGElf9DRv3fYp6jrCEPRygkY19N7Dp0M+JXQ786lejdaLl5f52ZMR3Oa2o8PcrKnxiOHYs40M3Nx3j/kO11NhxKmwEgGFXRoXr59GBc31VVFXVyd1ak/M/1dScOP8TFM6kgCHJKCE4534PXGZmc4Ay59zRyd5jZu8E9jvntpvZW1OeuhX4Cr608RVgM/DxNMe8DZ+EaGlpiX5XKCkpQTf4E3qGFXS7T7oupzB5N9RYjHjNJ6mfM8g8hmCwknkVA2DHiO84j9j7uuGKK/wI5KYm/1kvvujrRaurR6er6O/3pYQpfIGtGxq4/b8ZzhmYTwYjroz6yj5wwYypDQ0+GaTGnTr/E4zebt6shDDRk2b2F+PsB8A5978nePslwLuCRuNZQJ2Zfc8590rJwsy+Ddw11aBF8i3b47jyKnUgTLJB5JprfDvA0qUnNpIk6zITCbjpJnj0Udpf+juW1ByFxgb/eZ2d1NlR2o+dMtoglloHmkj49Q7a20fbH44dg+XLp9RwFlu3kgu/dYRHHquif6SMKhukrrqfobIqLpr/PFz1/vT9trM1/1MRmqwNYW6wtQCfApYE2yeBMyd6o3PuBufcqc655fgeSb9yzn3YzBanvOw9wNPTjF0kb4qq3Sddg8iBA/4KO10jSbCoTeJXPaw//FV2ujO5r/cC9u8J+pk0NtLt5tJc+Yf0X0ws5ucqWr36xPmLbrllyl/gjbfO45yVw7yq5jBzZw0yq2qY5bM6uP60H4yfXLI1/1MRymhgmpndA7w3WVVkZnOBHznnrsjoIL7K6LPOuXea2XeBVfgqo93AJ5xzEzZQF/vANJG8WrPGlwKeew527PAnx8FBnwjOOANOP9332EmO/G1uJvFve9l4+H9QX9FL/2AZDx8/FzDeOOdpZs2fTVdfFZturiG2bmXo4ScSEL+pg/ZHD9BMO60X7CR2/aXjJ5fUNoQ5c/zP29cHN99cdFVGoYxUNrNngXOdc/3B42rgSefc66Yd6RQoIYiEaP16v1DNiy+e/FxDg2/8bWnx9f0NDdDezvoHP8ABa2RehW9O3N9fz46+1zJYVsU7X/97Wjc05CQZTFshrzI3BVNNCJn2MvoukDCzH+Ov7N8D/OM04hORqFmwIH0yAD/9w7JlvuTwmtf4E2c8Tvv9y1hSvo9krfOiisNcOvdR9sw/my1PvyV3sU/XunVFmQBmKtNeRn9tZj8H3hTsWuecezy8sEQkdMmeRd+ZYJjQyMho9VFKW0DznS9yoKuGeRZ0Ex0aorvmVTRf0JCDwCUsGSUEM1sKHAB+nLrPOTfOZYWIRFrq9NQDAxO/9uKLfVVRsk4+FqP1a3PYeMMAHP0DddZD94JX03XaWWy4ft7EnyWRlmmV0b8TjFIGaoBmYBcQ4UpCERlXPO5nKN2x44TdW/kom/ksHTTRRAcb+DrrujpO6k8bW7eSTSshHj/vlWEKG7SGR8HLtMro7NTHZvYG4BOhRCRSosJYJOmkz2zZSazt/8APf+inikiZgG4rH+U6vk4Nx1jIQbqo4zq+Dnu/x7o0gWh0ffGZ1hKazrnHgAuyHItIyQpjsryTPnPXQTZe10diV72fw2fMNBGb+Sw1HKOCETpo4jALOM4svrrjyhKcta80Zboewl+kbJ81s+8DnSHHJlIywpgs76TP3PcM9TUDxPdd7scXjNFBE2WM0EkDw5RTwRDGCH/gNBI33TeDn04KRaYlhLkpWzW+TeHdYQUlUmrCWCTppM882k3dnEHajy70A80WLRpdpAZoooODNFDGCOWMYICjjBr6iD+q5sJSkGmj8v9zzv0odYeZvQ/40TivF5EpyPZkeVs//iAP/8fZ9I7UMKfsOOe8po8z5tbR3V1Gc91B/6ILL4QHHvC9jIaH2XD863yC26higDJGGKKCISp4gz1OO+HO2hdG+4lMXaYlhBsy3Cci05DNRZK2fvxBrtt6JhUjA1QwzPGRSh56bh6PH15GV18VrYvv8QepqvLTUlxxBVx6Keve28sl5Y9QzjD9zKKKAd7Ioyye3x/q+AItNhQdk812+g5gDbDEzG5JeaqOzFdME5FJJCfLS71KHjtrc6Y2f/8UasqOU1vRT83wIN1uLn0jVfyuo5ZffKebWFsXtO9Je5Abt+5k4w3t1CfHF8xbStdpF4Q6viC1rQNGb+NxlRJybbIqo734RWzeBWxP2X8U+ExYQYmUouTJL5kUkg3KUz0pdgwsYGF5FwCzy/uZTT9uBA4O1/v5hdalmRI6GUMexhdosaHomDAhOOeeBJ40s39yzqlEIBKidMsSpC5BkKmmqkN0Dc6mtmx0jEHvSDVNVYeAyat+cj2+oOgWGypgE7YhmNkPg7uPm9lTY7ccxCdSMrLV9XTDn75M38gseoaqcSPQM1RN38gsNvzpy+EEPkPZbD+RmZmsyuja4PadYQciUuqyVXWy7vY3Aw+y+fun0DGwgKaqQ2z6098H+6Mnm+0nMjOTVRklF675c+fc51KfM7Mbgc+d/C6RwpePbpDZrDpZd/ubWXd78lEDcHoWIgyPpsGIhky7nb49zb53ZDMQkajIVzdIVZ1Ivk3WhvApM9sBnDGm/aAdUBuCFKUwppHIRFGt0ywFabI2hO8DPwf+F/D5lP1HnXOHQotKJI/y2Q1SVSeSTxOWEJxzXc653c65DznnXgD68Osi1AaL5ogUneZmX3efSt0gpRRkOtvplWb2W6AdeADYjS85iBQd1eVLqcq0UfmrwIXAc865ZuCPgF+HFpVIHqkuX0pVprOdDjrnDppZmZmVOefuC7qdihQl1eVLKco0IRwxs1rgQeCfzGw/GU5uZ2bl+PmQ9jjn3mlmC4A7gOX4qqf3O+cOTzVwERHJrkyrjN6Nb1D+DPAL4Hngygzfey3wTMrjzwP3OudeC9zLib2XRCQLEglYvx7WrPG3uZhKOh/HlOzKKCE453qdc8POuSHnXNw5d4tz7uBk7zOzU4E/Br6TsvvdQLJHdxy4aqpBS/HSSSW9qXwv+RhYpzUNisNkA9OOmll3mu2omXVP9N7A3wLXAyMp+5qSU2IEt4vGOfbVZtZmZm2dnVq+uRTopJLeVL+XfAysy9dgPsmuycYhzHXO1aXZ5jrn6iZ6r5m9E9jvnNs+0esmOPZtzrkW51xLY2PjdD5CCoxOKulN9XsJY33myeTjmJJ9mTYqT8clwLvMbA0wC6gzs+8BHWa22Dm3z8wWA/tDjEEKiBZKSW+q30su1hcYO/nf7Nn+GFrToLBl2qg8Zc65G5xzpzrnlgMfBH7lnPsw8FMgOcSnFfhJWDFIYYn6COF8tW9M9XsJe2BduiqsPXvghRc0mK/QhZYQJvA14O3ByOe3B49FIj1COJ/tG1P9XsIeWJeuCmvpUjj1VA3mK3TmnMt3DJNqaWlxbW1t+Q5DcmDrVti8GTo6oKnJL5Sybl2+o/IlgrHVMEeO+BPflvGXKM6afKzPMF4c73ufT0x1dXD66bBokX+8Zw/cfXfuY5Lxmdl251xLpq8Psw1BZEoSCbjjDli5Ei66yFeLJB/n+0oz3+0bURg5nSwlVVb6BHD8OLS1QUsLVFVFp2pPpi8fVUaSgVLsjx/lXkZRb9/IheTv5+yzYXAQnPPJYceO6FTtycwoIURQqfbHj3LXxSi3b+RK8vezaJEvFdTU+MQwOKj2gmKhKqMISr1ShtHbeLy4/+ly0V1yuqKyEHw+2xJSfz+LFvkt2Y5SzH+XpUQlhAiK8pVymKJ+FR6L+Qbku+/2t/lIBvksOUb99yMzp4QQQaVaX611CCaW7zYW/X6Kn6qMIiTZ5fKll2BoCM48E8491yeDri5fRVHsotCbJqry3dMJ9PspdiohRMTWrXDddf7Ef8opfiqAp56CRx7RlZh4pVpylNxRQoiIzZt9r43aWjCDhQuhsdHX1eajvlqiR3X4EjYlhIjo6IA5c/z948fh5Zfh0CF4/vni724qmVEdvoRNbQgR0dTkr/YqKqCz0zcamvkRoBs36h9fPNXhS5hUQoiIDRugr88nAzNfJTA0BKtWRWe0rogUN5UQIiI5gds11/iRn7Nnw/nnw+te55NDsY9BkMIUlUn3JDuUECJk3To/Wdhko3X1TyhRkBwoV19/4kA5VW8WLlUZRcxkPUnyPVpVJCnfA+Uk+5QQImayniT6J5SoKNUpVoqZqowiaKKeJNkarapqp/Hpu8lMlCcjlOlRCaHAZGO0qqqdxqfvJnMaKFd8lBAKTDb+CVXtND59N5nTQLnioyqjApONefmjMElaVOm7mRoNlCsuSggFaKb/hKr7HZ++GyllqjIqQar7HTV27eqWFn03UrqUEEqQ6n69dA3Id9wBH/iAvhspTaoyKlH5rvuNQtfO8daubmvzU45nWxR+5ijGItERWgnBzGaZWcLMnjSznWb25WD/l8xsj5k9EWxrwopBoikqXTtzObAqKj9z1GKRaAmzyqgfeJtz7lxgFXCFmV0YPPcN59yqYLs7xBgkgqLStTOXK5BF5WeOWiwSLaElBOf1BA8rg82FdTwpHFGZ8iCXjetR+ZmjFotES6iNymZWbmZPAPuBXzrnHgme+rSZPWVmt5vZ/HHee7WZtZlZW2dnZ5hhSo5FZW3gXDauR+VnjlosEi2hJgTn3LBzbhVwKhAzs7OAW4EV+GqkfcDmcd57m3OuxTnX0tjYGGaYkmNR6vYai/kG5LvvDnft6ij9zFGKRaIlJ91OnXNHgPuBK5xzHUGiGAG+DahvQ4kpxW6vUfqZoxSLRIs5F061vpk1AoPOuSNmVgPcA9wIbHfO7Qte8xngjc65D070WS0tLa6trS2UOIuFuhGKyFhmtt0515Lp68MsISwG7jOzp4BH8W0IdwE3mdmOYP+lwGdCjKEkqBuhiGRDaAPTnHNPAeel2f+RsI5ZqpLdCAcG4De/gaNHoaoKbroJ7rwz39FJ2FQ6lGzR1BVFoL0djh/3I2z7+mD2bN9Y+MADKiUUO5UOJZuUEIpAczPs3AmVlVBdDWZ+y9Zgo0QC1q6FZcv8tnatTjhRoUFmkk1KCEUg2Y0waWDAb2edNfPBRokEXHMNbNsG5eV+27YNrr1WSSEKNMhMskkJoQjEYvCWt/grxN5emDXLT+NcXT3zwUbxuK+GmD3bf151NcyZA52dugqNAg0yk2xSQigS118Pr3kNXHQRXHyxb1TOxmCj9nbo7/efl1RV5ffpKjT/NHOSaIAAAArWSURBVMhMskkJoUiENdioudmXCgYGRvcNDGSn9BElYxfKKZTqMA0yk2wKbWBaNmlg2tRlqytisg1h925fbQRw7Jj/zG9+s/BPPIkEfO5z8PDDvspt0SJYsQIqKnRilcIXpYFpkifZ7IoYi8Ett8Dq1TA87LfVq4snGVx7LTzyyGiDeUcH7NgBQ0NqI5HSoxXTitB4K4HF49M7icdixTnALR73jePgSwRm/v7x47B374ntJiKlQCWEIpSLroiFWueeKtlgXl3tG2TBVxsND8Phw8XVRiKSCSWEIhR2V8RiGR2bbDBPjuxOVomBH+SnnjpSapQQilDYXRGLZXRsays0NvokkCxRDQz4cRZf/nLht5GITJUSQhEKuytisYyOjcV84/jq1b5EMH8+XHkl/OxnsG5dvqMTyT01KhepWCy8K9zmZl9NlGyshsIdHVusDeYi06ESgkyZRseKFCclBJkyjY4VKU6qMpJpCbNKSkTyQyUEEREBlBBERCSghCAiIoASgoiIBJQQREQEUC8jKWHZWjNCpFiEVkIws1lmljCzJ81sp5l9Odi/wMx+aWa/DW7nhxWDyHiKZYI+kWwKs8qoH3ibc+5cYBVwhZldCHweuNc591rg3uCxSE4VywR9ItkUWkJwXk/wsDLYHPBuIPlvFweuCisGkfFkOkFfMaz7IJKpUBuVzazczJ4A9gO/dM49AjQ55/YBBLeLwoxBJJ1M1oxQtZKUmlATgnNu2Dm3CjgViJnZWZm+18yuNrM2M2vrTK5zKDJDySv+xx+Hhx6C558ff4I+VStJqclJt1Pn3BHgfuAKoMPMFgMEt/vHec9tzrkW51xLY2NjLsKUIpd6xb9yJZx+Ojz3HOzcmX6CvmJZ90EkU2H2Mmo0s3nB/RrgMuBZ4KdA8jqsFfhJWDGIpBp7xb9iBVx0EZx3HmzZcnKX07CXIhWJmjBLCIuB+8zsKeBRfBvCXcDXgLeb2W+BtwePRUI31St+rfsgpSa0gWnOuaeA89LsPwj8UVjHFRnPVFd6S677kDp4bcMGDV6T4qWRylIyWlt9GwL4kkF3t7/i37Bh/Pdo3QcpJZrLSEqGVnoTmZhKCFJSdMUvMj6VEEREBFBCEBGRgKqMRIqYpviWqVAJQaRIaS4mmSolBJEipbmYZKqUEESKlOZikqlSQhApUpqLSaZKCUGkSGkuJpkqJQSRIqWR2TJV6nYqUsQ0MlumQiUEEREBlBBERCSghCAiIoASgoiIBJQQREQEAHPO5TuGSZlZJ/BCHkNoAA7k8fjpKKbMRDEmiGZciikzUYwJ0se1zDnXmOkHFERCyDcza3POteQ7jlSKKTNRjAmiGZdiykwUY4LsxKUqIxERAZQQREQkoISQmdvyHUAaiikzUYwJohmXYspMFGOCLMSlNgQREQFUQhARkYASgoiIAEoIJzCz95nZTjMbMbOWlP0xM3si2J40s/ekPHe+me0ws9+Z2S1mZjmK6e1mtj049nYze1sEYlpoZveZWY+ZfWvMe0KNaaK4guduCI69y8z+Sy7jSjnWuWb2UHC8n5lZXcpzaeMLm5mtMrOHg7/tNjOLpTyXl5iCY9+R8j+328yeiEhc/z047k4zuynfMZnZl8xsT8p3tWZGMTnntAUb8HrgDOB+oCVl/2ygIri/GNif8jgBXAQY8HPgHTmK6TzgVcH9s4A9Kc/lK6Y5wGrgk8C3xrwn1JgmietM4EmgGmgGngfKcxVXShyPAm8J7n8c+Mpk8eXgb/6e5M8MrAHuz3dMaWLcDGzMd1zApcB/AtXB40URiOlLwGfT7J9WTCohpHDOPeOc25Vm/zHn3FDwcBbgAMxsMVDnnHvI+d/CPwJX5Simx51ze4OHO4FZZlad55h6nXPbgOOp+3MR00RxAe8G/tk51++cawd+B8RyFVeKM4AHg/u/BN47UXwhxpHKAcmSSj2Q/JvKZ0yvCEps7wd+EIG4PgV8zTnXD+Cc2x+BmMYzrZiUEDJkZm80s53ADuCTQYJYAryU8rKXgn259l7g8eAPNSoxpcp3TEuAP6Q5fq7jehp4V3D/fcBpk8SXC/8DuNnM/gB8HbghAjGlehPQ4Zz7bfA4n3GdDrzJzB4xswfM7IIIxATwaTN7ysxuN7P5M4mp5FZMM7P/BE5J89RfOud+Mt77nHOPACvN7PVA3Mx+jq9mOOmluYopeO9K4Ebg8uSufMeU7uOyERNMO67xjp+1uF450ATx4auJbjGzjcBPgYFJ4suKSWL6I+Azzrl/MbP3A38PXBZ2TJPFlfK7/BCjpQPCjmuS76oCmA9cCFwA/NDMXp3nmG4FvhIc7yv46rWPTzemkksIzrnLZvj+Z8ysF19v/xJwasrTpzJa5A49JjM7Ffgx8FHn3PPB7rzGNI6sxATTjuslRq/GU4+ftbiSMojvcgAzOx3440niy4qJYjKzfwSuDR7+CPhOLmKaLK4gtgrgT4DzU3bn87v6FPCvQfViwsxG8BPK5S2mMfF9G7greDitmFRllAEzaw7+ODGzZfi64N3OuX3AUTO7MKjr/Cgw1avn6cY0D/h34Abn3K+T+/MZ03giENNPgQ8GbSzNwGuBRK7jMrNFwW0Z8AXg/04UX1hxjLEXeEtw/21AsmomnzElXQY865xLrdbLZ1z/hv+Okgm9Cj+7aN5iCtrBkt6Dr5Zk2jHloiW8ULbgC30J6Ac6gP8I9n8E33D7BPAYcFXKe1qCX8LzwLcIRn/nIKYvAL1BTMltUT5jCp7bDRwCeoLXnJmLmDKI6y+DY+8ipSdRLuJKOda1wHPB9rXUY40XXw7+5lcD2/E9Uh4Bzs93TCnH/wd8e93Y/fn6rqqA7wV/L48Bb4tATN/Ft2s+hU8Ci2cSk6auEBERQFVGIiISUEIQERFACUFERAJKCCIiAighiIhIQAlBSoKZ9YTwme8ys88H968yszOn8Rn325iZWUXyRQlBZJqccz91zn0teHgVfoZJkYKlhCAlxbybzexp8+sSfCDY/9bgav1OM3vWzP4pGL2Mma0J9m0zv2bCXcH+j5nZt8zsYvykdTcHc9KvSL3yN7MGM9sd3K8xs38OJiO7A6hJie1y8+slPGZmPzKz2tx+O1LqSm4uIyl5fwKsAs7Fz0PzqJklp6Q+D1iJn87h18AlZtYG/B3wZudcu5n9YOwHOud+Y2Y/Be5yzt0JYOOvs/Mp4Jhz7hwzOwc/4hUza8CPPr/MOddrZp8D/gLYlI0fWiQTSghSalYDP3DODQMdZvYAfubKbvz8Ri8BmF+hazl+Co7fOz+nPPiZN6+ewfHfDNwC4Jx7ysyeCvZfiK9y+nWQTKqAh2ZwHJEpU0KQUjPREpn9KfeH8f8f011Sc4jRKtlZY55LN1+MAb90zn1omscTmTG1IUipeRD4gJmVm1kj/op9olkgnwVebWbLg8cfGOd1R4G5KY93Mzpt89oxx/+vAGZ2FnBOsP9hfBXVa4LnZgczaorkjBKClJof42eGfBL4FXC9c+7l8V7snOsD/hz4hZltw8+i2pXmpf8MXGdmj5vZCvzqY58ys9/g2yqSbgVqg6qi6wmSkXOuE/gY8IPguYeB183kBxWZKs12KjIJM6t1zvUEvY62AL91zn0j33GJZJtKCCKT+7OgkXknfiH6v8tzPCKhUAlBREQAlRBERCSghCAiIoASgoiIBJQQREQEUEIQEZHA/wf+MJxwSO7LjwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "canada = train_df.query('country == \"Canada\"')\n",
    "usa = train_df.query('country == \"USA\"')\n",
    "plt.scatter(canada[\"longitude\"], canada[\"latitude\"], color=\"red\", alpha=0.6)\n",
    "plt.scatter(usa[\"longitude\"], usa[\"latitude\"], color=\"blue\", alpha=0.6)\n",
    "plt.ylabel(\"latitude\")\n",
    "plt.xlabel(\"longitude\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###  Gaussian Naive Bayes\n",
    "\n",
    "- Our cities dataset has continuous-valued features.\n",
    "- But so far, we've only seen how to use Naive Bayes for discrete features.\n",
    "- We can either discretize our continuous features into discrete bins (with counts), or...\n",
    "- Use _Gaussian_ naive Bayes (read more [here](https://machinelearningmastery.com/naive-bayes-for-machine-learning/) and [here](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Gaussian_naive_Bayes))\n",
    "- This sounds complicated but is actually quite simple:\n",
    "    - Assume each feature is normally distributed \n",
    "    - Calculate the mean ($\\mu_k$) and standard deviation ($\\sigma_k$) for each feature for each class\n",
    "    - Use the following equation to calculate the conditional probability of observing feature value $v$ in class $C_k$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src='./images/gaus_nb.png' width=\"400\">\n",
    "\n",
    "\n",
    "Source: [Wikipedia](https://en.wikipedia.org/wiki/Naive_Bayes_classifier#Gaussian_naive_Bayes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f8421a30880>,\n",
       "        <matplotlib.axes._subplots.AxesSubplot object at 0x7f84218c3580>]],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAa4ElEQVR4nO3df7RdZX3n8feHmEIkINCEawjYaEEqkhIWtxm6WM5cBDSCCI6DA0UIAzNxZkoHO3EwqFOhODPRJcq0ZVmDWDKKYtbCFIy2NY3csc6iYCKBhAYmCCkSMokgCBe7aC9854/9XNg5Offefc7d58eT+3mtddY5+9lnn/O9+zzne/d5nr2fRxGBmZnl54BeB2BmZu1xAjczy5QTuJlZppzAzcwy5QRuZpYpJ3Azs0w5gVcgaYekMzv8HiOS3lLj64WkY+t6PbN2vweS3iHpkRrjGJL0ZF2vlzMn8D4REbMj4jEASbdK+nSvYzJrR+PBQ0T8TUQcX1rf8QOi6cIJ3MwsU07gLZB0oKQbJT2VbjdKOjCtG5L0pKTlkvZI2iXp35S2/VVJ35b0vKQfSfq0pB+W1oekYyUtAy4Grk7NKt8ury89f6+jdEn/Jb3nU5IubxL35yQ9IWm3pD+VNKtze8r2Z5IWS7pH0nOpzv2JpF9J636QnvZAqr//utzkIemrwJuAb6f1VzdrEikfpUualer7s5L+DvithuceJekOST+T9Lik/9TpfdAvnMBb8wngVGARcBKwGPhkaf0bgTcA84ErgJskHZ7W3QS8mJ6zNN32ERGrgNuAz6ZmlXMnC0rSEuCjwFnAcUDjz9PPAG9NcR+b4vuDyV7XbBwvA78PzAF+GzgD+I8AEfHP03NOSvX3m+UNI+IS4Ang3LT+sxXe71PAr6fbuyl9dyQdAHwbeICiXp8BfETSu9v/8/LhBN6ai4E/jIg9EfEz4DrgktL6f0rr/ykivguMAMdLmgF8APhURPwyIv4OWF1jXB8E/iwitkbEi8C1YyskCfh3wO9HxM8j4gXgvwMX1vj+No1ExKaI+NuIGI2IHcCXgH/Rwbf8IPDfUv39KfBHpXW/BcyNiD+MiH9M/Ug3M03q9+t6HUBmjgL+vrT896lszDMRMVpa/iUwG5hLsa9/WlpXflxHXJsa4hozF3g9sKnI5QAImFHj+9s0IumtwOeBQYq69Tr2rn91O4q9vy/l+v1rwFGSniuVzQD+poPx9A0fgbfmKYoKM+ZNqWwyPwNGgaNLZcdM8PxmQ0T+kuLLMuaNpce7Gl7vTaXHTwP/ALw9Ig5LtzdExOwKcZs180XgYeC4iDgU+DjFQUFVjfX7RUp1O/1inVtaP1H9/inweKluHxYRh0TE2S3Eky0n8NZ8A/ikpLmS5lC0I39tso0i4mXgW8C1kl4v6TeASyfYZDfQeE74ZuB3JM1Ibd7ln6xrgMsknSDp9RRthmPv/QrFT8ovSDoSQNL86dJGaB1xCPA8MJLq8n9oWN+s/k60/v8CB0k6R9JMin6lA0vr1wDXSDpc0tHA75XW3Qc8L+ljqbNzhqQTJe3V0bm/cgJvzaeBjcCDwBbgx6msiispOjj/H/BVin8GL43z3FuAE1Iv/5+nsquAc4HnKNrix8qJiL8AbgS+Dzya7ss+lsr/VtLzwF8Dx2PWno8CvwO8QHFw8M2G9dcCq1P9/WCT7f8HxYHQc5I+GhG/oOgE/TKwk+KIvHxWynUUzSaPA9+j+P4Arx4cnUvRQf84xS/OL1N81/Z78oQOvSHpM8AbI6Lp2ShmZpPxEXiXSPoNSb+pwmKK0wzX9jouM8uXz0LpnkMomk2OAvYANwB39jQiM8uam1DMzDLlJhQzs0x1tQllzpw5sWDBgtpf98UXX+Tggw+u/XXr0u/xQf/HWI5v06ZNT0fE3Ek26QtTrfP9/rmU5RQr5BXvuHU+Irp2O+WUU6IT7r777o68bl36Pb6I/o+xHB+wMbpYb6dym2qd7/fPpSynWCPyine8Ou8mFDOzTDmBm5llygnczCxTTuBmZplyAjczy5QTuJlZpion8DRM4/2S1qXlIyStl7Q93R8+2WuYmVl9WjkCvwrYVlpeAWyIiOOADWnZzMy6pFICT4Oon0Mxzu6Y83htXsfVwPn1hmZmZhOpein9jcDVFCPqjRmIiF0AEbFrbLaXRpKWAcsABgYGGB4ebj/acYyMjHTkdevS6fi27PxFy9ssnL/3ePfTfR9a7yxY8Z2Wt9mx8pwORJKfSRO4pPcCeyJik6ShVt8gIlYBqwAGBwdjaKjll5jU8PAwnXjdunQ6vsva+QJcPLTX8nTfh2Y5qnIEfhrwPklnAwcBh0r6GrBb0rx09D2PYoxrMzPrkknbwCPimog4OiIWABcC34+IDwF3AWPTgS3FkxOYmXXVVM4DXwmcJWk7cFZaNjOzLmlpPPCIGAaG0+NngDPqD8nMzKrwlZhmZplyAjczy5QTuJlZppzAzcwy5QRuZpYpJ3Azs0w5gZuZZcoJ3MwsU07gZmaZcgI3M8uUE7iZWaacwM3MMuUEbmaWKSdwM7NMOYGbmWXKCdzMLFOTJnBJB0m6T9IDkh6SdF0qv1bSTkmb0+3szodr1h2Sdkjakur2xlR2hKT1kran+8N7HadNb1WOwF8C3hkRJwGLgCWSTk3rvhARi9Ltux2L0qw3Tk91ezAtrwA2RMRxwIa0bNYzVSY1jogYSYsz0y06GpVZfzoPWJ0erwbO72EsZtXmxJQ0A9gEHAvcFBH3SnoPcKWkS4GNwPKIeLZzoZp1VQDfkxTAlyJiFTAQEbsAImKXpCObbShpGbAMYGBggOHh4baDGBkZmdL23dRurMsXjra8TR37JKd9Ox5FVD+YlnQYsBb4PeBnwNMUFf16YF5EXN5km3JlPuX222+vIey9jYyMMHv27Npfty6djm/Lzl+0vM3C+W/YazmnfXj66advKjVrdISkoyLiqZSk11PU+bsi4rDSc56NiAnbwQcHB2Pjxo1txzE8PMzQ0FDb23dTu7EuWPGdlrfZsfKclrdplNO+ldS0zrc6K/1zkoaBJRHxudKL3wysG2ebVcAqKCpzJ3ZYv38QnY7vsna+ABcP7bU83fdho4h4Kt3vkbQWWAzsljQvHX3PA/Z0LSCzJqqchTI3HXkjaRZwJvBwqsBj3g9s7UyIZt0l6WBJh4w9Bt5FUb/vApampy0F7uxNhGaFKkfg84DVqR38AGBNRKyT9FVJiyiaUHYAH+5cmGZdNQCslQTFd+TrEfGXkn4ErJF0BfAEcEEPYzSbPIFHxIPAyU3KL+lIRGY9FhGPASc1KX8GOKP7EZk111IbuO0/GjuOli8cnbQtvY6OIzOrjy+lNzPLlBO4mVmmnMDNzDLlBG5mlikncDOzTDmBm5llygnczCxTTuBmZplyAjczy5QTuJlZppzAzcwy5QRuZpYpJ3Azs0w5gZuZZcoJ3MwsU07gZmaZqjIn5kGS7pP0gKSHJF2Xyo+QtF7S9nQ/4ezcZmZWrypH4C8B74yIk4BFwBJJpwIrgA0RcRywIS2bmVmXTJrAozCSFmemWwDnAatT+Wrg/I5EaGZmTVWaEzPNSL8JOBa4KSLulTQQEbsAImKXpCPH2XYZsAxgYGCA4eHhWgIvGxkZ6cjr1qXT8S1fODrl1xiYNfnr9HIf9/tnbNYLlRJ4RLwMLJJ0GLBW0olV3yAiVgGrAAYHB2NoaKidOCc0PDxMJ163Lp2Ob7LJiKtYvnCUG7ZMXB12XDw05fdpV79/xma90NJZKBHxHDAMLAF2S5oHkO731B6dmZmNq8pZKHPTkTeSZgFnAg8DdwFL09OWAnd2KkgzM9tXlSaUecDq1A5+ALAmItZJugdYI+kK4Anggg7GaWZmDSZN4BHxIHByk/JngDM6EZSZmU3OV2KamWXKCdxsHJJmSLpf0rq07KuPra84gZuN7ypgW2nZVx9bX3ECN2tC0tHAOcCXS8W++tj6SqULecymoRuBq4FDSmVdv/o4pytQ2421nSuJ//i21s9aXjj/DXst57Rvx+MEbtZA0nuBPRGxSdJQq9vXefVxTlegthtrHVcSV9F4JXFO+3Y8TuBm+zoNeJ+ks4GDgEMlfY109XE6+vbVx9ZzbgM3axAR10TE0RGxALgQ+H5EfAhffWx9xgncrLqVwFmStgNnpWWznnETitkEImKYYgA3X31sfSfrBL4gdX4sXzhauSNkx8pzOhmSmVnXuAnFzCxTTuBmZplyAjczy5QTuJlZppzAzcwy5QRuZpapKnNiHiPpbknbJD0k6apUfq2knZI2p9vZnQ/XzMzGVDkPfBRYHhE/lnQIsEnS+rTuCxHxuc6FZ2Zm46kyJ+YuYGwIzRckbQPmdzowMzObWEtXYkpaQDHB8b0UI7ZdKelSYCPFUfqzTbapbWzkRmPjCA/Mqj6mcC/G/+30uMPtjKfcqMo+7OXYyfvD2M1mdaucwCXNBu4APhIRz0v6InA9EOn+BuDyxu3qHBu50WWlS+lv2FLtT2kcE7gbOj3ucB3jKVfZh73Yd2P2h7GbzepW6SwUSTMpkvdtEfEtgIjYHREvR8QrwM3A4s6FaWZmjaqchSLgFmBbRHy+VD6v9LT3A1vrD8/MzMZTpd3hNOASYIukzans48BFkhZRNKHsAD7ckQjNzKypKmeh/BBQk1XfrT8cMzOryldimpllygnczCxTTuBmZplyAjczy5QTuJlZppzAzcwy5QRuZpYpJ3Azs0w5gZuZZcoJ3MwsU07gZmaZcgI3M8uUE7hZA0kHSbpP0gNpIu/rUvkRktZL2p7uD+91rDa9OYGb7esl4J0RcRKwCFgi6VRgBbAhIo4DNqRls55xAjdrEIWRtDgz3QI4D1idylcD5/cgPLNXtTSpsdl0IWkGsAk4FrgpIu6VNBARuwAiYpekI8fZtraJvHOazLndWOuYlLuKxthy2rfjcQI3ayIiXgYWSToMWCvpxBa2rW0i75wmc2431jom5a6icVLunPbteKrMiXmMpLslbUsdOlelcnfo2H4vIp4DhoElwO6xuWDT/Z4ehmZWqQ18FFgeEW8DTgV+V9IJuEPH9lOS5qYjbyTNAs4EHgbuApampy0F7uxNhGaFKnNi7gLG2v1ekLQNmE/RoTOUnraa4ijlYx2J0qy75gGrUzv4AcCaiFgn6R5gjaQrgCeAC3oZpFlLbeCSFgAnA/cCXe/QaTTW+TEwq3pHSC86LTrdWVJHJ1CVfdjLDp9udjhFxIMU9byx/BngjK4EYVZB5QQuaTZwB/CRiHheajZR/b7q7NBpNNb5sXzhKDdsqfanNHZkdEOnO0vq6ASqsg97se/G7A8dTmZ1q3QeuKSZFMn7toj4Vip2h46ZWQ9VOQtFwC3Atoj4fGmVO3TMzHqoSrvDacAlwBZJm1PZx4GVuEPHzKxnqpyF8kNgvAZvd+hMIwvabGvfsfKcmiMxM/BYKGZm2XICNzPLlBO4mVmmnMDNzDLlBG5mlikncDOzTDmBm5llygnczCxTTuBmZplyAjczy5QTuJlZppzAzcwy5QRuZpYpJ3Azs0w5gZuZZaqlSY2tuvLY2csXjlaet9JjZ5tZVT4CNzPLVJU5Mb8iaY+kraWyayXtlLQ53c7ubJhmZtaoyhH4rcCSJuVfiIhF6fbdesMyM7PJTJrAI+IHwM+7EIuZmbVgKp2YV0q6FNgILI+IZ5s9SdIyYBnAwMAAw8PDU3jLvS1fOArAwKzXHk+mzvefSDmeTsdX9bUn0kqMrapjn4+MjHTtszPLRbsJ/IvA9UCk+xuAy5s9MSJWAasABgcHY2hoqM233NfYmR3LF45yw5Zqf8qOi+t7/4lc1nAWSifjq3qGy0RaibFVdezz4eFh6qw7ZvuDts5CiYjdEfFyRLwC3Awsrjcss96RdIykuyVtk/SQpKtS+RGS1kvanu4P73WsNr21lcAlzSstvh/YOt5zzTI0StEs+DbgVOB3JZ0ArAA2RMRxwIa0bNYzk/5mlvQNYAiYI+lJ4FPAkKRFFE0oO4APdzBGs66KiF3ArvT4BUnbgPnAeRTfBYDVwDDwsR6EaAZUSOARcVGT4ls6EItZ35G0ADgZuBcYSMmdiNgl6chxtqmt4z6nztt2Y+1U53mjxthy2rfj8aX0ZuOQNBu4A/hIRDwvqdJ2dXbc59R5226sdXTCV9HYmZ7Tvh2PL6U3a0LSTIrkfVtEfCsV7x7r/0n3e3oVnxk4gZvtQ8Wh9i3Atoj4fGnVXcDS9HgpcGe3YzMrcxOK2b5OAy4BtkjanMo+DqwE1ki6AngCuKBH8ZkBTuBm+4iIHwLjNXif0c1YzCbiJhQzs0w5gZuZZcoJ3MwsU07gZmaZcgI3M8vUtDsLZUEbV315omEz60c+Ajczy5QTuJlZppzAzcwyNe3awM2sc9rpY7L2+QjczCxTTuBmZpmaNIFL+oqkPZK2lso8uauZWY9VOQK/FVjSUObJXc3MemzSBB4RPwB+3lB8HsWkrqT782uOy8zMJtHuWSiVJneFeid4bTQ2GerArM5OjDrViVpbia9Xk8J2ch/W8ZnvDxPQmtWt46cR1jnBa6OxyVCXLxzlhi2d+1MaJ0OtojxRayvxTfW92tXJfdjO39Rof5iA1qxu7Z6F4sldzcx6rN0E7sldzcx6rMpphN8A7gGOl/RkmtB1JXCWpO3AWWnZzMy6aNJGz4i4aJxVntzVbD/W6mXxRSd4/47O0fj3LF84Wqn/qJ+Hk/aVmGZmmXICNzPLlBO4mVmmnMDNzDLVvz0O05THUzazqnwEbmaWKSdwsyY8jLLlwAncrLlb8TDK1uecwM2a8DDKlgN3YppVV2kY5TqHUO7lMLqtDi/c6WGd61Y13n4extgJvAKfGWKtqHMI5V4Oo9vqMMWdHta5blXjrWM45E5xE4pZdR5G2fqKE7hZdR5G2fpKPr93zLooDaM8BMyR9CTwKYphk9ekIZWfAC7odlxuzrMyJ3CzJjyMsuXATShmZplyAjczy9SUmlAk7QBeAF4GRiNisI6gzMxscnW0gZ8eEU/X8DpmZtYCN6GYmWVqqkfgAXxPUgBfSleg7aXOy4objV0G2++X8PZ7fNDZGOv4zHt5SblZv5pqAj8tIp5KY0Ksl/RwGgToVXVeVtxo7FLffr+Et9/jg87GWMelyL28pNysX02pCSUinkr3e4C1wOI6gjIzs8m1ncAlHSzpkLHHwLuArRNvZWZmdZnKb+YBYK2ksdf5ekT8ZS1RmZnZpNpO4BHxGHBSjbGYmVkLfBqhmVmmnMDNzDLlBG5mlikncDOzTPXN1SUeqN7MrDV9k8DNppsqBy3LF462PLmwTR9uQjEzy5QTuJlZppzAzcwy5TZwM7MJdOsEix0rz2l5Gx+Bm5llygnczCxTTuBmZplyAjczy5QTuJlZpnwWinVcO7347fTIm003PgI3M8vUlBK4pCWSHpH0qKQVdQVl1q9c562fTGVS4xnATcB7gBOAiySdUFdgZv3Gdd76zVSOwBcDj0bEYxHxj8DtwHn1hGXWl1znra8oItrbUPpXwJKI+Ldp+RLgn0XElQ3PWwYsS4vHA4+0H+645gBPd+B169Lv8UH/x1iO79ciYm63A+hRne/3z6Usp1ghr3ib1vmpnIWiJmX7/DeIiFXAqim8z+SBSBsjYrCT7zEV/R4f9H+MfRJf1+t8n/zdleQUK+QXbzNTaUJ5EjimtHw08NTUwjHra67z1lemksB/BBwn6c2SfgW4ELirnrDM+pLrvPWVtptQImJU0pXAXwEzgK9ExEO1RdaajjbR1KDf44P+j7Hn8fWozvf8725BTrFCfvHuo+1OTDMz6y1fiWlmlikncDOzTGWVwCVdIOkhSa9IGiyVnyVpk6Qt6f6dpXXD6dLnzel2ZC9iTOuuSZdgPyLp3aXyU1Lsj0r6I0nNTlfrRKzfLO2XHZI2p/IFkv6htO5PuxHPODFeK2lnKZazS+ua7s8cSTpI0n2SHkj157pUfoSk9ZK2p/vDex3rGEkzJN0vaV1a7udYd6Tv2GZJG1NZ38ZbWURkcwPeRnFhxDAwWCo/GTgqPT4R2Flat9dzexjjCcADwIHAm4GfADPSuvuA36Y4z/gvgPf0YN/eAPxBerwA2NrrzzvFci3w0Sbl4+7PHG/ps5+dHs8E7gVOBT4LrEjlK4DP9DrWUsz/Gfg6sC4t93OsO4A5DWV9G2/VW1ZH4BGxLSL2uaotIu6PiLHzcR8CDpJ0YHejezWWpjFSXHJ9e0S8FBGPA48CiyXNAw6NiHuiqEn/Czi/iyGTjvg/CHyjm+87RU33Z49jalsURtLizHQLir9zdSpfTZfrxngkHQ2cA3y5VNyXsU4gt3j3kVUCr+gDwP0R8VKp7M/ST6f/2q3miSbmAz8tLT+Zyuanx43l3fQOYHdEbC+VvTn9PP7fkt7R5XgaXSnpQUlfKf3MHW9/Zis1SWwG9gDrI+JeYCAidgGk+442AbbgRuBq4JVSWb/GCsU/w++lJtaxYQ76Od5K+m5CB0l/DbyxyapPRMSdk2z7duAzwLtKxRdHxE5JhwB3AJdQHOV2O8bxLsOudHl2uyrGehF7H33vAt4UEc9IOgX4c0lvj4jn64qraozAF4HrKfbJ9RRNPZfT4f3WCxHxMrBI0mHAWkkn9jqmZiS9F9gTEZskDfU6nopOi4inUh/YekkP9zqgOvRdAo+IM9vZLv2kWwtcGhE/Kb3eznT/gqSvU/zMnlICbzPG8S7DfjI9biyvxWSxSnod8C+BU0rbvAS8lB5vkvQT4K3AxrriaiXGMZJuBtalxf32svaIeE7SMLAE2C1pXkTsSs1te3obHQCnAe9LHcoHAYdK+hr9GSsAY02sEbFH0lqKPNC38Va1XzShpCOW7wDXRMT/KZW/TtKc9Hgm8F5ga2+i5C7gQkkHSnozcBxwX/rp9oKkU1PzzqXAhL80anYm8HBEvNqMI2muirGvkfSWFOtjXYzpVemLNeb9vPb5Nd2f3Y6vLmmfH5YezyJ9LhR/59L0tKV0t240FRHXRMTREbGAYjiB70fEh+jDWAEkHZx+gSPpYIpf6Fvp03hb0ute1FZuFF/gJymODncDf5XKPwm8CGwu3Y4EDgY2AQ9SdG7+Tzp8psJ4MaZ1n6A4W+IRSmeaAIMUFeonwJ+QrpDt0j69Ffj3DWUfSPvrAeDHwLk9/My/CmxJn+FdwLzJ9meON+A3gfvT37mV184I+lVgA7A93R/R61gb4h7itbNQ+jJW4C2pLj+Q6vUn+jneVm6+lN7MLFP7RROKmdl05ARuZpYpJ3Azs0w5gZuZZcoJ3MwsU07gZmaZcgI3M8vU/weCweBLez871QAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_train.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Gaussian naive Bayes assumes normality\n",
    "    - Are our features normal?\n",
    "    - Not really but in practice we transform our data to try and make it more normal\n",
    "    - Scikit-learn provides the `PowerTransformer()` for this process\n",
    "    - From the [docs](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.PowerTransformer.html#sklearn.preprocessing.PowerTransformer): \"*...Power transforms are a family of parametric, monotonic transformations that are applied to make data more Gaussian-like.*\"    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import PowerTransformer\n",
    "\n",
    "pt = PowerTransformer(method='yeo-johnson')\n",
    "X_train[['longitude', 'latitude']] = pt.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[<matplotlib.axes._subplots.AxesSubplot object at 0x7f84223bde80>,\n",
       "        <matplotlib.axes._subplots.AxesSubplot object at 0x7f8422099760>]],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAEICAYAAABGaK+TAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAV9ElEQVR4nO3df7CkVZ3f8fdHZBWFVQxwHQF3ksi6MU7EqlliyiR7I84uCxq1Eo1KEKPJmB8kuDVGWN2KuLqbcSNqJUuZHX/EiUuMVCkBWU1k0VtqlaKMQQZEF0tHBCYz/kIYTLk78M0f/VzTXPrO7du3+3afO+9XVVf386O7v88zp79z7nnOeU6qCklSex417QAkSaMxgUtSo0zgktQoE7gkNcoELkmNMoFLUqNM4ENIsi/J8yf8HYeS/JUxfl4ledq4Pk8a9XeQ5O8k+eYY45hPcte4Pq9lJvAZUVXHV9W3AZJ8KMnbpx2TNIqllYeq+nxVPb1v+8QrREcLE7gkNcoEvgpJHpPkPUnu6R7vSfKYbtt8kruS7EhyMMn+JP+k771/KcknktyX5CtJ3p7kC33bK8nTkmwHzgfe2DWrfKJ/e9/+D6ulJ/m33Xfek+Q1A+J+Z5I7kxxI8p+THDe5M6WNLMlZSb6Y5N6uzP1hkl/otn2u2+1rXfn9R/1NHkk+DDwV+ES3/Y2DmkT6a+lJjuvK+4+TfB341SX7PiXJx5J8P8l3kvybSZ+DWWECX503A88BzgSeBZwF/E7f9icDTwBOBV4LXJHkxG7bFcAD3T4Xdo9HqKpdwJXAH3TNKi9cKagk5wBvALYBZwBL/zx9B/DLXdxP6+L7dyt9rrSMB4HfAk4C/hZwNvAvAarq73b7PKsrvx/tf2NVXQDcCbyw2/4HQ3zfW4C/2j1+g77fTpJHAZ8AvkavXJ8NvD7Jb4x+eO0wga/O+cDvVtXBqvo+8Fbggr7tf9Ft/4uq+iRwCHh6kmOAfwC8pap+WlVfB3aPMa6XAf+lqm6tqgeAyxY3JAnwz4DfqqofVdX9wO8DLx/j9+soUlV7qupLVXW4qvYBfwT82gS/8mXA73Xl93vAf+zb9qvAyVX1u1X15911pPdxlJTvR087gMY8Bfhu3/J3u3WLflhVh/uWfwocD5xM71x/r29b/+txxLVnSVyLTgYeB+zp5XIAAhwzxu/XUSTJLwPvArbSK1uP5uHlb9yewsN/L/3l+5eApyS5t2/dMcDnJxjPzLAGvjr30Cswi57arVvJ94HDwGl9604/wv6DbhH5U3o/lkVP7nu9f8nnPbXv9Q+A/wv89ap6Yvd4QlUdP0Tc0iDvBb4BnFFVvwi8iV6lYFhLy/cD9JXt7i/Wk/u2H6l8fw/4Tl/ZfmJVnVBV564inmaZwFfnI8DvJDk5yUn02pH/eKU3VdWDwMeBy5I8LsmvAK86wlsOAEv7hN8MvDLJMV2bd/+frFcBr07yjCSPo9dmuPjdD9H7k/LdSU4BSHLq0dJGqIk4AbgPONSV5X+xZPug8nuk7X8GPDbJeUmOpXdd6TF9268CfjvJiUlOA/5137YvA/cluaS72HlMkmcmediFzo3KBL46bwduAm4B9gJf7dYN4yJ6Fzj/D/Bhev8Z/GyZfT8APKO7yv8/unUXAy8E7qXXFr+4nqr6FPAe4DPAt7rnfpd067+U5D7gT4GnI43mDcArgfvpVQ4+umT7ZcDurvy+bMD7/z29itC9Sd5QVT+hdxH0/cDd9Grk/b1S3kqv2eQ7wKfp/X6An1eOXkjvAv136P3F+X56v7UNL07oMB1J3gE8uaoG9kaRpJVYA18nSX4lyd9Iz1n0uhlePe24JLXLXijr5wR6zSZPAQ4ClwPXTDUiSU2zCUWSGmUTiiQ1al2bUE466aTavHnzen7lunnggQd4/OMfP+0wJmLWjm3Pnj0/qKqTV95z+tZS5mftvA+jxZhh9uNersyvawLfvHkzN91003p+5bpZWFhgfn5+2mFMxKwdW5LvrrzXbFhLmZ+18z6MFmOG2Y97uTJvE4okNcoELkmNMoFLUqNM4JLUKBO4JDXKBC5JjTKBS1KjTOCS1CgTuCQ1yrsRTsjmS/9kpPft23nemCOR1s/eu3/Cq1dZ9i3zo7MGLkmNMoFLUqNM4JLUKBO4JDXKBC5JjRo6gSc5Jsn/TnJdt/ykJNcnuaN7PnFyYUqSllpNDfxi4Pa+5UuBG6rqDOCGblmStE6GSuBJTgPOA97ft/pFwO7u9W7gxeMNTZJ0JMPWwN8DvBF4qG/dXFXtB+ieTxlzbJKkI1hxJGaSFwAHq2pPkvnVfkGS7cB2gLm5ORYWFlb7EU04dOjQw45tx5bDI33OLJ6fpccmaTYMM5T+ucDfT3Iu8FjgF5P8MXAgyaaq2p9kE3Bw0JurahewC2Dr1q01yxOHrsXSSVFXO5x40b7z51fcZ73N+oSv0tFqxSaUqvrtqjqtqjYDLwc+U1X/GLgWuLDb7ULgmolFKUl6hLX0A98JbEtyB7CtW5YkrZNV3Y2wqhaAhe71D4Gzxx+SJGkYjsSUluHgNc06E7i0PAevaaaZwKUBHLymFpjApcEcvKaZ55Rq0hKzMnitxQFUc8etfhDbLBxji+caTODSIDMxeK3FAVT/6cpruHzv6tLKLAxea/Fcg00o0iM4eE2tMIFLw3PwmmaKTSjSERzNg9c2j3A/nx1bJhCIlmUNXJIaZQKXpEaZwCWpUSZwSWqUCVySGmUCl6RGmcAlqVErJvAkj03y5SRfS3Jbkrd26y9LcneSm7vHuZMPV5K0aJiBPD8DnldVh5IcC3whyae6be+uqndOLjxJ0nJWTOBVVcChbvHY7lGTDEqStLKhhtInOQbYAzwNuKKqbkzym8BFSV4F3ATsqKofD3jvWG6tOeuW3o5ytbfUXDSL56fVW21KG91QCbyqHgTOTPJE4OokzwTeC7yNXm38bcDlwGsGvHcst9acdUtvR/nqEe4jAbNxa82lWr3VprTRraoXSlXdS+/GPudU1YGqerCqHgLeB5w1gfgkScsYphfKyV3NmyTHAc8HvtHd0H7RS4BbJxOiJGmQYZpQNgG7u3bwRwFXVdV1ST6c5Ex6TSj7gNdNLkxJ0lLD9EK5BXj2gPUXTCQiSdJQHIkpSY0ygUtSo0zgktQoE7gkNcoELkmNMoFLUqNM4JLUqKHuhSJJk7J5hPsG7dt53gQiaY81cElqlAlckhplApekRpnAJalRJnBJapQJXJIaNTPdCO1KJEmrYw1ckho1zJRqj03y5SRfS3Jbkrd265+U5Pokd3TPJ04+XEnSomFq4D8DnldVzwLOBM5J8hzgUuCGqjoDuKFbliStkxUTePUc6haP7R4FvAjY3a3fDbx4IhFKkgYa6iJmN6HxHuBpwBVVdWOSuaraD1BV+5Ocssx7twPbAebm5lhYWBj4HTu2HF518Mt91jQcOnToYfGMcjwwW8e0aOmxSZoNQyXwqnoQODPJE4Grkzxz2C+oql3ALoCtW7fW/Pz8wP1ePUovlPMHf9Y0LCws0H9soxwPzNYxLVp6bJJmw6p6oVTVvcACcA5wIMkmgO754NijkyQta5heKCd3NW+SHAc8H/gGcC1wYbfbhcA1kwpSWk/2vFIrhqmBbwI+m+QW4CvA9VV1HbAT2JbkDmBbtyxtBPa8UhNWbAOvqluAZw9Y/0Pg7EkEJU1TVRWwXM+r+W79bnrNiZesc3jSzzkSUxogyTFJbqZ3bef6qroReFjPK2BgzytpvczMvVCkWbKWnlfDdp1dybS7b47SFXbuuNG70K7GuM/LtM/1qEzg0hFU1b1JFujredWNe1i259WwXWdXMu3um6N0hd2x5TCX7518Whl3d9tpn+tR2YQiLWHPK7XCGrj0SJuA3d0I5EcBV1XVdUm+CFyV5LXAncBLpxmkZAKXlrDnlVphE4okNcoELkmNMoFLUqNsA58xzg0qaVjWwCWpUSZwSWqUCVySGmUCl6RGmcAlqVHDzMhzepLPJrm9m53k4m79ZUnuTnJz9zh38uFKkhYN043wMLCjqr6a5ARgT5Lru23vrqp3Ti48SdJyhpmRZz+weBP7+5PcDpw66cAkSUe2qjbwJJvp3eTnxm7VRUluSfJBJ3iVpPU19EjMJMcDHwNeX1X3JXkv8DZ6cwW+DbgceM2A9w01O8kos3jM0gwaS2f0WI9ZSRZN+jy0OluJtNENlcCTHEsveV9ZVR8HqKoDfdvfB1w36L3Dzk4yyuwf456VYy2WzugxyvGMatLnodXZSqSNbpheKAE+ANxeVe/qW7+pb7eXALeOPzxJ0nKGqYE/F7gA2NvN0g3wJuAVSc6k14SyD3jdRCKUJA00TC+ULwAZsOmT4w9HkjQsR2JKUqNM4JLUKBO4JDXKBC5JjTKBS1KjTOCS1CgTuCQ1ygQuSY0ygUtSo0zgktQoE7gkNcoELkmNMoFLUqNM4JLUKBO4JDXKBC5JjRpmSrXTk3w2ye1Jbktycbf+SUmuT3JH9+ys9JK0joapgR8GdlTVXwOeA/yrJM8ALgVuqKozgBu6ZUnSOlkxgVfV/qr6avf6fuB24FTgRcDubrfdwIsnFaQk6ZGGmdT455JsBp4N3AjMVdV+6CX5JKcs857twHaAubk5FhYWBn72ji2HVxMKwLKfdSR77/7Jqt+z5dQnrLjPoUOHHhbPKMczqlHOw2osPTZJs2HoBJ7keOBjwOur6r5k0DzHj1RVu4BdAFu3bq35+fmB+7360j8ZNpSf23f+4M86kkl9z8LCAv3HNsr3jGqU87AaS49N0mwYqhdKkmPpJe8rq+rj3eoDSTZ12zcBBycTorS+vHCvVgzTCyXAB4Dbq+pdfZuuBS7sXl8IXDP+8KSp8MK9mjBMDfy5wAXA85Lc3D3OBXYC25LcAWzrlqXmeeFerVixDbyqvgAs1+B99njD0Sg2j9Kuv/O8CUSy8Uzywv1Kpn3xeJQL8XPHrc8F/HGfl2mf61GtqheKdDSZ9IX7lUz74vEoF+J3bDnM5Xsnn1bGfeF+2ud6VA6llwbwwr1aYAKXlvDCvVphE4r0SIsX7vcmublb9yZ6F+qvSvJa4E7gpVOKTwJM4NIjeOFerbAJRZIaZQKXpEaZwCWpUSZwSWqUCVySGmUCl6RGmcAlqVEmcElqlAlckhrV9EjMUW6jKkkbhTVwSWrUMFOqfTDJwSS39q27LMndS2bokSSto2Fq4B8Czhmw/t1VdWb3+OR4w5IkrWTFBF5VnwN+tA6xSJJWYS0XMS9K8irgJnozeP940E7Dzg+4HvPojWqYufKWzqk3y8cDq5tTsNX5AqWNbtQE/l7gbUB1z5cDrxm047DzA44y/956GWb+vaVz6s3y8cDq5hRsdb5AaaMbqRdKVR2oqger6iHgfcBZ4w1LkrSSkRL44sSunZcAty63ryRpMlZsQknyEWAeOCnJXcBbgPkkZ9JrQtkHvG6CMUqSBlgxgVfVKwas/sAEYpEkrYIjMSWpUSZwSWpU0zezko42o9zAbd/O8yYQiWaBNXBJapQJXJIaZQKXpEaZwCWpUSZwSWqUCVySGmUCl6RGmcAlqVEmcElqlAlckhplApekRpnAJalRKybwJB9McjDJrX3rnpTk+iR3dM8nTjZMSdJSw9yN8EPAHwL/tW/dpcANVbUzyaXd8iXjD0/auFa6s+COLYfHMjn2KHcwVBtWrIFX1eeAHy1Z/SJgd/d6N/DiMcclSVrBqPcDn6uq/QBVtT/JKcvtmGQ7sB1gbm6OhYWFgfvt2HJ4xFAmb7mY+x06dOhh+83y8cBwx7Ro6bEdDZJ8EHgBcLCqntmtexLwUWAzvblgX1ZVP55WjNLEJ3Soql3ALoCtW7fW/Pz8wP3G8afipOw7f37FfRYWFug/tlk+HhjumBYtPbajxIew6VAzbtReKAeSbALong+OLyRp+mw6VAtGrYFfC1wI7OyerxlbRNLsGqrpcFzNhnPHzX5T3FLrFfO4m/RabSZcMYEn+QgwD5yU5C7gLfQS91VJXgvcCbx0kkFKLRlXs+GOLYe5fG9b09auV8yraQIcRqvNhCue6ap6xTKbzh5zLDNrmG5Y4+rypZl2IMmmrvZt06GmzpGY0vAWmw7BpkPNABO4NEDXdPhF4OlJ7uqaC3cC25LcAWzrlqWpaauBTVonNh2qBdbAJalRJnBJapQJXJIaZRu4pOaMcofFfTvPm0Ak02UNXJIaZQKXpEaZwCWpUSZwSWqUCVySGmUCl6RGmcAlqVEmcElqlAlckhq1ppGYSfYB9wMPAoeraus4gpIkrWwcQ+n/XlX9YAyfI0laBZtQJKlRa62BF/DpJAX8UTeZ68OMa4buWdfaDOKrmYG71Rm7pY1urQn8uVV1T5JTgOuTfKOqPte/w7hm6J51rc0gvppZvVudsVva6NbUhFJV93TPB4GrgbPGEZQkaWUjJ/Akj09ywuJr4NeBW8cVmCTpyNbyN/8ccHWSxc/5b1X1P8cSlSRpRSMn8Kr6NvCsMcYiSVoFuxFKUqNM4JLUKBO4JDXKBC5JjTKBS1Kj2hk6KElrsPkIo713bDm87GjwfTvPm1RIa2YNXJIaZQKXpEaZwCWpUbaBH6WO1B641GL74Cy3BUqTsprfyqL1+q1YA5ekRpnAJalRJnBJapQJXJIa5UVMTdwsXwSSJmG9yrw1cElq1JoSeJJzknwzybeSXDquoKRZZZnXLFnLnJjHAFcAvwk8A3hFkmeMKzBp1ljmNWvWUgM/C/hWVX27qv4c+O/Ai8YTljSTLPOaKamq0d6Y/EPgnKr6p93yBcDfrKqLluy3HdjeLT4d+Obo4c60k4AfTDuICZm1Y/ulqjp5vb90CmV+1s77MFqMGWY/7oFlfi29UDJg3SP+N6iqXcCuNXxPE5LcVFVbpx3HJGzkY1uldS3zLZ73FmOGduNeSxPKXcDpfcunAfesLRxpplnmNVPWksC/ApyR5C8n+QXg5cC14wlLmkmWec2UkZtQqupwkouA/wUcA3ywqm4bW2Tt2cjNRBv52IY2hTLf4nlvMWZoNO6RL2JKkqbLkZiS1CgTuCQ1ygS+Rht1aHWS05N8NsntSW5LcvG0YzraJPkPSb6R5JYkVyd54rRjOpIWfwutl3PbwNegG1r9Z8A2el3MvgK8oqq+PtXAxiDJJmBTVX01yQnAHuDFG+HYWpHk14HPdBdP3wFQVZdMOayBWv0ttF7OrYGvzYYdWl1V+6vqq93r+4HbgVOnG9XRpao+XVWHu8Uv0et3Pqua/C20Xs5N4GtzKvC9vuW7aOgff1hJNgPPBm6cbiRHtdcAn5p2EEfQ/G+hxXLuhA5rM9TQ6pYlOR74GPD6qrpv2vFsNEn+FHjygE1vrqprun3eDBwGrlzP2Fap6d9Cq+XcBL42G3podZJj6RXqK6vq49OOZyOqqucfaXuSC4EXAGfXbF+wava30HI59yLmGiR5NL0LN2cDd9O7cPPKjTAiNUmA3cCPqur1047naJTkHOBdwK9V1fenHc+RtPpbaL2cm8DXKMm5wHv4/0Orf2/KIY1Fkr8NfB7YCzzUrX5TVX1yelEdXZJ8C3gM8MNu1Zeq6p9PMaQjavG30Ho5N4FLUqPshSJJjTKBS1KjTOCS1CgTuCQ1ygQuSY0ygUtSo0zgktSo/wePBOxz6J3v9wAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "X_train.hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Let's visualize probabilistic predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Text(0.5, 1.0, 'Naive Bayes decision boundary')"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEHCAYAAABGNUbLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nOydd7zkVP3+35/MLVvZ3nfvForUhUWKKEvnK11FpQlSbChYflb0q1/AAmsBG3Yp0pEqKAtSRIog0pcibN+7vfdbJ5/fHyfJJJkkk8yduXf37jz7yk5yenJnnvPkOScnoqrUUEMNNdSwY8Hq6QbUUEMNNdTQ/aiRfw011FDDDoga+ddQQw017ICokX8NNdRQww6IGvnXUEMNNeyAqJF/DTXUUMMOiBr5lwER+ZaI/LGn27EtQERURHbpYhkzReTcFOk2i8iUrtQVU+4CETmm0uWW0Y7LROTmnm5HDTsG6nq6AT0BEVkA9AWmqOoWJ+yTwNmqekSp/Kp6RZXa9QTwHqATyAOvAhep6qxq1LetQFWPT5luQLXbUkMNOwp2ZOVfB3yxpxsRgYsdkhsGPAHc1LPNqWF7gIjkeroNNWxf2JHJ/8fAV0VkcFSkiPxcRJpFZKOIvCgi031x3u25iDwkIheH8r4qIqc6+7uLyCMislZE3haR09I0TlU7gduBPX3lHiQiz4rIehFZJiLXiEiDE/crEbkq1I4HRORLzv5YEblbRFaJyHwR+UKo3Becc10hIlfHtUtEvubUvVRELgjFNYrIT0RkkVPOb0Wkry/+AyLyilPPXBE5zgl/wrnzQkR2EZF/isgGEVktInf48nsWk4gMEpEbnfNZKCLfFhHLiTtPRJ522rLOOd9SdxcHisibTvrrRaSPr95Picgc5294v4iMdcInOW2q86X1n0tiO0RksnOum0TkEWB46HreKSLLnWvxpIjs5Yu7QUR+IyIPisgW4MvONfe35cMi8kqJ865hG4KITCx3y1yZqu5wG7AAOAa4B/i+E/ZJ4AlfmrMx6rsO+AqwHOjjxF0G3Ozsfxx4xpdvT2A90Aj0B5qB851y9gdWA3vFtOsJ4JPOfgPwA+BJX/y7MbZQHTAJeAv4khN3ELAUsJzj4cBWYBSmk38R+D+n3CnAPOD9TtpngXOc/QHAe2LadxywAtjbObdbAQV2ceJ/BtwPDAUGAg8AV/ratwE41mnPOGD3iPO+DfhfJ00f4FBf/f66bgT+4tQzCXgH+IQTdx7QAXwKyAGfda6NJHwfXgcmOG1/xve9OMr5m+3v/E1/6f5NnHoVqIv5Gya2w7nuVzvlHgZswvleOfEXOOfX6FzbV3xxNzjX832+a/UmcLwvzb3AV3r691bbMnFTHrDL2TLX1dMn20MXeAGG/Pd2fkAjCJF/RJ51wL7O/mUUyH8gsAWY6Bz/ALjO2T8deCpUzu+AS2PqeAJD2OuBdqdtRye06UvAvb7jt4Bjnf2LgQed/YOBRaG83wSud/afBC4Hhpe4btcBM3zHuznktwsgznXY2Rd/CDDfd94/TThvlzBvBH4PjI9I59aVA9qAPX1xn3H/fhjSneOL6+fkHZ3wfbjQd3wCMNfZvxb4kS9uAIbQJ5GO/CPbATRhxnb6++JvxUf+oTYOdvIOco5vAG4MpfkGcIuzP9T5Lo3p6d9bbUu/Od+360LbPzCdwlrgUeAW53OtE/4EDudk2XZk2wdVfR34K3BJOE5EviIibzm33OuBQYRuy50yNgF/A85wgs7A/HEAJgIHOzbNeqecj2F+/HH4gqoOxii5k4C7RGSq06bdROSvjhWwEbgi1KY/Ye5YcD7d8YKJwNhQO76FuSsA+ASGyP8rIv8RkZNi2jYWcyfjYqFvfwSG3F701fGQEw5GVc9NOG8XX8d0JM+LyBtha8nBcMwdjL/+hZi7CRfL3R1V3ersJg0Yh89rrLM/1l+Pqm4G1oTqSkJcO8YC69SZcOCrFzAevojMcOyxjZgOCoJ/b3+bAW4GThaRAcBpGOGxLGU7a9gGoKqfUNUL3A34ITAVuAqYoKrHqOrHVPUYzG/qx078D7PWtUOTv4NLMbfl3o9ZjL//DcwPaIhDxhswpBSF24AzReQQzCyifzjhzcA/VXWwbxugqp8t1ShVtVX1KWAO8D9O8G+A/wK7qupOGAL3t+lm4AMisi+wB3Cfrx3zQ+0YqKonOHXNVtUzgZGYL9FdItI/olnLMF84F02+/dVAC8bScusYpIUZOs3AzinOe7mqfkpVx2LU/K+leCrpaoz69vucTcCSUuUnIHxeS539pf56nOsyzKnLJe5+vrxJHbsfy4Ahoevsv55nAR/A3KEOwtxlQPDvHViSV1WXYKykDwHnUJss0BswA3hdVb8eEgqo6hZVvQSYRY38s0NV5wB3AF/wBQ/E3JKvAupE5P+AnRKKeRBDEN8F7lBV2wn/K7CbiJwjIvXOdqCI7JGmbU5nsifwhq9dG4HNIrI7xkP2n8ti4D+YH/3dqtriRD0PbBSRb4hIX0dV7i0iBzr1nC0iI5x2r3fy5COa9GfgPBHZU0T6YTpOt24b+APwUxEZ6ZQ7TkTe7yS5FjhfRI4WEcuJ2z3inD8qIuOdw3UYggu0RVXzTlt+ICIDncGuL2M6v3JxkYiMF5GhmE7VHWi+1Wn3fiLSiLnb+reqLlDVVZhO4Gznml5Aig7OOYeFwAvA5SLSICKHAif7kgzEWFtrMJ1L2unFN2LunvbBeP41bN84DDMGlYSngekl0hRhhyd/B9/FDGC6eBiYiRlEXAi0UnyL7UFV2zCDx8dgyMIN34RR7WdgFORyTA/dmNCWa8Q8zLQZQ+LfVtWZTtxXMYpwE4Zo74jI/yfMD99TfQ5ZngzsB8zHKOc/YhQlmIHcN5w6fw6coaqtEec5EzPw+DjmjuTxUJJvOOHPOVbFo8C7nLzPYwa+f4q5i/onQeXu4kDg305b7ge+qKrzI9J9HqO852G+/Ldi/NFycSvwd6e8ecD3nXY/BnwHuBuj1nemYPGBuWv8Goak9wL+laHOszDjMWsxHemNvrgbMd+9JZiB3OdSlnkv5rreG1aKNWyXaKRgQcZhDMmcEgl31kENvQQichhGAU/y3YHUsANBROYCn1HVR3u6LTV0DSLyJHAAcJiqvhARvz/wFPCSqmZS/zvkE769FSJSj3lw7Y814t8xISIfxlhl4buyGrZPfBfjQvxLRG7FjCcux4wtHYGZQGI56TKhpvx7CZxxhBcwS0Icp6obe7hJNXQzxCwPsifmmY2He7g52y2cu+evYp6rGQucr6o3lMizD3AN5nmWtZipzd/TChCsiHzEKW8wxQP+6zHTlO/MWm5N+fcSqOpbBMctatjBoCnWpaohFQZgHvq7keA4TCREZCfgEczzMgdixrluwIxJXRWfMx1U9S4ReQgzi2saZqxuA/AyZmxncznl1pR/DTXUUEMMnIkHFycpfxH5LGYixyh3hp2IfBszG298JdR/NVAV5T+4vlFHN/QrnbCGGmrY4fH21vWrVXVE6ZTxOHjwKN3Q0Z62vjcwM/hc/F5Vf9+F6g/BPFDX4gt7GPge5vmMqNlqqZBlzR5n+nBqVIX8Rzf04w97H1mNomuooYZehsOevzcTaUVh3aZN/GLArqnSHs/6VlU9oKt1+jAaWBwKW+GLK5v8MdOO4x4uDSPT1P2a519DDTVs91AUzXf0bBOCkJjwrLiBYvIXTKeyL2aJlmcwz9dkQo38a6ihhu0ekrOoH9K3dEIoLMpRObhTL/0Y6XyuoAtQ1U/ExTlTu/8PuAizPlcm1J7wraGGGnoMUpfW0ShRjpiy0mxVwLPAdPG9AwKzdPlSCgvyVRyq2qGq3wHexnkiPQtq5F9DDTVUHVUnYgGrTlJtJYsSGeCs5bQfhiObnOMmJ/5KEXnMl+VWzPLZNzhrZp2KWSn46m6a6fMvzANfmVAj/xpqqKEi6Ga1Haq8osr/AMwc+pcxq/Re7uy7T9GOwbeAn6q6Lykai3nQ8leY+f2xb8SrMPoSsdx8KdQ8/xpqqCETuoXMs0JA6ivTLlV9goQZNqp6XkTYLMwKnD2B32Ne+ZoJNfKvoYYaAqgWuVe101CFfLp5/tsjnOXdL6D4Cd9rVTXtiq8B1Mi/hhp2UGyXJB9Xp2VRPyDlbJ/tDCIyA/OOhvD4wTTgEyIyQ1W/lbXcmudfQw29FNWc9VJOuVZOIrdKQATEklTb9gQROQ3zvoj5mPeM74JZe2hX4NOYdz5c4qTLhJryr6GG7RjVVNnllJ2GzKvTZkFyvVLLXox5o+DBqrraFz4PmCci92MWobsY83a71KiRfw01bOOoto3SEyQvucxVlqgsXZu2Q+wL3BQifg+qukpE7sKs658JNfKvoYYeRnd45OXWUQ2Sr8r5qqKdvXLAt57SzyRvBhqyFlwj/xpqqDK6awC0pwge0pN8UV2VesI3J711wHcucLyIfDPq7XwiYgHHU1vbp4Yauh/dObulK3VtSyRvVXzgVRCrV3r+twA/AO4Rka+o6lw3QkR2xTxMtjeQebZPjfxrqCEB3T1tsdrknqaOOD++qyQvMfuVwvY2kyclrgaOA04GThKRxcAyzFPG4zEPoz1BGU8T18i/hh0WPfWkalfr3V5JPmrfqtDAr0g17iZ6HqraLiLHYN4p/ElgMjDBiZ4P/BH4sarms5ZdI/8aeiV6egmCbZ3g4/JmtWuiCD2R5B1rxguXyv2deqnyR1U7gRnADOd9wYOADaq6sSvl1si/hu0KPU3qlWpDlmmJVSd4qI6SjyB6r60SjOsyVNHOHn2ZS7fAIfwukb6LGvnXsE1gWyB1P7qb4NPUWW2ShzKJPkbNRxG95d0dAJYgUplBWslZ1PXvUzphDR5q5F9DVbCtkbmLSrar0uQO2Qk+sh1RHUHCYGsp66YSJA8gYmHlBBFfWTXbJxFZXuAeRqkXutfIv4ZU2FbJ3I9Kt7GcJ0a7jeAhk4oPH6cheX98gOhDlo1lCZ6A96l5P9G7JO+tsSPipK3Q30x67VTPLC9wDyPxgtTIfwfE9kDkYVSjzdUid6ggwUNFSN6/X8qXh+xq3uzjEb2/c/ETvXifFlhg5Sq3zkNvVP6YF7hPwbwrYAPwEua9wKOA/TGDv09hOolM2KHIf1NnO9+b8zzf2eUgBtZlfhq6x7E9knYpVOucyl3npRLknlROdxG8qcsrrCg+NcmbxLEk7+YpRfLuvtcxWDnEqixh91Ly/xHmNY1XAZerqrfUg4j0B76DWd3zM6r6dpaCdyjyv3v5HF7euIq7l8/hvPF7Vr2+3kjWaVDt8+7KAl5Z2lZtgocKk7wpMBCfRPL++iup5pOIXsQyq286+Sun/BV652yfGcDrqvr1cITTEVzivOjlh8AHsxRcHfKXbY/4NnW2c+/yOdwHfGz5HD4yftftUv33BLrzb7mtEHup8qpF8OHjSih5fxtKkbzZ961/75B0IDyC6K2c5aSzCkSfs4z/L86+ZfZNWitw99BViGVR169Xru1zGPC7EmmeBi7MWvAOo/zvXjKbk1U5DjhBlbuXzuG8puqr/20NPdEpV2Kp3azt7klyhwoQvCm4OL5KJO8ep1fzhuitOmeA1zL1kLMKaS3LSesQvXecc9S/VdE1+Hup7dOIeTF8EsY46TJhhyD/TZ3t3LNsDs87i+JdqjYHL53Nh8fusl2q/23lrqpib2Eq43y6Su6QjeChtIIPhyURfkVJ3mQIzLKBEt68lyc90Ys7QOsSfQlFHyB4K2fak8uZk7ecO4UKre8gvXdht5eA00TkGlV9IRwpIvsDpzvpMmGHIH9X9e/iHO9C96r/bYWsk1DpF2GUe85pX/JRNrFDanKHbAo+fBxF8P40RQRvIoNpYpS8Kd+ZSumUX0mSj/LnDaHnfOkjFH1A2UuB6HM501l4xJ9DLXOnUBFIr1X+3wVmAv8SkVuBfwDLgdHAEZiXuFhOukzo1eQvdVKk+l246v8jTb3L+6/m24y60olleXNTmnqqRe5RYV0heIhX8f50pUjeHBfqKGXZFNKkJHqfP+8NxIpTXl3OO79URO/FF4geAbXqDOG7YZaFVmplN1XI974BX1V9VETOxPj+5wAf90cD64ELVfWRrGX3+gHfu5pnc4gqOcwSeC5ywMGq3LV4NudP2quHWleM7n4VXZdf5J3xt5u2vkqRO1SY4E1FRen8C5QlrV+Txa4x+9GzbPxxUSQfyGuJR9oBf74cog9YN+IRuQTI3woSvVge0Xvh4h5XSPlbFrm+vXLAF1W9S0Qewszmcef2bwBeBu5V1c3llNurlT9A89YNvJGrY3pMfH7LhkzlbYvvCa1kR1vOu1Wz1F/y+lWB2KPCwsdZFDxkJ3iIGHh16grbNSa+oOS9uBDJm/B03nyibSPi+Pc5z8cnYPnEKHqzjnKsolfJFRG9Sg4Vy+yTI5+rDAWJcy16KxyCv9nZKoKqkL+IbDMkednU9/V0E0qimndJXXlRdkVJHRJf2ZeF3KPC0xB+2QRvEgXSBtfK8TKb4xiSN8fplLyJT+/NV5boExS9WGjOMhfHb924hG4Zcsf5tCXnxdti0dphM//1Fbzz8mJmv9LMojeXFf3dyoL4L3bvgojsA3wOmIR5reNVqjo/MVMK9HrlXwn0tIXVFQIPlJPxPFJ34KUGX7tI7GnD0tgzkJ7gTRHxKt7UGW3VuOWElby7n4rkvfpKDMI6KrzItokgd/GsmAhFn4LoEUml6m2pIy8Wq5dt5J2XFzD75WZmv7SIhW8uJd9h3jsybPxQpuw/JejHdgG9ccBXRN6DeVNXvS/4DBGZpqrNIjIZeAP4nKrekKXsqpF/TxNmT6JSZB1Zdje85g8om9BddCexh9OXInd/+lIK3tSdQcU7+SPtGshE8nFTKkuqeb8/H/LdA7NuRCpC9EoO27Jobc0z7/XlvPPKYma/3MzclxexfqVZer6hTz0TpzZx9CeOZtK0nZk4bWcGDh9MHuHKv1xC1yG9kvyB72MGdk/BdAKnAtcBlwAXqep8EXkDOAmzDlBq9FrlX00C7goq1SlmttXSDrSWSepJcWkJP61yh2wEb4qKVvCQTPJ+Enc//SRv0screfc4s2WThej9it5P9FGK3iXsnAWEfHrxxZewb/JisWLReofoFzPnlWaa31pKvtPMqhvRNJzd3vsuJu8/hUnTdmbMuyZg1dWTR1C1yKvQkrewtVKErZDvrFBZ2xSmAXer6t+c45tE5HzgKF+aWcDhWQuu3myfbZR8w+juO5Quj4Vk8eFTKKFSaikr2aciddO42HzhV/uVRe6mIBOWQsH7y0uyagr70Urey98VkocQufusnih/HimeR+8OxvoVva8TcBW9mYKZTPS2lWPr5nbmvLaUd15uZs4ri5n3yiI2rTVrjDX2b2TS1Ikc+5n/YdK0KUzcdwoDhg+iUy1UDcG3q2B3WtiArYJti/msEPmLCLk+vfJlLnXAklDYa8DBvmP3he6ZC648ZPu0fao+SN0VyyblLW2aW99KqXfITuzQNXIPJEmp3s1xAsE7ZYWtGrNvSNtNm5bkvTYlDcA67Sup5sMDseJaQFY6oo9V9BJp3+RtYfHcNbzzilH0c15exNLZK1FVAEbvPIq9j96HydOMqh+163jEynmq3lahpVOwNUj2ebWwbbAV8iqoLeSDj9+UD19H3cvwKrB3KGwD4J/XOgzI/JBDdWb70A1EWg1Ua3nhMr6UWfzLSqt3SEfqUfkTid1kiMwbSe6mwEJ4CYI3YeltmmD6BKvGy1sgea89XVDzgdk2Qsi2KU30hK0bx6P3iN4jfZ9PjxSp+vWrtzL71UXMfqWZua8sZt5rzbRubgOg/+B+TN5vEtNOOpBJ+06mab+d6btTf/JqoUhB1TsWjq0Y0ndUfRTZ2yqogm1DXqkYeulUzx8D94rI+1X1YScs70aKSB1wAsb6yYSqKf9qEWlXUA4JZ0E5A07dRfKxD1KmIHTIRurhMrIQu2lrNnL3h4cVvBsXtmlMfLSKd9sZZ9cEykzy5f0kHx6ETbJt3IFYsXCfjA3kcTsAKweIp+hxiD0wIOt0AC7Rt7flmT9rBe84RD/3lUWsXrwOgFydxbjdx3HwBw8y9s20nRk+aTRglVT1LtnnFdQ2nUDedkje1wnYNuRtM4Jp22arFETiv/9llvc54GsYS+UN4Euq+lRC+vcDl2GUehvwDPA1VX2nC83YCPwZ+KuIXIsZ9J3k1Pch4CJgPPDNrAVXifyFXJ2F2hXs1tNU2023feXWU0miT3wqPkYBpSF1yEbsprryyN2EB9vs/wEnEbx7HKngIZWKd9tbRPJuw0oo+SKid5+Edb35BDUfIHqP1F2iJ9m68Sl6REI+PZ59Y2OxbME63nllMXNfXczcV5tZ9GZhUHbo2CFM2m8Sh597FBP3m8z4vSdR36fRU/WqQrtH7KXJXrWg7P1kr2oUvkv2qpC31duvCERiv/flFSenAz/HzK9/2vmcKSJ7quqiiPSTgb8Av8AswzAA8yKWB8FbVqwcPI6R0wp8CvPiFnW2u4BO4HuqekvWgqvn+Vvb59SrrrY5bf5US5okfJlj7ZoY9RM5AJ+B1IuSZyF2U5gTlo3co+LDPrwblkrFQ6RV45WbhuTjLJuwmvfPtony5wNhMYo+JdGr5Ni4diuzX13CnFcN2c97tZktG1oAaOzXwKSpEznmU8cwad/JTNx/ZwaOGGII2zwhha1Ca158tk2EhUOxsi8o/GhlHyZ721ZsW51wKgNVxK7obJ8vAzeo6h+c48+LyHHAZ4lW2e/GzMX/pqrmAUTkSuBxERmuqqvLbMdlEPkO3zbMQPBjqlrWk3LV8/zrCoRQ7TuAancymdeeSqlAEm2ahFvY2JlUUlxvJkI3FQfjMxC7CQ8Stn8/E7lD0UBrYT9ewXtt9hO8kzas4r26yyF5P6G7M3J8qj5RzftIPzzrBp91AxJL9O1tncx7YzlzXlvi2Termtd65zR2tzFMO35/Jk2bzMR9JzNql3FILhfw6tvyyareJfTw4GzeloJf7xC4n/yN6ldnXz3Cd8lenXh1wioCS5AKzfYRkQYMmf8kFPV34L0x2V7ADLp+UkT+CPQDzgX+0wXiR1W/V27eUqia8g8Q5rY2+GtZ2Ko8OG8+N7y2gEXrl9M0eDTnTZ3ECVMmJxNvho4mqRxIIHFTUer6o1erLGpMMD6UIAux+9sRIPkQufvDshK8l8cry/LOKVbFO3WXTfKlLJsoNR/pzzsNCw3URip6p5Nw7wTcmTdQ8OntvLJk3hpmv7aEua8uYd6rzTT/d1nQvtl3ItPPPoyJ+05m/D6TaOjXD8Aj+w4VbEfVq0oR2SvFg7PGyimQvavm45S92xEUEX4eQE3nkFdUlbwz0lspYShI0Xc6AcNFxL82/u9V9ff+eMzajytC+VYAx0QVqKoLRORY4E7gV5hvwcvA8Wkb1d2omufvPWa+jcESwVbl/z3yHM8uaaSj80As7mXOuoP57jNv8+iC5/n5se8pPcAZhxJfwES1HxMXWWQGMjfxRZX54opJvWi/i8TuD48jdxNWrOC9/YQB10J5ZRB8QMmnJHn/bJuwbeMRu9sh5AKevavoTYdgpliqCLZlZuOsXrqRObOWekQ/f9ZiWreY2Td9d+rLpH0mcOxnjmXi1ElMmjbF2DdIwKsPq3oNWTiuDx/n1xfiKXQCeR/pa9DGiSL7fN4ch8neVlDbdtQ/lYFQ9JtIwGpVPSBFunDPJBFhJkJkNHAtcCNwGzAQs8b+n0XkKNWKnWnFUKWF3SC3rc328ZHjg+/M4dkljbR0PkwfduFelA8xk5bOOfxryf8wc/4CTtptZydbuvNImkmU2B9EqfaIDFFTZ9MSumlDCVIPtSXSvimH3APlBtW7SRtB8E6ewCydKC/eOecuk3zUAKyTTj31DpG2jY/oNRxWguhB2Li+lbmzmo1HP2sJc19tZuOqTQDk6nNM2HMc7zn1YCZOm8zEqZMZMWUUSF3Aq2/zDcqGVb1r8dgeiUfPxIki+ygbJ4rsDZkHlb266V2yN72A6RAw4WorGs2lZaGCUz1XY6ZTjg6Fj6T4bsDFRcAW/4vWReRsoBljFT1dTkNEZGI5+QBUdWFSfJWWd3AeWOkhlJrS+afXmmnp/BF1XMMp2BwHnIzNPfyals5L+dOsr/HBvXctUUkJSyeG8VOROESOG6Ql9KK0MWRf5MVTTOz++DTkbsJD5O5EhWfSuGVFEjwUq3g3TVdJPq2a9yt6v22Tkuhd68Z9mKplawfz3ljG3NeWMO+1Jcx7bTGrFq3xrsOoKSPZ67A9mDh1EhP3nczYPcaTa2j0qXroUMshdb+Sx7Ftii2crpB9wc93iV8L5XoK3h28VW/aZt7pAFD11L0CdqeNUvD6VU0ZlUKlBnxVtV1EXgRcG8fFscDdMdn64Zt/78A97kqvNA8iB3zTILHe6in/hor1whWDSz7Lt2wGJlHHufwAMxviClp4gKvJ8wTLNm9BcrmSD6olWvpx0y0jMhURdxKRQyyZh/NmJfVgWDyx++OLlLvZiRjUTbBoIKDgvfgIgneLL1fJpyL5CNsG7+EoKTwZ64VRpOjb220W/ncFc2e5RN/M0jkrPY/b9ekPPWs6TVMn0rT3JBoH9vdsm7xD+J2uFRNS9erOwAlbOFA0OOsndjctGWyccsnepLWLyd62vaeFK0b+IkhDRZd3uBqzjs7zmPn6F2JepP5bU51cCRykqkc76f8G/D8RuRS4FWP7XIFR/i92oR03AFOAwzBP9r6EufsYReHFLk9hOolMqNp6/nX1QeVfsVH9orqy5xkzcAAbtl7BKdiB9/oa9f9Dxu40kFxDof2lHh7JsiRCEpFHxSd1BIHb3AhS96eJ9PVDxG7igmlilbsvf5R698oIEbzXFr+Cd/NFqXi3vrRK3iNyPBIvGoD1dQhRaj4t0btxHZ3K4jmrmPvaEubPWsK8WUvMgKyzdPHAYQOYNLWJaSe8m6apk5i4z0QGjBiMreb8XFXfbluOci+oerMfYeGQguxLePZRyt4/I6ekb59g5cQRvtl3ynHsoYqhgrP+VPUOERkGfBvzkNfrwAk+K2UMsAe+oOUAACAASURBVLMv/eMichbwdcyDYS3Ac8BxqrqlC035EfAv4Crgcn9ZItIf+A5m7v9nVPXtLAVXbcDXqi8UrbaWfd+SqrqMf/Sz3j2R7z14Lz8I+Y1X0MJfuYeP7X8EOafzSiT+ONJPo+6LSD50p1CCzIvDo2yedKQeOI4g9kC9GcjdS+Pz8osI3s2XleDDSt5P8iElH6nm/bNtfLaNf9YN4gyg+hR93sYQ/etLmT9rKfNnLWbRm0vpaDOWQ9+d+jJx7wkc+8mjaNpnIk1TJzF43HDACqj6trzlEbtGDMx6Fo1vymWUheMuk+CSepKyz0r2cb69q+7tEmQPBAi/UKbzu3PLrAAyzvZJBVX9NfDrmLjzIsJuB26vaCNgBvC6fyzBV98W4BIROQT4IeY1j6lRvame9SU8/3Lv9yrQu6/etIXDIPK9voeKsmrzFnIN0ZcmrjOIHGwKBZUi/NJkHizU/12PI/VAWApid8OLfHfffiS5O/m9fFkUvHsyWUk+SsmL68X71X+B/MNqPtqfN2nzeWXxvDXMn7WE+a8vZd6sJSx6cyntLe2AWc2yaa/xHHHOYca62WcSwyaOAKkLEH2HR+I+Tz5G1QfIW8Uh7Gi/3iNzSLRxksg+zSBt2MpJq+5dYvcI3+V8VW+2T8UgVFT5b0M4DPPy9iQ8jbGlMqFqtk+ujKmeYRVQym7JCpfk5q9Zy1v9+vDezjwtnZ3YtmJZQt+6OhrqcgxYvaao88pC+qWsmzgyN2mj86UidV/ZScQeKLvojiNicNXfML8944THKnivPUEV77Uji5L3p/emUzoWTYyad0lfHQXvJ3pFnAemjOLvzCtL5qxm3hvLmP/6UhbMWszCN5cViL5fAxP2Gs/0M99L0z4TmbDPREZOGY1YuYB90wGoXazqXbfDHZiNI3v//Pq0yj48z94ledsj52Lf3sTbnm+f1crBOZ8owvfinTZ5nYCf8J12VBLb44oCKdCIGWtIwhgnXSZUh/wtIddYHwjTSq7gFKgr+63e7y74aEJ5MV+g2C9WPHlHlRe4NY0icnMQWW8coQfj4kndny6R2N2ywwOsXrgvXxoF78+XluBDSj5A8p5Pn0DyMbYNYgZjF89dxbxZS1nwxlIWvL6U5reW0t5qVsVt7NfAhD3HM/2MQ2iaOokJeztEnysmegJPxBareg1YN8UWjju/Hor9er+ytx21HmfjdJXsy1X34LeIgnaO+5t32+rWW+g8KswJVeKYHsZLwGkico2qvhCOFJH9gdOddJnQjbZPd079TNchJFmE0YO1aaydogBf2mKyNnVFE3/UXPs0pB7IE7o7SEXu/niv7gSC99JHELybvhySj1TyhTQFb96UHVbziNDS0smit1cw/41lLHxzGQteX8Lit5fR2W4GY/sMaKRprwkc9rH3MWHviUzYeyKjpoyCXJ1H5qpCBwK237pxvXkp8urTWDjQdbIvnpETXDbBb+VA8qycWLKHzHYOpCB8u5CnIhBBGjKL3+0B3wVmAv8SkVuBfwDLMc8gHAF8DPPr+m7Wgqtn+9TXl05YbaToA2IHiaJsnqyDthFEbtIlkHkoPEzqgfxJxO7EBywZN05C+V317u7H2TP+/B7pJqh4t45UJO+GCe6LRjySD6h5QrNtCgp/84ZW5r+5mAVvLDVE/8ZSls0tTK/sP7gfTXuN5+jzj2DC3k1M2GcSw5uGe9aNn+jVBtWgfROl6j1FT2EWTpxf71flYbKPs3HskKIPzrU3GeOmYHZV3QMBwk9r57h1lyL8ij/02gttH1V9VETOxPj+5wAf90cD64ELVfWRrGVXb7ZPY33levXU1Zb+4yfaRBH508zcSVzILe6p2yhVH+HhRxF6IEmUYnfT+VU7FIjdSSdhgnbzBxR9mNwhUsH78qYi+CglH2PZeCTvpLMVVi/dwII3l7HwreUsfGsZC99YwhpnXXqAIaMHMWGvCex//DTG79XE+L2aGDpuqBnoDSl6P9FHDcqWUvV+C6eUX1+IT6fsu4XsoXw7xw335TFxdqS6dwm/UGYFlX/lnvDdpqCqd4nIQ8CHMO/0HYSZ8/8ycK+qbi6n3Oop/4b6bl3PP/VgT5YvSIrB3EiiNgcp0khR2lSkDtHE7sb7VXsorFiVm91EgveXUUTwpg3lk7ypK0j0zhx7hPa2PEvmrTIk/6axbprfWuotVWyejB3BlGmTOfzs6UzYq4mxe05gp2E7OUTt1Ad0qhhFTtC+8Q/Kxqn6zGTv2TDFNk5Y2Qf9+mTfPmzldEXdAwHCL7JzcM83WuFHEj7EKvwowq+oQEwh/rZXOAR/k7NVBNVb2K2+PvXc/lJfgLJn/STki+0somygInsngsAj8geiIgZp4wkdIkndl6fIbw9/hpW7W0aY3P3pYxW8KSiW4N2yM5O8FUi7fvVmFr61nEVvr2DRW8to/u8yls5e4a1e2dCnnnG7j+XdJ+7P+D3HM37PCYzZfRx9+vUpIvoOu7D2TZjok1R9gMjBe5Aq7Nf7p10WKXtKk314+mVa3z4L2YOvLp+dU1D70XaOyRfv37vxJq58wq+4ONz21k7rMrKs7aOqC8W81nGce5yUvjrkb1lYjQ3BhiX8obvSX6dW/HEdQQTZFyVN8PJjyT+KyP3hEXcGEu4Mokg9EB8idij47m76BPWvgXr8x10heFN/Esm3tXeyZPYqmt9eQfPbyz2i37i6cPc6ZPQgxu8xjr2P2pvxe0xg/B7jGT55FJZl3lDlkjhAmzO10iX6gHp3yR+XqIMzcMJefZKqd7fCHUC0jZOV7F0rJzzfHpLJ3o0v184x0Vqk7iEF4We0dKIIv9KqX+p75YDvPNJTpAVMAd5y8iTaHFWzfSQ04JvY+lJTtMrx8pLuFtJ2BEUdSwSJR+UNk7k/PlL5F5M6xBG7sxOVN8LaiVTvTh3FcWGbppAnC8m7lk3eVlY0r6P5nZUsdoi++b/LWbFgNbYzv7G+sZ4xu45inyP3ZvweYxm3+3jG7TmefkMGGvL0Eb2Ns7Y8BVIvqPnC7BuHm0uq+rLJnmJl71o3frLX0H744aquqHtTSjo7xz3uKYXvL68ovJIdQO/0/G8gmz7ekDZP1Wwf6htKpwsg7ktQ5n1B0h1BGmsnwc4pVvvJPn4Rmfvj/YQeyB9xFxCKC5N3IYxQvBsXtIr8Ch4oi+BdUl67bAOLZ6+k+Z2VLHlnBYvfWc6S2Su9h6QARjQNY9zuY9n/hGmM22M84941jmGTRpLL5byyXKLvtAtr3lAm0Ud69eZEiyycsF/vkr2bN6zs05C9avBFJlnJHvfcItQ9BAk/anaOlyaFf++2w81jytoOCN+F//fRi6Cqn8iYfgWQKk/1PP+oObeV9uTipmmG2pI6PInwIdAPRRI5FJO5v9zIjiHa2okldudQi+4k4uwZNy6C4L2iCwRv6ilF8qtYMmclS2avZPE7K1g6ZwUtm1q9Uxo0YifGvmsM0898L+PeNZZxu49j9K5jaOjf1xBQiOg7PDXvtDVE9OEBWb994yl1l/BLqHqI8OvTkr1n3cSTfdSsnHLUvZum1JO1heNuJnzwLmg5hF/phz6FCMFWQyKqqPwd26cKnXx0nfFRGtdJFNk84QTpvP7OvM2Xr/odN//1BVrbW+nT2IezTzqAq7/6Werq6oKEHshbTOrJ6aKUe7Acbwm9BAXvluUneCiQcmfeZuWi9Sydu4qlc1ayePZKls5dybK5K2nd3Oad98BhAxiz62gO/uCBjN1tNGN2G8fYd431LJsoNW/UdzLR+y0dO0T0YVXvEa5P1YfJ3lX5UWQf59m7ZO/37eNn5JiKU5M9RBK+/+UmJf17SE34JqmWTfim6niVHx7PiyP9MOFX/A6gd9o+VUOVyN9Cw4Mv1bjVg9K3emn9/SQrJ3Ts70w683l2PeHTrFizE+bVnVNpaX2NP9x1KQ88cRFvP/xH6urqkgnd3x4JhScReyGaeHJ3EklhgNRNt2VjG8vmr2bZvFUsnbuKZXNXsWzeKlYsWOU9AQswaOROjNllNId8+GDG7jqa0buOZcyuY+g/bKcCofiIvtN26klB9EmKPpLoTWWJqr6gyCn48y5RO/mrRfZAlwZrzXXL6N9716RyCj9YdhyZl6fyq2L7QOmxw+0Q29+bvESgroee8E3oDCLvAIrSB4+D9k5x+FdmXO0Q/38A92USuwInsnz1AXz1x7/h6u98NdR5RCt/9avwGHL3548ieLccl+Db2zpYvmANKxasZfmC1SxfYMh++bxVgdk1Vs5ixMThjNl5JHsftTdjdh7F6F1GMXqXsfTdqR82UqTmXZKH7Io+DdlHefWpyN7J5396Ng3ZV8LKAWIJP87OMdcvhvCdAqup8M1+12yd4nTFRFzcKVSwExCBhqzjjNsFssz2CaP7Z/uoCHZdD027irlM6Yg/WIAWqf9ir/7m+/8NXEOB+F30AS7n5vsu5seXNSSSOhBJ7IX0ofEAJJC+ZWsHKxeuZuWidSxfuIaVi9awYsEaVixYzdql6wM/uoHDBjB6ykimHr03IyePZMzOoxg5ZTQjJo4gV18XSfIdtq99JUgebz8b0Repev++R8LxRF+Osi+H7M35xXv3QBHhR83O8cqJIvwQaacifDeuBOEHy40nfH869zwjw8tQ+VV7+DPNGOD2hxson/wTUbV3+NoZlb9EfUnKHcCJ9fiLyyuuI9regQIB+stqbWsBpsY0ZB9a21qwcw0xpO6rz0fs7qebtrPTZt3yjaxasp6Vi9axavFaVi5ay6pFa1jZvNZ74beLAUP6M3LScHY5aGdGThrBqEkjGTVlJCMmjqTvoP6G4E0lHsl7g6AxJO+GadFngeSdIitC9NC9ZA/xhJ9W3Xvnn+Dfu3m7YumYMgpx/nZ1h61TnK58lV8xC0ikt67tk2m2TxZUSflbAeUvKWb5dOUrEDugS4iwvQZFdQLxM3v8d08a6ByEPo39aGl9DWP1hDGLPo39yNc1BPKGVX5baydrlq5jzbKNrF66njWL17F66XpWL17H6sXrWLtsvTcvHkAsYejYIYxoGsbUo/di+IThjJw4nJGTRjC8aST9BvUzhB5B8p1aIG0w+0CRmneyRRK9F0c6ojfl0z1k748vQfhhsgd/umJ1b+IrSPhORJzCd8O6Svhmf9tS+dWZ7tkrlX/VUDXP3875/bdqD8RkJH8oUvzFx/4yo2f5KMLpHzmCG26+FDiRoPXTClzKR085koVz17FuxUbWLtvAuhUbWbN8A+uWb2Dt0vWsXbaezeu2BuoWEQaP2olh44ay8wGTOWj8MIaNH8rw8cMYPmEYQ8YOI9dg7qxcm8ZtjyJ0hAgeXNItEDdhxR4i+UAYBZL3witI9Hjll6fsIQvZmwKT1D0UCD/OznHTxC2n4LapuwnfH1cNlW/Slkf6VRvoddELp3pud8s7KEJnLtn2SXM3EFl26t49bpZPiOTDA7xFA75BsldVWja1sXHtFjau3cIHDz2d5/4yh7ZNx9HAITTSSAPLaWAZfeRdLL2znUvuvCpQZv/B/RgyZjBDxgxm8rTJDBk7mKFjhzJ0zBCGjhvC4NFDqWuo89oSVvBgFikrqPDSJO8ex3nzXjyliR43jUYQvf84huz9fJaW7N06S1k5gbZ5hF9a3UNpwg8P2Hr7ZRA+UPSkrSlv21D5kcfVIP1KPvtT7c6lZ5BlwLfnl3dQEfJWfaSP76EKnXTSGEGQIJX21g5aNnfQsrmVlk2tbNnYwtaNrWzZ0MLWjS1s3tjK5nVb2LJ+K1vWb2XTuq1sXruFTeu2kO/IB8oez0QQ6OAt2mmjw+pkaNMwDj1mOkPHDGanEYMYMnowg0cNYvDoQdT36RNok5/c3fAOLZBvcfvDVg1Ekbx/P0rNm2OCHUAKonfLiVP1EG3hmHA3TzTZF9ocTfbu3y9O3UMx4Qc7hmT/3hxXjvAL7fURvnfdukb4/rhIlQ/eOcURflFcCcKPShNVZlw6X4b4uHJQ1qoC2wVuIBtb9uzyDhtWbebG7z9IfWM99Q056urrqGvIkavLUVdnYdXlsHJCLmdh5czTqJZIsLk+daa2Yudt8nnbfHbadHba5Ds66WzP09HeSWd7Jx1tZmtv7aC9tYOO1g7aWjto39pO65Y22ra20bKljbYt7QEPPQq5+hz9B/ej/6B+9B/cj+FNw5m070QGDhvAgKFm22nYQPoP6c9zLz3NrX++j2UrljBmzDgu+ORZnPyBU51FyIqJvTNArMXk7oYHFTyECd5NF9yX4nAKpO7FhcndIXz32E/sQBHRu+mgQPReWQ6p+4ke4pW9P19asvfa6XUaPl8+hZ3jlouqR7pJUzILbewa4XtxKQnfpNl+VH5cWl+m+LguQXrlQ15ZB3y1p5d32LqplX/e/jyd7Z1FKrkaEBHqGuqob6wzHU6fehr7Op/9GhkwbADDJgyjsX8jjf0a6TOgD30HOJ8D+9BnQF/67tSXfjv1o9+gvvQf1I+Gvg2ICBoz2Gsj2LbNRZ/+HM88vZCWrZcAU1mz+jX+95tX8uCDj/PL3/4GEfP6yryt/O3+u/jTtbewfNliRo8ez8c/cTbHn/JRLMsKEDz4O4XgvhtX3HmEiDRCzUfuK8V2TfjYR8RQ2q93w8JkDz6C9ZO9L00UwfvJH9LZOUDJAdtAexIUvpuvQPChpRWcnarYOpBuxo57AQim9ZcdDo88rtbMne5YbrkXev7VRFXIf/SUkVzzt58AYNs2+Y48+Y48nc6nbdvYnTb5fB47b36M4S+/OHcCYllYlnmxuJWzyNXlyNU5n/U5cvV11BW9L5gAaQfDg18QO3x35Hx/nfW4gnZLyHp54L57HOJ/Cv8DXi1bT+SZJw/lgb/cxwkfOA3btvnihZ/muWcWBDqJy741g4dmPsrVv/6D10n4Cd6ts4j88RFvSM0X8kTsu0TvHUcTvZsWDNG7x0kWjntciuxNOi0i+2AdweOkwVqTrjyF76arOOEHrnG8ynfrC+TvgrXjT+svOxweebzdKf0o9D7lnwYichhwpKpeniVf9QZ81SlaQBqgrsGprFKds+/71hnz3Yua6RMZpsmDvlHpbITr/3iLQ+bFD3i1tHyTG/74C95/yhn87S93OsQf6iRaTuS5p6bzt/vv4fhTTids1Zg6g21KInk3vUeyvoLSED2UUPVeXh+5O/+lJftCGzVUT7K6N2XH+/dufBbCN6UXP3jl1l/K1im0oZjwTT2VtXag9Kwdf/mRcWWQfrjMUmmdDPFxDiq+GkMvfJmLCxHpDwwBipUufBD4koj8CfcnWWKmD1SZ/P3Dkb7IqiGOtGPTa/JdQLCpxQOuy5ctJukBr+XLFpO3LW667qaETuISbrnulxx70hmBcwiTvAkrHGuIA5KI3gsL59HiDiOO7CGo7L3jON/eSZfk3QfzOGkqQPimbdHTMk3pQR/fK9cj+PJVvqmr69aO2U8m/aL0FbZ3wmWWSuvLFB9HFUgfzFInvXDAV0SmADcDB5MsnRUzM8hFydugKpG/WYu9FKyMPUGRRZNQfzTSqP74eA2FjxozgbVr4h/wGjVmAp1qsWJZM2k6iThf3v8ZVu+xYVrEE7EDs276sF/vnXOInMshe/+xv44oO6dQdnmE77Y7TPhu3jjCLw4LknU5Kt+t0582LemnsXb85UfGbUOkD9UhfgPprQ95/RQ4CHgSWITjSIewn7PdkKXg6jzkpUJerZJUXfG7vqQ4jbKAktNETa/05zvzvHO58jszaGkpfsCrT98ZnHH+xeRti5GjmxI7iZGjJ9BpO2v8RJC8v/44ovfnNWHxqt7Lk0D2bpqo6ZcmToPpAmQd3wF0B+G76f2EHy6/lMovtN2JK1Pl+9Mm+flmv0b6ZUPorQO+7wVuV9Wz4xKIyP8B+6nqBVkKrp7yd8hMJImSK1hnBLkX2lNa8Zt00fsmfXF5x5x0Bo8+9DDPPzOd1pZLgH2AWfTpO4OD3rcLRx1/BnlbOO3c8/jRpTNojekkPvLxL3grWLptiyX/EkTvfoYFaLlkX0hfyBD27v11RnUAXSF8L7+vMWl8fC/MR9ZRKt+7NhEq3yTtedIvSm8nxEUSevm+flJ6J1N8nIPuWmm57LXAtm0MAd6pRsFVW9jNs320SkvS4RVfVnz4+xw1z750Wovv//xPPPrgHdxxwzWsXN7MyNETOO3cL3D0CadjqwUKRx5/Jo899BAvPlvcSbz7kF05/DjTSUQRfZR1429TrKp3Tt5PwN71yEj24baksXPALdtH4BUifK+9FVb5/s9qDOKG47vi6YfjIo97u9oPo3faPkK01dNlVE35x3yfqo40M3UKYcn5kzqIAqma1TD9qjqvSt4uPHOi5Lj8pzfy2Mw7uPvGQifx4XO+yBHHnYFgOe96TSb7IsL30ngNLEn2EFLMMYQfp+79YeH2RCl8U1/xtEwgta1T1OYUKj9wXbqB9IP1ljGQayKKygrXEY6LPK4m6ZuMyfH0APFD9A96+8f5wEsl0twHLMhacJU8/4Ltk4Qsd2lZ/q5Js35KEX44TREZ+/fzNv/3pXN58dm5tLZ8A5jKujWvcfVlM/jnww9x6dV/Mg9wqQA5jjr+LI447qxAuYp56YgdcacRRfZ+ovfHlSJ7N7iYsDVW3bvp0il8U0NWwi/UWyi4lMo3YdEzdrzzjrJ2fDvbA+nH1RN37G9v6XRlEH8VSF9TlJkKImh9D71AqopQ1RtTpHlNRNpKpQujW2b7RE75dBNWpL7kXiSS8CPCAl/DUHxwjr/BY3+7wyH+J/HP329tOZEXn53OYw/ewZHHn1VcT8i68e8nqXp/fIirgmrdDsb5CTxO3ZvPYjWfVeEX6iuQa5gQsqp89zPK2vHq8C5YdUk/WH886QfrKk36/vT+OqLiIo+7SPpx6X0Z4+Mc9BjxA71lto+InAu8qKqvp0ibw8zx/xxwJBmfcqva1bLtwpa3xdtsrcwWKNMmcut0tryGNttwgX/L24L6tryarSOv/P2B27jw9GM49fB3ceHpx/DoA7eS71Tuuul6R/EXz99vbbmEe26+LrLNke112+Fsbrg6W97dnHiXM71rnDfp/GUVrr9R07atvjA3v/t0rBbedRsR5l4zU49iSF/J5921l9TLazs2Sz4fXERNbSfM6RzUKbxQjnPOvvTqlkfhWL3zV+fcCl849y7DvQbujhvuxdtaWH7BbY/TJvcduFFqX31lQcjX96n9LMTvT+/WExsXOjbpu672t2/id8oUK9WWFiLyORGZLyKtIvKiiEwvkV5E5Esi8l8RaRORZSIyI+NpXAecXKKesSJyGbAQuBM4Ang8Yz1VUv4+VVkcmZAxLODLvDOIm/kT+3WLUfm2bXP5lz/OS8/NpbXla8Ai1q+9mRnf+iK/vPI7qN1J0vz9VcsXFcjPV0fetnny4dv4y63XsXrlIoaPbOKUMy9g+v+ciThfzihlHwiPUPfuZ9RgbTBNtMJ3j4vuLkIDtyas2GNPpfJ9lYanaXrlRvj54bZWS+m77QqkqZLFU5SnSmo/quxS6Z1M8XE+bAvED1R0qqeInA78HKOqn3Y+Z4rInqq6KCbbVcBJwNeAWcAgYEzGqjcC745p09FOO07GPOm7BvgJ8HtVnZOxnmrO9sn2h7AgkuzL+pokfZ8j2hX+Xbi/h8cfvM0h/ieATwJzgG8DU9my6TXgIiB+/v7wUU2Bso0at/nh18/hlf/Moc0ZJ9iw9jV+feUMnnlsJl+98kYsy8pE9iZdacJ3j6PC3M8oH98tL4qEo7x8kz+a9KP8fDd9GtL3zq2LpB9Ik8HiMenKmMXjP5FQen89sfE14k+GSCZVnwJfBm5Q1T84x58XkeOAzwLfLK5e3gV8Hpiqqm/5ol7OWO9DwOki8iDwI+AV4DzgQmA3zLfzSeB3wN2q2pGxfA9V8/zVTib/8Pz/tF+JpPn8gfISO4A0+YV7bnZtnXsxxB/09k0nfSlRb/Fq7DuDk8/8Eo5D4tX75EO3OcQfLKut9URefX46zzxyO+871hkUriDhh8OLy8JXZ7HKL85fTPr+AVx//aVI3/1MRfq+Sivu6/vTpFT7gfOk68QfadfUiD9N4el+2AbDReQF3/HvVfX37oGINGDU909C+f6OeegqCh/ALK9wnIj8DaNn/wl8TVVXpm0YRlH2xaj74zDfNAtYB/wM+J2qvp2hvFhUx/NXn0ft28IeezlbsIz4reALF29AyXEEVVi9YhHG1rkWiPL2zwWGI3IAcDfmWYw7qavfk7rcKq776aV85eNH84+Zt9DRaWMr3H/7dY7iLx4naGu9hAduuzbgfdtOmwvee8G/Rwsef5SH788XDlcN+/guURT88rCXb/Lbnr0T9vO9fG7n4KT31tlx/XxfHv+qmlGeftC/r4yvb9ocofaLOrsIbx9ivX3/w1ppp3CWms3TFX9/hyF+MGv71NWn2oDVqnqAb/t9qLjhGFtlRSh8BTA6pglTgInAGRilfg6wO/CASPpbElVdq6ofxKzlcyPQjvlm9sE88DUsbVmlUDXlHxmeumMuH3bCnUFU/Ulhw0c1sX7ta5hxlQXAUZjlNZow70s4E/gtffodztgJv2LVioV0dnTQ0TacLZv/F5jKhnWv8dsZM3j28Zl89YobWeN1KFHYhzUrFxUp+M58nmcfu42H7ryeNasWMXREE8d95Hzec1RhjCBK5ccp/yRrx4RH+/mQ3t4x5RXUfpTSNzWnUPshpe9vayW9/WA7/HV13eYpypPR5gm3PTld/A+tNxK/gVRjeYfwxZKIMBcW0Aico6rvAIjIOcDbwIHAvzNVrPoCcJ6I/D/g4xjb5+PAuSLyBvB74EZV3ZCl3HCDq4Jo1V7ZLaqOsMr3z6iJCg+EucoVs5185gU09LkS6ABuwdyRzXQ+f4bp3GcxdsJe/Oj6xzj/i5eTz4+mo+NfwIcx1tCHaWt9ilefpuj5yAAAIABJREFUf4enH7mdYSObMOMEUZjF0JFNAYXfmc/z8++cw3VX/ZL571zExnUzWfDORVx/9S/4xaUfN+9EiFH5RdfCU/Tu7zxe6Rer6GKLxyV+v9J3LZ6w2nfzeHnJoPYJdgap1L57er68pQZ1ewPxJyFr+ihsm8Tv1IWVakuB1ZinasMqfyTFdwMulgGdLvE7mA10YtRiWVDVdar6c1XdAzOd8zZgF8xg9FIRuUFE4qyoRFSN/KPslrgpmVFpu5ovLk2Y4N0t74uznfzTjz2TCRMbgJ2AIKHDU8A71NV/g5PO+AQKPFDC0vnr7ddywmkX0NhnBtAaStNKY58ZHP/RC/AevFL416O38fqLc2lrfZJwh/LGi7N57h93FBFnOaQf9VBWKdJ3id/7g0Nqi8cty5c1qPaha95+DPHHTeH0zs+ry94uib/XPbWbFoKxflJspaCq7cCLwLGhqGMxRBCFZ4A6EdnZFzYF466UXFs/DVT1SVX9GDAeM6NoCUaBPi0is7KW163kn5Xc03YOUWQXJnd3CxC8j+jDCVVBxEKlEbicKEKHS2hotHnfMWeAksrSOeToM9nngF1p7DOdwjjB3TT2mc7eB+zKe440Zblte+iu62lrje9Q/n7XtRUh/bDah2iLB4p9/bDad2vy58lk8/gqL3dQN5Am48BuABUk/nJQCeJPUUl5+UoV250vV8nADylxNcZ2+aSI7CEiPwfGAr8FEJErReQxX/pHMcswXCci00RkGmbO/r+BF6ggVHWNql6lqrthOqQ7MTOBMiGV5y8iuwG/AUap6t4iMhU4RVW/H924wG+wW5H4/Y+JC38htra08o0LDmDx/KWYu79PAV8ArsB0tG6fuQ/19XUgFrbCsJFNbFgXP/Vz6MgmRCy+9L2bePax23nwzl+xZuUiho1o4viPfpH3HHWGV5bbprUrkzuUtauafQQavAbB356G4rL5+oG8EWrfX18S8acl/ei47iH+shV/CMWDtckdQ1c6im3N5+9uuAO+FStP9Q4RGYaZ2z0GeB04QQtvyBoD7OxLb4vIScAvMNMCW4BHgC9rFXtBVX0ceFxERmbNm3bA9w+Y24zfORW+JiK3ApHkDyVIOAZW6I6sXCEDpCb6cF2tra187MhRmEvTiLFnGoCtwMUYz/9WTAcwi2GjmrzfzwmnXcAffjyDttaIqZ99ZnDCR79kDkU45JgzOeToM4vq1xCBDR05gY3r4zuUISMmBPKlGcz1p+8O4g+rff/5+ctOjCtF/BF5o1Rz1PcikvjLQQLJ7kg+v6m3J3qLyg74quqvgV/HxJ0XEbYM+GhFG5ESmm06KZDe9umnqs+HwjqTW5OwxSBsx5RVtq+OpFu+qLoU+Oq50zCzvMYBl2K0xKXOcR3wAIb8DaGfeNonvLzvOep0xjb1AaZRx1TM3Z6xdPY5YDcOPur0Ir+5M2/z9CO38L+fOpxPHzuAb3/6cJ559Bby+Tyq8D+nXkBD45XA9cB06ugHTAeup6HxSo790AXbNPFHIZaPMlo9wUKJjUua1RNMVzydM1xmT9o98WnLrKtHCLpakIov79DbkfZKrHYGMhRARD6CGd2ORaK/loa8M3Yepby9uI4lXOyyRfMw4yn/oYFbEDZQz20YIh/npPw2jX2mM/XA3Tjk6DPMWIJt87PvnMuSha3UMRCLWeQ4CpGLGdtUz8WXXV94ctdH/L+49Byuv/qXLJs7EjvfytK5o/nTT3/BNZefi23bHHDYafTttwG4mhyDsWjBYihwFX36beTd0z8a7MC2MeKvtMfvR5TdU2h/hN0TamNcuWmIPwnVsnt6QvWXg55R/aBIqq0Gg7S2z0WYeaW7i8gSYD4Q+1oxP0p9/9JOzc36PU4SQ5FRXmB/4DJgBTme4R7gVJ6igxWYgd9PARv41Nd/wHuPPsMQOvDMI7cx64U5tLc9QB924V7gQ9i06sssXXQyzz/xZ9579FmBtj33+G288eJc2lrvd/IoH2Imra1zeOPFk/n3P+5AVWltGQLcTz1NKFDPTNpYxNbNx/OtC/Zly5qF2HVDqKuzGTFmD4754PkcePgZ3iqH2wvxp0El7R4/sto9SV5/b8K27vV7EGqqPiNSXS1VnaeqxwAjgN1V9VBVXZCUJ+1vo5Ij9KUso8gbh6LAPDCVBj7GBzHPV38AqOcczBu48oDNocee5azVb9r34J3X0db6Deq4hlOwOQ44GZscv6at9RJm/vk6r40u3Jk8cXn+fve1/P0eU26On1BHB/cB9XRgcTWdHd9m06oNYLej7YfSunU8zXPh5mt+ye+uOM88jbsdkVSc6k/OFB8Vpfqj6usyuuD1R+fZfmf49CQ05b8aDBKVv4h8OSYcAFW9Oim//3sZHsytFLrED5EROeBRcjzDd52Q7wF/4Sk6eAxooK6uuM9cs3IRMIk6zuUHtABwBS08wNXkeYI1q4ILAaq6M3ni86xd1eyknkQ9Z3IqpjP6EPBnfkYbD1PHSu+OIc8c4GQ62i7mrVd+xYtP3sH+h57GC0/eweP338D61c0MHj6BI08+l/0PPQ3LsrpN9ZcDtW1m/edunvjbr9m0bhlgMXDwUA4/8fPsuf+piFX4O6S2Tkq1p8KWTw3dA0VQq/e9zKWaKKX8BzrbAZjV7MY524XAnlkqKjmImxFpy8tC/LaC5LbQwFf4IOYxOpxPo/6/AnRy5EmnmiJ8ZQwb2UQdV3AKdiCfUfI/ZNiIpqL2Di2RZ+iICQwZMYEcl1FHB99x0vwfRv3X8xlOQQJ3DHAJcCPtrZfw6H3X8YcZ53Hbr39F89yL2bRhJs1zL+b23/yK6350AXY33tPHWj5x6W2bP//+c9x/82/ZsOZybPs5bPu3bFjbjwduvoK7r70406DotopqdRY7XidUG/DNisQroaqXq+rlmIWO9lfVr6jqVzAr3o0vp8JKdADVdDIGDppEjhZP9bv4HlBHC9DJuV/8RVG+o04+nTru8xS8iytooYF7OOrk04vyHHHiaYl5Dj/xDA4/4TTq+SunEuyMTgRyzOYHjlS9ghbquRqzttQiYB9WLXuL/74yj/a24NPB7W1P8d9X5/LS03/OdnG6Ea+/cA9z35qH2s8TfLL6WVTHMe+tV3jzpft6tpE1bFPozQO+InKIiPxBRF4QkdnO5x9E5D3llpm2G2zCrC7noh2YlLkyqYz9U5FyIvJbAh0bFjEdY/7M92054FCgT66Durpit2zDikUcJjH5BNavaC5qc6k8G1YuYnXzO+TAU/0uRmHuRqLuGMyfaxb5fB3tbdFPB7e3XcI/HrihxEWqPFzLsNTf79+P3Uq+89vEPVmdzw/i+Sduq0YTewWk8oucbfPorcrfeRvYM8AFwDTMshHTnON/icgV5ZSbdrbPTcDzInIvxjD5EGa50VSolt/vlpt0J+BWHZkkYo2+BmyeRdg3prxGnyksUrAzVix4gzk7DeWg9jba21tRO49YORoa+lDf0MjO84Ov5BSBlQvfZK6bp60V1TwiORoa+1BX38iUBW+wdOFbvJdCBwHmLQI3U7xMoBkvuIc8v6ShzwwsS+hIeDp4/epmLCs4o0MsQW3FEscGEzEWgnOylgi2KmIZ39+9hG4+7xNB8R0LqQfuATauW0zSk82wno3rCwsaeu3sKiw8399fpoiVyvfP2o6KtXtHh0DG90dtFxCR0zAP2M7HPFT7BLAc84TxkZgnkC8RkVdUNdOtfCryV9UfiMhMzNNFAOerauIbaqpF+GnqiuoMYjuBUAdw7rev43c//DltrU9hVOZ66vgIndxMY5+T+fTXvxTI4pLa535wrynfV1bUkgIuqQJ85nv3FHnh/jJU4Y+XfZhZm9YG3iDR2rqFA9tbAh0CmA7ifShP5b7HHvsdwdpVA2ieG/908ODhEyLCex6WwE5DxjtvS4tuOwymvn4r1/74Q2xY/gqDRu/HwUeezZ7TPugrR4pm/LidUXchXF+5ZF9OPlWbuW/NZNYL97Nl0zL6DxzDPgecws57HG+WAu9FM36yzArcznAxsAo4WFVX+8LnAfNE5H7M0hMXA5UnfxFpwixzeq8/TOPfZRlTTpbUpRH3x/Z3BuHfeWQn4As89NgzefYfD/La89Npa72EOh7D4nEac/sz9aAjee/RZ3hZ/B2A2x7/vuUc2KqBDP4OwO08giqzUNYnL7vbKa/Q4msv+zCz3vgXB3W00t7Whm3nvROwrDomTx7LZ751A8//83Zu/dUM2iOWm2honMFRp1xs2tPN6t8tz7sOTgP81+Dgo8/igZu+R76zuO0wA1jA5o2T2LJ2LBb/Zs2SPA/c/F0euvn/MXjs/hx8xMdMR+D+QSLu8iqmurPe1pQqzrJSD2bHdWaqNo/c978sWbiCzo5vAlNp2foaT/39Cua9/QzHfvD726n7HQMR1KrSW2l7FvsCN4WI34OqrhKRu4CPZS047dX6G4WfTl9gMuYlBXvFZegOyzGuDv/vMO6uIPJd8QJWzuJrV97EM4/czn23/IyVs5/hHpTTrbVceMkvyNW5D04VdyR+DvDvewrUlyGuAzDt10AHEI77xGV3B+4O/Oflf8L3oMPP4KVnZvLfV6bT3noJxi6ZRUPjDHbfbxf2P/S0AhGn7QCqDLeeqQeeypsvPsLs1w9E7cu8tps732aQ8eQ7Z9KHnZ2prv9BsFBs1i99k7/eeg1PzvwVn/76TKQ+OAUwijAtsQoPeqWwfixLYh/02hasnzlvznSI/2n8rwvt7DiRJQvfx9y3HmKXPY4rWU74e7EtY3v081OgHthSIs1mzAJkmZD2Ia99VHWqs+0KHIR5o30ypJs3t1op3rwTluAW1VTLspj+/rN4z/sO58ONfTgOOMWyeOTPP/PyiOU9QBuo3l+ff98S8e4E3Az+NgTziReWJs49r2C7BMvKceG3/sTZF3+epl2uYeDgE2ja+RrOuujzfPIb15PL5QJlWqFvg1gRg7PegK149bmn5M/jfRI6Dp2HV3aocrEsTvvMb/jAxz/H4GGXYVmHYFkXMnjYVgYNHQ16GXX8klNocaa6WljY7AMIm8ghbFxn8cvL34Pd2RlsZOB0CoFWV8gjQe1ISIGkGYyV8B8jIV+4fIBZL/zFUfzFA+adHd9i1gt/KfzxKowMby2sZK29dbbPXOD4uFdBOuHHY14yngll/ZVU9SXMq8nikUC4ld4CdcZ0CnEdQlRZAmzdtJ6Hb/8Zl7aZaZiXtrXw8B0/Zcum9cG0lm/zV5u2EyBUt48czVZM9P4OwL+F2wVgWTkOOuJMvvWzv/PDm97ikp/9nQMPP4NczgqVVboDcNtU7Q7Aa5PaNM/7D5s3rsXWPFaugSl7HkJH+xbMw3FX8QPnvusK/j975x1nRXW+8e+ZubtL22XpHXaxIYqABcRujImxxdgSu4kxGo2aXoxiYmzxF7HEioqgqKgo2GJv2KgiVXqv0rbClnvn/P44U860W3bvGiW+fq5378yZM3ML73Oe53nPGYs2wBfAi0AhS4Cjqa+Dd1/6m/aG/Neon88X2ueg79f/DRppjK1gn1EJOl375kZt9UbSGeZqf3YRgUNfy9hNq32eRCksLwRuFIMQYi9gMrA/arXJnCKrT0II8Vvt8Xt7Oect8e39CTUuJ+frkREcYh5RgOAc89pTd3CK9E++OsmyeG3CKPf4EGhEsIGoc6jz2CCgXY/7efkSvh8EvIfIiQWA8PqPScLZMgChXYjzPnQA1I9pKgBYqRSjrj2CWVNmkGy8B6R6/uzDmdTtrLInxzX6vp8TUXfMVkxAYtowMG/Gmx7gRrwvPXyj/1wBIJjw07GBFh79ty3uQbrbhar9sLuM/iXhxRvjHt+wGIW6beApwGIhxCohxKdCiFWosc5JqAqgtKstREW231Cx9ihCeQA/THeAnpjjEmC+HpkAxswBFISA2poKXptwFyPr/ZOvbqjfxRsT7mRndYVKZRGJWyVX0rKBKBDQgSAIAnqyjwOBbFiAk5odABCG8PWTSqV4dvQ1/O4ne/Cbs7rzh3P34LnRV5NMJqMBANyEp7MA9/0653AeOQDAa8/+hdqq1sAM/JO8ZmJZXUgwyZ3g5sTfgbVABXAzUMhGoAfJxp2+a/RJhO61eBvzCQB6ZJJ/ohJ7LgCgx6CDf0ii4BaibheaKLiFQQdr/3yzSNRf/9G/QAozq8c3KaS6peR3gb8Cq4A+KNm9j/36r8DxdrucIlvDd6GU8jl9gxDiLNTtwyLjq/6xOP8e4nwzEbHPFNGln68+eQeHpZKRpZSHJpO89vQozr5MzQF2RxJaX4bAe2HqC5UFLigQhr3Rsg/WP0JtdoH9HLzy6O1OH5b9h7ScRGsvs2AnpGRDIyMvHUzVjlaomxEdQGPDXD5+8wbmzRjKyAdmu5PbDPsNJS3J7I+fZcqr46jYvpbSjn044gcXMOSwsxD2uwlWAblvxL0w7wO0LIt50yby6Vvj2bB6PuoeQmHNOsF+HMHi6O/HvvqRqKHS86QQBW3cChqf+e4Y7s71Cc94bZYBLIT7YwuauZlKP6PM36jqn+h2Xt97DvwBKxZ/xPrVh5NsvBbHME8U3EKvfj3YIwuzNxi5mr/Zzo3IV2R5c/ZvXEgpk6gSt9uEECVAe6BSSlnVnH6zTf5/IZzoo7a5ETc4cZJOc8OKmdGhnzeY7KMAQkS03bRiPovbFnMExZHn2HvFPLcvU/iPTQsGpr/aSFp2ntCuT0owhVB9Cm+ugJsrA1eujgmWUYa3ZwKBiY/+xk78M9CrQ+AkqnYczOSxv+PMn9+tzmmoRD1u1M9YMnelPYv4AGoq5/LCo7cxf/qbnP/rh9Vy126yD3xfFr4yUCktJtx/GcsWrqKx/s/AJcRp1gmqmIpgaKKQVLKeUqASKFUfMSV2u5uBl/mMPQ88x77uCABAfTc6AAB2Keo3FwCEMPjej25m+RevMXfGv7Q6/7PYY98TwrJMC9X9f5UA8A00c3MOO+H7kr4Q4ijgWHspnqwj06qeP0BJqb2EEPqCNiWkuZOXIH9JPi6y6l/7LehgEQUQOjD87o5Jof3ua+1v/d+5syMTGOiswEnEDhAEr1n15wcC93ih65eZ2YAHLOo5CAJT33kFNWaOWk7h70x792rO/oVK/pYFn330jJ34p6CDRUP9SSyZdyRzpz7PgYefjWU4bzY9C5g39Xk78TuT665GadbhSV51XE6iYAFHfOfHFLxxF/sBS4F/a21W4kx6gw9nvsKy+Z/SvkNvDjnmJ+w7+FS1oqn75eMDAPAS69cJALINHQD2HHgSew48yd2Xts8sAKAppZ9fBQBI/jeSf0wcg7rdYP6SP7ABdQurU4FZ2vZq4DexR4nwyD9DsUOTI+PqkG6tfRo9yO1LhIAh9Fr723QAQ+vH0o/R2+j1+MK5Jq+dwwp0RuDsimMEQSCIG/U723VwcEHKPm+yoZZ01SHJxlr7/gUSw5B88Oq4tOsGTfnPvznwiB9rjEP6WIBlWXw+9Vk+fv0JKrevo76ujsaGM/HKlX+I+j1HTfK6gaGH/YDKTYtZ37Yj03ZVIq2UvSSH94VJCoBiRCrJrtqr2FXbj9efvZVFc97l9IvvIWoSnqPL6zKQ4wFY0mo2AICXgHMFAEf/1xlAsE+vbfTkr4yg8o0FgG9kGWfGEEK8l0WzMrvtO6ic/aqUckKmg9ImfynlHGCOEOJJW3fKKgRgGul+YNn2FHdh3p9mmr6kFJHnigIMNSM3qBPpx3jAEJzJmzUgpAEDlxUEj8MvDzUVCCAeHEwBBYVtaaiPX04hUdDWfs/KYK7Ymn7tnYpta9X57C0G2EAjsZIW4+/+OUvnraCx4XCgEbUS6VjUMlJt7OvYilpA9ka8SV430K6knhPOvgUzkXDf2+hbTmfjmmtQpnAwngfuA96lseEkVi0+goWzJ7PfQac3TQbSNDj/xLw0AKAaad9D0wBAHZO7B5CpbeDAbyQAfAMrebKJI8k+Yx5jP58nhOgqpQwvP6xFWodECOGsFTFbCDE3+EjbsUjzsE9sCpnTwzkuU2mnU8kjDBl+CIlp+B+GANPwHk4FTG11Bddf/j1qqiswhHQfvmMNQg+92kagkqspvP3q2lQ797x2G4GdK5z2AkxTtTcMR1Kzz+H2YVcM2Sd0j3OuxVD79GPUa2/7Yd89BTXSDleHwA0cdvypCENgmKq9WhcovpSwQ6c+7twAKSWzPnqaW64+mD+fX861F/Vj4awpNDYkUIsVHmwf1wdl8r6HWqrEWXvoCuAQCgqvZtixh/Lb2z/BLCx0fqOAvhCchSqN1m92vxJYbffVisaGvzDjAzUwEoaBMIzI+RdOlZJ+HkMYXjVQRCWQEIarpzufu9Yo1D54nuA+53V43kB2VUBx8wui+gwcGL/PjqYUdbRkCajEzOrxTQopZUJKaaZ7oO47i/13T+Az4LJMfWeSfa6xn0/O6YqFSrTBaPbXHhiZpxtDuF9xjAwudcNYSF9fpn3cS0/cwdzp7/LK+FGcd+WN3nm1Pg39BAGmoM7jbdMlZucP6ba3zy39bUN+qfC3d++o5SSs4Hml1BiBf5+U0pWFzr3ybmZ/sj+VOw5GSYfeSLukQwNnXXqnnRwVVTjutIt56t7baKiPXjfomFOuVEnNsnjsXxez8LP3QHYHHkQl6bkosKkHXgG6Ap8QNJvVyL+eYceexcnn3e4mDyklmIZiOZZlLwT3ud3nMhL0waAOk+6keBJlUVmoX+Egtm9ZDlKqUbTzbLMA9bk7H1JuLCBfMlCUnBMlA2XLAIDcWcA3iAH8j2v+7huXUm4WQrxBOlnejrT5WErpTAO8Qkq5Wn+ghmOxV+KMZE0BCUM9okbJzXk4/ernCj2MaGYQZASm4WcZu2p28OL4u5gsJS8+cRe1NTv8be1HwpQhxqBG/tmxBOd6guzAmScQYgbaqN8wFCswTX9/IVbgPuxjAozAMASFhQn+9dR8vnPqURQWXY0QwygsuppjTzmS25+YS1FRwr4mRUuGHX0O+w7dg8JWR6JklSXA8xQWHcmAIXtw8FE/xjBg1odPs/Czt0H2QNlH/pp9NW2kCFWuHGU23wiUM/uTz3n6/suR0rI/A5u5CDBMgxHHn49p/hE1y/0lErzBi0gKmQRMADrazwDzaKi3eOGxq+z+7LkSNgsACC7FkS0L8E++i2EBzpcIvv4gOxbge61dc1wbve/I7XlgALmygPwzgN12eYdsQuIf5tYT/scUimy/geMjtv0g3QF+WSX9wxRW2kfG40MyTvT+jMChJeNJ40ZxiiU5ATWz98Un7vQAQpOLlLxig4KZHhSca4sCBCeBO9fgXpu2XzjJPgIMdtVWcNs1x1O3s4KE6QcWL+kLDQzCQFBYWMBF19zL6FfXMObNbTz0yhouuOpeCgsL3Lamcw0Jg8uve5zzf3W1t27Qnvdy3q+u4tI/j8U0TQzD4LVn7gBZhGKmrfBkme+gZqUXAOvsv6NiELCFxvopLFuwgvkzJoUmpwkhOGD4GSQKLeBPJLiXU7E4ATiZFIXsixr53wTsBG4DeRsrF6/mi89f9BJ8AATSSUF6kvcBQAwIOJFOBgqCQOy+LGSgOFmnyQDQAjJQvgFASpHVY3cLKeU/bMnHideBX2c6LlOp5y9RI/z+AY2/GCXWRh+HGvmKvJV7ZteP+8VGfL/Bahf1SvjaOgZqdVUFkx6/m+nauj7Dxt3FmRf/muKSUq2t6s/Ck4p8spItJwUlKOc6Lal8D/3cToQlGpXIpfQkHkt653396TtYMOtdXp8wijN/caOSiQzvnJb0S0QGyvx15gBYjjwUOLcjCah/OM61SIQE0zAYcdw5DP+OqqNX/UvtfJKKrVtQSd/R489Hjc7/hCf/XI+SKN8lPB6ZB6SAS2ms/xMfvXEvQw8/G2n/J1DnMU2TRCJBPWUkuMi9NeYtwCtIDLpjsQHYAzgWuJDGhmKmv3cX+x+s7snsSDPOTGQnmxm2rhGUguyNqo193XGGsLNRSssFAEsvDYsxg4FQ+akT2VYDZSsDxVUOaQfmXQbKJwBYu+eoPueQUk4Hpmdql0nzfwp4DbgVdWdwJ6qllNtjjxJgGuFfQL6/mtBPNAA2vt+wnuQDdfFB0Hj+MTXq19eNOdmymDTuTi6++m9uW6cfwzmX/u84KvGLCFCIAATVp98gSAcGNTUVvPa0kqjOnXAnJ533W9oVlyJlwDOwwUD3CyK9AukluWyBAFRfTnvDeYPCQhm3c1G/x0UkKCXJcahpWY62fzDq5nAX44Wzdv+tqGqdNWzbvMy3XpC0FHoJoH3H3tRX38KpNPi+uzOAibxNHRuB41BTV5T2X7VjvZegrSRvPP9X5k57k2RyJ4lEGwYN+x7Hn/YPjEQiFgR0PyAdCDgVQepziweBXLyASG8g4AXktRy0hXyA5oYkfuLntxEdmTT/SinlKinlObbOvwv1ObcT6gYvkSEIV+okMkg7TXmoPuOrgxKGjPECgu0sTPuxs2Y7z4+7h+sD6/qMrN/FC+PuZmfNDk+20a8nID2ZhvRLS65sEm7ne5jqkTD90lGob1O6ctCrT6pF6ByJ6rWnR/n8ADPiOgwDzASuRGSankSUMD15KGEKV/JRf6uHaQrAYtp7T3PTVcfz258M4Karj2fmB08hsGyZStChc28UUbwBeJgE5Ri8j+lbh0pNJFOTxj3/AEYAewMXosYe47FSjoyl5BcjYWAYBgLBsGPOsNf8afR9dzegRjkGN6A8hDH2nnmUdOitkl0qxT03HM1nH80i2fhv1EJy/2b2xzO57x/HIlOpsBwU4wcoL8IIyUG5+gG+9hFeQDopKM4LCMtFIlIKSlsNlIUM1BQfoLnxP6z5NymyvZPXKahV43oCXwL9UCvKRd7MRQCm0EceLVuAm07Hi9sVPMb5UTz76F2MiFnXZ3gyycSxd/HzX4+0+wgfr/ftjpSFvk8hzES7AAAgAElEQVQ9O+v46CNo0AhKoOpH3+70L6SkpqqCV566ixm6RPXUnZx6wW9p267UZQdOl7pU5LAV4Sg+UcyAaFZgpSzu/fuFLJi1nPq6PwH7U1VxP4/c/iceG/U7epXtz3dPu5iBBw7jw9eeQQHAGhIk7ZuvjCLFb1Gjf8BeiV/p8qtRk8hvAs7FGaXDagyzQCVee86AtOwRrCGo2b6WI+xivuB3dxTwLqOp41LUnII6CgpvZcR3L8UwBf955q/aQnL+iqPaqoN5a/L1fP/MW+0vL1oO8iXRNHKQsLzfR65MIEoKcvY5r78yKcgBgDRM4CtjAbupnt+Ske3aPjeh1sx6W0o5VAhxLHBObGuf7NOyid85XzpRKepHISMASUrB2qULmNu2mKPaxnS2ZL6r07syDiC19yl1Hd/Z5ib7aGDIChRc6cg9PS+Nv4NTrPDS068+OYpzr/g7oOiw27+W9DOCgeYtSEu9dryCT96eYCf+KahZuUrLl/Jeko0HsHrpXMb86wak3ICSff5EAb/gBNuIPQWLFxhFihvtK58HOGzrXuAn+InpPKA1Xbrv6VXkSKkkNKk+3YoNi1hR1JYD6mtpZX9c+upMvYGVjCRFMQVFR7DHvuXsP/xMBDDn0zdIt7zF3GlX8f2zbrU/9/x4AlmDQITEk84P+LpJQQ4DaGkQ+HZUn1tkm/wbpZTbhBCGEMKQUr4nhPhnXGOBJCFScbu/sogzgIPGkPuTFvDP0U/6DggzBICUr2/9n4QHBsJv/jrwIDyY8Ebkwt/OTc4eEPhBQbWprargpSfuco1pJ26o38Ww8Xfyowt/Q7uSUoSr1WufgQ0IznQMze9VGr702gsJ0snDNjN4/bkx1Nf9DlVU8DhQjloBxBs1qzWoRgEzSHAdgnr62XtvYRcvu6P/ViQKbsJMQP2ueuB0/Im/DrgJ0yzi2FN/aktOKt9YUmCikthPfj+BNyf8jYYX/8Us1EyXPbVelgEH8BoFHffi+2f+gf0POcNNgslkhuUtkjsxTcNNgJaj80sQpomUinFIS93f01k3KA4EACx7gJQZBLRLScMEMvkB2S4P0ZIsIPR+8hSKlH6b/HOJbJN/hRCiHTAFeFII8SVpFnYDiSEsxFcx6vedNfDlay/1H4YR117vKwAcOmDoo2J8iZtAcrf3RVyHAw5RwJANKEgJk8eNYkQyFb20cSrJS+NHccGv/h4rGUWBAXjsIBIMbGawecMK4FJUJU4RaqHBE1GrcZ6D+pSfQJV41pHgHiYBF9hb9gROIckLXIooWMagQwbwi798yoM3X8y86YdgBe7bK4zNDDzoKHv+gBqFWgjXUHeAYPv6xSwpaMWhjXWRn8vhQHLYCQw57Gz7OPU5FBS0pbEhzfIWiTauDi6lxDTta8iyQigIAr41g7IGgezkoG8CCLQEAKS+2nTzjY9sk/8PUcOv36DuEt8eXL4eCgHuyP+rAoB0idwKlHTaB6Q93q/Ti4yA4Wx3QCIWIILgoDEHDyy8UX46trBu2QIWtCvmKE3c0N9Watk8ZRBj5xyXNYgQIATZgV7l40pFdheN9Q3UVm9AjfLLUD8Fp2zzNuA/qMS/BjiABDdxKo3KkMZbb/8W6nmF5znrivs4+sSfI4TBr254nKnvTuDFx29k+9Z1gEGHzl049fxbOOioc9Ticpb9Hlx5zQOC8//8LHdedQAzNy7lmIjvqRoo/vx1jHNvtt+P+m/o4T9g+ns3kG4hOcMU9npLwjVGHBCAaDYQBwJupRKaJBQAAey96j1+A0HgK2UBguynLWXZoxBXAH8AegALgF9LKT/M4ri9UMRTSCnbNfMa+mVupcIuysk6skr+Ukr97vHjMrWvq9nFzJemkaxvIFmfpLEhSSppkWpMkUqmsFKW/SyxLEv9qAIjUbALH0xVXaEqO0xVaVJgYiZMEoUmiYIEicIEiQKTRFEBBUUFFLQqoLB1IUWt1XNBqyKK2hZR1KaIRIE9Vgzk8OCNIPQkb+lVDxE6PngJ29CG+SFACQJEDHvQwSAKGBwmcNMDz7jtdVCQWuJ2YEP97f1j1ctA1WcvtNeqvWv8aoCQSln86swhQGugL1EGqVpPZ4K9/1MS3MPNdovrUObRD1GW7tEFhdRu30BBgamAxjA54nvncfjx53m+iHY90sJeHU8qtmIDgWE470vSrc8A6mq2U084CoGuvffBTAhlaNsa9ykX/B8LPzuEmsrw8hbt2tdzwk9uQ5gmpmWp6zKFAkht9c8QECCaJAlZWpm0BwT6RDHLPUcuIODu+6pAIAcpKB+RT9lHCPFj4G7UPKeP7OfXhBADpZRr0hxXiPrxTwGOzsOlrCCdoemPnD7NTJO8qgmNkdUuQEopSyL2UbGpgoevSo8RZsJQid0w3AXJhFM+B3YCklgpibQsLEtipaxIMyqXSBQlaNWmiFbtWtG6uIjW7VrRurg1rUta06bEfm7fhral6tGutC1tS9up1x3aYia8iXQOYEQCRUC6gTBARIFDmDl4wBACAv1vDRRkgCno8pGUUFVdyd+u+Qk33DWBdiUdfD5CkCE4r6UUmEje+8+TfLmpGrXy5t/wEv9qChhKI7NRZZn3AReQ4HecSspnSB8HjEDQtk07CgqK2HPlPC8JSDW6ltrDeR+GlEjDAQJb6xcOEKgcI6XgwmsnukCgjyylZbfTzAxLqqlihYUF/PnumbzyxB+Z9eHVNDbWUlDQlgOP+AE/+MltmKa9gqi9nhCWpZbFFvjYgAME3sg2ezaQqy+QKwhAwDRuARBoqhTU3FDKZP6SP/BbYKyU8mH79VVCiBOAX6JqkuPinygK/AH5Sf5jCSd/AXQHBgPdUBNul+XacaYlnaNvZZUhuvTpyO0P/I5EkUlBYQKzQI3QzQKDRMLEML0fd7aykD5qTSUtkjaLSDakSDYkSTYkaWxI0VDXqJ53NdJQ10j9rgbqdzZQV1tP/a5G6mrq2VVbT11tPbuq69hVU8eOjTvYsGQjO6t2sbO6Li3AtClpTXGndhR3bEdx53aUdCpWz11KKOlcTPsuJZR0aU9JlxJatS2yr917v47W73oI+ohdOOZlGBx8rMGViqKBwfe30CZk2aDwwtg7mD31PV4Ydyc/u+ZvdluHNUQzhJrKHfzjN2ezaNEKoAtquWXPIC3kPGAHhQymgbdRtfq3kmCrO+p34mbgVcPk/15YSXFJRy9x4yR5Z3Ss3mgQDKTENXkVGAhlSttAALisQGcEGELNObMlFxdAbF1LGAlOv2QUP/rpKHUt9u9AWrY45KzjL0G6a/yDaSdLHQiEm2xlSBYSIpoNCPu+orlKQobGFHSw82UMbUwYlIRCzKAZINBcP6A5ka8lne3R+0HAvwK73gQOS3PcSahFMA8kem3xnENKeUma8xWgFNQrUWZbTpGt5p9TFBaZ9OnfIbxDSiAFVvMqgQpM7IVtClBrw0RHJn3efa39+C3LYmdNA7WVu6it2ElNRR01O2qpqdhJ1fad1OyopXq7emxZtYVlM1dQs31n5I+9Vbsi2nctoUO39pR2b09p1/aUdi+lQ49SSrt3oEOPUtp3KcEwDT+LcMBBZw9CPUexBh0YdCkpChSqqyp47rF/M1lKzht7D+dccjXFJaVu4g2yBAcQJo27g8+nvUeKQlRacWbt7gWsxuRjXgBOpxKDkVjUUsAGjjETmKlkyHg90jR569m7OOsXN0ZWE0lLhMHAZgU416o9DA0InPJVjxU48hCeT2CAMxVF2oAgHFZgMwIMvzSE0XQgCMtC+NhAJiAwbcBQ8xuCQNB8SSjKF9AniTlA8HUGgRxG/p2FEDO116OllKP1/aif0ObAcZtRN1MPhRCiB2o98tOllNVp10rKU0gpG4HrhRDfRZXj/ziX41sk+SMlRkrNshRRX35LnDLNhx21T+ozFD2NBRNo3y5B++IS6F1i747W8Z1+rZRF5fadVG2toXJrLZVbqqn4soodm6up2FLFjs1VLJuxgoot1STr/UVShmlQ2q2Ejj070KlXBzr27EDH3h3p2KMjnfp0olOvjrRqW4TEiGQN6YAhChQmPHo3p1jObOAUzz12F7/4zfUuS8CWe6RWeVRVVcnEx0bxIvAjGkjRDjgEZe6eRCHncRpwAs5SCq9RRx9MqplV1JojC4si+d0+K+aRMNXHb0n7/Ug8MJD6hDMPDGQADHJhBQ4QgOYT2NUnKgGr30JzgMC5CU82/kBzZKGc2UAezeFcQSDY1j6AfIWUkLKyTrhbpZQHZ24W+tmKiG1OjAcekFJOzfYi8hifoCba5BQtkvyFlJipBm9DSwNABpSVUT+y0CHxhq+v8sf3tzrGFNC5Uys6d2oFA7qEj3eTMNTs2Mm2TVVs31jJdvt528YKtm+oZMXs1cx6bQ7JBj8zatehLZ37dKRT70506duJTn060rlPF/V3r44kChMKHITwA4PwQEBKQWVlFRMevY9p9eqGLSPr6hg+5l7Ov/QKSko6uOxASkEymeT2G//Ai8+9CA1fusn9FOAF1pDiY9QiaYdgMt8t/boBmAwY7E8d1fQtH8JDz7wV8g+cpO0mZD2B4wcDgOraCu669iyuufk5WrcrjQADPNYQwwqwWYwCAO3cujzkTsCy21vSAwJDndCyC1mklux0IDCc6wowAm9BPg8IfBVLmiwE6t9RftmA95uKk4T0G1tk4wvkrTqo2ZHXpRu2omqYuwe2dyXMBpz4DnC0EOIG94LAEEIkUUvij445Lh/RGsVWcoqWSf4Ekn9c5PojyJJKxbOADAwAL1F76kk42Ye78hu/UWAhhRqvlZYWUVrahT327eo/xn5OSUnllhq+XFfBtvUVbF1fwZZ1O9i6bjvrF61nztvzfOAgDEHHnqV06duZrv0606VfZ7r060q38i506deZwjatXXbw5CN3c3JoNnCKCY/cyxW/u1YxCCloSKU44YhhbN3SmgJKMPnS1e1vAV6mihTrgFoKqeQ08PX5I+A5XqeOdmzZtFaN7jX/ABk0k9HAgEgweC2waqnODFTCdRJLFBgI9/xBiSiKFUiJDwyCPoGhA4FUhnEcEICfEUB6IGiqPxBiA4oi+n6fzWEDmXyBqBH+Vw0C+ar2kVI2CCFmoZayf07bdTxq0amoGBR4/UPUTSqGAevzcmHxMRrvZhVZR4vJPiKZRfJviRAiDf5HJf/ANl+Cj/cGXLNVa++OPKIAw90UDRRuJZAw6NylNZ27tIEDe/nbIkhJyY7N1WxZu4Mv1+7gyzXb2bx6G1+u3sbsN+ZSvV2vyoX2XUvoVt6FDr1LeevFt7ixsSOrqaYHtRRiMbKujkMfvY+LL72MkvbtkULwj7//ga1bWgMTMRnAWfiTuxr97yRFFSa7QhM+RgKTkBg00q1HbwrMVMhD8I3alX6lAYL0sYOaav+qpade8FvaFpf62IFl9xn0DCwryi/wg4G0z5cNK3CAAPBYgW0YO0CAIVzD2AECEBg2sKRjBNFGsXpfsf6AIcJsgLAsJLR9+WID2fgC+ppHcZJQPiLPWDIKeEIIMR1VTXM5am2zBwGEELcCw6SUx6lzy/n6wUKIgwEruL2pIYTYB3Vv00NR86y2AVOB+6WUnzelz5bT/P+LyT/NztCWkCQk4veLiOSv2jhJPwwcHmCEASEIFlIYaUHCqQbq2q0NXbu1Zb+De4MQPnCora5n85ptbF61nU2rt7F51TY2rdrKrP/MpldyAA8JeAglKXSilu5Uc1D9Tu6+5jbO/eWF9NijG5Oeex34NwVcRAKVzPVQo/8aDEyOhDQLqdVx5jlnUWBYnqQEGghkBoSUFLz8RHjV0p/88sa0UpFl6GAgfNVEDlhYFv4+0rACEO49ClzACLACHQjAvgZLAwJbLlasQLXHUAnZNYud2cu5AgFg2ZeaiyxkmuDcUjEKCL4qNtDcUF99/sBESvmMEKITampKD2A+cKI2kaoHSvds8RBCnIdailbP13ugWMUVQojLpZSP5dpvC2n+FiLpX1a3xWp8M5lGcWAg4uUd8BJ09H5/IheBzdGA4SVxoV93DEj4AEL4t6km3n6dPRS3Myke2JU9B3bzAcPll13JZ9PmUGC1pjDVhtROgwqKWU0xra2ubHi/gn+9fw8AwxlGLe9RT5Ke7M0CqtlGFZ2pxcBZJsFkKik+QdXERdUEdwW2b1xNgZFyvQTvWZmg6cFAsLNyBy+Nv8t/Y53xd3L6Rb+mdTtnjkK0VBQCg4BMJLVtcRKRzgqiJKJ4eQhfGWl6ICCyaqipjCAbfyBbWail2UDeQEBCMs8pRkp5P3B/zL6LMxw7FlWj36wQQuwHPIoaX/0GNXmsClWG+j5wFzBaCLFQSjktl75bbORPoz3yb8GJHb5IBwKxu4Kjfu+nLgKvQ+fQwUNvF0zUvv1RSVwDkAigcM8bAImKqhp+9uu/8Ojd/6R9SUkse3DA4eH777TZg8H/3XEPmx95jLH19VzUqohuP/8ZF53zMzYu/5L1y7fyfzeNpnWqnk50JyXKudfuzZIpdlHNLqrYJarp2SpFxz7FrFq3mIq6XVj2P3TDMGjdujWFhUWsWbKQAiOpZBzhlJ56ZahBQNAZgiUFk8bdEbqxzkmWxUvj7+SCq/6GtKKlIvAAwbIiwMAnEwkfC8gWDPwSUYAVSGeCWZgVOEBgnz7gE6iiX5mOEag3FwsE7lIdOQJBU2ShprKBKEmo+ZFfGelrEr9DfdqnSCmXgvsZ7pRSviaEmA0sRS1DcWYuHbec5t9YH9rWIpFJN4wd+WeuAApLQtpr364IUPABghG9LzSizwIk7DYPPvo4H06dyehHx/Hnqy9zj9cZRJA9SGFQWVXNo2PGMbVefT/X19Vz6JixXH7JBQw5opwhR/TnlTkTeXXyy8wFussCVlPCGoqZSwnPUkyXNn2Qu4Ra9mYpDC3aix77dqXn3j3sR0967dODjr07IwwTi6RKumgJXoiMgFBVWcnzY++JXLV0+Li7OPtn19CuuNRL8hHeQcoSbgWPzg4sGcEM1IeVExhYlgxJRJlYgZCAbYJHsoKgT6ADAYDNCr4KIMgkCzWVDeRd78c2z3e/OBZ43Un8wZBSbhJCvISqNsopWiz5y8YcNP9M7KAp9cBpflyRP7xQog8YxxGj+9Bx2mb/yN/+290UAAO9Tw0I4kCioqaWh554lslScsHjz3DFeWdS2r5dgEFoTEFjDg89MiZc8ZOyeOThx/jjr68AAZvXrHG1/C000oZtDGAbA4B3gF0DD2T8Y+NYv2IL65Z+yfplW1i3eDPLZixn2uRZ7tsubF1Ir72703OfHvQe0Ite+/Sk5z69KO7S3p3Elg4QnhmT5sY6qSQvjLuTS64ZqSQky+kHn7ZvGpKgkZyyf27ZgkEqFV9W6vcN8scKouQh3TBuCSCInz8Q7w9kIwvpETeDOB/RUuPL/3J0J77CyIm1QMdcO26h5G9BQ9SyWnqbJn5TuYwYYkBDgrpnYfgA/8tgmzSVQCKQvOPkoFCS107rjfSjwUKtfSS4/7GntYlaFg+Oe5prf3lhQPoJMAChpKLR4ya4o34nrq+vZ8Rj47niwrNo376EDWvXsUUIBkd8R0IIuqxeQ3Fr2Gf/ruw9qLt6A7a/UFtdz7plX7J28ZesW7KZdUs2M/ed+Xz8rCdHlnRuR8+9e9B73170HtCL3vv2pPtePSlsVeSCgoXBmiULmNe2mKNjbqwjlsyj0K4ksoQRMpQ9QJD2a8UOEi47kEo20qSilKV5B9j6uxGWidQS0nY/dmNPZpJNAANPA7ekxgqk9MxqjRUYtlbusAIhcYEA25x2gADQAEeN6qOAwLljm7uOkH2sJa2QP+ABgdCWtdYiCyDI730e1cBhN4xa0i1joGIv4ucfxEaLJH8pJbLRnsn6ddD8nSZxv43gjzCSGcSM9sEt7bN3+raH2qeRhYIj/TAbgIrqWh54ejJT6xWzuq6+gRFPvcAVPz6Z0pJir58IBvHA2Kdi1/8flkry4Jjx/OWqnzPn3cmAnzVIH5swkMl6V0pyDGiJoKSNYN/B3RkwpCc6KFRsqWHN0i9Zu3izeizayJQnP6ahzp4Jbgi69+9K73170WtAT/rs24u//2MU7bt3UOfzeQKOnCBIkUL9w7dc6u+2FWrE75rIhvBG+/b2lOGXikw3qavPxpGNXDCwsmMGql20X5CycpOIYllBOnnI5xOAyBEIQtKQkFkYxemBIJ0/kI/YTUf+G1DL40aGEOIY4FSUKZxTtJzsk2bk31yHPyeTyFeHH9wX7iey7zTGb6gLI4IdRCR6P/uwn4NgEfIIDO59alKkAXr/+Of568/P8bMIH4MwWLJ0BfPatuZYWoffI9BuyTKMxvpI1hBmE2FgkMJZgiIMCp06FtLx0D4MHtEPBxRSKcmm1TtYs3gTaxZtYs0Xm1g5eyUzXv7Mu6YObem9b0/6DOxF731703tgH7rv2QOjsAAkmBiuZAQOINjXI/WKougKI50dWFIgDelOPktJQSJGKtJN5CAzSKUE0nQSvmIGOhiYLhg48xCyYwWplPRKSU2RlhVE+gQaEPgWvLOTdiYg0BlBJqM4EgiyYQNNDCnzX+3zNYmPgbOEEAVSreXjxPFCiMGoxL8ZtQ55TtFiyd9qaMzcrqndN7eDGLopXPPRt9H/MgAOYVNYRLeNkHvcpukAw5YzMAQVNTt5YOJ/mNrg91Ouq29gxLOvcMXp36NDcTvQtX+tv8f/8Ts7YWvXLgy/vNRQ5zt3ZU0tF/35Zsbdfj2lxe00EGg6MFRU1/Lzq3/PI/++gz79SuldVsKIEwbggEJtdT1rFm1m9aJNrF60iTULN/LBeI8lmAUmPffuTp+Bve1HL3oP7EdRSRuQkMBJ9DbQpgMEww8IljQ8YNBG7K7EE5CKonyDhJ34U7YO5TeQw2CgA0E6icgwLJ88lA0rMJ3XQXnISfqaPBQEAgMbICLmEcRJQ+mAQK1hJN2fWt4jz0ziaxLPoJaGHo66rwCoFDgCtQTFi8A1UsqNuXbcMrKP1bLJP6vI8EOIopySmJG/kYY9BPbrfcvAdfjBwGBH7U4uvmMMY39/CaVt23j7IyUhuO+ZVxiRsmIM0BT3P/MK1150OpFMQmcRusxkJ+ML/zaKx2/8A6XFbbWEbnDf48/wwYw53D/uWa697PzI472RfhgURAQojH54DB9+OoPRj4zlj9f8MsQUSloL9juwBwMP6uleRyol2bBymwKEhRtZvXAj899bwCfPeV5C5z6d6Lu/AoS++/Wm9/79KO5aihDCxxBUEncSvuECgQIBzyAOsgMXDIzswMCUnjmcstIwA8vrT7rAEPQLJJY0M3sFlpW2lDSTPBTlE4TWGiJmQlkWjMCTmzQgyFPsjrKPlPJ9YGBg87FADbBE+m+0lVN85SN/mScPQARH3JkPSL8/IulHAYTI4A8E2YNvanvg2PsmvcmUeYu5f/Jb/PmsHwR8AOE7XgqDJavWM69VEce2KrLbuI0BaLtiDThmrnMunUloLMI9zhDc+/QkPvhsPvc9PYlrLz4TtaaMQUXtTh6Y8LKqKnp6MleeeWIIHBxJSfcrhAMEEcCwo3onDz0+QfU59kmuOP8M2rcviWUK0jBBQAKDsrIS+pW3R544AClMEILtX1azauEmVn1hg8KC9Xz22hz3Yyzp3I6++/Wmz/596LdfH/oM6keHPl1U0rLZgeMlxAJCDDuwXB9CuqP8lFQySVAqckb1qWyYgaUvdie1fc416ck+CAZm2DSOYQUOEAAhVhAlD1kWCEMtw+qTh5yvXsYwAkBY4dJRXRpqbjjfzf9CyCxuJZlNtJjhm8ow8m8qBng5P7d7AmTyCaLARCXKLNoGQcIIA4J3HXa55s5dPPTaFCYDF/znAy777uGUtm3tv1bD8AHJo9dc6Cb1kDlsH2fV1/tH+kE2IQyf3FRRs5MHnn+DyRIumPgaV5z6XUpLVHnNvU9N9lUV3ff0JP7607PCspLPYA4DQ2VNLRddexvjbvsrDzz5gr9SaexTXHvFTyMkpLB8JIUZ8hS6diyk85FlHHRUuS3BCWprGlj9xSZWLdzIygUbWLNwI28+tISULQq3KWntMYT9+9B3UBmd+3ejQBhY+M1lSytF9QDAskHD0fojvANdKtJ8gwK8ZJ+Q+EzkbMDAqySyj7HCEpFq07QKonTykGH3m9Yn0IAgfomJKGmo+bGb1vkDIIQ4DLgYdaOY9kAlMBsYI6X8tCl9tuDIP5m5XVO6zqGtb9SdrqEhiAaTGG/ASAVeB5O9/zgZwgqDB159n5Ol5ATgRMvigf+8xx9PO15r4yTv6Eoh5/1U7qrjZ/9+gjHXXOSCh7r0aJAQTmWSnVTvfe41TrGv4yTL4r6J/+Hac0+lsnYnD7zwBlNtEL+uoZERPnAQIVnJkYSCwHDfE8/xwcy5jHrkScZMet1fqTR+Ilf++GRKnVnKDlswTIISUhAUFJMSGAGW0L61waADe7Dfwb1cQGhoSLJm8RZWLtjIqgUbWLVgA+8/8SGN9v0VitoW0XdgL/oO6ku/QX3os38/uu7Rg4RhxvoHTmVRkB34tkeBQYRU5ICBAwBxYJCy9G3pzeNUKrNxnIkVxJnGYXko7BPoQAAgLBEJBPmK3XXkL4S4G7iKcBobClwihPi3lPKaXPttwZF/yyT/rCNLVShduVlI4klzTLRXENym+qvcWcvodz5lml0Oe31jkkPf+oRLjx5O+zatEEIb8bvqTYQpbAjue/ldpixcxv0vv8OfTjveu+ZApZArH2nvqaJ2Fw+++p4/wb/4Nr/8wZHc//K7nCLDVUX3Pfcq1573Q9W9fi5nNO6+bwUOq7/cxp3jJzFZSn783CucJIxwpdITE7n20nPDMlIGCUkaRlpQ0KWjhAkDBnZi7/26IMVgpDBIJiXrl21lxfwNrJivAOHDpz/mnTHq8yhsXUjf/XrRb1Bf+p1xzxcAACAASURBVA7qQ99B5XTdozsFRpghRAICfkDwvINoqciZYxBlIvvnFTivPTYQxQxM02urg4Fp6hPNbDDIwAriTWNCrMCRhwzpAQEonyBuLkFzQ0pINu8GgV/LEEJcgVrNcxnqDqgfoKp7uqHWT/wr6v7CS6WU98Z2FBEtNvJPNf6Xk78W2ZaGhpO6+jVV7qrjigkv88C5P6R961b+JpGyUIAJuAlX9ffQOx+HyzWl5KG3Pub3Jx2Dx0K0yiDDv4Y/QOXOOh5++xMlHb35Mb84djjt27T2gEIDCf26nOu5/+V3Q7N9T7QsRk18nbFvf+qCghPXNTQy4qV3+OWJR1Harp17Dp+0FJCVLr/5Pk6yFLM4IZmia4BhXVffwIhnXubKM06gtLidX0aK8xbc82RiCkZAOjJ8LCEhYI+9SinfuyPfOWOQmjlrwfrlW1k+fyMr569n1fwNfDjhExoeU59FUZtC+u7X22YI/eg3uIzOZV1JGKaSgnyGsgcEubCDoG8QKi+1PGDQmUMUM9BZgQcG6auIdIkolZI5s4KwPCSifQIJ6RZgzz12S9nnSmATMFxKuUPbvgpYJYR4EViAWnL6v5/8pQQrBxjOVPef38Wf0o/23dDO+fCUGXyyfA0PfzCNX3/nsNjjo65TtfU+i6r6Bh6dMoNpST84Xt+Y5NAPpvOzww9yASbIPPzegeDBNz/ypCMpefCNT/j9iccAKU+CdxO/n01U7tzF6Dc/Ympj+DoOev1DDiV6qeZhqRT3v/gOfz7zBLuvQOWPc05hsHrLdqZ9sZy59vE3oxYj/ylQovU5PJXivgkvce2Fp9t9eglbaMk7EhhMgzimoGQgzUA2AizBMO1lsgWWbSBLYVDevz1l/UuxTttPAUJKsn7FNlbM38DyeRtYOW89Hz71Me/UvQ+oezX33b8PZYP60m9wGf0O6EfHPl1AGD5DWWcHXvL3vIOgkWxZas6Bs3idtDwT2WMJ0czAKyX1mIEDBv6yUzwJKsIvME2tiihHVuDIQ37TOLqMNB+xm8o+/VH3GN4RtVNKWSmEmAhclmvHLTjyz5D8m/ql52lKeLazC6saGhg79TMmA+d/+hkXHTKE9na1TRwoRfZttx39/jRGWNGzbIenLB5+bxq/+e7hoX6CQFBZX8cjH8zwS0fvT+WSIxR4BIFCvwaAB17/mEOt6LLRQy2LWYkExwZZjh1tVq8nVVfvk5h0ecmRln5512P8CP+NYI4DjissoF1RoX2s+l/b5auRdXWenKTLPJqMpINCUBIKsgUl95vpQcGWjgxXUvJYgi4blfcvpqz/QI794X5YhkkqaSmGMG8jy+etZ+W89bz3+BQa698FoG37NvQ7oA/9DuhL2QFl9BlcTmm3UmVa28k95VYVGZ40JA2cUlM/O1Cv00lFQeM45TIFESET6RKRAwyCVAqizWOlzWcnEWmVSgEwiGQFeQgHQHfD2AZkqptvtNvlFC028k81tJQA17R+DdP/I8v0O3Fy55iPZ/pG1499MourjxymElTMpaTzBJZv3srcwkIG76qjFJVsU0AFUFpYQOGmLVg6cLoJO+Xre/R70yI0ecnod6fyu+8dqR0fvi5hGCzbsJl5RYUcY1eN7mpoZFdjktYFCdoUFnBM/748eMnZkQwCwGps9L1GCJ+8tGZ7JdOXrHJH/U7cDLwrBB/968+UFrf1GdKyod43YUwxCE1KagYoqFG/ko8E9n7DjJWOfCzB8FiBAwgJw6T/nu0p37Mjx52+P5Zhkmy0WLt0C8vmbmDFvPWsmLuONx96x60yat+lRIHB4L70G1xO38FltC0tzoodOPMOLGn4pKKUlC4YRJnIDhh4zEDfJ0hZUgMRpxQzXEmk+wWplCPf5CYRRbECkSfN3zHJd8N4AfihEOI6KWVdcKcQohB1y8hJuXbcctU+mUb+LRhRUwBS2V6ONjquqqtn3Mw5TLclrJHJFIfOnMMFQw+gxKm1j+rCjB7NCAF3nfZ97p4yje0z5zBOk8YuTJh0Hro/Vx81PJI16Syjsq6esZ9+FpaOkkkO/XgWFw8f4klHEQxACMG955xqbzOo3FXHYbc9yHPA+ULw5h8upX3rVlqCT8cijEiAuPLBpzmKeOnogVfe5U8/+l6INbjMLiAnCS3Bu20jgMHV+APAIAxvVB9iCkbCBwrCYQYxLEH3Ehxz2RImBUKw1z6l7LVPR6yzD0AKQX29xapFm1k+dz3L5yiGMO/dBa5Z2rlPJ8oG96V8cD/6Di6nz/79KGzTKgQIQXYQZyRb9hpsHhCEfQM/M7CBIA0zsCxIJOw1iSwoKPBXEjlg4EpEUmJZVtasIB8h2T0NX5ShOxx4TwjxF2CKlNISatR1JHArauz411w7brmR/9d0oY3M/oE3fBg7fQ7fT6b8hqiUjJ3xOb869OBAv97fIaAJAMrjM+cyLfBLHZlMceiMz7lgyCBK24blFp1MjPl4JiNiJJvhVooxH8/imqMP1a5NS956R3aJ6+j3p2m3SZSu9BRZbmr4+xF6krbPVbmrjjkr19Ee/yLjDsPpUFRI6zUbSNXXe6N0+3p8/Tojeac8VQOGKI/BA4ksQcEe+QvR6AcFLckLjW2EQEFjCbpsJA3D9RESpmDgoM4MHNQV6/yDsAyTndX1rFiwSQHC3PWsmL2Sma/Mdt9Lz717UHZAX8qGlNFvcH967tMTkSjQJCG9akhgGdHsQK8qivMNdBM5SiZSzEAdk5DehDMdDMyEdM1jy3LMYwNpSfv4sERkWZZvktm3ERtzgULUbSPfBZJCiG1AJ7z8vRGYE1AchJSyLF3HLbew29c0+TupXWT4zVXV1/PE5/OYEdg+Mpli+Ox5nDdoP0qKokf/YYDxAGXcjDlpErfFuJmf86sRGrBEgNWKLduZW1jI0c75AvsLNm3B0o3cNMax42k4YHR9Msmhn3zGxcOHKm9DmwMRLBv1b3MSMzz49iccaRj8O4Ju/SphMujwg/j9iUeTqk/hmNM+WUkvbw0AQ9Bn0D0G571GsYXMoGBEG81u8o9gCYbhXaOPFaQHhPatBEMO6c7gQ3qplUcNk4qtO1k+dz3L5q5nxdz1zHlrLh8/O1V9n0UF9N2/N+VD+lE2pD99DyijU98u7qQ3nR2opG95TCBgJDu+gQIDmQUY2GzCJxOFwcByTWO/eWw6bUISkTe3IC8hnZLW3S4soB5V3aPHugzHZfxgW27kn/qafxNaXopiA4/Pns+wVLQxe6hl8fhnc7nykIMiu9Y/9SDIrNi6nbmFBW7iDobx5TaSDZZ2Tf7v0DAE/zrxeN+2KJkr6ZOOUujzBHRZyvE0/OzG4tGPZsazh0DpqOrU27Zs4xblJ8S8x6J1m0jVO0s5h5lDnKTkvgeDkNcQBwyZ2IIwjQygYPeRBhQcP8ExmIXtJ4RYgmFguH/7fYQu7Q06H92X4UeXYQmDlDD4ck0Fy1xAWMcH4z/i7UffB9Rqp2VD+ilAGKwAoW1Hzz9I6ctWuA8jZCS7UpFW/RM0kfWKopRMLxNZtnfggQH2qF83j/0SUb5id9T8pZQtdpP4FtT8v57JP4plRo0Yvti8hemo5fQcU3YHkBCCdgUFiO07sgI4Z1KLE7d99zu+10aEOSyTam36KHbiDqYjMr4fxNS/BMOZ3OWM3g2vj6r69BLURQd53obvMrWRdpQfcM9ZJ4X323LQVc+9yk2nHu9OAoxiDs72nIEhyBjSyUgRoJAVUzDMSOkIjQGEWIJTceQkfDfxm3Y7DxAchmAKk949W9Gr914cdeIALMOgMSmVoTxHAcLyz9ey4P0v3ATapV9nygf3o3xoOX2HlNN7376YRUUx7CBeKtIXmosFA81MjpOJZCQzsIEipVf9ND9242qfFouWSf746VzeqF1M5DIPIFtCUtuY5BTgKW3bEOALKTl37725bMiQjABnmJmpaAriy1ftnBz9/lIh4IiStNKBxdgZc9NKUI9Nm+1JUL5riGcS7iVFAMSjH83kkxVrefSjmfz62BG+feGJaMHXBlF+QzpgCEpJsWwhA1PIBApeJZENCqbp9qkzAEcaEs7cBPc4DVgchuDKRR5DSAiTvfbuwF77dECePRjLMNhZ28jKBRtZ+vl6ls1Zx9JpS5n+krqVpllg0mdgL8qHlFE2pJx+g8vpXN4dtdAzLjvwgUEmqShgImdiBmEw8JiBU1aar8gni/hfiBac5PXVjfzzAS56gt1QU8PcrVsYr+2vQN0o80Xg3IUL+cneAyguLKS6oYG/fDSFW484iuLCQl8/mYDGbZumYRSAVNU38Kf3P+CfxxxNSVQtfioKMBQK6ICxMicJyvuMDUO4TAJA6sRBK03VAaKmsYFx0z9X8yWmzeaiAwdR0sZeyC6CPYSkpQhwiGINzQKGHCUkJRnFgIJpRkpHvqojw/MDdHPZNZZ1HyEOEApNBh/YjQMO7oFkOJZhsG1zLUvnrGfZ5+tYPncdn06cxnvjpgDQpn0b+g/pR9nQcsqHlNNvSDmtnXLTNOwgsqpIKzEN+gZeeamwZWC/TKSDQT5CSvgvFhh+I+MrGfl/E0K/3r9/8nGoTPEe4GTgBOB7qRRPffEFvxh0AE8tXMjMzZvd11HvO46ZxH1GmQBk/NwFzNi4iSfnL+SyIUNC+2MZh2GQ0pL4rccdG3t9DkhESVBRbCLoUegAIQwYM/UzTra8+RJjps3mqsOH29ebHXPIBhgyMQZdSooDhZC30ESmEOUn+FiC7iVoCT8jIMR4CKoU1aRbpwRdjyvn8O/ugSUMklKwdulWls1dz9LZa1k+Zx2v3r3IHSl3K+9C+dAyyob0p2xoOT0H9EUkCsJgYGT2DXQw8M01SDjMIEImylPka8rA/0q0nOYf96Um8/QNJfIzYoiKTTXVfIGSecCuIQZ3wtJNwPDFizixbxnPLFnEZOC8xYv4yV77UFxYGOovHRBGJd50oFDd0MCERV+oc37xhctA9DZxRELELJ8b5YOE5KhICcqTnoImtw4QVfX1jJ89n2kpb77E8FnzuGDwIEqKirAaA6qUXWEUJS1FeQ9pwSETY8gIDIbzNnzeQl5BwccSHGDQvATHXNZlo4CxrNiA4Z+LYANCAYI9+7djjz0H8v0z9scSBrU7k6yYv5Els9exfM46vpjyBVNfULVtBUUF9BvUh/5Dyygb2p+yof1p36NT1uwgSipyfAOdGTiTy/IV36o+uUULJX8ik7yVyuO3k5Kxk6maGyf068+6ZYu5156Acg+wHQILsVncNH0qp0jFBk6SFk8vXsTP9xuU28m0RG9EAIEeVXX1XPzW63xfSrcm32EcTgSBw7esdQwriQILYYhI2hEEiijPwm9yp3j8s7mhiqKTpGTczDlcMezACAbhsAfNN3LXutNZgt0uDXPwJf0sGUNQRsqaLTQBFELSkWGmZQlBnyAWEIKSkWH6XrcvNBlyUDcGH9IDyaGkhGDLxhqWzVnPks/Xsmz2Wt4bN4XGh9VyFaXd2mtgUE6fQeUUtWqVkR1kAwb5iq9nicnXN1pG8yfPiT4mWuocayqrmJ9IcBSqTG1HY2NomYJfWxZHbN/Kk/brkZbF8KWL+XH/vSkuCI/+nahubOC66Z9w07DDQu0sZ/wcw2rGLpzPhp21dHPPmWL44kWcvcfe7npDwZCWpLqhgb9++hE3jzgiNDM5Dizitjt4UNXQwF+mTOG2Y46mJMB2dICoqm/gqfkLmR6o+R+ZSjF87gLO2W8gJUVFIQYRZA/B6wAPHKI8hyAwxDGGXNhCOm8hL0whyBIM0/UShL1fusk9AyAYCdzS04BkpFcZqWUyTBLCoGeXAnp8bw+O/P5eWMKgsTHFykWqumipDQifva7ukmYmDHoN6EX/oeWUHajYQae+3UAYsUZyKgYM8hLymyc1/7ejxUb+Ml/yzn8h/jb0MPfvR5fMZ/2qxZiW5auIeQI4jTAbmLBkMZcM2D+276eXLGLW1s3p26XCI9rqxgaeX7aEF4ELgL9p53xm8SJ+vq/GOALg8dTiL5j55eYQM4liGtmCwVPzFzBz8yaemr+AXxwwONCH1+/4uQtiK4oOtSyenDOfyw8cEikzeYneM6uzBwcLwxAZJaUQOGRiDDmwhbDhnCNTSJjuNbmg4FQSaSxBGoYLBCHZyGYBzohfmIY7QS2KIeiAUCAM9t23A/sO7Ig8bzCWSFCxo46lc9az5LO1LJuzjqkvTOP9J5SZ3K5DW8qHltHfBoO+B5TRql27jFJRPkKye8o+QojxwCP2vXzzGi2X/HcT531NdSULzARHaaNZS0p2JMNsYKRlMXzlYs7qt1fk6L+6sYGJK5covT5NO2Enb53ZjFu0gBMduQclRY10zrl8MWeVe4xDz3XVjQ08t3SxOmeAmWRiGvabVX1qQFHd0OB5HYsW8eM99/ExCh0oVlVUMK+ggKPi+t++g1Sjah/0JEISUxpw0JmDY06nUmSUlIJyUixjaA4oBHyFIFOIBYTGLFlCBCC4PoKpDGG3/FQ3loOSkQsUwmUIhgscyj/oXGzS6cg+DD+6HxKTpIS1y7exZPY6ln6+jmWz1zLv3QXu++25Tw8fO+hS3gNhmiSl4YJBPkJKaPgGDzjTxDnAuUKIJcAjwFgp5dZ8dCxaojZ235KOcszBx+W936aGyLM5/Oiy+WxYtZh7I0pqrhQGvcv24Wd7hkf1jy6bT8PqJYyzUlxoGLTq57VLt9xEdWMDp7/7Ep/Zuvky1Lr476DWxb/SUOe8ZG+7L+39PrJoHvUrvHO27j8gknFk7Z8kBA8vmEvd0sWMS6k+2+y9b0ZG4UTkPQ9i2ge3R1Yl6Z9b0HvQJSJN9/F91sIIt3WlIPdg9ToLtqDLSLq3ECkhNRUUAizBMA2/l2AYIXPZkY0yMgSHCdiAQIRkJAOAIIXpLmkhMamuqWfp3E0s+WwtSz9fx/LP17Kzaheg7p9cPrSM/gftQfnQ/vQd3J8hg3rPklL6F8vKMXrvcbC85vZpWbX945mJZp/vqwohRArYBbQGBNCAqjgfLaV8pzl9t1ip59cp8i1BrazcwSwkRyYK/VqyHVZ1pe+cIiGUbLN6CdMtu+LFshi+Zgln9VMj8XRM6ZkVS0KG6XHAsYZBW1N9hVZNpdveObfDNHznjGEc2Zpl1XX1PLd0savhR3kdaRlFhMGdjdQkDBHZzu9Jp0IA4YCDT1aKYA3p5KS0UlI6GSkLtpAPULA0UDC0CWZBluAzl+2/hRnhI6STjBxAcEpOhQgBQodWBgcP78FBI3qDXWq6ftUOlx0snb2GV+5UpaYi6h9QE2N3lH3seBr4P+BC+3EmcJYQYgWKDYyRUn6Za6ctdg/fr8LwbWo0t0qoZ9tiPty+iVN6lPHLPQ/I2F4mJc+uXMwpBD0CybMrl/Cz/vvFHuuCRmD7zcDbQjDm8B+4STcIIM+sWMIpMuKcK5bwsz33942AdbCqbmxg5OefcuOQESGQeGbp4og+Y7yODBVZFjItQOgMIggE2W7TwSFOVsoZGCI8hmgZKTMopGMKUfJRJkCQcSzBNLMCBNdYjmIImiyktidcphEsObXMhAsGpmFQ3rcNZf0GcPzpA5GY1NQ2sHTuJlZ9sYm3bn2O5kf+lor4OoaUcglwHXCdEOI44CLgdFQq+LsQ4mXgYSnlG9n2+T8x8g9Gc4CpOtnApLXLEMCktUs5t88+FCcKffuvn/8JQghu3G8ExYlCqpMNPL92qTsCd2KkZTF87RLO6hOt/QM8t3YJI6SMWYIhHjyCTMN3TodxEH3OZ1Ys4bNtX7ogofc5Ma7PpjKKCHPbO1b7nnSQyJI9pCt7Vcfor/2swQcM6kTaXAf7mjVgkA6QhoAhji34ZSRdQsrEFJoCCkHpyE30AXM55B/oxrLpjPpjGIIGCKYGHr45CEYCaZiUFhkcPKwbBw3vwSW3kpf4Xyn2seWed4S6ufuZKCD4EXC6EGIligncnKmf/8nk35x4cvUiQDIZOBvJk2sWcXl/b/Q/ce1S5lVtQwIT1y3lp2X7MXHt0vgELiXPrlnCT8uiR/9rapThfGRgu5MsrNrK0DHQfNCYDJynyVIAz65ewggrTZ8aWDisIlvJLRuQSAsQQQaRhj1kIysFSxB11pCeMYQrk4LAkA1biPIVgqAQxRTSVR7poGAkor0En2ykVxs5PoIOCHGSUSaGEPQP8hBSQmNjfrO/nVz/gFpLfwHwaynlhzFtjwF+AwwD2qOsubuklGPyelFaSClrgLHAWCFEOUoSugb4B4oRpI1vSz1ziOpkA5PXL+N01MSu04BJ65ZxXt8B7gh/4roltEaJdOeuW8qZvfdi7c7oBO5EKiaBA4zczys73VhXyy9mvk15u/bcvP9hLuOIYjLNAY1TcCau+UFibXUlC+z5D1Hh8x2yrPbKBSTSAkQu4JCBOQSBID0wpGEMEcCQLSjE+QpZy0dp/AQhBFZjNEuwDJsVxMlGOQGC4a8yspN/yD/IU+QHRlQIIX4M3A1cAXxkP78mhBgopVwTcchhwDzgdtTNVb4PjBZC1Ekpn4pon89r7YY98kcBT1bx7cg/h3hqrRr1X2+/HglMRvLkqkVcVjaIiauX0F9aDEYlzxMsi4lrlnL9PiMy9h1MflEVSv/8YgY7rSTzq7a5rCIudNCIiyBoBOUpR5Y6o9eetC8q4oZBmfvMJonr7y0XkMjUdyop4yu7YuSlbMAhk3ykv7aSfs8gCAyZZCR1TR4o+oHB8lciZSEf5QoKWbEE0/DaugAQ9hGE6UhDMZKRAwh5Ciu/mv9vUWWVD9uvrxJCnAD8EvhLsLGU8pbApgeEEMcCZ+BfHDgvIYQoQt279yLgeyjs2wrcATyc5lA3vk3+WUZ1soHJG5dzOn6z8zRg0oZlnNq9PxM3LMUEnrX334jkkA1LOaPnnj5fIJsIJrqNdbXMr97msopz1i3ljO7x/TalvHXi2qVhUxqZEWiCkclQzxUgIDNIZMsgYtlDOuaQQVKKk5PSMYZcZCSHLcRVImUtH+UqHcWwBCfhZw0IOkPQq4wcXyAP4dx0PsvoLISYqb0eLaUc7bwQ6qboBwH/Chz3JmqEn22UkPmOWzmFEGIEKuGfDZSiftLvAaOBSVLKZJrDffFt8s8ynlq7CKmN+p0YCUxCcsuSae6oX0+e37dSPL9hGRf3Hdis89++dAb7AENRrCJTv7nKbtXJBl5YH2NK2/JVtgCWraGeDiSyvX4HJLJhEJnYQyxzyJY1pGEMUVKS20+2bCGlswDPW8jEFKKM5qxAIcZgFmYGc1mTjXRj2Z2PEACEfEUOyztszVDn3xllZW0ObN8MfDebEwghTkZVZB+e7UWliW5CiOtQk/v3Qs1U3Azchpr9u6IpnX6r+WcZ07dv4jCIXqYA+KSmglZ4o34nboImj/6dUKP+7bQBJuaxXz2eX5/elJ64Zmks0DR1Et1XBRLZAkRzwOGrBoYQW4iRkKJ9hTAoZJKO4qqO0stGpq/ayHJWM9WMZcPMn+zTApWewR5FxLZQCCEOR0k9V0spg1XaTYmTURP7JYp9jAZelLJ56yh8O/LPMspaF7Mg1cAwK4W0UhRr+6qBQmAE0eBwiJVi4rqlXNw7/eg/Lonqo/6WYBUAa3dVpTeld8ab0rmO0nONbECiOQCRD3DIBzBkMp/1v3MBhVj5KAIQXKM5g3QU5yX4AaHRDwgJg6CxbOVN9pHU5+/WsVtRH2H3wPauhNmAL4QQRwD/AUZKKR/Iw7VIlIH8GGqUvzoPfQLfJv+s47q9D6U62cD5n7/OdLwkDKqm6wBgCuoeABI1RHBCAsnaiozniEosUaN+J5zR/+ld9wiN/nNNtNcPyGxKNzdyYYO5Xn9zWES+wCFuUJitnBQ5r0EDhmzYQjoJyWUBKSfhWh5oRMhHmUzmdCwhnZcQko3yFPla4V1K2SCEmAUcD+gz0I4Hno87TghxFPAq8Dcp5V35uRrOAF6WMtMNYXOPb5N/DvH8pmWx0sgxwqBHj70ASG5cyjjtu7pAGBS2LW3SOW9fMZO+KKEvjlU8v2lZiFU0R3bL91pITYmWYhOZQKLlwCG+j8h/1QHGkKuMFPW3ZzjH+AoRTAFyk46awhLyFXmu9hkFPCGEmA58DFwO9AQeBBBC3AoMk1IeZ78+BpX47weeFEI4rCElpdzS1IuQUr7Y5HeQIb5N/jnEup3ppZG62go+r97K9ABI3yAthm1alrY6Jy621NWyA8X7BkfsF4DYWZVTn5kiV+D4b4JFUyqH0kW+waE5wBDJGCLM53QyUlNBIdpojvcT4lhCyEsIzE3I28hf5nc9fynlM0KITqglFXoA84ETNdmlB7CHdsjFQBvg9/bDidVAWd4uLI/xbfLPIa7b+9C0+8euW0jrqi3RprCUkSP0TDH+wBPdvp/csJjzeu7j9lGdbOAfy6ZzTf8Dc+oz39EUlvFVAkY+ASJXcMiGNbQUMIRkpJi7xkXNW8hFPtJBISwdWe754lhCPkKS95E/Usr7USP5qH0XR7y+OKrt1zW+Tf55jEzMINXEEfrGulqe2aDW5D9XYxDPb1rG7KotTQKV/3a0tEmca2Qr7WSKdOCQC2toDmPIWkbKYDg3iSlkIR1FykZ5iN14XbcWiW+Tfx4jEzNoaty+YiYn4S258LwNAJM2LQsBwu4WXxeQyAc45MIamsMYZEpG3h8ixBbywBTCRnPKtz1rltDMkFLS0PDtXXxziW+T/9c8NtbVsrBmOxPs145/0GClOMW5s1cTJaXdKf7bINHS4JCtnJSZMYSPy0lCagYopK88an4Y4tuhfy7xbfL/msftK2byI/z1/cdbFpM3L2e2zXObYyj/r0W+5J18njefwJDpPLmAAkRISOmqkHIwmnXpKC+RZ8P3fyH+v707j5WzKuM4/v21WEpT0dBCCrhUIUZWJVBApJUQEFINEWKsBhNKWAwlbgioiRIj4YU85wAACSpJREFURo0VFGWpWE0TWZUlgRIExJQdaYHSgiBVwMjSYikUEHqxvY9/nHfwvdPZ3lnunXnv75NMet/3PXNmbm7zvOc95znnOPj3sepWf8UMgk/F1pu09LL1Xxlc/u7uB5b6BjMWTxCd3pC6cWMYOTMl991qdCFtyeqp14XU7kBzp8oc+yXtQVpU7mDSmj6vAH8BLo6Ix9up08G/j/3kqRXMYWR+/6vAZaS/el6vW/+DPLjcC6P5BNHJU0O9G0ORbqSOupBa7D7qVFDelr+kk0lZR9UjJPsDX5J0em710ZY5+Pep1za/xZrXN/AicHju/Kukv3g300lb+S7jYXC5mzrt5un153T6tNCNm0JXUz0jeKt7yzv0DUkHAZcAG0mbtNwMPAfsSsoBOQe4WNLKiFhepG4H/z517dq/c6gmcGHVhLGvAXcBB06YyLY1VkRsN5202Xfx4HL3dHtiWjuf0+mNoRtdSFta7GJrhYCJ5RzwPZM0MDK7qnvnSeBJSbcCD2fl5hWp2MG/TzWaMzAF2GvqtJ6lluZVWv2VWcseXB4dvX5y6PaNodnTQpGlt9tV0m6f2aQVPGv260fEE5JuoI2lox38+9RoBPZWVFr9ozm4bI31442h2U2h10u8R3mzfd4NrGlSZg1pV69CHPytrupWf4Vb//2r1wPRzeYXVGt1wLlz0fXlHfrEemBakzI7ARuKVtzNPY+tZKpXMa288oPLNlhic9R8jXa9w1vi7Ve3RERLrwGzCmi06xjZ9dVFK3bL3+rq1VpF1n969cQwWllPwwFD5VzeYTFwuqRdI+K56ouSdiHl/C8uWrGDv9XVL+MONvaKdvd0Umc7yprtExHXAdc1uP48I7PBW+bgb2ZtG62WfQvfZBC7dMaUg7+Z9cSo3hhKmu0j6f3tvrfZfr8O/mY26rqd+hnAcAmDP/AU9WbNNdcwocfB38wGXgRlXc9/Ce0H/4Yc/M1s4Iko64DvSb2q28HfzAZe2sO3lC3/nnHwN7PBV9IB315y8DezUihr8Je0D7AAmAn8AzgvIp5u+KYWOPibWQmUM89f0sHAMuAdudOfl7RfRPxL0geAx4AFEbGkSN1e28fMBt7wMAwNbWnpNWB+QBrSOAbYHpgPvAv4FkD2BPAY8OmiFbvlb2YDTwq2mVDKAd/9gGsj4qbs+HeSTmTkkg6rgU8UrdjB38wGX1DKbh9SjK5e0G0VcFDu+AVg53YqNjMbaCXewP0RYO+qcxuB7XLH04D/Fq3Yff5mVgrDMdzSa8AsBI6SdFTu3NsDF5K2Aebi9fzNbFyKKGvL/1Xg98BSSb8hZf7MBJB0LHA68B7g20UrdvA3s4E3HDC0aeAyeVrxZ9LaPgGcApya/RzANcBm4NyIuLxoxQ7+ZjbwBGwzsZQt/+9Re2G3IdJA8O0R8UI7FTv4m1kJBMPDA9ef31REnNuruj3ga2alEMPR0qtVkhZIelrSJkkPSqq3nXWl/D6S7pD0pqTnJJ0jaTS3MyvEwd/MBl5Ed4O/pHnABcAPSROt7gVulvS+OuW3B24D1gGzgK8AZwFndPJ7STpV0stV2T5I+rqkFdnr1HbqdrePmQ28iGBoaHM3qzwDWBIRv86OvyzpaOA0amfWHA9MAU6IiDeBRyXtAZwh6fxofwbaUcAm4E+VE5K+CPwUeIMUwxdJWhsRNxSp2MHfzAbetpMmsPvM7ZoXBO5scl3SJGB/UoDNuxU4pM7bPgbclQX+iluAc0mpme2uwvkR4O6IyKcyLQDWAnsBE0kzfr8KjH3w/9sbr6yf88D1DTcPNjPLtL1JecVLLz5+y29/fsD0FotPlrQid3xpRFyaO55OCqrrqt63DjiiTp0zgGdrlK9cazf4zwCeqRxk3UuzgF9GxCvZuRtJC78V0pPgHxE79qJeM7NaIuLoXlRbdawa55qVr3W+iE2MTPU8hDRWe0fu3Hpgh6IVe8DXzGyk9aQlFGZUnd+JrZ8GKtbWKU+D97Tin0A+y+jY7N97qj7npaIVO/ibmeVExFvAg8CRVZeOJGX91HIfMFvS5Kryz5PrtmnDFcAsSTdKWgycBNwREetzZfYF1hSt2MHfzGxr5wPzJZ0saQ9JFwC7AIsAJP1I0u258leQsm+WSNpb0nGkDVc6yfQh+7zlpMXbTgReJqWQkn2PHUjdQlcXrdjZPjaCpNcjYmqX6zwG2DMifizpM8CTEfHXgnUsA86MiBXNypp1KiKuljQN+A5prfxHgbkRUUlk2RnYLVd+o6QjgYuAFaQgfR7pJtLJ9/iPpENIA81TSBlF63PXNzBybf+WqaQbIFibehH8q+pfAiyNiGsKvm8ZDv5mXeNuH6tJyUJJj0panc14RNJhkpZJukbSE5Iur0xhlzQ3O3e3pF9IWpqdny/pwqwFcwywUNJKSbtldR2QlZsu6Zns5+0kXSVplaSryW1eIemTku6T9JCkP0jq2c3KbCx5hq+NheOAj5ImmUwHlkuqzI/ZjzTB5HlS1sHHs7zpXwFzIuJpSVdWVxgR90q6gVzLv8HSJ6cBb0TEvpL2BR7Kyk8nPYofkT0Sf5M0G/P73filzfpMz2b4uuVv9RwKXBkRWyJiHSmveFZ27YGIeDYihoGVpBmMHwaeiojKZJatgn9Bc4DLACJiFWkWI8DBwJ7APZJWAifQhUlCZn2q0Qzf95I2cnmBNMO3ELf8rZ5GqxEO5X7eQvp/1O7qhZv5fyNkctW1WgNSAm6LiC+0+Xlmg6RnM3zd8rd67gTmSZooaUdSS/yBBuWfAD4oaWZ2PK9OudeAd+aOnyGtowLw2arPPx5A0t6kXGaA+0ndTLtn16ZI+lALv4/ZIPIMXxt115O6Wh4hbSV3dkSsrVc4W9BqAfBHSXeTZjVurFH0KuAsSQ9L2o3Ud3mapHtJYwsVlwBTJa0Czia78UTEv4H5wJXZtftJXU5mZdSzGb5O9bSukTQ1Il7Psn8uAtZExM/G+nuZDSpJ3wAWAjeRGlTzgTsj4vBcmfuBTRFxWJG63edv3XSKpBOAScDDpOwfM2vfIuBzpBm+ABvo0gxft/zNzPqYpInUmeHbUb0O/mZm448HfM3MxiEHfzOzccjB38xsHHLwNzMbhxz8zczGIQd/M7Nx6H/mkhR1Ve6FmQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "from sklearn.naive_bayes import GaussianNB\n",
    "model = GaussianNB()\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "plt.figure()\n",
    "ax = plt.gca()\n",
    "plot_classifier(X_train, y_train, model, ax=ax, proba=True);\n",
    "\n",
    "plt.ylabel(\"latitude\");\n",
    "plt.xlabel(\"longitude\");\n",
    "plt.title(\"Naive Bayes decision boundary\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Predictions in darker colours: the model is more confident\n",
    "- Predictions in lighter colours: the model is less confident"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 5. Naive Bayes on multi-class problems "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use `sklearn`'s inbuilt 20 newsgroups dataset. It comprises around 18,000 newsgroups posts on 20 topics. So it's a 20-class classification problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.datasets import fetch_20newsgroups\n",
    "\n",
    "newsgroups_train = fetch_20newsgroups(subset='train', remove=('headers','footers','quotes'))\n",
    "newsgroups_test = fetch_20newsgroups(subset='test', remove=('headers','footers','quotes'))\n",
    "X_news_train, y_news_train = newsgroups_train.data, newsgroups_train.target\n",
    "X_news_test, y_news_test = newsgroups_test.data, newsgroups_test.target"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are the 20 topics (targets or classes) in the dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['alt.atheism',\n",
       " 'comp.graphics',\n",
       " 'comp.os.ms-windows.misc',\n",
       " 'comp.sys.ibm.pc.hardware',\n",
       " 'comp.sys.mac.hardware',\n",
       " 'comp.windows.x',\n",
       " 'misc.forsale',\n",
       " 'rec.autos',\n",
       " 'rec.motorcycles',\n",
       " 'rec.sport.baseball',\n",
       " 'rec.sport.hockey',\n",
       " 'sci.crypt',\n",
       " 'sci.electronics',\n",
       " 'sci.med',\n",
       " 'sci.space',\n",
       " 'soc.religion.christian',\n",
       " 'talk.politics.guns',\n",
       " 'talk.politics.mideast',\n",
       " 'talk.politics.misc',\n",
       " 'talk.religion.misc']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "newsgroups_train.target_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Let's try `DummyClassifier`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.897788</td>\n",
       "      <td>0.195109</td>\n",
       "      <td>0.053027</td>\n",
       "      <td>0.053033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.878481</td>\n",
       "      <td>0.222103</td>\n",
       "      <td>0.053027</td>\n",
       "      <td>0.053033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.902842</td>\n",
       "      <td>0.201383</td>\n",
       "      <td>0.053027</td>\n",
       "      <td>0.053033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.916285</td>\n",
       "      <td>0.204620</td>\n",
       "      <td>0.053027</td>\n",
       "      <td>0.053033</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.930153</td>\n",
       "      <td>0.187737</td>\n",
       "      <td>0.053050</td>\n",
       "      <td>0.053027</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.897788    0.195109    0.053027     0.053033\n",
       "1  0.878481    0.222103    0.053027     0.053033\n",
       "2  0.902842    0.201383    0.053027     0.053033\n",
       "3  0.916285    0.204620    0.053027     0.053033\n",
       "4  0.930153    0.187737    0.053050     0.053027"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_multi = make_pipeline(CountVectorizer(), DummyClassifier(strategy=\"most_frequent\"))\n",
    "scores = cross_validate(pipe_multi, X_news_train, y_news_train, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The scores are very low because we have 20 different classes!! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Let's try `MultinomialNB`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "pipe_multi = make_pipeline(CountVectorizer(), MultinomialNB())\n",
    "scores = cross_validate(pipe_multi, X_news_train, y_news_train, return_train_score=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.971424</td>\n",
       "      <td>0.212912</td>\n",
       "      <td>0.583738</td>\n",
       "      <td>0.741465</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.943519</td>\n",
       "      <td>0.225378</td>\n",
       "      <td>0.570040</td>\n",
       "      <td>0.739587</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.952782</td>\n",
       "      <td>0.216721</td>\n",
       "      <td>0.583297</td>\n",
       "      <td>0.749309</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.974784</td>\n",
       "      <td>0.205032</td>\n",
       "      <td>0.571365</td>\n",
       "      <td>0.731963</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.963277</td>\n",
       "      <td>0.195200</td>\n",
       "      <td>0.588859</td>\n",
       "      <td>0.739063</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.971424    0.212912    0.583738     0.741465\n",
       "1  0.943519    0.225378    0.570040     0.739587\n",
       "2  0.952782    0.216721    0.583297     0.749309\n",
       "3  0.974784    0.205032    0.571365     0.731963\n",
       "4  0.963277    0.195200    0.588859     0.739063"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The validation scores are low but much better than the `DummyClassifier`.  \n",
    "\n",
    "Let's try to predict on an example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe_multi.fit(X_news_train, y_news_train); "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Hello All!\n",
      "\n",
      "    It is my understanding that all True-Type fonts in Windows are loaded in\n",
      "prior to starting Windows - this makes getting into Windows quite slow if you\n",
      "have hundreds of them as I do.  First off, am I correct in this thinking -\n",
      "secondly, if that is the case - can you get Windows to ignore them on boot and\n",
      "maybe make something like a PIF file to load them only when you enter the\n",
      "applications that need fonts?  Any ideas?\n",
      "\n",
      "\n",
      "Chris\n",
      "Prediction index: 5 and prediction class: comp.windows.x\n"
     ]
    }
   ],
   "source": [
    "print(X_news_test[100])\n",
    "prediction = pipe_multi.predict([X_news_test[100]])[0]\n",
    "print('Prediction index: %d and prediction class: %s' %(prediction, newsgroups_train.target_names[prediction]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And here are prediction probabilities. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[7.26904475e-19, 1.48743104e-10, 4.16538604e-28, 2.96340768e-10,\n",
       "        1.91391647e-14, 1.00000000e+00, 4.34194388e-28, 2.40916532e-18,\n",
       "        9.74009600e-22, 1.72603033e-31, 5.61184330e-31, 8.76853929e-14,\n",
       "        1.00705690e-16, 2.70331089e-13, 2.36762710e-21, 5.91222397e-16,\n",
       "        3.61589511e-15, 1.49715839e-13, 2.92789387e-16, 1.20413840e-24]])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_multi.predict_proba([X_news_test[100]])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Generative models\n",
    " \n",
    "- Naive Bayes is a **generative model** because it's modeling the joint distribution over the features $X$ and labels $y$.\n",
    "- In the next lecture, we'll look at a **discriminative model**, which models $P(y\\mid X)$ rather than $P(X,y)$.\n",
    "- Right now just get familiar with these terms. We'll talk more about them later in the program. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### General comments on naive Bayes\n",
    "\n",
    "- Surprising accuracy \n",
    "- A fast and robust way to learn the corresponding parameters\n",
    "- Scales great; learning a naive Bayes classifier is just a matter of counting how many times each attribute co-occurs with each class\n",
    "- Can be easily used for multi-class classification. \n",
    "- It's closely related to linear classifiers we'll see in the next lecture. \n",
    "    - When we take the logarithms, the products turn into summations. \n",
    "- Can provides a informative set of features from which to predict the class (next class)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### General comments on naive Bayes\n",
    "\n",
    "- Assumes that spammers generate e-mails by picking words at random. It means that sentences have no syntax and content. Is that a fair assumption? \n",
    "    - oversimplification \n",
    "    - sometimes the best theories are the most oversimplified, provided their predictions are accurate, because they explain the most with the least. \n",
    "\n",
    "- Although naive Bayes is known as a decent classifier, it is known to be a **bad estimator**, so the probability outputs from `predict_proba` are not to be taken too seriously.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### TRUE/False questions for class discussion\n",
    "\n",
    "> 1. Laplace smoothing helps us deal with unknown words, the words that do not occur in the vocabulary of the training set. \n",
    "\n",
    "Varada's answer: False\n",
    "\n",
    "> 2. Naive Bayes should only be used if the \"naive\" conditional independence assumption holds for your problem.\n",
    "\n",
    "Varada's answer: leaning towards False\n",
    "\n",
    "> 3. Naive Bayes should only be used when the features (i.e. `X`-values) are binary.\n",
    "\n",
    "Varada's answer: False.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:571]",
   "language": "python",
   "name": "conda-env-571-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}