ArchetypeAI.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "10e68289",
   "metadata": {},
   "source": [
    "# Analyzing Character Archetypes in TV Shows and Movies\n",
    "\n",
    "This project involves fine-tuning a pre-trained language model to classify character descriptions into archetypes such as \"hero,\" \"sidekick,\" \"mentor,\" and \"villain.\" Using Hugging Face's Transformers and a custom dataset, the model is trained to predict the archetype of a character based on a brief description. The project showcases text classification using NLP and can be deployed for users to classify new character descriptions automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7ccdd202",
   "metadata": {},
   "source": [
    "### Prototype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "ba6dfea4",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\lnewb\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\torchvision\\datapoints\\__init__.py:12: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().\n",
      "  warnings.warn(_BETA_TRANSFORMS_WARNING)\n",
      "C:\\Users\\lnewb\\AppData\\Local\\Programs\\Python\\Python311\\Lib\\site-packages\\torchvision\\transforms\\v2\\__init__.py:54: UserWarning: The torchvision.datapoints and torchvision.transforms.v2 namespaces are still Beta. While we do not expect major breaking changes, some APIs may still change according to user feedback. Please submit any feedback you may have in this issue: https://github.com/pytorch/vision/issues/6753, and you can also check out https://github.com/pytorch/vision/issues/7319 to learn more about the APIs that we suspect might involve future changes. You can silence this warning by calling torchvision.disable_beta_transforms_warning().\n",
      "  warnings.warn(_BETA_TRANSFORMS_WARNING)\n"
     ]
    }
   ],
   "source": [
    "from transformers import pipeline "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e4271466",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Device set to use cpu\n"
     ]
    }
   ],
   "source": [
    "archetypes = [\"Warrior\", \"Child\", \"Orphan\", \"Creator\", \"Caregiver\", \"Mentor\", \"Joker\", \"Magician\", \"Ruler\", \"Rebel\", \"Lover\", \"Seducer\"]\n",
    "\n",
    "classifier = pipeline(\"zero-shot-classification\", model=\"facebook/bart-large-mnli\")\n",
    "\n",
    "def prototype_classify_character(character_description):\n",
    "    result = classifier(character_description, candidate_labels=archetypes)\n",
    "    return result['labels'][0]  # Return the highest scoring archetype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b730e144",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Now try on an example\n",
    "character_description =\\\n",
    "\"\"\"\n",
    "Samwise Gamgee from The Lord of the Rings is a loyal, steadfast companion to Frodo Baggins. \n",
    "He begins as a humble gardener but proves to be one of the most courageous and selfless characters in the story. \n",
    "Despite being seemingly ordinary, Sam consistently demonstrates extraordinary bravery, especially when supporting \n",
    "Frodo on their perilous journey to destroy the One Ring. His deep sense of duty, unwavering friendship, and determination \n",
    "keep him by Frodo’s side, even in the darkest moments, making him a true hero in his own right, despite never seeking fame \n",
    "or glory.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "3fa01e3f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The character's archetype is: Caregiver\n"
     ]
    }
   ],
   "source": [
    "archetype = prototype_classify_character(character_description)\n",
    "print(f\"The character's archetype is: {archetype}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c80ed708",
   "metadata": {},
   "source": [
    "### Load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "92494c1b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.preprocessing import LabelEncoder\n",
    "import datasets "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "dc5f3fcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "arch_data = pd.read_csv(\"archetype_data.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1364d4fd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>description</th>\n",
       "      <th>archetype</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>A decorated general who leads an army into bat...</td>\n",
       "      <td>warrior</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>A skilled mercenary who takes on dangerous mis...</td>\n",
       "      <td>warrior</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>A young knight eager to prove themselves on th...</td>\n",
       "      <td>warrior</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>A seasoned veteran who commands a group of sol...</td>\n",
       "      <td>warrior</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>A fearless bounty hunter with a reputation for...</td>\n",
       "      <td>warrior</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                         description archetype\n",
       "0  A decorated general who leads an army into bat...   warrior\n",
       "1  A skilled mercenary who takes on dangerous mis...   warrior\n",
       "2  A young knight eager to prove themselves on th...   warrior\n",
       "3  A seasoned veteran who commands a group of sol...   warrior\n",
       "4  A fearless bounty hunter with a reputation for...   warrior"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arch_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d308357a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A decorated general who leads an army into battle, renowned for their tactical genius and unshakeable confidence.'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "arch_data['description'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "51834367",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Axes: xlabel='archetype'>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAh8AAAHkCAYAAACe4E2WAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjguMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy81sbWrAAAACXBIWXMAAA9hAAAPYQGoP6dpAABFF0lEQVR4nO3dd3RUZeLG8WdCKpCEACEQSKiR3gRRRBGBpQkisIKC0nHxh7QoTQE3qARxkbKL4FIFl6KuYGEX0CABQ0cQUFooJihN2BCKhEDe3x85GRkTItFw70z8fs6Zc8idm5kHyEyeufe97+swxhgBAABYxMvuAAAA4I+F8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsJS33QF+KSMjQz/88IMCAwPlcDjsjgMAAG6DMUYXL15UeHi4vLxyP7bhduXjhx9+UEREhN0xAADAb5CcnKxy5crluo/blY/AwEBJmeGDgoJsTgMAAG5HamqqIiIinL/Hc+N25SPrVEtQUBDlAwAAD3M7QyYYcAoAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKW+7A/xeFUavytfHOz7pkXx9PCn/M0r5n9MTMgIACgaOfAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApTx+enX8sTCdPgB4Po58AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYKk/lY9asWapTp46CgoIUFBSkxo0b67///a/z/qtXr2rQoEEqUaKEihYtqi5duuj06dP5HhoAAHiuPJWPcuXKadKkSdq5c6d27Nih5s2bq2PHjvrmm28kScOHD9cnn3yi999/X/Hx8frhhx/UuXPnOxIcAAB4pjxNMtahQweXr1977TXNmjVLW7ZsUbly5TRv3jwtWbJEzZs3lyQtWLBA1atX15YtW3TfffflX2oAAOCxfvOYjxs3bmjZsmW6fPmyGjdurJ07dyo9PV0tW7Z07lOtWjVFRkZq8+bNt3yctLQ0paamutwAAEDBlefp1ffu3avGjRvr6tWrKlq0qFasWKEaNWpo9+7d8vX1VbFixVz2DwsL06lTp275eLGxsYqJiclzcAC/nadMAc90+vnDEzJK/H/nF0/ImOcjH1WrVtXu3bu1detWPfvss+rVq5e+/fbb3xxgzJgxunDhgvOWnJz8mx8LAAC4vzwf+fD19VWVKlUkSQ0aNND27ds1ffp0devWTdeuXVNKSorL0Y/Tp0+rdOnSt3w8Pz8/+fn55T05AADwSL97no+MjAylpaWpQYMG8vHxUVxcnPO+gwcPKikpSY0bN/69TwMAAAqIPB35GDNmjNq2bavIyEhdvHhRS5Ys0fr167VmzRoFBwerX79+io6OVvHixRUUFKTBgwercePGXOkCAACc8lQ+zpw5o549e+rkyZMKDg5WnTp1tGbNGv3pT3+SJE2dOlVeXl7q0qWL0tLS1Lp1a7311lt3JDgAAPBMeSof8+bNy/V+f39/zZw5UzNnzvxdoQAAQMHF2i4AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYivIBAAAsRfkAAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYKk/lIzY2Vvfcc48CAwNVqlQpPfbYYzp48KDLPs2aNZPD4XC5DRw4MF9DAwAAz5Wn8hEfH69BgwZpy5Yt+uyzz5Senq5WrVrp8uXLLvsNGDBAJ0+edN4mT56cr6EBAIDn8s7LzqtXr3b5euHChSpVqpR27typpk2bOrcXLlxYpUuXzp+EAACgQPldYz4uXLggSSpevLjL9n/9618qWbKkatWqpTFjxujKlSu3fIy0tDSlpqa63AAAQMGVpyMfN8vIyNCwYcPUpEkT1apVy7m9e/fuKl++vMLDw7Vnzx6NGjVKBw8e1Icffpjj48TGxiomJua3xgAAAB7mN5ePQYMGad++ffryyy9dtj/zzDPOP9euXVtlypRRixYtdOTIEVWuXDnb44wZM0bR0dHOr1NTUxUREfFbYwEAADf3m8rHc889p08//VQbNmxQuXLlct333nvvlSQlJibmWD78/Pzk5+f3W2IAAAAPlKfyYYzR4MGDtWLFCq1fv14VK1b81e/ZvXu3JKlMmTK/KSAAAChY8lQ+Bg0apCVLluijjz5SYGCgTp06JUkKDg5WQECAjhw5oiVLlqhdu3YqUaKE9uzZo+HDh6tp06aqU6fOHfkLAAAAz5Kn8jFr1ixJmROJ3WzBggXq3bu3fH199fnnn2vatGm6fPmyIiIi1KVLF40dOzbfAgMAAM+W59MuuYmIiFB8fPzvCgQAAAo21nYBAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYivIBAAAsRfkAAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUnkqH7GxsbrnnnsUGBioUqVK6bHHHtPBgwdd9rl69aoGDRqkEiVKqGjRourSpYtOnz6dr6EBAIDnylP5iI+P16BBg7RlyxZ99tlnSk9PV6tWrXT58mXnPsOHD9cnn3yi999/X/Hx8frhhx/UuXPnfA8OAAA8k3dedl69erXL1wsXLlSpUqW0c+dONW3aVBcuXNC8efO0ZMkSNW/eXJK0YMECVa9eXVu2bNF9992Xf8kBAIBH+l1jPi5cuCBJKl68uCRp586dSk9PV8uWLZ37VKtWTZGRkdq8eXOOj5GWlqbU1FSXGwAAKLh+c/nIyMjQsGHD1KRJE9WqVUuSdOrUKfn6+qpYsWIu+4aFhenUqVM5Pk5sbKyCg4Odt4iIiN8aCQAAeIDfXD4GDRqkffv2admyZb8rwJgxY3ThwgXnLTk5+Xc9HgAAcG95GvOR5bnnntOnn36qDRs2qFy5cs7tpUuX1rVr15SSkuJy9OP06dMqXbp0jo/l5+cnPz+/3xIDAAB4oDwd+TDG6LnnntOKFSu0bt06VaxY0eX+Bg0ayMfHR3Fxcc5tBw8eVFJSkho3bpw/iQEAgEfL05GPQYMGacmSJfroo48UGBjoHMcRHBysgIAABQcHq1+/foqOjlbx4sUVFBSkwYMHq3HjxlzpAgAAJOWxfMyaNUuS1KxZM5ftCxYsUO/evSVJU6dOlZeXl7p06aK0tDS1bt1ab731Vr6EBQAAni9P5cMY86v7+Pv7a+bMmZo5c+ZvDgUAAAou1nYBAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYivIBAAAsRfkAAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUnkuHxs2bFCHDh0UHh4uh8OhlStXutzfu3dvORwOl1ubNm3yKy8AAPBweS4fly9fVt26dTVz5sxb7tOmTRudPHnSeVu6dOnvCgkAAAoO77x+Q9u2bdW2bdtc9/Hz81Pp0qV/cygAAFBw3ZExH+vXr1epUqVUtWpVPfvsszp37twt901LS1NqaqrLDQAAFFz5Xj7atGmjRYsWKS4uTq+//rri4+PVtm1b3bhxI8f9Y2NjFRwc7LxFRETkdyQAAOBG8nza5dc88cQTzj/Xrl1bderUUeXKlbV+/Xq1aNEi2/5jxoxRdHS08+vU1FQKCAAABdgdv9S2UqVKKlmypBITE3O838/PT0FBQS43AABQcN3x8nHixAmdO3dOZcqUudNPBQAAPECeT7tcunTJ5SjGsWPHtHv3bhUvXlzFixdXTEyMunTpotKlS+vIkSMaOXKkqlSpotatW+drcAAA4JnyXD527Nihhx9+2Pl11niNXr16adasWdqzZ4/eeecdpaSkKDw8XK1atdIrr7wiPz+//EsNAAA8Vp7LR7NmzWSMueX9a9as+V2BAABAwcbaLgAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYivIBAAAsRfkAAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYivIBAAAsRfkAAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFgqz+Vjw4YN6tChg8LDw+VwOLRy5UqX+40xGj9+vMqUKaOAgAC1bNlShw8fzq+8AADAw+W5fFy+fFl169bVzJkzc7x/8uTJmjFjhmbPnq2tW7eqSJEiat26ta5evfq7wwIAAM/nnddvaNu2rdq2bZvjfcYYTZs2TWPHjlXHjh0lSYsWLVJYWJhWrlypJ5544velBQAAHi9fx3wcO3ZMp06dUsuWLZ3bgoODde+992rz5s05fk9aWppSU1NdbgAAoODK1/Jx6tQpSVJYWJjL9rCwMOd9vxQbG6vg4GDnLSIiIj8jAQAAN2P71S5jxozRhQsXnLfk5GS7IwEAgDsoX8tH6dKlJUmnT5922X769Gnnfb/k5+enoKAglxsAACi48rV8VKxYUaVLl1ZcXJxzW2pqqrZu3arGjRvn51MBAAAPleerXS5duqTExETn18eOHdPu3btVvHhxRUZGatiwYXr11VcVFRWlihUraty4cQoPD9djjz2Wn7kBAICHynP52LFjhx5++GHn19HR0ZKkXr16aeHChRo5cqQuX76sZ555RikpKXrggQe0evVq+fv7519qAADgsfJcPpo1ayZjzC3vdzgcmjBhgiZMmPC7ggEAgILJ9qtdAADAHwvlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYivIBAAAsRfkAAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYivIBAAAsRfkAAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJbK9/Lx17/+VQ6Hw+VWrVq1/H4aAADgobzvxIPWrFlTn3/++c9P4n1HngYAAHigO9IKvL29Vbp06Tvx0AAAwMPdkTEfhw8fVnh4uCpVqqQePXooKSnplvumpaUpNTXV5QYAAAqufC8f9957rxYuXKjVq1dr1qxZOnbsmB588EFdvHgxx/1jY2MVHBzsvEVEROR3JAAA4EbyvXy0bdtWjz/+uOrUqaPWrVvrP//5j1JSUvTee+/luP+YMWN04cIF5y05OTm/IwEAADdyx0eCFitWTHfddZcSExNzvN/Pz09+fn53OgYAAHATd3yej0uXLunIkSMqU6bMnX4qAADgAfK9fLzwwguKj4/X8ePHtWnTJnXq1EmFChXSk08+md9PBQAAPFC+n3Y5ceKEnnzySZ07d06hoaF64IEHtGXLFoWGhub3UwEAAA+U7+Vj2bJl+f2QAACgAGFtFwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYivIBAAAsRfkAAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACxF+QAAAJaifAAAAEtRPgAAgKUoHwAAwFKUDwAAYCnKBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACwFOUDAABYivIBAAAsRfkAAACWonwAAABLUT4AAIClKB8AAMBSlA8AAGApygcAALAU5QMAAFiK8gEAACx1x8rHzJkzVaFCBfn7++vee+/Vtm3b7tRTAQAAD3JHysfy5csVHR2tl19+WV999ZXq1q2r1q1b68yZM3fi6QAAgAe5I+XjzTff1IABA9SnTx/VqFFDs2fPVuHChTV//vw78XQAAMCDeOf3A167dk07d+7UmDFjnNu8vLzUsmVLbd68Odv+aWlpSktLc3594cIFSVJqauptPV9G2pXfmdjV7T5vXuR3Rin/c3pCRon/7/ziCRkl/r/ziydklPj/zi92Zczaxxjz6w9o8tn3339vJJlNmza5bB8xYoRp1KhRtv1ffvllI4kbN27cuHHjVgBuycnJv9oV8v3IR16NGTNG0dHRzq8zMjJ0/vx5lShRQg6HI1+eIzU1VREREUpOTlZQUFC+PGZ+84SMkmfkJGP+8YScZMw/npCTjPknv3MaY3Tx4kWFh4f/6r75Xj5KliypQoUK6fTp0y7bT58+rdKlS2fb38/PT35+fi7bihUrlt+xJElBQUFu/YMgeUZGyTNykjH/eEJOMuYfT8hJxvyTnzmDg4Nva798H3Dq6+urBg0aKC4uzrktIyNDcXFxaty4cX4/HQAA8DB35LRLdHS0evXqpYYNG6pRo0aaNm2aLl++rD59+tyJpwMAAB7kjpSPbt266ezZsxo/frxOnTqlevXqafXq1QoLC7sTT/er/Pz89PLLL2c7veNOPCGj5Bk5yZh/PCEnGfOPJ+QkY/6xM6fDmNu5JgYAACB/sLYLAACwFOUDAABYivIBAAAsRfkAAACWKnDl4/r161q0aFG2Sc4AOxljlJSUpKtXr9odBQBsV+DKh7e3twYOHMibfD64fv26JkyYoBMnTtgdJVfp6eny9vbWvn377I5yS8YYValSRcnJyXZHuSWKOwCr2L62y53QqFEj7d69W+XLl7c7Sq6uX7+uiRMnqm/fvipXrpzdcbLx9vbWG2+8oZ49e9odJVc+Pj6KjIzUjRs37I5yS15eXoqKitK5c+cUFRVld5wcZRX3/fv32x0lR507d77tfT/88MM7mCRvrl27pjNnzigjI8Nle2RkpE2Jfpaenq5q1arp008/VfXq1e2O49HS09P1l7/8RePGjVPFihXtjuP2CmT5+L//+z9FR0crOTlZDRo0UJEiRVzur1Onjk3JXHnCL/fmzZsrPj5eFSpUsDtKrl566SW9+OKLWrx4sYoXL253nBxNmjRJI0aM0KxZs1SrVi274+TInYv7zWtGGGO0YsUKBQcHq2HDhpKknTt3KiUlJU8l5U46fPiw+vbtq02bNrlsN8bI4XC4RVn28fHhKHE+8fHx0b///W+NGzfO7ii/Ki4uTnFxcTmW4vnz51uSoUBOMubllf1sksPhcKsXfZaOHTuqc+fO6tWrl91RcjR79mzFxMSoR48eORa5Rx991KZkrurXr6/ExESlp6erfPny2XJ+9dVXNiX7WUhIiK5cuaLr16/L19dXAQEBLvefP3/epmQ/e++99zRmzBgNHz7crYv7qFGjdP78ec2ePVuFChWSJN24cUP/93//p6CgIL3xxhs2J5SaNGkib29vjR49WmXKlMm2SnfdunVtSuZq4sSJOnTokObOnStvb/f5PBoSEnLbK5u7w2tHknr16qV69epp+PDhdke5pZiYGE2YMEENGzbM8edyxYoVluQokOXju+++y/V+d/pU5+6/3HMqclncqcjFxMTkev/LL79sUZJbe+edd3K93x0KqKcU99DQUH355ZeqWrWqy/aDBw/q/vvv17lz52xK9rMiRYpo586dqlatmt1RctWpUyfFxcWpaNGiql27drb3ILtOYf3a6+Vm7vDakaRXX31VU6ZMUYsWLXJ8Px8yZIhNyX5WpkwZTZ48WU8//bStOQpk+fAknvLLHX8MnlLcQ0JCtHDhQnXs2NFl+0cffaTevXvrf//7n03JfnbPPfdo6tSpeuCBB+yOkqtfW/BzwYIFFiXxfLmN9XA4HDp69KiFaXJWokQJbdu2TZUrV7Y1R4EtH4sXL9bs2bN17Ngxbd68WeXLl9e0adNUsWLFbG9YuD1Xr16Vv7+/3TFytXPnTueAyZo1a6p+/fo2J3J15MgRLViwQEeOHNH06dNVqlQp/fe//1VkZKRq1qxpdzyPER0drUWLFunFF19Uo0aNJElbt27VpEmT9PTTT+vNN9+0OaG0bt06jR07VhMnTlTt2rXl4+Pjcn9QUJBNyTwTr538MWrUKBUtWtT+sSmmAHrrrbdMyZIlzauvvmoCAgLMkSNHjDHGLFiwwDRr1szmdJ7l+vXrZsKECSY8PNwUKlTI+W85duxYM3fuXJvT/ez06dPm4YcfNg6Hw4SEhJiQkBDjcDhM8+bNzZkzZ+yOZ4wxZv369SYgIMC0bNnS+Pr6Ov8tY2NjTZcuXWxO97PExETz3HPPmRYtWpgWLVqYwYMHm8TERLtjubhx44Z5/fXXTXh4uHE4HMbhcJjw8HDz+uuvm+vXr9sdzxhjnLm8vLxcblnb3El6err57LPPzOzZs01qaqoxxpjvv//eXLx40eZkmTzltZMlLS3NHDhwwKSnp9sdJZshQ4aYYsWKmaZNm5rnnnvODB8+3OVmlQJZPqpXr25WrFhhjDGmaNGizh/UvXv3mhIlStiYLGfr16837du3N5UrVzaVK1c2HTp0MBs2bLA7ljHGmJiYGFOpUiXz7rvvuhS5ZcuWmfvuu8/mdD/r2rWradiwofn222+d27755hvTsGFD88QTT9iY7Gf33XefmTJlijHG9edy69atpmzZsnZGc1q9erXx9fU1jRo1cr4ZNWrUyPj5+Zm1a9faHS9HFy5cMBcuXLA7Rjbr16/P9eYujh8/bqpVq2YKFy7s8gFjyJAh5i9/+YvN6TJ5wmvHGGMuX75s+vbtawoVKuTyb/ncc8+Z2NhYm9Nlatas2S1vDz/8sGU5CmT58Pf3N8ePHzfGuP6gHjp0yPj7+9sZLZvFixcbb29v07VrVzN9+nQzffp007VrV+Pj42P+9a9/2R3PVK5c2Xz++efGGNd/y/3795tixYrZGc1FUFCQ2bZtW7btW7duNcHBwdYHykGRIkXM0aNHjTGu/5bHjh0zfn5+dkZzqlevnhk1alS27aNGjTL169e3IRHutI4dO5qnnnrKpKWlufxcfvHFF6ZKlSo2p8vkCa8dYzILW4MGDczGjRtNkSJFnDlXrlxp6tWrZ3M69+I+11Xlo4oVK+Y4V8Hq1avdbiKd1157TZMnT3a5NGvIkCF688039corr6h79+42ppO+//57ValSJdv2jIwMpaen25AoZxkZGdnOqUuZ197/8jp2uxQrVkwnT57MNiht165dKlu2rE2pXO3fv1/vvfdetu19+/bVtGnTrA90k/r169/2pZfucGl1litXrigpKUnXrl1z2e4uly1v3LhRmzZtkq+vr8v2ChUq6Pvvv7cplStPeO1I0sqVK7V8+XLdd999Lj+rNWvW1JEjR2xM5n4KZPmIjo7WoEGDdPXqVRljtG3bNi1dulSxsbGaO3eu3fFcHD16VB06dMi2/dFHH9WLL75oQyJXNWrU0MaNG7MVuQ8++MCtBnM2b95cQ4cO1dKlSxUeHi4pszgNHz5cLVq0sDldpieeeEKjRo3S+++/L4fDoYyMDCUkJOiFF15wm4nmQkNDtXv37myzsO7evVulSpWyKVWmxx57zNbnz6uzZ8+qT58++u9//5vj/e5yJVtGRkaOWU6cOKHAwEAbEmXnCa8dKfP/PKfXyeXLl2+7OFthx44deu+993IsxZZdWm33oZc75d133zVVqlRxDvoqW7asWw2QzFK5cmUze/bsbNtnzZrlFoc8V65caYKDg82kSZNM4cKFzRtvvGH69+9vfH193WoMQFJSkqlXr57x8fExlSpVMpUqVTI+Pj6mfv36Jjk52e54xpjMQWj9+/c33t7exuFwGB8fH+Pl5WWeeuoptxkkGRMTY4oVK2YmTZpkNmzYYDZs2GBiY2NNsWLFzIQJE+yO51G6d+9umjRpYrZv326KFCli1q5daxYvXmyqVq1qPv30U7vjOXXt2tUMGDDAGJN5SuPo0aPm4sWLpnnz5qZ37942p8vkCa8dY4x58MEHzYwZM4wxP/9bGpM55qN169Z2RnNaunSp8fHxMe3btze+vr6mffv25q677jLBwcGW/n8XyPJx8+Czy5cvm9OnTzu/Pnz4sB2Rbumtt94yvr6+ZuDAgWbRokVm0aJF5i9/+Yvx8/PLsZTYYcOGDaZly5YmNDTUBAQEmCZNmpg1a9bYHSubjIwMs3btWjNjxgwzY8YM89lnn9kdKUdJSUlm1apVZvny5ebQoUN2x3GRkZFh3nzzTVO2bFmX4j5t2jSTkZFhdzyPUrp0abN161ZjjDGBgYHm4MGDxhhjPvroI9OkSRM7o7lITk42NWrUMNWrVzfe3t7mvvvuMyVKlDBVq1Z1ee90B999953bvnaMMWbjxo2maNGiZuDAgcbf398MHTrU/OlPfzJFihQxO3bssDueMcaY2rVrm3/84x/GmJ/Hz2RkZJgBAwaY8ePHW5ajQJaPBx54wFy9ejXb9gMHDrjVyOgsH374oWnSpIkpXry4KV68uGnSpIlZuXKl3bE8yjvvvJPj/3laWpp55513bEjk+VJTU52XXbqDkJAQc/bsWWOMMcWKFXNeUp3TzR0EBgaaY8eOGWOMiYyMNF9++aUxxpijR4+agIAAG5Nll56ebhYvXmxGjBhhnn32WTNnzhxz5coVu2PlKCMjw62LcGJiounfv7+55557TPXq1U2PHj3Mnj177I7lVLhwYefPZfHixZ3Zvv32W1O6dGnLchTIMR9FixZVp06d9PHHHzvXKti/f7+aN2+url272pwuu06dOqlTp052x8hR//799dRTT6lZs2Z2R8lVnz591KZNm2znWy9evKg+ffq4xXnhLl26qFGjRho1apTL9smTJ2v79u16//33bUqWM3c5359l6tSpzkx2D369HVWrVtXBgwdVoUIF1a1bV2+//bYqVKig2bNnq0yZMnbHc8qaPPCpp56yO0qu5s2bp6lTp+rw4cOSpKioKA0bNkz9+/e3OZmrypUra86cOXbHuKWQkBBdvHhRklS2bFnt27dPtWvXVkpKiq5cuWJdEMtqjoWuXLli7r//ftO1a1eTkZFh9u7da0qVKmXpBCq3q2LFiubHH3/Mtv1///ufqVixog2JXD366KPGz8/PlCtXzrzwwgtm165ddkfKkcPhyHEysd27d7vNJ+GSJUvm+Aloz549plSpUjYkylS/fn1z/vx5Y0zmpbb169e/5Q23b/HixWbBggXGGGN27NhhSpYsaby8vIy/v79ZtmyZveFuEhgYaHr27GnWrl1rbty4YXecHI0bN84UKVLEjB492nz00Ufmo48+MqNHjzZFixY148aNszVb1jwzt3NzB08++aRzzpQJEyaY0NBQ079/f1O+fHnTqVMny3IU2OnVU1JS1KxZM0VFRWnDhg3q2bOnW6x0+UteXl46depUtk/sp0+fVmRkpNLS0mxK9rP//e9/ev/997VkyRJt3LhR1apVU48ePdS9e3dVqFDB1mxZl19+/fXXqlmzpsuqnDdu3NCxY8fUpk2bHC8ftVpAQIB2796dbTG0AwcOqH79+vrpp59syRUTE6MRI0aocOHCHrFAX5aMjAwlJibmuCx406ZNbUp1a1euXNGBAwcUGRmpkiVL2h3HacWKFVqyZIlWrVql4OBgdevWTU899ZQaNmxodzSn0NBQzZgxQ08++aTL9qVLl2rw4MH68ccfbUqW+R7+a1eyGDdamPH8+fO6evWqwsPDlZGRocmTJ2vTpk2KiorS2LFjFRISYkmOAlM+UlNTs207efKk/vSnP6l9+/aaNGmSc7s7rKnw8ccfS8q8fPCdd95RcHCw874bN24oLi5On332mQ4ePGhXxBydOHFCS5cu1fz583X48GFdv37d1jxZvyxjYmL0/PPPq2jRos77fH19VaFCBXXp0iXbHAZ2aNSokdq3b6/x48e7bP/rX/+qTz75RDt37rQpWaYbN24oISFBderUUbFixWzN8mu2bNmi7t2767vvvtMv38Lc5U3e01y8eFEffPCBli5dqnXr1qlSpUp66qmnsv282qFYsWLavn17tkvADx06pEaNGiklJcWeYJLi4+Nve9+HHnroDibxLAWmfNyqfWb99dxtWfCs1Wyzct3Mx8dHFSpU0JQpU9S+fXs74uUoPT1dq1at0rvvvqtVq1apePHibjMJ0TvvvKNu3bq59cJ3n3zyiTp37qzu3burefPmkqS4uDgtXbpU77//vlvMY+Hv76/9+/fnujqnO6hXr57uuusuxcTEqEyZMtle+zeXebvcuHFDCxcuVFxcXI5HZ9atW2dTsl/37bffqkePHtqzZ49bvF8OHjxYPj4+2RYMfOGFF/TTTz9p5syZNiXzDKmpqc4P3Tl9UL+ZVR/OC8yA0y+++MLuCHmS9UZUsWJFbd++3a0Ow/7SF198oSVLlujf//63MjIy1LlzZ3366afOX6DuoFevXnZH+FUdOnTQypUrNXHiRH3wwQcKCAhQnTp19Pnnn7vNJ6JatWrp6NGjbl8+Dh8+rA8++CDH2XfdxdChQ7Vw4UI98sgjqlWrlltNMpWTq1ev6uOPP9aSJUu0evVqhYWFacSIEbbliY6Odv7Z4XBo7ty5Wrt2re677z5JmasYJyUlucVg8iwbNmzI9X67TgeGhITo5MmTKlWqlIoVK3bLD+pWfjgvMEc+cGeULVtW58+fV5s2bdSjRw916NBBfn5+dsfK5saNG5o6deotZ+07f/68Tck8y+rVqzVmzBi98soratCggYoUKeJyvzucspQyZ7QdOXKk2rRpY3eUWypZsqQWLVqkdu3a2R0lV2vWrNGSJUu0cuVKeXt7689//rN69Ohh+7iZhx9++Lb2czgcbnMUKeuI9s1u/kVv11Gk+Ph4NWnSRN7e3r96msiqD0IFtnykpKRo3rx52r9/v6TMufX79u3rFodjfykuLu6Wh2bnz59vU6pMc+bM0eOPP+72YwDGjx+vuXPn6vnnn9fYsWP10ksv6fjx41q5cqXGjx+vIUOG2B3RI9z85nnzm6Y7nLLcs2eP889HjhzR2LFjNWLECNWuXTvbuj7usG5KeHi41q9fr7vuusvuKLkqXLiw2rdvrx49eqhdu3Y5rpGE23PhwgWXr9PT07Vr1y6NGzdOr732mtss9eAOCmT52LFjh1q3bq2AgAA1atRIkrR9+3b99NNPWrt2re6++26bE/4sJiZGEyZMUMOGDXM8d71ixQqbknmWypUra8aMGXrkkUcUGBio3bt3O7dt2bJFS5YssTvir46Kd4dz6+7yqSgnWf9+t3rLcrdxXVOmTNHRo0f1j3/8w61PuVy8eNHt5nS5lcTERB05ckRNmzZVQECA8//b3cXHxys6Otr2QeWStGDBAhUtWlSPP/64y/b3339fV65csewUdoEsHw8++KCqVKmiOXPmOC+9vH79uvr376+jR4/+6nk5K5UpU0aTJ0/W008/bXcUp86dO2vhwoUKCgpS586dc93XskWIfkWRIkW0f/9+RUZGqkyZMlq1apXuvvtuHT16VPXr18/2icQOH330kcvXWZ+K3nnnHcXExKhfv342Jfs5T5s2bTR79uxsVxW4g+++++629/3lQohW+eXrZd26dSpevLhq1qyZ7YiCu7x2pMziu3LlSueR4ho1aqhjx44qVKiQzckynTt3Tl27dtUXX3whh8Ohw4cPq1KlSurbt69CQkI0ZcoUuyPm6sCBA2rYsKEuXbpkdxTdddddevvtt7Od1oqPj9czzzxj2RWWBWbA6c127NjhUjwkydvbWyNHjnSra9cl6dq1a7r//vvtjuEiODjY+WnCHU9T5aRcuXI6efKkIiMjVblyZecRru3bt7vNGJWOHTtm2/bnP/9ZNWvW1PLly20vHz4+Pi6nNtzNzYUiNjZWYWFh6tu3r8s+8+fP19mzZ7PNImuVX75e3HXm4pslJiaqXbt2+v77751z0MTGxioiIkKrVq1S5cqVbU4oDR8+XD4+PkpKSlL16tWd27t166bo6Gi3KR+/fP0YY3Ty5ElNmjRJ9erVsyfULyQlJeU4oLx8+fJKSkqyLohl05lZqFSpUjkufLZ69WpbZ5LMyciRI1ktNB+MGjXKvPbaa8YYY5YtW2a8vb1NlSpVjK+vrxk1apTN6XJ35MgRU6RIEbtjGGOMGTZsmNv/exljTPny5U1CQkK27Vu2bDEVKlSwIZHnatu2rWnTpo05d+6cc9uPP/5o2rRpY9q1a2djsp+FhYWZ3bt3G2N+XgzNGPd67RiTOdOyl5eXc1HGrFvjxo3N/v377Y5njDEmIiLCfPTRR9m2r1y50tK1zwrkkY9u3bqpX79++tvf/uY8qpCQkKARI0ZkmyHPblevXtU///lPff7556pTp062Q7O/vK4dObt5Erlu3bopMjJSmzdvVlRUlDp06GBjstz99NNPmjFjhsqWLWt3FEmZpyfnz5+vzz//PMerXdzl5/HUqVM5ro8SGhqqkydP2pDo1s6cOeM8lF21atVssxnbLT4+Xlu2bFHx4sWd20qUKKFJkyapSZMmNib72eXLl1W4cOFs28+fP+82RzbT09PVrFkzzZ4925nJy8tLoaGhbjX/0JNPPqkhQ4YoMDDQeUVTfHy8hg4dqieeeMKyHAWyfPztb3+Tw+FQz549df36dRlj5Ovrq2effdbll5Q72LNnj/Nw3L59+1zuc4eBVFnTl/+Sw+GQv7+/qlSpot69e9/2ZXFWady4sRo3bmx3DBchISHZriC5ePGiChcurHfffdfGZD/bt2+fc0D2oUOHXO5zh5/HLBEREUpISMh2+DghIUHh4eE2pXKVmpqqQYMGadmyZc4BsIUKFVK3bt00c+ZMtzml6efn51xo7GaXLl1yi5mBpcxxfIsWLdIrr7wiKfNnMWtqcHd57/Hx8dHevXvl5eVl25ij2/HKK6/o+PHjatGihXNoQkZGhnr27KmJEydalqNADjjNcuXKFR05ckRS5tUQOTVn5G7MmDGaNWuWateu7XLl0J49e9S7d299++23iouL04cffpjjmAYrLV68WLNnz9axY8e0efNmlS9fXtOmTVPFihVtzyZJCxcudPkFnvWp6N5777VsPYWCYvLkyZo8ebLeeOMNl9liR44cqeeff15jxoyxOWHmEbhdu3bp73//u7MIb968WUOHDlW9evW0bNkymxNm6tmzp7766ivNmzfP+RrfunWrBgwYoAYNGmjhwoX2BpT0zTffqHnz5rr77ru1bt06Pfroo/rmm290/vx5JSQkuMW4FClzbIqfn5/bfcjNyaFDh/T1118rICBAtWvXtrwwFZjykZcrNIoWLaqaNWtq4MCBbvPpw10vIRswYIAiIyM1btw4l+2vvvqqvvvuO82ZM0cvv/yyVq1apR07dtiUUpo1a5bGjx+vYcOG6bXXXtO+fftUqVIlLVy4UO+8847HzYCL3BljNHr0aM2YMcM5oZy/v79GjRrlFmuRSJlXYK1Zs0YPPPCAy/aNGzeqTZs2unz5sk3JXKWkpKhXr1765JNPnKd909PT1bFjRy1YsMD2OX6yrsKKjY3VZ599pq+//lqXLl3S3XffrUGDBuV4+s0ugwcP1qJFixQVFeXWpy3dQYEpH3369NGMGTMUGBioPn365LpvWlqaNm/erNq1azsXeLOLu19CFhwcrJ07d2abxjoxMVENGjTQhQsXdODAAd1zzz05Hrq1So0aNTRx4kQ99thjCgwM1Ndff61KlSpp3759atasmW2rXubl6hF3mBhLyrxa7FYzxbrT5aFS5qmB/fv3KyAgQFFRUW5z/l+SIiMjtWrVKtWuXdtl+549e9SuXTudOHHCpmQ5S0xMdF5qW716dbeauj40NNS58qo7y+0UkJ0zsUZHR+uVV15RkSJFXKatz4lVBanAjPlYsGBBjn++lW+//Vb33HPPnYx0W9z9EjJ/f39t2rQp2xvRpk2bnIOoMjIybB9QdezYMdWvXz/bdj8/P1s/YdarV89lYix3n2Rs2bJl6tmzp1q3bq21a9eqVatWOnTokE6fPu2Wl40WLVrULV7HORk7dqyio6O1ePFilS5dWlLmQNkRI0ZkO5JotV/7BXTzkUJ3+LT+1FNPad68eW5/OsNdj7Du2rVL6enpzj/fipVH2wtM+cirqlWratOmTXbH0Nq1a7VmzRqVK1fOZXtUVFSeJlW6UwYPHqyBAwdq586dzjf57du3a+7cuXrxxRclZa4NYfc17BUrVtTu3buznbdcvXq1S6mz2rFjx5x/3rVrl1544QWNGDHCZQzAlClTNHnyZLsiupg4caKmTp2qQYMGKTAwUNOnT1fFihX1l7/8xa0Ob3uCWbNmKTExUZGRkYqMjJSUOceCn5+fzp49q7ffftu571dffWVpttx+Ad3MHU79Sp5zFZa7urkUuUtB+sOWj0KFCqlu3bp2x3D7S8jGjh2rihUr6h//+IcWL14sKbO4zZkzR927d5ckDRw4UM8++6ydMRUdHa1Bgwbp6tWrMsZo27ZtWrp0qWJjYzV37lzbct1chh5//HHNmDHDZaGxOnXqKCIiQuPGjdNjjz1mQ0JXR44c0SOPPCJJ8vX11eXLl+VwODR8+HA1b95cMTExNif0HO7w/3kr7vIL6HZ5ylVYuH1/2PLhLjzhErIePXqoR48et7w/ICDAwjQ569+/vwICAjR27FhduXJF3bt3V3h4uKZPn27pteu52bt3b44zC1asWFHffvutDYmyCwkJcY7dKVu2rPbt26fatWsrJSVFV65csTmd57hx44Yefvhh1alTx/YBmwWBp5Uld9apU6dfnT6he/fuztlu75Ts6//CUpMnT9Y///lPtW3bVteuXdPIkSNVq1YtbdiwQa+//rrd8SRljobPOs2StTT9V199pe+//97mZJmuX7+uRYsWqWXLljp8+LAuXbqkU6dO6cSJE7ZPWX6z6tWrKzY21mUQ57Vr1xQbG2vrqaGbNW3aVJ999pmkzCM1Q4cO1YABA/Tkk0+yImceFCpUSK1atdL//vc/u6MALoKDg7Vu3Tp99dVXcjgccjgc2rVrl9atW6fr169r+fLlqlu3rhISEu5ojgJztYsnS0lJ0cyZM93yErI9e/aoZcuWCg4O1vHjx3Xw4EFVqlRJY8eOVVJSkhYtWmR3REmZy4Lv37/frSf32bZtmzp06CBjjPPKlj179sjhcOiTTz5xzrFgp/Pnz+vq1asKDw93HoHLuspg7NixzEeSBw0bNtTrr79OaYNbGT16tFJTU/WPf/xDXl6Zxx8yMjI0dOhQBQYG6rXXXtPAgQP1zTff6Msvv7xjOSgfbuDq1avas2ePzpw5o4yMDJf7Hn30UZtSZWrZsqXuvvtuTZ482eUS1k2bNql79+46fvy4rfmyNGvWTMOGDXPr8+xS5hiff/3rXzpw4ICkzKMh3bt3zzaAzi49e/bUww8/rKZNm7rNxE2eavXq1RozZoxeeeWVHAdJBgUF2ZQMf2ShoaFKSEjQXXfd5bL90KFDuv/++/Xjjz9q7969evDBB5WSknLHcjDmw2arV6/W008/rfPnz+uXPdDhcNh++eX27dtdRuVnKVu2rE6dOmVDopz93//9n55//nmdOHEixzd6d5lDo0iRInrmmWfsjnFLvr6+io2NVb9+/VS2bFk99NBDatasmR566CG3n2PB3WQNLH700UezTavvDq9t/DFdv35dBw4cyFY+Dhw44PyZ9Pf3v+MDeSkfNhs8eLC6du2q8ePHKywszO442fj5+Sk1NTXb9kOHDik0NNSGRDnLGlQ6ZMgQ57as+TXc6Y1+8eLFevvtt3X06FHnFPBTp05VpUqV3GIK+Kwrg77//ntt2LBB8fHxmjJlivNSW3ebGMudMUgS7ujpp59Wv3799OKLL7pMnzBx4kT17NlTUuZCczVr1ryjOSgfNjt9+rSio6PdsnhImZ/aJkyYoPfee09S5i/0pKQkjRo1Sl26dLE53c9unk/DXd08Bfyrr77qLEQhISGaNm2aW5SPLCEhISpRooRCQkJUrFgxeXt7u1XZ9AQPPfSQUlJSNG/ePOfMoTVq1FC/fv3cZlkH/PFMnTpVYWFhmjx5sk6fPi1JCgsL0/DhwzVq1ChJUqtWrdSmTZs7moMxHzbr27evmjRp4lZXZdzswoUL+vOf/6wdO3bo4sWLCg8P16lTp9S4cWP95z//cZuxCrGxsQoLC1Pfvn1dts+fP19nz551vqjs5K5TwN/sxRdf1Pr167Vr1y5Vr17dedqladOmDDbNox07dqhNmzby9/d3WZTxp59+0tq1a53zVgB2yTqqbcf4I8qHza5cuaLHH39coaGhql27tnNhpyw3n0awU0JCgsvVOC1btrQ7kosKFSpoyZIluv/++122b926VU888YRbHBkJCAjQgQMHVL58eZfycfjwYdWpU0c//fST3RGdK+0OHz5cnTt3znZeGLfvwQcfVJUqVTRnzhzn0uXXr19X//79dfToUW3YsMHmhIB9OO1is6VLl2rt2rXy9/fX+vXrXQb5OBwOW8tHenq6AgICtHv3bjVp0kRNmjSxLcuvOXXqVI6XJoeGhurkyZM2JMrOXaeAv9muXbsUHx+v9evXa8qUKfL19XUe/WjWrBllJA927NjhUjwkydvbWyNHjlTDhg1tTIY/ug8++OCWi0daNdU/k4zZ7KWXXlJMTIwuXLig48eP69ixY87b0aNHbc3m4+OjyMhItxmsmZuIiIgcJ8VJSEhQeHi4DYmyy5oCfvny5c4p4F977TWNGTNGI0eOtDueJKlu3boaMmSIPvzwQ509e1b/+c9/5Ovrq0GDBrlNQfIUQUFBSkpKyrY9OTlZgYGBNiQCpBkzZqhPnz4KCwvTrl271KhRI5UoUUJHjx5V27ZtrQtiYKuQkBCTmJhod4xbmjt3rmnXrp05d+6c3VFy9frrr5sSJUqY+fPnm+PHj5vjx4+befPmmRIlSpiJEyfaHc/p3XffNVWqVDEOh8M4HA5Trlw5M3fuXLtjOWVkZJidO3eaKVOmmA4dOpiQkBBTqFAhU79+fTNs2DC743mUwYMHm3Llyplly5aZpKQkk5SUZJYuXWrKlStnhg4danc8/EFVrVrVLFmyxBhjTNGiRc2RI0eMMcaMGzfODBo0yLIcjPmw2fDhwxUaGupcIdbd1K9fX4mJiUpPT1f58uWzDTC1ejXOWzHGaPTo0ZoxY4bzMKK/v79GjRql8ePH25wu008//SRjjAoXLqwrV65o3759SkhIUI0aNdS6dWu740nKvMrl0qVLqlu3rvN0y4MPPsj6JL/BtWvXNGLECM2ePVvXr1+XlHk08dlnn9WkSZPcYuFI/PHcPBt0qVKl9Nlnn6lu3bo6fPiw7rvvPp07d86SHIz5sNmNGzc0efJkrVmzRnXq1Mk24NTupaLdfcbQLA6HQ6+//rrGjRun/fv3KyAgQFFRUW71Bt+xY0d17txZAwcO1LVr1/Too4/Kx8dHP/74o958803bVwaWpHfffVcPPvggs2/mA19fX02fPl2xsbE6cuSIJKly5co5rmINWKV06dI6f/68ypcvr8jISG3ZskV169bVsWPHsk10eSdx5MNmua1c63A4tG7dOgvT4E4qWbKkc/KeuXPn6u9//7t27dqlf//73xo/frxzLggAuFP69++viIgIvfzyy5o5c6ZGjBihJk2aaMeOHercubPmzZtnSQ7KB2CRwoUL68CBA4qMjFTXrl1Vs2ZNvfzyy0pOTlbVqlVZsh7AHZeRkaGMjAznVVjLly9XQkKCoqKiNHDgwGxH3+8UrnZBrm7cuKG//e1vatSokUqXLq3ixYu73HD7qlSpopUrVyo5OVlr1qxRq1atJElnzpzhNAcAS3h5een69evatm2bPv30UwUEBKhly5YqX768Vq9ebVkOxnwgVzExMZo7d66ef/55jR07Vi+99JKOHz+ulStXus1ATk8xfvx4de/eXcOHD1eLFi3UuHFjSdLatWtVv359m9MB+CPIWsw0p4GlVq6DxWkX5Kpy5cqaMWOGHnnkEQUGBmr37t3ObVu2bNGSJUvsjuhRTp06pZMnT6pu3bry8so88Lht2zYFBQWpWrVqNqcDUNBFRUWpVatWti9mSvlArooUKaL9+/crMjJSZcqU0apVq3T33Xfr6NGjql+/vi5cuGB3RADAbQoKCtKuXbtUuXJlW3Mw5gO5KleunHN68sqVK2vt2rWSMhfIcqfLWAEAv+7Pf/6z1q9fb3cMjnwgd6NHj1ZQUJBefPFFLV++XE899ZQqVKigpKQkDR8+XJMmTbI7IgDgNrnLYqaUD+TJli1btGnTJkVFRalDhw52xwEA5MG8efM0cOBA+fv7q0SJEtkWM7VqTTHKB3IVGxursLAw9e3b12X7/PnzdfbsWY0aNcqmZACAvCpdurSGDBmi0aNHOwe924ExH8jV22+/neNVGDVr1tTs2bNtSAQA+K2uXbumbt262Vo8JMoHfsWpU6dUpkyZbNtDQ0OdA1EBAJ6hV69eWr58ud0xmGQMuYuIiFBCQoIqVqzosj0hIUHh4eE2pQIA/Bbuspgp5QO5GjBggIYNG6b09HQ1b95ckhQXF6eRI0fq+eeftzkdACAv9u7d65xRed++fS733Tz49E5jwClyZYzR6NGjNWPGDF27dk2S5O/vr1GjRjG9OgDgN6F84LZcunRJ+/fvV0BAgKKiophgDADwm1E+AACApbjaBQAAWIryAQAALEX5AAAAlqJ8APhdHA6HVq5caXcMAB6E8gHAds2aNdOwYcPsjgHAIpQPALeUnp5udwQABRDlA/gDWb16tR544AEVK1ZMJUqUUPv27XXkyBFJ0vHjx+VwOLR8+XI99NBD8vf317/+9S9JmasY16xZU35+fipTpoyee+45l8f98ccf1alTJxUuXFhRUVH6+OOPXe7ft2+f2rZtq6JFiyosLExPP/20fvzxR0lS7969FR8fr+nTp8vhcMjhcOjYsWOqUqWK/va3v7k8zu7du+VwOJSYmCgp85TPrFmz1LZtWwUEBKhSpUr64IMPXL4nOTlZXbt2VbFixVS8eHF17NhRx48fz7d/UwB5R/kA/kAuX76s6Oho7dixQ3FxcfLy8lKnTp2UkZHh3Gf06NEaOnSo9u/fr9atW2vWrFkaNGiQnnnmGe3du1cff/yxqlSp4vK4MTEx6tq1q/bs2aN27dqpR48eOn/+vCQpJSVFzZs3V/369bVjxw6tXr1ap0+fVteuXSVJ06dPV+PGjTVgwACdPHlSJ0+eVGRkpPr27asFCxa4PM+CBQvUtGlTl+cfN26cunTpoq+//lo9evTQE088of3790vKPHLTunVrBQYGauPGjUpISFDRokXVpk0b54y9AGxgAPxhnT171kgye/fuNceOHTOSzLRp01z2CQ8PNy+99NItH0OSGTt2rPPrS5cuGUnmv//9rzHGmFdeecW0atXK5XuSk5ONJHPw4EFjjDEPPfSQGTp0qMs+33//vSlUqJDZunWrMcaYa9eumZIlS5qFCxe6PPfAgQNdvu/ee+81zz77rDHGmMWLF5uqVauajIwM5/1paWkmICDArFmzJtd/GwB3Dkc+gD+Qw4cP68knn1SlSpUUFBSkChUqSJKSkpKc+zRs2ND55zNnzuiHH35QixYtcn3cOnXqOP9cpEgRBQUF6cyZM5Kkr7/+Wl988YWKFi3qvFWrVk2SnKd8chIeHq5HHnlE8+fPlyR98sknSktL0+OPP+6yX+PGjbN9nXXk4+uvv1ZiYqICAwOdz128eHFdvXo11+cGcGexqi3wB9KhQweVL19ec+bMUXh4uDIyMlSrVi2XUxBFihRx/jkgIOC2HveXy3I7HA7nqZxLly6pQ4cOev3117N9X5kyZXJ93P79++vpp5/W1KlTtWDBAnXr1k2FCxe+rUxZz92gQQPn2JWbhYaG3vbjAMhflA/gD+LcuXM6ePCg5syZowcffFCS9OWXX+b6PYGBgapQoYLi4uL08MMP/6bnvfvuu/Xvf/9bFSpUkLd3zm85vr6+unHjRrbt7dq1U5EiRTRr1iytXr1aGzZsyLbPli1b1LNnT5evs5YMv/vuu7V8+XKVKlVKQUFBvyk/gPzHaRfgDyIkJEQlSpTQP//5TyUmJmrdunWKjo7+1e/761//qilTpmjGjBk6fPiwvvrqK/3973+/7ecdNGiQzp8/ryeffFLbt2/XkSNHtGbNGvXp08dZOCpUqKCtW7fq+PHj+vHHH51HTQoVKqTevXtrzJgxioqKynaKRZLef/99zZ8/X4cOHdLLL7+sbdu2Oa/G6dGjh0qWLKmOHTtq48aNOnbsmNavX68hQ4boxIkTt/13AJC/KB/AH4SXl5eWLVumnTt3qlatWho+fLjeeOONX/2+Xr16adq0aXrrrbdUs2ZNtW/fXocPH77t5w0PD1dCQoJu3LihVq1aqXbt2ho2bJiKFSsmL6/Mt6AXXnhBhQoVUo0aNRQaGuoyBqVfv366du2a+vTpk+Pjx8TEaNmyZapTp44WLVqkpUuXqkaNGpKkwoULa8OGDYqMjFTnzp1VvXp19evXT1evXuVICGAjhzHG2B0CAG5l48aNatGihZKTkxUWFuZyn8Ph0IoVK/TYY4/ZEw7Ab8KYDwBuKS0tTWfPntVf//pXPf7449mKBwDPxWkXAG5p6dKlKl++vFJSUjR58mS74wDIR5x2AQAAluLIBwAAsBTlAwAAWIryAQAALEX5AAAAlqJ8AAAAS1E+AACApSgfAADAUpQPAABgKcoHAACw1P8DH7h2AtarRgUAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "arch_data['archetype'].value_counts().plot(kind=\"bar\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6b6dc29d",
   "metadata": {},
   "outputs": [],
   "source": [
    "arch_data['arch_int'] = LabelEncoder().fit_transform(arch_data['archetype'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "34c3e055",
   "metadata": {},
   "outputs": [],
   "source": [
    "train_split, test_split = train_test_split(arch_data, train_size = 0.8)\n",
    "\n",
    "train_df = pd.DataFrame({\n",
    "    \"label\": train_split.arch_int.values,\n",
    "    \"text\": train_split.description.values\n",
    "})\n",
    "train_df = datasets.Dataset.from_dict(train_df)\n",
    "\n",
    "test_df = pd.DataFrame({\n",
    "    \"label\": test_split.arch_int.values,\n",
    "    \"text\": test_split.description.values\n",
    "})\n",
    "test_df = datasets.Dataset.from_dict(test_df)\n",
    "\n",
    "dataset_dict = datasets.DatasetDict({\"train\":train_df, \"test\":test_df})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c0b1003",
   "metadata": {},
   "source": [
    "### Preprocess data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "801a8c86",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer, DataCollatorWithPadding"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "65cec6e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenizer = AutoTokenizer.from_pretrained(\"distilbert/distilbert-base-uncased\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "6cdaf39f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocess_function(examples):\n",
    "    return tokenizer(examples[\"text\"], truncation=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "abe8ffad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "61e2ab2b7c9842e7995d7d30f340346f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/294 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "5790cb26c1cb4d65a5bf036651b12050",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Map:   0%|          | 0/74 [00:00<?, ? examples/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "tokenized_data = dataset_dict.map(preprocess_function, batched=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "feb1b543",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_collator = DataCollatorWithPadding(tokenizer=tokenizer)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff804b6f",
   "metadata": {},
   "source": [
    "### Train model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "96e61677",
   "metadata": {},
   "outputs": [],
   "source": [
    "import evaluate\n",
    "import numpy as np\n",
    "from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer\n",
    "import accelerate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "296cc26a",
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = evaluate.load(\"accuracy\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "f19004c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(eval_pred):\n",
    "    predictions, labels = eval_pred\n",
    "    predictions = np.argmax(predictions, axis=1)\n",
    "    return accuracy.compute(predictions=predictions, references=labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "73d15982",
   "metadata": {},
   "outputs": [],
   "source": [
    "id2label = {\n",
    "    0: \"Caregiver\",\n",
    "    1: \"Child\",\n",
    "    2: \"Creator\",\n",
    "    3: \"Joker\",\n",
    "    4: \"Lover\",\n",
    "    5: \"Magician\",\n",
    "    6: \"Mentor\",\n",
    "    7: \"Orphan\",\n",
    "    8: \"Rebel\",\n",
    "    9: \"Ruler\",\n",
    "    10: \"Seducer\",\n",
    "    11: \"Warrior\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "57147f69",
   "metadata": {},
   "outputs": [],
   "source": [
    "label2id = {\n",
    "    \"Caregiver\": 0,\n",
    "    \"Child\": 1,\n",
    "    \"Creator\": 2,\n",
    "    \"Joker\": 3,\n",
    "    \"Lover\": 4,\n",
    "    \"Magician\": 5,\n",
    "    \"Mentor\": 6,\n",
    "    \"Orphan\": 7,\n",
    "    \"Rebel\": 8,\n",
    "    \"Ruler\": 9,\n",
    "    \"Seducer\": 10,\n",
    "    \"Warrior\": 11\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "634ffe5d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "model = AutoModelForSequenceClassification.from_pretrained(\n",
    "    \"distilbert/distilbert-base-uncased\", num_labels=12, id2label=id2label, label2id=label2id\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a52dbae4",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='74' max='74' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [74/74 01:01, Epoch 2/2]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.721277</td>\n",
       "      <td>0.797297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.680634</td>\n",
       "      <td>0.783784</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=74, training_loss=0.021499144064413535, metrics={'train_runtime': 62.4119, 'train_samples_per_second': 9.421, 'train_steps_per_second': 1.186, 'total_flos': 4524878938608.0, 'train_loss': 0.021499144064413535, 'epoch': 2.0})"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_args = TrainingArguments(\n",
    "    output_dir=\"archetype_ai_output\",\n",
    "    num_train_epochs=2,\n",
    "    eval_strategy=\"epoch\",\n",
    "    load_best_model_at_end=True\n",
    ")\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    processing_class=tokenizer,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_data[\"train\"],\n",
    "    eval_dataset=tokenized_data[\"test\"],\n",
    "    compute_metrics=compute_metrics\n",
    ")\n",
    "\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b3a456ab",
   "metadata": {},
   "source": [
    "### Tune Hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "81999c18",
   "metadata": {},
   "outputs": [],
   "source": [
    "import optuna\n",
    "from datasets import load_metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "bde50abd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# initialise a new model for each tuning run\n",
    "def model_init():\n",
    "    return AutoModelForSequenceClassification.from_pretrained(\n",
    "        \"distilbert/distilbert-base-uncased\",\n",
    "        num_labels=12,  \n",
    "        id2label=id2label,  \n",
    "        label2id=label2id   \n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "3614d858",
   "metadata": {},
   "outputs": [],
   "source": [
    "# function to control the search of hyperparams to tune\n",
    "def hp_space(trial):\n",
    "    return {\n",
    "        \"learning_rate\": trial.suggest_float(\"learning_rate\", 1e-5, 5e-4, log=True),\n",
    "        \"per_device_train_batch_size\": trial.suggest_categorical(\"per_device_train_batch_size\", [8, 16, 32]),\n",
    "        \"num_train_epochs\": trial.suggest_categorical(\"num_train_epochs\", [2, 3, 5]),\n",
    "        \"weight_decay\": trial.suggest_float(\"weight_decay\", 0.01, 0.1)\n",
    "    }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "eb7cc406",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the training args for each tuning run\n",
    "tuning_training_args = TrainingArguments(\n",
    "    output_dir=\"archetype_ai_tuning\",\n",
    "    eval_strategy=\"epoch\",\n",
    "    save_strategy=\"epoch\",\n",
    "    learning_rate=2e-5,  # Placeholder; will be overridden by the hyperparameter search\n",
    "    per_device_train_batch_size=16,  # Placeholder; will be overridden by the hyperparameter search\n",
    "    num_train_epochs=3,  # Placeholder; will be overridden by the hyperparameter search\n",
    "    logging_dir=\"./logs\",\n",
    "    load_best_model_at_end=True,\n",
    "    metric_for_best_model=\"eval_accuracy\",\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "cd3df56e",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\lnewb\\AppData\\Local\\Temp\\ipykernel_34940\\904913785.py:1: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead.\n",
      "  tuning_trainer = Trainer(\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "tuning_trainer = Trainer(\n",
    "    model_init=model_init,\n",
    "    args=tuning_training_args,\n",
    "    train_dataset=tokenized_data[\"train\"],\n",
    "    eval_dataset=tokenized_data[\"test\"],\n",
    "    tokenizer=tokenizer,\n",
    "    compute_metrics=compute_metrics,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "015e4acc",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 14:49:47,999] A new study created in memory with name: no-name-0d086b68-b04b-4dce-b606-c79262363f5b\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='111' max='111' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [111/111 01:35, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.252723</td>\n",
       "      <td>0.121622</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.200581</td>\n",
       "      <td>0.081081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.128614</td>\n",
       "      <td>0.162162</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 14:51:25,053] Trial 0 finished with value: 0.16216216216216217 and parameters: {'learning_rate': 0.00038320877984687816, 'per_device_train_batch_size': 8, 'num_train_epochs': 3, 'weight_decay': 0.04881692616594958}. Best is trial 0 with value: 0.16216216216216217.\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='111' max='111' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [111/111 01:48, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.827929</td>\n",
       "      <td>0.527027</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.066615</td>\n",
       "      <td>0.702703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.694251</td>\n",
       "      <td>0.810811</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 14:53:15,294] Trial 1 finished with value: 0.8108108108108109 and parameters: {'learning_rate': 0.0002359879189651834, 'per_device_train_batch_size': 8, 'num_train_epochs': 3, 'weight_decay': 0.03466857047662752}. Best is trial 1 with value: 0.8108108108108109.\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='185' max='185' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [185/185 03:13, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.126455</td>\n",
       "      <td>0.513514</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.666884</td>\n",
       "      <td>0.662162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.285545</td>\n",
       "      <td>0.797297</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.067227</td>\n",
       "      <td>0.851351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.026299</td>\n",
       "      <td>0.797297</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 14:56:31,115] Trial 2 finished with value: 0.7972972972972973 and parameters: {'learning_rate': 3.413801169081619e-05, 'per_device_train_batch_size': 8, 'num_train_epochs': 5, 'weight_decay': 0.030881000317279868}. Best is trial 1 with value: 0.8108108108108109.\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='185' max='185' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [185/185 03:14, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.376522</td>\n",
       "      <td>0.418919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.226388</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.087121</td>\n",
       "      <td>0.500000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.002788</td>\n",
       "      <td>0.540541</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.970025</td>\n",
       "      <td>0.554054</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 14:59:47,493] Trial 3 finished with value: 0.5540540540540541 and parameters: {'learning_rate': 1.1220207156767666e-05, 'per_device_train_batch_size': 8, 'num_train_epochs': 5, 'weight_decay': 0.07786745231790623}. Best is trial 1 with value: 0.8108108108108109.\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='57' max='57' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [57/57 01:23, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.269610</td>\n",
       "      <td>0.324324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.075977</td>\n",
       "      <td>0.337838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.978718</td>\n",
       "      <td>0.459459</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:01:13,690] Trial 4 finished with value: 0.4594594594594595 and parameters: {'learning_rate': 3.4760394226941686e-05, 'per_device_train_batch_size': 16, 'num_train_epochs': 3, 'weight_decay': 0.09377720136894309}. Best is trial 1 with value: 0.8108108108108109.\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='38' max='38' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [38/38 00:51, Epoch 2/2]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.322362</td>\n",
       "      <td>0.351351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.241563</td>\n",
       "      <td>0.472973</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:02:07,010] Trial 5 finished with value: 0.47297297297297297 and parameters: {'learning_rate': 2.8887721062153714e-05, 'per_device_train_batch_size': 16, 'num_train_epochs': 2, 'weight_decay': 0.047672912854303466}. Best is trial 1 with value: 0.8108108108108109.\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='37' max='111' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [ 37/111 00:31 < 01:06, 1.11 it/s, Epoch 1/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.810747</td>\n",
       "      <td>0.391892</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:02:40,202] Trial 6 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='30' max='30' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [30/30 01:08, Epoch 3/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.056965</td>\n",
       "      <td>0.283784</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.571015</td>\n",
       "      <td>0.716216</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.338696</td>\n",
       "      <td>0.756757</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:03:51,756] Trial 7 finished with value: 0.7567567567567568 and parameters: {'learning_rate': 0.0001636652316810582, 'per_device_train_batch_size': 32, 'num_train_epochs': 3, 'weight_decay': 0.06472856087762864}. Best is trial 1 with value: 0.8108108108108109.\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='50' max='50' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [50/50 01:54, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.002821</td>\n",
       "      <td>0.378378</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.362751</td>\n",
       "      <td>0.756757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.860549</td>\n",
       "      <td>0.824324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.725967</td>\n",
       "      <td>0.824324</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.602076</td>\n",
       "      <td>0.824324</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:05:49,596] Trial 8 finished with value: 0.8243243243243243 and parameters: {'learning_rate': 0.00018227242233440232, 'per_device_train_batch_size': 32, 'num_train_epochs': 5, 'weight_decay': 0.08360522534360566}. Best is trial 8 with value: 0.8243243243243243.\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='38' max='57' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [38/57 00:51 < 00:26, 0.70 it/s, Epoch 2/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.267918</td>\n",
       "      <td>0.337838</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.071317</td>\n",
       "      <td>0.324324</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:06:42,733] Trial 9 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='10' max='50' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [10/50 00:19 < 01:36, 0.41 it/s, Epoch 1/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.140760</td>\n",
       "      <td>0.243243</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:07:04,912] Trial 10 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='10' max='20' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [10/20 00:19 < 00:23, 0.42 it/s, Epoch 1/2]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.088067</td>\n",
       "      <td>0.243243</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:07:26,856] Trial 11 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='10' max='50' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [10/50 00:19 < 01:36, 0.42 it/s, Epoch 1/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.194209</td>\n",
       "      <td>0.283784</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:07:48,973] Trial 12 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='185' max='185' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [185/185 02:44, Epoch 5/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.690266</td>\n",
       "      <td>0.567568</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.066026</td>\n",
       "      <td>0.662162</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.714207</td>\n",
       "      <td>0.810811</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.698134</td>\n",
       "      <td>0.770270</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.580568</td>\n",
       "      <td>0.810811</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:10:34,675] Trial 13 finished with value: 0.8108108108108109 and parameters: {'learning_rate': 0.00019372605935676665, 'per_device_train_batch_size': 8, 'num_train_epochs': 5, 'weight_decay': 0.030848103050948397}. Best is trial 8 with value: 0.8243243243243243.\n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='10' max='20' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [10/20 00:19 < 00:24, 0.41 it/s, Epoch 1/2]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.075462</td>\n",
       "      <td>0.297297</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:10:56,983] Trial 14 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='37' max='111' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [ 37/111 00:30 < 01:04, 1.15 it/s, Epoch 1/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.827462</td>\n",
       "      <td>0.459459</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:11:28,786] Trial 15 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='20' max='50' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [20/50 00:41 < 01:09, 0.43 it/s, Epoch 2/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.251811</td>\n",
       "      <td>0.391892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.935316</td>\n",
       "      <td>0.527027</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:12:13,140] Trial 16 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='37' max='111' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [ 37/111 00:38 < 01:20, 0.92 it/s, Epoch 1/3]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.521039</td>\n",
       "      <td>0.094595</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:12:52,809] Trial 17 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='20' max='50' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [20/50 00:45 < 01:16, 0.39 it/s, Epoch 2/5]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.261573</td>\n",
       "      <td>0.391892</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>1.953997</td>\n",
       "      <td>0.527027</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:13:41,675] Trial 18 pruned. \n",
      "Some weights of DistilBertForSequenceClassification were not initialized from the model checkpoint at distilbert/distilbert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight', 'pre_classifier.bias', 'pre_classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='19' max='38' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [19/38 00:26 < 00:29, 0.65 it/s, Epoch 1/2]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "      <th>Accuracy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>2.027436</td>\n",
       "      <td>0.324324</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[I 2025-01-04 15:14:10,047] Trial 19 pruned. \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Hyperparameters: {'learning_rate': 0.00018227242233440232, 'per_device_train_batch_size': 32, 'num_train_epochs': 5, 'weight_decay': 0.08360522534360566}\n"
     ]
    }
   ],
   "source": [
    "best_trial = tuning_trainer.hyperparameter_search(\n",
    "    direction=\"maximize\",\n",
    "    backend=\"optuna\",\n",
    "    hp_space=hp_space,\n",
    "    n_trials=20  \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "8b5e898e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Hyperparameters: {'learning_rate': 0.00018227242233440232, 'per_device_train_batch_size': 32, 'num_train_epochs': 5, 'weight_decay': 0.08360522534360566}\n"
     ]
    }
   ],
   "source": [
    "print(\"Best Hyperparameters:\", best_trial.hyperparameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "3e9dd3bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('archetype_ai_tokenizer\\\\tokenizer_config.json',\n",
       " 'archetype_ai_tokenizer\\\\special_tokens_map.json',\n",
       " 'archetype_ai_tokenizer\\\\vocab.txt',\n",
       " 'archetype_ai_tokenizer\\\\added_tokens.json',\n",
       " 'archetype_ai_tokenizer\\\\tokenizer.json')"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tuning_trainer.model.save_pretrained(\"archetype_ai\")\n",
    "tuning_trainer.processing_class.save_pretrained(\"archetype_ai_tokenizer\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c4806c0",
   "metadata": {},
   "source": [
    "## Inference"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "937de8cc",
   "metadata": {},
   "outputs": [],
   "source": [
    "classify_character =  AutoModelForSequenceClassification.from_pretrained(\"archetype_ai\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "4c15a492",
   "metadata": {},
   "outputs": [],
   "source": [
    "classify_character_tokenizer = AutoTokenizer.from_pretrained(\"archetype_ai_tokenizer\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "89a7d91c",
   "metadata": {},
   "outputs": [],
   "source": [
    "new_description =\\\n",
    "\"\"\"\n",
    "Megan from Bridesmaids is a bold, brash, and unapologetically confident \n",
    "woman who enjoys pushing boundaries and making people laugh. \n",
    "She's often the life of the party, offering her unfiltered thoughts \n",
    "and humorous insights, even in uncomfortable situations. Though she \n",
    "may come across as a bit reckless and self-serving at times, she shows \n",
    "her true loyalty and care for her friends when it counts, balancing her \n",
    "outrageous behavior with unexpected moments of warmth and sincerity.\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "66ad2e71",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Device set to use cpu\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[{'label': 'Joker', 'score': 0.27674412727355957}]"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "archetype_ai = pipeline(\"text-classification\", model=classify_character, tokenizer=classify_character_tokenizer)\n",
    "archetype_ai(new_description)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53c7925f",
   "metadata": {},
   "source": [
    "### Gradio"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "698f20e7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gradio as gr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "a7d16c8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "* Running on local URL:  http://127.0.0.1:7861\n",
      "\n",
      "To create a public link, set `share=True` in `launch()`.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div><iframe src=\"http://127.0.0.1:7861/\" width=\"100%\" height=\"500\" allow=\"autoplay; camera; microphone; clipboard-read; clipboard-write;\" frameborder=\"0\" allowfullscreen></iframe></div>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gr.Interface.from_pipeline(archetype_ai).launch()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b4e0741",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}