pytorch and tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c161a893",
   "metadata": {},
   "source": [
    "## Load dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d846fa7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "b222e950",
   "metadata": {},
   "outputs": [],
   "source": [
    "dataset = load_dataset(\"SetFit/bbc-news\", split = \"train\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "d022aca3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'text': 'wales want rugby league training wales could follow england s lead by training with a rugby league club.  england have already had a three-day session with leeds rhinos  and wales are thought to be interested in a similar clinic with rivals st helens. saints coach ian millward has given his approval  but if it does happen it is unlikely to be this season. saints have a week s training in portugal next week  while wales will play england in the opening six nations match on 5 february.  we have had an approach from wales   confirmed a saints spokesman.  it s in the very early stages but it is something we are giving serious consideration to.  st helens  who are proud of their welsh connections  are obvious partners for the welsh rugby union  despite a spat in 2001 over the collapse of kieron cunningham s proposed £500 000 move to union side swansea. a similar cross-code deal that took iestyn harris from leeds to cardiff in 2001 did go through  before the talented stand-off returned to the 13-man code with bradford bulls. kel coslett  who famously moved from wales to league in the 1960s  is currently saints  football manager  while clive griffiths - wales  defensive coach - is a former st helens player and is thought to be the man behind the latest initiative. scott gibbs  the former wales and lions centre  played for st helens from 1994-96 and was in the challenge cup-winning team at wembley in 1996.',\n",
       " 'label': 2,\n",
       " 'label_text': 'sport'}"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7c7e8359",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'type': None,\n",
       " 'format_kwargs': {},\n",
       " 'columns': ['text', 'label', 'label_text'],\n",
       " 'output_all_columns': False}"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset.format # note type = None"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ccf78f4",
   "metadata": {},
   "source": [
    "## Convert dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dced6d6c",
   "metadata": {},
   "source": [
    "### Option 1: format using with_format()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d9e4165",
   "metadata": {},
   "source": [
    "**Pytorch**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "15e9588e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'type': 'torch',\n",
       " 'format_kwargs': {},\n",
       " 'columns': ['text', 'label', 'label_text'],\n",
       " 'output_all_columns': False}"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pt_format = dataset.with_format(type=\"torch\")\n",
    "pt_format.format # note type = Torch"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b813a48",
   "metadata": {},
   "source": [
    "**Tensorflow**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "aafb5af1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'type': 'tensorflow',\n",
       " 'format_kwargs': {},\n",
       " 'columns': ['text', 'label', 'label_text'],\n",
       " 'output_all_columns': False}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tf_format = dataset.with_format(type=\"tf\")\n",
    "tf_format.format # note type = tensorflow"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3db59d1",
   "metadata": {},
   "source": [
    "### Option 2: set our tokenizer to return the desired format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "132ec21e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import AutoTokenizer\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"bert-base-uncased\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9237f411",
   "metadata": {},
   "source": [
    "**Pytorch**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e2ff8856",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': tensor([[ 101, 3575, 2215, 4043, 2223, 2731, 3575, 2071, 3582, 2563, 1055, 2599,\n",
       "         2011, 2731, 2007, 1037, 4043, 2223, 2252,  102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(dataset[0]['text'], max_length=20, padding='max_length', truncation=True, return_tensors=\"pt\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7ba1860",
   "metadata": {},
   "source": [
    "**Tensorflow**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d94acbd8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'input_ids': <tf.Tensor: shape=(1, 20), dtype=int32, numpy=\n",
       "array([[ 101, 3575, 2215, 4043, 2223, 2731, 3575, 2071, 3582, 2563, 1055,\n",
       "        2599, 2011, 2731, 2007, 1037, 4043, 2223, 2252,  102]])>, 'token_type_ids': <tf.Tensor: shape=(1, 20), dtype=int32, numpy=array([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]])>, 'attention_mask': <tf.Tensor: shape=(1, 20), dtype=int32, numpy=array([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])>}"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer(dataset[0]['text'], max_length=20, padding='max_length', truncation=True, return_tensors=\"tf\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0bc37d1",
   "metadata": {},
   "source": [
    "### Option 3: apply transformations after tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "84032af7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# tokenize our dataset\n",
    "def tokenize_function(examples):\n",
    "    return tokenizer(examples['text'], max_length=50, padding='max_length', truncation=True) # shorter max length to speed up training\n",
    "\n",
    "tokenized = dataset.map(tokenize_function, batched=True)\n",
    "tokenized = tokenized.remove_columns(['text', 'label_text'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "483a0df2",
   "metadata": {},
   "source": [
    "**Pytorch**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7e8053bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_pt = tokenized.with_format(type='torch', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f1ac44e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_pt = tokenized_pt.train_test_split(test_size=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "74460045",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'label': tensor(4),\n",
       " 'input_ids': tensor([  101,  8359,  2000,  4895,  3726,  4014,  7521,  2933,  2047,  7711,\n",
       "          2006,  3171, 16836,  1998, 12347,  3675, 14703,  2097,  2022,  2112,\n",
       "          1997,  2231,  3488, 11521,  2006,  6928,  1012,  2188,  3187,  2798,\n",
       "          8359,  4122,  2000,  8970,  1037,  2685,  2291,  2005,  3171, 16836,\n",
       "          1998,  3623, 23702,  2015,  1997,  3478, 11386, 24071,  1012,   102]),\n",
       " 'token_type_ids': tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "         0, 0]),\n",
       " 'attention_mask': tensor([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "         1, 1])}"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_pt['train'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3bca795",
   "metadata": {},
   "source": [
    "**Tensorflow**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "e823cf59",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install --no-deps tensorflow (running without no deps broke numpy)\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "6daed7a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_tf = tokenized.with_format(type='tf', columns=['input_ids', 'token_type_ids', 'attention_mask', 'label'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "43fde49a",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_tf = tokenized_tf.train_test_split(test_size=0.3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "eb6efabb",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'label': <tf.Tensor: shape=(), dtype=int64, numpy=2>,\n",
       " 'input_ids': <tf.Tensor: shape=(50,), dtype=int64, numpy=\n",
       " array([  101,  5982, 11124,  2015, 23439,  6904,  6834,  2346,  5823,\n",
       "         3306, 19938, 12849,  9153,  2015,  5982, 11124,  2015,  2038,\n",
       "         6380,  4447,  2008,  2002,  8275,  2094,  1037,  5013,  5638,\n",
       "         3489,  5823,  2000,  4468,  1037, 23799,  3231,  2420,  2077,\n",
       "         1996,  2707,  1997,  1996,  3783,  1012,  5982, 11124,  2015,\n",
       "         1998,  3507, 19938,  5736,   102], dtype=int64)>,\n",
       " 'token_type_ids': <tf.Tensor: shape=(50,), dtype=int64, numpy=\n",
       " array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,\n",
       "        0, 0, 0, 0, 0, 0], dtype=int64)>,\n",
       " 'attention_mask': <tf.Tensor: shape=(50,), dtype=int64, numpy=\n",
       " array([1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "        1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,\n",
       "        1, 1, 1, 1, 1, 1], dtype=int64)>}"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenized_tf['train'][0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca9d7990",
   "metadata": {},
   "source": [
    "## Load model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "d68fc084",
   "metadata": {},
   "outputs": [],
   "source": [
    "# pip install tf-keras"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d124fb4f",
   "metadata": {},
   "source": [
    "**Pytorch**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "bd0aa01d",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.bias', 'classifier.weight']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import AutoModelForSequenceClassification\n",
    "model_pt = AutoModelForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaae6a4b",
   "metadata": {},
   "source": [
    "**Tensorflow**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "b7569e8b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "WARNING:tensorflow:From C:\\Users\\lnewb\\anaconda3\\envs\\huggingface_course_env\\Lib\\site-packages\\tf_keras\\src\\losses.py:2976: The name tf.losses.sparse_softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.sparse_softmax_cross_entropy instead.\n",
      "\n",
      "WARNING:tensorflow:From C:\\Users\\lnewb\\anaconda3\\envs\\huggingface_course_env\\Lib\\site-packages\\tf_keras\\src\\backend.py:873: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.\n",
      "\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "All PyTorch model weights were used when initializing TFBertForSequenceClassification.\n",
      "\n",
      "Some weights or buffers of the TF 2.0 model TFBertForSequenceClassification were not initialized from the PyTorch model and are newly initialized: ['classifier.weight', 'classifier.bias']\n",
      "You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.\n"
     ]
    }
   ],
   "source": [
    "from transformers import TFAutoModelForSequenceClassification\n",
    "model_tf = TFAutoModelForSequenceClassification.from_pretrained(\"bert-base-uncased\", num_labels=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff8601c3",
   "metadata": {},
   "source": [
    "## Train model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bccf8b6",
   "metadata": {},
   "source": [
    "**Pytorch**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "c65da530",
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import TrainingArguments, Trainer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "df8baf68",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='216' max='216' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [216/216 06:39, Epoch 2/2]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Epoch</th>\n",
       "      <th>Training Loss</th>\n",
       "      <th>Validation Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.291571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>No log</td>\n",
       "      <td>0.151326</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": [
       "TrainOutput(global_step=216, training_loss=0.47597309395119, metrics={'train_runtime': 404.9987, 'train_samples_per_second': 4.232, 'train_steps_per_second': 0.533, 'total_flos': 44041454705400.0, 'train_loss': 0.47597309395119, 'epoch': 2.0})"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "training_args = TrainingArguments(\n",
    "    output_dir=\"pt_model\",\n",
    "    eval_strategy=\"epoch\",\n",
    "    learning_rate=2e-5,\n",
    "    num_train_epochs=2,\n",
    "    weight_decay=0.01\n",
    ")\n",
    "\n",
    "trainer = Trainer(\n",
    "    model=model_pt,\n",
    "    args=training_args,\n",
    "    train_dataset=tokenized_pt['train'],\n",
    "    eval_dataset=tokenized_pt['test']\n",
    ")\n",
    "\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e32f6f9",
   "metadata": {},
   "source": [
    "**Tensorflow**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "2d4af110",
   "metadata": {},
   "outputs": [],
   "source": [
    "tokenized_tf = tokenized_tf['train'].to_tf_dataset(\n",
    "    columns=['input_ids', 'attention_mask'],\n",
    "    label_cols=[\"label\"],\n",
    "    shuffle=True,\n",
    "    batch_size=16\n",
    ") # you could also run this directly on tokenized without doing the other steps above in option 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "39e9e1c5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/2\n",
      "54/54 [==============================] - 148s 2s/step - loss: 0.8043 - accuracy: 0.8506\n",
      "Epoch 2/2\n",
      "54/54 [==============================] - 132s 2s/step - loss: 0.1920 - accuracy: 0.9790\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tf_keras.src.callbacks.History at 0x1f4a2ea4dd0>"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model_tf.compile(optimizer=tf.keras.optimizers.Adam(learning_rate=2e-5),\n",
    "                 loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
    "                 metrics=['accuracy'])\n",
    "\n",
    "model_tf.fit(tokenized_tf, epochs=2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "huggingface_course_env",
   "language": "python",
   "name": "huggingface_course_env"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}