Named_Entity_Recognition-LSTM-CNN-CRF-Tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial we will demonstrate how to implement a state of the art  Bi-directional LSTM-CNN-CRF architecture (Published at ACL'16. [Link To Paper](http://www.aclweb.org/anthology/P16-1101)) for Named Entity Recognition using Pytorch. \n",
    "\n",
    "The main aim of the tutorial is to make the audience comfortable with pytorch using this tutorial and give a step-by-step walk through of the Bi-LSTM-CNN-CRF architecture for NER. Some familiarity with pytorch (or any other deep learning framework) would definitely be a plus. \n",
    "\n",
    "The agenda of this tutorial is as follows:\n",
    "\n",
    "1. Getting Ready with the data \n",
    "2. Network Definition. This includes\n",
    "    * CNN Encoder for Character Level representation.\n",
    "    * Bi-directional LSTM for Word-Level Encoding.\n",
    "    * Conditional Random Fields(CRF) for output decoding\n",
    "3. Training \n",
    "4. Model testing\n",
    "\n",
    "This tutorial draws its content/design heavily from [this](https://github.com/ZhixiuYe/NER-pytorch) Github implementation of NER model. We reuse their data preprocessing/Model creation methodology. This helps in focussing more on explaining model architecture and it's translation from formulae to code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Authors:**\n",
    "[**Anirudh Ganesh**](https://www.linkedin.com/in/anirudh-ganesh95/),\n",
    "[**Peddamail Jayavardhan Reddy**](https://www.linkedin.com/in/jayavardhan-reddy-peddamail-6b4125a0/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preparation\n",
    "\n",
    "The paper uses the English data from CoNLL 2003 shared task\\[1\\], which is present in the \"data\" directory of this project. We will later apply more preprocessing steps to generate tag mapping, word mapping and character  mapping. The data set contains four different types of named entities: PERSON, LOCATION, ORGANIZATION, and MISC and uses the BIO tagging scheme\n",
    "\n",
    "BIO tagging Scheme:\n",
    "\n",
    "    I - Word is inside a phrase of type TYPE\n",
    "    B - If two phrases of the same type immediately follow each other, the first word of the second phrase will have tag B-TYPE \n",
    "    O - Word is not part of a phrase\n",
    "    \n",
    "Example of English-NER sentence available in the data:\n",
    "    \n",
    "    U.N.         NNP  I-NP  I-ORG \n",
    "    official     NN   I-NP  O \n",
    "    Ekeus        NNP  I-NP  I-PER \n",
    "    heads        VBZ  I-VP  O \n",
    "    for          IN   I-PP  O \n",
    "    Baghdad      NNP  I-NP  I-LOC \n",
    "    .            .    O     O \n",
    "    \n",
    "Data Split(We use the same split as mentioned in paper):\n",
    "\n",
    "    Training Data - eng.train\n",
    "    Validation Data - eng.testa\n",
    "    Testing Data - eng.testb\n",
    "    \n",
    "\n",
    " To get started we first import the necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "from collections import OrderedDict\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import init\n",
    "from torch.autograd import Variable\n",
    "from torch import autograd\n",
    "\n",
    "import time\n",
    "import _pickle as cPickle\n",
    "\n",
    "import urllib\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.dpi'] = 80\n",
    "plt.style.use('seaborn-pastel')\n",
    "\n",
    "import os\n",
    "import sys\n",
    "import codecs\n",
    "import re\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Define constants and paramaters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now define some constants and parameters that we will be using later"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#parameters for the Model\n",
    "parameters = OrderedDict()\n",
    "parameters['train'] = \"./data/eng.train\" #Path to train file\n",
    "parameters['dev'] = \"./data/eng.testa\" #Path to test file\n",
    "parameters['test'] = \"./data/eng.testb\" #Path to dev file\n",
    "parameters['tag_scheme'] = \"BIOES\" #BIO or BIOES\n",
    "\n",
    "# parameters['train']=\"./data1/train.bieso.txt\"\n",
    "# parameters['dev']=\"./data1/test.bieso.txt\"\n",
    "# parameters['test']=\"./data1/valid.bieso.txt\"\n",
    "# parameters['tag_scheme'] = \"BIOES\" #BIO or BIOES\n",
    "\n",
    "\n",
    "\n",
    "parameters['lower'] = True # Boolean variable to control lowercasing of words\n",
    "parameters['zeros'] =  True # Boolean variable to control replacement of  all digits by 0 \n",
    "parameters['char_dim'] = 30 #Char embedding dimension\n",
    "parameters['word_dim'] = 100 #Token embedding dimension\n",
    "parameters['word_lstm_dim'] = 200 #Token LSTM hidden layer size\n",
    "parameters['word_bidirect'] = True #Use a bidirectional LSTM for words\n",
    "parameters['embedding_path'] = \"./data/glove.6B.100d.txt\" #Location of pretrained embeddings\n",
    "parameters['all_emb'] = 1 #Load all embeddings\n",
    "parameters['crf'] =1 #Use CRF (0 to disable)\n",
    "parameters['dropout'] = 0.5 #Droupout on the input (0 = no dropout)\n",
    "parameters['epoch'] =  50 #Number of epochs to run\"\n",
    "parameters['weights'] = \"\" #path to Pretrained for from a previous run\n",
    "parameters['name'] = \"self-trained-model\" # Model name\n",
    "parameters['gradient_clip']=5.0\n",
    "parameters['char_mode']=\"CNN\"\n",
    "models_path = \"./models/\" #path to saved models\n",
    "\n",
    "#GPU\n",
    "parameters['use_gpu'] = torch.cuda.is_available() #GPU Check\n",
    "use_gpu = parameters['use_gpu']\n",
    "\n",
    "# parameters['reload'] = \"./models/pre-trained-model\" \n",
    "parameters['reload'] = \"./models/trained-model-cpu\"\n",
    "\n",
    "#Constants\n",
    "START_TAG = '<START>'\n",
    "STOP_TAG = '<STOP>'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#paths to files \n",
    "#To stored mapping file\n",
    "mapping_file = './data/mapping.pkl'\n",
    "\n",
    "#To stored model\n",
    "name = parameters['name']\n",
    "model_name = models_path + name #get_name(parameters)\n",
    "\n",
    "if not os.path.exists(models_path):\n",
    "    os.makedirs(models_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Load data and preprocess"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Firstly, the data is loaded from the train, dev and test files into a list of sentences.\n",
    "\n",
    "Preprocessing:\n",
    "\n",
    "    * All the digits in the words are replaced by 0\n",
    "    \n",
    "Why this preprocessing step?\n",
    "    * For the Named Entity Recognition task, the information present in numerical digits doesnot help in predicting the entity. So, we replace all the digits by 0. So, now the model can concentrate on more important alphabets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def zero_digits(s):\n",
    "    \"\"\"\n",
    "    Replace every digit in a string by a zero.\n",
    "    \"\"\"\n",
    "    return re.sub('\\d', '0', s)\n",
    "\n",
    "def load_sentences(path, zeros):\n",
    "    \"\"\"\n",
    "    Load sentences. A line must contain at least a word and its tag.\n",
    "    Sentences are separated by empty lines.\n",
    "    \"\"\"\n",
    "    sentences = []\n",
    "    sentence = []\n",
    "    for line in codecs.open(path, 'r', 'utf8'):\n",
    "        line = zero_digits(line.rstrip()) if zeros else line.rstrip()\n",
    "        if not line:\n",
    "            if len(sentence) > 0:\n",
    "                if 'DOCSTART' not in sentence[0][0]:\n",
    "                    sentences.append(sentence)\n",
    "                sentence = []\n",
    "        else:\n",
    "            word = line.split()\n",
    "            assert len(word) >= 2\n",
    "            sentence.append(word)\n",
    "    if len(sentence) > 0:\n",
    "        if 'DOCSTART' not in sentence[0][0]:\n",
    "            sentences.append(sentence)\n",
    "    return sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_sentences = load_sentences(parameters['train'], parameters['zeros'])\n",
    "test_sentences = load_sentences(parameters['test'], parameters['zeros'])\n",
    "dev_sentences = load_sentences(parameters['dev'], parameters['zeros'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Update tagging scheme"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Different types of tagging schemes can be used for NER. We update the tags for train, test and dev data ( depending on the parameters \\[ tag_scheme \\] ).\n",
    "\n",
    "In the paper, the authors use the tagging Scheme ( BIOES ) rather than BIO (which is used by the dataset). So, we need to first update the data to convert tag scheme from BIO to BIOES.\n",
    "\n",
    "BIOES tagging scheme:\n",
    "\n",
    "    I - Word is inside a phrase of type TYPE\n",
    "    B - If two phrases of the same type immediately follow each other, the first word of the second phrase will have tag B-TYPE \n",
    "    O - Word is not part of a phrase\n",
    "    E - End ( E will not appear in a prefix-only partial match )\n",
    "    S - Single"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def iob2(tags):\n",
    "    \"\"\"\n",
    "    Check that tags have a valid BIO format.\n",
    "    Tags in BIO1 format are converted to BIO2.\n",
    "    \"\"\"\n",
    "    for i, tag in enumerate(tags):\n",
    "        if tag == 'O':\n",
    "            continue\n",
    "        split = tag.split('-')\n",
    "        if len(split) != 2 or split[0] not in ['I', 'B']:\n",
    "            return False\n",
    "        if split[0] == 'B':\n",
    "            continue\n",
    "        elif i == 0 or tags[i - 1] == 'O':  # conversion IOB1 to IOB2\n",
    "            tags[i] = 'B' + tag[1:]\n",
    "        elif tags[i - 1][1:] == tag[1:]:\n",
    "            continue\n",
    "        else:  # conversion IOB1 to IOB2\n",
    "            tags[i] = 'B' + tag[1:]\n",
    "    return True\n",
    "\n",
    "def iob_iobes(tags):\n",
    "    \"\"\"\n",
    "    the function is used to convert\n",
    "    BIO -> BIOES tagging\n",
    "    \"\"\"\n",
    "    new_tags = []\n",
    "    for i, tag in enumerate(tags):\n",
    "        if tag == 'O':\n",
    "            new_tags.append(tag)\n",
    "        elif tag.split('-')[0] == 'B':\n",
    "            if i + 1 != len(tags) and \\\n",
    "               tags[i + 1].split('-')[0] == 'I':\n",
    "                new_tags.append(tag)\n",
    "            else:\n",
    "                new_tags.append(tag.replace('B-', 'S-'))\n",
    "        elif tag.split('-')[0] == 'I':\n",
    "            if i + 1 < len(tags) and \\\n",
    "                    tags[i + 1].split('-')[0] == 'I':\n",
    "                new_tags.append(tag)\n",
    "            else:\n",
    "                new_tags.append(tag.replace('I-', 'E-'))\n",
    "        else:\n",
    "            raise Exception('Invalid IOB format!')\n",
    "    return new_tags\n",
    "\n",
    "def update_tag_scheme(sentences, tag_scheme):\n",
    "    \"\"\"\n",
    "    Check and update sentences tagging scheme to BIO2\n",
    "    Only BIO1 and BIO2 schemes are accepted for input data.\n",
    "    \"\"\"\n",
    "    for i, s in enumerate(sentences):\n",
    "        tags = [w[-1] for w in s]\n",
    "        # Check that tags are given in the BIO format\n",
    "        if not iob2(tags):\n",
    "            s_str = '\\n'.join(' '.join(w) for w in s)\n",
    "            raise Exception('Sentences should be given in BIO format! ' +\n",
    "                            'Please check sentence %i:\\n%s' % (i, s_str))\n",
    "        if tag_scheme == 'BIOES':\n",
    "            new_tags = iob_iobes(tags)\n",
    "            for word, new_tag in zip(s, new_tags):\n",
    "                word[-1] = new_tag\n",
    "        else:\n",
    "            raise Exception('Wrong tagging scheme!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "update_tag_scheme(train_sentences, parameters['tag_scheme'])\n",
    "update_tag_scheme(dev_sentences, parameters['tag_scheme'])\n",
    "update_tag_scheme(test_sentences, parameters['tag_scheme'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Create Mappings for Words, Characters and Tags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we have updated the tag scheme. We now have a list of sentences which are words along with their modified tags. Now, we want to map these individual words, tags and characters in each word, to unique numerical ID's so that each unique word, character and tag in the vocabulary is represented by a particular integer ID. To do this, we first create a functions that do these mapping for us"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Why mapping is important?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These indices for words, tags and characters help us employ matrix (tensor) operations inside the neural network architecture, which are considerably faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dico(item_list):\n",
    "    \"\"\"\n",
    "    Create a dictionary of items from a list of list of items.\n",
    "    \"\"\"\n",
    "    assert type(item_list) is list\n",
    "    dico = {}\n",
    "    for items in item_list:\n",
    "        for item in items:\n",
    "            if item not in dico:\n",
    "                dico[item] = 1\n",
    "            else:\n",
    "                dico[item] += 1\n",
    "    return dico\n",
    "\n",
    "def create_mapping(dico):\n",
    "    \"\"\"\n",
    "    Create a mapping (item to ID / ID to item) from a dictionary.\n",
    "    Items are ordered by decreasing frequency.\n",
    "    \"\"\"\n",
    "    sorted_items = sorted(dico.items(), key=lambda x: (-x[1], x[0]))\n",
    "    id_to_item = {i: v[0] for i, v in enumerate(sorted_items)}\n",
    "    item_to_id = {v: k for k, v in id_to_item.items()}\n",
    "    return item_to_id, id_to_item\n",
    "\n",
    "def word_mapping(sentences, lower):\n",
    "    \"\"\"\n",
    "    Create a dictionary and a mapping of words, sorted by frequency.\n",
    "    \"\"\"\n",
    "    words = [[x[0].lower() if lower else x[0] for x in s] for s in sentences]\n",
    "    dico = create_dico(words)\n",
    "    dico['<UNK>'] = 10000000 #UNK tag for unknown words\n",
    "    word_to_id, id_to_word = create_mapping(dico)\n",
    "    print(\"Found %i unique words (%i in total)\" % (\n",
    "        len(dico), sum(len(x) for x in words)\n",
    "    ))\n",
    "    return dico, word_to_id, id_to_word\n",
    "\n",
    "def char_mapping(sentences):\n",
    "    \"\"\"\n",
    "    Create a dictionary and mapping of characters, sorted by frequency.\n",
    "    \"\"\"\n",
    "    chars = [\"\".join([w[0] for w in s]) for s in sentences]\n",
    "    dico = create_dico(chars)\n",
    "    char_to_id, id_to_char = create_mapping(dico)\n",
    "    print(\"Found %i unique characters\" % len(dico))\n",
    "    return dico, char_to_id, id_to_char\n",
    "\n",
    "def tag_mapping(sentences):\n",
    "    \"\"\"\n",
    "    Create a dictionary and a mapping of tags, sorted by frequency.\n",
    "    \"\"\"\n",
    "    tags = [[word[-1] for word in s] for s in sentences]\n",
    "    dico = create_dico(tags)\n",
    "    dico[START_TAG] = -1\n",
    "    dico[STOP_TAG] = -2\n",
    "    tag_to_id, id_to_tag = create_mapping(dico)\n",
    "    print(\"Found %i unique named entity tags\" % len(dico))\n",
    "    return dico, tag_to_id, id_to_tag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found 17493 unique words (203621 in total)\n",
      "Found 75 unique characters\n",
      "Found 19 unique named entity tags\n"
     ]
    }
   ],
   "source": [
    "dico_words,word_to_id,id_to_word = word_mapping(train_sentences, parameters['lower'])\n",
    "dico_chars, char_to_id, id_to_char = char_mapping(train_sentences)\n",
    "dico_tags, tag_to_id, id_to_tag = tag_mapping(train_sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Preparing final dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The function prepare dataset returns a list of dictionaries ( one dictionary per each sentence )\n",
    "\n",
    "Each of the dictionary returned by the function contains\n",
    "    1. list of all words in the sentence\n",
    "    2. list of word index for all words in the sentence\n",
    "    3. list of lists, containing character id of each character for words in the sentence\n",
    "    4. list of tag for each word in the sentence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lower_case(x,lower=False):\n",
    "    if lower:\n",
    "        return x.lower()  \n",
    "    else:\n",
    "        return x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "14041 / 3250 / 3453 sentences in train / dev / test.\n"
     ]
    }
   ],
   "source": [
    "def prepare_dataset(sentences, word_to_id, char_to_id, tag_to_id, lower=False):\n",
    "    \"\"\"\n",
    "    Prepare the dataset. Return a list of lists of dictionaries containing:\n",
    "        - word indexes\n",
    "        - word char indexes\n",
    "        - tag indexes\n",
    "    \"\"\"\n",
    "    data = []\n",
    "    for s in sentences:\n",
    "        str_words = [w[0] for w in s]\n",
    "        words = [word_to_id[lower_case(w,lower) if lower_case(w,lower) in word_to_id else '<UNK>']\n",
    "                 for w in str_words]\n",
    "        # Skip characters that are not in the training set\n",
    "        chars = [[char_to_id[c] for c in w if c in char_to_id]\n",
    "                 for w in str_words]\n",
    "        tags = [tag_to_id[w[-1]] for w in s]\n",
    "        data.append({\n",
    "            'str_words': str_words,\n",
    "            'words': words,\n",
    "            'chars': chars,\n",
    "            'tags': tags,\n",
    "        })\n",
    "    return data\n",
    "\n",
    "train_data = prepare_dataset(\n",
    "    train_sentences, word_to_id, char_to_id, tag_to_id, parameters['lower']\n",
    ")\n",
    "dev_data = prepare_dataset(\n",
    "    dev_sentences, word_to_id, char_to_id, tag_to_id, parameters['lower']\n",
    ")\n",
    "test_data = prepare_dataset(\n",
    "    test_sentences, word_to_id, char_to_id, tag_to_id, parameters['lower']\n",
    ")\n",
    "print(\"{} / {} / {} sentences in train / dev / test.\".format(len(train_data), len(dev_data), len(test_data)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are  done with the preprocessing step for input data. It ready to be given as input to the model ! ! !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Load Word Embeddings\n",
    "\n",
    "Now, We move to the next step of loading the pre-trained word embeddings.\n",
    "\n",
    "The paper uses glove vectors 100 dimension vectors trained on the ( Wikipedia 2014 + Gigaword 5 ) corpus containing 6 Billion Words. The word embedding file ( glove.6B.100d.txt ) is placed in the data folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loaded 400000 pretrained embeddings.\n"
     ]
    }
   ],
   "source": [
    "all_word_embeds = {}\n",
    "for i, line in enumerate(codecs.open(parameters['embedding_path'], 'r', 'utf-8')):\n",
    "    s = line.strip().split()\n",
    "    if len(s) == parameters['word_dim'] + 1:\n",
    "        all_word_embeds[s[0]] = np.array([float(i) for i in s[1:]])\n",
    "\n",
    "#Intializing Word Embedding Matrix\n",
    "word_embeds = np.random.uniform(-np.sqrt(0.06), np.sqrt(0.06), (len(word_to_id), parameters['word_dim']))\n",
    "\n",
    "for w in word_to_id:\n",
    "    if w in all_word_embeds:\n",
    "        word_embeds[word_to_id[w]] = all_word_embeds[w]\n",
    "    elif w.lower() in all_word_embeds:\n",
    "        word_embeds[word_to_id[w]] = all_word_embeds[w.lower()]\n",
    "\n",
    "print('Loaded %i pretrained embeddings.' % len(all_word_embeds))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Storing Processed Data for Reuse\n",
    "\n",
    "We can store the preprocessed data and the embedding matrix for future reuse. This helps us avoid the time taken by the step of preprocessing, when we are trying to tune the hyper parameters for the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "word_to_id:  17493\n"
     ]
    }
   ],
   "source": [
    "with open(mapping_file, 'wb') as f:\n",
    "    mappings = {\n",
    "        'word_to_id': word_to_id,\n",
    "        'tag_to_id': tag_to_id,\n",
    "        'char_to_id': char_to_id,\n",
    "        'parameters': parameters,\n",
    "        'word_embeds': word_embeds\n",
    "    }\n",
    "    cPickle.dump(mappings, f)\n",
    "\n",
    "print('word_to_id: ', len(word_to_id))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The model that we are presenting is a complicated one, since its a hybridized network using LSTMs and CNNs. So in order to break down the complexity, we have attempted to simplify the process by splitting up operations into individual functions that we can go over part by part. This hopefully makes the whole thing more easily digestable and gives a more intuitive understanding of the whole process."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Initialization of weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start with the init_embedding function, which just initializes the embedding layer by pooling from a random sample. \n",
    "\n",
    "The distribution is pooled from $-\\sqrt{\\frac{3}{V}}$ to $+\\sqrt{\\frac{3}{V}}$ where $V$ is the embedding dimension size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_embedding(input_embedding):\n",
    "    \"\"\"\n",
    "    Initialize embedding\n",
    "    \"\"\"\n",
    "    bias = np.sqrt(3.0 / input_embedding.size(1))\n",
    "    nn.init.uniform(input_embedding, -bias, bias)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the initialization above, except this is for the linear layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_linear(input_linear):\n",
    "    \"\"\"\n",
    "    Initialize linear transformation\n",
    "    \"\"\"\n",
    "    bias = np.sqrt(6.0 / (input_linear.weight.size(0) + input_linear.weight.size(1)))\n",
    "    nn.init.uniform(input_linear.weight, -bias, bias)\n",
    "    if input_linear.bias is not None:\n",
    "        input_linear.bias.data.zero_()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the initialization scheme for the LSTM layers. \n",
    "\n",
    "The LSTM layers are initialized by uniform sampling from $-\\sqrt{\\frac{6}{r+c}}$ to $+\\sqrt{\\frac{6}{r+c}}$. Where $r$ is the number of rows, $c$ is the number of columns (based on the shape of the weight matrix)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def init_lstm(input_lstm):\n",
    "    \"\"\"\n",
    "    Initialize lstm\n",
    "    \n",
    "    PyTorch weights parameters:\n",
    "    \n",
    "        weight_ih_l[k]: the learnable input-hidden weights of the k-th layer,\n",
    "            of shape `(hidden_size * input_size)` for `k = 0`. Otherwise, the shape is\n",
    "            `(hidden_size * hidden_size)`\n",
    "            \n",
    "        weight_hh_l[k]: the learnable hidden-hidden weights of the k-th layer,\n",
    "            of shape `(hidden_size * hidden_size)`            \n",
    "    \"\"\"\n",
    "    \n",
    "    # Weights init for forward layer\n",
    "    for ind in range(0, input_lstm.num_layers):\n",
    "        \n",
    "        ## Gets the weights Tensor from our model, for the input-hidden weights in our current layer\n",
    "        weight = eval('input_lstm.weight_ih_l' + str(ind))\n",
    "        \n",
    "        # Initialize the sampling range\n",
    "        sampling_range = np.sqrt(6.0 / (weight.size(0) / 4 + weight.size(1)))\n",
    "        \n",
    "        # Randomly sample from our samping range using uniform distribution and apply it to our current layer\n",
    "        nn.init.uniform(weight, -sampling_range, sampling_range)\n",
    "        \n",
    "        # Similar to above but for the hidden-hidden weights of the current layer\n",
    "        weight = eval('input_lstm.weight_hh_l' + str(ind))\n",
    "        sampling_range = np.sqrt(6.0 / (weight.size(0) / 4 + weight.size(1)))\n",
    "        nn.init.uniform(weight, -sampling_range, sampling_range)\n",
    "        \n",
    "        \n",
    "    # We do the above again, for the backward layer if we are using a bi-directional LSTM (our final model uses this)\n",
    "    if input_lstm.bidirectional:\n",
    "        for ind in range(0, input_lstm.num_layers):\n",
    "            weight = eval('input_lstm.weight_ih_l' + str(ind) + '_reverse')\n",
    "            sampling_range = np.sqrt(6.0 / (weight.size(0) / 4 + weight.size(1)))\n",
    "            nn.init.uniform(weight, -sampling_range, sampling_range)\n",
    "            weight = eval('input_lstm.weight_hh_l' + str(ind) + '_reverse')\n",
    "            sampling_range = np.sqrt(6.0 / (weight.size(0) / 4 + weight.size(1)))\n",
    "            nn.init.uniform(weight, -sampling_range, sampling_range)\n",
    "\n",
    "    # Bias initialization steps\n",
    "    \n",
    "    # We initialize them to zero except for the forget gate bias, which is initialized to 1\n",
    "    if input_lstm.bias:\n",
    "        for ind in range(0, input_lstm.num_layers):\n",
    "            bias = eval('input_lstm.bias_ih_l' + str(ind))\n",
    "            \n",
    "            # Initializing to zero\n",
    "            bias.data.zero_()\n",
    "            \n",
    "            # This is the range of indices for our forget gates for each LSTM cell\n",
    "            bias.data[input_lstm.hidden_size: 2 * input_lstm.hidden_size] = 1\n",
    "            \n",
    "            #Similar for the hidden-hidden layer\n",
    "            bias = eval('input_lstm.bias_hh_l' + str(ind))\n",
    "            bias.data.zero_()\n",
    "            bias.data[input_lstm.hidden_size: 2 * input_lstm.hidden_size] = 1\n",
    "            \n",
    "        # Similar to above, we do for backward layer if we are using a bi-directional LSTM \n",
    "        if input_lstm.bidirectional:\n",
    "            for ind in range(0, input_lstm.num_layers):\n",
    "                bias = eval('input_lstm.bias_ih_l' + str(ind) + '_reverse')\n",
    "                bias.data.zero_()\n",
    "                bias.data[input_lstm.hidden_size: 2 * input_lstm.hidden_size] = 1\n",
    "                bias = eval('input_lstm.bias_hh_l' + str(ind) + '_reverse')\n",
    "                bias.data.zero_()\n",
    "                bias.data[input_lstm.hidden_size: 2 * input_lstm.hidden_size] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### CRF Layer\n",
    "\n",
    "We have two options: \n",
    "\n",
    "* softmax: normalize the scores into a vector such that can be interpreted as the probability that the word belongs to class. Eventually, the probability of a sequence of tag $y$ is the product of all tags.\n",
    "\n",
    "\n",
    "* linear-chain CRF: the first method makes local choices. In other words, even if we capture some information from the context thanks to the bi-LSTM, the tagging decision is still local. We don’t make use of the neighbooring tagging decisions. Given a sequence of words $w_1,…,w_m$, a sequence of score vectors $s_1,…,s_m$ and a sequence of tags $y_1,…,y_m$, a linear-chain CRF defines a global score $C \\in \\mathbb{R}$ such that\n",
    "\n",
    "$$% <![CDATA[\n",
    "\\begin{align*}\n",
    "C(y_1, \\ldots, y_m) &= b[y_1] &+ \\sum_{t=1}^{m} s_t [y_t] &+ \\sum_{t=1}^{m-1} T[y_{t}, y_{t+1}] &+ e[y_m]\\\\\n",
    "                    &= \\text{begin} &+ \\text{scores} &+ \\text{transitions} &+ \\text{end}\n",
    "\\end{align*} %]]>$$\n",
    "\n",
    "where $T$ is a transition matrix in $R^{9×9}$ and $e,b \\in R^9$ are vectors of scores that capture the cost of beginning or ending with a given tag. The use of the matrix $T$ captures linear (one step) dependencies between tagging decisions.\n",
    "\n",
    "The motivation behind CRFs was to generate sentence level likelihoods for optimal tags. What that means is for each word we estimate maximum likelihood and then we use the Viterbi algorithm to decode the tag sequence optimally.\n",
    "\n",
    "\n",
    "**Advantages of CRF over Softmax:**\n",
    "* Softmax doesn't value any dependencies, this is a problem since NER the context heavily influences the tag that is assigned. This is solved by applying CRF as it takes into account the full sequence to assign the tag. \n",
    "* *Example: I-ORG cannot directly follow I-PER.*\n",
    "\n",
    "<img src = \"https://github.com/TheAnig/NER-LSTM-CNN-Pytorch/raw/master/images/crf_mode.png\"></img>\n",
    "<a href=\"https://arxiv.org/pdf/1508.01991.pdf\">(Image Source)</a>\n",
    "\n",
    "The figure shows a simple CRF network, in our case we have the inputs feeding in from our BiLSTMs, but otherwise the structure largely remains the same."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Evaluation schemes: Forward pass and Viterbi algorithm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the CRF computes a conditional probability. Let $y$ be a tag sequence and $x$ an input sequence of words. Then we compute\n",
    "\n",
    "$$P(y|x) = \\frac{\\exp{(\\text{Score}(x, y)})}{\\sum_{y'} \\exp{(\\text{Score}(x, y')})}$$\n",
    "\n",
    "Where the score is determined by defining some log potentials $\\log \\psi_i(x,y)$ such that\n",
    "\n",
    "$$\\text{Score}(x,y) = \\sum_i \\log \\psi_i(x,y)$$\n",
    "\n",
    "In our model, we define two kinds of potentials: emission and transition. The emission potential for the word at index $i$ comes from the hidden state of the Bi-LSTM at timestep $i$. The transition scores are stored in a $|T|x|T|$ matrix $P$, where $T$ is the tag set. In my implementation, $P_{j,k}$ is the score of transitioning to tag $j$ from tag $k$. So:\n",
    "\n",
    "$$\\text{Score}(x,y) = \\sum_i \\log \\psi_\\text{EMIT}(y_i \\rightarrow x_i) + \\log \\psi_\\text{TRANS}(y_{i-1} \\rightarrow y_i)$$\n",
    "$$= \\sum_i h_i[y_i] + \\textbf{P}_{y_i, y_{i-1}}$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Helper Functions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we define some helper functions for numerical operations and score calculations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def log_sum_exp(vec):\n",
    "    '''\n",
    "    This function calculates the score explained above for the forward algorithm\n",
    "    vec 2D: 1 * tagset_size\n",
    "    '''\n",
    "    max_score = vec[0, argmax(vec)]\n",
    "    max_score_broadcast = max_score.view(1, -1).expand(1, vec.size()[1])\n",
    "    return max_score + torch.log(torch.sum(torch.exp(vec - max_score_broadcast)))\n",
    "    \n",
    "def argmax(vec):\n",
    "    '''\n",
    "    This function returns the max index in a vector\n",
    "    '''\n",
    "    _, idx = torch.max(vec, 1)\n",
    "    return to_scalar(idx)\n",
    "\n",
    "def to_scalar(var):\n",
    "    '''\n",
    "    Function to convert pytorch tensor to a scalar\n",
    "    '''\n",
    "    return var.view(-1).data.tolist()[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Helper function to calculate score\n",
    "\n",
    "This is a score function for our sentences. \n",
    "\n",
    "This function takes two things, a list of ground truths that tell us what the corresponding tags are, the other are the features which contains the supposed tagged parts of the function. Which is then used to compute the score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def score_sentences(self, feats, tags):\n",
    "    # tags is ground_truth, a list of ints, length is len(sentence)\n",
    "    # feats is a 2D tensor, len(sentence) * tagset_size\n",
    "    r = torch.LongTensor(range(feats.size()[0]))\n",
    "    if self.use_gpu:\n",
    "        r = r.cuda()\n",
    "        pad_start_tags = torch.cat([torch.cuda.LongTensor([self.tag_to_ix[START_TAG]]), tags])\n",
    "        pad_stop_tags = torch.cat([tags, torch.cuda.LongTensor([self.tag_to_ix[STOP_TAG]])])\n",
    "    else:\n",
    "        pad_start_tags = torch.cat([torch.LongTensor([self.tag_to_ix[START_TAG]]), tags])\n",
    "        pad_stop_tags = torch.cat([tags, torch.LongTensor([self.tag_to_ix[STOP_TAG]])])\n",
    "\n",
    "    score = torch.sum(self.transitions[pad_stop_tags, pad_start_tags]) + torch.sum(feats[r, tags])\n",
    "\n",
    "    return score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Implementation of Forward Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_alg(self, feats):\n",
    "    '''\n",
    "    This function performs the forward algorithm explained above\n",
    "    '''\n",
    "    # calculate in log domain\n",
    "    # feats is len(sentence) * tagset_size\n",
    "    # initialize alpha with a Tensor with values all equal to -10000.\n",
    "    \n",
    "    # Do the forward algorithm to compute the partition function\n",
    "    init_alphas = torch.Tensor(1, self.tagset_size).fill_(-10000.)\n",
    "    \n",
    "    # START_TAG has all of the score.\n",
    "    init_alphas[0][self.tag_to_ix[START_TAG]] = 0.\n",
    "    \n",
    "    # Wrap in a variable so that we will get automatic backprop\n",
    "    forward_var = autograd.Variable(init_alphas)\n",
    "    if self.use_gpu:\n",
    "        forward_var = forward_var.cuda()\n",
    "        \n",
    "    # Iterate through the sentence\n",
    "    for feat in feats:\n",
    "        # broadcast the emission score: it is the same regardless of\n",
    "        # the previous tag\n",
    "        emit_score = feat.view(-1, 1)\n",
    "        \n",
    "        # the ith entry of trans_score is the score of transitioning to\n",
    "        # next_tag from i\n",
    "        tag_var = forward_var + self.transitions + emit_score\n",
    "        \n",
    "        # The ith entry of next_tag_var is the value for the\n",
    "        # edge (i -> next_tag) before we do log-sum-exp\n",
    "        max_tag_var, _ = torch.max(tag_var, dim=1)\n",
    "        \n",
    "        # The forward variable for this tag is log-sum-exp of all the\n",
    "        # scores.\n",
    "        tag_var = tag_var - max_tag_var.view(-1, 1)\n",
    "        \n",
    "        # Compute log sum exp in a numerically stable way for the forward algorithm\n",
    "        forward_var = max_tag_var + torch.log(torch.sum(torch.exp(tag_var), dim=1)).view(1, -1) # ).view(1, -1)\n",
    "    terminal_var = (forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]).view(1, -1)\n",
    "    alpha = log_sum_exp(terminal_var)\n",
    "    # Z(x)\n",
    "    return alpha"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Viterbi decode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Viterbi decode is basically applying dynamic programming to choosing our tag sequence. Let’s suppose that we have the solution $\\tilde{s}_{t+1} (y^{t+1})$ for time steps $t + 1, ...., m$ for sequences that start with $y^{t+1}$ for each of the possible $y^{t+1}$. Then the solution $\\tilde{s}_t(y_t)$ for time steps $t, ..., m$ that starts with $y_t$ verifies \n",
    "\n",
    "$$ % <![CDATA[\n",
    "\\begin{align*}\n",
    "\\tilde{s}_t(y_t) &= \\operatorname{argmax}_{y_t, \\ldots, y_m} C(y_t, \\ldots, y_m)\\\\\n",
    "            &= \\operatorname{argmax}_{y_{t+1}} s_t [y_t] + T[y_{t}, y_{t+1}] + \\tilde{s}_{t+1}(y^{t+1})\n",
    "\\end{align*} %]]>$$\n",
    "\n",
    "Then, we can easily define the probability of a given sequence of tags as\n",
    "\n",
    "$$ \\mathbb{P}(y_1, \\ldots, y_m) = \\frac{e^{C(y_1, \\ldots, y_m)}}{Z} $$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Implementation of Viterbi Algorithm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def viterbi_algo(self, feats):\n",
    "    '''\n",
    "    In this function, we implement the viterbi algorithm explained above.\n",
    "    A Dynamic programming based approach to find the best tag sequence\n",
    "    '''\n",
    "    backpointers = []\n",
    "    # analogous to forward\n",
    "    \n",
    "    # Initialize the viterbi variables in log space\n",
    "    init_vvars = torch.Tensor(1, self.tagset_size).fill_(-10000.)\n",
    "    init_vvars[0][self.tag_to_ix[START_TAG]] = 0\n",
    "    \n",
    "    # forward_var at step i holds the viterbi variables for step i-1\n",
    "    forward_var = Variable(init_vvars)\n",
    "    if self.use_gpu:\n",
    "        forward_var = forward_var.cuda()\n",
    "    for feat in feats:\n",
    "        next_tag_var = forward_var.view(1, -1).expand(self.tagset_size, self.tagset_size) + self.transitions\n",
    "        _, bptrs_t = torch.max(next_tag_var, dim=1)\n",
    "        bptrs_t = bptrs_t.squeeze().data.cpu().numpy() # holds the backpointers for this step\n",
    "        next_tag_var = next_tag_var.data.cpu().numpy() \n",
    "        viterbivars_t = next_tag_var[range(len(bptrs_t)), bptrs_t] # holds the viterbi variables for this step\n",
    "        viterbivars_t = Variable(torch.FloatTensor(viterbivars_t))\n",
    "        if self.use_gpu:\n",
    "            viterbivars_t = viterbivars_t.cuda()\n",
    "            \n",
    "        # Now add in the emission scores, and assign forward_var to the set\n",
    "        # of viterbi variables we just computed\n",
    "        forward_var = viterbivars_t + feat\n",
    "        backpointers.append(bptrs_t)\n",
    "\n",
    "    # Transition to STOP_TAG\n",
    "    terminal_var = forward_var + self.transitions[self.tag_to_ix[STOP_TAG]]\n",
    "    terminal_var.data[self.tag_to_ix[STOP_TAG]] = -10000.\n",
    "    terminal_var.data[self.tag_to_ix[START_TAG]] = -10000.\n",
    "    best_tag_id = argmax(terminal_var.unsqueeze(0))\n",
    "    path_score = terminal_var[best_tag_id]\n",
    "    \n",
    "    # Follow the back pointers to decode the best path.\n",
    "    best_path = [best_tag_id]\n",
    "    for bptrs_t in reversed(backpointers):\n",
    "        best_tag_id = bptrs_t[best_tag_id]\n",
    "        best_path.append(best_tag_id)\n",
    "        \n",
    "    # Pop off the start tag (we dont want to return that to the caller)\n",
    "    start = best_path.pop()\n",
    "    assert start == self.tag_to_ix[START_TAG] # Sanity check\n",
    "    best_path.reverse()\n",
    "    return path_score, best_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def forward_calc(self, sentence, chars, chars2_length, d):\n",
    "    \n",
    "    '''\n",
    "    The function calls viterbi decode and generates the \n",
    "    most probable sequence of tags for the sentence\n",
    "    '''\n",
    "    \n",
    "    # Get the emission scores from the BiLSTM\n",
    "    feats = self._get_lstm_features(sentence, chars, chars2_length, d)\n",
    "    # viterbi to get tag_seq\n",
    "    \n",
    "    # Find the best path, given the features.\n",
    "    if self.use_crf:\n",
    "        score, tag_seq = self.viterbi_decode(feats)\n",
    "    else:\n",
    "        score, tag_seq = torch.max(feats, 1)\n",
    "        tag_seq = list(tag_seq.cpu().data)\n",
    "\n",
    "    return score, tag_seq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Details fo the Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 1. CNN model for generating character embeddings\n",
    "\n",
    "\n",
    "Consider the word 'cat', we pad it on both ends to get our maximum word length ( this is mainly an implementation quirk since we can't have variable length layers at run time, our algorithm will ignore the pads).\n",
    "\n",
    "We then apply a convolution layer on top that generates spatial coherence across characters, we use a maxpool to extract meaningful features out of our convolution layer. This now gives us a dense vector representation of each word. This representation will be concatenated with the pre-trained GloVe embeddings using a simple lookup.\n",
    "\n",
    "\n",
    "<img src = \"https://github.com/TheAnig/NER-LSTM-CNN-Pytorch/raw/master/images/cnn_model.png\"></img>\n",
    "<a href=\"http://www.aclweb.org/anthology/P16-1101\">Image Source</a>\n",
    "\n",
    "\n",
    "This snippet shows us how the CNN is implemented in pytorch\n",
    "\n",
    "`self.char_cnn3 = nn.Conv2d(in_channels=1, out_channels=self.out_channels, kernel_size=(3, char_embedding_dim), padding=(2,0))`\n",
    "\n",
    "##### 2. Rest of the model (LSTM based) that generates tags for the given sequence\n",
    "\n",
    "The word-embeddings( glove+char embedding ) that we generated above, we feed to a bi-directional LSTM model. The LSTM model has 2 layers, \n",
    "* The forward layer takes in a sequence of word vectors and generates a new vector based on what it has seen so far in the forward direction (starting from the start word up until current word) this vector can be thought of as a summary of all the words it has seen. \n",
    "\n",
    "* The backwards layer does the same but in opposite direction, i.e., from the end of the sentence to the current word.\n",
    "\n",
    "The forward vector and the backwards vector at current word concatanate to generate a unified representation.\n",
    "\n",
    "<img src = \"https://github.com/TheAnig/NER-LSTM-CNN-Pytorch/raw/master/images/lstm_model.png\"></img>\n",
    "<a href=\"http://www.aclweb.org/anthology/P16-1101\">Image Source</a>\n",
    "\n",
    "This snippet shows us how the BiLSTM is implemented in pytorch\n",
    "\n",
    "`self.lstm = nn.LSTM(embedding_dim+self.out_channels, hidden_dim, bidirectional=True)`\n",
    "\n",
    "Finally, we have a linear layer to map hidden vectors to tag space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Main Model Implementation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The get_lstm_features function returns the LSTM's tag vectors. The function performs all the steps mentioned above for the model.\n",
    "\n",
    "Steps:\n",
    "1. It takes in characters, converts them to embeddings using our character CNN.\n",
    "2. We concat Character Embeeding with glove vectors, use this as features that we feed to Bidirectional-LSTM. \n",
    "3. The Bidirectional-LSTM generates outputs based on these set of features.\n",
    "4. The output are passed through a linear layer to convert to tag space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_lstm_features(self, sentence, chars2, chars2_length, d):\n",
    "    \n",
    "    if self.char_mode == 'LSTM':\n",
    "        \n",
    "            chars_embeds = self.char_embeds(chars2).transpose(0, 1)\n",
    "            \n",
    "            packed = torch.nn.utils.rnn.pack_padded_sequence(chars_embeds, chars2_length)\n",
    "            \n",
    "            lstm_out, _ = self.char_lstm(packed)\n",
    "            \n",
    "            outputs, output_lengths = torch.nn.utils.rnn.pad_packed_sequence(lstm_out)\n",
    "            \n",
    "            outputs = outputs.transpose(0, 1)\n",
    "            \n",
    "            chars_embeds_temp = Variable(torch.FloatTensor(torch.zeros((outputs.size(0), outputs.size(2)))))\n",
    "            \n",
    "            if self.use_gpu:\n",
    "                chars_embeds_temp = chars_embeds_temp.cuda()\n",
    "            \n",
    "            for i, index in enumerate(output_lengths):\n",
    "                chars_embeds_temp[i] = torch.cat((outputs[i, index-1, :self.char_lstm_dim], outputs[i, 0, self.char_lstm_dim:]))\n",
    "            \n",
    "            chars_embeds = chars_embeds_temp.clone()\n",
    "            \n",
    "            for i in range(chars_embeds.size(0)):\n",
    "                chars_embeds[d[i]] = chars_embeds_temp[i]\n",
    "    \n",
    "    \n",
    "    if self.char_mode == 'CNN':\n",
    "        chars_embeds = self.char_embeds(chars2).unsqueeze(1)\n",
    "\n",
    "        ## Creating Character level representation using Convolutional Neural Netowrk\n",
    "        ## followed by a Maxpooling Layer\n",
    "        chars_cnn_out3 = self.char_cnn3(chars_embeds)\n",
    "        chars_embeds = nn.functional.max_pool2d(chars_cnn_out3,\n",
    "                                             kernel_size=(chars_cnn_out3.size(2), 1)).view(chars_cnn_out3.size(0), self.out_channels)\n",
    "\n",
    "        ## Loading word embeddings\n",
    "    embeds = self.word_embeds(sentence)\n",
    "\n",
    "    ## We concatenate the word embeddings and the character level representation\n",
    "    ## to create unified representation for each word\n",
    "    embeds = torch.cat((embeds, chars_embeds), 1)\n",
    "\n",
    "    embeds = embeds.unsqueeze(1)\n",
    "\n",
    "    ## Dropout on the unified embeddings\n",
    "    embeds = self.dropout(embeds)\n",
    "\n",
    "    ## Word lstm\n",
    "    ## Takes words as input and generates a output at each step\n",
    "    lstm_out, _ = self.lstm(embeds)\n",
    "\n",
    "    ## Reshaping the outputs from the lstm layer\n",
    "    lstm_out = lstm_out.view(len(sentence), self.hidden_dim*2)\n",
    "\n",
    "    ## Dropout on the lstm output\n",
    "    lstm_out = self.dropout(lstm_out)\n",
    "\n",
    "    ## Linear layer converts the ouput vectors to tag space\n",
    "    lstm_feats = self.hidden2tag(lstm_out)\n",
    "    \n",
    "    return lstm_feats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Funtion for Negative log likelihood calculation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a helper function that calculates the negative log likelihood. \n",
    "\n",
    "The functions takes as input the previously calulcated lstm features to use to calculate the sentence score and then perform a forward run score and compare it with our predicted score to generate a log likelihood. \n",
    "\n",
    "`Implementation detail: Notice we do not pump out any log conversion in this function that is supposedly about log likelihood calculation, this is because we have ensured that we get the scores from our helper functions in the log domain.`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_neg_log_likelihood(self, sentence, tags, chars2, chars2_length, d):\n",
    "    # sentence, tags is a list of ints\n",
    "    # features is a 2D tensor, len(sentence) * self.tagset_size\n",
    "    feats = self._get_lstm_features(sentence, chars2, chars2_length, d)\n",
    "\n",
    "    if self.use_crf:\n",
    "        forward_score = self._forward_alg(feats)\n",
    "        gold_score = self._score_sentence(feats, tags)\n",
    "        return forward_score - gold_score\n",
    "    else:\n",
    "        tags = Variable(tags)\n",
    "        scores = nn.functional.cross_entropy(feats, tags)\n",
    "        return scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Main Model Class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BiLSTM_CRF(nn.Module):\n",
    "\n",
    "    def __init__(self, vocab_size, tag_to_ix, embedding_dim, hidden_dim,\n",
    "                 char_to_ix=None, pre_word_embeds=None, char_out_dimension=25,char_embedding_dim=25, use_gpu=False\n",
    "                 , use_crf=True, char_mode='CNN'):\n",
    "        '''\n",
    "        Input parameters:\n",
    "                \n",
    "                vocab_size= Size of vocabulary (int)\n",
    "                tag_to_ix = Dictionary that maps NER tags to indices\n",
    "                embedding_dim = Dimension of word embeddings (int)\n",
    "                hidden_dim = The hidden dimension of the LSTM layer (int)\n",
    "                char_to_ix = Dictionary that maps characters to indices\n",
    "                pre_word_embeds = Numpy array which provides mapping from word embeddings to word indices\n",
    "                char_out_dimension = Output dimension from the CNN encoder for character\n",
    "                char_embedding_dim = Dimension of the character embeddings\n",
    "                use_gpu = defines availability of GPU, \n",
    "                    when True: CUDA function calls are made\n",
    "                    else: Normal CPU function calls are made\n",
    "                use_crf = parameter which decides if you want to use the CRF layer for output decoding\n",
    "        '''\n",
    "        \n",
    "        super(BiLSTM_CRF, self).__init__()\n",
    "        \n",
    "        #parameter initialization for the model\n",
    "        self.use_gpu = use_gpu\n",
    "        self.embedding_dim = embedding_dim\n",
    "        self.hidden_dim = hidden_dim\n",
    "        self.vocab_size = vocab_size\n",
    "        self.tag_to_ix = tag_to_ix\n",
    "        self.use_crf = use_crf\n",
    "        self.tagset_size = len(tag_to_ix)\n",
    "        self.out_channels = char_out_dimension\n",
    "        self.char_mode = char_mode\n",
    "\n",
    "        if char_embedding_dim is not None:\n",
    "            self.char_embedding_dim = char_embedding_dim\n",
    "            \n",
    "            #Initializing the character embedding layer\n",
    "            self.char_embeds = nn.Embedding(len(char_to_ix), char_embedding_dim)\n",
    "            init_embedding(self.char_embeds.weight)\n",
    "            \n",
    "            #Performing LSTM encoding on the character embeddings\n",
    "            if self.char_mode == 'LSTM':\n",
    "                self.char_lstm = nn.LSTM(char_embedding_dim, char_lstm_dim, num_layers=1, bidirectional=True)\n",
    "                init_lstm(self.char_lstm)\n",
    "                \n",
    "            #Performing CNN encoding on the character embeddings\n",
    "            if self.char_mode == 'CNN':\n",
    "                self.char_cnn3 = nn.Conv2d(in_channels=1, out_channels=self.out_channels, kernel_size=(3, char_embedding_dim), padding=(2,0))\n",
    "\n",
    "        #Creating Embedding layer with dimension of ( number of words * dimension of each word)\n",
    "        self.word_embeds = nn.Embedding(vocab_size, embedding_dim)\n",
    "        if pre_word_embeds is not None:\n",
    "            #Initializes the word embeddings with pretrained word embeddings\n",
    "            self.pre_word_embeds = True\n",
    "            self.word_embeds.weight = nn.Parameter(torch.FloatTensor(pre_word_embeds))\n",
    "        else:\n",
    "            self.pre_word_embeds = False\n",
    "    \n",
    "        #Initializing the dropout layer, with dropout specificed in parameters\n",
    "        self.dropout = nn.Dropout(parameters['dropout'])\n",
    "        \n",
    "        #Lstm Layer:\n",
    "        #input dimension: word embedding dimension + character level representation\n",
    "        #bidirectional=True, specifies that we are using the bidirectional LSTM\n",
    "        if self.char_mode == 'LSTM':\n",
    "            self.lstm = nn.LSTM(embedding_dim+char_lstm_dim*2, hidden_dim, bidirectional=True)\n",
    "        if self.char_mode == 'CNN':\n",
    "            self.lstm = nn.LSTM(embedding_dim+self.out_channels, hidden_dim, bidirectional=True)\n",
    "        \n",
    "        #Initializing the lstm layer using predefined function for initialization\n",
    "        init_lstm(self.lstm)\n",
    "        \n",
    "        # Linear layer which maps the output of the bidirectional LSTM into tag space.\n",
    "        self.hidden2tag = nn.Linear(hidden_dim*2, self.tagset_size)\n",
    "        \n",
    "        #Initializing the linear layer using predefined function for initialization\n",
    "        init_linear(self.hidden2tag) \n",
    "\n",
    "        if self.use_crf:\n",
    "            # Matrix of transition parameters.  Entry i,j is the score of transitioning *to* i *from* j.\n",
    "            # Matrix has a dimension of (total number of tags * total number of tags)\n",
    "            self.transitions = nn.Parameter(\n",
    "                torch.zeros(self.tagset_size, self.tagset_size))\n",
    "            \n",
    "            # These two statements enforce the constraint that we never transfer\n",
    "            # to the start tag and we never transfer from the stop tag\n",
    "            self.transitions.data[tag_to_ix[START_TAG], :] = -10000\n",
    "            self.transitions.data[:, tag_to_ix[STOP_TAG]] = -10000\n",
    "\n",
    "    #assigning the functions, which we have defined earlier\n",
    "    _score_sentence = score_sentences\n",
    "    _get_lstm_features = get_lstm_features\n",
    "    _forward_alg = forward_alg\n",
    "    viterbi_decode = viterbi_algo\n",
    "    neg_log_likelihood = get_neg_log_likelihood\n",
    "    forward = forward_calc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model Initialized!!!\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/zl/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:6: UserWarning: nn.init.uniform is now deprecated in favor of nn.init.uniform_.\n",
      "  \n",
      "/Users/zl/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:25: UserWarning: nn.init.uniform is now deprecated in favor of nn.init.uniform_.\n",
      "/Users/zl/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:30: UserWarning: nn.init.uniform is now deprecated in favor of nn.init.uniform_.\n",
      "/Users/zl/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:38: UserWarning: nn.init.uniform is now deprecated in favor of nn.init.uniform_.\n",
      "/Users/zl/anaconda3/lib/python3.7/site-packages/ipykernel_launcher.py:41: UserWarning: nn.init.uniform is now deprecated in favor of nn.init.uniform_.\n"
     ]
    }
   ],
   "source": [
    "#creating the model using the Class defined above\n",
    "model = BiLSTM_CRF(vocab_size=len(word_to_id),\n",
    "                   tag_to_ix=tag_to_id,\n",
    "                   embedding_dim=parameters['word_dim'],\n",
    "                   hidden_dim=parameters['word_lstm_dim'],\n",
    "                   use_gpu=use_gpu,\n",
    "                   char_to_ix=char_to_id,\n",
    "                   pre_word_embeds=word_embeds,\n",
    "                   use_crf=parameters['crf'],\n",
    "                   char_mode=parameters['char_mode'])\n",
    "print(\"Model Initialized!!!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model reloaded : ./models/trained-model-cpu\n"
     ]
    }
   ],
   "source": [
    "#Reload a saved model, if parameter[\"reload\"] is set to a path\n",
    "if parameters['reload']:\n",
    "    if not os.path.exists(parameters['reload']):\n",
    "        print(\"downloading pre-trained model\")\n",
    "        model_url=\"https://github.com/TheAnig/NER-LSTM-CNN-Pytorch/raw/master/trained-model-cpu\"\n",
    "        urllib.request.urlretrieve(model_url, parameters['reload'])\n",
    "    model.load_state_dict(torch.load(parameters['reload']))\n",
    "    print(\"model reloaded :\", parameters['reload'])\n",
    "\n",
    "if use_gpu:\n",
    "    model.cuda()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Training Paramaters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Initializing the optimizer\n",
    "#The best results in the paper where achived using stochastic gradient descent (SGD) \n",
    "#learning rate=0.015 and momentum=0.9 \n",
    "#decay_rate=0.05 \n",
    "\n",
    "learning_rate = 0.015\n",
    "momentum = 0.9\n",
    "number_of_epochs = parameters['epoch'] \n",
    "decay_rate = 0.05\n",
    "gradient_clip = parameters['gradient_clip']\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate, momentum=momentum)\n",
    "\n",
    "#variables which will used in training process\n",
    "losses = [] #list to store all losses\n",
    "loss = 0.0 #Loss Initializatoin\n",
    "best_dev_F = -1.0 # Current best F-1 Score on Dev Set\n",
    "best_test_F = -1.0 # Current best F-1 Score on Test Set\n",
    "best_train_F = -1.0 # Current best F-1 Score on Train Set\n",
    "all_F = [[0, 0, 0]] # List storing all the F-1 Scores\n",
    "eval_every = len(train_data) # Calculate F-1 Score after this many iterations\n",
    "plot_every = 2000 # Store loss after this many iterations\n",
    "count = 0 #Counts the number of iterations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Helper functions for evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_chunk_type(tok, idx_to_tag):\n",
    "    \"\"\"\n",
    "    The function takes in a chunk (\"B-PER\") and then splits it into the tag (PER) and its class (B)\n",
    "    as defined in BIOES\n",
    "    \n",
    "    Args:\n",
    "        tok: id of token, ex 4\n",
    "        idx_to_tag: dictionary {4: \"B-PER\", ...}\n",
    "\n",
    "    Returns:\n",
    "        tuple: \"B\", \"PER\"\n",
    "\n",
    "    \"\"\"\n",
    "    \n",
    "    tag_name = idx_to_tag[tok]\n",
    "    tag_class = tag_name.split('-')[0]\n",
    "    tag_type = tag_name.split('-')[-1]\n",
    "    return tag_class, tag_type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_chunks(seq, tags):\n",
    "    \"\"\"Given a sequence of tags, group entities and their position\n",
    "\n",
    "    Args:\n",
    "        seq: [4, 4, 0, 0, ...] sequence of labels\n",
    "        tags: dict[\"O\"] = 4\n",
    "\n",
    "    Returns:\n",
    "        list of (chunk_type, chunk_start, chunk_end)\n",
    "\n",
    "    Example:\n",
    "        seq = [4, 5, 0, 3]\n",
    "        tags = {\"B-PER\": 4, \"I-PER\": 5, \"B-LOC\": 3}\n",
    "        result = [(\"PER\", 0, 2), (\"LOC\", 3, 4)]\n",
    "\n",
    "    \"\"\"\n",
    "    \n",
    "    # We assume by default the tags lie outside a named entity\n",
    "    default = tags[\"O\"]\n",
    "    \n",
    "    idx_to_tag = {idx: tag for tag, idx in tags.items()}\n",
    "    \n",
    "    chunks = []\n",
    "    \n",
    "    chunk_type, chunk_start = None, None\n",
    "    for i, tok in enumerate(seq):\n",
    "        # End of a chunk 1\n",
    "        if tok == default and chunk_type is not None:\n",
    "            # Add a chunk.\n",
    "            chunk = (chunk_type, chunk_start, i)\n",
    "            chunks.append(chunk)\n",
    "            chunk_type, chunk_start = None, None\n",
    "\n",
    "        # End of a chunk + start of a chunk!\n",
    "        elif tok != default:\n",
    "            tok_chunk_class, tok_chunk_type = get_chunk_type(tok, idx_to_tag)\n",
    "            if chunk_type is None:\n",
    "                # Initialize chunk for each entity\n",
    "                chunk_type, chunk_start = tok_chunk_type, i\n",
    "            elif tok_chunk_type != chunk_type or tok_chunk_class == \"B\":\n",
    "                # If chunk class is B, i.e., its a beginning of a new named entity\n",
    "                # or, if the chunk type is different from the previous one, then we\n",
    "                # start labelling it as a new entity\n",
    "                chunk = (chunk_type, chunk_start, i)\n",
    "                chunks.append(chunk)\n",
    "                chunk_type, chunk_start = tok_chunk_type, i\n",
    "        else:\n",
    "            pass\n",
    "\n",
    "    # end condition\n",
    "    if chunk_type is not None:\n",
    "        chunk = (chunk_type, chunk_start, len(seq))\n",
    "        chunks.append(chunk)\n",
    "\n",
    "    return chunks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluating(model, datas, best_F,dataset=\"Train\"):\n",
    "    '''\n",
    "    The function takes as input the model, data and calcuates F-1 Score\n",
    "    It performs conditional updates \n",
    "     1) Flag to save the model \n",
    "     2) Best F-1 score\n",
    "    ,if the F-1 score calculated improves on the previous F-1 score\n",
    "    '''\n",
    "    # Initializations\n",
    "    prediction = [] # A list that stores predicted tags\n",
    "    save = False # Flag that tells us if the model needs to be saved\n",
    "    new_F = 0.0 # Variable to store the current F1-Score (may not be the best)\n",
    "    correct_preds, total_correct, total_preds = 0., 0., 0. # Count variables\n",
    "    \n",
    "    for data in datas:\n",
    "        ground_truth_id = data['tags']\n",
    "        words = data['str_words']\n",
    "        chars2 = data['chars']\n",
    "        \n",
    "        if parameters['char_mode'] == 'LSTM':\n",
    "            chars2_sorted = sorted(chars2, key=lambda p: len(p), reverse=True)\n",
    "            d = {}\n",
    "            for i, ci in enumerate(chars2):\n",
    "                for j, cj in enumerate(chars2_sorted):\n",
    "                    if ci == cj and not j in d and not i in d.values():\n",
    "                        d[j] = i\n",
    "                        continue\n",
    "            chars2_length = [len(c) for c in chars2_sorted]\n",
    "            char_maxl = max(chars2_length)\n",
    "            chars2_mask = np.zeros((len(chars2_sorted), char_maxl), dtype='int')\n",
    "            for i, c in enumerate(chars2_sorted):\n",
    "                chars2_mask[i, :chars2_length[i]] = c\n",
    "            chars2_mask = Variable(torch.LongTensor(chars2_mask))\n",
    "        \n",
    "        \n",
    "        if parameters['char_mode'] == 'CNN':\n",
    "            d = {} \n",
    "\n",
    "            # Padding the each word to max word size of that sentence\n",
    "            chars2_length = [len(c) for c in chars2]\n",
    "            char_maxl = max(chars2_length)\n",
    "            chars2_mask = np.zeros((len(chars2_length), char_maxl), dtype='int')\n",
    "            for i, c in enumerate(chars2):\n",
    "                chars2_mask[i, :chars2_length[i]] = c\n",
    "            chars2_mask = Variable(torch.LongTensor(chars2_mask))\n",
    "\n",
    "        dwords = Variable(torch.LongTensor(data['words']))\n",
    "        \n",
    "        # We are getting the predicted output from our model\n",
    "        if use_gpu:\n",
    "            val,out = model(dwords.cuda(), chars2_mask.cuda(), chars2_length, d)\n",
    "        else:\n",
    "            val,out = model(dwords, chars2_mask, chars2_length, d)\n",
    "        predicted_id = out\n",
    "    \n",
    "        \n",
    "        # We use the get chunks function defined above to get the true chunks\n",
    "        # and the predicted chunks from true labels and predicted labels respectively\n",
    "        lab_chunks      = set(get_chunks(ground_truth_id,tag_to_id))\n",
    "        lab_pred_chunks = set(get_chunks(predicted_id,\n",
    "                                         tag_to_id))\n",
    "\n",
    "        # Updating the count variables\n",
    "        correct_preds += len(lab_chunks & lab_pred_chunks)\n",
    "        total_preds   += len(lab_pred_chunks)\n",
    "        total_correct += len(lab_chunks)\n",
    "    \n",
    "    # Calculating the F1-Score\n",
    "    p   = correct_preds / total_preds if correct_preds > 0 else 0\n",
    "    r   = correct_preds / total_correct if correct_preds > 0 else 0\n",
    "    new_F  = 2 * p * r / (p + r) if correct_preds > 0 else 0\n",
    "\n",
    "    print(\"{}: new_F: {} best_F: {} \".format(dataset,new_F,best_F))\n",
    "    \n",
    "    # If our current F1-Score is better than the previous best, we update the best\n",
    "    # to current F1 and we set the flag to indicate that we need to checkpoint this model\n",
    "    \n",
    "    if new_F>best_F:\n",
    "        best_F=new_F\n",
    "        save=True\n",
    "\n",
    "    return best_F, new_F, save"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Helper function for performing Learning rate decay"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def adjust_learning_rate(optimizer, lr):\n",
    "    \"\"\"\n",
    "    shrink learning rate\n",
    "    \"\"\"\n",
    "    for param_group in optimizer.param_groups:\n",
    "        param_group['lr'] = lr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training Step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If `parameters['reload']` is set, we already have a model to load of off, so we can skip the training. We have originally specified a pre-trained model since training is an expensive process, but we encourage readers to try this out once they're done with the tutorial."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "#parameters['reload']=False\n",
    "\n",
    "if not parameters['reload']:\n",
    "    tr = time.time()\n",
    "    model.train(True)\n",
    "    for epoch in range(1,number_of_epochs):\n",
    "        for i, index in enumerate(np.random.permutation(len(train_data))):\n",
    "            count += 1\n",
    "            data = train_data[index]\n",
    "\n",
    "            ##gradient updates for each data entry\n",
    "            model.zero_grad()\n",
    "\n",
    "            sentence_in = data['words']\n",
    "            sentence_in = Variable(torch.LongTensor(sentence_in))\n",
    "            tags = data['tags']\n",
    "            chars2 = data['chars']\n",
    "            \n",
    "            if parameters['char_mode'] == 'LSTM':\n",
    "                chars2_sorted = sorted(chars2, key=lambda p: len(p), reverse=True)\n",
    "                d = {}\n",
    "                for i, ci in enumerate(chars2):\n",
    "                    for j, cj in enumerate(chars2_sorted):\n",
    "                        if ci == cj and not j in d and not i in d.values():\n",
    "                            d[j] = i\n",
    "                            continue\n",
    "                chars2_length = [len(c) for c in chars2_sorted]\n",
    "                char_maxl = max(chars2_length)\n",
    "                chars2_mask = np.zeros((len(chars2_sorted), char_maxl), dtype='int')\n",
    "                for i, c in enumerate(chars2_sorted):\n",
    "                    chars2_mask[i, :chars2_length[i]] = c\n",
    "                chars2_mask = Variable(torch.LongTensor(chars2_mask))\n",
    "            \n",
    "            if parameters['char_mode'] == 'CNN':\n",
    "\n",
    "                d = {}\n",
    "\n",
    "                ## Padding the each word to max word size of that sentence\n",
    "                chars2_length = [len(c) for c in chars2]\n",
    "                char_maxl = max(chars2_length)\n",
    "                chars2_mask = np.zeros((len(chars2_length), char_maxl), dtype='int')\n",
    "                for i, c in enumerate(chars2):\n",
    "                    chars2_mask[i, :chars2_length[i]] = c\n",
    "                chars2_mask = Variable(torch.LongTensor(chars2_mask))\n",
    "\n",
    "\n",
    "            targets = torch.LongTensor(tags)\n",
    "\n",
    "            #we calculate the negative log-likelihood for the predicted tags using the predefined function\n",
    "            if use_gpu:\n",
    "                neg_log_likelihood = model.neg_log_likelihood(sentence_in.cuda(), targets.cuda(), chars2_mask.cuda(), chars2_length, d)\n",
    "            else:\n",
    "                neg_log_likelihood = model.neg_log_likelihood(sentence_in, targets, chars2_mask, chars2_length, d)\n",
    "            loss += neg_log_likelihood.data[0] / len(data['words'])\n",
    "            neg_log_likelihood.backward()\n",
    "\n",
    "            #we use gradient clipping to avoid exploding gradients\n",
    "            torch.nn.utils.clip_grad_norm(model.parameters(), gradient_clip)\n",
    "            optimizer.step()\n",
    "\n",
    "            #Storing loss\n",
    "            if count % plot_every == 0:\n",
    "                loss /= plot_every\n",
    "                print(count, ': ', loss)\n",
    "                if losses == []:\n",
    "                    losses.append(loss)\n",
    "                losses.append(loss)\n",
    "                loss = 0.0\n",
    "\n",
    "            #Evaluating on Train, Test, Dev Sets\n",
    "            if count % (eval_every) == 0 and count > (eval_every * 20) or \\\n",
    "                    count % (eval_every*4) == 0 and count < (eval_every * 20):\n",
    "                model.train(False)\n",
    "                best_train_F, new_train_F, _ = evaluating(model, train_data, best_train_F,\"Train\")\n",
    "                best_dev_F, new_dev_F, save = evaluating(model, dev_data, best_dev_F,\"Dev\")\n",
    "                if save:\n",
    "                    print(\"Saving Model to \", model_name)\n",
    "                    torch.save(model.state_dict(), model_name)\n",
    "                best_test_F, new_test_F, _ = evaluating(model, test_data, best_test_F,\"Test\")\n",
    "\n",
    "                all_F.append([new_train_F, new_dev_F, new_test_F])\n",
    "                model.train(True)\n",
    "\n",
    "            #Performing decay on the learning rate\n",
    "            if count % len(train_data) == 0:\n",
    "                adjust_learning_rate(optimizer, lr=learning_rate/(1+decay_rate*count/len(train_data)))\n",
    "\n",
    "    print(time.time() - tr)\n",
    "    plt.plot(losses)\n",
    "    plt.show()\n",
    "\n",
    "if not parameters['reload']:\n",
    "    #reload the best model saved from training\n",
    "    model.load_state_dict(torch.load(model_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Model Testing\n",
    "\n",
    "This is where we provide our readers with some fun, they can try out how the trained model functions on the sentences that you throw at it. Feel free to play around.\n",
    "\n",
    "\n",
    "##### LIVE: PRODUCTION!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Prediction:\n",
      "word : tag\n",
      "John : PER\n",
      "lives : NA\n",
      "in : NA\n",
      "New : LOC\n",
      "York : NA\n",
      "and : NA\n",
      "works : NA\n",
      "for : NA\n",
      "European : ORG\n",
      "Union : NA\n",
      "\n",
      "\n",
      "Donald : PER\n",
      "is : NA\n",
      "the : NA\n",
      "president : NA\n",
      "of : NA\n",
      "USA : LOC\n",
      "\n",
      "\n",
      "Zhaoshiyan : NA\n",
      "is : NA\n",
      "from : NA\n",
      "Chongqing : LOC\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "model_testing_sentences = ['John lives in New York and works for European Union',\n",
    "                           'Donald is the president of USA',\n",
    "                          'Zhaoshiyan is from Chongqing']\n",
    "\n",
    "#parameters\n",
    "lower=parameters['lower']\n",
    "\n",
    "#preprocessing\n",
    "final_test_data = []\n",
    "for sentence in model_testing_sentences:\n",
    "    s=sentence.split()\n",
    "    str_words = [w for w in s]\n",
    "    words = [word_to_id[lower_case(w,lower) if lower_case(w,lower) in word_to_id else '<UNK>'] for w in str_words]\n",
    "    \n",
    "    # Skip characters that are not in the training set\n",
    "    chars = [[char_to_id[c] for c in w if c in char_to_id] for w in str_words]\n",
    "    \n",
    "    final_test_data.append({\n",
    "        'str_words': str_words,\n",
    "        'words': words,\n",
    "        'chars': chars,\n",
    "    })\n",
    "\n",
    "#prediction\n",
    "predictions = []\n",
    "print(\"Prediction:\")\n",
    "print(\"word : tag\")\n",
    "for data in final_test_data:\n",
    "    words = data['str_words']\n",
    "    chars2 = data['chars']\n",
    "\n",
    "    d = {} \n",
    "    \n",
    "    # Padding the each word to max word size of that sentence\n",
    "    chars2_length = [len(c) for c in chars2]\n",
    "    char_maxl = max(chars2_length)\n",
    "    chars2_mask = np.zeros((len(chars2_length), char_maxl), dtype='int')\n",
    "    for i, c in enumerate(chars2):\n",
    "        chars2_mask[i, :chars2_length[i]] = c\n",
    "    chars2_mask = Variable(torch.LongTensor(chars2_mask))\n",
    "\n",
    "    dwords = Variable(torch.LongTensor(data['words']))\n",
    "\n",
    "    # We are getting the predicted output from our model\n",
    "    if use_gpu:\n",
    "        val,predicted_id = model(dwords.cuda(), chars2_mask.cuda(), chars2_length, d)\n",
    "    else:\n",
    "        val,predicted_id = model(dwords, chars2_mask, chars2_length, d)\n",
    "\n",
    "    pred_chunks = get_chunks(predicted_id,tag_to_id)\n",
    "    temp_list_tags=['NA']*len(words)\n",
    "    for p in pred_chunks:\n",
    "        temp_list_tags[p[1]]=p[0]\n",
    "        \n",
    "    for word,tag in zip(words,temp_list_tags):\n",
    "        print(word,':',tag)\n",
    "    print('\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Xuezhe Ma and Eduard Hovy. 2016. ** End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF .** In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: LongPapers). Association for Computational Linguistics, Berlin, Germany ** (https://arxiv.org/pdf/1603.01354.pdf) **\n",
    "\n",
    "2) Official PyTorch Tutorial : [** Advanced: Making Dynamic Decisions and the Bi-LSTM CRF **](http://pytorch.org/tutorials/beginner/nlp/advanced_tutorial.html#sphx-glr-beginner-nlp-advanced-tutorial-py)\n",
    "\n",
    "3) [** Sequence Tagging with Tensorflow **](https://guillaumegenthial.github.io/sequence-tagging-with-tensorflow.html)  using bi-LSTM + CRF with character embeddings for NER and POS by Guillaume Genthial\n",
    "\n",
    "4) Github Repository - [** Reference Github Repository **](https://github.com/jayavardhanr/End-to-end-Sequence-Labeling-via-Bi-directional-LSTM-CNNs-CRF-Tutorial)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}