Natural_Language_Processing_Fundamentals_in_Python_2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Natural Language Processing Fundamentals in Python\n",
    "## Named Entity Recognition?\n",
    "\n",
    "* NLP task to identify important named entities in the text\n",
    "    * People, places, organizations\n",
    "    * Dates, states, works of art\n",
    "    * ... and other categories!\n",
    "* Can be used alongside topic identification\n",
    "    * ... or on its own!\n",
    "* Who? What? When? Where?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "wiki_text_bug.txt       wiki_text_debugging.txt wiki_text_malware.txt\r\n",
      "wiki_text_computer.txt  wiki_text_exception.txt wiki_text_program.txt\r\n",
      "wiki_text_crash.txt     wiki_text_hopper.txt    wiki_text_reversing.txt\r\n",
      "wiki_text_debugger.txt  wiki_text_language.txt  wiki_text_software.txt\r\n"
     ]
    }
   ],
   "source": [
    "#you can see that which article in  the file\n",
    "!ls data/NLP/Wikipedi_article\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### nltk and the Stanford CoreNLP Library\n",
    "* The Stanford CoreNLP library:\n",
    "* Integrated into Python via nltk\n",
    "* Java based\n",
    "* Support for NER as well as coreference and dependency trees\n",
    "\n",
    "```python\n",
    "import nltk\n",
    "\n",
    "In [2]: sentence = '''In New York, I like to ride the Metro to visit MOMA \n",
    "                      and some restaurants rated well by Ruth Reichl.'''\n",
    "\n",
    "In [3]: tokenized_sent = nltk.word_tokenize(sentence)\n",
    "In [4]: tagged_sent = nltk.pos_tag(tokenized_sent)\n",
    "In [5]: tagged_sent[:3]\n",
    "Out[5]: [('In', 'IN'), ('New', 'NNP'), ('York', 'NNP')]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[('In', 'IN'), ('New', 'NNP'), ('York', 'NNP')]\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "\n",
    "sentence = '''In New York, I like to ride the Metro to visit MOMA \n",
    "                      and some restaurants rated well by Ruth Reichl.'''\n",
    "\n",
    "tokenized_sent = nltk.word_tokenize(sentence)\n",
    "#adding a tag for proper name , pronoun, adjective...\n",
    "tagged_snt =nltk.pos_tag(tokenized_sent)\n",
    "print(tagged_snt[:3])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(S\n",
      "  In/IN\n",
      "  (GPE New/NNP York/NNP)\n",
      "  ,/,\n",
      "  I/PRP\n",
      "  like/VBP\n",
      "  to/TO\n",
      "  ride/VB\n",
      "  the/DT\n",
      "  (ORGANIZATION Metro/NNP)\n",
      "  to/TO\n",
      "  visit/VB\n",
      "  (ORGANIZATION MOMA/NNP)\n",
      "  and/CC\n",
      "  some/DT\n",
      "  restaurants/NNS\n",
      "  rated/VBN\n",
      "  well/RB\n",
      "  by/IN\n",
      "  (PERSON Ruth/NNP Reichl/NNP)\n",
      "  ./.)\n"
     ]
    }
   ],
   "source": [
    "# return the sentence as tree\n",
    "print(nltk.ne_chunk(tagged_snt))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(NE Uber/NNP)\n",
      "(NE Beyond/NN)\n",
      "(NE Apple/NNP)\n",
      "(NE Uber/NNP)\n",
      "(NE Uber/NNP)\n",
      "(NE Travis/NNP Kalanick/NNP)\n",
      "(NE Tim/NNP Cook/NNP)\n",
      "(NE Apple/NNP)\n",
      "(NE Silicon/NNP Valley/NNP)\n",
      "(NE CEO/NNP)\n",
      "(NE Yahoo/NNP)\n",
      "(NE Marissa/NNP Mayer/NNP)\n"
     ]
    }
   ],
   "source": [
    "file_name = 'data/NLP/New_articles/uber_apple.txt'\n",
    "with open(file_name, 'r') as file:\n",
    "    ## Tokenize the article into sentences: sentences\n",
    "    sentences = nltk.sent_tokenize(file.read())\n",
    "    # Tokenize each sentence into words: token_sentences\n",
    "    token_sentences = [nltk.word_tokenize(sent) for sent in sentences]\n",
    "    # Tag each tokenized sentence into parts of speech: pos_sentences\n",
    "    pos_sentences = [nltk.pos_tag(sent) for sent in token_sentences]\n",
    "    # Create the named entity chunks: chunked_sentences\n",
    "    chunk_sentences = nltk.ne_chunk_sents(pos_sentences, binary=True)\n",
    "    \n",
    "    for sent in chunk_sentences:\n",
    "        #print(sent)\n",
    "        for chunk in sent:\n",
    "            #The hasattr() method returns true if an object has the given named attribute and false \n",
    "            #if it does not.\n",
    "            if hasattr(chunk, 'label')  and chunk.label() == 'NE':\n",
    "                print(chunk)\n",
    "                      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['GPE', 'PERSON', 'ORGANIZATION']\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAADuCAYAAAAQjk2ZAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VNX9//HXZ5YsEDIJmyAio6KoECUBN0RR0Vprq8W1tlV/2mqtWm2r1ViVXq22dFFb/VptbWvBtWpV1Ki1VUERrbKIG7Lv+zpkss/M+f1xJxAIEBJmcu7MfJ6Pxzwkk5k77yB5z5l7zz1XjDEopZTyLp/tAEoppXZPi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTxOi1oppTwuYDuAUjvlhIqB7kAIKADygPzkf3f8cxCIA027uTUC1cBmYBNQjRMxnfcDKdVxYoz+W1WdwAn5gX5Af2D/5H/3BXrgFnLzrTR5S/cgIg5E2Fbczf9dAyxP3pZt/bMTaUhzHqV2Ke1FLSL7AQ8Ch+PuankF+BkwApgILAQKgVeMMTe2eN5XgTuBYqAemAP8zBizNPn9ALAaeMQYc0uL500Ciowxw5NfDwd+b4w5SUROAm40xnxdRB4FyltELQEKjTH7tNjWLOALY8xFya8fBI7HHcUdkMwEcBfw9eTP8JyI5AG/Bb4BJIAvgGuMMcuT2zHAvcaYG5Jf35jM7LT7L9hLnNA+wCDgUOAgYADbirkv4LcXbq+tZ1t5LwS+3HpzIqttBlPZL62jFhER4HngIWPM2SLiB/4C3A1UAe8mS7MQmCkiLxhj3hORIcADwFnGmNnJbZ0FhIGlyc1/BbcoLxCRn5vt33F6i8gZxpjXdpXNGHNZi5w+YBIwocV9h+G+sZwoIl2NMTXGmGuS3wvjlvLQFo//eovN/wroBhxijImLyGXA8yJyTDJnA3COiPzaGLN+T/4uvaRsfFkfYEjzrSie6PX+0uUn4r7ZZaueydvQVt9xQptx/y02l/ds4FOcyMLODKiyV7o/Xp4C1BtjHgVIltZPgEXA280PMsbUicjHuB+NAW4GftVc0snHvLTDti8C/gj8EDgWeL/F934H3Abssqh38HNgvTHmry3u+zbwGHAYcBbw1J5sSES6AJcBBxhj4snsj4rI5bh/H28CMdw3rJ8At+5hRivKxpcVA0fj/h0fm/xzr5aPifp9TCvI/2J4fUM2F/XulADHJG/buAU+I3mbBnyIE1nU6elUxkt3UQ8Gpre8wxizRUSWAgOb7xORUuBg4J0Wz/v9rjaaHIGPBn6A+0tyEdsX9fvAGBE5GfcA0i6JyNHA94GKHb51IXAa7kf5a9nDosb9uZYaY7bscP803J/rzeTXDwKfiMhv93C7aVc2vsyPm/FY3NI5Fnc3Rpuzg+4rLdnwxKo16Q2YeUpw35xP2XqPE1oHfAj8D/ff+we6/1u1Jd1FLcDOdoI333+CiHyCW4bjjDGt9vWJSA/ccusC/MUY83vc/cFvG2NqReRfwO0i8pPmEWzSXbij6pt3GU6kCHfU/D1jzMYW9x8FrDPGLBGR5cDfRaTUGLMpBT8zsPUNawJwHVC3B9tNubLxZQIcibsb6TTcYi7qyLY+yc87ol6krsCYwhRGzEa9gDOTN4A6nNBU3E+Yb+OOumO2wilvSndRfw6c2/IOESnGPcC0gG37qA8BpiT3UX+cfF4FMMsYswEY2nzALbmZi4DjRWRx8usewMnAf5tfxxjzloj8Erd8duUB4CVjzJs73H8RcGiL7Rcnf46/0rb5wAAR6WaMaTmarwBe3uGxf8D9WPzoHmw3JcrGl/XGLebTgVOBPinZsEjoyeKiqZdHqkekZHu5o/nT4ejk11Gc0BTc0v4PTmSmtWTKM9Jd1G8C40TkEmPMhOTBxHuAfwC1zQ8yxswVkV/jjn4vwp0x8YKIfNBiP3UX2Fr0I4H+xpiG5H2XJZ+3taiT7gYexj1Kvx0ROQ93NHnsDvf7gPOBI4wxK5L3nYw7Om+zqI0xNSIyHrhXRK5K7pe/JJn/rR0eu1FEngG+B/y9rW13RNn4siBwAm4xfwX3Z5Z0vNY/QsWFl0d2u6dJta0I+Gry9huc0DLcN/iJwNs4kSab4ZQdaS1qY4wRkTHAn0Tkdtx9na/iHrw7boeHPwzcKCIHGGM+FZHrgQki0g3YgDvb4xfAOcBbzSWdNBH4rYjk7/D6r4rIul3Euxu3PD90J6dsdQuwormkk94BDheRvsaYVXvwo9+Cu499rogkcGcCjNlhZkqze3D3gadM2fiyAO4I7QJgDO685LTb5PMNXRIILBsQi/XvjNfLEf2Bq5O3LTih13D/vb+KE4lYTaY6jZ7wkiWSBwJPwj0IOgZ3KlmnO7WmdvJ9a9ePsvHaOaYJd0rpk8C/cPSjTDbTos5gZePLfLi7NS7E/aSxz+6fkX5+Y1bOWLysj0/XkelMtcCLuAfG/4MTibfxeJVhtKgzUNn4sgG4+7UvA/azHKeV36xdP/1rNbXDbOfIUatxR9kTcCKzbIdRqaFFnSGS+53PBq7AnUrn2RHrgKam919ZvmrHYxCq880CHgIex4nU2A6jOk6L2uPKxpf1xT2x50rc9TK8z5iGd5euqC1JJDrlIKZq02bcKaAP4kQW2A6j2k+L2qPKxpeNAK7HPTAYtByn3b4T2TK5cuNmPajoLQncZRUeAN7QZV4zhxa1x5SNLzsNd/2PjC65gkRizkdLlg+ynUPt0hzctXL+rqewe58WtQckT+U+C7egj7IcJ2UmrFz9ZXlD46G2c6jdWoF7gtlfcCL1tsOondOitig59/kC3BOAhliOk3Ll9fXvTFi19kTbOdQeWY276uTDOJHath6sOpcWtQXJgr4U9wzGgW08PGOJMZs/WrKsIN9QYDuL2mNrcc+q/ZPOFPEOz07xylZl48tOx5029TeyuKQBjEjJ0926zbCdQ7VLb9xdIYtwQlclL6GmLNMRdScpG192OO66Hl+1naUz9YjHZ0xaumLHtb5V5vgMuAEn8obtILlMizrNysaX9cS99uOVZPY1AzvGGPPq8lUr+sdinjuDUrXLq7iF/aXtILlIizpNysaX5eHOg74VCFmOY9VXamon36MLNWWDGO4qlw5OZIPtMLlEizoNysaXnYS7dvVBlqN4gt+Y5TMXL+snaVoHW3W6jcCPcSKP2Q6SK7SoU6hsfFkR7oGYq9BS2s49a9bN+Eptne6rzi6vAT/AiSyzHSTbaVGnSPKMwkeAAbazeFG4sWnqyytW6WW6sk81UAk8pKekp48W9V4qG19WjDub4/u2s3iaMfVTlq6oDyUSJbajqLR4B/g+TmSe7SDZSOdR74Wy8WVn4F6IV0u6LSIFfykp/sR2DJU2JwKf4ISusx0kG+mIugPKxpfl446ir7GdJZMUJhKzP1yy/DDbOVTavQxcpjNDUkdH1O1UNr7sQOA9tKTbrc7nO2xWft4c2zlU2n0DmIUT0nVeUkSLuh0OuedHZwDTAb3MVAfdV1qyxnYG1Sn6AW/hhG7GCekMqL2kuz72QLiyygfcDVT6uyz4vLDf4zEJ1B1pO1cmEmM2TVu8rEse5NvOojrNy8AlOJHNtoNkKi3qNoQrq0qBp4DTW94fCE37qKDPi93FF9OTWtrppg2b3r94S7VeUzG3zAbOxIkssh0kE2lR70a4supg4HXgwJ0/IhHP6/nfqXk93z5YxPTpzGyZrGcsPv3tZSt091HuWQuchRP5n+0gmUaLehfClVXDcRei6dXmg6WxtqDvCx8GimdWiFCc9nCZzpjE68tXru4Xi+9rO4rqdHW4u0Gesx0kk+jBxJ0IV1adCrzNnpQ0gMnrUr/ywpNq5t3WFKs54B1jaEprwEwn4vtjaclc2zGUFYXAMzihm2wHySQ6ot5BuLLqQmACkNfRbfjy1i4p3G/CSslbf6yIrvmxMwFjls1YvGw/Xagpp/0ZuBonkrAdxOu0qFsIV1Zdi3tl5pR80vB3WfhFQb/HG32B2qGp2F62uW/Nuo9Pra3Tv5vcNh64XMt697Sok8KVVXcAY9Ox7UBo+kcFfV4oEV/s4HRsvy3L/7ac6o+rCRQHOPhuN0IsGmPZQ8toWt9EsGeQ/a/eH3/X1tc12DRlE+teXgdAr2/0onRkKYmmBEv/uJSmTU10P6U7PUb3AGDFoyvofkp3CgcU7lGugxobp764YrUu1KQm4J7JqGW9C7qPmvSWNEAsMuyo6Jw7D2pYN3qKMbIqXa+zK6UjSwnfEN7uvvVV6yk6rIhDfnMIRYcVsa5qXavnxaIx1k5cy4G3H8hBYw9i7cS1xGviRD+LUhguZOAvB7Jp0iYA6pbWgWGPSxpgQTBYvsUnkb364VQ2uAR4FCekfbQLOf8XE66s+hFpLOltfL7G9aeNjM65s6Rpc8VkY9iS/td0dR3UtdVoecvMLZSMdBeyKxlZwpYZreNEP4tSNLiIQFEAf1c/RYOLqP60GvELiaYEJrHt09ja59fSe0zv9gUTKXwkFNKFmhS4Zf0PLeudy+m/lHBl1Xdw90l3HhMsrF91wSh3hsiBk42hsVNfPykWiREsCQIQLAkS2xJr/ZhNMYLdg1u/DpYGiW2KUTS4iFgkxsI7F9Lzaz3ZMnMLheFCgqXBVttoyzPFRT06/lOoLHMxblnrAeYdBGwHsCVcWXUm8A8szTow8aIedUuvHCV565YW7jdhuS9v3XFemyGyq+MX4hf6X9XffUzMsPiexex//f6semoVTRuaKDm+hOLyPZtOXuvzHf5ZXt68IY2NVvbfK8+5GFgO/Nx2EC/JyRF1uLJqJPAsHnijMo299q9deMOIuiVXzk7Eus7srNcNhAI0bXanezdtbiJQ3PqvItg9SNPGbVPCmzY1ESjd/nEb3tpAyfEl1M2vcwv86v6se6n1/u7dubd7Safvt1eedgtO6HLbIbwk54o6XFl1BPAK7sR7z4jXHXh4zbzby+tWXDDNJAJpv0pG8dBiNk9x18jZPGXzTkfARUOKiH4WJV4T33oQsWhI0bbMNXGqZ1VTcnwJicbE1n9Niab2HbyfVpA/uBE7u4CUZz2MExptO4RX5NT0vHBlVQkwAzjAdpbdSyTyer41Na/nmweJmL57u7VlDy2j5ssaYtEYgeIAvb/Zm+JhxSx7cBlNG5sIdg/S/5r+BIoC1C2qY+PbG+l3eT8ANr2ziXWvtJied0Lp1u2uenIVxRXFdD20K4nGBEv+uITYphjdT+5Oj9Pat+v55+s3fnBRdfTYvf1ZVVaJACNwIl/YDmJbzhR1uLJKgIm4i5pnBmmqK+jz4oeB0PShIoRsx0mnXrHYtLeWrRxuO4fynMXAMTiRtbaD2JRLuz5uIpNKGpIzRM4fVTPv9nis5iBrM0Q6wzq/v2JlwK/7qtWOwsCEXJ8JkhNFHa6sGoW78H9GMvGu3euWXjGqZsGNa+INvaYaQ/Z9DBLx3a8LNamdOx24wXYIm7J+10e4sqoPMBPImvWi/YWLZhfs93i9L1BTbjtLKgWMWTJj8bL9daEmtRNNuPurp9kOYkNWF3W4ssoPvIV7KfusEyieOa2g7/PF4ms6xHaWVLl/zbpZJ9fqZc7UTs0HynEiUdtBOlu27/q4jiwtaYDYlvLh0Tl3DGxYe9oUY3wrbedJhQdKQzn3S6j22EDgT7ZD2JC1I+pwZVV/4AugqK3HZgVpqs/vM/GDYGhaeUbPEDGmduqS5bFuxuiVctSunIMTecF2iM6UzSPq+8mVkgYwwYKGVeedFJ13eyJWM3CyMTTYjtQhIl3+XlI8y3YM5Wl/wAl1sR2iM2VlUYcrq84Cvmk7hxXxrqV1S78/qmbBjeviDb3fy8QZIk8Vd+tuO4PytP2BW22H6ExZt+sjXFlVhLvLo7/tLF7gL1w8u2C/x+t8gWiF7Szt8c8Vq+Yf3tg00HYO5VmNwBCcSNqXW/CCbBxR34GW9FbxuvBhNfNuq6hb8a3pJhGcYzvPnrqve+kK2xmUp+UBD9gO0VmyakQdrqwaCMzGA6vieVMikddj0vt5vf4bFkn0s51md8SY9dMXLwsFof2LXKtcci5O5HnbIdIt20bUt6AlvRs+X+OGU46PzrmjR+OmoyYbg2cvg2VEej7frWiG7RzK8+7MhdPLs6aow5VVA3AXHVdtMcGChtXnjorOHWti0YM9O0PkkZLirPn3qdJmMDkwcSCbfhFuRj8mt0+iS0ndsu+Nqlnws3Xx+n08N0Nkjd9fsdrvX207h/K8rJ8BkhX7qMOVVfsCC4F821kyma9wyZzC/R6L+gLRYbazNDurOjr57vUbR9nOoTzvDJzI67ZDpEu2jKh/hpb0XkvUDRhUM++2YXXLv+2ZGSKvFXUdYDuDygi32Q6QThk/og5XVvUAlgI5daZS+hmT12Py1LxebwwQSexnM8mDq9d+cmJd/RE2M6iMMBIn8p7tEOmQDSPqb6ElnQYijRtOOj46585ejZuOnmwMm20lub+0ZIut11YZ5TLbAdIlG4r6u7YDZDUTyG9Yfc6o6NyxxKKHTDKG+s6OMCcvWB4Vqe7s11UZ53yckKcuWp0qGV3U4cqqgwC9IGpnSHQpqVt2+Uk1C27aEK/vM8UY2nep8b0h0vXRkC7UpNpUTJZO1cvooga+bTtArjFN3fvVLvrxyNrFV89LNHXrtKttPFncLXOXblWd6RLbAdIh04v6O7YD5KpE/f6DaubfOrxu+bdnmETel+l+vajfV/ZlXnBBul9HZbzTcEJZc9m9Zhlb1OHKquHAINs5cl2s+oiK6Jw7BtWvOWOqMb7l6Xyt+0pL0rp9lRX8wPm2Q6RaxhY1cK7tAKqZSNPGUSPcGSLHTDaGTel4lfcLCw6PQSwd21ZZ5XTbAVItk4v6BNsB1A5MIL9h9ZhR0bljfemYIWJEer1Y1HV6KrepstJJOKE82yFSKSOLOlxZlQ8Mt51D7UKiSyg5Q2RjvL7Pe6mcIfLn0uxfKU3tta7AcbZDpFJGFjUwDD1l3PNMU/d9axf9+PjaxdfMT9UMkdV+f8Vav39tKralslpWfeLO1KI+3nYAtecS9f0PcWeIfGeGSeTN3quNiQT+rzT0RYqiqex1ou0AqaRFrTpNrLqsIjrnjkPr13xtr2aIvKILNam2HWU7QCp1qKhFZB8ReVJEForIdBF5X0TGiMhJIhIRkZkiMltEfpF8fPP9H7e4nboXuUfsxXOVVSJNG08cEf3yzt6Nm46dbAwb27uFJpEDphQWfJqOdCprlOCEetsOkSrtLmoREeBF4B1jzIHGmGG4CyM1r7D2rjGmHPdg33dFZFiL+4e2uP23I4HDlVX9gF4dea7ykkBew+pvjorO/YU/Vn3oJGOoa8+z7y8t8exlxJRnZM15Fh0ZUZ8CNBpjHm6+wxizxBiz3RWBjTE1wHTgoL2L2MoBKd6esilRGKpb/v9Oqpl/8+Z4fd89XkNkdl5waI1INN3xVEY7xHaAVOnIhWAHA21edFREeuAumPRL3BHwCSLycYuHnGuM6cgpwZ4t6tiWdayvupd4dBMiPoqGnk7x8LOJ11WzfuJviG1ZQ6B4H3p+sxJ/QVGr50c/fZPI+08DEDruWxSVjcbEmlj7/C+JV6+nW/mZdKs4E4ANrz9At/KvkbdPqt8H7TCx0r61i67v6ytYNq9wv8civuCW3U+/FCkaHyqecvXmyMhOiqgyT9YU9V4fTBSRB0Vkloh8lLzrBBGZCbwBjDPGfJ68f8ddHx1dt8G7B5J8fkpP/h79rniYPhf/nuoZVTSuX8qWD56lIHwk/a58hILwkWz54NlWT43XVRN570n6XHwvfS65j8h7TxKvj1K3aAZ5fQbS9/L/o3qWe6WhxrULwZisKemWEvX9D66Z//Phdcsunmniebud3fFEcZEu1KR2J6eL+nOgovkLY8w1wGi27Td+1xhTbowZ1nL3SAp5dsGVQFF38vsMBMCX34Vgj/7EqzdQO/9/dB0yGoCuQ0ZTO++DVs+tXzSDgnA5/sJu+AuKKAiXU79wOuLzY5oaIBHf+tjN7z5OaGR2r0cViw4uj86947D6NWdONca3dGeP2eL3l80NBhd1djaVMfrbDpAqHSnqt4ACEflhi/s68wor+3Tia3VYLLKGxjULyd93EPGazQSKugNumSdqWl8sJVa9AX9xz61f+7v1IFa9gYIDyonXbGbVhBsIHXMutfP+R94+Awl069FpP4s9Ik0bTxgR/fLOPo0bj3vHGDbs+Ij7upfstMSVwj1DMSu0ex+1McaIyDeB+0TkJmAdUAPc3MZTd9xHfZcx5rn2vj7g+Sk3icY61r3wK7qPvgJf/p6+h7W+dqUIiM9Pr7N+5j4iHmPNM2Ppfe7tbHzzEeJb1tF1yGi6HHxMCtN7USCvYc3ZJzasO31L4b7/nOwvmn20CIUAUwsLDotBLNCx4y0qu+VuUQMYY1bhTsnbmUk7efwkIFX7EwtStJ20MPEY6174FV0PP4kug9zp3v6uJcSiGwkUdScW3Yiva0mr5wW69aR+6bapwfHqDQT3L9vuMdUzqygaMpqGFV8i/iA9z76Z1Y/fmANFnZQoKK5bfukoCWxeVdh/wjRf/srjEyK9Xy7q+tGYaE1WneCgUiJrijoTz0yMt/0QO4wxbHjtjwR79Kf46DFb7+8y8BhqPnsTgJrP3qTLwNbFWnBABXWLZxKvj7oHERfPpOCArYcC3Pvmf0TXIadgYg3J4bZgYk3p/8E8xsRK+tYuuu6E2sXXLkg0FX/0cEmo8y4LpjJJ1hS1GNP6I7eXhSur3sGjC67UL/+cNU/cTLBX2C1SoPTES8jbdxDrJ44jtmUdgeJe9Dz7FvyF3WhYNY/ox6/R44zrAIh+8gaR990ZIaHjLqDoiNO2bnvjm4/Q5eBjKdi/DBNrZO2/fkm8egNF5WdQPOwbnf6zekl58Uv/ea7h6VP8gt92FuU5AZyIZwd3eyoTi/ot4GTbOZRd+TTWf9//6vQfBF4JFUvtENt5lGfl4UQy/mNnJh6Ayfh3R9Vxg2TporGBx5Ye5/v8CJ/o4lxqt2qzoaQhM4taL8WUYwLEmi7x/2fatYEXC7tL9VA8fHaq8pTW82AzVCYWdVa8Q6q2hWXVstsDjy88yffx4X4xWXXFDtUptKgtWm07gEofH4n4+f5J038aeM7Xm80VItlzdpnqdFrUFnV0jRDlYfuyftXPg0/M+arvo0EBSRxtO4/KClrUFi20HUClijFn+abOuCn4z3g/1g8Toa/tRCqrdPgqQl6TiUWtI+oM15PN624OPP3FN/3vHRCU+LC2n6FUh3ze9kMygxa16jSn+qbNuiXwVO2BsmqYCKNs51FZL2uKOuNOeAEIV1ZtALrbzqHaFiK6+YbAs7Mu9L/dP19iB9rOo3JKH5zIGtshUiETR9QAc3GvHqM8aqTv089uDTweOVSWVejoWVmwIVtKGjK3qN9Di9pzulJXfV3ghY8v9r/Ru4s06mndyqas2e0BmVvUbwI32A6hXOUyb84vghPWHikLykW8uWCWyjnv2w6QSpla1O/inqEYtB0kVxXQUPcD/yvTvx94tbSb1A0GBtnOpFQLb9gOkEoZeTARIFxZ9R4wwnaOXHO4LF7wi+CEFUfLl0eKpOxiEEqlUg3QHSfSaDtIqmTqiBrcazdqUXeCILHGy/yvT/thYGJRqdQcAWTf5c8zzJz1cS58rm7r1ws3Jbjz5HwuOTKPC5+rZfFmQ7hEeOa8LpQWSqvnj/+4kbvedXvsthPyuHRoHg0xw9lP17J8i+Hqo/K4+qg8AK58uY4fDs+jvG/GLPc9KZtKGjK/qG+zHSKbHSgrl4wNTFh8ou/TwT4x+qboIYN6+vn4qiIA4glDv3ujjDk0yLgpDYw+IEDlyHzGTWlg3JQGfnPa9lev21hnuGNyA9OuLEKAYX+JctagIO8ujTGsr59Xv5NPxZ9ruPqoPGatjpMwZFJJA/zbdoBUy+Sinop7Ln/rCxCqDvMTj13kf2va9YHn83oSKRdhgO1MavfeXBTnoO4+BpT4mDgnxqRL3QsqX3pkkJPG1/Kb07Z//L/nxzjtwADdkyPt0w4M8Pr8GCUFUBeDWIsLm93+dgMPf93Tlyndmawr6ky8ZiIAi8ed2QA8ZTtHtthP1q18OHjv5Ln5l2y4K/josb0kUiFC68/MynOe/qyJi4a4x9XXRBP07eb+Wvft5mNtTevLSa6oTtA/tO1Xf79iHyuqE5x2UIDV0QTH/LWGm47P56U5TQzr62ffbhlVEzNxInNth0i1TB5RA/wd+KHtEJlKSCTG+KZMvzH4LH3ZUCHCvrYzqfZpjBtemhPj16Pz9/g5O5s/IEDAJzx5rjsab4obTn+8lpcu6sJP/13P0kiCS44MctYgz0+0+pvtAOmQUW+VO1o87sxpwCe2c2Safdi49r7gg5Pn5V+y8t68h4/aVzYcJXph2Iz02rwYFX197FPk/irvU+RjVbU7il5VnaB319a/4vsV+1gW2TbSXr4l0WrU/KePGrn0yCDvL4uT54d/nlfIXe80pPEnSYl64AnbIdIho4s66e+2A2QGY87w/W/m5Lwff/BB/rWlY/zvjQpIYj/bqdTeearFbg+Asw4JMH6WexGk8bOaOHtQ6w/Npw8M8MbCGJvqDJvqDG8sjHH6wG2P21RneGVejEuODFLbZPAJiEC99y+C9zxOJGvWoG4p03d9ADwO/BbIsx3Ei0rZsvFngWc+Pc//zoA8iZXbzqNSp7bJ8J+Fcf789cKt91WOzOOC5+r428wm9g8Jz57v7sqYtjLOw9Ma+etZhXQvFG4/MZ+jHokCMPbE/K0HFgHunNzAbSfkIyKcPjDAgx81UvZQDVcN8/yvWFbu9oAMPuGlpXBl1TPA+bZzeMko38ef3hp4ovpgWTFMhD122MZvAAAKvElEQVTfgalUZloAHIwTyfxC24lsGFED/A4tarpRE7k+8Pys7/r/27dAmsps51GqE43L1pKGLBlRA4QrqyYCZ9nOYcPRMvuLscHHNg6WxRUidLGdR6lOtgg4BCfi/b3oHZQtI2qA24FvQG7M/e1Cfc3VgYkzLvP/u2dXqT/cdh6lLLo7m0sasmhEDbmxr/oIWTBvbPCxVcNk7lARim3nUcqyhcCgbC/qbBpRA4wFzoHsmhOcT2P95f7Xpl8VeLk4JLVlwMG2MynlEVk/moYsG1EDhCurJgAX286RCofIskVjA48tHeH77AifUGo7j1IeMwcYkgtFnW0jaoBfAOdCZh5UCxBr+q7/v9N+FHihoDvVQ0U4wHYmpTzq2lwoacjCETVAuLLqp8A9tnO0xwBZvfz2wOMLTvbNPNwvppftPEp53LM4kQva+yQRiQOf4g5SZwOXGmNqW9zf7GljzDgRmQT0xT09vRG4whjzcXJblwM/AQzuWd63GmMmiogAtwKXJr+3ArjWGPN58nmLgenGmHOTX58HfN0Y8/92lTsbR9QAfwAuAI6xHWR3fCTi5/knT/9p4DnfPmyqEEFP6VaqbVtwC7Ij6owxQwFE5AngKuDelvfvxHeMMdNE5DLcczZOE5H9cMu4whgTEZEioHmAdQ3uRU2OTL4JfAV4SUQGG2Pqk48Znvx6jy7Cmw1rfbSyeNyZCeBy3HdAz+nLhtX3Bx+YPDf/kjW/DT5ydB/ZNFwkO/9fKJUGN+FEVqRgO+8CA9vx+PeBfsk/9waqgSiAMSZqjFmU/N7NwI+MMbXJ772Bu37+d1ps6/fAz/f0hbO2HBaPO/ML4C7bObYx5hu+qdPfybv+g6n5P+p5lv/9UQFJ6LKiSrXPO8Bf9nYjIhIAzmDb7o5CEfm4xe3CnTztq8CLyT/PAtYAi0TkURH5RnK7xUBXY8yCHZ47DRjc4utngAoR2aM3imzd9dFsHHAecIStAD2IrL858PTnY/xTwkGJD7OVQ6kssAm4dC9PFS8UkY+Tf36XbQs57W7XxxMi0hV32m8FgDEmLiJfBY4CRgP3icgw3N0oOyO4+6ubxXF3o9wCvNZW6Kw8mNhSuLKqAvcjS6cu/TXaN33WLYEnaw6SVcNFdGU/pVLgmziRiXuzARGJGmOK2nH/JOBG3BH0OOAAY8w5O3nccOBRY0yZiCwDRhljFrb4/gRgsjHmb8mDicOBCPAF8BBwxO4OJmbtro9mi8edOQO4rjNeq5hoxAn84505+Zcu+FvePUcO9K0aoSWtVEr8YW9Lem8YY5pwL6Z9rIgcJiL7ikhFi4cMBZYk//w74H4RKQQQkVOBkcCTO9nmfcCP23r9bN/1AcDicWf+OVxZNQy4Ih3bP9732ee3Bh7fdJgsHSbCiel4DaVy2IfATWl+jZa7RABeN8ZUtnyAMaZORO7BHWHfCfxeRPbFnbq3DncGCcADQCnwaXLa32rgbGNM3U5e92+4bwC7lfW7PpqFK6vygMnAsanYXlfqotcGXpx5qf+N3l2kYVAqtqmUamUzUI4TWWw7iE05U9QA4cqqfXGPvvbt6DbKZd6cscHH1g6V+UNF6Ja6dEqpHcSBs3EiVbaD2JZTRQ0QrqwaAbxNOw4uFtBQd4W/asaVgaqSblI3uO1nKKVS4CqcyJ9th/CCnCtqgHBl1ZVAm/8ADpMlC8YGJiw/xjf7SJ9Q0gnRlFKuX+FEbrUdwitysqgBwpVVDu4CTtsJEmu81P/vadcEJnYtleiRnZ9MqZw3ASdyqe0QXpKzRQ0Qrqz6A3A9wAGycunYwGOLRvk+GewT09NyNKVy1X+AM3EiTbaDeElOTM/bjZ+M8H0W+0PwwZN6EakQYX/bgZTKYdOAc7WkW8vpETUATsgHTGD7BVOUUp1rKnAGTmSL7SBelPVnJrbJiSRw1419ynYUpXLUZOB0Leld06IGcCJx3Mt3/cNyEqVyzX9wR9JR20G8THd97MgJ3YW7ILhSKr2qcPdJN9gO4nVa1DvjhK7GPV9fP3EolR7PABfjRDx5cQ+v0SLaGSfyJ+B83MVWlFKpdSfwLS3pPacj6t1xQiOBl3BXwlJK7Z164DKcyNO2g2QaLeq2OKFDgH8BQ2xHUSqDrcZd+P9/toNkIt310RYnMhd3adQn23qoUmqnZgFHa0l3nI6o28MJXYt7TbSg7ShKZYgJwNU4kRrbQTKZFnV7OaHjgGfZdtl4pVRr1cAPcSJP2A6SDbSoO8IJ9QaeAE61HUUpD5qGO6tjge0g2UL3UXeEE1kLfAW4BtCPdEq5DPB7YISWdGrpiHpvOaGDcE89H2k5iVI2rQS+hxN53XaQbKQj6r3ljhxGATegJ8io3GOAh4HDtaTTR0fUqeSEDgXGA0fbjqJUJ5gNXIETec92kGynI+pUciJfAscBPwQ2WE6jVLo0Ag4wVEu6c+iIOl2cUHfgl8APAL/lNEqlyrvAlclBieokWtTp5oSG4q7EpwcbVSabD1TiRP5lO0gu0qLuLE7o28DvgH1tR1GqHTbgrnb3kF7L0B4t6s7khAqBa4GbAL3SufKyBuB+4G6cSMR2mFynRW2DEyoCrgduBEosp1GqpQTu9UNvxYkssR1GubSobXJCJcBPgR8D3SynUbktjrtC5N04kTm2w6jtaVF7gRPqAfwMd1pfseU0Krc0Ao8Dv8aJzLcdRu2cFrWXOKFuwPeB64Cw3TAqy0WBvwD34kRW2A6jdk+L2ouckB84B3e3yLGW06jssgj3lO+/4kQ22g6j9owWtdc5oWNxC/sc9MQZ1TEJ4HXgQeB1nEjCch7VTlrUmcIJ9QMuBS4DBlpOozLDeuDvwMM4kUW2w6iO06LORE7oRNzCPh/oajmN8pYE8Dbu4mDP4EQaLOdRKaBFncnc+dgXApcDIyynUXZNw51e9zROZJXtMCq1tKizhXsBg/OTtwrLaVTnmI9bzk/q3OfspkWdjdzSPhcYAxwDiN1AKoU+B14BnseJfGg7jOocWtTZzgn1Bc5O3kYBhXYDqXZqBCbhlvMrelAwN2lR5xInlI+73OppyVs5Otr2otW40+leBt7AiUQt51GWaVHnMifUExjNtuLe326gnLUcmLz15kTmWs6jPEaLWm3jhMK4Z0I238qBPJuRstRiti/mhXbjKK/Tola75u4qKWdbcR+DrkHSXouA6cCMrTcnss5uJJVptKhV+zihEDAYGLLDf/exGcsDosA84Eu2L+XNVlOprKBFrVLD3d89OHk7EBiAO/oeAPSyFyylmoCFwNxWNyey0mYwld20qFX6OaEubF/cA3BH4D13uJVgZxZKDNgIrAJWAiuS/215WwGs1QWNlA1a1Mo73OVdeyRvPYEioCB5K9zJnwtxiz3exq0RqMbdPdH8383J2yad/qa8TotaKaU8zmc7gFJKqd3TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/TolZKKY/7/+u1NU70YAWtAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from collections import  defaultdict\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "file_name = 'data/NLP/New_articles/uber_apple.txt'\n",
    "with open(file_name, 'r') as file:\n",
    "    ## Tokenize the article into sentences: sentences\n",
    "    sentences = nltk.sent_tokenize(file.read())\n",
    "    # Tokenize each sentence into words: token_sentences\n",
    "    token_sentences = [nltk.word_tokenize(sent) for sent in sentences]\n",
    "    # Tag each tokenized sentence into parts of speech: pos_sentences\n",
    "    pos_sentences = [nltk.pos_tag(sent) for sent in token_sentences]\n",
    "    # Create the named entity chunks: chunked_sentences\n",
    "    chunk_sentences = nltk.ne_chunk_sents(pos_sentences, binary=False)\n",
    "    ner_categories = defaultdict(int)\n",
    "\n",
    "    # Create the nested for loop\n",
    "    for sent in chunk_sentences:\n",
    "        for chunk in sent:\n",
    "            if hasattr(chunk, 'label'):\n",
    "                ner_categories[chunk.label()] += 1\n",
    "\n",
    "    # Create a list from the dictionary keys for the chart labels: labels\n",
    "    labels = list(ner_categories.keys())\n",
    "    print(labels)\n",
    "\n",
    "    # Create a list of the values: values\n",
    "    values = [ner_categories.get(l) for l in labels]\n",
    "\n",
    "    # Create the pie chart\n",
    "    plt.pie(values, labels=labels, autopct='%1.1f%%', startangle=140)\n",
    "\n",
    "    # Display the chart\n",
    "    plt.show()\n",
    "   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### SpaCy\n",
    "* NLP library similar to gensim, with different implementations\n",
    "* Focus on creating NLP pipelines to generate models and corpora\n",
    "* Open-source, with extra libraries and tools\n",
    "    * Displacy\n",
    "    \n",
    "    \n",
    "```python\n",
    "\n",
    "In [1]: import spacy\n",
    "In [2]: nlp = spacy.load('en')\n",
    "In [3]: nlp.entity\n",
    "Out[3]: <spacy.pipeline.EntityRecognizer at 0x7f76b75e68b8>\n",
    "In [4]: doc = nlp(\"\"\"Berlin is the capital of Germany; \n",
    "                  and the residence of Chancellor Angela Merkel.\"\"\")\n",
    "In [5]: doc.ents\n",
    "Out[5]: (Berlin, Germany, Angela Merkel)\n",
    "In [6]: print(doc.ents[0], doc.ents[0].label_)\n",
    "Berlin GPE\n",
    "\n",
    "```\n",
    "\n",
    "**Why use SpaCy for NER(Name Entity Recognation)?**\n",
    "* Easy pipeline creation\n",
    "* Different entity types compared to nltk\n",
    "* Informal language corpora\n",
    "    * Easily find entities in Tweets and chat messages\n",
    "* Quickly growing!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Berlin GPE\n",
      "Germany GPE\n",
      "Angela Merkel PERSON\n"
     ]
    }
   ],
   "source": [
    "#if \"en\" dosen't work for you\n",
    "#sudo python -m spacy download en\n",
    "nlp = spacy.load(\"en\")\n",
    "nlp.entity\n",
    "doc = nlp(\"Berlin is the capital of Germany; and the residence of Chancellor Angela Merkel.\")\n",
    "doc.ents\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.label_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uber GPE\n",
      "Uber PERSON\n",
      "Apple ORG\n",
      "Uber PERSON\n",
      "Uber PERSON\n",
      "Uber PERSON\n",
      "Travis Kalanick GPE\n",
      "Uber GPE\n",
      "Tim Cook PERSON\n",
      "Apple ORG\n",
      "Millions CARDINAL\n",
      "Uber’s ORG\n",
      "Silicon Valley’s LOC\n",
      "Yahoo ORG\n",
      "Marissa Mayer PERSON\n",
      "186 MONEY\n"
     ]
    }
   ],
   "source": [
    "file_name = 'data/NLP/New_articles/uber_apple.txt'\n",
    "with open(file_name, 'r') as file:\n",
    "    nlp = spacy.load(\"en\")\n",
    "    nlp.entity\n",
    "    ff = nlp(file.read())\n",
    "    for ent in ff.ents:\n",
    "        print(ent.text, ent.label_)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### What is polyglot?\n",
    "* NLP library which uses word vectors\n",
    "* Why polyglot?\n",
    "    * Vectors for many different languages\n",
    "    * More than 130!\n",
    "    \n",
    "* polyglot uses the language detection model\n",
    "* When we call the entities of text, we see alist of chunk with  the label, representing by the symbol startin the I \n",
    "    * I-ORG : Organizatin\n",
    "    * I-PER : Person\n",
    "    * I-LOC : Location\n",
    "\n",
    "```python\n",
    "In [1]: from polyglot.text import Text\n",
    "\n",
    "In [2]: ẗext = \"\"\"El presidente de la Generalitat de Cataluña,\n",
    "                  Carles Puigdemont, ha afirmado hoy a la alcaldesa \n",
    "                  de Madrid, Manuela Carmena, que en su etapa de \n",
    "                  alcalde de Girona (de julio de 2011 a enero de 2016) \n",
    "                  hizo una gran promoción de Madrid.\"\"\"\n",
    "\n",
    "In [3]: ptext = Text(text)\n",
    "In [4]: ptext.entities\n",
    "Out[4]: \n",
    "[I-ORG(['Generalitat', 'de']),\n",
    " I-LOC(['Generalitat', 'de', 'Cataluña']),\n",
    " I-PER(['Carles', 'Puigdemont']),\n",
    " I-LOC(['Madrid']),\n",
    " I-PER(['Manuela', 'Carmena']),\n",
    " I-LOC(['Girona']),\n",
    " I-LOC(['Madrid'])]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}