DataCamp Intermediate Importing Data in Python.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2e9b5034-c640-4244-9ec6-23e6c698a9e0",
   "metadata": {},
   "source": [
    "## Intermediate Importing Data in Python\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7603d12f-585a-4b71-b035-6803e1311738",
   "metadata": {},
   "source": [
    "## Course Description\n",
    "\n",
    "As a data scientist, you will need to clean data, wrangle and munge it, visualize it, build predictive models and interpret these models. Before you can do so, however, you will need to know how to get data into Python. In the prequel to this course, you learned many ways to import data into Python: from flat files such as .txt and .csv; from files native to other software such as Excel spreadsheets, Stata, SAS, and MATLAB files; and from relational databases such as SQLite and PostgreSQL. In this course, you'll extend this knowledge base by learning to import data from the web and by pulling data from Application Programming Interfaces— APIs—such as the Twitter streaming API, which allows us to stream real-time tweets."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eec9bc87-5769-48fc-b924-b877685f7b31",
   "metadata": {},
   "source": [
    "##  Importing data from the Internet\n",
    "Free\n",
    "0%\n",
    "\n",
    "The web is a rich source of data from which you can extract various types of insights and findings. In this chapter, you will learn how to get data from the web, whether it is stored in files or in HTML. You'll also learn the basics of scraping and parsing web data.\n",
    "\n",
    "    Importing flat files from the web    50 xp\n",
    "    Importing flat files from the web: your turn!    100 xp\n",
    "    Opening and reading flat files from the web    100 xp\n",
    "    Importing non-flat files from the web    100 xp\n",
    "    HTTP requests to import files from the web    50 xp\n",
    "    Performing HTTP requests in Python using urllib    100 xp\n",
    "    Printing HTTP request results in Python using urllib    100 xp\n",
    "    Performing HTTP requests in Python using requests    100 xp\n",
    "    Scraping the web in Python    50 xp\n",
    "    Parsing HTML with BeautifulSoup    100 xp\n",
    "    Turning a webpage into data using BeautifulSoup: getting the text    100 xp\n",
    "    Turning a webpage into data using BeautifulSoup: getting the hyperlinks    100 xp\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3e16fa3-36bf-45af-8ca7-4256cbeeaabc",
   "metadata": {},
   "source": [
    "##  Interacting with APIs to import data from the web\n",
    "0%\n",
    "\n",
    "In this chapter, you will gain a deeper understanding of how to import data from the web. You will learn the basics of extracting data from APIs, gain insight on the importance of APIs, and practice extracting data by diving into the OMDB and Library of Congress APIs.\n",
    "\n",
    "    Introduction to APIs and JSONs    50 xp\n",
    "    Pop quiz: What exactly is a JSON?    50 xp\n",
    "    Loading and exploring a JSON    100 xp\n",
    "    Pop quiz: Exploring your JSON    50 xp\n",
    "    APIs and interacting with the world wide web    50 xp\n",
    "    Pop quiz: What's an API?    50 xp\n",
    "    API requests    100 xp\n",
    "    JSON–from the web to Python    100 xp\n",
    "    Checking out the Wikipedia API    100 xp\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58aa6317-f066-45c7-91cf-9b6f3a7bd956",
   "metadata": {},
   "source": [
    "##  Diving deep into the Twitter API\n",
    "0%\n",
    "\n",
    "In this chapter, you will consolidate your knowledge of interacting with APIs in a deep dive into the Twitter streaming API. You'll learn how to stream real-time Twitter data, and how to analyze and visualize it.\n",
    "\n",
    "    The Twitter API and Authentication    50 xp\n",
    "    API Authentication    100 xp\n",
    "    Streaming tweets    100 xp\n",
    "    Load and explore your Twitter data    100 xp\n",
    "    Twitter data to DataFrame    100 xp\n",
    "    A little bit of Twitter text analysis    100 xp\n",
    "    Plotting your Twitter data    100 xp\n",
    "    Final Thoughts    50 xp \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a9e9c2-e789-4b03-a847-3033769fdbb5",
   "metadata": {},
   "source": [
    "## Importing flat files from the web\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**You're now able to import data in Python from all sorts of file types: flat files such as .txt and .csv, other file types such as pickle files, Excel spreadsheets and MATLAB files.  You've also gained valuable experience in querying relational databases to import data from them using SQL.  You have really come a ong way, congratulations.  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "However, all of these skills involve importing data from files that you have locally.  Much of the time as a Data Scientist, these skills won't be quite enough because you won't always have the data that you need.  You will need to import from the WWW.  Say , for example, you want to import Wine Quality dataset from the Machine Learning Repository hosted by the University of California, Irvine.  How do you get this file form the web?  \n",
    "\n",
    "Now you could use your favourite web browser of choice to navigateto the relevant URL, point and clickon the appropriate hyperlinks to download the file but this poses a couple of series problems.  Firstly, it isnt written in codeand so poses reproducibility issues.  If anothet Data Scientist wanted to reproduce your workflow, he would necessarily to do so outside Python code.  Secondly, it is not scalable.  If you want to download hundred or one thousand such files, it would take 100 or 1000 times as long, respectively, whereas if you wrote it in code, your workflow could scale.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "As reproducibility and scalability are situated at the very heart of Data Science, you're going to learn in this chapter how to use Python code to import and locally save datasets from the WWW.  You'll also learn how to load such datasets into Pandas dataframes directly from the web, whether they be flat files or otherwise.  Then you'll place these skills in the wider context of making HTTP request.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "In particular, you'll make HTTP GET requests, which in plain English means getting data from the web.  You'll use these new request skills to learn the basics of scraping HTML from the internet and you'll use the wonderful Python package BeautifulSoup to parse the HTML and turn it into data.  \n",
    "\n",
    "# \"urllib\" and \"request\" packages\n",
    "Now there are a number of great packages to help us import the web data: herein, you'll become familiar with the \"rullib\" and \"request\" packages.  We'll first check out the \"urllib\": This module provides a high-level interface for fetching data across the WWW.  \n",
    "\n",
    "# \"urlopen()\" function just like Python build-in \"open()\" function\n",
    "In particular, the \"urlopen()\" function is similar to the build-in function \"open()\", but accepts Universal Resource Locations (URLs) instead of filenames.  Lets now dive directly in to importing data from the web with an example, importing the Wine Quality dataset for white wine.  Dont be jealous: in the first interactive exercise, it will be your job to import the red wine dataset.  \n",
    "\n",
    "# \"urlretrieve()\" function from \"urllib.request\" sub-package\n",
    "All we have done here is imported a fnction called \"urlretrieve()\" from the \"request\" subpackage of the \"urllib\" package, we assigned the relevant URLas a string to the variable \"url\".  We than use the \"urlretrieve()\" function to write the contents of the url to a file \"winequality-white.csv\".  \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "30815f82-1dbb-400f-830b-b12f36c3c7a2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['__add__', '__class__', '__class_getitem__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'count', 'index']\n",
      "['__add__', '__class__', '__class_getitem__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmul__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', 'count', 'index']\n"
     ]
    }
   ],
   "source": [
    "print(dir(tuple))\n",
    "\n",
    "print(dir(white_quality))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "f5f77dbd-7f1f-4be0-9428-23fccdac1738",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'tuple'>\n",
      "2\n",
      "winequality-white.csv\n",
      "Date: Mon, 24 Jan 2022 15:55:25 GMT\n",
      "Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.2k-fips SVN/1.7.14 Phusion_Passenger/4.0.53 mod_perl/2.0.11 Perl/v5.16.3\n",
      "Last-Modified: Fri, 16 Oct 2009 21:36:53 GMT\n",
      "ETag: \"408ea-4761431c10740\"\n",
      "Accept-Ranges: bytes\n",
      "Content-Length: 264426\n",
      "Connection: close\n",
      "Content-Type: application/x-httpd-php\n",
      "\n",
      "\n",
      "(4898, 12)\n",
      "Index(['fixed acidity', 'volatile acidity', 'citric acid', 'residual sugar',\n",
      "       'chlorides', 'free sulfur dioxide', 'total sulfur dioxide', 'density',\n",
      "       'pH', 'sulphates', 'alcohol', 'quality'],\n",
      "      dtype='object')\n",
      "   fixed acidity  volatile acidity  citric acid  residual sugar  chlorides  \\\n",
      "0            7.0              0.27         0.36            20.7      0.045   \n",
      "1            6.3              0.30         0.34             1.6      0.049   \n",
      "2            8.1              0.28         0.40             6.9      0.050   \n",
      "3            7.2              0.23         0.32             8.5      0.058   \n",
      "4            7.2              0.23         0.32             8.5      0.058   \n",
      "\n",
      "   free sulfur dioxide  total sulfur dioxide  density    pH  sulphates  \\\n",
      "0                 45.0                 170.0   1.0010  3.00       0.45   \n",
      "1                 14.0                 132.0   0.9940  3.30       0.49   \n",
      "2                 30.0                  97.0   0.9951  3.26       0.44   \n",
      "3                 47.0                 186.0   0.9956  3.19       0.40   \n",
      "4                 47.0                 186.0   0.9956  3.19       0.40   \n",
      "\n",
      "   alcohol  quality  \n",
      "0      8.8        6  \n",
      "1      9.5        6  \n",
      "2     10.1        6  \n",
      "3      9.9        6  \n",
      "4      9.9        6  \n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from urllib.request import urlretrieve\n",
    "\n",
    "\n",
    "url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv'\n",
    "\n",
    "white_quality = urlretrieve(url, 'winequality-white.csv')\n",
    "\n",
    "\n",
    "print(type(white_quality))\n",
    "\n",
    "print(len(white_quality))\n",
    "\n",
    "print(white_quality[0])\n",
    "\n",
    "print(white_quality[1])\n",
    "\n",
    "\n",
    "\n",
    "w_quality = pd.read_csv(white_quality[0], delimiter=';') ############################################################\n",
    "\n",
    "print(w_quality.shape)\n",
    "print(w_quality.columns)\n",
    "\n",
    "print(w_quality.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "e7f2773f-2db0-4324-b880-7afd5226c48c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function urlretrieve in module urllib.request:\n",
      "\n",
      "urlretrieve(url, filename=None, reporthook=None, data=None)\n",
      "    Retrieve a URL into a temporary location on disk.\n",
      "    \n",
      "    Requires a URL argument. If a filename is passed, it is used as\n",
      "    the temporary file location. The reporthook argument should be\n",
      "    a callable that accepts a block number, a read size, and the\n",
      "    total file size of the URL target. The data argument should be\n",
      "    valid URL encoded data.\n",
      "    \n",
      "    If a filename is passed and the URL points to a local resource,\n",
      "    the result is a copy from local file to new file.\n",
      "    \n",
      "    Returns a tuple containing the path to the newly created\n",
      "    data file as well as the resulting HTTPMessage object.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(urlretrieve)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7402c42-0e36-44f8-881b-e2dbee5f5d9e",
   "metadata": {},
   "source": [
    "## Importing flat files from the web: your turn!\n",
    "\n",
    "You are about to import your first file from the web! The flat file you will import will be 'winequality-red.csv' from the University of California, Irvine's Machine Learning repository. The flat file contains tabular data of physiochemical properties of red wine, such as pH, alcohol content and citric acid content, along with wine quality rating.\n",
    "\n",
    "The URL of the file is\n",
    "\n",
    "'https://s3.amazonaws.com/assets.datacamp.com/production/course_1606/datasets/winequality-red.csv'\n",
    "\n",
    "After you import it, you'll check your working directory to confirm that it is there and then you'll load it into a pandas DataFrame.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Import the function urlretrieve from the subpackage urllib.request.\n",
    "    Assign the URL of the file to the variable url.\n",
    "    Use the function urlretrieve() to save the file locally as 'winequality-red.csv'.\n",
    "    Execute the remaining code to load 'winequality-red.csv' in a pandas DataFrame and to print its head to the shell.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "8a109602-0d61-4f3a-9457-063707323da6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   fixed acidity  volatile acidity  citric acid  residual sugar  chlorides  \\\n",
      "0            7.4              0.70         0.00             1.9      0.076   \n",
      "1            7.8              0.88         0.00             2.6      0.098   \n",
      "2            7.8              0.76         0.04             2.3      0.092   \n",
      "3           11.2              0.28         0.56             1.9      0.075   \n",
      "4            7.4              0.70         0.00             1.9      0.076   \n",
      "\n",
      "   free sulfur dioxide  total sulfur dioxide  density    pH  sulphates  \\\n",
      "0                 11.0                  34.0   0.9978  3.51       0.56   \n",
      "1                 25.0                  67.0   0.9968  3.20       0.68   \n",
      "2                 15.0                  54.0   0.9970  3.26       0.65   \n",
      "3                 17.0                  60.0   0.9980  3.16       0.58   \n",
      "4                 11.0                  34.0   0.9978  3.51       0.56   \n",
      "\n",
      "   alcohol  quality  \n",
      "0      9.4        5  \n",
      "1      9.8        5  \n",
      "2      9.8        5  \n",
      "3      9.8        6  \n",
      "4      9.4        5  \n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from urllib.request import urlretrieve\n",
    "\n",
    "\n",
    "url = 'https://s3.amazonaws.com/assets.datacamp.com/production/course_1606/datasets/winequality-red.csv'\n",
    "\n",
    "urlretrieve(url, 'winequality-red.csv')  ############################################################################\n",
    "\n",
    "\n",
    "r_quality = pd.read_csv('winequality-red.csv', sep=';')\n",
    "print(r_quality.head())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f38e7a72-0afa-488a-8f6b-c352e9893629",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import package\n",
    "from urllib.request import urlretrieve\n",
    "\n",
    "# Import pandas\n",
    "import pandas as pd\n",
    "\n",
    "# Assign url of file: url\n",
    "url = 'https://s3.amazonaws.com/assets.datacamp.com/production/course_1606/datasets/winequality-red.csv'\n",
    "\n",
    "#####################################################################################################################\n",
    "# Save file locally\n",
    "urlretrieve(url, 'winequality-red.csv')\n",
    "\n",
    "# Read file into a DataFrame and print its head\n",
    "df = pd.read_csv('winequality-red.csv', sep=';')\n",
    "print(df.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd0610f4-4b1c-4e16-8a5f-13b6bfdabd86",
   "metadata": {},
   "source": [
    "## Opening and reading flat files from the web\n",
    "\n",
    "You have just imported a file from the web, saved it locally and loaded it into a DataFrame. If you just wanted to load a file from the web into a DataFrame without first saving it locally, you can do that easily using pandas. In particular, you can use the function pd.read_csv() with the URL as the first argument and the separator sep as the second argument.\n",
    "\n",
    "The URL of the file, once again, is\n",
    "\n",
    "'https://s3.amazonaws.com/assets.datacamp.com/production/course_1606/datasets/winequality-red.csv'\n",
    "\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Assign the URL of the file to the variable url.\n",
    "    Read file into a DataFrame df using pd.read_csv(), recalling that the separator in the file is ';'.\n",
    "    Print the head of the DataFrame df.\n",
    "    Execute the rest of the code to plot histogram of the first feature in the DataFrame df.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "3a776c51-4ab1-498d-87dd-3656dfc8b7aa",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   fixed acidity  volatile acidity  citric acid  residual sugar  chlorides  \\\n",
      "0            7.4              0.70         0.00             1.9      0.076   \n",
      "1            7.8              0.88         0.00             2.6      0.098   \n",
      "2            7.8              0.76         0.04             2.3      0.092   \n",
      "3           11.2              0.28         0.56             1.9      0.075   \n",
      "4            7.4              0.70         0.00             1.9      0.076   \n",
      "\n",
      "   free sulfur dioxide  total sulfur dioxide  density    pH  sulphates  \\\n",
      "0                 11.0                  34.0   0.9978  3.51       0.56   \n",
      "1                 25.0                  67.0   0.9968  3.20       0.68   \n",
      "2                 15.0                  54.0   0.9970  3.26       0.65   \n",
      "3                 17.0                  60.0   0.9980  3.16       0.58   \n",
      "4                 11.0                  34.0   0.9978  3.51       0.56   \n",
      "\n",
      "   alcohol  quality  \n",
      "0      9.4        5  \n",
      "1      9.8        5  \n",
      "2      9.8        5  \n",
      "3      9.8        6  \n",
      "4      9.4        5  \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEcCAYAAAAoSqjDAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMCwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8/fFQqAAAACXBIWXMAAAsTAAALEwEAmpwYAAAbQklEQVR4nO3df5gdVZ3n8feHBORHMCEEWyBZGjWiDBkQGoiDs9sBmQmghp0HWGf4kSDzZFUUZwxqHHWG4ZHZOI7L4uiCeUATXCQyyI/IDwUDLasOCOFXElAIGCQhgISQpYOAge/+UafP3DS3O7e7b3V133xez3OfvnWq6tQ59yb3c+tU3SpFBGZmZgA7VN0AMzMbORwKZmaWORTMzCxzKJiZWeZQMDOzzKFgZmaZQ8FahqQDJN0v6UVJ50i6RNKXStjOGknvb3Kd/bZVUkh6RyPLmg3F2KobYNZEnwVuj4hDqm7IQEXERwezrKRO4P9ExOQSmmXbIe8pWCvZD1hVdSPMRjOHgrUESbcBM4BvSOqW9E5JiyR9Oc3/nKS7JI1N0x+TtErSzpJ2kDRf0mOSNki6StLEmrpPl/REmveFbbTjBEn3Sfp/kp6UdF6v+e+T9AtJL6T5c1J5bmua/oyk9ZKekvSRXnUskvRlSbsBNwP7pD53S9pH0kuS9qxZ/lBJv5O04+BeXdueOBSsJUTE0cD/BT4REeMi4pFei3wVeAX4oqSpwD8Bp0XEy8AngROB/wLsA2wEvgkg6UDgYuD0NG9PoL+hms3AGcAE4ATgY5JOTHXtR/Eh/q/AXsAhwP29K5A0EzgXOBaYCtQ9fhERm4HjgKdSn8dFxFNAF3BKzaKnA0si4g/9tNsMcCjYdiIiXqf4sD4HWAr8c0Tcl2Z/FPhCRKyNiFeA84CT0l7FScANEXFHmvcl4PV+ttMVESsi4vWIeBC4kiJsAP4K+ElEXBkRf4iIDRFxf51qTgG+ExEr0wf/eQPs7mLgNABJY4C/BL47wDpsO+VQsO1GRKwBbgfaSXsCyX7AtWlI5wXgYeA1oI1i7+DJmjo2Axv62oakIyXdnoZrNlEEzqQ0ewrwWANN3WqbwBMNrFPreuBASftT7G1siohfDrAO2045FGy7IekE4L3AMorhpB5PAsdFxISax84RsQ5YT/Fh3lPHrhRDSH35HsWeyJSIGA9cAqhmO29voKlbbRP4T/0s+4bLHKchsaso9hZOx3sJNgAOBdsuSJoEXAr8NTAb+KCk49PsS4AL0pg/kvaSNCvNuxr4QDpAvBNwPv3/v9kdeD4iXpZ0BMWQUY8rgPdLOkXSWEl7SjqkTh1XAXMkHZhC6B/62d4zwJ6SxvcqvxyYA3wIh4INgEPBthcLgesj4qaI2ACcBVyaztK5iOLb/S2SXgTuBI4EiIhVwNkUewDrKQ5Cr+1nOx8Hzk/1/D3FBzyprt8CxwPzgOcpDjIf3LuCiLgZ+F/AbcDq9LeuiPgVxXGLx9Pw1z6p/OcUxz7ujYiBDj/Zdky+yY5Za0qn6X4vIi6tui02ejgUzFqQpMOBWymObbxYdXts9PDwkVmLkbQY+AnwNw4EGyjvKZiZWeY9BTMzyxwKZmaWjepLZ0+aNCna29urbkZTbd68md12263qZjSd+zW6tGK/WrFPMLh+LV++/LmI2KvevFEdCu3t7dxzzz1VN6Opurq66OzsrLoZTed+jS6t2K9W7BMMrl+S+vztioePzMwscyiYmVnmUDAzs8yhYGZmmUPBzMwyh4KZmWUOBTMzyxwKZmaWjeofr9nAtM+/sbJtL5rZer8kNWtF3lMwM7PMoWBmZplDwczMMoeCmZllDgUzM8scCmZmljkUzMwscyiYmVnmUDAzs8yhYGZmmUPBzMwyh4KZmWWlhoKkNZJWSLpf0j2pbKKkWyU9mv7ukcol6euSVkt6UNKhZbbNzMzeaDj2FGZExCER0ZGm5wPLImIqsCxNAxwHTE2PucDFw9A2MzOrUcXw0SxgcXq+GDixpvzyKNwJTJC0dwXtMzPbbikiyqtc+g2wEQjgWxGxUNILETEhzRewMSImSLoBWBARP0vzlgGfi4h7etU5l2JPgra2tsOWLFlSWvur0N3dzbhx40qpe8W6TaXU24j9x48prV9VKvP9qlIr9qsV+wSD69eMGTOW14zebKXsm+y8LyLWSXoLcKukX9XOjIiQNKBUioiFwEKAjo6O6OzsbFpjR4Kuri7K6tOcim+y02rvFZT7flWpFfvVin2C5ver1OGjiFiX/j4LXAscATzTMyyU/j6bFl8HTKlZfXIqMzOzYVJaKEjaTdLuPc+BPwNWAkuB2Wmx2cD16flS4Ix0FtJ0YFNErC+rfWZm9kZlDh+1AdcWhw0YC3wvIn4k6W7gKklnAU8Ap6TlbwKOB1YDLwFnltg2MzOro7RQiIjHgYPrlG8AjqlTHsDZZbXHzMy2zb9oNjOzzKFgZmaZQ8HMzDKHgpmZZQ4FMzPLHApmZpY5FMzMLHMomJlZ5lAwM7PMoWBmZplDwczMMoeCmZllDgUzM8scCmZmljkUzMwscyiYmVnmUDAzs8yhYGZmmUPBzMwyh4KZmWUOBTMzyxwKZmaWORTMzCxzKJiZWeZQMDOzzKFgZmaZQ8HMzDKHgpmZZQ4FMzPLSg8FSWMk3SfphjS9v6S7JK2W9H1JO6XyN6Xp1Wl+e9ltMzOzrQ3HnsKngIdrpr8CXBgR7wA2Amel8rOAjan8wrScmZkNo1JDQdJk4ATg0jQt4Gjg6rTIYuDE9HxWmibNPyYtb2Zmw0QRUV7l0tXA/wB2B84F5gB3pr0BJE0Bbo6IgyStBGZGxNo07zHgyIh4rledc4G5AG1tbYctWbKktPZXobu7m3HjxpVS94p1m0qptxH7jx9TWr+qVOb7VaVW7Fcr9gkG168ZM2Ysj4iOevPGNqVVdUj6APBsRCyX1NmseiNiIbAQoKOjIzo7m1b1iNDV1UVZfZoz/8ZS6m3Eopm7ldavKpX5flWpFfvVin2C5vertFAAjgI+JOl4YGfgzcBFwARJYyNiCzAZWJeWXwdMAdZKGguMBzaU2D4zM+ultGMKEfH5iJgcEe3Ah4HbIuJU4HbgpLTYbOD69HxpmibNvy3KHNsyM7M3qOJ3Cp8DPi1pNbAncFkqvwzYM5V/GphfQdvMzLZrZQ4fZRHRBXSl548DR9RZ5mXg5OFoj5mZ1edfNJuZWeZQMDOzzKFgZmaZQ8HMzDKHgpmZZcNy9pHZinWbKvlF9ZoFJwz7Ns1GM+8pmJlZ5lAwM7PMoWBmZplDwczMMoeCmZllDgUzM8scCmZmljkUzMwscyiYmVnmUDAzs8yhYGZmmUPBzMwyh4KZmWUOBTMzyxwKZmaWORTMzCxzKJiZWeZQMDOzzKFgZmaZQ8HMzDKHgpmZZQ2FgqRljZSZmdnoNra/mZJ2BnYFJknaA1Ca9WZg35LbZmZmw2xbewr/HVgOvCv97XlcD3yjvxUl7Szpl5IekLRK0j+m8v0l3SVptaTvS9oplb8pTa9O89uH2DczMxugfkMhIi6KiP2BcyPibRGxf3ocHBH9hgLwCnB0RBwMHALMlDQd+ApwYUS8A9gInJWWPwvYmMovTMuZmdkw6nf4qEdE/KukPwHaa9eJiMv7WSeA7jS5Y3oEcDTwV6l8MXAecDEwKz0HuBr4hiSleszMbBiokc9cSd8F3g7cD7yWiiMiztnGemMohpveAXwT+CpwZ9obQNIU4OaIOEjSSmBmRKxN8x4DjoyI53rVOReYC9DW1nbYkiVLGuzq6NDd3c24ceNKqXvFuk2l1NuItl3gmd8P/3an7Tu+1PrLfL+q1Ir9asU+weD6NWPGjOUR0VFvXkN7CkAHcOBAv7VHxGvAIZImANdSHJsYkohYCCwE6OjoiM7OzqFWOaJ0dXVRVp/mzL+xlHobMW/aFr62otF/bs2z5tTOUusv8/2qUiv2qxX7BM3vV6O/U1gJvHWwG4mIF4DbgfcCEyT1fDpMBtal5+uAKQBp/nhgw2C3aWZmA9doKEwCHpL0Y0lLex79rSBpr7SHgKRdgGOBhynC4aS02GyKM5kAlqZp0vzbfDzBzGx4Nbo/f94g6t4bWJyOK+wAXBURN0h6CFgi6cvAfcBlafnLgO9KWg08D3x4ENs0M7MhaPTso58OtOKIeBB4T53yx4Ej6pS/DJw80O2YmVnzNBQKkl6kOJ0UYCeK00s3R8Sby2qYmZkNv0b3FHbveS5JFL8pmF5Wo8zMrBoDvkpqFK4D/rz5zTEzsyo1Onz0FzWTO1D8buHlUlpkZmaVafTsow/WPN8CrKEYQjIzsxbS6DGFM8tuiJmZVa/Rm+xMlnStpGfT4weSJpfdODMzG16NHmj+DsUvjvdJjx+mMjMzayGNhsJeEfGdiNiSHouAvUpsl5mZVaDRUNgg6TRJY9LjNHyxOjOzltNoKHwEOAV4GlhPccG6OSW1yczMKtLoKannA7MjYiOApInAv1CEhZmZtYhG9xT+uCcQACLieepc7M7MzEa3RkNhB0l79EykPYXhv42WmZmVqtEP9q8B/y7p39L0ycAF5TTJzMyq0ugvmi+XdA9wdCr6i4h4qLxmmZlZFRoeAkoh4CAwM2thA750tpmZtS6HgpmZZQ4FMzPLHApmZpY5FMzMLPMP0CrQPv/GPufNm7aFOf3MNzMrk/cUzMwscyiYmVnmUDAzs8yhYGZmmUPBzMwyh4KZmWWlhYKkKZJul/SQpFWSPpXKJ0q6VdKj6e8eqVySvi5ptaQHJR1aVtvMzKy+MvcUtgDzIuJAYDpwtqQDgfnAsoiYCixL0wDHAVPTYy5wcYltMzOzOkoLhYhYHxH3pucvAg8D+wKzgMVpscXAien5LODyKNwJTJC0d1ntMzOzNxqWYwqS2inu6XwX0BYR69Osp4G29Hxf4Mma1damMjMzGyaKiHI3II0DfgpcEBHXSHohIibUzN8YEXtIugFYEBE/S+XLgM9FxD296ptLMbxEW1vbYUuWLCm1/WVYsW5Tn/PadoFnfj+MjRkmVfVr2r7jS62/u7ubcePGlbqNKrRiv1qxTzC4fs2YMWN5RHTUm1fqtY8k7Qj8ALgiIq5Jxc9I2jsi1qfhoWdT+TpgSs3qk1PZViJiIbAQoKOjIzo7O8tqfmn6u7bRvGlb+NqK1rskVVX9WnNqZ6n1d3V1MRr/DW5LK/arFfsEze9XmWcfCbgMeDgi/mfNrKXA7PR8NnB9TfkZ6Syk6cCmmmEmMzMbBmV+dTsKOB1YIen+VPZ3wALgKklnAU8Ap6R5NwHHA6uBl4AzS2ybmZnVUVoopGMD6mP2MXWWD+Dsstpj26f+LlPeDH1d6nzNghNK3a5ZWfyLZjMzyxwKZmaWORTMzCxzKJiZWeZQMDOzzKFgZmaZQ8HMzDKHgpmZZQ4FMzPLHApmZpY5FMzMLHMomJlZ5lAwM7PMoWBmZplDwczMMoeCmZllDgUzM8scCmZmljkUzMwscyiYmVnmUDAzs8yhYGZmmUPBzMwyh4KZmWUOBTMzyxwKZmaWORTMzCxzKJiZWeZQMDOzbGxZFUv6NvAB4NmIOCiVTQS+D7QDa4BTImKjJAEXAccDLwFzIuLestpmVrb2+TdWtu01C06obNs2+pW5p7AImNmrbD6wLCKmAsvSNMBxwNT0mAtcXGK7zMysD6WFQkTcATzfq3gWsDg9XwycWFN+eRTuBCZI2rustpmZWX3DfUyhLSLWp+dPA23p+b7AkzXLrU1lZmY2jEo7prAtERGSYqDrSZpLMcREW1sbXV1dzW5a6eZN29LnvLZd+p8/Wrlfw6cZ/ye6u7tH5f+t/rRin6D5/RruUHhG0t4RsT4NDz2bytcBU2qWm5zK3iAiFgILATo6OqKzs7PE5pZjTj8HIedN28LXVlSW1aVxv4bPmlM7h1xHV1cXo/H/Vn9asU/Q/H4N9/DRUmB2ej4buL6m/AwVpgObaoaZzMxsmJR5SuqVQCcwSdJa4B+ABcBVks4CngBOSYvfRHE66mqKU1LPLKtdZmbWt9JCISL+so9Zx9RZNoCzy2qLmZk1xr9oNjOzzKFgZmaZQ8HMzDKHgpmZZSPrBGszG7JmXIxv3rQt/f6eph5fiK81eE/BzMwyh4KZmWUOBTMzyxwKZmaWORTMzCxzKJiZWeZQMDOzzKFgZmaZQ8HMzDKHgpmZZQ4FMzPLtttrHzXj+jBmZq3GewpmZpY5FMzMLHMomJlZ5lAwM7PMoWBmZplDwczMsu32lFQza64qT/P2rUCbx3sKZmaWORTMzCxzKJiZWeZQMDOzzKFgZmaZQ8HMzLIRdUqqpJnARcAY4NKIWFBxk8zM+tSKp+GOmFCQNAb4JnAssBa4W9LSiHio2paZ2UjXyIfzvGlbmONL5m/TSBo+OgJYHRGPR8SrwBJgVsVtMjPbrigiqm4DAJJOAmZGxF+n6dOBIyPiE72WmwvMTZMHAL8e1oaWbxLwXNWNKIH7Nbq0Yr9asU8wuH7tFxF71ZsxYoaPGhURC4GFVbejLJLuiYiOqtvRbO7X6NKK/WrFPkHz+zWSho/WAVNqpienMjMzGyYjKRTuBqZK2l/STsCHgaUVt8nMbLsyYoaPImKLpE8AP6Y4JfXbEbGq4mZVoVWHxtyv0aUV+9WKfYIm92vEHGg2M7PqjaThIzMzq5hDwczMMofCCCJpgqSrJf1K0sOS3lt1m4ZK0t9KWiVppaQrJe1cdZsGS9K3JT0raWVN2URJt0p6NP3do8o2DlQfffpq+jf4oKRrJU2osImDUq9fNfPmSQpJk6po21D01S9Jn0zv2SpJ/zyUbTgURpaLgB9FxLuAg4GHK27PkEjaFzgH6IiIgyhOIPhwta0akkXAzF5l84FlETEVWJamR5NFvLFPtwIHRcQfA48Anx/uRjXBIt7YLyRNAf4M+O1wN6hJFtGrX5JmUFz94eCI+CPgX4ayAYfCCCFpPPCfgcsAIuLViHih0kY1x1hgF0ljgV2Bpypuz6BFxB3A872KZwGL0/PFwInD2aahqteniLglIrakyTspfjM0qvTxXgFcCHwWGJVn2PTRr48BCyLilbTMs0PZhkNh5Ngf+B3wHUn3SbpU0m5VN2ooImIdxbeW3wLrgU0RcUu1rWq6tohYn54/DbRV2ZgSfAS4uepGNIOkWcC6iHig6rY02TuBP5V0l6SfSjp8KJU5FEaOscChwMUR8R5gM6NvKGIraXx9FkXg7QPsJum0altVnijO7x6V30DrkfQFYAtwRdVtGSpJuwJ/B/x91W0pwVhgIjAd+AxwlSQNtjKHwsixFlgbEXel6aspQmI0ez/wm4j4XUT8AbgG+JOK29Rsz0jaGyD9HdKu+0ghaQ7wAeDUaI0fM72d4svJA5LWUAyJ3SvprZW2qjnWAtdE4ZfA6xQXyRsUh8IIERFPA09KOiAVHQOM9ntJ/BaYLmnX9M3lGEb5wfM6lgKz0/PZwPUVtqUp0s2uPgt8KCJeqro9zRARKyLiLRHRHhHtFB+kh6b/d6PddcAMAEnvBHZiCFeDdSiMLJ8ErpD0IHAI8E/VNmdo0l7P1cC9wAqKf2+j9lIDkq4E/h04QNJaSWcBC4BjJT1KsWc0qu4W2EefvgHsDtwq6X5Jl1TayEHoo1+jXh/9+jbwtnSa6hJg9lD27nyZCzMzy7ynYGZmmUPBzMwyh4KZmWUOBTMzyxwKZmaWORTMzCxzKJjZiCDp3ZIuSZeP/1jV7dleORQMAEnnpHs4XCHpF02q8zxJ5zahnrrtqa2/Z5l0T4qPD2Ibu6SLiY1pdLkhbGtQ66V1m/LeDKTuXq/zTpLuSFe9rV3mEklH9bVeIyLi4Yj4KHAKcFR/27PyOBSsx8eBYyPi1IgYUdcnaqQ9NctMoOjLQH2E4voxrw1guQFvK13uY+Jg1pO0Q5nvTYOv86sU9434b71mTae4zPaQSPoQcCNw0za2ZyVxKBjpMgZvA25Wcae07lR+eLr71s6Sdkt3dToozTtN0i/TZRC+1fMNW9IXJD0i6WfAAX1s7zpJy1N9c3vNOyNt8wFJ301l3TXz69Zfs8wC4O2pXV+VdL6kv6lZ7gJJn6rTrFOpuW6RpC9J+rWkn6m4Y9y5dZbbalt99U1Se6rrcmAlxT0zBrPelF6vxRteq0Zf68G+zsl16XXoWfbdwCMR8Vq99VI/fiVpUZp3haT3S/q5ijvWHdFTV0QsjYjjauvvvT0rWUT44QfAGmBSet5dU/5linsifBP4fCp7N/BDYMc0/b+BM4DDKK5xtCvwZmA1cG6dbU1Mf3eh+LDbM03/EcWdvib1Wq47/e2z/ppl2oGVNdtqB+5Nz3cAHuvZXs0yOwFP10wfDtwP7ExxDaBHgXPrLLfVtvrqW1rudWD6UNbr1c+6r1Ujr/VQXuc0fwzwu5rpT1PsQdVdL/VjCzAtvQfLKa7XI4pLq1+X6ukEvg58Czi7r+35Ue7D43S2LecDdwMvU9xaE4qrnR4G3F2MhrALxSWjJwLXRrqypqSlfdR5jqT/mp5PAaYCG4CjgX+LiOcAIqL3Hab+tMH6s4hYI2mDpPdQ3ADnvojY0GuxScALNdNHAddHxMvAy5J+2MdyjfbtaeCJiOhveGWg623rteqv3sO3sW6/r3MUewSvSto9Il4E/hw4k+JYQF/r/SYiVqTyVRS3MA1JKyhCg4joArp6d6DO9qxEDgXblj2BccCOFN+cN1N8w1scEVvdu7d2mKYvkjoprib63oh4SVJXqrdMlwJzgLdSfEPt7fcNtqHf5bbRt83NXm9bSn6t30QRmLsCEyLiKfV/X5dXap6/XjP9Oo19Dr2J4ouJlczHFGxbvgV8ieLuW19JZcuAkyS9BUDSREn7AXcAJ6o4Q2d34IN16hsPbEwfUu+iOEDZ4zbgZEl79tTba91G6n+RYsin1rUUNzs/HPhx7xUiYiMwRlLPB+bPgQ+mYynjKG42U2+53tvqr2/9tbHR9Wpt67Xqr94hvc5pveeiuHHSDOD2RtYbrF7bs5J5T8H6JOkM4A8R8T0VB5J/IenoiLhN0heBWyTtAPyBYgz4TknfBx6gGE66u061PwI+Kulh4NfUnLESEaskXQD8VNJrwH0U3/B75t+7rfojYkM6gLkSuDkiPhMRr0q6HXgh+j676BbgfcBPIuLuNPTxIPAMxTj5pjrLbbUt4It99a2/Nja6Xq86+n2tkrqvdRNe5xkUZwgBHEdxz4yG3p9Bqt2elcz3U7CWl4LrXuDkiHi0j2UOBf42Ik5P0+MiojsNj9wBzE0felsttz2SdA0wPyIekXQvcGSZ3+Jrt1fWNuw/ePjIWpqkAynOglnWVyBA8S0XuF3/8eO1hZLupwiTH6T59ZbbrkjaieJsoUcAIuLQkgNhq+1Z+bynYGZmmfcUzMwscyiYmVnmUDAzs8yhYGZmmUPBzMwyh4KZmWUOBTMzyxwKZmaWORTMzCz7/7jfW8GPOG61AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Import packages\n",
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "\n",
    "# Assign url of file: url\n",
    "url = 'https://s3.amazonaws.com/assets.datacamp.com/production/course_1606/datasets/winequality-red.csv'\n",
    "\n",
    "#urlretrieve(url, 'winequality-red.csv')\n",
    "\n",
    "\n",
    "# Read file into a DataFrame: df\n",
    "df = pd.read_csv(url, sep=';') ######################################################################################\n",
    "\n",
    "\n",
    "# Print the head of the DataFrame\n",
    "print(df.head())\n",
    "\n",
    "# Plot first column of df\n",
    "#pd.DataFrame.hist(df.ix[:, 0:1]) ###################################################################################\n",
    "\n",
    "pd.DataFrame.hist(df.iloc[:, :1])\n",
    "\n",
    "plt.xlabel('fixed acidity (g(tartaric acid)/dm$^3$)')\n",
    "plt.ylabel('count')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2860f4aa-ae71-438c-81bb-8357ba6f4bb8",
   "metadata": {},
   "source": [
    "## Importing non-flat files from the web\n",
    "\n",
    "# Congrats! You've just loaded a flat file from the web into a DataFrame without first saving it locally using the pandas function pd.read_csv(). This function is super cool because it has close relatives that allow you to load all types of files, not only flat ones. In this interactive exercise, you'll use pd.read_excel() to import an Excel spreadsheet.\n",
    "\n",
    "The URL of the spreadsheet is\n",
    "\n",
    "'http://s3.amazonaws.com/assets.datacamp.com/course/importing_data_into_r/latitude.xls'\n",
    "\n",
    "Your job is to use \"pd.read_excel()\" to read in all of its sheets, print the sheet names and then print the head of the first sheet using its name, not its index.\n",
    "\n",
    "Note that the output of pd.read_excel() is a Python dictionary with sheet names as keys and corresponding DataFrames as corresponding values.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Assign the URL of the file to the variable url.\n",
    "#    Read the file in url into a dictionary xls using \"pd.read_excel()\" recalling that, in order to import all sheets you need to pass \"None: to the argument \"sheet_name=\".\n",
    "    Print the names of the sheets in the Excel spreadsheet; these will be the keys of the dictionary xls.\n",
    "    Print the head of the first sheet using the sheet name, not the index of the sheet! The sheet name is '1700'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "6bd4b666-9a3c-4699-b969-5d91bc6bb9ce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'dict'>\n",
      "dict_keys(['1700', '1900'])\n",
      "                 country       1700\n",
      "0            Afghanistan  34.565000\n",
      "1  Akrotiri and Dhekelia  34.616667\n",
      "2                Albania  41.312000\n",
      "3                Algeria  36.720000\n",
      "4         American Samoa -14.307000\n"
     ]
    }
   ],
   "source": [
    "url = 'http://s3.amazonaws.com/assets.datacamp.com/course/importing_data_into_r/latitude.xls'\n",
    "\n",
    "\n",
    "exc_df = pd.read_excel(url, sheet_name=None)  #######################################################################\n",
    "\n",
    "#print(exc_df.sheet_names)   # dict' object has no attribute 'sheet_names'\n",
    "print(type(exc_df))\n",
    "\n",
    "print(exc_df.keys())  ###############################################################################################\n",
    "\n",
    "\n",
    "\n",
    "#df_1700 = exc_df.parse('1700')\n",
    "df_1700 = exc_df['1700']  ###########################################################################################\n",
    "\n",
    "print(df_1700.head())\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27e1a685-9ca0-4e51-b473-b8e3c58bbb61",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "\n",
    "with open('pickled_fruit.pkl', 'rb') as file:\n",
    "    data = pickle.load(file)\n",
    "    \n",
    "print(data)\n",
    "#{'peaches',: 13, 'apples': 4, 'oranges': 11}\n",
    "\n",
    "\n",
    "\n",
    "exc_df = pd.ExcelFile('en_US.xlsx')   ###############################################################################\n",
    "\n",
    "print(exc_df.sheet_names)\n",
    "#['1960-1966', '1967-1974', '1975-2011']\n",
    "\n",
    "\n",
    "#####################################################################################################################\n",
    "df1 = exc_df.parse('1960-1966')  #sheet name, as a string\n",
    "\n",
    "df2 = exc_df.parse(1)  #sheet index, as a float\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c8f2ef7-8b40-40a7-abc5-f359b16f7715",
   "metadata": {},
   "source": [
    "## HTTP requests to import files from the web\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**Congrats on importing your first web data.  In order to import files from the web, we used the \"urlretrieve()\" function from \"urllib.request\".  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "Lets now unpack this a bit and, in the process, understand a few things about how the internet works.  URL standas for Uniform or Universal Resource Locator and all they really are are reference to web resources.  The vast majority of URLs are web addresses, but they can refer to few other things, such as file transfer protocols (FTP) and database access.  We'll currently focus on those URLs that are web addresses OR the locations of the websites.  \n",
    "\n",
    "Such a URL consistsof 2 parts, a protocal identifier \"http\" or \"https\" and a resource name such as \"datacamp.com\".  The combination of protocol identifier and resource name uniquely specifies the web address.  To explain URLs, I have introduced yet another acronym \"http\", which itself stands for  HyperText Transfer Protocol.  Wikipedia provides a great description of HTTP.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "\"The HyperText Transfer Prptocol (HTTP) is an application protocol for distributed, collaborative, hypermedia informationn systems.  HTTP is the foundation of data communication for the World Wide Web.\"  Note that HTTPS is a more secure form of HTTP.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "Each time you go to a website, you are actually sending an HTTP request to a server.  This request is known as a GET request, by far the most common type of HTTP request.  We are actually performing a GET request when using the function \"urlretrieve()\".  The ingenuity of \"urlretrieve()\" also lies in fact that it not only makes a GET request but also saves the relevant data locally.  \n",
    "\n",
    "In the following, you'll learn how to make more GET requests to store web data in your environment.  In particular, you'll figure out how to get the HTML data from a webpage.  HTML stands for HyperText Markup Language and is the standard markup language for the web.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "To extract the HTML from the Wikipedia home page, you import the necessary functions - \"urlopen\", \"Request\", specify the URL, package the GET request using the function \"Request()\", send the request and catch the response using the function \"urlopen()\".  This returns an HTTPResponse object, which has an associated \".read()\" method.  You then apply this read() method to the response, which returns the HTML as a string, which you store in the variable html.  Finally, you remember to be polite and close the response.  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "Now we are going to do the same, however here we'll use the \"Requests\" package which provides a wonderful API for making requests.  According to the \"Requests\" package website.  \"Requests allows you to send organic, grass-fed HTTP/1.1 requests, without the need for a manual labor.\"  And the following organizations claim to use the Requests internally: Her Majesty's Goverment, Amazon, Google, Twillo, NPR, Obama for America, Twitter, Sony, and Federal US Institutions that prefer to be unnamed.  Moreover, \"Requests is one of the most downloaded Python packages of all time, pulling in over 7,000,000 downloads every month.  All the cool kids are doing it.\"  \n",
    "\n",
    "\n",
    "# Lets now see the Requests at work.  \n",
    "Here, you import the package requests, specify the url, package the request, send the request and catch the response with a single function \"requests.get()\"; and finally you apply the \".text()\" method to the response which returns the HTML as a string.  \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "acdd09c8-b810-472b-864d-f2201b830bce",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'http.client.HTTPResponse'>\n",
      "<class 'bytes'>\n",
      "b'<!DOCTYPE html><html lang=en-US><head><meta charset=\"UTF-8\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><link rel=profile href=https://gmpg.org/xfn/11><link rel=manifest href=/superpwa-manifest.json><link rel=prefetch href=/superpwa-manifest.json><meta name=\"theme-color\" content=\"#D5E0EB\"><meta name=\\'robots\\' content=\\'index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1\\'><title>Develop Perfect Memory With the Memory Palace Technique - Litemind</titl'\n",
      "<!DOCTYPE html><html lang=en-US><head><meta charset=\"UTF-8\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><link rel=profile href=https://gmpg.org/xfn/11><link rel=manifest href=/superpwa-manifest.json><link rel=prefetch href=/superpwa-manifest.json><meta name=\"theme-color\" cont\n"
     ]
    }
   ],
   "source": [
    "from urllib.request import Request, urlopen\n",
    "\n",
    "\n",
    "url = 'https://en.wikipedia.org/wiki/Regular_expression'\n",
    "url = 'https://litemind.com/memory-palace/'\n",
    "\n",
    "\n",
    "request = Request(url)\n",
    "\n",
    "response = urlopen(request)   #######################################################################################\n",
    "\n",
    "html = response.read()   ############################################################################################\n",
    "\n",
    "response.close()\n",
    "\n",
    "\n",
    "print(type(response))\n",
    "print(type(html))\n",
    "\n",
    "print(html[:500])\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "import requests\n",
    "\n",
    "\n",
    "url = 'https://litemind.com/memory-palace/'\n",
    "\n",
    "r = requests.get(url)  ##############################################################################################\n",
    "\n",
    "text = r.text\n",
    "\n",
    "\n",
    "print(text[:300])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10fbf909-a25e-4141-835b-c976d0b09609",
   "metadata": {},
   "source": [
    "## Performing HTTP requests in Python using urllib\n",
    "\n",
    "Now that you know the basics behind HTTP GET requests, it's time to perform some of your own. In this interactive exercise, you will ping our very own DataCamp servers to perform a GET request to extract information from the first coding exercise of this course, \"https://campus.datacamp.com/courses/1606/4135?ex=2\".\n",
    "\n",
    "In the next exercise, you'll extract the HTML itself. Right now, however, you are going to package and send the request and then catch the response.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Import the functions urlopen and Request from the subpackage urllib.request.\n",
    "    Package the request to the url \"https://campus.datacamp.com/courses/1606/4135?ex=2\" using the function Request() and assign it to request.\n",
    "    Send the request and catch the response in the variable response with the function urlopen().\n",
    "    Run the rest of the code to see the datatype of response and to close the connection!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "7f575f34-b81b-4bdd-aa9e-b4bb408205af",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'<!doctype html><html lang=\"en\"><head><link rel=\"apple-touch-icon-precomposed\" sizes=\"57x57\" href=\"/campus/apple-touch-icon-57x57.png\"><link rel=\"apple-touch-icon-precomposed\" sizes=\"114x114\" href=\"/campus/apple-touch-icon-114x114.png\"><link rel=\"apple-touch-icon-precomposed\" sizes=\"72x72\" href=\"/cam'\n",
      "<!doctype html><html lang=\"en\"><head><link rel=\"apple-touch-icon-precomposed\" sizes=\"57x57\" href=\"/campus/apple-touch-icon-57x57.png\"><link rel=\"apple-touch-icon-precomposed\" sizes=\"114x114\" href=\"/campus/apple-touch-icon-114x114.png\"><link rel=\"apple-touch-icon-precomposed\" sizes=\"72x72\" href=\"/campus/apple-touch-icon-72x72.png\"><link rel=\"apple-touch-icon-precomposed\" sizes=\"144x144\" href=\"/campus/apple-touch-icon-144x144.png\"><link rel=\"apple-touch-icon-precomposed\" sizes=\"60x60\" href=\"/campu\n"
     ]
    }
   ],
   "source": [
    "from urllib.request import Request, urlopen\n",
    "\n",
    "\n",
    "url = 'https://campus.datacamp.com/courses/1606/4135?ex=2'\n",
    "\n",
    "request = Request(url)\n",
    "\n",
    "response = urlopen(request)\n",
    "\n",
    "text = response.read()\n",
    "response.close()\n",
    "\n",
    "print(text[:300])\n",
    "\n",
    "\n",
    "\n",
    "import requests\n",
    "\n",
    "r = requests.get(url)\n",
    "\n",
    "content = r.text  ###################################################################################################\n",
    "\n",
    "print(content[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ed3d8f2-f1cf-4a7c-9878-ad4b16e205dd",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import packages\n",
    "from urllib.request import urlopen, Request\n",
    "\n",
    "# Specify the url\n",
    "url = \"https://campus.datacamp.com/courses/1606/4135?ex=2\"\n",
    "\n",
    "# This packages the request: request\n",
    "request = Request(url)\n",
    "\n",
    "# Sends the request and catches the response: response\n",
    "response = urlopen(request)\n",
    "\n",
    "# Print the datatype of response\n",
    "print(type(response))\n",
    "\n",
    "# Be polite and close the response!\n",
    "response.close()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d98d85e5-0bfa-42a4-86c3-7c8e52985390",
   "metadata": {},
   "source": [
    "## Printing HTTP request results in Python using urllib\n",
    "\n",
    "You have just packaged and sent a GET request to \"https://campus.datacamp.com/courses/1606/4135?ex=2\" and then caught the response. You saw that such a response is a http.client.HTTPResponse object. The question remains: what can you do with this response?\n",
    "\n",
    "Well, as it came from an HTML page, you could read it to extract the HTML and, in fact, such a http.client.HTTPResponse object has an associated \"read()\" method. In this exercise, you'll build on your previous great work to extract the response and print the HTML.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Send the request and catch the response in the variable \"response\" with the function \"urlopen()\", as in the previous exercise.\n",
    "    Extract the response using the read() method and store the result in the variable html.\n",
    "    Print the string html.\n",
    "    Hit submit to perform all of the above and to close the response: be tidy!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "677dd883-a993-45aa-9ff0-cb99201a9e72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'http.client.HTTPResponse'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "b'<!doctype html><html lang=\"en\"><head><link rel=\"apple-touch-icon-precomposed\" sizes=\"57x57\" href=\"/campus/apple-touch-icon-57x57.png\"><link rel=\"apple-touch-icon-precomposed\" sizes=\"114x114\" href=\"/campus/apple-touch-icon-114x114.png\"><link rel=\"apple-touch-icon-precomposed\" sizes=\"72x72\" href=\"/campus/apple-touch-icon-72x72.png\"><link rel=\"apple-touch-icon-precomposed\" sizes=\"144x144\" href=\"/camp'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from urllib.request import Request, urlopen\n",
    "\n",
    "\n",
    "url = 'https://campus.datacamp.com/courses/1606/4135?ex=2'\n",
    "\n",
    "response = urlopen(Request(url))\n",
    "\n",
    "print(type(response))\n",
    "\n",
    "\n",
    "html = response.read()\n",
    "response.close()  ###################################################################################################\n",
    "\n",
    "\n",
    "html[:400]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "18d4096b-6a71-4545-b139-7d355f3b1472",
   "metadata": {},
   "source": [
    "## Performing HTTP requests in Python using requests\n",
    "\n",
    "Now that you've got your head and hands around making HTTP requests using the urllib package, you're going to figure out how to do the same using the higher-level requests library. You'll once again be pinging DataCamp servers for their \"http://www.datacamp.com/teach/documentation\" page.\n",
    "\n",
    "Note that unlike in the previous exercises using urllib, you don't have to close the connection when using requests!\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Import the package requests.\n",
    "    Assign the URL of interest to the variable url.\n",
    "    Package the request to the URL, send the request and catch the response with a single function requests.get(), assigning the response to the variable r.\n",
    "    Use the text attribute of the object r to return the HTML of the webpage as a string; store the result in a variable text.\n",
    "    Hit submit to print the HTML of the webpage.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "cde98585-291c-4678-adbd-ae1885ef5ae2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE HTML>\\n<html lang=\"en-US\">\\n<head>\\n  <meta charset=\"UTF-8\" />\\n  <meta http-equiv=\"Content-Type\" content=\"text/html; charset=UTF-8\" />\\n  <meta http-equiv=\"X-UA-Compatible\" content=\"IE=Edge,chrome=1\" />\\n  <meta name=\"robots\" content=\"noindex, nofollow\" />\\n  <meta name=\"viewport\" content=\"width'"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "\n",
    "url = 'http://www.datacamp.com/teach/documentation'\n",
    "\n",
    "r = requests.get(url)    # Just beautiful\n",
    "\n",
    "r.text[:300]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af2755d3-706a-408c-b5b4-bde55f5c6d01",
   "metadata": {},
   "source": [
    "## Scraping the web in Python\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**You have just scraped HTML data from the web and you're done so using 2 different packages, urllib and requests.  You also saw that requests provided a higher-level interface in that you needed to write fewer lines of code to retrieve the relevant HTML as a string.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "You've got the HTML of your page of interest but, generally HTML is a humble-jumble mix of both unstructured and structured data.  A few words of these terms: Structured data is data that has a pre-defined data model or that is organized in a defined manner.  Unstructured data is data that does not possess either of these properties.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "HTML is interesting because, although much of it is unstructured text, it does contain tags that determine that where, for example, headings can be found, and hyperlinks.  In general, to turn HTML that you have scraped from the WWW into useful data, you'll need to parse it and extract structured data from it.  \n",
    "\n",
    "In this video and the next few interactive exercises, we'll providea brief introduction to how you can perform such tasks using the Python package BeautifulSoup.  Lets now check out the pacakge's website.  The first words at the top are: \"You didn't write that awful page.  You're just trying to get some data out of it.  Beautiful Soup is here to help.  Since 2004, its been saving programmers hours or daysof work on quick-turnaround screen scraping projects.\"  Firstly, a word on the named of the package: BeautifulSoup.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "In web development, the term \"tag soup\" refers to structurally or syntactically incorrect HTML code written for a web page.  What Beautiful Soup does best is to make \"tag soup\" beautiful again and to extract information from it with ease.  In fact, the main object created and queried when using this package is called BeautifulSoup and it has very important associated method called \"prettify()\".  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "Lets now see BeautifulSoup in beautiful Action.  Once again you use requests to scrape the HTMLfrom the web.  Then you create a BeautifulSoup object from the resulting HTML and prettify it.  Printing the prettified Soup and the origina lHTML, you can see that for example, that the prettified Soup is indented in the way you would expect properly written HTML to be.  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "You'll explorea few of the method that you can apply to your soupified HTML in the following exercises, such as \".title\" and \".get_text()\", which extract the title and text, respectively.  You'll also work with the Soupy method \".find_all()\" in order to extract the URLs of all the hyperlinks in the HTML.  And these are merely a few of many methods existing in BeautifulSoup to extract data from HTML.  If, after completing these exercises, you find yourself thirsting for more BeautifulSoup, just Google what you want.  \n",
    "\n",
    "\n",
    "Now its your turn to jump into the deep end of the proverbial soup bowl.  Happy hackinng\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "4690a6e7-f3e2-403e-a3f0-73d24bee17a1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'<!DOCTYPE html><html lang=en-US><head><meta charset=\"UTF-8\"><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"><link rel=profile href=https://gmpg.org/xfn/11><link rel=manifest href=/superpwa-manifest.json><link rel=prefetch href=/superpwa-manifest.json><meta name=\"theme-color\" cont'\n",
      "<!DOCTYPE html>\n",
      "<html lang=\"en-US\">\n",
      " <head>\n",
      "  <meta charset=\"utf-8\"/>\n",
      "  <meta content=\"width=device-width, initial-scale=1\" name=\"viewport\"/>\n",
      "  <link href=\"https://gmpg.org/xfn/11\" rel=\"profile\"/>\n",
      "  <link href=\"/superpwa-manifest.json\" rel=\"manifest\"/>\n",
      "  <link href=\"/superpwa-manifest.json\" rel=\"prefetch\"/>\n",
      "  <meta content=\"#D5E0EB\" name=\"theme-color\"/>\n",
      "  <meta content=\"index, follow, max-image-preview:large, max-snippet:-1, max-video-preview:-1\" name=\"robots\"/>\n",
      "  <title>\n",
      "   Develop Perfect Memory With the Memory Palace Technique - Litemind\n",
      "  </title>\n",
      "  <link href=\"https://litemind.com/memory-palace/\" rel=\"canonical\"/>\n",
      "  <meta content=\"en_US\" property=\"og:locale\"/>\n",
      "  <meta content=\"article\" property=\"og:type\"/>\n",
      "  <meta content=\"Develop Perfect Memory With the Memory Palace Technique - Litemind\" property=\"og:title\"/>\n",
      "  <meta content=\"The Memory Palace is one of the most powerful memory techniques I know. It’s not only effective, but also fun to use — and not hard to learn at all. The Me\n",
      "\n",
      "\n",
      "<class 'str'>\n",
      "10480\n",
      "Develop Perfect Memory With the Memory Palace Technique - Litemind    Skip to content   Litemind    Main Menu  Decision Making Brainpower Creativity Productivity Personal Development   Search    Search for:   Develop Perfect Memory With the Memory Palace TechniqueBrainpowerThe Memory Palace is one of the most powerful memory techniques I know. It’s not only effective, but also fun to use — and not hard to learn at all. The Memory Palace has been used since ancient Rome, and is responsible for some quite incredible memory feats. Eight-time world memory champion Dominic O’Brien, for instance, was able to memorize 54 decks of cards in sequence (that’s 2808 cards), viewing each card only once. And there are countless other similar achievements attributed to people using the Memory Palace technique or variations of it. Even in fiction, there are several references to the technique. In Thomas Harris’ novel Hannibal, for example, serial killer Hannibal Lecter uses Memory Palaces to store amaz\n"
     ]
    }
   ],
   "source": [
    "from urllib.request import Request, urlopen\n",
    "\n",
    "\n",
    "url = 'https://litemind.com/memory-palace/'\n",
    "\n",
    "response = urlopen(Request(url))\n",
    "html = response.read()\n",
    "response.close()\n",
    "\n",
    "print(html[:300])\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "soup = BeautifulSoup(html)\n",
    "\n",
    "pretty_soup = soup.prettify()\n",
    "\n",
    "print(pretty_soup[:1000])\n",
    "print('\\n')\n",
    "\n",
    "\n",
    "text = soup.get_text()\n",
    "\n",
    "print(type(text))\n",
    "print(len(text))\n",
    "\n",
    "print(soup.get_text()[:1000])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "677299b1-d84e-470a-bb2e-b95c5d910234",
   "metadata": {},
   "source": [
    "## Parsing HTML with BeautifulSoup\n",
    "\n",
    "In this interactive exercise, you'll learn how to use the BeautifulSoup package to parse, prettify and extract information from HTML. You'll scrape the data from the webpage of Guido van Rossum, Python's very own Benevolent Dictator for Life. In the following exercises, you'll prettify the HTML and then extract the text and the hyperlinks.\n",
    "\n",
    "The URL of interest is url = 'https://www.python.org/~guido/'.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Import the function BeautifulSoup from the package bs4.\n",
    "    Assign the URL of interest to the variable url.\n",
    "#    Package the request to the URL, send the request and catch the response with a single function \"requests.get()\", assigning the response to the variable r.\n",
    "#    Use the \".text\" attribute of the object r to return the HTML of the webpage as a string; store the result in a variable html_doc.\n",
    "#    Create a BeautifulSoup object soup from the resulting HTML using the function \"BeautifulSoup()\".\n",
    "#    Use the method \".prettify()\" on soup and assign the result to pretty_soup.\n",
    "    Hit submit to print to prettified HTML to your shell!\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "id": "ee137436-397c-47c7-a33e-9f46a4abd153",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<HTML>\n",
      "\n",
      "<HEAD>\n",
      "<TITLE>Guido's Personal Home Page</TITLE>\n",
      "</HEAD>\n",
      "\n",
      "<BODY BGCOLOR=\"#FFFFFF\" TEXT=\"#000000\">\n",
      "\n",
      "<!-- Built from main -->\n",
      "<H1>\n",
      "<a href=\"pics.html\"><img border=\"0\" src=\"images/IMG_2192.jpg\"><\n",
      "ns and interviews (all about Python), some\n",
      "pictures of me,\n",
      "my new blog, and\n",
      "my old\n",
      "blog on Artima.com.  I am\n",
      "@gvanrossum on Twitter.\n",
      "\n",
      "I am retired, working on personal projects (and maybe a book).\n",
      "I have worked for Dropbox, Google, Elemental Security, Zope\n",
      "Corporation, BeOpen.com, CNRI, CWI, and SARA.  (See\n",
      "my resume.)  I created Python while at CWI.\n",
      "\n",
      "How to Reach Me\n",
      "You can send email for me to guido (at) python.org.\n",
      "I read everything sent there, but I receive too much email to respond\n",
      "to everything.\n",
      "\n",
      "My Name\n",
      "My name often poses difficulties for Americans.\n",
      "\n",
      "Pronunciation: in Dutch, the \"G\" in\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "import requests\n",
    "\n",
    "\n",
    "url = 'https://www.python.org/~guido/'\n",
    "\n",
    "r = requests.get(url)\n",
    "html = r.text\n",
    "\n",
    "print(html[:200])\n",
    "\n",
    "\n",
    "soup = BeautifulSoup(html)\n",
    "pretty_soup = soup.prettify()\n",
    "\n",
    "\n",
    "text = soup.get_text()\n",
    "print(text[300:900])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd050d00-4ef3-4326-97b9-bed78dec3891",
   "metadata": {},
   "source": [
    "## Turning a webpage into data using BeautifulSoup: getting the text\n",
    "\n",
    "As promised, in the following exercises, you'll learn the basics of extracting information from HTML soup. In this exercise, you'll figure out how to extract the text from the BDFL's webpage, along with printing the webpage's title.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    In the sample code, the HTML response object html_doc has already been created: your first task is to Soupify it using the function BeautifulSoup() and to assign the resulting soup to the variable soup.\n",
    "    Extract the title from the HTML soup soup using the attribute title and assign the result to guido_title.\n",
    "    Print the title of Guido's webpage to the shell using the print() function.\n",
    "    Extract the text from the HTML soup soup using the method get_text() and assign to guido_text.\n",
    "    Hit submit to print the text from Guido's webpage to the shell.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "id": "4003111f-3e85-412d-b625-64e9e27afccd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>Guido's Personal Home Page</title>\n",
      "['DEFAULT_INTERESTING_STRING_TYPES', '__bool__', '__call__', '__class__', '__contains__', '__copy__', '__delattr__', '__delitem__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattr__', '__getattribute__', '__getitem__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__setitem__', '__sizeof__', '__str__', '__subclasshook__', '__unicode__', '__weakref__', '_all_strings', '_find_all', '_find_one', '_is_xml', '_lastRecursiveChild', '_last_descendant', '_should_pretty_print', 'append', 'attrs', 'can_be_empty_element', 'cdata_list_attributes', 'childGenerator', 'children', 'clear', 'contents', 'decode', 'decode_contents', 'decompose', 'decomposed', 'default', 'descendants', 'encode', 'encode_contents', 'extend', 'extract', 'fetchNextSiblings', 'fetchParents', 'fetchPrevious', 'fetchPreviousSiblings', 'find', 'findAll', 'findAllNext', 'findAllPrevious', 'findChild', 'findChildren', 'findNext', 'findNextSibling', 'findNextSiblings', 'findParent', 'findParents', 'findPrevious', 'findPreviousSibling', 'findPreviousSiblings', 'find_all', 'find_all_next', 'find_all_previous', 'find_next', 'find_next_sibling', 'find_next_siblings', 'find_parent', 'find_parents', 'find_previous', 'find_previous_sibling', 'find_previous_siblings', 'format_string', 'formatter_for_name', 'get', 'getText', 'get_attribute_list', 'get_text', 'has_attr', 'has_key', 'hidden', 'index', 'insert', 'insert_after', 'insert_before', 'interesting_string_types', 'isSelfClosing', 'is_empty_element', 'known_xml', 'name', 'namespace', 'next', 'nextGenerator', 'nextSibling', 'nextSiblingGenerator', 'next_element', 'next_elements', 'next_sibling', 'next_siblings', 'parent', 'parentGenerator', 'parents', 'parserClass', 'parser_class', 'prefix', 'preserve_whitespace_tags', 'prettify', 'previous', 'previousGenerator', 'previousSibling', 'previousSiblingGenerator', 'previous_element', 'previous_elements', 'previous_sibling', 'previous_siblings', 'recursiveChildGenerator', 'renderContents', 'replaceWith', 'replaceWithChildren', 'replace_with', 'replace_with_children', 'select', 'select_one', 'setup', 'smooth', 'sourceline', 'sourcepos', 'string', 'strings', 'stripped_strings', 'text', 'unwrap', 'wrap']\n",
      "\n",
      "\n",
      "Guido's Personal Home Page\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Guido van Rossum - Personal Home Page\n",
      "\n",
      "\n",
      "\"Gawky and proud of it.\"\n",
      "Who I Am\n",
      "Read\n",
      "my \"King's\n",
      "Day Speech\" for some inspiration.\n",
      "\n",
      "I am the author of the Python\n",
      "programming language.  See also my resume\n",
      "and my publications list, a brief bio, assorted writings, presentations and interviews (all about Python), some\n",
      "pictures of me,\n",
      "my new blog, and\n",
      "my old\n",
      "blog on Artima.com.  I am\n",
      "@gvanrossum on Twitter.\n",
      "\n",
      "I am retired, working on personal projects (and maybe a book).\n",
      "I have worked for Dropbox, Google, Elemental Security, Zope\n",
      "Corporation, BeOpen.com, CNRI, CWI, and SAR\n"
     ]
    }
   ],
   "source": [
    "url = 'https://www.python.org/~guido/'\n",
    "\n",
    "\n",
    "r = requests.get(url)\n",
    "html = r.text\n",
    "\n",
    "soup = BeautifulSoup(html)\n",
    "pretty_soup = soup.prettify()\n",
    "\n",
    "\n",
    "title = soup.title\n",
    "print(title)\n",
    "print(dir(title))\n",
    "\n",
    "text = soup.getText()\n",
    "print(text[:600])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f2cd348-ba96-48f2-bc07-f4afd9d66637",
   "metadata": {},
   "source": [
    "## Turning a webpage into data using BeautifulSoup: getting the hyperlinks\n",
    "\n",
    "In this exercise, you'll figure out how to extract the URLs of the hyperlinks from the BDFL's webpage. In the process, you'll become close friends with the soup method \".find_all()\".\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "#    Use the method find_all() to find all hyperlinks in soup, remembering that hyperlinks are defined by the HTML tag <a> but passed to find_all() without angle brackets; store the result in the variable a_tags.\n",
    "#    The variable a_tags is a results set: your job now is to enumerate over it, using a for loop and to print the actual URLs of the hyperlinks; to do this, for every element link in a_tags, you want to print() link.get('href').\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f03bc53-d5b5-4fe6-a7f7-4382d7cf9806",
   "metadata": {},
   "source": [
    "# *******************************************************************************************************************"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "id": "3711669e-301b-46a7-9bd3-fa77e6ba5ec1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "23\n",
      "[<a href=\"pics.html\"><img border=\"0\" src=\"images/IMG_2192.jpg\"/></a>, <a href=\"pics.html\"><img border=\"0\" height=\"216\" src=\"images/guido-headshot-2019.jpg\" width=\"270\"/></a>, <a href=\"http://www.washingtonpost.com/wp-srv/business/longterm/microsoft/stories/1998/raymond120398.htm\"><i>\"Gawky and proud of it.\"</i></a>, <a href=\"images/df20000406.jpg\">Who I Am</a>, <a href=\"http://neopythonic.blogspot.com/2016/04/kings-day-speech.html\">\"King's\n",
      "Day Speech\"</a>, <a href=\"http://www.python.org\">Python</a>, <a href=\"Resume.html\">resume</a>, <a href=\"Publications.html\">publications list</a>, <a href=\"bio.html\">brief bio</a>, <a href=\"http://legacy.python.org/doc/essays/\">writings</a>, <a href=\"http://legacy.python.org/doc/essays/ppt/\">presentations</a>, <a href=\"interviews.html\">interviews</a>, <a href=\"pics.html\">pictures of me</a>, <a href=\"http://neopythonic.blogspot.com\">my new blog</a>, <a href=\"http://www.artima.com/weblogs/index.jsp?blogger=12088\">old\n",
      "blog</a>, <a href=\"https://twitter.com/gvanrossum\">@gvanrossum</a>, <a href=\"Resume.html\">resume</a>, <a href=\"guido.au\">sound clip</a>, <a href=\"http://legacy.python.org/doc/essays/\">essays</a>, <a href=\"images/license.jpg\"><img align=\"center\" border=\"0\" height=\"75\" src=\"images/license_thumb.jpg\" width=\"100\"/>\n",
      "Python license.</a>, <a href=\"http://www.cnpbagwell.com/audio-faq\">http://www.cnpbagwell.com/audio-faq</a>, <a href=\"http://sox.sourceforge.net/\">SOX</a>, <a href=\"images/internetdog.gif\">\"On the Internet, nobody knows you're\n",
      "a dog.\"</a>]\n",
      "pics.html\n",
      "pics.html\n",
      "http://www.washingtonpost.com/wp-srv/business/longterm/microsoft/stories/1998/raymond120398.htm\n",
      "images/df20000406.jpg\n",
      "http://neopythonic.blogspot.com/2016/04/kings-day-speech.html\n",
      "http://www.python.org\n",
      "Resume.html\n",
      "Publications.html\n",
      "bio.html\n",
      "http://legacy.python.org/doc/essays/\n",
      "http://legacy.python.org/doc/essays/ppt/\n",
      "interviews.html\n",
      "pics.html\n",
      "http://neopythonic.blogspot.com\n",
      "http://www.artima.com/weblogs/index.jsp?blogger=12088\n",
      "https://twitter.com/gvanrossum\n",
      "Resume.html\n",
      "guido.au\n",
      "http://legacy.python.org/doc/essays/\n",
      "images/license.jpg\n",
      "http://www.cnpbagwell.com/audio-faq\n",
      "http://sox.sourceforge.net/\n",
      "images/internetdog.gif\n"
     ]
    }
   ],
   "source": [
    "links = soup.find_all('a')\n",
    "\n",
    "print(len(links))\n",
    "print(links)\n",
    "\n",
    "\n",
    "for i in links:\n",
    "    print(i.get('href'))  ###########################################################################################\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b33e6f7b-590f-4715-bec9-620f7d206ff1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import packages\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "# Specify url\n",
    "url = 'https://www.python.org/~guido/'\n",
    "\n",
    "# Package the request, send the request and catch the response: r\n",
    "r = requests.get(url)\n",
    "\n",
    "# Extracts the response as html: html_doc\n",
    "html_doc = r.text\n",
    "\n",
    "# create a BeautifulSoup object from the HTML: soup\n",
    "soup = BeautifulSoup(html_doc)\n",
    "\n",
    "# Print the title of Guido's webpage\n",
    "print(soup.title)\n",
    "\n",
    "# Find all 'a' tags (which define hyperlinks): a_tags\n",
    "\n",
    "\n",
    "# Print the URLs to the shell\n",
    "for ____ in ____:\n",
    "    ____"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "345b7ed4-ca9f-48ce-84a1-1cc2c08f8977",
   "metadata": {},
   "source": [
    "## Introduction to APIs and JSONs\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**In this chapter, you'll explore pulling data from the web even further, by learning how to interactwith APIs, or Application Programming Interfaces.  \n",
    "\n",
    "# An API is a set of protocols and routines for building and interacting with software applications.  \n",
    "In particular, you'll learn how to use the Open Movies Database API and, in the next chapter, the Twitter API to pull data from both applications, while learning about API interaction best practices.  A standard form for transferring data through APIs is the JSON file format, so in this video, we'll focus our attention squarely on these.  Then we'll move onto actually getting data from APIs.  \n",
    "\n",
    "JSON is an acronym that is short for JavaScript Object Notation.  It is a file format that arose out a growing need for real-time server-to-browser communication that wouldn't necessarily rely on Flash or Java and was first specified and also popularized by Douglas Crockford, an American programmer and entrepreneur.  \n",
    "\n",
    "On of the cool things about JSONs is that they're human readable, that is, they can naturally read by human unlike, for example, pickle files, as we saw in the previous course.  As they're human readable, lets check one out.  Below you can see a JSON form the OMDB or Open Movie Database API, in particular, this is JSON containing information about the movie Snakes on a Plane.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "First notice hat the JSON consists of name-value pairs separated by commas.  This will remind you of the key-value pairs in a Python dictionary.  We'll see in a minute that for this reason, when loading JSONs into Python, it is nature to store them in a dict.  The keys in JSONs will always be strings enclosed in quotation marks.  The values can be strings, integers, arrays or even objects.  Such an object can even be a JSON and then you have nested JSONs but we won't go further into these here.  \n",
    "\n",
    "In the case of the Snakes on a Plane JSON, all the values are strings and we can see this from the quotation marks.  The value corresponding to the key \"Title\" is the movie title as a string: \"Snakes on a Plane\".  The value corresponding to the key \"Year\" is the year of release as a string \"18 Aug 2006\".  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "You'll soon learn how to use the OMDB API and Python to automate retrieval of such data, but first you'll figure out how to load JSONs form local directory.  \n",
    "\n",
    "# Lets say we have the JSON stored in our working directory as \"snakes.json\".  \n",
    "To load the JSON into out Python environment, we need first import the package \"json\" and then \"open()\" a connection to the file and use the function \"json.load()\" to load the JSON.  If we then check the datatype of j_data by executing \"type()\" function, we can see that Python cleverly imported the JSON as a dictionary.  To print the key-value pairs to the console, we can then iterate over the key-value pairs using a for loop.  \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e3d3142-cbe3-4326-aa7e-d533910cf459",
   "metadata": {},
   "outputs": [],
   "source": [
    "{\n",
    "  \"Actors\": \"Samuel L. Jackson, Julianna Margulies, Nathan Phillips, Rachel Blanchard\",\n",
    "  \"Awards\": \"3 wins & 7 nominations.\",\n",
    "  \"Country\": \"Germany, USA, Canada\",\n",
    "  \"Director\": \"David R. Ellis\",\n",
    "  \"Genre\": \"Action, Adventure, Crime\",\n",
    "  \"Language\": \"English\",\n",
    "  \"Rated\": \"R\",\n",
    "  \"Released\": \"18 Aug 2006\",\n",
    "  \"Runtime\": \"105 min\",\n",
    "  \"Title\": \"Snakes on a Plane\",\n",
    "  \"Type\": \"movie\",\n",
    "  \"Writer\": \"John Heffernan (screenplay), Sebastian Gutierrez(screenplay), David Dalessandro (story), John Heffernan (story)\",\n",
    "  \"Year\": \"2006\",\n",
    "  \"imdbID\": \"tt0417148\",\n",
    "  \"imdbRating\": \"5.6\",\n",
    "  \"imdbVotes\": \"114,668\"\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "6d3cd0cc-0b46-4c72-b36b-e3bb3cd4dd61",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Actors': 'Samuel L. Jackson, Julianna Margulies, Nathan Phillips, Rachel Blanchard', 'Awards': '3 wins & 7 nominations.', 'Country': 'Germany, USA, Canada', 'Director': 'David R. Ellis', 'Genre': 'Action, Adventure, Crime', 'Language': 'English', 'Rated': 'R', 'Released': '18 Aug 2006', 'Runtime': '105 min', 'Title': 'Snakes on a Plane', 'Type': 'movie', 'Writer': 'John Heffernan (screenplay), Sebastian Gutierrez(screenplay), David Dalessandro (story), John Heffernan (story)', 'Year': '2006', 'imdbID': 'tt0417148', 'imdbRating': '5.6', 'imdbVotes': '114,668'}\n",
      "<class 'dict'>\n",
      "\n",
      "\n",
      "Actors:  Samuel L. Jackson, Julianna Margulies, Nathan Phillips, Rachel Blanchard\n",
      "Awards:  3 wins & 7 nominations.\n",
      "Country:  Germany, USA, Canada\n",
      "Director:  David R. Ellis\n",
      "Genre:  Action, Adventure, Crime\n",
      "Language:  English\n",
      "Rated:  R\n",
      "Released:  18 Aug 2006\n",
      "Runtime:  105 min\n",
      "Title:  Snakes on a Plane\n",
      "Type:  movie\n",
      "Writer:  John Heffernan (screenplay), Sebastian Gutierrez(screenplay), David Dalessandro (story), John Heffernan (story)\n",
      "Year:  2006\n",
      "imdbID:  tt0417148\n",
      "imdbRating:  5.6\n",
      "imdbVotes:  114,668\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "\n",
    "\n",
    "with open('snakes.json', 'r') as j_file:\n",
    "    j_data = json.load(j_file)  #####################################################################################\n",
    "    \n",
    "    \n",
    "print(j_data)\n",
    "print(type(j_data))\n",
    "\n",
    "print('\\n')\n",
    "\n",
    "\n",
    "for k,i in j_data.items():\n",
    "    #print(k, i)\n",
    "    print(k +': ', i)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "686e116b-18ee-4fde-915e-6d55969e677a",
   "metadata": {},
   "source": [
    "## Pop quiz: What exactly is a JSON?\n",
    "\n",
    "Which of the following is NOT true of the JSON file format?\n",
    "Answer the question\n",
    "50XP\n",
    "Possible Answers\n",
    "\n",
    "    JSONs consist of key-value pairs.\n",
    "    1\n",
    "    JSONs are human-readable.\n",
    "    2\n",
    "    The JSON file format arose out of a growing need for real-time server-to-browser communication.\n",
    "    3\n",
    "#    The function json.load() will load the JSON into Python as a list.\n",
    "    4\n",
    "    The function json.load() will load the JSON into Python as a dictionary.\n",
    "    5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd7807b3-d567-4d90-8931-cc13d4513e96",
   "metadata": {},
   "source": [
    "## Loading and exploring a JSON\n",
    "\n",
    "Now that you know what a JSON is, you'll load one into your Python environment and explore it yourself. Here, you'll load the JSON 'a_movie.json' into the variable json_data, which will be a dictionary. You'll then explore the JSON contents by printing the key-value pairs of json_data to the shell.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "#    Load the JSON 'a_movie.json' into the variable json_data within the context provided by the with statement. To do so, use the function json.load() within the context manager.\n",
    "#    Use a for loop to print all key-value pairs in the dictionary json_data. Recall that you can access a value in a dictionary using the syntax: dictionary[key].\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9304d766-4061-4de5-b049-c1dc3cdc8737",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load JSON: json_data\n",
    "with open(\"a_movie.json\") as json_file:\n",
    "    ____\n",
    "\n",
    "# Print each key-value pair in json_data\n",
    "for k in json_data.keys():\n",
    "    print(k + ': ', ____)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a46bfff3-ee36-4c96-8daa-4fd2d51ec92f",
   "metadata": {},
   "source": [
    "## Pop quiz: Exploring your JSON\n",
    "\n",
    "Load the JSON 'a_movie.json' into a variable, which will be a dictionary. Do so by copying, pasting and executing the following code in the IPython Shell:\n",
    "\n",
    "import json\n",
    "with open(\"a_movie.json\") as json_file:\n",
    "    json_data = json.load(json_file)\n",
    "\n",
    "Print the values corresponding to the keys 'Title' and 'Year' and answer the following question about the movie that the JSON describes:\n",
    "\n",
    "Which of the following statements is true of the movie in question?\n",
    "Instructions\n",
    "50 XP\n",
    "Possible Answers\n",
    "\n",
    "    The title is 'Kung Fu Panda' and the year is 2010.\n",
    "    The title is 'Kung Fu Panda' and the year is 2008.\n",
    "    The title is 'The Social Network' and the year is 2010.\n",
    "    The title is 'The Social Network' and the year is 2008.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e48f5c1c-bf6f-48e3-9d63-2215509a2d53",
   "metadata": {},
   "source": [
    "## APIs and interacting with the world wide web\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**Congrats on making it through your crash course in JSONs.  JSONs are everywhere\n",
    "# *******************************************************************************************************************\n",
    "JSONs are everywhere and one of the main motivating reasons for getting to know how to work with them as a Data Scientist is that much of the data that you'll get from APIs are packaged as JSONs.  In this video, you'll learn what APIs are, why they are so important, and see a number of illustrative examples.  In the subsequent interactive exercises, you'll gain valuable practice connecting to a variety of APIs, pulling and parsing data from them.  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "So what is an API and why are they so important?  Simply put, an API is a set of protocols and routines for building and interacting with software applications.  Another way to think of it is that an API is a bunch of code that allows 2 software programs to communicate with each other.  \n",
    "\n",
    "For example, if you wanted to stream Twitter data by writing some Python code, you would use the Twitter API.  If you wanted to automate pulling and processing information from Wikipedia in your programming language of choice, you would do so using the Wikipedia API.  Using such APIs have now become standard ways of interacting with such applications.  \n",
    "\n",
    "Twitter ha an API that is used by marketing companies along with social scientists engaged in research concerning social networks.  Uber, Facebook and Instagram all have APIs.  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "Now lets figure out how to connect to an API and how to pull data from it.  In this example, we'll pull movie data from the Open Movie Database, or OMDB API.  Once again, you'll use the ever-elegant requests library.  We first import requests, and assign URL of interest to the variable \"url\".  We then package and send request to the URL, which describes your API query, and catch the response in one line of code.  \n",
    "# *******************************************************************************************************************\n",
    "Another really cool aspect of the requests package is that the Response objects such as \"r\", have an associated method \".json()\", which is a built-in JSON decoder for when we're dealing with JSON data.  This returns a dictionary and we can then print all the key-value pairs to check out what weulled from the OMDB API.  \n",
    "# *******************************************************************************************************************\n",
    "\n",
    "# Now the last thingto discuss is how the URL we used actually pulled data from the API.  To do so, lets break it up into chucks.  \n",
    "\n",
    "The HTTP signifies that we're making an HTTP request, the \"www.omdbapi.com\" that we are querying the OMDB API.  Then there is the \"?t=hackers\" which is really interesting part and something we havn't discussed yet in this course.  This string that begins with a question mark (?) is called a Query String.  Query Strings are parts of URLs that do not necessarily fit into the conventional hierarchical path structure.  What follows the question mark (?) in the query string is the query we are making to the OMDB API.  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "The query we just made was simple: querying t equals hackers, asks the API to return the data about the movie with the title Hackers.  The \"t\" in the query stood for title.  We knew that this was how perform such a query from the documentation on the OMDB API's homepage.  Under Usage here, they state explicitly  that Send all data requests to: http://www.omdbapi.com/?, they also have a Query String parameters table that shows how to query a particular title or a particular movie id.  \n",
    "# *******************************************************************************************************************\n",
    "\n",
    "\n",
    "There is also worth mentioning that there is nothing special about this URL and so you can also navigate to it in your browser of choice.  It will generaly look like a JSON file opened though your browser like a CSV on GitHub page.  \n",
    "Now you know all bout APIs and have a basic practical understanding of how to query them, lets get ready writing some Python to extract some data from a number of APIs.  \n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "08cfafa3-444f-4f52-8699-90908b4445e9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Response': 'False', 'Error': 'No API key provided.'}\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "\n",
    "url = 'https://www.omdbapi.com/?t=hackers'\n",
    "\n",
    "r = requests.get(url)\n",
    "\n",
    "#j_data = json.load(r)\n",
    "j_data = r.json()  ##################################################################################################\n",
    "\n",
    "print(j_data)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e325fc87-4be5-4012-acbb-ce7a3c75eb7b",
   "metadata": {},
   "source": [
    "# Note the difference here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "df21206d-9da8-48c8-9f0c-99110aa7f834",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<title>Guido's Personal Home Page</title>\n",
      "rking on personal projects (and maybe a book).\n",
      "I have worked for Dropbox, Google, Elemental Security, Zope\n",
      "Corporation, BeOpen.com, CNRI, CWI, and SAR\n"
     ]
    }
   ],
   "source": [
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "url = 'https://www.python.org/~guido/'\n",
    "\n",
    "\n",
    "r = requests.get(url)\n",
    "html = r.text  ######################################################################################################\n",
    "\n",
    "soup = BeautifulSoup(html)\n",
    "#pretty_soup = soup.prettify()\n",
    "\n",
    "title = soup.title\n",
    "print(title)\n",
    "\n",
    "text = soup.getText()\n",
    "print(text[450:600])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16019c71-e791-448d-8c7f-dd3eb2543721",
   "metadata": {},
   "source": [
    "## Pop quiz: What's an API?\n",
    "\n",
    "Which of the following statements about APIs is NOT true?\n",
    "Answer the question\n",
    "50XP\n",
    "Possible Answers\n",
    "\n",
    "    An API is a set of protocols and routines for building and interacting with software applications.\n",
    "    1\n",
    "    API is an acronym and is short for Application Program interface.\n",
    "    2\n",
    "    It is common to pull data from APIs in the JSON file format.\n",
    "    3\n",
    "#    All APIs transmit data only in the JSON file format.\n",
    "    4\n",
    "    An API is a bunch of code that allows two software programs to communicate with each other.\n",
    "    5"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "831b30e3-74aa-48b2-a521-d5182bd3e55b",
   "metadata": {},
   "source": [
    "## API requests\n",
    "\n",
    "Now it's your turn to pull some movie data down from the Open Movie Database (OMDB) using their API. The movie you'll query the API about is The Social Network. Recall that, in the video, to query the API about the movie Hackers, Hugo's query string was 'http://www.omdbapi.com/?t=hackers' and had a single argument t=hackers.\n",
    "\n",
    "Note: recently, OMDB has changed their API: you now also have to specify an API key. This means you'll have to add another argument to the URL: apikey=72bc447a.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Import the requests package.\n",
    "    Assign to the variable url the URL of interest in order to query 'http://www.omdbapi.com' for the data corresponding to the movie The Social Network. The query string should have two arguments: apikey=72bc447a and t=the+social+network. You can combine them as follows: apikey=72bc447a&t=the+social+network.\n",
    "    Print the text of the response object r by using its text attribute and passing the result to the print() function.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "8bf77321-8527-4789-bfd8-ddb56beb64d0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Title': 'Blade', 'Year': '1998', 'Rated': 'R', 'Released': '21 Aug 1998', 'Runtime': '120 min', 'Genre': 'Action, Horror, Sci-Fi', 'Director': 'Stephen Norrington', 'Writer': 'David S. Goyer', 'Actors': 'Wesley Snipes, Stephen Dorff, Kris Kristofferson', 'Plot': 'A half-vampire, half-mortal man becomes a protector of the mortal race, while slaying evil vampires.', 'Language': 'English, Russian, Serbian', 'Country': 'United States', 'Awards': '5 wins & 11 nominations', 'Poster': 'https://m.media-amazon.com/images/M/MV5BOTk2NDNjZWQtMGY0Mi00YTY2LWE5MzctMGRhZmNlYzljYTg5XkEyXkFqcGdeQXVyMTAyNjg4NjE0._V1_SX300.jpg', 'Ratings': [{'Source': 'Internet Movie Database', 'Value': '7.1/10'}, {'Source': 'Rotten Tomatoes', 'Value': '57%'}, {'Source': 'Metacritic', 'Value': '47/100'}], 'Metascore': '47', 'imdbRating': '7.1', 'imdbVotes': '261,993', 'imdbID': 'tt0120611', 'Type': 'movie', 'DVD': '09 Nov 2004', 'BoxOffice': '$70,087,718', 'Production': 'N/A', 'Website': 'N/A', 'Response': 'True'}\n"
     ]
    }
   ],
   "source": [
    "url = 'http://www.omdbapi.com/?t=hackers'\n",
    "url = 'http://www.omdbapi.com/?i=tt3896198&apikey=8c241015'\n",
    "url = 'http://www.omdbapi.com/?t=Blade&apikey=8c241015'\n",
    "\n",
    "\n",
    "\n",
    "import json\n",
    "import requests\n",
    "\n",
    "r = requests.get(url)\n",
    "\n",
    "j_data = r.json()\n",
    "\n",
    "print(j_data)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "67afc1e7-fa3f-4c76-bba0-07effcab4c16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\"Title\":\"The Social Network\",\"Year\":\"2010\",\"Rated\":\"PG-13\",\"Released\":\"01 Oct 2010\",\"Runtime\":\"120 min\",\"Genre\":\"Biography, Drama\",\"Director\":\"David Fincher\",\"Writer\":\"Aaron Sorkin, Ben Mezrich\",\"Actors\":\"Jesse Eisenberg, Andrew Garfield, Justin Timberlake\",\"Plot\":\"As Harvard student Mark Zuckerberg creates the social networking site that would become known as Facebook, he is sued by the twins who claimed he stole their idea, and by the co-founder who was later squeezed out of the business.\",\"Language\":\"English, French\",\"Country\":\"United States\",\"Awards\":\"Won 3 Oscars. 172 wins & 186 nominations total\",\"Poster\":\"https://m.media-amazon.com/images/M/MV5BOGUyZDUxZjEtMmIzMC00MzlmLTg4MGItZWJmMzBhZjE0Mjc1XkEyXkFqcGdeQXVyMTMxODk2OTU@._V1_SX300.jpg\",\"Ratings\":[{\"Source\":\"Internet Movie Database\",\"Value\":\"7.7/10\"},{\"Source\":\"Rotten Tomatoes\",\"Value\":\"96%\"},{\"Source\":\"Metacritic\",\"Value\":\"95/100\"}],\"Metascore\":\"95\",\"imdbRating\":\"7.7\",\"imdbVotes\":\"670,561\",\"imdbID\":\"tt1285016\",\"Type\":\"movie\",\"DVD\":\"11 Jan 2011\",\"BoxOffice\":\"$96,962,694\",\"Production\":\"N/A\",\"Website\":\"N/A\",\"Response\":\"True\"}\n"
     ]
    }
   ],
   "source": [
    "# Import requests package\n",
    "import requests\n",
    "\n",
    "# Assign URL to variable: url\n",
    "url = 'http://www.omdbapi.com/?apikey=72bc447a&t=the+social+network'\n",
    "\n",
    "# Package the request, send the request and catch the response: r\n",
    "r = requests.get(url)\n",
    "\n",
    "# Print the text of the response\n",
    "print(r.text)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "178d43e5-c867-483e-a7cc-437bcd767e41",
   "metadata": {},
   "source": [
    "## JSON–from the web to Python\n",
    "\n",
    "Wow, congrats! You've just queried your first API programmatically in Python and printed the text of the response to the shell. However, as you know, your response is actually a JSON, so you can do one step better and decode the JSON. You can then print the key-value pairs of the resulting dictionary. That's what you're going to do now!\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Pass the variable url to the requests.get() function in order to send the relevant request and catch the response, assigning the resultant response message to the variable r.\n",
    "    Apply the json() method to the response object r and store the resulting dictionary in the variable json_data.\n",
    "    Hit submit to print the key-value pairs of the dictionary json_data to the shell.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "344ca083-8f19-41d6-b0e9-cb35b3a0ff37",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title:  Blade Runner 2049\n",
      "Year:  2017\n",
      "Rated:  R\n",
      "Released:  06 Oct 2017\n",
      "Runtime:  164 min\n",
      "Genre:  Action, Drama, Mystery\n",
      "Director:  Denis Villeneuve\n",
      "Writer:  Hampton Fancher, Michael Green, Philip K. Dick\n",
      "Actors:  Harrison Ford, Ryan Gosling, Ana de Armas\n",
      "Plot:  Young Blade Runner K's discovery of a long-buried secret leads him to track down former Blade Runner Rick Deckard, who's been missing for thirty years.\n",
      "Language:  English, Finnish, Japanese, Hungarian, Russian, Somali, Spanish\n",
      "Country:  United States, United Kingdom, Canada, Spain\n",
      "Awards:  Won 2 Oscars. 101 wins & 164 nominations total\n",
      "Poster:  https://m.media-amazon.com/images/M/MV5BNzA1Njg4NzYxOV5BMl5BanBnXkFtZTgwODk5NjU3MzI@._V1_SX300.jpg\n",
      "Ratings:  [{'Source': 'Internet Movie Database', 'Value': '8.0/10'}, {'Source': 'Rotten Tomatoes', 'Value': '88%'}, {'Source': 'Metacritic', 'Value': '81/100'}]\n",
      "Metascore:  81\n",
      "imdbRating:  8.0\n",
      "imdbVotes:  524,957\n",
      "imdbID:  tt1856101\n",
      "Type:  movie\n",
      "DVD:  16 Jan 2018\n",
      "BoxOffice:  $92,054,159\n",
      "Production:  N/A\n",
      "Website:  N/A\n",
      "Response:  True\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import json\n",
    "\n",
    "\n",
    "url = url = 'http://www.omdbapi.com/?t=Blade+Runner+2049&apikey=8c241015'\n",
    "\n",
    "r = requests.get(url)\n",
    "j_data = r.json()\n",
    "\n",
    "\n",
    "for k,v in j_data.items():\n",
    "    print(k+': ', v)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "355bcae6-329e-484d-b471-acda1f0b4f31",
   "metadata": {},
   "source": [
    "## Checking out the Wikipedia API\n",
    "\n",
    "You're doing so well and having so much fun that we're going to throw one more API at you: the Wikipedia API (documented here). You'll figure out how to find and extract information from the Wikipedia page for Pizza. What gets a bit wild here is that your query will return nested JSONs, that is, JSONs with JSONs, but Python can handle that because it will translate them into dictionaries within dictionaries.\n",
    "\n",
    "The URL that requests the relevant query from the Wikipedia API is\n",
    "\n",
    "https://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&exintro=&titles=pizza\n",
    "\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Assign the relevant URL to the variable url.\n",
    "    Apply the \".json()\" method to the response object \"r\" and store the resulting dictionary in the variable json_data.\n",
    "    The variable pizza_extract holds the HTML of an extract from Wikipedia's Pizza page as a string; use the function print() to print this string to the shell.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61ff7b5d-5d15-4bb1-9fbc-07ffab842fd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "\n",
    "url = 'https://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&exintro=&titles=pizza'\n",
    "url = 'https://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&exintro=&titles=coffee'\n",
    "\n",
    "proxies = {\"https\": \"socks4://5.160.184.243:4145\"}\n",
    "#proxies = {\"https\": \"https://164.68.105.235:3128\"}\n",
    "#proxies = {\"https\": \"socks5://5.189.224.84:10000\"}\n",
    "#proxies = {\"https\": \"https://129.226.52.93:443\"}\n",
    "\n",
    "\n",
    "r = requests.get(url, proxies=proxies)\n",
    "\n",
    "j_data = r.json()\n",
    "print(j_data)\n",
    "\n",
    "\n",
    "# Above code works o Colab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1b2f8147-eb31-49ef-966c-a11f68c77619",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import package\n",
    "import requests\n",
    "\n",
    "# Assign URL to variable: url\n",
    "url = 'https://en.wikipedia.org/w/api.php?action=query&prop=extracts&format=json&exintro=&titles=pizza'\n",
    "\n",
    "# Package the request, send the request and catch the response: r\n",
    "r = requests.get(url)\n",
    "\n",
    "# Decode the JSON data into a dictionary: json_data\n",
    "json_data = r.json()\n",
    "\n",
    "\n",
    "\n",
    "# Print the Wikipedia page extract\n",
    "pizza_extract = json_data['query']['pages']['24768']['extract']\n",
    "print(pizza_extract)\n",
    "\n",
    "\n",
    "# Anyway, we know its a nested dict, so nested key calling is obvious, just checking the url"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b748a6fa-3ac1-4729-b76f-e1e02315e7e0",
   "metadata": {},
   "source": [
    "## The Twitter API and Authentication\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**Congratulations on interacting with very first APIs and getting data from them.  You are on the home stretch now.  As a final deep dive, you're going to stream data from the Twitter API.  You'll learn how to filter incoming tweets for keywords, you'll learn about the principlesof API authentication and OAuth.  You'll also learn the basics of the packages tweepy, which many people in PythonLand use to interact with the Twitter API.  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "One of the first major differences between the Twitter API and all the APIs we have seen so far is that you were able to access all the others anonymously and Twitter requires that you have an account.  In order to gain access to the Twitter API, one need to create a Twitter account if you dont already have one, log into Twitter Apps and click \"Create a New App\" - you'll need to agree to a varietyof termsand conditions here, then go to your \"Keys and Access Tokens\" tab and copy your API key, your API secret, your Access Token and your Access Token secret.  These are the Authentication credentials that will allow you to access the Twitter API from Python.  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "In the following interactive exercises, we won't require that you create your own Twitter account and App: we'll do mock run-through of how you would stream data and analyze as if you done so.  It is now important to mention that Twitterhas a number of APIs.  Firstly, they have REST API; we won't go into the gory details of REST APIs here but I'll say 2 things - one, REST is short for Representational State Transfer; 2, Twitter's REST API allows the user to \"read and write Twitter data\"; in order to \"monitor or process Tweets in real-time\", that is to stream Twitter data, however, we'll want to use Twitter's Streaming API.  \n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "In particular, we'll use the public stream which Twitter's API documentation states \"Stream of the public data flowing through Twitter.\"  The Public Stream itself contains a number of options.  As we want to read and process Tweets, we'll want to use the GET statuses/sample API, which \"Returns a small random sample of all public streams.\"  \n",
    "If you wanted to access absolutely \"All public statuses\", you would need to use Twitter's Firehose API, which is not publicly available and would most likely cost you a pretty penny.  \n",
    "\n",
    "One last point to note before we begin streaming Tweets: Tweets are returned to us as JSONs and they contain numerous possible fields.  Check out the Twitter tweet field guide on Google.  You can get Tweet text, user, langurage, time of Tweet, among many other fields.  \n",
    "# ( here can you recall we use \"field names\" to access columns in a NumPy array of flat files )\n",
    "\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "Lets now see how to access and stream data from the Twitter API.  For first-time Python tweet-streamers, we usually  recommend the package \"tweepy\", which has nice balance of usability and capability.  Lets now use it to stream some tweets.  \n",
    "\n",
    "\n",
    "# First off, it has an OAuth handler which takes care of all of that nasty stuff for you: all you need to do is pass the API Key and Secret to the handler and then to pass access credentials using the \"set_access_token()\" method.  \n",
    "\n",
    "After this you'll need to define your Twitter stream listener Class.  We wouldn't necessarily expect you to be able to do this yourdelf so we are going to do that for you both here and in the interactive exercises that follow.  \n",
    "\n",
    "# *******************************************************************************************************************\n",
    "Here we define a Tweet listener that creates a file called \"tweets.txt\", collects streaming tweets and writes them to the file \"tweets.txt\"; once 100 tweets have been streamed, the listener closes the file and stops listening.  Now that we have written our Twitter Stream Listener Class, all you need to do is to create an instance of it and authenticate it.  \n",
    "\n",
    "# You can then stream tweets that containing keywordsof choice by applying the \".filter()\" method to the object stream.  In the following exercises, you'll practice writing Python code to stream tweets and then you'll do some basic analysis of these tweets to see how often particular keywords are mentioned.  Happy streaming\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "631da2bb-0d9a-48bf-ac48-074a468174bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tweepy, json\n",
    "\n",
    "\n",
    "access_token = ''\n",
    "access_token_secret = ''\n",
    "consumer_key = ''\n",
    "consumer_secret = ''\n",
    "\n",
    "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)  ##########################################################\n",
    "auth.set_access_token(access_token, access_token_secret)  ###########################################################\n",
    "\n",
    "\n",
    "\n",
    "#####################################################################################################################\n",
    "\n",
    "# st_class.py\n",
    "class MyStreamListener(tweepy.StreamListener):\n",
    "    \n",
    "    def __init__(self, api=None):\n",
    "        super(MyStreamListener, self).__init__()\n",
    "        self.num_tweets = 0\n",
    "        self.file = open('tweets.txt', 'w')\n",
    "        \n",
    "    def on_status(self, status):\n",
    "        tweet = status._json\n",
    "        self.file.write(json.dumps(tweet) + '\\\\n')\n",
    "        tweet_list.append(status)\n",
    "        \n",
    "        self.num_tweets += 1\n",
    "        if self.num_tweets < 100:\n",
    "            return True\n",
    "        else:\n",
    "            return False\n",
    "        self.file.close()\n",
    "        \n",
    "#####################################################################################################################\n",
    "        \n",
    "\n",
    "# Create Streaming object and authenticate\n",
    "l = MyStreamListener()\n",
    "stream = tweepy.Stream(auth, l)\n",
    "stream.apply(tracks=['apples', 'orages'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9baf54f0-91a8-414c-96f8-a57dc09fb8d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tweepy\n",
    "import json\n",
    "\n",
    "class MyStreamListener(tweepy.StreamListener):\n",
    "    def on_status(self, status):\n",
    "        print(status.text)\n",
    "\n",
    "\n",
    "\n",
    "access_token = \"1092294848-aHN7DcRP9B4VMTQIhwqOYiB14YkW92fFO8k8EPy\"\n",
    "access_token_secret = \"X4dHmhPfaksHcQ7SCbmZa2oYBBVSD2g8uIHXsp5CTaksx\"\n",
    "consumer_key = \"nZ6EA0FxZ293SxGNg8g8aP0HM\"\n",
    "consumer_secret = \"fJGEodwe3KiKUnsYJC3VRndj7jevVvXbK2D5EiJ2nehafRgA6i\"\n",
    "\n",
    "\n",
    "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set_access_token(access_token, access_token_secret)\n",
    "\n",
    "\n",
    "myStreamListener = MyStreamListener()\n",
    "myStream = tweepy.Stream(auth = auth, listener=myStreamListener)\n",
    "\n",
    "\n",
    "print(myStream.body) ###\n",
    "\n",
    "# Filter Twitter Streams to capture data by the keywords:\n",
    "myStream.filter(track=['Japan', 'Iran'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6e57c0da-6130-4513-8d0c-ae4d32deb2a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tweepy\n",
    "import json\n",
    "\n",
    "\n",
    "class MyStreamListener(tweepy.StreamListener):\n",
    "\n",
    "    def __init__(self, api=None):\n",
    "        super(MyStreamListener, self).__init__()\n",
    "        self.num_tweets = 0\n",
    "        self.file = open('tweets.txt', \"w\")\n",
    "\n",
    "    def on_status(self, status):\n",
    "        tweet = status._json\n",
    "        self.file.write( json.dumps(tweet) + '\\n' )\n",
    "        self.num_tweets += 1\n",
    "        \n",
    "        # Stops streaming when it reaches the limit\n",
    "        if self.num_tweets <= 50:\n",
    "            if self.num_tweets % 10 == 0: # just to see some progress...\n",
    "                print('Numer of tweets captured so far: {}'.format(self.num_tweets))\n",
    "            return True\n",
    "        else:\n",
    "            return False\n",
    "        self.file.close()\n",
    "\n",
    "    def on_error(self, status):\n",
    "        print(status)\n",
    "\n",
    "\n",
    "# Store OAuth authentication credentials in relevant variables (these are DataCamp tokens)\n",
    "access_token = \"1092294848-aHN7DcRP9B4VMTQIhwqOYiB14YkW92fFO8k8EPy\"\n",
    "access_token_secret = \"X4dHmhPfaksHcQ7SCbmZa2oYBBVSD2g8uIHXsp5CTaksx\"\n",
    "consumer_key = \"nZ6EA0FxZ293SxGNg8g8aP0HM\"\n",
    "consumer_secret = \"fJGEodwe3KiKUnsYJC3VRndj7jevVvXbK2D5EiJ2nehafRgA6i\"\n",
    "\n",
    "# Pass OAuth details to tweepy's OAuth handler\n",
    "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set_access_token(access_token, access_token_secret)\n",
    "\n",
    "# Initialize Stream listener\n",
    "l = MyStreamListener()\n",
    "\n",
    "# Create you Stream object with authentication\n",
    "stream = tweepy.Stream(auth, l)  ####################################################################################\n",
    "print(dir(stream)) ###\n",
    "print(stream.body) ###\n",
    "# Filter Twitter Streams to capture data by the keywords:\n",
    "stream.filter(track=['Japan', 'Iran'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2783c75-40f9-4d3e-b1e8-c6322d7df1e8",
   "metadata": {},
   "source": [
    "## API Authentication\n",
    "\n",
    "The package tweepy is great at handling all the Twitter API OAuth Authentication details for you. All you need to do is pass it your authentication credentials. In this interactive exercise, we have created some mock authentication credentials (if you wanted to replicate this at home, you would need to create a Twitter App as Hugo detailed in the video). Your task is to pass these credentials to tweepy's OAuth handler.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Import the package tweepy.\n",
    "#    Pass the parameters \"consumer_key\" and \"consumer_secret\" to the function \"tweepy.OAuthHandler()\".\n",
    "#    Complete the passing of OAuth credentials to the OAuth handler auth by applying to it the method \"set_access_token()\", along with arguments \"access_token\" and \"access_token_secret\".\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53b014e2-e8af-46dd-b7fd-7c4b40957171",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import package\n",
    "import tweepy\n",
    "\n",
    "\n",
    "# Store OAuth authentication credentials in relevant variables\n",
    "access_token = \"1092294848-aHN7DcRP9B4VMTQIhwqOYiB14YkW92fFO8k8EPy\"\n",
    "access_token_secret = \"X4dHmhPfaksHcQ7SCbmZa2oYBBVSD2g8uIHXsp5CTaksx\"\n",
    "consumer_key = \"nZ6EA0FxZ293SxGNg8g8aP0HM\"\n",
    "consumer_secret = \"fJGEodwe3KiKUnsYJC3VRndj7jevVvXbK2D5EiJ2nehafRgA6i\"\n",
    "\n",
    "\n",
    "# Pass OAuth details to tweepy's OAuth handler\n",
    "auth = tweepy.OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set.access_token(access_token, access_token_secret)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86f7cddf-31e8-4d40-a82d-c3f3898a7260",
   "metadata": {},
   "source": [
    "## Streaming tweets\n",
    "\n",
    "Now that you have set up your authentication credentials, it is time to stream some tweets! We have already defined the tweet stream listener class, MyStreamListener, just as Hugo did in the introductory video. You can find the code for the tweet stream listener class here.\n",
    "\n",
    "Your task is to create the Streamobject and to filter tweets according to particular keywords.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Create your Stream object with authentication by passing tweepy.Stream() the authentication handler auth and the Stream listener l;\n",
    "    To filter Twitter streams, pass to the track argument in stream.filter() a list containing the desired keywords 'clinton', 'trump', 'sanders', and 'cruz'.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb7648db-78e9-42e2-a694-b355cf7a9ef7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Initialize Stream listener\n",
    "l = MyStreamListener()\n",
    "\n",
    "# Create your Stream object with authentication\n",
    "stream = tweepy.Stream(____, ____)\n",
    "\n",
    "# Filter Twitter Streams to capture data by the keywords:\n",
    "stream.filter(____)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32f09658-2bf3-4caa-ac00-6132cd0bc5df",
   "metadata": {},
   "source": [
    "## Load and explore your Twitter data\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "Now that you've got your Twitter data sitting locally in a text file, it's time to explore it! This is what you'll do in the next few interactive exercises. In this exercise, you'll read the Twitter data into a list: \"tweets_data\".\n",
    "\n",
    "Be aware that this is real data from Twitter and as such there is always a risk that it may contain profanity or other offensive content (in this exercise, and any following exercises that also use real Twitter data).\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Assign the filename 'tweets.txt' to the variable \"tweets_data_path\".\n",
    "    Initialize \"tweets_data\" as an empty list to store the tweets in.\n",
    "#    Within the for loop initiated by for line in \"tweets_file\":, load each tweet into a variable, tweet, using \"json.loads()\", then append tweet to tweets_data using the append() method.\n",
    "    Hit submit and check out the keys of the first tweet dictionary printed to the shell.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8c0e2dbf-a7f4-421b-9f5c-8d7edd72c9ef",
   "metadata": {},
   "outputs": [
    {
     "ename": "IndentationError",
     "evalue": "expected an indented block (4097757246.py, line 5)",
     "output_type": "error",
     "traceback": [
      "\u001b[0;36m  File \u001b[0;32m\"/tmp/ipykernel_2281/4097757246.py\"\u001b[0;36m, line \u001b[0;32m5\u001b[0m\n\u001b[0;31m    for i in file:\u001b[0m\n\u001b[0m                  ^\u001b[0m\n\u001b[0;31mIndentationError\u001b[0m\u001b[0;31m:\u001b[0m expected an indented block\n"
     ]
    }
   ],
   "source": [
    "tweets_data_path = 'tweets.txt'\n",
    "\n",
    "tweets_data = []\n",
    "with open('tweets_data_path', 'r') as file:\n",
    "    for i in file:\n",
    "        tweet = json.loads(i)\n",
    "        tweets_data.append(tweet)\n",
    "        \n",
    "        \n",
    "print(tweets_data[:3])\n",
    "\n",
    "\n",
    "# Need to try it on my Colab env and my own Twitter API"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b3cbccc-0556-400c-98ce-e04274e3e570",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import package\n",
    "import json\n",
    "\n",
    "# String of path to file: tweets_data_path\n",
    "\n",
    "\n",
    "# Initialize empty list to store tweets: tweets_data\n",
    "\n",
    "\n",
    "# Open connection to file\n",
    "tweets_file = open(tweets_data_path, \"r\")\n",
    "\n",
    "# Read in tweets and store in list: tweets_data\n",
    "for line in tweets_file:\n",
    "    ____\n",
    "    ____\n",
    "\n",
    "# Close connection to file\n",
    "tweets_file.close()\n",
    "\n",
    "# Print the keys of the first tweet dict\n",
    "print(tweets_data[0].keys())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6da924c4-a7c0-4dec-b3df-f63857b7a9c6",
   "metadata": {},
   "source": [
    "## Twitter data to DataFrame\n",
    "\n",
    "Now you have the Twitter data in a list of dictionaries, \"tweets_data\", where each dictionary corresponds to a single tweet. Next, you're going to extract the text and language of each tweet. The text in a tweet, \"t1\", is stored as the value \"t1['text']\"; similarly, the language is stored in \"t1['lang']\". Your task is to build a DataFrame in which each row is a tweet and the columns are 'text' and 'lang'.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "#    Use \"pd.DataFrame()\" to construct a DataFrame of tweet texts and languages; to do so, the first argument should be \"tweets_data\", a list of dictionaries. The second argument to \"pd.DataFrame()\" is a list of the keys you wish to have as columns. Assign the result of the pd.DataFrame() call to \"df\".\n",
    "    Print the head of the DataFrame.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a507d91c-dae4-4862-be02-9dd83f43177f",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame()\n",
    "\n",
    "for i in tweets_data:\n",
    "    df['text'] = i['text']\n",
    "    df['lang'] = i['lang']\n",
    "    \n",
    "    \n",
    "print(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e49a90f0-dcb1-417c-aefd-441a58e3af15",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import package\n",
    "import pandas as pd\n",
    "\n",
    "# Build DataFrame of tweet texts and languages\n",
    "df = pd.DataFrame(____, columns=____)  ##############################################################################\n",
    "\n",
    "# Print head of DataFrame\n",
    "print(df.head())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04e50436-6e94-45f7-a166-60eff4449a9b",
   "metadata": {},
   "source": [
    "## A little bit of Twitter text analysis\n",
    "\n",
    "Now that you have your DataFrame of tweets set up, you're going to do a bit of text analysis to count how many tweets contain the words 'clinton', 'trump', 'sanders' and 'cruz'. In the pre-exercise code, we have defined the following function word_in_text(), which will tell you whether the first argument (a word) occurs within the 2nd argument (a tweet).\n",
    "\n",
    "# *******************************************************************************************************************\n",
    "import re\n",
    "\n",
    "def word_in_text(word, text):\n",
    "    word = word.lower()\n",
    "    text = text.lower()\n",
    "    match = re.search(word, text)\n",
    "\n",
    "    if match:\n",
    "        return True\n",
    "    return False\n",
    "# *******************************************************************************************************************\n",
    "\n",
    "You're going to iterate over the rows of the DataFrame and calculate how many tweets contain each of our keywords! The list of objects for each candidate has been initialized to 0.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Within the for loop for index, row in df.iterrows():, the code currently increases the value of clinton by 1 each time a tweet (text row) mentioning 'Clinton' is encountered; complete the code so that the same happens for trump, sanders and cruz.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7078001b-6c2d-4f82-b490-dad790476631",
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "def word_in_text(word, text):\n",
    "    word = word.lower()\n",
    "    text = text.lower()\n",
    "    match = re.search(word, text)\n",
    "\n",
    "    if match:\n",
    "        return True\n",
    "    return False\n",
    "\n",
    "\n",
    "\n",
    "# Initialize list to store tweet counts\n",
    "[clinton, trump, sanders, cruz] = [0, 0, 0, 0]\n",
    "\n",
    "# Iterate through df, counting the number of tweets in which\n",
    "# each candidate is mentioned\n",
    "for index, row in df.iterrows():\n",
    "    clinton += word_in_text('clinton', row['text'])\n",
    "    trump += word_in_text(____, ____)\n",
    "    sanders += word_in_text(____, ____)\n",
    "    cruz += word_in_text(____, ____)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00ba2c2c-3520-44a0-a25c-fad3876701b2",
   "metadata": {},
   "source": [
    "## Plotting your Twitter data\n",
    "\n",
    "Now that you have the number of tweets that each candidate was mentioned in, you can plot a bar chart of this data. You'll use the statistical data visualization library seaborn, which you may not have seen before, but we'll guide you through. You'll first import seaborn as sns. You'll then construct a barplot of the data using \"sns.barplot()\", passing it two arguments:\n",
    "\n",
    "    a list of labels and\n",
    "    a list containing the variables you wish to plot (clinton, trump and so on.)\n",
    "\n",
    "Hopefully, you'll see that Trump was unreasonably represented! We have already run the previous exercise solutions in your environment.\n",
    "Instructions\n",
    "100 XP\n",
    "\n",
    "    Import both \"matplotlib.pyplot\" and \"seaborn\" using the aliases plt and sns, respectively.\n",
    "#    Complete the arguments of \"sns.barplot()\":\n",
    "#        The first argument should be the list of labels to appear on the x-axis (created in the previous step).\n",
    "#        The second argument should be a list of the variables you wish to plot, as produced in the previous exercise (i.e. a list containing clinton, trump, etc).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "68f0f336-9c7e-4891-8215-90731fe1082a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import packages\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Set seaborn style\n",
    "sns.set(color_codes=True)\n",
    "\n",
    "# Create a list of labels:cd\n",
    "cd = ['clinton', 'trump', 'sanders', 'cruz']\n",
    "\n",
    "# Plot the bar chart\n",
    "ax = sns.barplot(cd, [clinton, trump, sanders, cruz])\n",
    "ax.set(ylabel=\"count\")\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0caf76ab-6afd-4c5b-9567-c29abcafca15",
   "metadata": {},
   "source": [
    "## Final Thoughts\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "**Wowee, congratulations.  We just completed our deep dive (if Twitter offered my dev account) into the Twitter API, in which we streamed tweets, processed them and visualized our results.  \n",
    "\n",
    "# If we made this far, you're now able to import data in Python from a wide array of sources.  To recapthe skill-set and data importing chops that you gained in this course and its prequel, you're now adept at importing basic text files and flat files, the basic bread and butter of any working Data Scientist's professional life, local files in other formats, such as Excel spreeasheet, SAS, Stata, and MATLAB files, pickle and HDF5 files.  These skills will make you even better collaborator for many working professional out there.  \n",
    "\n",
    "You've built up your basic skils at writing SQL queries and can now get all types of data out of relational databases.  You're able to pull data from the web: not only can you import basic files, but you know a number of ways to issue GET requests and even do some basic web scraping and HTML parsing.  \n",
    "\n",
    "On top of all of this, you've learnt how to pull data from APIs and have had hands-on experience doing so with dives into several examples.  You've learnt a great deal and successfully completed these two courses on Importing Data in Python.  Happy importing.  \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74e38df4-5af6-4a2d-aaf8-93346b5d5ee7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e840d8c3-475a-4c59-83e8-c6c7db570e3a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0f4bb0ba-5e51-4cbd-9af3-67d9aebc0b15",
   "metadata": {},
   "source": [
    "# Any one can offer an usable Twitter API keys? Please help"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f86e5dc2-8c26-4e9d-a439-0ae0902133ea",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42b132a3-026a-454f-9e58-e36611c45869",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e34785ea-95b4-4e8c-825c-7cad72e86c3d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "961d19b5-72d0-4f79-8786-b49a71703f56",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a11ee57f-5256-4948-8866-c0155fe5856f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5f131514-4f6a-4ebe-9ec3-a8b0bddec88f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f5c55e0-5d85-40a9-af70-92fd92260a10",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35876360-a5f6-453b-bf84-568abd75a3ad",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "42b1d911-a5c0-4863-b8c3-62a585951577",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61d23a86-0095-4222-8f54-0cf07a88f032",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4d3fd457-9d52-4fa5-b571-66d37265b0a8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "82de786b-38e6-4c55-9b24-0569c2cc5f85",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c428e3d-b902-4e68-b800-22992baece94",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}