speech_maker.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Speech Maker\n",
    "\n",
    "This notebook uses the [markovify](https://github.com/jsvine/markovify) library to generate Markov chains from a [dataset](https://floydhub.com/whatrocks/datasets/commencement) of popular commencement speeches. Among other things, this simple, extensible library is used by [many](https://twitter.com/MarkovPicard), [many](https://twitter.com/mashomatic) Twitter bots.\n",
    "\n",
    "### Part 1: Entire Dataset\n",
    "\n",
    "Let's first generate some sentences from our entire dataset of speeches."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import markovify\n",
    "\n",
    "SPEECH_PATH = '/floyd/input/speeches/'\n",
    "\n",
    "speech_dict = {}\n",
    "for speech_file in os.listdir(SPEECH_PATH):\n",
    "    with open(f'{SPEECH_PATH}{speech_file}') as speech:\n",
    "        contents = speech.read()\n",
    "        # Create a Markov model for each speech in our dataset\n",
    "        model = markovify.Text(contents)\n",
    "        speech_dict[speech_file] = model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = list(speech_dict.values())\n",
    "print(f'There are {len(models)} speeches in our dataset.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine the Markov models\n",
    "model_combination = markovify.combine(models)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate 3 sentences\n",
    "for i in range(3):\n",
    "    print(f'{i}: {model_combination.make_sentence()}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 2: Top 10 Schools by Speech Count\n",
    "\n",
    "Let's be elitist for a moment, shall we? Let's take a look at the top ten schools by speech count, and generate some sentences from these paragons of higher education."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "\n",
    "df = pd.read_csv('speech_metadata.csv')\n",
    "\n",
    "# Top 10 schools by speech count\n",
    "top_ten_schools = df.school.value_counts().head(10)\n",
    "top_ten_schools.plot.bar()\n",
    "\n",
    "# Make it a Python list\n",
    "top_ten_schools_list = top_ten_schools.index.tolist()\n",
    "\n",
    "# Make list of all speech filenames for these top 10 schools\n",
    "filtered_speech_filenames = []\n",
    "for index, row in df.iterrows():\n",
    "    if row['school'] in top_ten_schools_list:\n",
    "        filtered_speech_filenames.append(row['filename'])\n",
    "\n",
    "# Filter our { filename: markov_model } dict by these filenames\n",
    "filtered_speech_dict = { speech_file: speech_dict[speech_file] for speech_file in filtered_speech_filenames }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filtered_models = list(filtered_speech_dict.values())\n",
    "print(f'Using {len(filtered_models)} speeches!\\n')\n",
    "\n",
    "# Combine these models and print some new sentences\n",
    "filtered_model_combination = markovify.combine(filtered_models)\n",
    "for i in range(3):\n",
    "    print(f'{i}: {filtered_model_combination.make_sentence()}\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Part 3: One school at a time\n",
    "\n",
    "In this last section, let's play with a super fun [Jupyter widget extension](http://ipywidgets.readthedocs.io/en/latest/index.html) that lets us filter the speeches by each of the top ten schools. \n",
    "\n",
    "Why would you want to do this? Well, sometimes you really just want to generate a sentence from commencement speeches given at Stanford University. Or was it MIT? That's entirely up to you!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Actually, let's try these schools one at a time!\n",
    "import ipywidgets as widgets\n",
    "\n",
    "toggle = widgets.ToggleButtons(\n",
    "    options=top_ten_schools_list,\n",
    "    description='Schools: ',\n",
    "    disabled=False,\n",
    "    button_style='',\n",
    "    tooltips=[]\n",
    ")\n",
    "display(toggle)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Make list of all speech filenames for selected school\n",
    "filtered_speech_filenames = []\n",
    "for index, row in df.iterrows():\n",
    "    if row['school'] == toggle.value:\n",
    "        filtered_speech_filenames.append(row['filename'])\n",
    "\n",
    "# Filter our { filename: markov_model } dict by these filenames\n",
    "filtered_speech_dict = { speech_file: speech_dict[speech_file] for speech_file in filtered_speech_filenames }\n",
    "\n",
    "filtered_models = list(filtered_speech_dict.values())\n",
    "print(f'There are {len(filtered_models)} speeches from {toggle.value}!\\n')\n",
    "\n",
    "# Combine these models and print some new sentences\n",
    "filtered_model_combination = markovify.combine(filtered_models)\n",
    "for i in range(3):\n",
    "    print(f'{i}: {filtered_model_combination.make_sentence()}\\n')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}