demonstration.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demonstration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 - Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Film Data Setup\n",
    "\n",
    "Please refer to [this](./data/film_config_template.yaml) YAML file as template. The film data can be merged with your review data if you wish to.\n",
    "\n",
    "Once you have a film data YAML file, set up the path to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please replace the path with your own\n",
    "FILM_DATA_FILE = \"/Users/pipchang/Documents/VSC/Projects/DH-S/film_data/drive.yaml\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Review Data Setup\n",
    "\n",
    "Navigate to the film review page you want to scrape and copy the URL.\n",
    "\n",
    "Currently, the package supports the following pages:\n",
    "\n",
    "- IMDB audience review:\n",
    "\n",
    "Note: the sorting option does not matter as all reviews can be retrieved\n",
    "\n",
    "![Drive (2011) - IMDB](./data/imdb.png)\n",
    "\n",
    "- Rotten Tomatoes audience review:\n",
    "\n",
    "![Drive (2011) - Rotten Tomatoes](./data/rt.png)\n",
    "\n",
    "- Douban audience short review\n",
    "\n",
    "Note: due to website ristrictions, up to 600 short reviews can be retrieved under each sorting option when logged in (200 when not).\n",
    "\n",
    "It is recommended to run it several times, each under a different sorting option.\n",
    "\n",
    "![Drive (2011) - Douban Short](./data/db_short.png)\n",
    "\n",
    "- Douban audiencelong review\n",
    "\n",
    "Note: the scraper will first retrievd all long reviews' permalinks, then open each one to retrieve the review and the comments under the review\n",
    "\n",
    "It takes a bit longer to run. The sorting option does not matter.\n",
    "\n",
    "![Drive (2011) - Douban Long](./data/db_long.png)\n",
    "\n",
    "After getting the needed URLs, you can set them up like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "FILM_NAME = \"Drive\"\n",
    "IMDB_URL = \"https://www.imdb.com/title/tt0780504/reviews?sort=submissionDate&dir=desc&ratingFilter=0\"\n",
    "RT_URL = \"https://www.rottentomatoes.com/m/drive_2011/reviews?type=user\"\n",
    "DB_SHORT_URL = \"https://movie.douban.com/subject/2010972/comments?status=P\"\n",
    "DB_LONG_URL = \"https://movie.douban.com/subject/2010972//reviews?sort=hotest\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Folder Setup\n",
    "\n",
    "You need to specify the folder path where you want to store the scraped reviews. \n",
    "\n",
    "If the folder does not exist yet, it will be created automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Please replace this with your own\n",
    "OUTPUT_FOLDER = \"/Users/pipchang/Documents/VSC/Projects/DH-S/download/Drive\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 - Scraping\n",
    "\n",
    "Essentially, every ```Website``` class has functions ```fetch_reviews``` (retrieve reviews in the form of Beautifulsoup objects) and ```parse_reviews``` (parse Beautifulsoup objects into certain ```Review``` dataclasses).\n",
    "\n",
    "Note: you can run ```fetch_reviews``` in headless mode if you pass it the parameter ```headless_mode=True``` to run Chrome without an active window.\n",
    "\n",
    "Please refer to the [Documentation](https://film-review-scraper.readthedocs.io/en/latest/) for more specific parameters. \n",
    "\n",
    "Here are examples for each website:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IMDB\n",
    "from film_review_scraper import IMDB\n",
    "imdb = IMDB()\n",
    "imdb_review_blocks = imdb.fetch_reviews(url=IMDB_URL)\n",
    "imdb_reviews = imdb.parse_reviews(imdb_review_blocks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Rotten Tomatoes\n",
    "from film_review_scraper import RottenTomatoes\n",
    "rt = RottenTomatoes()\n",
    "rt_review_blocks = rt.fetch_reviews(url=IMDB_URL)\n",
    "rt_reviews = rt.parse_reviews(rt_review_blocks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Douban Short and Long\n",
    "from film_review_scraper import Douban\n",
    "db = Douban()\n",
    "\n",
    "db_short_review_blocks = db.fetch_reviews(\n",
    "    url=DB_SHORT_URL,\n",
    "    review_type=\"short\",\n",
    ")\n",
    "db_short_reviews = db.parse_reviews(review_blocks=db_short_review_blocks, parse_type=\"short\")\n",
    "\n",
    "db_long_review_blocks = db.fetch_reviews(\n",
    "    url=DB_LONG_URL,\n",
    "    review_type=\"long\",\n",
    ")\n",
    "db_long_reviews = db.parse_reviews(review_blocks=db_long_review_blocks, parse_type=\"long\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 - Storing Reviews\n",
    "\n",
    "You can store your scraped reviews with ```save_dataclass_to_jsonl```. For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from film_review_scraper import get_output_path, save_dataclass_to_jsonl\n",
    "\n",
    "# Specify the file name\n",
    "output_path = get_output_path(\n",
    "    folder_path=OUTPUT_FOLDER,\n",
    "    file_name=f\"imdb_{FILM_NAME}_newest\",  # You can change \"newest\" to something else based on the sorting option\n",
    "    file_type=\"jsonl\",\n",
    ")\n",
    "\n",
    "# Save to JSONL file\n",
    "save_dataclass_to_jsonl(objects=imdb_reviews, output_path=output_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4 - Merging Film Data & Review Data\n",
    "\n",
    "With all of your scraped review data in a folder, you can merge them all together and with film data into a single JSONL file.\n",
    "\n",
    "Each line of JSONL file is a dictionary of film data and review data.\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from film_review_scraper import get_output_path, FilmData, ReviewData, merge_film_and_review_data, save_dicts_to_jsonl\n",
    "\n",
    "# Specify the folder where the review data files are\n",
    "INPUT_FOLDER = \"/Users/pipchang/Documents/VSC/Projects/DH-S/download/Drive\"\n",
    "\n",
    "# Specify the folder where you want to store the merged file\n",
    "MERGED_FOLDER = \"/Users/pipchang/Documents/VSC/Projects/DH-S/download/Drive/Merged\"\n",
    "\n",
    "# Get film data\n",
    "film_data = FilmData.from_file(FILM_DATA_FILE)\n",
    "\n",
    "# Get review data\n",
    "review_data = ReviewData.from_folder(\n",
    "    folder_path=INPUT_FOLDER, film_name=FILM_NAME\n",
    ")\n",
    "\n",
    "# Merge data\n",
    "merged_data = merge_film_and_review_data(film_data=film_data, review_data=review_data)\n",
    "\n",
    "# Store merged data\n",
    "output_path = get_output_path(\n",
    "    folder_path=MERGED_FOLDER, \n",
    "    file_name=f'{FILM_NAME}_Merged', \n",
    "    file_type=\"jsonl\"\n",
    ")\n",
    "save_dicts_to_jsonl(list_of_dicts=merged_data, output_path=output_path)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "45a86756a88996e814c4432654fc41ed6880797c3f685411ffd1be84db613c4e"
  },
  "kernelspec": {
   "display_name": "Python 3.10.9 ('pipjenew')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.9"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}