exercises/Exercise-2-Genre-Classification-with-MFCCs.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DSC160 Data Science and the Arts - Twomey - Spring 2020 - [dsc160.roberttwomey.com](http://dsc160.roberttwomey.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2: Audio Classification with MFCCs\n",
    "\n",
    "This exercise walks you through feature extraction and genre classification based on MFCCs, using audio frames extracted from two instances of distinct audio styles (classical and rap music).\n",
    "\n",
    "It has two parts:\n",
    "- [Part 1](#Part-1:-Genre-Recognition). In this part you will load two audio files as genre examples, extract MFCC features from each, and implement a simple SVM classifier.\n",
    "  - [Step 1 - Load Files and Display Audio](#Step-1:-Load-Files-and-Display-Audio)\n",
    "  - [Step 2 - Extract Features](#Step-2:-Extract-Features)\n",
    "  - [Step 3 - Train a Classifier](#Step-3:-Train-a-Classifier)\n",
    "  - [Step 4 - Run the Classifier](#Step-4:-Run-the-Classifier)\n",
    "- [Part 2](#Part-2:-Extension). In this part you will extend the work from Part 1, either creating a new classifier using new genre examples and training examples from your own music collection, implementing a new classifier method, or experimenting with different features for the SVM and existing examples.\n",
    "  - [Part 2A - Code for your Extension](#2A.-Code-for-Extension)\n",
    "  - [Part 2B - Discussion of Results](#2B.-Discussion-of-Results)\n",
    "  \n",
    "Once you have completed both parts, you will submit your completed notebook as a pdf to gradescope for grading.\n",
    "\n",
    "Note: this is a simplified genre classification example. For a more comprehensive approach combining timbral, beat, and pitch features, see Tzanetakis and Cook ['Musical Genre Classification of Audio Signal'](https://pdfs.semanticscholar.org/4ccb/0d37c69200dc63d1f757eafb36ef4853c178.pdf) from IEEE Transactions on Audio and Speech Processing 2002. Many of the techniques described in that paper can be implemented using librosa and our numpy/scipy toolkits for your own Project 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import necessary modules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn\n",
    "\n",
    "import librosa\n",
    "import librosa.display\n",
    "\n",
    "from IPython.display import Audio\n",
    "\n",
    "import requests\n",
    "import os\n",
    "\n",
    "import sklearn\n",
    "import numpy as np\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 1: Genre Recognition\n",
    "(50 points total)\n",
    "\n",
    "This section walks you through three steps: \n",
    "1. Load and Display Audio Files\n",
    "2. Extracting features from an audio signal.\n",
    "3. Training a genre classifier.\n",
    "4. Using the classifier to classify the genre in a song."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1: Load Files and Display Audio\n",
    "(10 points)\n",
    "\n",
    "We will use two audio pieces as exemplars of distinct audio genres:\n",
    "\n",
    "- Johannes Brahms' ['Hungarian Dance #5 in G Minor'](https://www.youtube.com/watch?v=3X9LvC9WkkQ) (1885) \n",
    "- Busta Rhmyes' ['Hits for Days feat. J Holiday'](https://www.youtube.com/watch?v=B6bt3gWLV5g) (2016) \n",
    "  \n",
    "These have both been added to the course repository in an `audio` subdirectory under the current path:\n",
    "- `audio/brahms_hungarian_dance_5.mp3`\n",
    "- `audio/busta_rhymes_hits_for_days.mp3`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using [`librosa.load`](https://librosa.github.io/librosa/generated/librosa.core.load.html), load 120 seconds of Brahms :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `librosa.display.waveplot`, plot the time-domain waveform of the audio signal for Brahms:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the IPython.display Audio class, play the audio file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using [`librosa.feature.melspectogram`](https://librosa.github.io/librosa/generated/librosa.feature.melspectrogram.html), [`librosa.power_to_db`](https://librosa.github.io/librosa/generated/librosa.core.power_to_db.html), and [`librosa.display.specshow`](https://librosa.github.io/librosa/generated/librosa.display.specshow.html), calculate and display the mel spectogram with a logarithmic magnitude scale:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate mel spectogram\n",
    "# convert spectogram to log spectogram with power_to_db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show spectogram"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat the above steps for the Busta Rhymes song. \n",
    "\n",
    "Load the file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the waveform:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Play the audio file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calculate and display the mel spectogram with a logarithmic magnitude scale:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate\n",
    "\n",
    "# display"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you notice any difference between the wave form or spectogram for the classical and rap song? Is so, what? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```YOUR RESPONSE HERE```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2: Extract Features\n",
    "\n",
    "(10 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to work with MFCCs. For each of your audio files (starting with Brahms), use [`librosa.feature.mfcc`](https://librosa.github.io/librosa/generated/librosa.feature.mfcc.html) to calculate the MFCCs.\n",
    "\n",
    "(Note: you can experiment with `n_mfcc` to select a different number of coefficients, e.g. 12)\n",
    "\n",
    "Start with Brahms, using 12 coefficients, inputting the Brahms time series and Brahms sample rate as the arguments to the mfcc function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: We transpose the result to accommodate scikit-learn which assumes that each row is one observation, and each column is one feature dimension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfcc_brahms = mfcc_brahms.T\n",
    "mfcc_brahms.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using [`sklearn.preprocessing.StandardScaler`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html), scale the features to have zero mean and unit variance:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that the scaling worked (e.g. do we have a mean close to zero and std deviation close to 1 for each feature?):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfcc_brahms_scaled.mean(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfcc_brahms_scaled.std(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Repeat these calculations for Busta Rhymes. Use [`librosa.feature.mfcc`](https://librosa.github.io/librosa/generated/librosa.feature.mfcc.html) to calculate the MFCCs, inputting the Busta time series and Busta sample rate as the arguments to the mfcc function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: Transpose the result to accommodate scikit-learn which assumes that each row is one observation, and each column is one feature dimension:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfcc_busta = mfcc_busta.T\n",
    "mfcc_busta.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scale the resulting MFCC features to have approximately zero mean and unit variance. Re-use the scaler from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that the mean of the MFCCs for the second audio file is approximately equal to zero and the variance is approximately equal to one."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfcc_busta_scaled.mean(axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfcc_busta_scaled.std(axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3: Train a Classifier\n",
    "\n",
    "(15 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Concatenate all of the scaled feature vectors into one feature table using [`np.vstack`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.vstack.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "features.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Construct a vector of ground-truth labels using [`np.concatenate`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.concatenate.html), where 0 refers to the first audio file, and 1 refers to the second audio file. (use `np.zeros` and `np.ones` for brahms and busta rhymes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "labels = np.concatenate((np.zeros(len(mfcc_brahms_scaled)), np.ones(len(mfcc_busta_scaled))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a classifer model object using sklearn's Support Vector Machine [`sklearn.svm.SVC`](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Support Vector Machine\n",
    "model = sklearn.svm.SVC()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Train the classifier with your test data and labels using [`SVC.fit`](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html#sklearn.svm.SVC.fit):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4: Run the Classifier\n",
    "\n",
    "(15 points)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To test the classifier, we will extract an unused 10-second segment from the earlier audio fields as test excerpts:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_brahms_test, fs_brahms = librosa.load(filename_brahms, duration=10, offset=120)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_busta_test, fs_busta = librosa.load(filename_busta, duration=10, offset=120)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the waveform and listen to the audio for both of the test excerpts using `librosa.display.waveplot` and \n",
    "`Audio`. \n",
    "\n",
    "Start with Brahms test (wave plot and audio player):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wave plot\n",
    "\n",
    "# audio playback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next Busta test (waveplot and audio player):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wave plot\n",
    "\n",
    "# audio playback"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute MFCCs from both of the test audio excerpts folowing above using `librosa.feature.mfcc`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate brahms test mfccs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# calculate busta rhymes test mfccs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scale the test sample MFCCs using the previous scaler:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n",
    "mfcc_busta_test_scaled.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n",
    "mfcc_brahms_test_scaled.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Concatenate all test features together using `np.vstack`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_features = np.vstack((mfcc_brahms_test_scaled, mfcc_busta_test_scaled))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Concatenate all test labels together (using `np.concatenate`, with `np.zeros` for brahms and `np.ones` for busta rhymes):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_labels = np.concatenate((np.zeros(len(mfcc_brahms_test_scaled)), np.ones(len(mfcc_busta_test_scaled))))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the predicted labels using `model.predict`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.predict(test_features)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, compute the accuracy score of the classifier on the test data using `model.score`, based on predicted labels and test labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# your code here\n",
    "score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do you believe this classifier is performing well? If so, why? If not, why not?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```WRITE YOUR ANSWER HERE```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part 2: Extension\n",
    "(50 points)\n",
    "\n",
    "Extend this exercise in some aspect. Possible extension include:\n",
    "- Find a confounding example (hip hop song that samples classical music), calculate and classify MFCCs. What do you find are the classification results? Plot the label over time.\n",
    "- Create a new genre classifier by repeating the steps above, but this time use training data and test data from your own audio collection representing two or more different genres. For what genres and audio data styles does the classifier work well, and for which (pairs of) genres does the classifier fail?\n",
    "- Create a new genre classifier by repeating the steps above, but this time use a different machine learning classifier, e.g. random forest, Gaussian mixture model, Naive Bayes, k-nearest neighbor, etc. Adjust the parameters. How well do they perform?\n",
    "- Create a new genre classifier by repeating the steps above, but this time use different features. Consult the [librosa documentation on feature extraction](http://librosa.github.io/librosa/feature.html) for different choices of features. Which features work well? not well?\n",
    "\n",
    "### 2A. Code for Extension\n",
    "\n",
    "Write your code below with comments (25 points): "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2B. Discussion of Results\n",
    "\n",
    "(25 points total)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Describe your goals for the extension (1 paragraph, 10 points):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```REPLACE THIS WITH YOUR DESCRIPTION OF YOUR EXTENSIONS GOALS``` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Describe your results for the extension (1 paragraph, 10 points):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```REPLACE THIS WITH YOUR DESCRIPTION OF RESULTS```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Describe future directions and interesting research questions for this line of inquiry (1 paragraph, 5 points):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```REPLACE THIS WITH YOUR DESCRIPTION OF FUTURE DIRECTIONS```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "- Tzanetakis and Cook ['Musical Genre Classification of Audio Signals'](https://pdfs.semanticscholar.org/4ccb/0d37c69200dc63d1f757eafb36ef4853c178.pdf) from IEEE Transactions on Audio and Speech Processing 2002.- International Society for Music Information Retrieval (ISMIR) [https://ismir.net/](https://ismir.net/)\n",
    "- LibROSA [https://librosa.github.io/librosa/](https://librosa.github.io/librosa/)\n",
    "- SciPy 2015 Talk on Audio / MIR: https://www.youtube.com/watch?v=MhOdbtPhbLU\n",
    "  - [website](https://bmcfee.github.io/) [paper](https://bmcfee.github.io/papers/scipy2015_librosa.pdf)\n",
    "- Music Representation: https://musicinformationretrieval.com/audio_representation.html\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}