examples/advanced-audio-features.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "DSC160 Data Science and the Arts - Twomey - Spring 2020 - [dsc160.roberttwomey.com](http://dsc160.roberttwomey.com)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Advanced Audio Features\n",
    "\n",
    "This notebook demonstrates advanced audio features including frequency transforms (Fourier Transform, Constant-Q Transform), MFCCs, Spectral Statistics (spectral peak, spectral bandwidth, spectral rolloff), and Chroma and pitch band energy.\n",
    "\n",
    "It depends on the [numpy](https://numpy.org/), [matplotlib](https://matplotlib.org/), [seaborn](https://seaborn.pydata.org/), and [LibROSA](https://librosa.github.io/librosa/) libraries. \n",
    "\n",
    "The examples are adapted from the tutorials at [musicinformationretrieval.com](musicinformationretrieval.com), developed for the Stanford MIR workshops."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Basic imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "# visualization\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn\n",
    "\n",
    "# sound processing\n",
    "import librosa\n",
    "import librosa.display\n",
    "\n",
    "# to play audio inline in ipython/jupyter notebooks\n",
    "from IPython.display import Audio\n",
    "\n",
    "import numpy as np\n",
    "import scipy\n",
    "import sklearn\n",
    "\n",
    "import os, requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Fourier Transform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [Fourier Transform](https://en.wikipedia.org/wiki/Fourier_transform) is one of the most fundamental operations in applied mathematics and signal processing.\n",
    "\n",
    "It transforms our time-domain signal into the *frequency domain*. Whereas the time domain expresses our signal as a sequence of samples, the frequency domain expresses our signal as a *superposition of sinusoids* of varying magnitudes, frequencies, and phase offsets.\n",
    "\n",
    "Before we compute a FFT, let's load an audio file to work with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, sr = librosa.load(\"audio/c_strum.wav\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(x.shape)\n",
    "print(sr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "Audio(x, rate=sr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To compute a Fourier transform in NumPy or SciPy, we use [`scipy.fft`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.fft.fft.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X = scipy.fft(x) # scipy.fft.fft(x) \n",
    "X_mag = np.absolute(X)\n",
    "f = np.linspace(0, sr, len(X_mag)) # frequency variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "f.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "max(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the spectrum:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(13, 5))\n",
    "plt.plot(f, X_mag) # magnitude spectrum\n",
    "plt.xlabel('Frequency (Hz)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: the spectrum is symmetrical around sr/2. According to sampling frequency, the max frequency that can be captured by a digital signal with sampling rate of sr is sr/2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Zooming in, let's inspect the frequency bands at the lower end of the spectrum:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(13, 5))\n",
    "plt.plot(f[:5000], X_mag[:5000])\n",
    "plt.xlabel('Frequency (Hz)')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: this sample has six large peaks, likely corresponding to the six strings of the guitar sounding the chord.\n",
    "\n",
    "What is the value of the largest peak (leftmost) in Hz?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# index of the maximum value in time series\n",
    "max_pos = X_mag.argmax()\n",
    "\n",
    "# frequency at the same index\n",
    "f[max_pos]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is this frequency as a musical note?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "librosa.hz_to_note(f[max_pos])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "EXERCISE: Find the indices of the six largest peaks, and compute their corresponding frequencies and musical notes. Do those notes correspond to the expected components of a guitar C chord strum?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "See the [frequency-transforms.ipynb](frequency-transforms.ipynb) notebook for a more complete example of different feature transfer types."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mel Frequency Ceptral Coefficients"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In sound processing, the mel-frequency cepstrum (MFC) is a representation of the short-term power spectrum of a sound, based on a linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency.\n",
    "\n",
    "The [mel frequency cepstral coefficients](https://en.wikipedia.org/wiki/Mel-frequency_cepstrum) (MFCCs) of a signal are a small set of features (usually about 10-20) which concisely describe the overall shape of a spectral envelope (i.e. how the spectral envelope is evolving). In MIR, it is often used to describe timbre.\n",
    "\n",
    "They are derived from a type of cepstral representation of the audio clip (a nonlinear \"spectrum-of-a-spectrum\"). The difference between the cepstrum and the mel-frequency cepstrum is that in the MFC, the frequency bands are equally spaced on the mel scale, which approximates the human auditory system's response more closely than the linearly-spaced frequency bands used in the normal cepstrum.\n",
    "\n",
    "Let's load, display, and play an audio file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "# load\n",
    "x, fs = librosa.load('audio/simple_loop.wav')\n",
    "\n",
    "# display\n",
    "plt.figure(figsize=(16, 5))\n",
    "librosa.display.waveplot(x, sr=fs)\n",
    "plt.show()\n",
    "\n",
    "# play\n",
    "Audio(x, rate=fs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To copmute MFCCs across an audio signal, we can use [`librosa.feature.mfcc`](http://bmcfee.github.io/librosa/generated/librosa.feature.mfcc.html#librosa.feature.mfcc):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfccs = librosa.feature.mfcc(x, sr=fs)\n",
    "print(mfccs.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, `mfcc` computed 20 MFCCs over 130 frames.\n",
    "\n",
    "The very first MFCC, the 0th coefficient, does not convey information relevant to the overall shape of the spectrum. It only conveys a constant offset, i.e. adding a constant value to the entire spectrum. Therefore, many practitioners will discard the first MFCC when performing classification. For now, we will use the MFCCs as is.\n",
    "\n",
    "Let's display the MFCCs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16, 5))\n",
    "librosa.display.specshow(mfccs, sr=fs, x_axis='time')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's scale the MFCCs such that each coefficient dimension has zero mean and unit variance (using [`sklearn.preprocessing.scale`](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.scale.html):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mfccs = sklearn.preprocessing.scale(mfccs, axis=1)\n",
    "print(\"mean:\", mfccs.mean(axis=1))\n",
    "print(\"var:\", mfccs.var(axis=1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the scaled MFCCs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(16, 5))\n",
    "librosa.display.specshow(mfccs, sr=fs, x_axis='time')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: We can use these scaled MFCCs as direct features input to classifiers and clustering algorithms."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Spectral Features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For classification, we're going to be using new features in our arsenal: spectral moments (centroid, bandwidth, skewness, kurtosis) and other spectral statistics.\n",
    "\n",
    "[Moments](https://en.wikipedia.org/wiki/Moment_(mathematics)) is a term used in physics and statistics. There are raw moments and central moments. You are probably already familiar with two examples of moments: mean and variance. The first raw moment is known as the mean. The second central moment is known as the variance. \n",
    "\n",
    "These Spectral features are calculated on magnitudes of the frequency components of signals."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spectral Centroid"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load an audio file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x, sr = librosa.load('audio/simple_loop.wav')\n",
    "Audio(x, rate=sr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The **spectral centroid** ([Wikipedia](https://en.wikipedia.org/wiki/Spectral_centroid)) indicates at which frequency the energy of a spectrum is centered upon. This is like a weighted mean:\n",
    "\n",
    "$$ f_c = \\frac{\\sum_k S(k) f(k)}{\\sum_k S(k)} $$\n",
    "\n",
    "where $S(k)$ is the spectral magnitude at frequency bin $k$, $f(k)$ is the frequency at bin $k$.\n",
    "\n",
    "[`librosa.feature.spectral_centroid`](https://librosa.github.io/librosa/generated/librosa.feature.spectral_centroid.html#librosa.feature.spectral_centroid) computes the spectral centroid for each frame in a signal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spectral_centroids = librosa.feature.spectral_centroid(x, sr=sr)[0]\n",
    "spectral_centroids.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the time variable for visualization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = range(len(spectral_centroids))\n",
    "t = librosa.frames_to_time(frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define a helper function to normalize the spectral centroid for visualization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def normalize(x, axis=0):\n",
    "    return sklearn.preprocessing.minmax_scale(x, axis=axis)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plot the spectral centroid along with the waveform:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(14, 5))\n",
    "librosa.display.waveplot(x, sr=sr, alpha=0.4)\n",
    "plt.plot(t, normalize(spectral_centroids), color='r') # normalize for visualization purposes\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar to the zero crossing rate, there is a spurious rise in spectral centroid at the beginning of the signal. That is because the silence at the beginning has such small amplitude that high frequency components have a chance to dominate. One hack around this is to add a small constant before computing the spectral centroid, thus shifting the centroid toward zero at quiet portions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(14, 5))\n",
    "spectral_centroids = librosa.feature.spectral_centroid(x+0.01, sr=sr)[0]\n",
    "librosa.display.waveplot(x, sr=sr, alpha=0.4)\n",
    "plt.plot(t, normalize(spectral_centroids), color='r') # normalize for visualization purposes\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"mean spectral centroid: {:0.2f} Hz\".format(np.mean(spectral_centroids)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spectral Bandwidth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[`librosa.feature.spectral_bandwidth`](https://librosa.github.io/librosa/generated/librosa.feature.spectral_bandwidth.html#librosa.feature.spectral_bandwidth) computes the order-$p$ spectral bandwidth:\n",
    "\n",
    "$$ \\left( \\sum_k S(k) \\left(f(k) - f_c \\right)^p \\right)^{\\frac{1}{p}} $$\n",
    "\n",
    "where $S(k)$ is the spectral magnitude at frequency bin $k$, $f(k)$ is the frequency at bin $k$, and $f_c$ is the spectral centroid. When $p = 2$, this is like a weighted standard deviation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spectral_bandwidth_2 = librosa.feature.spectral_bandwidth(x+0.01, sr=sr)[0]\n",
    "spectral_bandwidth_3 = librosa.feature.spectral_bandwidth(x+0.01, sr=sr, p=3)[0]\n",
    "spectral_bandwidth_4 = librosa.feature.spectral_bandwidth(x+0.01, sr=sr, p=4)[0]\n",
    "\n",
    "plt.figure(figsize=(14, 5))\n",
    "librosa.display.waveplot(x, sr=sr, alpha=0.4)\n",
    "plt.plot(t, normalize(spectral_bandwidth_2), color='r')\n",
    "plt.plot(t, normalize(spectral_bandwidth_3), color='g')\n",
    "plt.plot(t, normalize(spectral_bandwidth_4), color='y')\n",
    "plt.legend(('p = 2', 'p = 3', 'p = 4'))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"mean spectral bandwidth: {:0.2f} Hz\".format(np.mean(spectral_bandwidth_2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spectral Rolloff"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Spectral rolloff** is the frequency below which a specified percentage of the total spectral energy, e.g. 85%, lies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[`librosa.feature.spectral_rolloff`](https://librosa.github.io/librosa/generated/librosa.feature.spectral_rolloff.html#librosa.feature.spectral_rolloff) computes the rolloff frequency for each frame in a signal:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spectral_rolloff = librosa.feature.spectral_rolloff(x+0.01, sr=sr)[0]\n",
    "\n",
    "plt.figure(figsize=(14, 5))\n",
    "librosa.display.waveplot(x, sr=sr, alpha=0.4)\n",
    "plt.plot(t, normalize(spectral_rolloff), color='r')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"mean spectral roloff: {:0.2f} Hz\".format(np.mean(spectral_rolloff)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spectral Contrast"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Spectral contrast** considers the spectral peak, the spectral valley, and their difference in each frequency subband. For more information:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [Jiang, Dan-Ning, Lie Lu, Hong-Jiang Zhang, Jian-Hua Tao, and Lian-Hong Cai. “Music type classification by spectral contrast feature.” In Multimedia and Expo, 2002. ICME‘02. Proceedings. 2002 IEEE International Conference on, vol. 1, pp. 113-116. IEEE, 2002.](http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.583.7201&rep=rep1&type=pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[`librosa.feature.spectral_contrast`](https://librosa.github.io/librosa/generated/librosa.feature.spectral_contrast.html) computes the spectral contrast for six subbands for each time frame:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "spectral_contrast = librosa.feature.spectral_contrast(x, sr=sr)\n",
    "spectral_contrast.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(14, 5))\n",
    "plt.imshow(normalize(spectral_contrast, axis=1), aspect='auto', origin='lower', cmap='coolwarm')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Constant-Q Transform"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike the Fourier transform, but similar to the mel scale, the [constant-Q transform](http://en.wikipedia.org/wiki/Constant_Q_transform) uses a logarithmically spaced frequency axis. For more information, read the original paper:\n",
    "\n",
    "- [Judith C. Brown, \"Calculation of a constant Q spectral transform,\" J. Acoust. Soc. Am., 89(1):425–434, 1991.](http://academics.wellesley.edu/Physics/brown/pubs/cq1stPaper.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "The constant in constant-Q is the ratio between frequency and resolution (i.e. the width of each frequency band changes as the frequency changes). \n",
    "\n",
    "Let's load a file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "x, sr = librosa.load('audio/simple_piano.wav')\n",
    "Audio(x, rate=sr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "notes"
    }
   },
   "source": [
    "To compute a constant-Q spectrogram, will use [`librosa.cqt`](https://librosa.github.io/librosa/generated/librosa.core.cqt.html#librosa.core.cqt):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fmin = librosa.midi_to_hz(36)\n",
    "hop_length = 512\n",
    "C = librosa.cqt(x, sr=sr, fmin=fmin, n_bins=72, hop_length=hop_length)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "logC = librosa.amplitude_to_db(np.abs(C))\n",
    "plt.figure(figsize=(15, 5))\n",
    "librosa.display.specshow(logC, sr=sr, x_axis='time', y_axis='cqt_note', fmin=fmin, cmap='coolwarm')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: each frequency bin corresponds to one MIDI pitch number."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logC.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print the magnitudes of each key in the Constant-Q spectrum:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "frame_width = (1.0 / sr * hop_length)\n",
    "frame_width"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pos = int(2.0 / frame_width)\n",
    "pos"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "print values at time frame `pos` in logC:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "logC[:,pos]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "plot frequency magnitudes at time frame `pos` as bar plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize=(15, 5))\n",
    "plt.bar(x=range(0,72), height=logC[:,pos])\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "NOTE: the scale of magnitudes goes from -55 db to 10 db. Read [more about decibels here](https://en.wikipedia.org/wiki/Decibel).\n",
    "\n",
    "Computing peaks at any point in the Constant-Q time series would reveal dominant notes as pitch peaks."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chroma"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A [chroma vector](https://en.wikipedia.org/wiki/Chroma_feature) is a typically a 12-element feature vector indicating how much energy of each pitch class {musical notes such as C, C#, D, D#, E, ..., B}, is present in the signal. A chromagram displays the chroma pitch magnitudes in the frequency spectrum of an audio signal over time.\n",
    "\n",
    "Librosa has a function to generate a chromagram: [`librosa.feature.chroma_stft`](https://librosa.github.io/librosa/generated/librosa.feature.chroma_stft.html#librosa.feature.chroma_stft):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chromagram = librosa.feature.chroma_stft(x, sr=sr, hop_length=hop_length)\n",
    "plt.figure(figsize=(15, 5))\n",
    "librosa.display.specshow(chromagram, x_axis='time', y_axis='chroma', hop_length=hop_length, cmap='coolwarm')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Librosa can also generate a chromagram using the Constant-Q frequency transform: [`librosa.feature.chroma_cqt`](https://librosa.github.io/librosa/generated/librosa.feature.chroma_cqt.html#librosa.feature.chroma_cqt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chromagram = librosa.feature.chroma_cqt(x, sr=sr, hop_length=hop_length)\n",
    "plt.figure(figsize=(15, 5))\n",
    "librosa.display.specshow(chromagram, x_axis='time', y_axis='chroma', hop_length=hop_length, cmap='coolwarm')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Chroma energy normalized statistics (CENS): The main idea of CENS features is that taking statistics over large windows smooths local deviations in tempo, articulation, and musical ornaments such as trills and arpeggiated chords. CENS are best used for tasks such as audio matching and similarity.\n",
    "\n",
    "[`librosa.feature.chroma_cens`](https://librosa.github.io/librosa/generated/librosa.feature.chroma_cens.html#librosa.feature.chroma_cens)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "chromagram = librosa.feature.chroma_cens(x, sr=sr, hop_length=hop_length)\n",
    "plt.figure(figsize=(15, 5))\n",
    "librosa.display.specshow(chromagram, x_axis='time', y_axis='chroma', hop_length=hop_length, cmap='coolwarm')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "- International Society for Music Information Retrieval (ISMIR) [https://ismir.net/](https://ismir.net/)\n",
    "- Laboratory for the Recognition and Organization of Speech and Audio at Columbia University: [LabROSA](https://labrosa.ee.columbia.edu/)\n",
    "  - LibROSA [https://librosa.github.io/librosa/](https://librosa.github.io/librosa/)\n",
    "- Brian McFee - SciPy 2015 Talk on Audio Processing and MIR with LibROSA: https://www.youtube.com/watch?v=MhOdbtPhbLU\n",
    "  - [website](https://bmcfee.github.io/) [paper](https://bmcfee.github.io/papers/scipy2015_librosa.pdf)\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}