notebooks/softvc-voice-conversion/softvc-voice-conversion.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c6eed66854c398c3",
   "metadata": {},
   "source": [
    "# SoftVC VITS Singing Voice Conversion and OpenVINO™\n",
    "This tutorial is based on [SoftVC VITS Singing Voice Conversion project](https://github.com/svc-develop-team/so-vits-svc). The purpose of this project was to enable developers to have their beloved anime characters perform singing tasks. The developers' intention was to focus solely on fictional characters and avoid any involvement of real individuals, anything related to real individuals deviates from the developer's original intention.\n",
    "\n",
    "The singing voice conversion model uses SoftVC content encoder to extract speech features from the source audio. These feature vectors are directly fed into [VITS](https://github.com/jaywalnut310/vits) without the need for conversion to a text-based intermediate representation. As a result, the pitch and intonations of the original audio are preserved.\n",
    "\n",
    "In this tutorial we will use the base model flow.\n",
    "\n",
    "\n",
    "#### Table of contents:\n",
    "\n",
    "- [Prerequisites](#Prerequisites)\n",
    "- [Use the original model to run an inference](#Use-the-original-model-to-run-an-inference)\n",
    "- [Convert to OpenVINO IR model](#Convert-to-OpenVINO-IR-model)\n",
    "- [Run the OpenVINO model](#Run-the-OpenVINO-model)\n",
    "- [Interactive inference](#Interactive-inference)\n",
    "\n",
    "\n",
    "### Installation Instructions\n",
    "\n",
    "This is a self-contained example that relies solely on its own code.\n",
    "\n",
    "We recommend  running the notebook in a virtual environment. You only need a Jupyter server to start.\n",
    "For details, please refer to [Installation Guide](https://github.com/openvinotoolkit/openvino_notebooks/blob/latest/README.md#-installation-guide).\n",
    "\n",
    "<img referrerpolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=5b5a4db0-7875-4bfb-bdbd-01698b5b1a77&file=notebooks/softvc-voice-conversion/softvc-voice-conversion.ipynb\" />\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a921abbc-41a4-4ae4-a20c-7541f0a69a4b",
   "metadata": {},
   "source": [
    "## Prerequisites\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1aaa6bb335f4efe",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%pip install -qU \"pip<24.1\"  # to fix fairseq install problem\n",
    "%pip install -q \"openvino>=2023.2.0\"\n",
    "!git clone https://github.com/svc-develop-team/so-vits-svc -b 4.1-Stable\n",
    "%pip install -q --extra-index-url https://download.pytorch.org/whl/cpu  tqdm librosa \"torch>=2.1.0\" \"torchaudio>=2.1.0\" faiss-cpu \"gradio>=4.19\" \"numpy>=1.23.5\" \"fairseq==0.12.2\" praat-parselmouth"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "19df02f3-a4e7-4063-8b8c-b05a68a63c6e",
   "metadata": {},
   "source": [
    "Download pretrained models and configs. We use a recommended encoder [ContentVec](https://arxiv.org/abs/2204.09224) and models from [a collection of so-vits-svc-4.0 models made by the Pony Preservation Project](https://huggingface.co/therealvul/so-vits-svc-4.0) for example. You can choose any other pretrained model from this or another project or [prepare your own](https://github.com/svc-develop-team/so-vits-svc#%EF%B8%8F-training)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5ca855b365c4e0e7",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Fetch `notebook_utils` module\n",
    "import requests\n",
    "\n",
    "r = requests.get(\n",
    "    url=\"https://raw.githubusercontent.com/openvinotoolkit/openvino_notebooks/latest/utils/notebook_utils.py\",\n",
    ")\n",
    "\n",
    "open(\"notebook_utils.py\", \"w\").write(r.text)\n",
    "from notebook_utils import download_file\n",
    "\n",
    "# ContentVec\n",
    "download_file(\n",
    "    \"https://huggingface.co/lj1995/VoiceConversionWebUI/resolve/main/hubert_base.pt\",\n",
    "    \"checkpoint_best_legacy_500.pt\",\n",
    "    directory=\"so-vits-svc/pretrain/\",\n",
    ")\n",
    "\n",
    "# pretrained models and configs from a collection of so-vits-svc-4.0 models. You can use other models.\n",
    "download_file(\n",
    "    \"https://huggingface.co/therealvul/so-vits-svc-4.0/resolve/main/Rainbow%20Dash%20(singing)/kmeans_10000.pt\",\n",
    "    \"kmeans_10000.pt\",\n",
    "    directory=\"so-vits-svc/logs/44k/\",\n",
    ")\n",
    "download_file(\n",
    "    \"https://huggingface.co/therealvul/so-vits-svc-4.0/resolve/main/Rainbow%20Dash%20(singing)/config.json\",\n",
    "    \"config.json\",\n",
    "    directory=\"so-vits-svc/configs/\",\n",
    ")\n",
    "download_file(\n",
    "    \"https://huggingface.co/therealvul/so-vits-svc-4.0/resolve/main/Rainbow%20Dash%20(singing)/G_30400.pth\",\n",
    "    \"G_30400.pth\",\n",
    "    directory=\"so-vits-svc/logs/44k/\",\n",
    ")\n",
    "download_file(\n",
    "    \"https://huggingface.co/therealvul/so-vits-svc-4.0/resolve/main/Rainbow%20Dash%20(singing)/D_30400.pth\",\n",
    "    \"D_30400.pth\",\n",
    "    directory=\"so-vits-svc/logs/44k/\",\n",
    ")\n",
    "\n",
    "# a wav sample\n",
    "download_file(\n",
    "    \"https://huggingface.co/datasets/santifiorino/spinetta/resolve/main/spinetta/000.wav\",\n",
    "    \"000.wav\",\n",
    "    directory=\"so-vits-svc/raw/\",\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6f53cf28ef37988f",
   "metadata": {},
   "source": [
    "## Use the original model to run an inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ea7218ff-e99c-4b05-ac54-99b097aab23f",
   "metadata": {},
   "source": [
    "Change directory to `so-vits-svc` in purpose not to brake internal relative paths."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "324125b043342928",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-11T16:23:39.582765900Z",
     "start_time": "2023-10-11T16:23:39.441185100Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%cd so-vits-svc"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "531f9121-2452-4dca-9da1-0e0272f54e0b",
   "metadata": {},
   "source": [
    "Define the Sovits Model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cba0ed03ac900ce",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-11T16:24:11.594409400Z",
     "start_time": "2023-10-11T16:23:48.236884800Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from inference.infer_tool import Svc\n",
    "\n",
    "model = Svc(\"logs/44k/G_30400.pth\", \"configs/config.json\", device=\"cpu\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "3e38baeb-14bd-4dbf-86de-36a7d69b0026",
   "metadata": {},
   "source": [
    "Define `kwargs` and make an inference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "364d72774d2863d0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-11T16:24:34.084985700Z",
     "start_time": "2023-10-11T16:24:22.823731700Z"
    },
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "kwargs = {\n",
    "    \"raw_audio_path\": \"raw/000.wav\",  # path to a source audio\n",
    "    \"spk\": \"Rainbow Dash (singing)\",  # speaker ID in which the source audio should be converted.\n",
    "    \"tran\": 0,\n",
    "    \"slice_db\": -40,\n",
    "    \"cluster_infer_ratio\": 0,\n",
    "    \"auto_predict_f0\": False,\n",
    "    \"noice_scale\": 0.4,\n",
    "}\n",
    "\n",
    "audio = model.slice_inference(**kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6d6882b1-1762-4909-ba58-66314fd24108",
   "metadata": {},
   "source": [
    "And let compare the original audio with the result."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a941d393fb0d9f5b",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import IPython.display as ipd\n",
    "\n",
    "# original\n",
    "ipd.Audio(\"raw/000.wav\", rate=model.target_sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8fe21dc8-794c-48b0-a9a1-1ca2280d65d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "# result\n",
    "ipd.Audio(audio, rate=model.target_sample)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ae244d48f7982d4c",
   "metadata": {},
   "source": [
    "## Convert to OpenVINO IR model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Model components are PyTorch modules, that can be converted with `ov.convert_model` function directly. We also use `ov.save_model` function to serialize the result of conversion.\n",
    "`Svc` is not a model, it runs model inference inside. In base scenario only `SynthesizerTrn` named `net_g_ms` is used. It is enough to convert only this model and we should re-assign `forward` method on `infer` method for this purpose.\n",
    "\n",
    "`SynthesizerTrn` uses several models inside it's flow, i.e. `TextEncoder`, `Generator`, `ResidualCouplingBlock`, etc., but in our case OpenVINO allows to convert whole pipeline by one step without need to look inside."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "259b86a26d06f881",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import openvino as ov\n",
    "import torch\n",
    "from pathlib import Path\n",
    "\n",
    "\n",
    "dummy_c = torch.randn(1, 256, 813)\n",
    "dummy_f0 = torch.randn(1, 813)\n",
    "dummy_uv = torch.ones(1, 813)\n",
    "dummy_g = torch.tensor([[0]])\n",
    "model.net_g_ms.forward = model.net_g_ms.infer\n",
    "\n",
    "net_g_kwargs = {\n",
    "    \"c\": dummy_c,\n",
    "    \"f0\": dummy_f0,\n",
    "    \"uv\": dummy_uv,\n",
    "    \"g\": dummy_g,\n",
    "    \"noice_scale\": torch.tensor(0.35),  # need to wrap numeric and boolean values for conversion\n",
    "    \"seed\": torch.tensor(52468),\n",
    "    \"predict_f0\": torch.tensor(False),\n",
    "    \"vol\": torch.tensor(0),\n",
    "}\n",
    "core = ov.Core()\n",
    "\n",
    "\n",
    "net_g_model_xml_path = Path(\"models/ov_net_g_model.xml\")\n",
    "\n",
    "if not net_g_model_xml_path.exists():\n",
    "    converted_model = ov.convert_model(model.net_g_ms, example_input=net_g_kwargs)\n",
    "    net_g_model_xml_path.parent.mkdir(parents=True, exist_ok=True)\n",
    "    ov.save_model(converted_model, net_g_model_xml_path)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e211edea-0515-4070-8d66-d5b3bfb398dc",
   "metadata": {},
   "source": [
    "## Run the OpenVINO model\n",
    "[back to top ⬆️](#Table-of-contents:)\n",
    "\n",
    "Select a device from dropdown list for running inference using OpenVINO."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b907034e797533dc",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import ipywidgets as widgets\n",
    "import openvino as ov\n",
    "\n",
    "core = ov.Core()\n",
    "\n",
    "device = widgets.Dropdown(\n",
    "    options=core.available_devices + [\"AUTO\"],\n",
    "    value=\"AUTO\",\n",
    "    description=\"Device:\",\n",
    "    disabled=False,\n",
    ")\n",
    "\n",
    "device"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a49a84e3-6ae2-4336-a77b-b5818ae1d985",
   "metadata": {},
   "source": [
    "We should create a wrapper for `net_g_ms` model to keep it's interface. Then replace `net_g_ms` original model by the converted IR model. We use `ov.compile_model` to make it ready to use for loading on a device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fb890ffe86cd0a84",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class NetGModelWrapper:\n",
    "    def __init__(self, net_g_model_xml_path):\n",
    "        super().__init__()\n",
    "        self.net_g_model = core.compile_model(net_g_model_xml_path, device.value)\n",
    "\n",
    "    def infer(self, c, *, f0, uv, g, noice_scale=0.35, seed=52468, predict_f0=False, vol=None):\n",
    "        if vol is None:  # None is not allowed as an input\n",
    "            results = self.net_g_model((c, f0, uv, g, noice_scale, seed, predict_f0))\n",
    "        else:\n",
    "            results = self.net_g_model((c, f0, uv, g, noice_scale, seed, predict_f0, vol))\n",
    "\n",
    "        return torch.from_numpy(results[0]), torch.from_numpy(results[1])\n",
    "\n",
    "\n",
    "model.net_g_ms = NetGModelWrapper(net_g_model_xml_path)\n",
    "audio = model.slice_inference(**kwargs)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "72924b58-e565-4855-8b1a-97be9144b943",
   "metadata": {},
   "source": [
    "Check result. Is it identical to that created by the original model. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bea8d8b65b7fb95d",
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import IPython.display as ipd\n",
    "\n",
    "ipd.Audio(audio, rate=model.target_sample)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "69937fb6-3da5-4e1e-b4e4-304ee5e6a421",
   "metadata": {},
   "source": [
    "## Interactive inference\n",
    "[back to top ⬆️](#Table-of-contents:)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52af2039-b06f-490e-aa7c-b8588a420fdc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import gradio as gr\n",
    "\n",
    "\n",
    "src_audio = gr.Audio(label=\"Source Audio\", type=\"filepath\")\n",
    "output_audio = gr.Audio(label=\"Output Audio\", type=\"numpy\")\n",
    "\n",
    "title = \"SoftVC VITS Singing Voice Conversion with Gradio\"\n",
    "description = f'Gradio Demo for SoftVC VITS Singing Voice Conversion and OpenVINO™. Upload a source audio, then click the \"Submit\" button to inference. Audio sample rate should be {model.target_sample}'\n",
    "\n",
    "\n",
    "def infer(src_audio, tran, slice_db, noice_scale):\n",
    "    kwargs[\"raw_audio_path\"] = src_audio\n",
    "    kwargs[\"tran\"] = tran\n",
    "    kwargs[\"slice_db\"] = slice_db\n",
    "    kwargs[\"noice_scale\"] = noice_scale\n",
    "\n",
    "    audio = model.slice_inference(**kwargs)\n",
    "\n",
    "    return model.target_sample, audio\n",
    "\n",
    "\n",
    "demo = gr.Interface(\n",
    "    infer,\n",
    "    [\n",
    "        src_audio,\n",
    "        gr.Slider(-100, 100, value=0, label=\"Pitch shift\", step=1),\n",
    "        gr.Slider(\n",
    "            -80,\n",
    "            -20,\n",
    "            value=-30,\n",
    "            label=\"Slice db\",\n",
    "            step=10,\n",
    "            info=\"The default is -30, noisy audio can be -30, dry sound can be -50 to preserve breathing.\",\n",
    "        ),\n",
    "        gr.Slider(\n",
    "            0,\n",
    "            1,\n",
    "            value=0.4,\n",
    "            label=\"Noise scale\",\n",
    "            step=0.1,\n",
    "            info=\"Noise level will affect pronunciation and sound quality, which is more metaphysical\",\n",
    "        ),\n",
    "    ],\n",
    "    output_audio,\n",
    "    title=title,\n",
    "    description=description,\n",
    "    examples=[[\"raw/000.wav\", 0, -30, 0.4, False]],\n",
    ")\n",
    "\n",
    "try:\n",
    "    demo.queue().launch(debug=True)\n",
    "except Exception:\n",
    "    demo.queue().launch(share=True, debug=True)\n",
    "# if you are launching remotely, specify server_name and server_port\n",
    "# demo.launch(server_name='your server name', server_port='server port in int')\n",
    "# Read more in the docs: https://gradio.app/docs/"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  },
  "openvino_notebooks": {
   "imageUrl": "https://github.com/openvinotoolkit/openvino_notebooks/assets/29454499/bb43bea1-6709-48e5-a928-b86061399849",
   "tags": {
    "categories": [
     "Model Demos",
     "AI Trends"
    ],
    "libraries": [],
    "other": [],
    "tasks": [
     "Audio-to-Audio",
     "Voice Conversion"
    ]
   }
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "state": {},
    "version_major": 2,
    "version_minor": 0
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}