ray-demo.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import multiprocessing as mp\n",
    "import time\n",
    "\n",
    "all_procs = []\n",
    "\n",
    "def background(command):\n",
    "    proc = mp.Process(target=os.system, args=(command,))\n",
    "    all_procs.append(proc)\n",
    "    proc.start()\n",
    "    \n",
    "def kill_all():\n",
    "    for proc in all_procs:\n",
    "        proc.terminate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sharded Dataset\n",
    "\n",
    "For the subsequent code, we assume that the Imagenet shards are stored in ./shards.\n",
    "\n",
    "If the shards do not exist, we generate them directly from the original Imagenet data using a small script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "test -f shards/imagenet-train-000000.tar || {\n",
    "    python3 ./convert-imagenet.py ./imagenet-data ./shards\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Server\n",
    "\n",
    "Since we are implementing distributed training, we need to be able to retrieve shards over the network.\n",
    "\n",
    "Here, we use a small web server to serve the shards; the web server is simply nginx running in a Docker container.\n",
    "\n",
    "In practice, you would use some kind of permanently installed web server, or even better the AIStore object store."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "docker ps | awk '$2==\"nginx\"{print $1}' | xargs docker kill"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "imagenetdir=/media/tmb/data1/gs/nvdata-imagenet\n",
    "docker run -it --rm -d -p 8080:80 --name web -v $imagenetdir:/usr/share/nginx/html nginx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "!curl http://$(hostname -i):8080/imagenet-train-000000.tar | tar tvf - | tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ray Cluster\n",
    "\n",
    "In this example, we use Ray for starting up distributed training. Ray is a distributed processing system for Python. We start a two node Ray cluster. Of course, you can start up as many nodes as you like, with as many GPUs as you like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    ". ./venv/bin/activate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "ray stop > /dev/null 2>&1 || true"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "ray start --head"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "ssh sedna \"cd $(/bin/pwd) && . ./venv/bin/activate && ray start --address=$(hostname -i):6379\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "ray status"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training Jobs\n",
    "\n",
    "Since Ray manages the cluster and handles the distributed computing aspect, starting up a multi-GPU distributed job is particularly simple and can be done with just a single command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "vscode": {
     "languageId": "shellscript"
    }
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "python3 imagenet.py raytrain --verbose --bucket=http://$(hostname -i):8080/ --backend=gloo --mname=resnet18"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.6 ('venv': venv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "e4bcf1c9b208164f37ac7931ad36e4bb0817a209136db5032682a51fb7927afc"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}