demos/02 Supervised Classification/02c Deploying a Supervised Classifier.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4a8ea6e8-2a48-4050-b90e-3785a0983381",
   "metadata": {},
   "source": [
    "## Supervised ML on Descartes Labs Platform: Deploying a Random Forest Classifier\n",
    "__________________\n",
    "This example will demonstrate a typical example of how to deploy and save the results of a simple supervised classifier model.\n",
    "\n",
    "The general steps covered in this notebook are:\n",
    "* Create a new [`Product`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/product.html#descarteslabs.catalog.Product) and associated [`Classified Band`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/band.html#descarteslabs.catalog.ClassBand)\n",
    "* Split up the training AOI into tiles\n",
    "* Define an asynchronous[`Function`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function) which iterates over each tile and:\n",
    "    * Retrieves the saved ML model from [02b Training a Supervised Classifier.ipynb](02b%20Training%20a%20Supervised%20Classifier.ipynb)\n",
    "    * Searches and rasters NAIP imagery as an ndarray\n",
    "    * Runs inference on retrieved imagery\n",
    "    * Stores predictions as a new [`Image`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/image.html#descarteslabs.catalog.Image)\n",
    "\n",
    "\n",
    "Optionally move on to [02d Interactive Deployment with Dynamic Compute.ipynb](02d%20Interactive%20Deployment%20with%20Dynamic%20Compute.ipynb) where we will use [`Dynamic Compute`](https://docs.descarteslabs.com/api/dynamic-compute.html) to visualize results and interactively deploy this model to new AOIs.\n",
    "\n",
    "**_Note:_** In order to run this example you must first complete the steps outlined in [02a Generate Training Data.ipynb](02a%20Generate%20Training%20Data.ipynb) and [02b Training a Supervised Classifier.ipynb](02b%20Training%20a%20Supervised%20Classifier.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "03ebd6ef-dd2d-4876-b77c-772d262a4408",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import datetime\n",
    "import sklearn\n",
    "import sys\n",
    "import uuid\n",
    "import yaml\n",
    "\n",
    "import descarteslabs as dl\n",
    "import descarteslabs.compute\n",
    "import descarteslabs.vector as dl_vector\n",
    "import geopandas as gpd\n",
    "import matplotlib.pyplot as plt\n",
    "import shapely.geometry as sgeom"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c37f5d34-0c46-43f7-bb6b-bbf941433c67",
   "metadata": {},
   "source": [
    "Defining global variables for reference throughout this example, including the NAIP product ID, a list of bands, a start and end date, resolution, and a name for our function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a04cae47-b8d6-415c-bef9-5d8bb4108523",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "with open(\"config.yaml\", \"r\") as file:\n",
    "    config = yaml.load(file, yaml.FullLoader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a337fc11-d455-4630-a73c-d64366a65ca0",
   "metadata": {},
   "outputs": [],
   "source": [
    "user_hash = dl.auth.Auth().namespace\n",
    "org = dl.auth.Auth().payload['org']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7803e0ed-3fe5-4178-8d5b-75959c22ecab",
   "metadata": {},
   "outputs": [],
   "source": [
    "major = sys.version_info.major\n",
    "minor = sys.version_info.minor\n",
    "compute_image = f\"python{major}.{minor}:latest\"\n",
    "compute_image"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9246e80c-33a1-40f9-b4f4-8e1fa3370cb2",
   "metadata": {},
   "source": [
    "As well as retrieve our training features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7662be1f-f90a-4e7c-a9fd-65fb99a56ea6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "table = dl_vector.Table.get(config[\"training_table_name\"])\n",
    "gdf = table.collect()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f672910-c8ef-42d0-b418-4bd32e320474",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Creating an Output Product\n",
    "Next we create a new product by first setting a unique ID:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c637ee6-90d3-4911-b17a-e220ebfbbab4",
   "metadata": {},
   "source": [
    "#### Note on Product ID Creation\n",
    "Since this is an example, where multiple users at the same organization may re-run this notebook, we intend to delete and overwrite the output product upon every iteration. Because of this we must reconstruct the product's _namespace_. In the following cell we will append either the current user's _organization_ or _user hash_ to the passed product ID:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "026adc26-6d5e-4151-b53d-7504ccfee7b5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result_product_id = f\"{org or user_hash}:rfc-results-{user_hash}\"\n",
    "result_product_id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63449f58-a582-4516-b418-611b8b43ff66",
   "metadata": {},
   "source": [
    "#### Note on Product Creation\n",
    "We do not always need to delete and overwrite our product on every iteration as in the following cell. This notebook is designed for demonstration purposes, where we do not care about preserving each prior product.\n",
    "\n",
    "In practice, as long as your product has a **unique** ID you may ignore the next cell and skip to the following."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ea911cb6-462b-4be4-90c2-66fb4d115705",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result_product = dl.catalog.Product.get(result_product_id)\n",
    "\n",
    "if result_product:\n",
    "    print(\"Product already exists, deleting old iteration\")\n",
    "    status = result_product.delete_related_objects()\n",
    "    if status:\n",
    "        status.wait_for_completion()\n",
    "    print(\"Deleted related objects\")\n",
    "    result_product.delete()\n",
    "    print(\"Deleted Product\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7004c8ca-4010-4562-9829-ed3798f1390b",
   "metadata": {},
   "outputs": [],
   "source": [
    "result_product = dl.catalog.Product.get_or_create(result_product_id)\n",
    "result_product.name = \"Testing RFC Outputs\"\n",
    "result_product.tags = [\"examples\"]\n",
    "result_product.readers = []\n",
    "result_product.save()\n",
    "result_product"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c12b8c24-111b-4e31-9d7a-bad8f234a709",
   "metadata": {},
   "source": [
    "Next we create a single band, which includes:\n",
    "* Unique ID\n",
    "* Data type\n",
    "* Valid data range\n",
    "* Display range\n",
    "* No data value\n",
    "* Resolution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "db4ff9ce-2744-4973-966c-3ee74363cbbe",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "preds_max = int(gdf.category_int.max())\n",
    "preds_min = int(gdf.category_int.min())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e16b0231-4cda-4dff-a9ec-fce1701f0c83",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Creating a band\n",
    "band = dl.catalog.ClassBand.get_or_create(\n",
    "    id=f\"{result_product.id}:class\",\n",
    "    band_index=0,\n",
    "    data_type=dl.catalog.DataType.BYTE,\n",
    "    data_range=[preds_min, preds_max],\n",
    "    display_range=[preds_min, preds_max],\n",
    "    nodata=preds_max + 1,\n",
    "    colormap_name=\"terrain\",\n",
    "    resolution=dl.catalog.Resolution(\n",
    "        value=config[\"resolution_m\"], unit=dl.catalog.ResolutionUnit.METERS\n",
    "    ),\n",
    ")\n",
    "band.save()\n",
    "band"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b05ffce7-85fc-4175-82e4-0210acfe813a",
   "metadata": {},
   "source": [
    "## Tiling Input AOI\n",
    "Here we will once again split the extent of our input features into tiles:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "772d7aa0-7358-4475-8d7b-7e93434484f5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gdf_geom = sgeom.box(*gdf[\"geometry\"].total_bounds)\n",
    "dltiles = dl.geo.DLTile.from_shape(\n",
    "    gdf_geom, resolution=config[\"resolution_m\"], tilesize=2048, pad=0\n",
    ")\n",
    "len(dltiles)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d486f69-6651-43a4-b131-76d3f38aeb7e",
   "metadata": {},
   "source": [
    "This time there is no need to filter out tiles that don't intersect features!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "18aaddae-e079-4eaa-9308-25ad7de2503b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(5, 5))\n",
    "dltile_gdf = gpd.GeoDataFrame(\n",
    "    {\n",
    "        \"geometry\": [dltile.geometry for dltile in dltiles],\n",
    "    },\n",
    "    crs=4326,\n",
    ")\n",
    "dltile_gdf.plot(ax=ax, facecolor=\"none\", edgecolor=\"grey\", linewidth=0.5)\n",
    "gdf.plot(ax=ax, column=\"category\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d407a85e-5f14-4c1f-b83c-4b54609ba601",
   "metadata": {},
   "source": [
    "## Defining the Batch Compute Function\n",
    "Now we can define a self contained python function which writes our outputs to our new product. The inputs here are:\n",
    "\n",
    "* A DLTile key\n",
    "* Output Product ID\n",
    "\n",
    "The overall steps are as follows:\n",
    "\n",
    "* Re-create a tile from the passed key\n",
    "* Retrieve our trained classifier as a blob\n",
    "* Search NAIP using Catalog over our tile\n",
    "* Mosaic the returned imagery as a numpy array\n",
    "* Run predict from our retrieved model\n",
    "* Create and upload our predictions to the output product"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd878ca0-4f6b-4277-87b1-08e3b4d0b03a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def write_rfc_to_catalog(dltile_key, product_id):\n",
    "\n",
    "    import datetime\n",
    "    import os\n",
    "    import pickle\n",
    "\n",
    "    import descarteslabs as dl\n",
    "    import numpy as np\n",
    "\n",
    "    user_id = dl.auth.Auth().namespace\n",
    "    blob_name = f\"training_rfc_model-{user_id}\"\n",
    "\n",
    "    print(f\"Processing: {dltile_key}\")\n",
    "    # Getting DLTile, finding scenes\n",
    "    dltile = dl.geo.DLTile.from_key(dltile_key)\n",
    "\n",
    "    blob = dl.catalog.Blob.get(name=blob_name)\n",
    "    blob.download(\"rfc.pickle\")\n",
    "    print(\"Retrieved Classifier\")\n",
    "\n",
    "    clf = pickle.load(open(f\"rfc.pickle\", \"rb\"))\n",
    "\n",
    "    pid = \"usda:naip:v1\"\n",
    "    bands = [\"nir\", \"red\", \"green\"]\n",
    "    start = \"2020-01-01\"\n",
    "    end = \"2021-01-01\"\n",
    "\n",
    "    print(\"Searching Images...\")\n",
    "    ###NAIP over this DLTile\n",
    "    naip_ic = (\n",
    "        dl.catalog.Product.get(pid)\n",
    "        .images()\n",
    "        .intersects(dltile)\n",
    "        .filter(start <= dl.catalog.properties.acquired < end)\n",
    "        .sort(\"acquired\")\n",
    "        .limit(None)\n",
    "    ).collect()\n",
    "    print(naip_ic)\n",
    "\n",
    "    naip_arr, raster_info = naip_ic.mosaic(bands=bands, bands_axis=-1, raster_info=True)\n",
    "    print(\"Rastered imagery\")\n",
    "    # Reshaping for sklearn:\n",
    "    nx, ny, nsamples = naip_arr.shape\n",
    "    in_ras_arr = naip_arr.reshape(-1, nsamples)\n",
    "\n",
    "    preds = clf.predict(in_ras_arr).reshape(nx, ny)\n",
    "    print(\"Predictions Complete\")\n",
    "    # Getting the Product--NOTE that this is hard coded in\n",
    "    out_product = dl.catalog.Product.get_or_create(product_id)\n",
    "\n",
    "    # Creating an image - note the required unique id corresponding to the DLTile\n",
    "    image = dl.catalog.Image(\n",
    "        product=out_product,\n",
    "        id=\"{}:class_{}\".format(product_id, dltile_key.replace(\":\", \"_\")),\n",
    "    )\n",
    "    print(f\"Writing to: {image.id}\")\n",
    "\n",
    "    image.acquired = datetime.datetime.now().isoformat()\n",
    "\n",
    "    upload = image.upload_ndarray(\n",
    "        ndarray=preds.astype(\"uint8\"),\n",
    "        raster_meta=raster_info,\n",
    "        overviews=[2, 4, 8, 16, 32, 64],\n",
    "        overview_resampler=dl.catalog.OverviewResampler.NEAREST,\n",
    "        overwrite=True,\n",
    "    )\n",
    "    upload.wait_for_completion()\n",
    "    print(\"Complete\")\n",
    "    # Cleaning up\n",
    "    os.remove(\"rfc.pickle\")\n",
    "    return image.id"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f833bd38-11a3-479e-889f-299542d58dd0",
   "metadata": {},
   "source": [
    "Now we format our input arguments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4dea560a-9c9a-4845-9d6d-2ab873525010",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "args = [(dltile.key, result_product.id) for dltile in dltiles]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "04293903-3bd3-43f1-841d-2418464ca6fd",
   "metadata": {},
   "source": [
    "And we can test our function locally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba2fd2cf-3160-4460-ac04-d3c65551ba32",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "img_id = write_rfc_to_catalog(*args[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "14fb2b1b-2536-4cdb-9d2a-f24790e5fed5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "img = dl.catalog.Image.get(img_id)\n",
    "ndarr = img.ndarray(\"class\")\n",
    "plt.imshow(ndarr[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4484ba6-6cae-4f2e-b179-30cbb6ae13ec",
   "metadata": {},
   "source": [
    "Once we are happy with the performance of our function we can save it to our Compute service.\n",
    "\n",
    "Note here that we must pass the specific scikit-learn _version_ as a requirement:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e4786b1-2274-4e87-8c82-5b7e9479b9d9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "async_func = dl.compute.Function(\n",
    "    write_rfc_to_catalog,\n",
    "    name=config[\"pred_func_name\"],\n",
    "    image=compute_image,\n",
    "    cpus=1,\n",
    "    memory=2,\n",
    "    timeout=300,\n",
    "    maximum_concurrency=20,\n",
    "    retry_count=1,\n",
    "    requirements=[f\"scikit-learn=={sklearn.__version__}\"],\n",
    ")\n",
    "\n",
    "async_func.save()\n",
    "print(f\"Saved {async_func.id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44ebbe15-2ed7-4473-896e-9b4517b72dd5",
   "metadata": {},
   "source": [
    "**_Take note of your Function ID!_**\n",
    "\n",
    "And finally map args to our Function to return a set of jobs:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae582f14-757a-4f04-9349-e08153e39f3b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "jobs = async_func.map(args)\n",
    "len(jobs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "288fea5a-4490-4e65-9128-22be022b5c8e",
   "metadata": {},
   "source": [
    "Navigate to [app.descarteslabs.com/compute](https://app.descarteslabs.com/compute) to track your progress! Or wait programmatically via:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e554240f-b21e-47c5-9058-325575d78be5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# async_func.wait_for_completion()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf571e86-6fb6-47d0-bbd6-d84690efcf3e",
   "metadata": {},
   "source": [
    "Once this function completes, you can navigate to [Explorer](https://app.descarteslabs.com/explorer) to view your results or move on to [02d Interactive Deployment with Dynamic Compute.ipynb](02d%20Interactive%20Deployment%20with%20Dynamic%20Compute.ipynb) where we will use [`Dynamic Compute`](https://docs.descarteslabs.com/api/dynamic-compute.html) to visualize our new product and interactively deploy this model to new AOIs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "819205d9-e829-44b5-8222-407e3a16e672",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}