guides/dynamic-compute/04 Advanced Computing Concepts.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5440ddaa-38cc-43b9-b4b9-4f2dc3fa4026",
   "metadata": {},
   "source": [
    "# Advanced Computing Concepts with Dynamic Compute\n",
    "__________________\n",
    "In previous tutorials we explored the fundementals of Dynamic Compute. In this notebook we will overview of a subset of more complex examples of the API, such as:\n",
    "* Interacting between Mosaics and ImageStacks\n",
    "* Grouping ImageStacks\n",
    "* More advanced batch-style, time series analysis\n",
    "\n",
    "For a more detailed look at all its classes and their available methods please visit the [`Dynamic Compute`](https://docs.descarteslabs.com/api/dynamic-compute.html) Documentation page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e373f284-2531-447a-8893-7e11f5ceae6b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import descarteslabs as dl\n",
    "import descarteslabs.dynamic_compute as dc\n",
    "from descarteslabs.dynamic_compute import Mosaic, ImageStack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "41de1fad-da78-47a4-8b20-1ffdf067f533",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datetime import date\n",
    "from shapely.wkt import loads\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "098b55dd-0cb0-430c-a292-ecbee235759a",
   "metadata": {},
   "source": [
    "First, we will set up some global variables and set up our input image stack and mosaic objects. Here we will work with a stack of Sentinel-2 imagery corresponding to the date range of May to August, 2022 and a mosaic of 2022's [Cropland Data Layer](https://app.descarteslabs.com/explorer/datasets/usda:cdl:v1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "511c0e03-fa2f-4f59-a2d2-48c9e6528986",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "start_date = \"2022-05-01\"\n",
    "end_date = \"2022-08-01\"\n",
    "pid = \"esa:sentinel-2:l2a:v1\"\n",
    "cdl_pid = \"usda:cdl:v1\"\n",
    "bands = \"nir red green blue\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "700cc06b-abcf-4ba0-9b5b-89bc6a896aea",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if float(dc.__version__[:3]) >= 1.3:\n",
    "    # this is the >=1.3.0 dynamic-compute version\n",
    "    s2_stack = ImageStack.from_product_bands(\n",
    "        pid, bands, start_datetime=start_date, end_datetime=end_date\n",
    "    ).filter(dl.catalog.properties.cloud_fraction < 0.2)\n",
    "else:\n",
    "    # this is the <1.3.0 dynamic-compute version\n",
    "    s2_stack = ImageStack.from_product_bands(\n",
    "        pid, bands, start_datetime=start_date, end_datetime=end_date\n",
    "    ).filter(lambda x: x.cloud_fraction < 0.2)\n",
    "\n",
    "cdl_mosaic = Mosaic.from_product_bands(\n",
    "    cdl_pid, \"class\", start_datetime=\"2021-01-01\", end_datetime=\"2022-01-01\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c49bdf65-4775-4af7-9b0e-ab49a042f1b9",
   "metadata": {},
   "source": [
    "## Interacting between ImageStacks and Mosaics\n",
    "In the next few cells we will calculate NDVI through our time period and then mask to our Cropland Data Layer's Corn class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "25e764cb-0706-44f4-9883-d4a886581774",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "nir, red = s2_stack.unpack_bands(\"nir red\")\n",
    "ndvi = (nir - red) / (nir + red)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "24839465-b9f9-4bb7-a103-8853fab931b6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ndvi_corn_mask = ndvi.mask(cdl_mosaic != 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c2d9b43e-e295-4ba9-9779-e4b10f020001",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "m = dc.map\n",
    "m.center = 41.34232959809853, -95.54491138405865\n",
    "m.zoom = 13"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bf305a5a-8813-4b0b-8b15-29cfa7bcf763",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "(\n",
    "    s2_stack.pick_bands(\"red green blue\")\n",
    "    .median(axis=\"images\")\n",
    "    .visualize(\"Sentinel-2 Composite\", m)\n",
    ")\n",
    "ndvi.mean(axis=\"images\").visualize(\"NDVI Composite\", m)\n",
    "cdl_mosaic.pick_bands(\"class\").visualize(\"CDL\", m, colormap=\"terrain\")\n",
    "ndvi_corn_mask.mean(axis=\"images\").visualize(\n",
    "    \"NDVI Composite Corn Mask\", m, colormap=\"magma\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5dc92dfe-2f94-4657-95fa-f052da7ca9c1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "135ea9b4-4806-4f25-93d2-a9592f8564d6",
   "metadata": {},
   "source": [
    "## Time Series Analysis and Aggregation with ImageStacks\n",
    "In the following cells we will demonstrate how you can utilize an image stack to aggregate time series summary statistics and ndarrays.  There are two options for aggregation:\n",
    "\n",
    "* __axis='pixels'__\n",
    "    * First, we'll call .compute over our stack along __axis='pixels'__ to calculate the mean value _across each AOI for each image._ When calling __axis='pixels'__ we are _aggregating the spatial dimension to a single value_.\n",
    "* __axis='images'__\n",
    "    * Next we will explore calling .compute over our stack along the __axis='images'__ to build simple data composites. When computing over the images axis we will be aggregating all images in our stack into a single-image equivalent."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b445eb7b-d82e-4ce2-84ac-48e8d8a69260",
   "metadata": {},
   "source": [
    "First we will define a tile over which we want to operate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac192f69-5d66-4d5c-9d7f-18ed6edbdf33",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "dltile = dl.geo.DLTile.from_latlon(\n",
    "    41.34232959809853, -95.54491138405865, tilesize=512, pad=0, resolution=10.0\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "969f093a-fc31-4064-b554-6f40031d38e0",
   "metadata": {},
   "source": [
    "Next retrieve the properties of our stack, such as __datetime__ and __image ID__, which we will keep track of for later:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a757f037-95be-4ad9-8cc0-036f46f9874e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ndvi_props = ndvi_corn_mask.properties.compute(dltile)\n",
    "ndvi_dates = [p[\"acquired\"].strftime(\"%Y-%m-%d\") for p in ndvi_props]\n",
    "ndvi_ids = [p[\"id\"] for p in ndvi_props]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22e6165b-3af7-4e79-af49-d61fce8028c1",
   "metadata": {},
   "source": [
    "Next we will explore __axis='pixels'__, where the returned value is a single statistic corresponding to an image's acquired date:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2a94980d-b49d-4970-b483-f54af563e58e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ndvi_mean = ndvi.mean(axis=\"pixels\").compute(dltile)\n",
    "ndvi_mean_list = ndvi_mean.ndarray[:, 0].tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d58ca0fc-0db1-4b9e-bab6-1e00ae1945ec",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(figsize=(10, 5))\n",
    "ax.plot(ndvi_dates, ndvi_mean_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a59f5ae2-aaa4-4176-b65a-339f9f95694d",
   "metadata": {},
   "source": [
    "Comparing __axis='pixels'__ to __axis='images'__, we'll see the resulting ndarray is the same shape as that of a 3D mosaic, in this case each pixel represents the mean value throughout the time period:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d64ceab5-c006-4ae9-bbf6-7c656ea4e0e1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ndvi_mean_arr = ndvi_corn_mask.mean(axis=\"images\").compute(dltile).ndarray\n",
    "dl.utils.display(ndvi_mean_arr, colormap=\"viridis\", figsize=(5, 5))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba01cd77-5a8f-4a3d-b8ca-1c1dc0d5d9c5",
   "metadata": {},
   "source": [
    "## Groupby\n",
    "We can also call [`ImageStack.groupby()`](https://docs.descarteslabs.com/api/dynamic-compute.html#descarteslabs.dynamic_compute.ImageStack.groupby) and aggregate over useful temporal dimensions, as well as any arbitary attribute. \n",
    "\n",
    "Here we will group our stack by date within our time period:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ead24dae-10d4-4a6e-aa71-8d2b37f5bcf8",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ndvi_corn_mask_groupby = ndvi_corn_mask.groupby(\n",
    "    lambda x: date(x.acquired.year, x.acquired.month, x.acquired.day)\n",
    ")\n",
    "type(ndvi_corn_mask_groupby)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c13adae3-b559-4544-b450-ce57e5560aaa",
   "metadata": {},
   "source": [
    "We then compute our grouped object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d72f31e0-360f-414f-a877-93a03f91a5d1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "groupby_results = list(ndvi_corn_mask_groupby.groups.compute(dltile))\n",
    "len(groupby_results)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ab33771-09f0-4211-9ce3-f892206ff5cb",
   "metadata": {},
   "source": [
    "And now we can iterate over each date present in our stack:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7e6268a5-c28d-4ca4-8667-2e2b6c69d153",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots(\n",
    "    nrows=len(groupby_results), figsize=(10, 10 * len(groupby_results))\n",
    ")\n",
    "\n",
    "for i, (date, date_stack) in enumerate(groupby_results):\n",
    "    ax[i].set_title(date.strftime(\"%Y-%m-%d\"))\n",
    "    ndarr = date_stack.median(axis=\"images\").compute(dltile)\n",
    "    ax[i].imshow(ndarr.ndarray[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ebb603a-3200-4e94-ad5f-1f1845c296f8",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}