guides/catalog/02 Creating and Managing Products.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "df9ead35-3ddf-4978-a556-6ecebac7893a",
   "metadata": {},
   "source": [
    "# Managing Products\n",
    "__________________\n",
    "The Descartes Labs Catalog API is a single interface through which you can discover existing raster datasets, search and retrieve their associated images, and manage your own datasets. \n",
    "\n",
    "Here we will outline the basic steps to create a new raster product with associated imagery. This example will also cover the basics of namespaces and organizations for managing access to your newly created product.\n",
    "\n",
    "This guide is meant to serve as an introduction to searching and retrieving raster data. For a more in depth overview of all Catalog classes and their capabilities please visit the [API Reference](https://docs.descarteslabs.com/descarteslabs/catalog/readme.html) and [Catalog Guide](https://docs.descarteslabs.com/guides/catalog.html) sections in our Documentation page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2790d71b-9377-448f-8d05-6aa138e97df7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import descarteslabs as dl\n",
    "from descarteslabs.catalog import (\n",
    "    Image,\n",
    "    DataType,\n",
    "    Product,\n",
    "    Resolution,\n",
    "    ResolutionUnit,\n",
    "    SpectralBand,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c661c1ac-b9ce-4e18-a741-fe49f5149085",
   "metadata": {},
   "source": [
    "## Creating a Product\n",
    "The only required attributes for a product are a *unique* ID and a name. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b631be2e-3f1e-45eb-9956-02471f0f0c2d",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Note on Product IDs\n",
    "We do not always need to append our user hash to the product ID as in the following cell. This notebook is designed for demonstration purposes where other users within your organization may have already run this example. In practice, as long as your product has a unique identifier string within your organization or user namespace you can define it as you'd like."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f29f859f-b93f-4dc9-9503-f5ee54297b59",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "##Setting namespace\n",
    "auth = dl.auth.Auth.get_default_auth()\n",
    "user_hash = auth.namespace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1829e262-a26e-494f-a9e2-65dc03ef7c26",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "pid = f\"sample-rgb-product:{user_hash}\"\n",
    "pid"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6ca0deff-3be9-4a64-bbde-22a894925de8",
   "metadata": {},
   "source": [
    "We can create our new dataset by calling [`Product.get_or_create()`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/product.html#descarteslabs.catalog.Product.get_or_create), giving it a name, [`tags`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/product.html#descarteslabs.catalog.Product.tags), and importantly saving everything through [`Product.save()`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/product.html#descarteslabs.catalog.Product.save):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a56aa3ec-019a-44f2-8a3a-179b6bdba424",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "product = Product.get_or_create(pid)\n",
    "product.name = \"Example RGB Product\"\n",
    "product.tags = [\"examples\"]\n",
    "product.save()\n",
    "product"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf22f30e-1f3e-4912-9b38-cc546ab62172",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### Finding your Product's Namespace\n",
    "\n",
    "Note the product ID in the above output. This prepended string is the new product's _namespace_. You can retrieve that namespace by the [`Auth`](https://docs.descarteslabs.com/descarteslabs/auth/readme.html) API, defaulting to the user's organization:\n",
    "\n",
    "    my-org-id:my-product-id\n",
    "\n",
    "or, if the user doesn't have an organization, their unique user hash:\n",
    "    \n",
    "    my-user-hash:my-product-id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afa69bb7-7442-4d4d-9466-a49d130e5bfd",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "org = auth.payload[\"org\"]\n",
    "org"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "deca51ed-b555-4014-8a85-c87e66471dd0",
   "metadata": {
    "tags": []
   },
   "source": [
    "##### _What if I don't have an organization?_\n",
    "\n",
    "If you do not have an organization, no problem! Your default namespace will simply be your user hash:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bb63e31a-fef8-4eba-9cab-ed2e3a06abd9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "default_product_namespace = org if org else user_hash\n",
    "default_product_namespace"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e640a52a-c521-463a-8212-b99703894072",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Creating bands\n",
    "Before adding any imagery to a product you must [`Band`s](https://docs.descarteslabs.com/descarteslabs/catalog/docs/band.html) that declare the structure of the data shared among all images in a product.\n",
    "\n",
    "We'll start with a [`SpectralBand`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/band.html#descarteslabs.catalog.SpectralBand), which defines data that lie somewhere on the electro-optical spectrum. Here we have a geotiff located under the path *\"data/rgb.tif\"*  which contains 3 spectral bands. We will create one band per channel here, for __red__, __green__, and __blue__, including supported wavelength properties:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d70a3a6-0e30-4d22-9610-03b84a7afb91",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# These values are in nanometers (nm)\n",
    "band_info_dict = {\n",
    "    \"red\": {\"wavelength_nm_min\": 650, \"wavelength_nm_max\": 680},\n",
    "    \"green\": {\"wavelength_nm_min\": 542.5, \"wavelength_nm_max\": 577.5},\n",
    "    \"blue\": {\"wavelength_nm_min\": 457.5, \"wavelength_nm_max\": 522.5},\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "034dcdfb-d5a2-4885-812d-df0439a17e5e",
   "metadata": {},
   "source": [
    "A band defines where its data is found in the files attached to the images associated to the product, uniquely identified by its name and product ID. \n",
    "\n",
    "The full id of the band is composed of the product ID and the name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40448583-5d5a-4638-bef9-c4ef1f76ff00",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "for i, (name, info) in enumerate(band_info_dict.items()):\n",
    "    band = SpectralBand(name=name, product=product)\n",
    "    band.band_index = i\n",
    "    band.file_index = 0\n",
    "    band.data_type = DataType.FLOAT64\n",
    "    band.data_range = (0, 1)\n",
    "    band.display_range = (0, 0.4)\n",
    "    band.resolution = Resolution(unit=ResolutionUnit.METERS, value=30.0)\n",
    "    band.wavelength_nm_min = info[\"wavelength_nm_min\"]\n",
    "    band.wavelength_nm_max = info[\"wavelength_nm_max\"]\n",
    "    band.save()\n",
    "    print(f\"Saved: {band.id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9bcda58-12a8-4ad0-9563-61df856b902d",
   "metadata": {},
   "source": [
    "#### More on Band Creation\n",
    "In the above example:\n",
    "* *band_index = 0* indicates that __red__ is the first band in the source file\n",
    "* *file_index = 0* indicates we expect the first source file passed to contain this band's data \n",
    "* Each band is expected to be represented by 64-bit floating point value ([`DataType.FLOAT64`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/types.html#descarteslabs.catalog.DataType))\n",
    "* *data_range* and *display_range* are the default from [Sentinel-2 L2A](https://app.descarteslabs.com/explorer/datasets/esa:sentinel-2:l2a:v1)\n",
    "* Each band is expected to have a [`Resolution`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/types.html#descarteslabs.catalog.Resolution) of 30 meters\n",
    "\n",
    "Note also that if we were to set *file_index = 1* it would expect a second Image in that position to represent that band. Since all 3 bands are in the same file, we kept that at 0.\n",
    "\n",
    "See the [`Band` Documentation page](https://docs.descarteslabs.com/descarteslabs/catalog/docs/band.html#bands) for more information about properties and other band types not covered in this example."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "587516dc-1472-4771-bb1e-8a25c8c09b18",
   "metadata": {},
   "source": [
    "## Uploading Images - File Paths\n",
    "[`Image.upload()`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/image.html#descarteslabs.catalog.Image.upload) uploads imagery to a product via a list of specified file paths. This returns an [`ImageUpload`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/image.html#descarteslabs.catalog.ImageUpload) object. Images are uploaded and processed asynchronously, so they are not available in the catalog immediately. With [`ImageUpload.wait_for_completion()`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/image.html#descarteslabs.catalog.ImageUpload.wait_for_completion) we wait until the upload is completely finished:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7fde391-1842-42d3-a7a7-146f3403f7e3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set any attributes that should be set on the uploaded images\n",
    "image = Image(product=product, name=\"image1\")\n",
    "# Set acquired date\n",
    "image.acquired = \"2023-06-15\"\n",
    "# Set metadata\n",
    "image.cloud_fraction = 0.1\n",
    "image.extra_properties = {\"foo\": \"bar\"}\n",
    "# Do the upload\n",
    "upload = image.upload(\"data/rgb.tif\")\n",
    "# Wait for completion\n",
    "upload.wait_for_completion()\n",
    "upload.status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac4353fc-4b35-4fff-9479-40314a5f3928",
   "metadata": {},
   "source": [
    "#### _Note: Check your Results!_\n",
    "You can now search and retrieve this imagery in either [Explorer](https://app.descarteslabs.com/explorer) or the Catalog API. We'll retrieve this image's data directly through the [`Image.ndarray()`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/image.html#descarteslabs.catalog.Image.ndarray) method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01d93307-729a-452a-a152-3edab862f0fc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "ndarr = image.ndarray(bands=[\"red\", \"green\", \"blue\"])\n",
    "dl.utils.display(ndarr, size=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a131832-a7a6-4093-9ea3-3feedfe78565",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Uploading Images - Numpy Arrays\n",
    "You can also call [`Image.upload_ndarray()`](https://docs.descarteslabs.com/examples-gallery/plot_create_product.html?#upload-ndarray-to-new-product) if you have your pixel data in-memory and not in a static file. Note you will need to manually specify raster metadata as that is not information contained within a numpy array."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5182f726-e20e-474e-a5b2-846a69385eb8",
   "metadata": {},
   "source": [
    "## Access Control\n",
    "By default only the creator of a product can read and modify it as well as read and modify the bands and imagery in it. To share access to a product with others you can modify its access control lists (ACLs):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4db9c19-4262-42b8-b12c-db3ea69c07cc",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "product.readers = [\"org:pga-tour\"]\n",
    "product.writers = [\"email:jane.doe@descarteslabs.com\", \"email:john.daly@gmail.com\"]\n",
    "product.owners = ['user:XYZ123']\n",
    "product.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9989dbfd-de20-433b-89e6-b1f41c9434c5",
   "metadata": {},
   "source": [
    "In the above example:\n",
    "* All users in the \"pga-tour\" organization can now find and retrieve data from this product\n",
    "* Two specific users identified by email can now update the product and add new imagery"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05452b44-377f-4d6f-b910-73d654c239da",
   "metadata": {},
   "source": [
    "### Transfer ownership\n",
    "Transferring ownership of a product to a new user requires cooperation from both the previous owner and the new owner and is a two-step effort. The first step is for the previous owner to add the new owner to the product:"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea00cd51-2053-40f9-be0f-0052e4f9724f",
   "metadata": {
    "tags": []
   },
   "source": [
    "    product.owners.append(f\"user:{new_owner_id}\")\n",
    "    product.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "803cbb1f-20cc-4da7-aab8-fc2fd2c3c8c4",
   "metadata": {},
   "source": [
    "##### __Note on Transferring Ownership__: \n",
    "You cannot use the \"email:\" variant as an owner. You will have to request the user ID from the new owner and use that instead. You can find your user ID in the profile drop-down on [app.descarteslabs.com](app.descarteslabs.com) or through [`dl.auth.Auth().namespace`](https://docs.descarteslabs.com/descarteslabs/auth/readme.html#descarteslabs.auth.Auth.namespace)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32b72536-817e-4525-acf5-2ce7c295258e",
   "metadata": {},
   "source": [
    "The second step is for the new owner to remove the previous owner:\n",
    "\n",
    "    product.owners.remove(f\"user:{old_owner_id}\")\n",
    "    product.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "017a7b42-b403-4717-abc6-de84216aff1f",
   "metadata": {},
   "source": [
    "\n",
    "For further information on access control please see the [Sharing Resources](https://docs.descarteslabs.com/guides/sharing.html) guide."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd2ab6a4-a73f-4fe2-821c-700bdc8f101c",
   "metadata": {},
   "source": [
    "## Deleting Bands and Products\n",
    "We can delete our Product through the [`Product.delete()`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/product.html#descarteslabs.catalog.Product.delete) method"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d23deb2-1960-4511-a0bd-65f1af16b62f",
   "metadata": {},
   "source": [
    "*__Note on Product Deletion__*: A Product can _only be deleted if it doesn’t have any associated bands or images_. Because the product we created still has images and bands we must first delete those related objects. \n",
    "\n",
    "There is a convenience method [`Product.delete_related_objects()`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/product.html#descarteslabs.catalog.Product.delete_related_objects) to delete all bands and images in a product. Be careful as this may delete a lot of data and can’t be undone!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "311dd372-e111-4367-ad48-29ab5b7617d9",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "status = product.delete_related_objects()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8667b9ab-6feb-4e50-8084-667ddad9501e",
   "metadata": {},
   "source": [
    "This kicks off a job that deletes bands and images in the background. You can wait for this to complete and then delete the product:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "345854be-d926-417c-b953-5ea670ec8c44",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "if status:\n",
    "    status.wait_for_completion()\n",
    "product.delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "536d3fad-10ab-473d-a160-fd4767bcec11",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}