guides/batch-compute/01 Hello World.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "ea5784c3-c69c-4c7c-b431-49d7508b176c",
   "metadata": {},
   "source": [
    "# Compute Functions and Jobs\n",
    "__________________\n",
    "The Compute module provides scalable compute capabilities to parallelize your analysis. Compute enables users to package and execute your Python code within nodes hosted on Descartes Lab's cloud infrastructure. These nodes offer the ability to access imagery at extremely high rates of throughput to execute computations over nearly any spatio-temporal scale. \n",
    "\n",
    "This example provides a light introduction to the basics of asynchronous computing with Descartes Labs. For a more detailed look at all its classes and their available methods please visit the [`Compute`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html) documentation page."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1fd838d-971c-44be-a4cd-865e842320bf",
   "metadata": {},
   "source": [
    "We'll start by importing the two primary objects within the API, as well as their associated status objects:\n",
    "\n",
    " * [`Function:`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function) dynamically created, serverless functions containing user specified, compiled code that you can submit many jobs to.\n",
    " * [`Job:`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Job) submitted request for a single invocation of a created function. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ff1ce274-85e8-4aa0-87b1-c7ed27681648",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from descarteslabs.auth import Auth\n",
    "from descarteslabs.catalog import Blob, properties as p, StorageType\n",
    "from descarteslabs.compute import Function, FunctionStatus, Job, JobStatus"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a48ca67b",
   "metadata": {},
   "source": [
    "We'll also need these imports for this example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a054f4ad-2c1c-4359-a8cf-58dda7011092",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "827ddcd7-7ed1-4572-b9a2-9071006c9a09",
   "metadata": {},
   "source": [
    "## Hello World\n",
    "Next, we'll create a very basic *hello_world* function which returns a string constructed from the passed argument:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90509f65-19fb-4032-9698-dac8ade9583b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def hello(arg):\n",
    "    print(f\"Hello, {arg}\")\n",
    "    return f\"hello {arg}\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05d45cbd-086c-4c77-8959-0a636bf1eaa5",
   "metadata": {},
   "source": [
    "### Creating an Asynchronous Compute Function\n",
    "To create the object, we simply need to call [`Function()`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function) and specify the desired parameters. \n",
    "\n",
    "The recommended minimum parameters are your Python function, a name, and the image that will be used to build the Function environment.  Some common attributes used to customize the performance of your scalable compute object include: \n",
    " * __image__ = Reference to the base Compute image to use, which must match your local Python version\n",
    " * __cpus__ = number of CPUs requested for a single job\n",
    " * __memory__ = max memory available for each job\n",
    " * __maximum_concurrency__ = max number of jobs to run in parallel\n",
    " * __timeout__ = max length a job can run in seconds\n",
    " * __retry_count__ = max number of times a job can be retried\n",
    " * __requirements__ = list of Python dependencies required by this function\n",
    "\n",
    "Once we have defined the object, we simply call [`Function.save()`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function.save) to complete the creation. \n",
    "\n",
    "\n",
    "*__Note__*: For all available **cpus** and **memory** options please visit the [Compute limits](https://docs.descarteslabs.com/guides/quota.html#general-limitations) page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9d5f0f1-a524-4b22-8850-07d7c60f6dd4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "major = sys.version_info.major\n",
    "minor = sys.version_info.minor\n",
    "image = f\"python{major}.{minor}:latest\"\n",
    "image"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3962281-bbce-4408-870a-e8a0fe4424d7",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Creating function\")\n",
    "async_func = Function(\n",
    "    hello,\n",
    "    name=\"my-compute-hello\",\n",
    "    image=image,\n",
    "    cpus=0.25,\n",
    "    memory=512,\n",
    "    maximum_concurrency=20,\n",
    "    timeout=300,\n",
    "    retry_count=0,\n",
    ")\n",
    "async_func.save()\n",
    "print(f\"Created: {async_func.id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2943774-2a1a-41ea-8a27-03a0c70bc0d2",
   "metadata": {},
   "source": [
    "*__Note__*: This function will take just a few minutes to build!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8f080cc-c554-4f49-bb81-4ae422c68dd0",
   "metadata": {},
   "source": [
    "## Submitting Jobs\n",
    "Now that we have a function, we can test it by creating and submitting a job. There are several ways you can submit jobs to a function:\n",
    " * `async_func(args)` - Pass arguments directly to the Function\n",
    " * [`async_func.map()`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function.map) - Submit multiple jobs efficiently (discussed in more detail within the next tutorial [02 Create Imagery.ipynb](./02%20Create%20Imagery.ipynb))\n",
    " \n",
    "Below we will invoke our asynchronous function by passing the string *\"Hello from my function!\"*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39f2012c-b0f2-407e-afa1-bf463a32ecc4",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# invoke the function\n",
    "print(\"Submitting a job\")\n",
    "job = async_func(\"Hello from my function!\")\n",
    "print(f\"Submitted: {job.id}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b70ed895-84be-4535-a9e9-ad2ea48d6a69",
   "metadata": {},
   "source": [
    "## Waiting for Completion\n",
    "\n",
    "Now that we have submitted a job, there are a few ways to wait for completion. It is highly recommended to navigate to the Compute monitor app at [app.descarteslabs.com/compute](https://app.descarteslabs.com/compute) anyways to track your progress. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c01b04bc-a20d-475d-b5cd-bb5f20d03371",
   "metadata": {},
   "source": [
    "1. Wait on an individual job via [`Job.wait_for_completion()`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Job.wait_for_completion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dcd9666f-ed11-4ac9-b8fa-47c851500eb1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "job.wait_for_completion()\n",
    "job.status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d31e75c-e506-44dc-a4b3-10ff640a6999",
   "metadata": {},
   "source": [
    "2. Wait for all pending and running jobs for the function to complete using [`Function.wait_for_completion()`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function.wait_for_completion)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81f85369-f78a-4597-b92c-dfbd0ba4e5a5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "async_func.wait_for_completion()\n",
    "async_func.status"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "531df871-c39f-4e25-abf8-851e9518b733",
   "metadata": {},
   "source": [
    "3. Navigate to the Compute monitor app at [app.descarteslabs.com/compute](https://app.descarteslabs.com/compute)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4269806e-6e07-4715-9321-cf159a8ca195",
   "metadata": {},
   "source": [
    "### Note - to continue this tutorial you must wait for your Function to complete!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "671110b4-03e8-44b5-97df-6242c94fdb0e",
   "metadata": {},
   "source": [
    "## Logging and Results - Functions\n",
    "When a function build is completed you can view the build logs by calling [`Function.build_log()`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function.build_log):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99f2b88c-6cbe-47ee-876b-354dc8c12d9f",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "async_func.build_log()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "660587b0-7560-4de6-ae14-284960e9a2d3",
   "metadata": {},
   "source": [
    "## Logging and Results - Jobs\n",
    "When a job is completed you can access both the logs and results by [`Job.log()`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Job.log) and [`Job.result()`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Job.result) respectively:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43d6fcc0-6087-4ccd-84a0-32d1cafc1f51",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "job.log()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af452a21-ed79-4152-bd34-9a482919611d",
   "metadata": {},
   "source": [
    "#### Note on Job Logs \n",
    "These are where you can debug your code if it is failing!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "546ea01b-f451-4350-a963-d91f8155132a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "job.result()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f65f19b-e22f-4857-86ee-f6081db41f63",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Logging and Results - Retrieving Results in Bulk\n",
    "\n",
    "In addition to accessing your build logs and results as shown above, you can also\n",
    "access these as [`Blob`](https://docs.descarteslabs.com/descarteslabs/catalog/docs/blob.html#descarteslabs.catalog.Blob)s, as all these artifacts are automatically stored in the Catalog. \n",
    "\n",
    "*__Note__*: Logs are retained only for a period of 30 days. Results are retained indefinitely, even after the Function\n",
    "and its Jobs have been deleted."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f8e90d3-27b3-4b1a-bb54-a6ce0727e4b6",
   "metadata": {},
   "source": [
    "In order to access these artifacts, you will need to form the correct IDs to retrieve them. All three require\n",
    "the correct [`StorageType`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Job) value (__logs__ for logs and __compute__ for results), namespace (available as\n",
    "[`Function.namespace`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function.namespace) while your function exists, or can be derived from your authentication details as shown below),\n",
    "and the name of the blob derived from your __Function.id__ and __Job.id__.\n",
    "\n",
    "\n",
    "__Function Build Log id:__\n",
    "\n",
    "    logs/{your-default-namespace}/{function-id}\n",
    "\n",
    "    \n",
    "__Job Execution Log id:__\n",
    "\n",
    "    logs/{your-default-namespace}/{function-id}/{job-id}\n",
    "\n",
    "    \n",
    "__Job Result id:__\n",
    "        \n",
    "    compute/{your-default-namespace}/{function-id}/{job-id}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74d02f84-fdce-4701-b3ee-c5f7dba2fe64",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "auth = Auth.get_default_auth()\n",
    "namespace = (auth.payload.get(\"org\", \"\") + \":\" + auth.namespace).lstrip(\":\")\n",
    "print(f\"Namespace: {namespace}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "325304a5-999c-4f1d-9436-70f55c878a5a",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"Results for {async_func.id}\")\n",
    "for b in (\n",
    "    Blob.search()\n",
    "    .filter(p.namespace == namespace)\n",
    "    .filter(p.name.startswith(f\"{async_func.id}/\"))\n",
    "    .filter(p.storage_type == StorageType.COMPUTE)\n",
    "):\n",
    "    print(f\"ID: {b.id}\")\n",
    "    print(b.data())\n",
    "    print(\"\\n\")\n",
    "\n",
    "print(f\"Job Execution Logs for {async_func.id}\")\n",
    "for b in (\n",
    "    Blob.search()\n",
    "    .filter(p.namespace == namespace)\n",
    "    .filter(p.name.startswith(f\"{async_func.id}/\"))\n",
    "    .filter(p.storage_type == StorageType.LOGS)\n",
    "):\n",
    "    print(f\"ID: {b.id}\")\n",
    "    print(b.data())\n",
    "    print(\"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "da55efe3-7d4d-41b5-9d08-181824f80062",
   "metadata": {},
   "source": [
    "_Note that the build log content is compressed!_"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fee121d2-045f-4410-9a3f-c4146a4bf990",
   "metadata": {},
   "source": [
    "## Searching Functions\n",
    "You can also search, filter, and sort your previously created functions. Here we will find all of our functions created today:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "234a7ecb-e4b4-4009-b3dd-01881f34c6a6",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "today = datetime.today().strftime(\"%Y-%m-%d\")\n",
    "\n",
    "for func in Function.search().filter(Function.creation_date > today):\n",
    "    print(func.id)\n",
    "    print(func.creation_date)\n",
    "    print(func.status)\n",
    "    print(len(list(func.jobs)))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5f57e69-7cce-402a-89cf-f7817b471c13",
   "metadata": {},
   "source": [
    "We can get active functions by specifying the **ready** status"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "824ad205-fe36-49d9-a488-5ffc0f9ba496",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "active_funcs = [\n",
    "    {\"name\": f.name, \"id\": f.id}\n",
    "    for f in Function.search().filter(Function.status == FunctionStatus.READY)\n",
    "]\n",
    "active_funcs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98ef21f2-f93f-4939-acf8-4ee5c95fe268",
   "metadata": {},
   "source": [
    "Here we can filter by prefix:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1302459-5cbe-4a31-943c-c323169789aa",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "my_funcs = [f.name for f in Function.search().filter(Function.name.startswith(\"my\"))]\n",
    "my_funcs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23d770ea-a648-4ccc-831e-f446ac3e92c4",
   "metadata": {},
   "source": [
    "## Deleting Functions\n",
    "\n",
    "In order to release all resources associated with a function you should delete it when you are done by calling [`Function.delete()`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function.delete).\n",
    "\n",
    "*__Notes on Deletion__*: \n",
    "*  Any jobs must have been completed before you can delete the function. When deleted, all associated jobs, build logs, and job logs will be deleted. \n",
    "* Results will not be deleted and will remain available unless you call [`Function.delete_jobs(delete_results=True)`](https://docs.descarteslabs.com/descarteslabs/compute/readme.html#descarteslabs.compute.Function.delete_jobs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e7c882a9-8b79-4e17-ad15-81ce8e560080",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "async_func.delete_jobs(delete_results=True)\n",
    "async_func.delete()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "37b05ddb-9800-46d7-ba78-82fe5986a41a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}