From d22af969d0c73a463bd29b055cb2184bf1244962 Mon Sep 17 00:00:00 2001 From: EOEPCA CI Date: Fri, 1 Mar 2024 14:59:34 +0000 Subject: [PATCH] Deployed 26a20c4 to current with MkDocs 1.5.3 and mike 2.0.0 --- current/index.html | 1 + current/search/search_index.json | 2 +- current/sitemap.xml | 58 +++++++++++++++---------------- current/sitemap.xml.gz | Bin 490 -> 489 bytes versions.json | 35 ++++++++++++++++++- 5 files changed, 65 insertions(+), 31 deletions(-) diff --git a/current/index.html b/current/index.html index b5bf86bc..91c06352 100644 --- a/current/index.html +++ b/current/index.html @@ -1285,6 +1285,7 @@

Deployment GuideEOEPCA Reference Implementation, by providing for each version…

diff --git a/current/search/search_index.json b/current/search/search_index.json index ec95f029..57864568 100644 --- a/current/search/search_index.json +++ b/current/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Deployment Guide","text":"

Changelog

This current version of the Deployment Guide represents the development tip that goes beyond the latest release verion v1.4.

The following provides a summary of changes since the last release (v1.4)\u2026

  • [FIX] Update Application Hub to chart version 2.0.57 to fix hard-coded ingress namespace

The Deployment Guide captures each release of the EOEPCA Reference Implementation, by providing for each version\u2026

  • Description of how each building-block is configured and deployed - see Deploy EOEPCA Components
  • Scripted deployment in which each building-block can be selectively deployed to form a system - see Getting Started

A full system deployment is described, in which components are deployed with complementary configurations that facilitate their integration as a coherent system. Nevertheless, each component can be cherry-picked from this system deployment for individual re-use.

The deployment is organised into the following sections:

  • Getting Started A quickstart guide with associated scripts to facilitate example deployments, which preempt the descriptions that follow later in the document. Scripts are provided in a variety of \u2018profiles\u2019 that deploy different combinations of building-blocks for different notional use cases.
  • Prepare Cluster Establish the Kubernetes cluster and other prerequisites for the deployment of the EOEPCA system.
  • Deploy EOEPCA Components Deployment of the EOEPCA components.
"},{"location":"cluster/cluster-prerequisites/","title":"Cluster Prerequisites","text":""},{"location":"cluster/cluster-prerequisites/#overview","title":"Overview","text":"

The following prerequisite components are assumed to be deployed in the cluster.

Note

The Scripted Deployment automatically deploys most of the components list here - in particular\u2026

  • Nginx Ingress Controller
  • Cert Manager
  • Letsencrypt ClusterIssuers
  • Minio Object Storage

The Sealed Secrets controller is not deployed - but can be added following the instructions below.

"},{"location":"cluster/cluster-prerequisites/#nginx-ingress-controller","title":"Nginx Ingress Controller","text":"
# Install the Nginx Ingress Controller helm chart\nhelm upgrade -i --version='<4.5.0' \\\n  --repo https://kubernetes.github.io/ingress-nginx \\\n  ingress-nginx ingress-nginx \\\n  --wait\n

Note

For Kubernetes version 1.22 and earlier the version of the Nginx Ingress Controller must be before v4.5.0.

To target the Nginx Ingress Controller the kubernetes.io/ingress.class: nginx annotation must be applied to the Ingress resource\u2026 

apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  annotations:\n    kubernetes.io/ingress.class: nginx\n    ...\n

"},{"location":"cluster/cluster-prerequisites/#cert-manager","title":"Cert Manager","text":"
# Install the Cert Manager helm chart\nhelm upgrade -i --namespace cert-manager --create-namespace \\\n  --repo https://charts.jetstack.io \\\n  --set installCRDs=true \\\n  cert-manager cert-manager\n
"},{"location":"cluster/cluster-prerequisites/#letsencrypt-certificates","title":"Letsencrypt Certificates","text":"

Once the Certificate Manager is deployed, then we can establish ClusterIssuer operators in the cluster to support use of TLS with service Ingress endpoints.

For Letsencrypt we can define two ClusterIssuer - for production and for staging.

NOTE that these require the cluster to be publicly accessible, in order for the http01 acme flow to verify the domain ownership. Local development deployments will typically not have public IP/DNS - in which case the system deployment can proceed, but without TLS support for the service endpoints.

"},{"location":"cluster/cluster-prerequisites/#production","title":"Production","text":"
apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n  name: letsencrypt-production\nspec:\n  acme:\n    # You must replace this email address with your own.\n    # Let's Encrypt will use this to contact you about expiring\n    # certificates, and issues related to your account.\n    email: eoepca.systemteam@telespazio.com\n    server: https://acme-v02.api.letsencrypt.org/directory\n    privateKeySecretRef:\n      # Secret resource that will be used to store the account's private key.\n      name: letsencrypt-production-account-key\n    # Add a single challenge solver, HTTP01 using nginx\n    solvers:\n      - http01:\n          ingress:\n            class: nginx\n
"},{"location":"cluster/cluster-prerequisites/#staging","title":"Staging","text":"
apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n  name: letsencrypt-staging\nspec:\n  acme:\n    # You must replace this email address with your own.\n    # Let's Encrypt will use this to contact you about expiring\n    # certificates, and issues related to your account.\n    email: eoepca.systemteam@telespazio.com\n    server: https://acme-staging-v02.api.letsencrypt.org/directory\n    privateKeySecretRef:\n      # Secret resource that will be used to store the account's private key.\n      name: letsencrypt-staging-account-key\n    # Add a single challenge solver, HTTP01 using nginx\n    solvers:\n      - http01:\n          ingress:\n            class: nginx\n

To exploit the specified ClusterIssuer the cert-manager.io/cluster-issuer annotation must be applied to the Ingress resource. For example\u2026 

apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  annotations:\n    kubernetes.io/ingress.class: nginx\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    ...\n

"},{"location":"cluster/cluster-prerequisites/#sealed-secrets","title":"Sealed Secrets","text":"

The EOEPCA development team maintain their deployment configurations in GitHub - for declarative, reproducible cluster deployments.

Various Secret are relied upon by the system services. Secrets should not be exposed by commit to GitHub.

Instead SealedSecret are committed to GitHub, which are encrypted, and can only be decrypted by the sealed-secret-controller that runs within the cluster. The sealed-secret-controller decrypts the SealedSecret to a regular Secret (of the same name) that can then be consumed by the cluster components.

The sealed-secret-controller is deployed to the cluster using the helm chart\u2026

helm install --version 2.1.8 --create-namespace --namespace infra \\\n  --repo https://bitnami-labs.github.io/sealed-secrets \\\n  eoepca-sealed-secrets sealed-secrets\n

Once the controller is deployed within the cluster, then the kubeseal command can be used to create a SealedSecret from a regular Secret, as follows\u2026

Create example Secret\u2026 

kubectl -n test create secret generic mysecret \\\n  --from-literal=password=changeme \\\n  --dry-run=client -o yaml \\\n  > mysecret.yaml\n

Create SealedSecret from Secret using kubeseal\u2026 

kubeseal -o yaml \\\n  --controller-name eoepca-sealed-secrets \\\n  --controller-namespace infra \\\n  < mysecret.yaml \\\n  > mysecret-sealed.yaml\n

"},{"location":"cluster/cluster-prerequisites/#references","title":"References","text":"
  • Sealed Secrets on GitHub
  • kubeseal Release
"},{"location":"cluster/cluster-prerequisites/#minio-object-storage","title":"MinIO Object Storage","text":"

Various building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service.

As a workaround, in the absence of an existing object storage, it is possible to use MinIO to establish an object storage service within the Kubernetes cluster. We use the minio helm chart provided by the MinIO Project.

# Install the minio helm chart\nhelm upgrade -i -f minio-values.yaml --namespace rm --create-namespace \\\n  --repo https://charts.min.io/ \\\n  minio minio \\\n  --wait\n

Note

The Kubernetes namespace rm is used above as an example, and can be changed according to your deployment preference.

The minio deployment is customised via the values file minio-values.yaml, for example\u2026

existingSecret: minio-auth\nreplicas: 2\n\ningress:\n  enabled: true\n  ingressClassName: nginx\n  annotations:\n    cert-manager.io/cluster-issuer: \"letsencrypt\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n    nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n  path: /\n  hosts:\n    - minio.192-168-49-2.nip.io\n  tls:\n    - secretName: minio-tls\n      hosts:\n        - minio.192-168-49-2.nip.io\n\nconsoleIngress:\n  enabled: true\n  ingressClassName: nginx\n  annotations:\n    cert-manager.io/cluster-issuer: \"letsencrypt\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n    nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n  path: /\n  hosts:\n    - console.minio.192-168-49-2.nip.io\n  tls:\n  - secretName: minio-console-tls\n    hosts:\n      - console.minio.192-168-49-2.nip.io\n\nresources:\n  requests:\n    memory: 1Gi\n\npersistence:\n  storageClass: standard\n\nbuckets:\n  - name: eoepca\n  - name: cache-bucket\n

Note

  • The example values assuming a TLS configuration using letsencrypt certificate provider
  • The admin credentials are provided by the Kubernetes secret named minio-auth - see below
  • The annotation nginx.ingress.kubernetes.io/proxy-body-size was found to be required to allow transfer of large files (such as data products) through the nginx proxy
"},{"location":"cluster/cluster-prerequisites/#minio-credentials-secret","title":"Minio Credentials Secret","text":"

The Minio admin credentials are provided via a Kubernetes secret that is referenced from the Minio helm chart deployment values. For example\u2026

kubectl -n rm create secret generic minio-auth \\\n  --from-literal=rootUser=\"eoepca\" \\\n  --from-literal=rootPassword=\"changeme\"\n

Note

The secret must be created in the same Kubernetes namespace as the Minio service deployment - e.g. rm namespce in the example above.

"},{"location":"cluster/cluster-prerequisites/#s3cmd-configuration","title":"s3cmd Configuration","text":"

The s3cmd can be configured for access to the MinIO deployment. The --configure option can be used to prepare a suitable configuration file for s3cmd\u2026

s3cmd -c mys3cfg --configure\n

In response to the prompts, the following configuration selections are applicable to the above settings\u2026

Access Key: eoepca\nSecret Key: changeme\nDefault Region: us-east-1\nS3 Endpoint: minio.192-168-49-2.nip.io\nDNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io\nEncryption password: \nPath to GPG program: /usr/bin/gpg\nUse HTTPS protocol: True\nHTTP Proxy server name: \nHTTP Proxy server port: 0\n

Save the configuration file, and check access to the S3 object store with\u2026

# Create a bucket\ns3cmd -c mys3cfg mb s3://eoepca\n\n# List buckets\ns3cmd -c mys3cfg ls\n

For example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube\u2026 

s3cmd -c deploy/cluster/s3cfg ls\n

"},{"location":"cluster/cluster-prerequisites/#references_1","title":"References","text":"
  • MinIO Website
  • MinIO Helm Chart
  • MinIO on GitHub
"},{"location":"cluster/helm-repositories/","title":"Helm Repositories","text":"

Note

This section identifies some helm chart repositories that can be referenced (for convenience) via helm add. Nevertheless, all helm commands included in the guide specifically reference the source helm repository via the --repo argument to the helm install command - and thus it is not specifically necessary to add these repositories in advance.

"},{"location":"cluster/helm-repositories/#eoepca-helm-charts","title":"EOEPCA Helm Charts","text":"

The EOEPCA building-blocks are engineered as containers for deployment to a Kubernetes cluster. Each building block defines a Helm Chart to facilitate its deployment.

The EOEPCA Helm Chart Repository is configured with helm as follows\u2026 

helm repo add eoepca https://eoepca.github.io/helm-charts/\n

"},{"location":"cluster/helm-repositories/#third-party-helm-charts","title":"Third-party Helm Charts","text":"

In addition to the EOEPCA Helm Chart Repository, a variety of third party helm repositories are relied upon, as identified below.

"},{"location":"cluster/helm-repositories/#cert-manager","title":"Cert Manager","text":"
helm repo add jetstack https://charts.jetstack.io\n
"},{"location":"cluster/helm-repositories/#nginx-ingress-controller","title":"Nginx Ingress Controller","text":"
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\n
"},{"location":"cluster/helm-repositories/#minio","title":"Minio","text":"
helm repo add minio https://charts.min.io/\n
"},{"location":"cluster/helm-repositories/#sealed-secrets-bitnami","title":"Sealed Secrets (Bitnami)","text":"
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets\n
"},{"location":"cluster/helm-repositories/#harbor","title":"Harbor","text":"
helm repo add harbor https://helm.goharbor.io\n
"},{"location":"cluster/helm-repositories/#repo-update","title":"Repo Update","text":"

Refresh the local repo cache, after helm repo add\u2026

helm repo update\n
"},{"location":"cluster/kubernetes/","title":"Kubernetes Cluster","text":"

The EOEPCA Reference Implementation has been developed with Kubernetes as its deployment target. The system components have been developed, deployed and tested using a cluster at version v1.22.5.

Note

The Scripted Deployment assumes that minikube is installed, and creates a minikube cluster under the profile eoepca.

"},{"location":"cluster/kubernetes/#rancher-kubernetes-engine-rke","title":"Rancher Kubernetes Engine (RKE)","text":"

The development, integration and test clusters have been established using Rancher Kubernetes Engine (RKE) at version v1.22.5.

An example of the creation of the EOEPCA Kubernetes clusters can be found on the GitHub Kubernetes Setup page. CREODIAS has been used for the development hosting infrastructure - which provides OpenStack infrastructure that is backed by Cloudferro. An example of the Terraform configurations used to automate the creation of the cloud infrastructure that underpins the RKE deployment can be found on the GitHub CREODIAS Setup page.

"},{"location":"cluster/kubernetes/#local-kubernetes","title":"Local Kubernetes","text":"

To make a full deployment of the EOEPCA Reference Implementation requires a multi-node node cluster with suitable resources. For example, the development cluster comprises:

  • 1 Master node (2 vCPU, 8 GB RAM)
  • 5 Worker nodes (4 vCPU, 16 GB RAM)
  • 1 NFS server (2 vCPU, 8 GB RAM)

Limited local deployment can be made using a suitable local single-node kuberbetes deployment using - for example using minikube\u2026

minikube -p eoepca start --cpus max --memory max --kubernetes-version v1.22.5\nminikube profile eoepca\n

With such a deployment it is possible to deploy individual building-blocks for local development, or building-blocks in combination - within the constraints of the local host resources.

"},{"location":"cluster/prerequisite-tooling/","title":"Prerequisite Tooling","text":"

There are some standard tools referenced in this guide. These are detailed in the following subsections.

"},{"location":"cluster/prerequisite-tooling/#docker","title":"docker","text":"

Docker faciliates the creation, management and execution of containers. Whilst not strictly necessary to support deployment to an existing/managed Kubernetes cluster, it can nevertheless be useful to have local access to the docker tooling. For example, if minikube is used to follow this guide using a local k8s cluster, then this is best achieved using minikube\u2019s docker driver.

Docker is most easily installed with\u2026 

curl -fsSL https://get.docker.com | sh\n

For convenience, add your user to the docker group\u2026 

sudo usermod -aG docker ${USER}\n

Logout/in to refresh your session\u2019s group permissions.

"},{"location":"cluster/prerequisite-tooling/#kubectl","title":"kubectl","text":"

Kubectl is the main tool for interaction with a Kubernetes cluster. The latest version can be installed with\u2026 

mkdir -p $HOME/.local/bin \\\n&& curl -fsSLo $HOME/.local/bin/kubectl \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\" \\\n&& chmod +x $HOME/.local/bin/kubectl\n

See the official kubectl installation documentation for more installation options.

"},{"location":"cluster/prerequisite-tooling/#helm","title":"helm","text":"

Helm is the Kubernetes package manager, in which components are deployed to a Kubernetes cluster via helm charts. The helm charts are instantiated for deployment via \u2018values\u2019 that configure the chart templates.

The latest helm version can be installed with\u2026 

export HELM_INSTALL_DIR=\"$HOME/.local/bin\" \\\n&& curl -sfL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash\n

See the official helm installation documentation for more installation options.

"},{"location":"cluster/prerequisite-tooling/#minikube","title":"minikube","text":"

Minikube is a tool that allows to create a local (single-node) Kubernetes cluster for development/testing. It is not designed for production use. In the absence of access to a \u2018full\u2019 Kubernetes cluster, this guide can be followed using minikube.

The latest version of minikube can be installed with\u2026 

mkdir -p $HOME/.local/bin \\\n&& curl -fsSLo $HOME/.local/bin/minikube \"https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\" \\\n&& chmod +x $HOME/.local/bin/minikube\n

See the official minikube installation documentation for more installation options.

"},{"location":"eoepca/ades-zoo/","title":"ADES (Processing)","text":"ADES - Application Deployment & Execution Service"},{"location":"eoepca/ades-zoo/#zoo-project-dru","title":"ZOO-Project DRU","text":"

Note

With EOEPCA release 1.4, the ADES implementation has been significantly reworked and fully aligned with the upstream ZOO-Project (GitHub). This zoo-project-dru version deprecates the previous proc-ades implementation.

With this transition, there are some functional changes to be aware of\u2026

  • Service Endpoint With zoo-project-dru the OGC API Processes endpoint is at the path /<username>/ogc-api/processes compared to the previous /<username>/wps3/processes.
  • Deployed Application Endpoint The endpoint for a deployed Application no longer appends the version of the Application Package. For example, previously the application convert-url at version 0.1.2 would result in the endpoint /<username>/wps3/processes/convert-url_0_1_2. With the new zoo-project-dru this same Application Package deployment will result in the endpoint /<username>/ogc-api/processes/convert-url.
  • Deployed Application Version The version of the deployed application is obtained from the Application Package CWL (ref. s:softwareVersion: 0.1.2), and is maintained within the metadata for the deployed process that is returned from the APIs Get Process Details request. In the case that multiple versions of the same Application Package are required to be simultaneously deployed, then this would have to be handled with different CWL documents in which the version is embedded in the workflow id (or some other technique that establishes uniqueness of id between variants).
DRU - Deploy, Replace, Undeploy - OFC API Processes Part 2

The ADES provides a platform-hosted execution engine through which users can initiate parameterised processing jobs using applications made available within the platform - supporting the efficient execution of the processing \u2018close to the data\u2019. Users can deploy specific \u2018applications\u2019 to the ADES, which may be their own applications, or those published by other platform users.

The ADES provides an implementation of the OGC API Processes - Part 1: Core and Part 2: Deploy, Replace, Undeploy (draft).

"},{"location":"eoepca/ades-zoo/#helm-chart","title":"Helm Chart","text":"

The EOEPCA deployment is aligned with the upstream implementation and so relies upon the upstream helm chart that is hosted at the ZOO-Project Helm Chart Repository - in particular the zoo-project-dru chart variant.

The chart is configured via values that are fully documented in the README for the zoo-project-dru chart.

helm install --version 0.2.6 --values ades-values.yaml \\\n  --repo https://zoo-project.github.io/charts/ \\\n  zoo-project-dru zoo-project-dru\n
"},{"location":"eoepca/ades-zoo/#values","title":"Values","text":"

The deployment must be configured for you environment. Some significant configuration values are elaborated here\u2026

"},{"location":"eoepca/ades-zoo/#cookie-cutter-template","title":"Cookie-cutter Template","text":"

The implementation zoo-project-dru provides the core capabilities for OGC API Processes Parts 1 & 2. The deployemnt of this core must be completed by inetgartion with the \u2018runner\u2019 that executes the processes as Application Packages, and integrates as necessary with other platform services - such as Catalogue, Workspace, etc.

Thus, zoo-project-dru is extensible by design via a \u2018cookie-cutter\u2019 that provides the template \u2018runner\u2019 for each Application Package process as it is deployed to the service.

For the purposes of our EOEPCA \u2018release\u2019 as covered by this guide, we provide eoepca-proc-service-template as a cookie-cutter implemetation that provides:

  • Integration with Kubernetes to run process Application packages, via the Calrissian CWL runner
  • Stage-in of inputs as STAC items, integrated as required with S3 object storage
  • Stage-out of outputs as a STAC Collection, integrated with S3 object storage and (optionally) user Workspace inetgration

The cookie-cutter template is identified in the helm values\u2026

cookiecutter:\n  templateUrl: https://github.com/EOEPCA/eoepca-proc-service-template.git\n  templateBranch: master\n

The function of the cookie-cutter template is supported some other aspects, that are elaborated below, which must be configured in collaboration with the expectations of the template. In particular\u2026

  • Template parameterisation that is passed through the core zoo-project-dru configuration [ref]
  • CWL \u2018wrapper\u2019 files that prepend and append the process Application Package CWL to perform stage-in and stage-out functions [ref]
"},{"location":"eoepca/ades-zoo/#zoo-project-dru-custom-configuration","title":"ZOO-Project DRU custom configuration","text":"

In order support our eoepca-proc-service-template cookie-cutter template, there is a custom zoo-project-dru container image that includes the python dependencies that are required by this template. Thus, the deployment must identify the custom container image via helm values\u2026

zoofpm:\n  image:\n    tag: eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762\nzookernel:\n  image:\n    tag: eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762\n

In addition, we can add values to the ZOO-Project DRU main.cfg configuration file via helm values. In this case we add some eoepca-specific values that match those that we know to be expected by our eoepca-proc-service-template cookie-cutter template. In this way we can effectively use helm values to pass parameters through to the template.

customConfig:\n  main:\n    eoepca: |-\n      domain=192-168-49-2.nip.io\n      workspace_prefix=ws\n

This is manifest in zoo\u2019s main.cfg in INI file configuration syntax\u2026

[eoepca]\ndomain=192-168-49-2.nip.io\nworkspace_prefix=ws\n

The presence or otherwise of the workspace_prefix parameter dicates whether or not the stage-out step will integrate with the user\u2019s Workspace for persistence of the processing results, and registration within the Workspace services.

In the case that workspace_prefix is not set, then the object storage specification in the helm values is relied upon\u2026

workflow:\n  inputs:\n    STAGEOUT_AWS_SERVICEURL: https://minio.192-168-49-2.nip.io\n    STAGEOUT_AWS_ACCESS_KEY_ID: eoepca\n    STAGEOUT_AWS_SECRET_ACCESS_KEY: changeme\n    STAGEOUT_AWS_REGION: RegionOne\n    STAGEOUT_OUTPUT: eoepca\n
"},{"location":"eoepca/ades-zoo/#stage-in-stage-out","title":"Stage-in / Stage-out","text":"

The ADES hosts applications that are deployed and invoked in accordance with the OGC Best Practise for Application Package. Thus, the ADES provides a conformant environment within which the application is integrated for execution. A key part of the ADES\u2019s role in this is to faciltate the provision of input data to the application (stage-in), and the handling of the results output at the conclusion of application execution (stage-out).

The zoo-project-dru helm chart provides a default implementation via the included files - main.yaml, rules.yaml, stagein.yaml and stageout.yaml.

The helm values provides a means through which each of these files can be overriden for reasons of integration with your platform environment\u2026

files:\n  # Directory 'files/cwlwrapper-assets' - assets for ConfigMap 'XXX-cwlwrapper-config'\n  cwlwrapperAssets:\n    main.yaml: |-\n      <override file content here>\n    rules.yaml: |-\n      <override file content here>\n    stagein.yaml: |-\n      <override file content here>\n    stageout.yaml: |-\n      <override file content here>\n

In the most part the default CWL wrapper files provided with the helm chart are suffient. In particular the stagein.yaml implements the stage-in of STAC items that are specified as inputs of type Directory in the Application Package CWL.

E.g. 

    inputs:\n      stac:\n        label: the image to convert as a STAC item\n        doc: the image to convert as a STAC item\n        type: Directory\n

Nevertheless, in this guide we provide an override of the stageout.yaml in order to organise the processing outputs into a STAC Collection that is then pushed to the designated S3 object storage, including support for the user\u2019s workspace storage and resource management services.

The custom stage-out embeds, within the CWL document, the python code required to implement the desired stage-out functionality. This should be regarded as an example that could be adapted for alternative behaviour.

cwlVersion: v1.0\nclass: CommandLineTool\nid: stage-out\ndoc: \"Stage-out the results to S3\"\ninputs:\n  process:\n    type: string\n  collection_id:\n    type: string\n  STAGEOUT_OUTPUT:\n    type: string\n  STAGEOUT_AWS_ACCESS_KEY_ID:\n    type: string\n  STAGEOUT_AWS_SECRET_ACCESS_KEY:\n    type: string\n  STAGEOUT_AWS_REGION:\n    type: string\n  STAGEOUT_AWS_SERVICEURL:\n    type: string\noutputs:\n  StacCatalogUri:\n    outputBinding:\n      outputEval: ${  return \"s3://\" + inputs.STAGEOUT_OUTPUT + \"/\" + inputs.process + \"/catalog.json\"; }\n    type: string\nbaseCommand:\n  - python\n  - stageout.py\narguments:\n  - $( inputs.wf_outputs.path )\n  - $( inputs.STAGEOUT_OUTPUT )\n  - $( inputs.process )\n  - $( inputs.collection_id )\nrequirements:\n  DockerRequirement:\n    dockerPull: ghcr.io/terradue/ogc-eo-application-package-hands-on/stage:1.3.2\n  InlineJavascriptRequirement: {}\n  EnvVarRequirement:\n    envDef:\n      AWS_ACCESS_KEY_ID: $( inputs.STAGEOUT_AWS_ACCESS_KEY_ID )\n      AWS_SECRET_ACCESS_KEY: $( inputs.STAGEOUT_AWS_SECRET_ACCESS_KEY )\n      AWS_REGION: $( inputs.STAGEOUT_AWS_REGION )\n      AWS_S3_ENDPOINT: $( inputs.STAGEOUT_AWS_SERVICEURL )\n  InitialWorkDirRequirement:\n    listing:\n      - entryname: stageout.py\n        entry: |-\n          import sys\n          import shutil\n          import os\n          import pystac\n\n          cat_url = sys.argv[1]\n\n          shutil.copytree(cat_url, \"/tmp/catalog\")\n          cat = pystac.read_file(os.path.join(\"/tmp/catalog\", \"catalog.json\"))\n\n          ...\n

The helm chart values provide the opportunity to pass through additional inputs - to satisfy the input specifications that are specified in the cwlwrapperAssets files\u2026

workflow:\n  inputs:\n    STAGEIN_AWS_SERVICEURL: http://data.cloudferro.com\n    STAGEIN_AWS_ACCESS_KEY_ID: test\n    STAGEIN_AWS_SECRET_ACCESS_KEY: test\n    STAGEIN_AWS_REGION: RegionOne\n    STAGEOUT_AWS_SERVICEURL: https://minio.192-168-49-2.nip.io\n    STAGEOUT_AWS_ACCESS_KEY_ID: eoepca\n    STAGEOUT_AWS_SECRET_ACCESS_KEY: changeme\n    STAGEOUT_AWS_REGION: RegionOne\n    STAGEOUT_OUTPUT: eoepca\n
"},{"location":"eoepca/ades-zoo/#node-selection","title":"Node Selection","text":"

The zoo-project-dru services uses a Node Selector to determine the node(s) upon which the processing execution is run. This is configured as a matching rule in the helm values, and must be tailored to your cluster.

For example, for minikube\u2026

workflow:\n  nodeSelector:\n    minikube.k8s.io/primary: \"true\"\n
"},{"location":"eoepca/ades-zoo/#ingress","title":"Ingress","text":"

Ingress can be enabled and configured to establish (reverse-proxy) external access to the zoo-project-dru services.

Hosturl

In the case that protection is enabled - e.g. via Resource Guard - then it is likely that ingress should be disabled here, since the ingress will instead be handled by the protection.

In this case, the hosturl parameter should be set to reflect the public url through the service will be accessed.

In the case that ingress is enabled then it is not necessary to specify the hosturl, since it will be taken from the ingress.hosts[0].host value.

Ingress disabled\u2026

ingress:\n  enabled: false\n  hosturl: zoo.192-168-49-2.nip.io\n

Ingress enabled\u2026

ingress:\n  enabled: true\n  annotations:\n    kubernetes.io/ingress.class: nginx\n    ingress.kubernetes.io/ssl-redirect: true\n    nginx.ingress.kubernetes.io/ssl-redirect: true\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  hosts:\n  - host: zoo-open.192-168-49-2.nip.io\n    paths:\n    - path: /\n      pathType: ImplementationSpecific\n  tls:\n  - hosts:\n    - zoo-open.192-168-49-2.nip.io\n    secretName: zoo-open-tls\n

The above example assumes that TLS should be enabled via Letsencrypt as certificate provider - see section Letsencrypt Certificates.

"},{"location":"eoepca/ades-zoo/#persistence","title":"Persistence","text":"

Various of the services deployed as part of zoo-project-dru rely upon dynamic provisioning of persistent storage volumes.

A number of helm values are impacted by this setting, which must be configured with the Storage Class appropriate to your cluster. For example, using the minikube standard storage class\u2026

workflow:\n  storageClass: standard\npersistence:\n  procServicesStorageClass: standard\n  storageClass: standard\n  tmpStorageClass: standard\npostgresql:\n  primary:\n    persistence:\n      storageClass: standard\n  readReplicas:\n    persistence:\n      storageClass: standard\nrabbitmq:\n  persistence:\n    storageClass: standard\n
"},{"location":"eoepca/ades-zoo/#built-in-iam","title":"Built-in IAM","text":"

ZOO-Project DRU has a built-in capability for Identity & Access Management (IAM), in which the zoo-project-dru service is configured as an OIDC client of an OIDC Identity Provider service.

This capability is disabled by the default deployment offered by this guide (ingress.enabled: false) - which instead (optionally) applies resource protection using the EOEPCA IAM solution. Nevertheless, the built-in IAM can be enabled and configured through helm values.

For example\u2026

iam: \n  enabled: true\n  openIdConnectUrl: https://keycloak.192-168-49-2.nip.io/realms/master/.well-known/openid-configuration\n  type: openIdConnect\n  name: OpenIDAuth\n  realm: Secured section\n
"},{"location":"eoepca/ades-zoo/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the zoo-project-dru service to provide access authorization decisions

"},{"location":"eoepca/ades-zoo/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install zoo-project-dru-protection identity-gatekeeper -f zoo-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"zoo\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the zoo-project-dru - in particular the specific ingress requirements for the zoo-project-dru-service\u2026

Example zoo-protection-values.yaml\u2026

fullnameOverride: zoo-project-dru-protection\nconfig:\n  client-id: ades\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: zoo.192-168-49-2.nip.io\n  name: zoo-project-dru-service\n  port:\n    number: 80\nsecrets:\n  # Values for secret 'zoo-project-dru-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  serverSnippets:\n    custom: |-\n      # Open access to some endpoints, including Swagger UI\n      location ~ /(ogc-api/api|swagger-ui) {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n
"},{"location":"eoepca/ades-zoo/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: ades above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example, with path protection for test users\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=ades \\\n  --name=\"ADES Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by ADES Gatekeeper\" \\\n  --resource=\"eric\" --uris='/eric/*' --scopes=view --users=\"eric\" \\\n  --resource=\"bob\" --uris='/bob/*' --scopes=view --users=\"bob\" \\\n  --resource=\"alice\" --uris='/alice/*' --scopes=view --users=\"alice\"\n
"},{"location":"eoepca/ades-zoo/#service-urls","title":"Service URLs","text":"

The zoo-project-dru service provides a mutil-user aware set of service interfaces at\u2026

  • OGC API Processes: https://zoo.192-168-49-2.nip.io/<username>/ogc-api/
  • Swagger UI: https://zoo.192-168-49-2.nip.io/swagger-ui/oapip/
"},{"location":"eoepca/ades-zoo/#usage-samples","title":"Usage Samples","text":"

See the Example Requests in the Processing Deployment for sample requests that cans be used to test your deployment, and to learn usage of the OGC API Processes.

"},{"location":"eoepca/ades-zoo/#debugging-tips","title":"Debugging Tips","text":"

This section includes some tips that may be useful in debugging errors with deployed application packages.

For debugging, establish a shell session with the zoofpm pod\u2026

$ kubectl -n zoo exec -it deploy/zoo-project-dru-zoofpm -c zoofpm -- bash\n
"},{"location":"eoepca/ades-zoo/#execution-logs","title":"Execution Logs","text":"

The logs are in the directory /tmp/zTmp\u2026

$ cd /tmp/zTmp/\n

In the log directory, each execution is characterised by a set of files/directories\u2026

  • <appname>_<jobid>_error.log <<START HERE The main log output of the job
  • <appname>_<jobid>.json The output (results) of the job
  • <jobid>_status.json The overall status of the job
  • <jobid>_logs.cfg Index of logs for job workflow steps
  • convert-url-c6637d4a-d561-11ee-bf3b-0242ac11000e (directory) Subdirectory with a dedicated log file for each step of the CWL workflow, including the stage-in and stage-out steps
"},{"location":"eoepca/ades-zoo/#deployed-process-executables","title":"Deployed Process \u2018Executables\u2019","text":"

When the process is deployed from its Application Package, then a representation is created using the configured cookiecutter.templateUrl.

It may be useful to debug the consequent process files, which are located under the path /opt/zooservices_user/<username>, with a dedicated subdirectory for each deployed process - i.e. /opt/zooservices_user/<username>/<appname>/.

For example\u2026

$ cd /opt/zooservices_user/eric/convert-url\n$ ls -l\ntotal 28\n-rw-rw-r-- 1 www-data www-data     0 Feb 27 11:17 __init__.py\ndrwxrwxr-x 2 www-data www-data  4096 Feb 27 11:17 __pycache__\n-rw-rw-r-- 1 www-data www-data  1408 Feb 27 11:17 app-package.cwl\n-rw-rw-r-- 1 www-data www-data 17840 Feb 27 11:17 service.py\n

Note

In the case that the cookie-cutter template is updated, then the process can be re-deployed to force a refresh against the updated template.

"},{"location":"eoepca/ades-zoo/#swagger-ui-openapi","title":"Swagger UI (OpenAPI)","text":"

The zoo-project-dru service includes a Swagger UI interactive representation of its OpenAPI REST interface - available at the URL https://zoo.192-168-49-2.nip.io/swagger-ui/oapip/.

"},{"location":"eoepca/ades-zoo/#application-package-example","title":"Application Package Example","text":"

For a (trivial) example application package see Example Application Package, which provides a description and illustration of the basics of creating an application that integrates with the expectations of the ADES stage-in and stage-out.

For further reference see\u2026

  • Application Packages
    • OGC Best Practise for Application Package
    • Example Application Package
  • Common Workflow Language (CWL)
    • Guide for CWL in Earth Observation
    • CWL Specification
    • CWL User Guide
"},{"location":"eoepca/ades-zoo/#additional-information","title":"Additional Information","text":"

Additional information regarding the ADES can be found at:

  • ZOO-Project DRU\u2026
    • Helm Chart
    • Documentation
  • Git Repositories\u2026
    • ZOO-Project Core OGC API Processes capability
    • eoepca-proc-service-template Cookie-cutter template for Application Package execution in Kubernetes
    • zoo-calrissian-runner Python library used by the eoepca-proc-service-template to aid orchestration of CWL application packages running in Kubernetes via Calrissian
    • pycalrissian Python library used by zoo-calrissian-runner to aid interfacing with Calrissian and Kubernetes
"},{"location":"eoepca/application-hub/","title":"Application Hub","text":"

The Application Hub provides a set of web-based tooling, including JupyterLab for interactive analysis, Code Server for application development, and the capability to add user-defined interactive dashboards.

"},{"location":"eoepca/application-hub/#helm-chart","title":"Helm Chart","text":"

The Application Hub is deployed via the application-hub helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values, which are detailed in the default values file for the chart.

helm install --version 2.0.57 --values application-hub-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  application-hub application-hub\n
"},{"location":"eoepca/application-hub/#values","title":"Values","text":"

The Application Hub supports many values to configure the service - ref. the default values file for the chart.

Typically, values for the following attributes may be specified:

  • The fully-qualified public URL for the service
  • Specification of Ingress for reverse-proxy access to the service
  • Storage class for persistence
  • Node selector rule - required by JupyterHub to spawn container workloads
  • Values for integration with the user workspace
  • Integration of JupyterHub with the Login Service (identity provider) via OpenID Connect configuration
  • OIDC client credentials from a secret

Example application-hub-values.yaml\u2026

ingress:\n  enabled: true\n  annotations: {}\n  hosts:\n    - host: applicationhub.192-168-49-2.nip.io\n      paths:\n        - path: /\n          pathType: ImplementationSpecific\n  tls:\n    - secretName: applicationhub-tls\n      hosts:\n      - applicationhub.192-168-49-2.nip.io\n  clusterIssuer: letsencrypt-production\n\njupyterhub:\n  fullnameOverride: \"application-hub\"\n  hub:\n    existingSecret: application-hub-secrets\n    extraEnv: \n        JUPYTERHUB_ENV: \"dev\"\n        JUPYTERHUB_SINGLE_USER_IMAGE: \"eoepca/pde-container:1.0.3\"\n        OAUTH_CALLBACK_URL: https://applicationhub.192-168-49-2.nip.io/hub/oauth_callback\n        OAUTH2_USERDATA_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/userinfo\n        OAUTH2_TOKEN_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/token\n        OAUTH2_AUTHORIZE_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/authorize\n        OAUTH_LOGOUT_REDIRECT_URL: \"https://applicationhub.192-168-49-2.nip.io\"\n        OAUTH2_USERNAME_KEY: \"preferred_username\"\n        STORAGE_CLASS: \"standard\"\n        RESOURCE_MANAGER_WORKSPACE_PREFIX: \"ws\"\n\n        JUPYTERHUB_CRYPT_KEY:\n          valueFrom:\n            secretKeyRef:\n              name: application-hub-secrets\n              key: JUPYTERHUB_CRYPT_KEY\n\n        OAUTH_CLIENT_ID:\n          valueFrom:\n            secretKeyRef:\n              name: application-hub-secrets\n              key: OAUTH_CLIENT_ID\n\n        OAUTH_CLIENT_SECRET:\n          valueFrom:\n            secretKeyRef:\n              name: application-hub-secrets\n              key: OAUTH_CLIENT_SECRET\n\n    image:\n      # name: eoepca/application-hub\n      # tag: \"1.2.0\"\n      pullPolicy: Always\n      # pullSecrets: []\n\n    db:\n      pvc:\n        storageClassName: standard\n\n  singleuser:\n    image:\n      name: jupyter/minimal-notebook\n      tag: \"2343e33dec46\"\n    profileList: \n    - display_name:  \"Minimal environment\"\n      description: \"To avoid too much bells and whistles: Python.\"\n      default: \"True\"\n    - display_name:  \"EOEPCA profile\"\n      description: \"Sample profile\"\n      kubespawner_override:\n        cpu_limit\": 4\n        mem_limit\": \"8G\"\n\nnodeSelector:\n  key: minikube.k8s.io/primary\n  value: \\\"true\\\"\n
"},{"location":"eoepca/application-hub/#client-and-credentials","title":"Client and Credentials","text":"

The Application Hub requires an OIDC client to be registered with the Identity Service (Keycloak) in order to enable user identity integration - ref. OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=application-hub \\\n  --name=\"Application Hub OIDC Client\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Application Hub for OIDC integration\"\n

Corresponding to this client, a secret application-hub-secrets must be created (ref. value jupyterhub.hub.existingSecret: application-hub-secrets)\u2026

kubectl -n proc create secret generic application-hub-secrets \\\n  --from-literal=JUPYTERHUB_CRYPT_KEY=\"$(openssl rand -hex 32)\" \\\n  --from-literal=OAUTH_CLIENT_ID=\"application-hub\" \\\n  --from-literal=OAUTH_CLIENT_SECRET=\"changeme\"\n
"},{"location":"eoepca/application-hub/#post-deployment-manual-steps","title":"Post-deployment Manual Steps","text":"

The deployment of the Application Hub has been designed, as far as possible, to automate the configuration. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026

  • Configure Groups and Users
"},{"location":"eoepca/application-hub/#groups-and-users","title":"Groups and Users","text":"

The default helm chart has some built-in application launchers whose assignments to example users (eric and bob) assume the existence of some JupyterHub groups - which must be replicated to exploit this configuration.

  • In a browser, navigate to the Application Hub - https://applicationhub.192-168-49-2.nip.io/
  • Login as the user eric (or bob) for admin access
  • Select the Admin menu (top of page)
  • Add groups group-1, group-2, group-3 to ApplicationHub, and add users eric, bob to these groups

This setup corresponds to the \u2018sample\u2019 configuration that is built=in to the help chart - see file config.yaml.

"},{"location":"eoepca/application-hub/#additional-information","title":"Additional Information","text":"

Additional information regarding the Application Hub can be found at:

  • Helm Chart
"},{"location":"eoepca/container-registry/","title":"Container Registry","text":"

To support the development (ref. Application Hub) and deployment/execution (ref. ADES) of user-defined applications, we deploy a container registry to host container images. This is provied by a deployment of the Harbor artefact repository.

"},{"location":"eoepca/container-registry/#helm-chart","title":"Helm Chart","text":"

Harbor is deployed via the harbor helm chart from the Harbor Helm Chart Repository.

helm install --version 1.7.3 --values harbor-values.yaml \\\n   --repo https://helm.goharbor.io \\\n  harbor harbor\n
"},{"location":"eoepca/container-registry/#values","title":"Values","text":"

The chart is configured via values that are fully documented on the Harbor website.

Example\u2026

expose:\n  ingress:\n    annotations:\n      kubernetes.io/ingress.class: nginx\n      cert-manager.io/cluster-issuer: letsencrypt-production\n      nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n\n      # from chart:\n      ingress.kubernetes.io/ssl-redirect: letsencrypt-production\n      ingress.kubernetes.io/proxy-body-size: \"0\"\n      nginx.ingress.kubernetes.io/ssl-redirect: letsencrypt-production\n      nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n\n    hosts:\n      core: harbor.192-168-49-2.nip.io\n    tls:\n      enabled: \"true\"\n      certSource: secret\n      secret:\n        secretName: \"harbor-tls\"\n\npersistence:\n  persistentVolumeClaim:\n    registry:\n      storageClass: standard\n    chartmuseum:\n      storageClass: standard\n    jobservice:\n      storageClass: standard\n    database:\n      storageClass: standard\n    redis:\n      storageClass: standard\n    trivy:\n      storageClass: standard\n\nexternalURL: https://harbor.192-168-49-2.nip.io\n# initial password for logging in with user \"admin\"\nharborAdminPassword: \"changeme\"\n\nchartmuseum:\n  enabled: false\ntrivy:\n  enabled: false\nnotary:\n  enabled: false\n

Note

  • We specify use of \u2018valid\u2019 certificates from Letsencrypt \u2018production\u2019. The Workspace API, which calls the Harbor API, expects valid certificates and will thus fail if presented with TLS certificates that fail validation.
  • The letsencrypt-production Cluster Issuer relies upon the deployment being accessible from the public internet via the expose.ingress.hosts.core DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller. The Workspace API will not like this.
"},{"location":"eoepca/container-registry/#container-registry-usage","title":"Container Registry Usage","text":"

After deployemnt Harbor is accessible via its web interface at https://harbor.192-168-49-2.nip.io/e.g. https://harbor.192-168-49-2.nip.io/.

Login as the admin user with the password specified in the helm values.

"},{"location":"eoepca/container-registry/#additional-information","title":"Additional Information","text":"

Additional information regarding the Container Registry can be found at:

  • Web Site
  • Helm Chart Repository
  • Helm Chart Description
  • Harbor Documentation
"},{"location":"eoepca/data-access/","title":"Data Access","text":"

The Data Access provides standards-based services for access to platform hosted data - including OGC WMS/WMTS for visualisation, and OGC WCS for data retrieval. This component also includes Harvester and Registrar services to discover/watch the existing data holding of the infrastructure data layer and populate/maintain the data access and resource catalogue services accordingly.

"},{"location":"eoepca/data-access/#helm-chart","title":"Helm Chart","text":"

The Data Access is deployed via the data-access helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are supplied with the instantiation of the helm release. The EOEPCA data-access chart provides a thin wrapper around the EOX View Server (vs) helm chart. The documentation for the View Server can be found here:

  • User Guide: https://vs.pages.eox.at/documentation/user/main/
  • Operator Guide: https://vs.pages.eox.at/documentation/operator/main/
helm install --version 1.4.0 --values data-access-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  data-access data-access\n
"},{"location":"eoepca/data-access/#values","title":"Values","text":"

The Data Access supports many values to configure the service. These are documented in full in the View Server - Operator Guide Configuration page.

"},{"location":"eoepca/data-access/#core-configuration","title":"Core Configuration","text":"

Typically, values for the following attributes may be specified to override the chart defaults:

  • The fully-qualified public URL for the service, ref. (global.ingress.hosts.host[0])
  • Metadata describing the service instance
  • Dynamic provisioning StorageClass for persistence
  • Persistent Volume Claims for database and redis components
  • Object storage details for data and cache
  • Container images for renderer and registrar
  • (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Data Access will not be protected by the identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.
global:\n  env:\n    REGISTRAR_REPLACE: \"true\"\n    CPL_VSIL_CURL_ALLOWED_EXTENSIONS: .TIF,.tif,.xml,.jp2,.jpg,.jpeg\n    AWS_ENDPOINT_URL_S3: https://minio.192-168-49-2.nip.io\n    AWS_HTTPS: \"FALSE\"\n    startup_scripts:\n      - /registrar_pycsw/registrar_pycsw/initialize-collections.sh\n  ingress:\n    enabled: true\n    annotations:\n      kubernetes.io/ingress.class: nginx\n      kubernetes.io/tls-acme: \"true\"\n      nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n      nginx.ingress.kubernetes.io/enable-cors: \"true\"\n      cert-manager.io/cluster-issuer: letsencrypt-production\n    hosts:\n      - host: data-access.192-168-49-2.nip.io\n    tls:\n      - hosts:\n          - data-access.192-168-49-2.nip.io\n        secretName: data-access-tls\n  storage:\n    data:\n      data:\n        type: S3\n        endpoint_url: http://data.cloudferro.com\n        access_key_id: access\n        secret_access_key: access\n        region_name: RegionOne\n        validate_bucket_name: false\n    cache:\n      type: S3\n      endpoint_url: \"https://minio.192-168-49-2.nip.io\"\n      host: \"minio.192-168-49-2.nip.io\"\n      access_key_id: xxx\n      secret_access_key: xxx\n      region: us-east-1\n      bucket: cache-bucket\n  metadata:\n    title: EOEPCA Data Access Service developed by EOX\n    abstract: EOEPCA Data Access Service developed by EOX\n    header: \"EOEPCA Data Access View Server (VS) Client powered by <a href=\\\"//eox.at\\\"><img src=\\\"//eox.at/wp-content/uploads/2017/09/EOX_Logo.svg\\\" alt=\\\"EOX\\\" style=\\\"height:25px;margin-left:10px\\\"/></a>\"\n    url: https://data-access.192-168-49-2.nip.io/ows\n  layers:\n    # see section 'Data-layer Configuration'\n  collections:\n    # see section 'Data-layer Configuration'\n  productTypes:\n    # see section 'Data-layer Configuration'\nvs:\n  renderer:\n    replicaCount: 4\n    ingress:\n      enabled: false\n    resources:\n      requests:\n        cpu: 100m\n        memory: 300Mi\n      limits:\n        cpu: 1.5\n        memory: 3Gi\n  registrar:\n    replicaCount: 1\n    config:\n      # see section 'Registrar Routes Configuration'\n    resources:\n      requests:\n        cpu: 100m\n        memory: 100Mi\n  harvester:\n    # see section 'Harvester Configuration'\n    replicaCount: 1\n    resources:\n      requests:\n        cpu: 100m\n        memory: 100Mi\n  client:\n    replicaCount: 1\n    ingress:\n      enabled: false\n  redis:\n    master:\n      persistence:\n        enabled: true\n        storageClass: standard\n  ingestor:\n    replicaCount: 0\n    ingress:\n      enabled: false\n  preprocessor:\n    replicaCount: 0\n  cache:\n    ingress:\n      enabled: false\n  scheduler:\n    resources:\n      requests:\n        cpu: 100m\n        memory: 100Mi\n

Note

The resources: above have been limited for the benefit of a minikube deployment. For a production deployment the values should be tuned (upwards) according to operational needs.

"},{"location":"eoepca/data-access/#registrar-routes-configuration","title":"Registrar Routes Configuration","text":"

The Data Access registrar component supports a number of different resource types. For each a dedicated \u2018backend\u2019 is configured to handle the specific registration of the resource type\u2026

vs:\n  registrar:\n    config:\n      #--------------\n      # Default route\n      #--------------\n      disableDefaultRoute: false\n      # Additional backends for the default route\n      defaultBackends:\n        - path: registrar_pycsw.backend.ItemBackend\n          kwargs:\n            repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n            ows_url: https://data-access.192-168-49-2.nip.io/ows\n      defaultSuccessQueue: seed_queue\n      #----------------\n      # Specific routes\n      #----------------\n      routes:\n        collections:\n          path: registrar.route.stac.CollectionRoute\n          queue: register_collection_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.CollectionBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        ades:\n          path: registrar.route.json.JSONRoute\n          queue: register_ades_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.ADESBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        application:\n          path: registrar.route.json.JSONRoute\n          queue: register_application_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.CWLBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        catalogue:\n          path: registrar.route.json.JSONRoute\n          queue: register_catalogue_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.CatalogueBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        json:\n          path: registrar.route.json.JSONRoute\n          queue: register_json_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.JSONBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        xml:\n          path: registrar.route.json.JSONRoute\n          queue: register_xml_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.XMLBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n
"},{"location":"eoepca/data-access/#data-layer-configuration","title":"Data-layer Configuration","text":"

Configuration of the service data-layer - as described in the View Server Operator Guide.

The data-access service data handling is configured by definition of productTypes, collections and layers\u2026

  • productTypes - Product Types Identify the underlying file assets as WCS coverages and their visual representation
  • collections - Data Collections Provides groupings into which products are organised
  • layers - Layers Specifies the hoe the product visual representations are exposed through the WMS service

For more information, see the worked example in section Data Specification for the example CREODIAS deployment.

"},{"location":"eoepca/data-access/#harvester","title":"Harvester","text":"

The Data Access service includes a Harvester component. The following subsections describe its configuration and usage.

"},{"location":"eoepca/data-access/#harvester-helm-configuration","title":"Harvester Helm Configuration","text":"

The Harvester can be configured through the helm chart values\u2026

vs:\n  harvester:\n    replicaCount: 1\n    config:\n      redis:\n        host: data-access-redis-master\n        port: 6379\n      harvesters:\n        - name: Creodias-Opensearch\n          resource:\n            url: https://datahub.creodias.eu/resto/api/collections/Sentinel2/describe.xml\n            type: OpenSearch\n            format_config:\n              type: 'application/json'\n              property_mapping:\n                start_datetime: 'startDate'\n                end_datetime: 'completionDate'\n                productIdentifier: 'productIdentifier'\n            query:\n              time:\n                property: sensed\n                begin: 2019-09-10T00:00:00Z\n                end: 2019-09-11T00:00:00Z\n              collection: null\n              bbox: 14.9,47.7,16.4,48.7\n          filter: {}\n          postprocess:\n            - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor\n          queue: register\n        - name: Creodias-Opensearch-Sentinel1\n          resource:\n            url: https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml\n            type: OpenSearch\n            format_config:\n              type: 'application/json'\n              property_mapping:\n                start_datetime: 'startDate'\n                end_datetime: 'completionDate'\n                productIdentifier: 'productIdentifier'\n            query:\n              time:\n                property: sensed\n                begin: 2019-09-10T00:00:00Z\n                end: 2019-09-11T00:00:00Z\n              collection: null\n              bbox: 14.9,47.7,16.4,48.7\n              extra_params:\n                productType: GRD-COG\n          filter: {}\n          postprocess:\n            - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel1Postprocessor\n          queue: register\n

The harvester.config.harvesters list defines a set of pre-defined harvesters which can be invoked in a later stage. The name property must be unique for each harvester and must be unique among all harvesters in the list. Each harvester is associated with a resource, an optional filter or postprocess function, and a queue.

The resource defines where each item is harvested from. This can be a file system, a search service, catalog file or something similar. The example above defines a connection to an OpenSearch service on CREODIAS, with associated default query parameters and a format configuration.

The filter allows to filter elements within the harvester, when the resource does not provide a specific filter. This filter can be supplied using CQL2-JSON.

The postprocess can adjust the harvested results. In this example the harvested items are not complete, and additional metadata must be retrieved from an object storage.

The queue defines where harvested items will be pushed into. Usually this is a registration queue, where the registrar will pick up and start registration according to its configuration.

"},{"location":"eoepca/data-access/#starting-the-harvester","title":"Starting the Harvester","text":"

The harvester can either do one-off harvests via the CLI or listen on a redis queue to run consecutive harvests whenever a harvesting request is received on that queue.

"},{"location":"eoepca/data-access/#one-off-harvests-via-the-cli","title":"One-off harvests via the CLI","text":"

In order to start a harvest from the CLI, the operator first needs to connect to the kubernetes pod of the harvester. Within that pod, the harvest can be executed like this\u2026 

python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\n

This will invoke the Creodias-Opensearch harvester with default arguments. When some values are to be overridden, the \u2013values switch can be used to pass override values. These values must be a JSON string. The following example adjusts the begin and end times of the query parameters\u2026 

python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch --values '{\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}'\n

"},{"location":"eoepca/data-access/#harvests-via-the-harvest-daemon","title":"Harvests via the harvest daemon","text":"

The harvester pod runs a service listening on a redis queue. When a message is read from the queue, it will be read as a JSON string, expecting an object with at least a name property. Optionally, it can also have a values property, working in the same way as with CLI --values.

To send a harvesting request via the redis queue, it is necessary to connect to the redis pod and execute the redis-cli there. Then the following command can be used to achieve the same result as above with CLI harvesting\u2026 

redis-cli LPUSH '{\"name\": \"Creodias-Opensearch\", \"values\": {\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}}'\n

"},{"location":"eoepca/data-access/#results-of-the-harvesting","title":"Results of the harvesting","text":"

The harvester produces a continous stream of STAC Items which are sent down via the configured queue. It is possible that the harvested metadata is not sufficient to create a fully functional STAC Item. In this case the postprocess must transform this intermediate item to a valid STAC Item. In our example, the postprocessor looks up the Sentinel-2 product file referenced by the product identifier which is then accessed on the object storage. From the stored metadata files, the STAC Items to be sent is created.

"},{"location":"eoepca/data-access/#storage","title":"Storage","text":"

Specification of PVCs and access to object storage.

"},{"location":"eoepca/data-access/#persistent-volume-claims","title":"Persistent Volume Claims","text":"

The PVCs specified in the helm chart values must be created.

"},{"location":"eoepca/data-access/#pvc-for-database","title":"PVC for Database","text":"
kind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n  name: data-access-db\n  namespace: rm\n  labels:\n    k8s-app: data-access\n    name: data-access\nspec:\n  storageClassName: standard\n  accessModes:\n    - ReadWriteMany\n  resources:\n    requests:\n      storage: 100Gi\n
"},{"location":"eoepca/data-access/#pvc-for-redis","title":"PVC for Redis","text":"
kind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n  name: data-access-redis\n  namespace: rm\n  labels:\n    k8s-app: data-access\n    name: data-access\nspec:\n  storageClassName: standard\n  accessModes:\n    - ReadWriteMany\n  resources:\n    requests:\n      storage: 1Gi\n
"},{"location":"eoepca/data-access/#object-storage","title":"Object Storage","text":"

The helm chart values expect specification of object storage details for:

  • data: to access the EO data of the underlying infrastructure
  • cache: a dedicated object storage bucket is used to support the cache function of the data access services
"},{"location":"eoepca/data-access/#platform-eo-data","title":"Platform EO Data","text":"

Specifies the details for the infrastructure object storage that provides direct access to the EO product files.

For example, the CREODIAS metadata catalogue provides references to product files in their eodata object storage - the access details for which are configured in the data access services:

global:\n  storage:\n    data:\n      data:\n        type: S3\n        endpoint_url: http://data.cloudferro.com\n        access_key_id: access\n        secret_access_key: access\n        region_name: RegionOne\n        validate_bucket_name: false\n
"},{"location":"eoepca/data-access/#data-access-cache","title":"Data Access Cache","text":"

The Data Access services maintain a cache, which relies on the usage of a dedicate object storage bucket for data persistence. This bucket must be created (manual step) and its access details configured in the data access services. Example based upon CREODIAS:

global:\n  storage:\n    cache:\n      type: S3\n      endpoint_url: \"https://cf2.cloudferro.com:8080/cache-bucket\"\n      host: \"cf2.cloudferro.com:8080\"\n      access_key_id: xxx\n      secret_access_key: xxx\n      region: RegionOne\n      bucket: cache-bucket\n

\u2026where xxx must be replaced with the bucket credentials.

"},{"location":"eoepca/data-access/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the data-access service to provide access authorization decisions

"},{"location":"eoepca/data-access/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install data-access-protection identity-gatekeeper -f data-access-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"rm\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the data-access - in particular the specific ingress requirements for the data-access backend services\u2026

Example data-access-protection-values.yaml\u2026

fullnameOverride: data-access-protection\nconfig:\n  client-id: data-access\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: data-access.192-168-49-2.nip.io\n  name: data-access-renderer\n  port:\n    number: 80\nsecrets:\n  # Values for secret 'data-access-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n    nginx.ingress.kubernetes.io/enable-cors: \"true\"\n    nginx.ingress.kubernetes.io/rewrite-target: /$1\n  serverSnippets:\n    custom: |-\n      # Open access to renderer...\n      location ~ ^/(ows.*|opensearch.*|coverages/metadata.*|admin.*) {\n        proxy_pass http://data-access-renderer.rm.svc.cluster.local:80/$1;\n      }\n      # Open access to cache...\n      location ~ ^/cache/(.*) {\n        proxy_pass http://data-access-cache.rm.svc.cluster.local:80/$1;\n      }\n      # Open access to client...\n      # Note that we use a negative lookahead to avoid matching '/.well-known/*' which\n      # otherwise appears to interfere with the work of cert-manager/letsencrypt.\n      location ~ ^/(?!\\.well-known)(.*) {\n        proxy_pass http://data-access-client.rm.svc.cluster.local:80/$1;\n      }\n
"},{"location":"eoepca/data-access/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: data-access above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=data-access \\\n  --name=\"Data Access Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Data Access Gatekeeper\"\n
"},{"location":"eoepca/data-access/#data-access-usage","title":"Data Access Usage","text":""},{"location":"eoepca/data-access/#default-harvesting","title":"Default Harvesting","text":"

At deployment time the harvester helm values include configuration that populates a default harvester configuration, that is prepared in the file /config.yaml in the harvester pod.

The Data Access and Resource Catalogue services are configured to properly interpret harvested data via these values specified in the instantiation of the helm release. See section Data-layer Configuration.

The harvesting of data can be triggered (post deployment), in accordance with this default configuration, by connecting to the rm/harvester service and executing the command\u2026 

python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\n

"},{"location":"eoepca/data-access/#ad-hoc-harvesting","title":"Ad-hoc Harvesting","text":"

Ad-hoc harvesting can be invoked by provision of a suitable config.yaml into the harvester pod, which can then be invoked as shown above for the default harvester configuration established at deploy time.

The helper script ./deploy/bin/harvest faciltates this\u2026

./deploy/bin/harvest <path-to-config-file>\n

See directory ./deploy/samples/harvester/ that contains some sample harvesting configuration files. For example\u2026

./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2019.09.10.yaml\n
"},{"location":"eoepca/data-access/#registration-of-collections","title":"Registration of Collections","text":"

The helper script ./deploy/bin/register-collection is provided to faciltate the registration of collections that are specfied in STAC Collection format.

./deploy/bin/register-collection <path-to-stac-collection-file>\n

See directory ./deploy/samples/collections/ that contains some same STAC Collection files. For example\u2026

./deploy/bin/register-collection ./deploy/samples/collections/S2MSI2A.json\n
"},{"location":"eoepca/data-access/#additional-information","title":"Additional Information","text":"

Additional information regarding the Data Access can be found at:

  • Helm Chart
  • Documentation:
    • User Guide
    • Operator Guide
  • Git Repository
"},{"location":"eoepca/iam-overview/","title":"IAM Overview","text":"

This guide includes two approaches for Identity & Access Management:

  • Keycloak Solution (NEW)
  • Gluu Solution (deprecated)

Until now, our IAM solution has been based solely upon Gluu.

In the course of the project Keycloak has emerged as a preferred solution across EO platforms.

Thus, we have introduced an IAM approach based upon Keycloak, whilst retaining the Gluu-based approach for reference, which will be deprecated.

"},{"location":"eoepca/identity-service/","title":"Identity Service","text":"

The Identity Service provides the platform Authorization Server for authenticated user identity and request authorization.

Identity Service is composed of:

  • Keycloak IAM Authorization Service - supporting OpenID Connect (OIDC), etc.
  • Postgres DB Relational database used by Keycloak for persistence
  • Identity API Service that provided a convenience API to simplify IAM management interactions with Keycloak. Provides endpoints to create clients and protect resources. Uses a keycloak python client which sends requests to Keycloak API
  • Identity API Gatekeeper Instance of Gatekeeper to \u2018protect\u2019 access requests to the Identity API service. Gatekeeper is a reusable component that provides the Policy Enforcement for requests to individual resource servers. A Gatekeeper instance should be configured and deployed for each application that requires protection by access policies.
"},{"location":"eoepca/identity-service/#helm-chart","title":"Helm Chart","text":"

The Identity Service is deployed via the identity-service helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values - the full set of available values can be tailored according the helm chart defaults, that can be found here\u2026

  • identity-service https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/values.yaml
  • identity-keycloak https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-keycloak/values.yaml
  • identity-postgres https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-postgres/values.yaml
  • identity-api https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-api/values.yaml
helm install --version 1.0.97 --values identity-service-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  identity-service identity-service\n
"},{"location":"eoepca/identity-service/#values","title":"Values","text":"

The deployment must be configured for you environment. Some significant configuration values are elaborated here\u2026

"},{"location":"eoepca/identity-service/#identity-keycloak","title":"identity-keycloak","text":""},{"location":"eoepca/identity-service/#secrets","title":"Secrets","text":"

Keycloak relies upon a secret identity-keycloak that provides\u2026

  • KEYCLOAK_ADMIN_PASSWORD - admin password for Keycloak
  • KC_DB_PASSWORD - password for connecting with Postgres DB This should match the POSTGRES_PASSWORD setting for identity-postgres (see below)

The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026

identity-keycloak:\n  secrets:\n    # Values for secret 'identity-keycloak'\n    # Note - if ommitted, these can instead be set by creating the secret independently.\n    kcDbPassword: \"changeme\"\n    keycloakAdminPassword: \"changeme\"\n
"},{"location":"eoepca/identity-service/#ingress","title":"Ingress","text":"

The details for ingress (reverse-proxy) to the Keycloak service - in particular the hostname and possible TLS - must be specified\u2026

identity-keycloak:\n  ingress:\n    enabled: true\n    className: nginx\n    annotations:\n      ingress.kubernetes.io/ssl-redirect: \"true\"\n      nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n      cert-manager.io/cluster-issuer: letsencrypt-production\n    hosts:\n      - host: keycloak.192-168-49-2.nip.io\n        paths:\n          - path: /\n            pathType: Prefix\n    tls:\n      - secretName: identity-keycloak-tls\n        hosts:\n          - keycloak.192-168-49-2.nip.io\n
"},{"location":"eoepca/identity-service/#identity-postgres","title":"identity-postgres","text":""},{"location":"eoepca/identity-service/#secrets_1","title":"Secrets","text":"

Postgres relies upon a secret identity-postgres that provides\u2026

  • POSTGRES_PASSWORD - superuser password for PostgreSQL
  • PGPASSWORD - password used for client connections to the DB

The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026

identity-postgres:\n  secrets:\n    # Values for secret 'identity-postgres'\n    # Note - if ommitted, these can instead be set by creating the secret independently.\n    postgresPassword: \"changeme\"\n    pgPassword: \"changeme\"\n
"},{"location":"eoepca/identity-service/#persistence","title":"Persistence","text":"

In order to persist data, Postgres requires a Persistent Volume Claim.

This can be specified as an existing volume claim - for example as described in the Persistence section.

identity-postgres:\n  volumeClaim:\n    name: eoepca-userman-pvc\n
"},{"location":"eoepca/identity-service/#identity-api","title":"identity-api","text":""},{"location":"eoepca/identity-service/#secrets_2","title":"Secrets","text":"

The Identity API relies upon a secret identity-api that provides\u2026

  • ADMIN_PASSWORD Admin password for Keycloak This should match the KEYCLOAK_ADMIN_PASSWORD setting for identity-keycloak (see above)

The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026

identity-api:\n  secrets:\n    # Values for secret 'identity-api'\n    # Note - if ommitted, these can instead be set by creating the secret independently\n    # e.g. as a SealedSecret via GitOps.\n    adminPassword: \"changeme\"\n

Note

It is also possible to set the value of ADMIN_PASSWORD directly as an environment variable. In this case it is necessary to set the secret as optional\u2026

identity-api:\n  secrets:\n    optional: true\n
"},{"location":"eoepca/identity-service/#environment-variables","title":"Environment Variables","text":"

The Identity API service can be configured via environment variables as follows\u2026

  • AUTH_SERVER_URL URL of the Keycloak Authorization Server. Can also be set via value configMap.authServerUrl
  • ADMIN_USERNAME Admin user for Keycloak
  • REALM The Keycloak realm
identity-api:\n  deployment:\n    # Config values that can be passed via env vars\n    extraEnv:\n      - name: AUTH_SERVER_URL  # see configMap.authServerUrl instead\n        value: https://keycloak.192-168-49-2.nip.io\n      - name: ADMIN_USERNAME\n        value: admin\n      - name: ADMIN_PASSWORD  # see secrets.adminPassword instead\n        value: changeme\n      - name: REALM\n        value: master\n
"},{"location":"eoepca/identity-service/#identity-api-gatekeeper","title":"identity-api-gatekeeper","text":""},{"location":"eoepca/identity-service/#secrets_3","title":"Secrets","text":"

gatekeeper relies upon a secret identity-api-protection that provides\u2026

  • PROXY_CLIENT_SECRET Password for the Keycloak client configured for use by this Gatekeeper instance - corresponding to config.client-id.
  • PROXY_ENCRYPTION_KEY Encryption Key used by Gatekeeper.

The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026

identity-api-gatekeeper:\n  secrets:\n    # Values for secret 'identity-api-protection'\n    # Note - if ommitted, these can instead be set by creating the secret independently.\n    clientSecret: \"changeme\"\n    encryptionKey: \"changemechangeme\"\n
"},{"location":"eoepca/identity-service/#configuration","title":"Configuration","text":"

Configuration of Gatekeeper via the file config.yaml that is mounted into the deployment\u2026

  • client-id ID of the Keycloak client to be used by this Gatekeeper instance.
  • discovery-url Discovery URL of the Keycloak Authorization Server
  • cookie-domain Domain in which this Gatekeeper instance creates cookies 
identity-api-gatekeeper:\n  config:\n    client-id: identity-api\n    discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n    cookie-domain: 192-168-49-2.nip.io\n
"},{"location":"eoepca/identity-service/#ingress_1","title":"Ingress","text":"

The details for ingress (reverse-proxy) to the Gatekeeper service that protects the Identity API\u2026

identity-api-gatekeeper:\n  targetService:\n    host: identity-api.192-168-49-2.nip.io\n  ingress:\n    annotations:\n      ingress.kubernetes.io/ssl-redirect: \"true\"\n      nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n      cert-manager.io/cluster-issuer: letsencrypt\n
"},{"location":"eoepca/identity-service/#identity-api-client","title":"Identity API Client","text":"

The Identity API is protected via an instance of Gatekeeper - which relies upon a Keycloak client having been created for authorization decision/enforcement flows between Gatekeeper and Keycloak.

As described in the \u2018create-client\u2019 section below, this can be achieved using the create-client helper script.

Note

At time of client creation, the Identity API is not yet protected with an ingress. Therefore, we use a port-forward to interface directly with the Identity API service.

$ kubectl -n um port-forward svc/identity-api \"9876\":http >/dev/null &\n$ portForwardPid=$!\n\n$ ./deploy/bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i http://localhost:9876 \\\n  -r master \\\n  -u admin \\\n  -p changeme \\\n  -c admin-cli \\\n  --id=identity-api \\\n  --name=\"Identity API Gatekeeper\" \\\n  --secret=changeme \\\n  --description=\"Client to be used by Identity API Gatekeeper\" \\\n  --resource=\"admin\" --uris='/*' --scopes=view --users=\"admin\"\n\n$ kill -TERM $portForwardPid\n
"},{"location":"eoepca/identity-service/#create-user-helper-script","title":"create-user Helper Script","text":"

The Keycloak Admin UI can be used to create users interactively.

Alternatvely there is a helper script create-user that can be used.

The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The create-user helper script requires some command-line arguments\u2026

$ ./deploy/bin/create-user -h\n\nCreate a new user.\ncreate-user -h | -a {auth_server} -r {realm} -c {client} -u {admin-username} -p {admin-password} -U {new-username} -P {new-password}\n\nwhere:\n    -h  show help message\n    -a  authorization server url (default: http://keycloak.192-168-49-2.nip.io)\n    -r  realm within Keycloak (default: master)\n    -u  username used for authentication (default: admin)\n    -p  password used for authentication (default: changeme)\n    -c  client id of the bootstrap client used in the create request (default: admin-cli)\n    -U  name of the (new) user to create\n    -P  password for the (new) user to create\n
"},{"location":"eoepca/identity-service/#protection-of-resources","title":"Protection of Resources","text":"

The Identity Service is capable of protecting resources using OpenID-connect/SAML clients, resources (URIs/scopes), policies (user based, role based, etc) and permissions (associations between policies and resources).

Creating and protecting resources can be done in multiple ways, as described in the following sections.

"},{"location":"eoepca/identity-service/#keycloak-admin-ui","title":"Keycloak Admin UI","text":"

To create and protect resources using the keycloak User Interface (UI), do the following steps:

  • (Optional) Create clients. Clients can be created using the keycloak user interface at http://keycloak.192-168-49-2.nip.io. You need to login as admin. To create a client: Login as admin in the keycloak UI > Clients > Create Client > Set a name > Next > Turn Client Authentication and Authorization On > Add the valid redirect URI\u2019s > Save.
  • (Optional) Create Users. Users > Add User. Then set a password for the user. Credentials > Set Password.
  • Select a client.
  • Create a Resource: Select Authorization tab > Resources > Create Resource.
  • Create a Policy: In client details, select Authorization > Policies > Create Policy > Select Policy Type (e.g.: User) > Select users > Save.
  • Create Authorization Scope: In client details, select Authorization > Scopes > Create authorization scope > Save.
  • Create a Permission: In client details, select Authorization > Permissions > Create Permission > Create Resource Based Permission > Select Resources to protect > Select Policies > Save.
"},{"location":"eoepca/identity-service/#create-client-helper-script","title":"create-client Helper Script","text":"

Alternatively, a script was developed to allow simultaneaously create a client, create resources and protect them.

The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The create-client helper script requires some command-line arguments\u2026

$ ./deploy/bin/create-client -h\n\nAdd a client with protected resources.\ncreate-client [-h] [-a] [-i] [-u] [-p] [-c] [-s] [-t | --token t] [-r] --id id [--name name] (--secret secret | --public) [--default] [--authenticated] [--resource name] [--uris u1,u2] [--scopes s1,s2] [--users u1,u2] [--roles r1,r2]\n\nwhere:\n    -h                    show help message\n    -a                    authorization server url - e.g. https://keycloak.192-168-49-2.nip.io\n    -i                    identity-api server url - e.g. https://identity-api.192-168-49-2.nip.io\n    -u                    username used for authentication\n    -p                    password used for authentication\n    -c                    client id (of the bootstrap client used in the create request)\n    -s                    client secret (of the bootstrap client used in the create request)\n    -t or --token         access token used for authentication\n    -r                    realm\n    --id                  client id (of the created client)\n    --name                client name (of the created client)\n    --secret              client secret (of the created client)\n    --public              public client (no client secret)\n    --default             add default resource - /* authenticated\n    --authenticated       allow access to the resource only when authenticated\n    --resource            resource name\n    --uris                resource uris - separated by comma (,)\n    --scopes              resource scopes - separated by comma (,)\n    --users               user names with access to the resource - separated by comma (,)\n    --roles               role names with access to the resource - separated by comma (,)\n

The script interacts with Identity API and therefore requires admin authorization. It accepts basic authentication with username and password with -u and -p parameters, respectively - or a bearer access token with -t parameter.

To generate the access token needed to use the script, you can get it through the login in the eoepca portal, by accessing the cookies in the browser. See section EOEPCA Portal for details regarding deployment/configuration of the eoepca-portal.

Or you can generate an access token using postman oauth2.0, as described in the Postman document Requesting an OAuth 2.0 token.

Script execution examples:

  1. With username/password 

    ./deploy/bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r master \\\n  -u admin \\\n  -p changeme \\\n  -c admin-cli \\\n  --id=myservice-gatekeeper \\\n  --name=\"MyService Gatekeeper\" \\\n  --secret=changeme \\\n  --description=\"Client to be used by MyService Gatekeeper\" \\\n  --resource=\"Eric space\" --uris=/eric/* --users=eric \\\n  --resource=\"Alice space\" --uris=/alice/* --users=alice \\\n  --resource=\"Admin space\" --uris=/admin/* --roles=admin\n

  2. With access token 

    ./deploy/bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r master \\\n  -t eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJXZWFIY2pscThPc1RUYjdlV0s5SjJTTDFBUDIyazZpajdlMGFlVHRNU2xRIn0.eyJleHAiOjE3MDAyNDM4MzgsImlhdCI6MTcwMDI0Mzc3OCwiYXV0aF90aW1lIjoxNzAwMjQxODYyLCJqdGkiOiI2MWI0ZGRhYy1mOWZjLTRmZjktOWQ4Zi01NWU1N2NlNmE5ODgiLCJpc3MiOiJodHRwczovL2lkZW50aXR5LmtleWNsb2FrLmRldmVsb3AuZW9lcGNhLm9yZy9yZWFsbXMvbWFzdGVyIiwiYXVkIjpbImFkZXMtcmVhbG0iLCJkZW1vLXJlYWxtIiwiZHVtbXktc2VydmljZS1yZWFsbSIsIm1hc3Rlci1yZWFsbSIsImFjY291bnQiLCJlb2VwY2EtcmVhbG0iXSwic3ViIjoiZTNkZTMyNGUtMGY0NS00MWUwLTk2YTctNTM1YzkxMTA1NTUyIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiZW9lcGNhLXBvcnRhbCIsIm5vbmNlIjoiMTIwMGJlNzAtZWI1Ni00Nzc2LThjODgtOWRiOWQxMDdiMGY2Iiwic2Vzc2lvbl9zdGF0ZSI6ImVmNGUwOTlmLTFmMDgtNDY3MC04ZmE2LTJiOGI3OGUwNWMzMSIsImFjciI6IjAiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiY3JlYXRlLXJlYWxtIiwiZGVmYXVsdC1yb2xlcy1tYXN0ZXIiLCJvZmZsaW5lX2FjY2VzcyIsImFkbWluIiwidW1hX2F1dGhvcml6YXRpb24iLCJ1c2VyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWRlcy1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LWlkZW50aXR5LXByb3ZpZGVycyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwiZGVtby1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LXJlYWxtIiwidmlldy1pZGVudGl0eS1wcm92aWRlcnMiLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwiZHVtbXktc2VydmljZS1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LXJlYWxtIiwidmlldy1pZGVudGl0eS1wcm92aWRlcnMiLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwibWFzdGVyLXJlYWxtIjp7InJvbGVzIjpbInZpZXctaWRlbnRpdHktcHJvdmlkZXJzIiwidmlldy1yZWFsbSIsIm1hbmFnZS1pZGVudGl0eS1wcm92aWRlcnMiLCJpbXBlcnNvbmF0aW9uIiwiY3JlYXRlLWNsaWVudCIsIm1hbmFnZS11c2VycyIsInF1ZXJ5LXJlYWxtcyIsInZpZXctYXV0aG9yaXphdGlvbiIsInF1ZXJ5LWNsaWVudHMiLCJxdWVyeS11c2VycyIsIm1hbmFnZS1ldmVudHMiLCJtYW5hZ2UtcmVhbG0iLCJ2aWV3LWV2ZW50cyIsInZpZXctdXNlcnMiLCJ2aWV3LWNsaWVudHMiLCJtYW5hZ2UtYXV0aG9yaXphdGlvbiIsIm1hbmFnZS1jbGllbnRzIiwicXVlcnktZ3JvdXBzIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX0sImVvZXBjYS1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LWlkZW50aXR5LXByb3ZpZGVycyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfX0sInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJzaWQiOiJlZjRlMDk5Zi0xZjA4LTQ2NzAtOGZhNi0yYjhiNzhlMDVjMzEiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInByZWZlcnJlZF91c2VybmFtZSI6ImFkbWluIn0.FK6DhVzpCRFmef2acD2Hmc149e1GTOCGz13dZA828crFbG8j4uhpkoNpiZqdyOPmDtMQ-OebNfjTAUaOt2sS1FmEIBgb9IddcpHKNJOquRjdzQNsX09bX8pFUq1haGwKh6_QmABNOBcT-kQNDSZO-aq7-8FoO9PYa0GWvBRcbcx0W_ngyb7xHglaZTElzcDPBcUTW6llVTTTFygn55smwdxTZ7-tEsMVGM5gNuHwJyLB51HI5KDWrwgUm1hqhhRzvcoutDEAB_HSEXGNNeF7fjP9Qx6q04b7fKOTtnIlXsu3oYW4va9y754llMSJ7w8U-y7yI6Tm2UdNMdYqju7hAA \\\n  -c admin-cli \\\n  --id=myservice-gatekeeper \\\n  --name=\"MyService Gatekeeper\" \\\n  --secret=changeme \\\n  --description=\"Client to be used by MyService Gatekeeper\" \\\n  --resource=\"Eric space\" --uris=/eric/* --users=eric \\\n  --resource=\"Alice space\" --uris=/alice/* --users=alice \\\n  --resource=\"Admin space\" --uris=/admin/* --roles=admin\n

"},{"location":"eoepca/identity-service/#using-identity-api","title":"Using Identity API","text":"

Also, an API was developed to interact more easily with the Keycloak API, that allows client, resource, policies and permissions management.

The API documentation can be found in its Swagger UI at the service endpoint - https://identity-api.192-168-49-2.nip.io/docs.

The Identity API is best used in combination with the eoepca-portal test aide, which can be used to establish a login sesssion in the browser to the benefit of the Identity API swagger UI. See section EOEPCA Portal for details regarding deployment/configuration of the eoepca-portal.

"},{"location":"eoepca/identity-service/#token-lifespans","title":"Token Lifespans","text":"

By default the Access Token Lifespan is 1 minute. With the current ADES (zoo-dru) implementation this presents a problem - since the access_token that is provided to the process execute request will (most likely) have expired by the time the ADES attempts to use the access_token in its call to the Workspace API to register the processing outputs. The lifespan of the token must outlive the duration of the processing execution - which we must assume can take a long time.

To avoid this potential problem, the Keycloak Admin web console can be used to increase this token lifespan.

Thus, the following settings are recommended to be updated following deployment\u2026

  • SSO Session Settings /admin/master/console/#/master/realm-settings/sessions
  • SSO Session Idle: 1 day
  • SSO Session Max: 1 day
  • Tokens /admin/master/console/#/master/realm-settings/tokens
  • Access Token Lifespan: 1 day
"},{"location":"eoepca/identity-service/#additional-information","title":"Additional Information","text":"

Additional information regarding the Identity Service can be found at:

  • Helm Chart
  • GitHub Repository
"},{"location":"eoepca/login-service/","title":"Login Service","text":"

The Login Service provides the platform Authorization Server for authenticated user identity and request authorization.

"},{"location":"eoepca/login-service/#helm-chart","title":"Helm Chart","text":"

The Login Service is deployed via the login-service helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the login-service chart.

helm install --version 1.2.8 --values login-service-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  login-service login-service\n
"},{"location":"eoepca/login-service/#values","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io
  • IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2
  • Kubernetes namespace for the login-service components
  • Initial password for the admin user Note that the password must meet the complexity: at least 6 characters and include one uppercase letter, one lowercase letter, one digit, and one special character
  • Name of Persistent Volume Claim for login-service persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.
  • TLS Certificate Provider, e.g. letsencrypt-production

Example login-service-values.yaml\u2026 

global:\n  domain: auth.192-168-49-2.nip.io\n  nginxIp: 192.168.49.2\n  namespace: um\nvolumeClaim:\n  name: eoepca-userman-pvc\n  create: false\nconfig:\n  domain: auth.192-168-49-2.nip.io\n  adminPass: Chang3me!\n  ldapPass: Chang3me!\n  volumeClaim:\n    name: eoepca-userman-pvc\nopendj:\n  volumeClaim:\n    name: eoepca-userman-pvc\n  resources:\n    requests:\n      cpu: 100m\n      memory: 300Mi\noxauth:\n  volumeClaim:\n    name: eoepca-userman-pvc\n  resources:\n    requests:\n      cpu: 100m\n      memory: 1000Mi\noxtrust:\n  volumeClaim:\n    name: eoepca-userman-pvc\n  resources: \n    requests:\n      cpu: 100m\n      memory: 1500Mi\noxpassport:\n  resources:\n    requests:\n      cpu: 100m\n      memory: 100Mi\nnginx:\n  ingress:\n    annotations:\n      cert-manager.io/cluster-issuer: letsencrypt-production\n    hosts:\n      - auth.192-168-49-2.nip.io\n    tls:\n      - hosts:\n          - auth.192-168-49-2.nip.io\n        secretName: login-service-tls\n

Note

The resources: above have been limited for the benefit of a minikube deployment. For a production deployment the values should be tuned (upwards) according to operational needs.

"},{"location":"eoepca/login-service/#post-deployment-manual-steps","title":"Post-deployment Manual Steps","text":"

The deployment of the Login Service has been designed, as far as possible, to automate the configuration. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026

  • Configure UMA Resource Lifetime
  • Configure Operator user
"},{"location":"eoepca/login-service/#uma-resource-lifetime","title":"UMA Resource Lifetime","text":"

The Login Service maintains a background service that \u2018cleans\u2019 UMA resources that are older than aa certain age - by default 30 days (2592000 secs). This lifetime does not fit the approach we are adopting, and so we must update this lifetime value to avoid the unexpected removal of UMA resources that would cause unexpected failures in policy enforcement.

  • In a browser, navigate to the Login Service (Gluu) - https://auth.192-168-49-2.nip.io/ - and login as the admin user
  • Open Configuration -> JSON Configuration -> OxAuth Configuration
  • Search for the setting umaResourceLifetime
  • Update the values of umaResourceLifetime to 2147483647
  • Select to Save Configuration
  • Restart the oxauth deployment\u2026 
    kubectl -n um rollout restart deploy/login-service-oxauth\n
"},{"location":"eoepca/login-service/#configure-operator-user","title":"Configure Operator user","text":"

The default resource protection establishes policy in which \u2018operator\u2019 privilege is required for some services, such as the Workspace API. Thus, we need to configure a user with this privilege. For convenience we add this attribute to the built-in admin user - but alternatively you may choose to create a new user for this role.

  • In a browser, navigate to the Login Service (Gluu) - https://auth.192-168-49-2.nip.io/ - and login as the admin user
  • Select Users -> Manage People and search for user admin
  • For user admin select Available User Claims -> gluuCustomPerson
  • Select Is Operator and ensure the value is set True
  • Select Update to confirm
"},{"location":"eoepca/login-service/#login-service-usage","title":"Login Service Usage","text":"

Once the deployment has been completed successfully, the Login Service is accessed at the endpoint https://auth.192-168-49-2.nip.io/, configured by your domain - e.g. https://auth.192-168-49-2.nip.io/.

Login as the admin user with the credentials configured in the helm values - ref. adminPass / ldapPass.

Typical first actions to undertake through the Gluu web interface include creation of users and clients.

"},{"location":"eoepca/login-service/#additional-information","title":"Additional Information","text":"

Additional information regarding the Login Service can be found at:

  • Helm Chart
  • Wiki
  • GitHub Repository
"},{"location":"eoepca/pdp/","title":"Policy Decision Point","text":"

The Policy Decision Point (PDP) provides the platform policy database and associated service for access policy decision requests.

"},{"location":"eoepca/pdp/#helm-chart","title":"Helm Chart","text":"

The PDP is deployed via the pdp-engine helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the pdp-engine chart.

helm install --version 1.1.12 --values pdp-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  pdp pdp-engine\n
"},{"location":"eoepca/pdp/#values","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io
  • IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2
  • Name of Persistent Volume Claim for pdp-engine persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.

Example pdp-values.yaml\u2026 

global:\n  nginxIp: 192.168.49.2\n  domain: auth.192-168-49-2.nip.io\nvolumeClaim:\n  name: eoepca-userman-pvc\n  create: false\n

"},{"location":"eoepca/pdp/#additional-information","title":"Additional Information","text":"

Additional information regarding the PDP can be found at:

  • Helm Chart
  • Wiki
  • GitHub Repository
"},{"location":"eoepca/persistence/","title":"Persistence","text":""},{"location":"eoepca/persistence/#overview","title":"Overview","text":"

The EOEPCA building-blocks rely upon Kubernetes Persistent Volumes for their component persistence. Components integrate with the storage provided in the cluster by means of configurable Persistent Volume Claims and/or dynamic Storage Class that are specfied as values at time of deployment. Some components require storage of type ReadWriteMany - which, for a multi-node cluster, implies a network-based storage solution.

Note

Local CLuster Storage For the purposes of the Scripted Deployment, the default Storage Class included with the local Kubernetes distribution can be used for all storage concerns - e.g. standard for minikube which provides the ReadWriteMany persistence that is required by the ADES.

"},{"location":"eoepca/persistence/#readwritemany-storage","title":"ReadWriteMany Storage","text":"

For the EOEPCA development deployment, an NFS server has been established to provide the persistence layer for ReadWriteMany storage.

"},{"location":"eoepca/persistence/#pre-defined-persistent-volume-claims","title":"Pre-defined Persistent Volume Claims","text":"

The EOEPCA development deployment establishes the following pre-defined Persistent Volume Claims, to provide a simple storage architecture that is organised around the \u2018domain areas\u2019 into which the Reference Implementation is split.

  • Resource Managment (resman) - persistentvolumeclaim/eoepca-resman-pvc
  • Processing & Chaining (proc) - persistentvolumeclaim/eoepca-proc-pvc
  • User Management (userman) - persistentvolumeclaim/eoepca-userman-pvc

NOTE that this is offered only as an example thay suits the approach of the development team. Each building-block has configuration through which its persistence (PV/PVC) can be configured according the needs of the deployment.

The following Kubernetes yaml provides an example of provisioning such domain-specific PersistentVolumeClaims within the cluster - in this case using the minikube built-in storage-class standard for dynamic provisioning\u2026

---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: eoepca-proc-pvc\n  namespace: proc\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: standard\n  resources:\n    requests:\n      storage: 5Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: eoepca-resman-pvc\n  namespace: rm\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: standard\n  resources:\n    requests:\n      storage: 5Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: eoepca-userman-pvc\n  namespace: um\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: standard\n  resources:\n    requests:\n      storage: 5Gi\n

Once established, these PersistentVolumeClaims are then referenced within the deployment configurations of the building-blocks.

"},{"location":"eoepca/persistence/#dynamic-readwritemany-storage-provisioning","title":"Dynamic ReadWriteMany Storage Provisioning","text":"

In addition to the pre-defined PV/PVCs, the EOEPCA Reference Implementation also defines NFS-based storage classes for dynamic storage provisioning:

  • managed-nfs-storage With a Reclaim Policy of Delete.
  • managed-nfs-storage-retain With a Reclaim Policy of Retain.

The building-blocks simply reference the required Storage Class in their volume specifications, to receive a Persistent Volume Claim that is dynamically provisioned at deployment time.

This is acheived through the nfs-provisioner helm chart, with the following typical configurations\u2026

Reclaim Policy Delete\u2026 

provisionerName: nfs-storage\nstorageClass:\n  name: managed-nfs-storage\n  create: true\n  reclaimPolicy: Delete\n  archiveOnDelete: false\n  allowVolumeExpansion: true\nnfs:\n  server: \"<your-nfs-ip-address-here>\"\n  path: /data/dynamic  # your NFS server path here\n

Reclaim Policy Retain\u2026 

provisionerName: nfs-storage-retain\nstorageClass:\n  name: managed-nfs-storage-retain\n  create: true\n  reclaimPolicy: Retain\n  allowVolumeExpansion: true\nnfs:\n  server: \"<your-nfs-ip-address-here>\"\n  path: /data/dynamic  # your NFS server path here\n

"},{"location":"eoepca/persistence/#clustered-storage-solutions","title":"Clustered Storage Solutions","text":"

Clustered storage approaches offer an alternative to NFS. Clustered Storage provides a network-attached storage through a set of commodity hosts whose storage is aggregated to form a distributed file-system. Capacity is scaled by adding additional nodes or adding additional storage to the existing nodes. In the context of a multi-node Kubernetes cluster, then it is typical that the same commodity nodes provide both the cluster members and storage resources, i.e. the clustered storage is spread across the Kubernetes worker nodes.

Candidate clustered storage solutions include:

  • GlusterFS GlusterFS is deployed as an operating system service across each node participating in the storage solution. Thus, with GlusterFS, the distributed storage nodes do not need to be one-and-the-same with the compute (cluster) nodes \u2013 although this may preferably be the case.
  • Longhorn Longhorn offers a solution that is similar to that of GlusterFS, except that Longhorn is \u2018cloud-native\u2019 in that its service layer deploys within the Kubernetes cluster itself. Thus, the storage nodes are also the cluster compute nodes by design.

All things being equal, Longhorn is recommended as the best approach for Kubernetes clusters.

"},{"location":"eoepca/registration-api/","title":"Registration API","text":"

The Registration API provides a REST API through which resources can be registered with both the Resource Catalogue and (as applicable) with the Data Access services.

"},{"location":"eoepca/registration-api/#helm-chart","title":"Helm Chart","text":"

The Registration API is deployed via the rm-registration-api helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the rm-registration-api chart.

helm install --version 1.4.0 --values registration-api-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  registration-api rm-registration-api\n
"},{"location":"eoepca/registration-api/#values","title":"Values","text":"

The Registration API supports many values to configure the service - as described in the Values section of the chart README.

Typically, values for the following attributes may be specified:

  • The fully-qualified public URL for the service
  • (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Registration API will not be protected by the identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.
  • Values for integration with the workspace-api and data-access services

Example registration-api-values.yaml\u2026

fullnameOverride: registration-api\n\ningress:\n  enabled: false\n  hosts:\n    - host: registration-api-open.192-168-49-2.nip.io\n      paths: [\"/\"]\n  tls:\n    - hosts:\n        - registration-api-open.192-168-49-2.nip.io\n      secretName: registration-api-tls\n\n# some values for the workspace API\nworkspaceK8sNamespace: rm\nredisServiceName: \"data-access-redis-master\"\n
"},{"location":"eoepca/registration-api/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the registration-api service to provide access authorization decisions

"},{"location":"eoepca/registration-api/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install registration-api-protection identity-gatekeeper -f registration-api-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"rm\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the registration-api - in particular the specific ingress requirements for the registration-api backend service\u2026

Example registration-api-protection-values.yaml\u2026

fullnameOverride: registration-api-protection\nconfig:\n  client-id: registration-api\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: registration-api.192-168-49-2.nip.io\n  name: registration-api\n  port:\n    number: 8080\nsecrets:\n  # Values for secret 'registration-api-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n    nginx.ingress.kubernetes.io/enable-cors: \"true\"\n  serverSnippets:\n    custom: |-\n      # Open access...\n      location ~ ^/ {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n
"},{"location":"eoepca/registration-api/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: registration-api above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=registration-api \\\n  --name=\"Registration API Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Registration API Gatekeeper\"\n
"},{"location":"eoepca/registration-api/#additional-information","title":"Additional Information","text":"

Additional information regarding the Registration API can be found at:

  • Helm Chart
  • GitHub Repository
"},{"location":"eoepca/resource-catalogue/","title":"Resource Catalogue","text":"

The Resource Catalogue provides a standards-based EO metadata catalogue that includes support for OGC CSW / API Records, STAC and OpenSearch.

"},{"location":"eoepca/resource-catalogue/#helm-chart","title":"Helm Chart","text":"

The Resource Catalogue is deployed via the rm-resource-catalogue helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the rm-resource-catalogue chart.

helm install --version 1.4.0 --values resource-catalogue-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  resource-catalogue rm-resource-catalogue\n
"},{"location":"eoepca/resource-catalogue/#values","title":"Values","text":"

The Resource Catalogue supports many values to configure the service - as described in the Values section of the chart README.

Typically, values for the following attributes may be specified:

  • The fully-qualified public URL for the service
  • Dynamic provisioning StorageClass for database persistence
  • (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Resource Catalogue will not be protected by the identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.
  • Metadata describing the Catalogue instance
  • Tuning configuration for PostgreSQL - see values db.config.XXX.

Example resource-catalogue-values.yaml\u2026

global:\n  namespace: rm\n# For protected access disable this ingress, and rely upon the identity-gatekeeper\n# for ingress with protection.\ningress:\n  # Enabled for unprotected 'open' access to the resource-catalogue.\n  enabled: true\n  name: resource-catalogue\n  host: resource-catalogue.192-168-49-2.nip.io\n  tls_host: resource-catalogue.192-168-49-2.nip.io\n  tls_secret_name: resource-catalogue-tls\n  annotations:\n    cert-manager.io/cluster-issuer: letsencrypt-production\ndb:\n  volume_storage_type: standard\n  # config:\n  #   enabled: true\n  #   shared_buffers: 2GB\n  #   effective_cache_size: 6GB\n  #   maintenance_work_mem: 512MB\n  #   checkpoint_completion_target: 0.9\n  #   wal_buffers: 16MB\n  #   default_statistics_target: 100\n  #   random_page_cost: 4\n  #   work_mem: 4MB\n  #   cpu_tuple_cost: 0.4\npycsw:\n  config:\n    server:\n      url: https://resource-catalogue.192-168-49-2.nip.io/\n    manager:\n      transactions: \"true\"\n      allowed_ips: \"*\"\n

Note

The above example values enable transactions (write-access) to the catalogue from any IP address. This is convenient for testing/demonstration of the capability, but should be disbaled or restricted for operational deployments.

"},{"location":"eoepca/resource-catalogue/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the resource-catalogue service to provide access authorization decisions

"},{"location":"eoepca/resource-catalogue/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install resource-catalogue-protection identity-gatekeeper -f resource-catalogue-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"rm\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the resource-catalogue - in particular the specific ingress requirements for the resource-catalogue-service\u2026

Example resource-catalogue-protection-values.yaml\u2026

fullnameOverride: resource-catalogue-protection\nconfig:\n  client-id: resource-catalogue\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: resource-catalogue.192-168-49-2.nip.io\n  name: resource-catalogue-service\n  port:\n    number: 80\nsecrets:\n  # Values for secret 'resource-catalogue-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n    nginx.ingress.kubernetes.io/enable-cors: \"true\"\n  serverSnippets:\n    custom: |-\n      # Open access...\n      location ~ ^/ {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n
"},{"location":"eoepca/resource-catalogue/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: resource-catalogue above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=resource-catalogue \\\n  --name=\"Resource Catalogue Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Resource Catalogue Gatekeeper\"\n
"},{"location":"eoepca/resource-catalogue/#resource-catalogue-usage","title":"Resource Catalogue Usage","text":"

The Resource Catalogue is initially populated during the initialisation of the Data Access service. See section Data-layer Configuration.

The Resource Catalogue is accessed at the endpoint https://resource-catalogue.192-168-49-2.nip.io/, configured by your domain - e.g. https://resource-catalogue.192-168-49-2.nip.io/.

"},{"location":"eoepca/resource-catalogue/#loading-records","title":"Loading Records","text":"

As described in the pycsw documentation, ISO XML records can be loaded into the resource-catalogue using the pycsw-admin.py admin utility\u2026

pycsw-admin.py load_records -c /path/to/cfg -p /path/to/records\n

The /path/to/records can either be a single metadata file, or a directory containing multiple metadata files.

This is most easily achieved via connection to the pycsw pod, which includes the pycsw-admin.py utility and the pycsw configuration file at /etc/pycsw/pycsw.cfg\u2026

kubectl -n rm cp \"<metadata-file-or-directory>\" \"<pycsw-pod-name>\":/tmp/metadata\nkubectl -n rm exec -i \"<pycsw-pod-name>\" -- pycsw-admin.py load-records -c /etc/pycsw/pycsw.cfg -p /tmp/metadata\n

The name of the pycsw pod can be obtained using kubectl\u2026

kubectl -n rm get pod --selector='io.kompose.service=pycsw' --output=jsonpath={.items[0].metadata.name}\n

To facilitate the loading of records via the pycsw pod, a helper script load-records has been provided in the git repository that hosts this document\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n./deploy/bin/load-records \"<metadata-file-or-directory>\"\n

The helper script identifies the pycsw pod, copies the metadata files to the pod, and runs pycsw-admin.py load-records within the pod to load the records.

"},{"location":"eoepca/resource-catalogue/#additional-information","title":"Additional Information","text":"

Additional information regarding the Resource Catalogue can be found at:

  • Helm Chart
  • pycsw Documentation
  • GitHub Repository
"},{"location":"eoepca/resource-protection-gluu/","title":"Resource Protection (Gluu)","text":"

EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes:

  • Login Service (Authorization Server)
  • Policy Decision Point (PDP)
  • Policy Enforcement Point (PEP)

Building Blocks that act as a Resource Server are individually protected by a Policy Enforcement Point (PEP). The PEP enforces the authorization decision in collaboration with the Login Service and Policy Decision Point (PDP).

The PEP expects to interface to a client (user agent, e.g. browser) using User Managed Access (UMA) flows. It is not typical for a client to support UMA flows, and so the PEP can be deployed with a companion UMA User Agent component that interfaces between the client and the PEP, and performs the UMA Flow on behalf of the client.

The Resource Guard is a \u2018convenience\u2019 component that deploys the PEP & UMA User Agent as a cooperating pair.

The Resource Guard \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, the Resource Guard deploys with an Ingress specification that:

  • Configures the auth_request module to defer access authorization to the uma-user-agent service
  • Configures the ingress rules (host/path) for the target Resource Server
"},{"location":"eoepca/resource-protection-gluu/#helm-chart","title":"Helm Chart","text":"

The Resource Guard is deployed via the resource-guard helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the resource-guard chart.

It is expected to deploy multiple instances of the Resource Guard chart, one for each Resource Server to be protected.

helm install --version 1.3.1 --values myservice-guard-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  myservice-guard resource-guard\n
"},{"location":"eoepca/resource-protection-gluu/#values","title":"Values","text":"

The helm chart is deployed with values that are passed through to the subcharts for the pep-engine and uma-user-agent. Typical values to be specified include:

  • Host/domain details for the Login Service and PDP, e.g. auth.192-168-49-2.nip.io
  • IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2
  • Name of Persistent Volume Claim for pep-engine persistence, e.g. myservice-pep-pvc
  • TLS Certificate Provider, e.g. letsencrypt-production
  • Optional specification of default resources with which to initialise the policy database for the component
  • Ingress rules definition for reverse-proxy to the target Resource Server
  • Name of Secret that contains the client credentials used by the uma-user-agent to interface with the Login Service. See section Client Secret below

Example myservice-guard-values.yaml\u2026 

#---------------------------------------------------------------------------\n# Global values\n#---------------------------------------------------------------------------\nglobal:\n  context: myservice\n  domain: 192-168-49-2.nip.io\n  nginxIp: 192.168.49.2\n  certManager:\n    clusterIssuer: letsencrypt-production\n#---------------------------------------------------------------------------\n# PEP values\n#---------------------------------------------------------------------------\npep-engine:\n  configMap:\n    asHostname: auth\n    pdpHostname: auth\n  customDefaultResources:\n  - name: \"Eric's space\"\n    description: \"Protected Access for eric to his space in myservice\"\n    resource_uri: \"/ericspace\"\n    scopes: []\n    default_owner: \"d3688daa-385d-45b0-8e04-2062e3e2cd86\"\n  volumeClaim:\n    name: myservice-pep-pvc\n    create: false\n#---------------------------------------------------------------------------\n# UMA User Agent values\n#---------------------------------------------------------------------------\numa-user-agent:\n  nginxIntegration:\n    enabled: true\n    hosts:\n      - host: myservice\n        paths:\n          - path: /(.*)\n            service:\n              name: myservice\n              port: 80\n          - path: /(doc.*)\n            service:\n              name: myservice-docs\n              port: 80\n    annotations:\n      nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n      nginx.ingress.kubernetes.io/enable-cors: \"true\"\n      nginx.ingress.kubernetes.io/rewrite-target: /$1\n  client:\n    credentialsSecretName: \"myservice-agent\"\n  logging:\n    level: \"debug\"\n  unauthorizedResponse: 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"'\n#---------------------------------------------------------------------------\n# END values\n#---------------------------------------------------------------------------\n

"},{"location":"eoepca/resource-protection-gluu/#client-credentials","title":"Client Credentials","text":"

The uma-user-agent requires Client Credentials for its interactions with the login-service. The uma-user-agent expects to read these credentials from the file client.yaml, in the form\u2026

client-id: <my-client-id>\nclient-secret: <my-secret>\n
"},{"location":"eoepca/resource-protection-gluu/#client-registration","title":"Client Registration","text":"

To obtain the Client Credentials required by the uma-user-agent it is necessary to register a client with the login-service, or use the credentials for an existing client.

A helper script is provided to register a basic client and obtain the required credentials. The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The register-client helper script requires some command-line arguments\u2026

Usage:\n  register_client <authorization-server-hostname> <client-name> [<redirect-uri> [<logout-uri>]]\n

For example\u2026

./deploy/bin/register-client auth.192-168-49-2.nip.io myclient\n\nINFO: Preparing docker image... [done]\nClient successfully registered.\nMake a note of the credentials:\nclient-id: a98ba66e-e876-46e1-8619-5e130a38d1a4\nclient-secret: 73914cfc-c7dd-4b54-8807-ce17c3645558\n

Or to register OIDC redirect URLs\u2026 

./deploy/bin/register-client auth.192-168-49-2.nip.io myclient https://portal.192-168-49-2.nip.io/oidc/callback/ https://portal.192-168-49-2.nip.io/logout\n

The script writes the \u2018client credentials\u2019 to stdout - in the expected YAML configuration file format - which can be redirected to file\u2026 

./deploy/bin/register-client auth.192-168-49-2.nip.io myclient | tee client.yaml\n
\u2026writes the client credentials to the file client.yaml.

NOTE that the register-client helper relies upon docker to build and run the script.

"},{"location":"eoepca/resource-protection-gluu/#client-secret","title":"Client Secret","text":"

The client.yaml configuration file is made available via a Kubernetes Secret\u2026

kubectl -n myservice-ns create secret generic myservice-agent \\\n  --from-file=client.yaml \\\n  --dry-run=client -o yaml \\\n  > myservice-agent-secret.yaml\n
apiVersion: v1\nkind: Secret\nmetadata:\n  name: myservice-agent\n  namespace: myservice-ns\ndata:\n  client.yaml: Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4\n

The resource-guard deployment is configured with the name of the Secret through the helm chart value client.credentialsSecretName.

"},{"location":"eoepca/resource-protection-gluu/#user-id-token","title":"User ID Token","text":"

As described in the README for the Resource Guard, it is necessary for a request to a protected resource to provide the User ID Token in the request header.

"},{"location":"eoepca/resource-protection-gluu/#obtaining-the-user-id-token","title":"Obtaining the User ID Token","text":"

In the simple case of a user with username/password held within the Login Service, the User ID Token can be obtained as follows:

curl --location --request POST 'https://auth.192-168-49-2.nip.io/oxauth/restv1/token' \\\n--header 'Cache-Control: no-cache' \\\n--header 'Content-Type: application/x-www-form-urlencoded' \\\n--data-urlencode 'scope=openid user_name is_operator' \\\n--data-urlencode 'grant_type=password' \\\n--data-urlencode 'username=<username>' \\\n--data-urlencode 'password=<password>' \\\n--data-urlencode 'client_id=<client-id>' \\\n--data-urlencode 'client_secret=<client-password>'\n

The User ID Token is included in the id_token field of the json response.

Alternatively, OAuth/OIDC flows can be followed to authenticate via external identity providers.

"},{"location":"eoepca/resource-protection-gluu/#user-id-token-in-http-requests","title":"User ID Token in HTTP requests","text":"

The Resource Guard protection supports presentation of the User ID Token via the following HTTP request headers (in order of priority)\u2026

  • Authorization header as a bearer token - in the form: Authorization: Bearer <token>
  • X-User-Id header
  • Cookie: auth_user_id=<token>

    Note that the name of the cookie is configurable

"},{"location":"eoepca/resource-protection-gluu/#additional-information","title":"Additional Information","text":"

Additional information regarding the Resource Guard can be found at:

  • Helm Chart
  • README
  • GitHub Repository:
    • pep-engine
    • uma-user-agent
"},{"location":"eoepca/resource-protection-keycloak/","title":"Resource Protection (Keycloak)","text":"

EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes:

  • Keycloak - Identity Service (Authorization Server)
  • Gatekeeper - Policy Enforcement

Building Blocks that act as a Resource Server are individually protected by a dedicated Gatekeeper instance that enforces the authorization decision in collaboration with the Identity Service (Keycloak).

Gatekeeper \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, Gatekeeper deploys with an Ingress specification that:

  • Configures the auth_request module to defer access authorization to the gatekeeper service
  • Configures the ingress rules (host/path) for the target Resource Server
"},{"location":"eoepca/resource-protection-keycloak/#helm-chart","title":"Helm Chart","text":"

Each Gatekeeper is deployed via the identity-gatekeeper helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values - the full set of available values can be seen at https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/values.yaml.

It is expected to deploy multiple instances of the Gatekeeper chart, one for each Resource Server to be protected.

helm install --version 1.0.10 --values myservice-gatekeeper-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  myservice-protection identity-gatekeeper\n
"},{"location":"eoepca/resource-protection-keycloak/#values","title":"Values","text":"

The helm chart is deployed with values that customise the service for the specific needs of the resource-server under protection and the deployment target platform. Typical values to be specified include:

  • Host/domain details for the Keycloak Identity Service, e.g. keycloak.192-168-49-2.nip.io
  • Credentials for the Keycloak client to be used by Gatekeeper (ideally via secret)
  • TLS Certificate Provider, e.g. letsencrypt-production
  • Ingress rules definition for reverse-proxy to the target Resource Server

Example myservice-protection-values.yaml\u2026 

nameOverride: myservice-protection\nconfig:\n  client-id: myservice\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: myservice.192-168-49-2.nip.io\n  name: myservice\n  port:\n    number: 80\nsecrets:\n  # Values for secret 'myservice-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  serverSnippets:\n    custom: |-\n      # Open access to some endpoints, including Swagger UI\n      location ~ ^/(docs|openapi.json|probe) {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n

"},{"location":"eoepca/resource-protection-keycloak/#client-credentials","title":"Client Credentials","text":"

Gatekeeper requires Client Credentials for its interactions with the Keycloak identity-service. These credentials must be supplied by the secret named <myservice>-protection. The secret can be created directly by the helm chart - via the values secrets.clientSecret and secrets.encryptionKey - or perhaps more securely the secret can be created independently (e.g. via a SealedSecret).

"},{"location":"eoepca/resource-protection-keycloak/#client-registration","title":"Client Registration","text":"

The Keycloak client can be created directly in the Keycloak admin console - e.g. via https://keycloak.192-168-49-2.nip.io/admin.

As an aide there is a helper script create-client. The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The create-client helper script requires some command-line arguments\u2026

$ ./deploy/bin/create-client -h\n\nAdd a client with protected resources.\ncreate-client [-h] [-a] [-i] [-u] [-p] [-c] [-s] [-t | --token t] [-r] --id id [--name name] (--secret secret | --public) [--default] [--authenticated] [--resource name] [--uris u1,u2] [--scopes s1,s2] [--users u1,u2] [--roles r1,r2]\n\nwhere:\n    -h                    show help message\n    -a                    authorization server url - e.g. https://keycloak.192-168-49-2.nip.io\n    -i                    identity-api server url - e.g. https://identity-api.192-168-49-2.nip.io\n    -u                    username used for authentication\n    -p                    password used for authentication\n    -c                    client id (of the bootstrap client used in the create request)\n    -s                    client secret (of the bootstrap client used in the create request)\n    -t or --token         access token used for authentication\n    -r                    realm\n    --id                  client id (of the created client)\n    --name                client name (of the created client)\n    --secret              client secret (of the created client)\n    --public              public client (no client secret)\n    --default             add default resource - /* authenticated\n    --authenticated       allow access to the resource only when authenticated\n    --resource            resource name\n    --uris                resource uris - separated by comma (,)\n    --scopes              resource scopes - separated by comma (,)\n    --users               user names with access to the resource - separated by comma (,)\n    --roles               role names with access to the resource - separated by comma (,)\n

For example\u2026

./deploy/bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=myservice \\\n  --name=\"Gatekeeper for myservice\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Gatekeeper for myservice\" \\\n  --resource=\"eric\" --uris='/eric/*' --scopes=view --users=\"eric\" \\\n  --resource=\"bob\" --uris='/bob/*' --scopes=view --users=\"bob\" \\\n  --resource=\"alice\" --uris='/alice/*' --scopes=view --users=\"alice\"\n
"},{"location":"eoepca/resource-protection-keycloak/#user-tokens","title":"User Tokens","text":"

Requests to resource server endpoints that are protected by Gatekeeper must carry an Access Token that has been obtained on behalf of the requesting user. The access_token is carried in the request header\u2026

Authorization: Bearer <access_token>\n

The Access Token for a given user can be obtained with a call to the token endpoint of the Keycloak Identity Service - supplying the credentials for the user and the pre-registered client\u2026

curl -L -X POST 'https://keycloak.192-168-49-2.nip.io/realms/master/protocol/openid-connect/token' \\\n  -H 'Cache-Control: no-cache' \\\n  -H 'Content-Type: application/x-www-form-urlencoded' \\\n  --data-urlencode 'scope=openid profile email' \\\n  --data-urlencode 'grant_type=password' \\\n  --data-urlencode 'username=<username>' \\\n  --data-urlencode 'password=<password>' \\\n  --data-urlencode 'client_id=admin-cli'\n

A json response is returned, in which the field access_token provides the Access Token for the specified <username>.

"},{"location":"eoepca/resource-protection-keycloak/#additional-information","title":"Additional Information","text":"

Additional information regarding the Gatekeeper can be found at:

  • Container Image
  • Helm Chart
  • GitHub Repository
"},{"location":"eoepca/user-profile/","title":"User Profile","text":"

The User Profile represents the user\u2019s \u2018account\u2019 within the platform.

"},{"location":"eoepca/user-profile/#helm-chart","title":"Helm Chart","text":"

The User Profile is deployed via the user-profile helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the user-profile chart.

helm install --version 1.1.12 --values user-profile-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  user-profile user-profile\n
"},{"location":"eoepca/user-profile/#values","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io
  • IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2
  • Name of Persistent Volume Claim for user-profile persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.

Example user-profile-values.yaml\u2026 

global:\n  domain: auth.192-168-49-2.nip.io\n  nginxIp: 192.168.49.2\nvolumeClaim:\n  name: eoepca-userman-pvc\n  create: false\n

"},{"location":"eoepca/user-profile/#user-profile-usage","title":"User Profile Usage","text":"

The User Profile is accessed through the /web_ui path of the Login Service, e.g. http://auth.kube.guide.eoepca.org/web_ui.

"},{"location":"eoepca/user-profile/#additional-information","title":"Additional Information","text":"

Additional information regarding the User Profile can be found at:

  • Helm Chart
  • Wiki
  • GitHub Repository
"},{"location":"eoepca/workspace/","title":"Workspace","text":"

The Workspace provides protected user resource management that includes dedicated storage and services for resource discovery and access.

"},{"location":"eoepca/workspace/#workspace-api","title":"Workspace API","text":"

The Workspace API provides a REST service through which user workspaces can be created, interrogated, managed and deleted.

"},{"location":"eoepca/workspace/#helm-chart","title":"Helm Chart","text":"

The Workspace API is deployed via the rm-workspace-api helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the um-workspace-api chart.

helm install --version 1.4.2 --values workspace-api-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  workspace-api rm-workspace-api\n
"},{"location":"eoepca/workspace/#values","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • The fully-qualified public URL for the service
  • (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Workspace API will not be protected by the identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.
  • Prefix for user projects in OpenStack
  • Details for underlying S3 object storage service
  • Identification of secret that provides the client credentials for resource protection
  • Whether flux components should be installed - otherwise they must already be present - Flux Dependency
  • Name of the ConfigMap for user workspace templates - See User Workspace Templates

Example workspace-api-values.yaml\u2026 

fullnameOverride: workspace-api\ningress:\n  enabled: true\n  annotations:\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    kubernetes.io/ingress.class: nginx\n    nginx.ingress.kubernetes.io/enable-cors: \"true\"\n    nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n  hosts:\n    - host: workspace-api-open.192-168-49-2.nip.io\n      paths: [\"/\"]\n  tls:\n    - hosts:\n        - workspace-api-open.192-168-49-2.nip.io\n      secretName: workspace-api-open-tls\nfluxHelmOperator:\n  enabled: true\nprefixForName: \"ws\"\nworkspaceSecretName: \"bucket\"\nnamespaceForBucketResource: \"rm\"\ns3Endpoint: \"https://minio.192-168-49-2.nip.io\"\ns3Region: \"RegionOne\"\nharborUrl: \"https://harbor.192-168-49-2.nip.io\"\nharborUsername: \"admin\"\nharborPasswordSecretName: \"harbor\"\nworkspaceChartsConfigMap: \"workspace-charts\"\nbucketEndpointUrl: \"http://minio-bucket-api:8080/bucket\"\nkeycloakIntegration:\n  enabled: true\n  keycloakUrl: \"https://keycloak.192-168-49-2.nip.io\"\n  realm: \"master\"\n  identityApiUrl: \"https://identity-api.192-168-49-2.nip.io\"\n  workspaceApiIamClientId: \"workspace-api\"\n  defaultIamClientSecret: \"changeme\"\n

Note

  • The Workspace API assumes a deployment of the Harbor Container Regsitry, as configured by the harborXXX values above.See section Container Registry.
  • The password for the harbor admin user must be created as described in the section Harbor admin Password.
  • The keycloakIntegration allows the Workspace API to apply protecion (for the specified workspace owner) to the services within newly created workspaces.
  • The workspace-api initiates the creation of a storage \u2018bucket\u2019 for each workspace - the actual bucket creation being abstracted via a webhook - the URL of which is specified in the value bucketEndpointUrl. See section Bucket Creation Webhook for details.
"},{"location":"eoepca/workspace/#harbor-admin-password","title":"Harbor admin Password","text":"

The password for the harbor admin user is provided to the workspace-api via the specified secret - defined as harbor above.

This secret must be created - for example as follows\u2026

kubectl -n rm create secret generic harbor \\\n  --from-literal=HARBOR_ADMIN_PASSWORD=\"changeme\"\n
"},{"location":"eoepca/workspace/#flux-dependency","title":"Flux Dependency","text":"

Workspaces are created by instantiating the rm-user-workspace helm chart for each user/group. The Workspace API uses Flux CD as a helper to manage these subordinate helm charts - via flux resources of type HelmRelease. Thus, it is necessary to deploy within the cluster the aspects of flux that support this helm chart management - namely the flux helm-controller, source-controller and the Kubernetes Custom Resource Definitions (CRD) for HelmRelease and HelmRepository.

In case you are not already using flux within your clsuter, then the Workspace API helm chart can be configured to deploy the required flux components\u2026 

fluxHelmOperator:\n  enabled: true  # true = install flux for me, false = I already have flux\n

"},{"location":"eoepca/workspace/#user-workspace-templates","title":"User Workspace Templates","text":"

The Workspace API instantiates for each user a set of services, including a Resource Catalogue and Data Access services. These user services are instantiated via helm using templates. The templates are provided to the Workspace API in a ConfigMap that is, by default, named workspace-charts. Each file in the config-map is expected to be of kind HelmRelease. During creation of a new workspace, the Worksapce API applies each file to the cluster in the namespace of the newly created namespace.

The default ConfigMap that is included with this guide contains the following templates for instantiation of user-specific components:

  • Data Access: template-hr-data-access.yaml
  • Resource Catalogue: template-hr-resource-catalogue.yaml
  • Protection: template-hr-resource-protection.yaml

Each of these templates is expressed as a flux HelmRelease object that describes the helm chart and values required to deploy the service.

In addition, ConfigMap templates are included that provide specific details required to access the user-scoped workspace resources, including access to S3 object storage and container registry:

  • S3 client configuration: template-cm-aws-config.yaml
  • S3 client credentials: template-cm-aws-credentials.yaml
  • Container registry configuration: template-cm-docker-config.yaml

These ConfigMaps are designed to be mounted as files into the runtime environments of other components for user workspace integration. In particular the Application Hub makes use of this approach to provide a user experience that integrates with the user\u2019s workspace resources.

"},{"location":"eoepca/workspace/#templates-configmap","title":"Templates ConfigMap","text":"

The templates are provided to the Workspace API as a ConfigMap in the namespace of the Workspace API deployment\u2026

(for full examples see https://github.com/EOEPCA/deployment-guide/tree/eoepca-v1.4/deploy/eoepca/workspace-templates)

apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: workspace-charts\ndata:\n  template-hr-resource-catalogue.yaml: |\n    apiVersion: helm.toolkit.fluxcd.io/v2beta1\n    kind: HelmRelease\n    metadata:\n      name: rm-resource-catalogue\n    spec:\n      interval: 5m\n      chart:\n        spec:\n          chart: rm-resource-catalogue\n          version: 1.3.1\n          sourceRef:\n            kind: HelmRepository\n            name: eoepca\n            namespace: rm\n      values:\n        ...\n  template-hr-data-access.yaml: |\n    apiVersion: helm.toolkit.fluxcd.io/v2beta1\n    kind: HelmRelease\n    metadata:\n      name: vs\n    spec:\n      interval: 5m\n      chart:\n        spec:\n          chart: data-access\n          version: 1.3.1\n          sourceRef:\n            kind: HelmRepository\n            name: eoepca\n            namespace: rm\n      values:\n        ...\n  template-hr-resource-protection.yaml: |\n    apiVersion: helm.toolkit.fluxcd.io/v2beta1\n    kind: HelmRelease\n    metadata:\n      name: resource-protection\n    spec:\n      interval: 5m\n      chart:\n        spec:\n          chart: identity-gatekeeper\n          version: 1.0.11\n          sourceRef:\n            kind: HelmRepository\n            name: eoepca\n            namespace: ${NAMESPACE}\n      values:\n        ...\n  template-cm-aws-config.yaml: |\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      name: aws-config\n    data:\n      aws-config: |\n        [default]\n        region = {{ s3_region }}\n        s3 =\n          endpoint_url = {{ s3_endpoint_url }}\n        s3api =\n          endpoint_url = {{ s3_endpoint_url }}\n        [plugins]\n        endpoint = awscli_plugin_endpoint\n  template-cm-aws-credentials.yaml: |\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      name: aws-credentials\n    data:\n      aws-credentials: |\n        [default]\n        aws_access_key_id = {{ access_key_id }}\n        aws_secret_access_key = {{ secret_access_key }}\n  template-cm-docker-config.yaml: |\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      name: docker-config\n    data:\n      docker-config: |\n        {\n          \"auths\": {\n            \"{{ container_registry_host }}\": {\n              \"auth\": \"{{ container_registry_credentials }}\"\n            }\n        }\n

Notice the use of workspace template parameters {{ param_name }} that are used at workspace creation time to contextualise the workspace for the owning user. See section Workspace Template Parameters for more information.

"},{"location":"eoepca/workspace/#helmrepositories-for-templates","title":"HelmRepositories for Templates","text":"

As can be seen above, the HelmRelease templates rely upon objects of type HelmRepository that define the hosting helm chart repository. Thus, in support of the workspace templates, appropriate HelmRepository objects must be provisioned within the cluster. For example, in support of the above examples that rely upon the EOEPCA Helm Chart Repository\u2026

apiVersion: source.toolkit.fluxcd.io/v1beta1\nkind: HelmRepository\nmetadata:\n  name: eoepca\n  namespace: rm\nspec:\n  interval: 2m\n  url: https://eoepca.github.io/helm-charts/\n
"},{"location":"eoepca/workspace/#workspace-template-parameters","title":"Workspace Template Parameters","text":"

The Workspace API uses the jinja2 templating engine when applying the resources for a user workspace. The current parameters are currently supported:

  • workspace_name The name of the workspace - {{ workspace_name }} used to ensure unique naming of cluster resources, such as service ingress
  • default_owner The uuid of the owner of the workspace - {{ default_owner }} used to initialise the workspace protection
  • S3 Object Storage details\u2026
    • {{ s3_endpoint_url }}
    • {{ s3_region }}
    • {{ access_key_id }}
    • {{ secret_access_key }}
  • Container Registry details\u2026
    • {{ container_registry_host }}
    • {{ container_registry_credentials }}
"},{"location":"eoepca/workspace/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the workspace-api service to provide access authorization decisions

"},{"location":"eoepca/workspace/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install workspace-api-protection identity-gatekeeper -f workspace-api-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"rm\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the workspace-api - in particular the specific ingress requirements for the workspace-api backend service\u2026

Example workspace-api-protection-values.yaml\u2026

fullnameOverride: workspace-api-protection\nconfig:\n  client-id: workspace-api\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: workspace-api.192-168-49-2.nip.io\n  name: workspace-api\n  port:\n    number: 8080\nsecrets:\n  # Values for secret 'workspace-api-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  serverSnippets:\n    custom: |-\n      # Open access to some endpoints, including Swagger UI\n      location ~ ^/(docs|openapi.json|probe) {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n
"},{"location":"eoepca/workspace/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: workspace-api above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example, with path protection for the admin user\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=\"workspace-api\" \\\n  --name=\"Workspace API Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Workspace API Gatekeeper\" \\\n  --resource=\"admin\" --uris='/*' --scopes=view --users=\"admin\"\n
"},{"location":"eoepca/workspace/#workspace-api-usage","title":"Workspace API Usage","text":"

The Workspace API provides a REST interface that is accessed at the endpoint https://workspace-api.192-168-49-2.nip.io/. See the Swagger Docs - /docs.

Note

If the Workspace API has been protected (via Gatekeeper with Keycloak), then requests must be supported by an access_token carried in the HTTP header Authorozation: Bearer <token>. This diminishes the utility of the swagger UI.

"},{"location":"eoepca/workspace/#additional-information","title":"Additional Information","text":"

Additional information regarding the Workspace API can be found at:

  • Helm Chart
  • Wiki
  • GitHub Repository
"},{"location":"eoepca/workspace/#bucket-creation-webhook","title":"Bucket Creation Webhook","text":"

With helm chart version 1.3.1 of the workspace-api the approach to bucket creation has been re-architected to use a webhook approach.

"},{"location":"eoepca/workspace/#approach","title":"Approach","text":"

During workspace creation the workspace-api needs to create an object storage bucket for the user. The method by which the bucket is created is a function of the hosting infrastructure object storage layer - i.e. there is no \u2018common\u2019 approach for the workspace-api to perform the bucket creation.

In order to allow this bucket creation step to be customised by the platform integrator, the workspace-api is configured with a webhook endpoint that is invoked to effect the bucket creation on behalf of the workspace-api.

The workspace-api is configured by the following value in its helm chart deployment, e.g\u2026 

bucketEndpointUrl: \"http://my-bucket-webhook:8080/bucket\"\n

The webhook service must implement the following REST interface\u2026

method: POST content-type: application/json data: 

{\n  bucketName: str\n  secretName: str\n  secretNamespace: str\n}\n

There are two possible approaches to implement this request, distinguished by the response code\u2026

  • 200 The bucket is created and the credentials are included in the response body. In this case only the supplied bucketName is relevant to fulfil the request.
  • 201 The bucket will be created (asychronously) and the outcome is provided by the webhook via a Kubernetes secret, as per the secretName and secretNamespace request parameters

200 Response

In case 200 response, the response body should communicate the credentials with an application/json content-type in the form\u2026 

{\n    \"bucketname\": \"...\",\n    \"access_key\": \"...\",\n    \"access_secret\": \"....\",\n    \"projectid\": \"...\",\n}\n

In this case the workspace-api will create the appropriate bucket secret using the returned credentials.

201 Response

In case 201 response, the secret should be created in the form\u2026 

data:\n  bucketname: \"...\"\n  access: \"...\"\n  secret: \"...\"\n  projectid: \"...\"\n

In this case the workspace-api will wait for the (asynchronous) creation of the specified secret before continuing with the workspace creation.

Overall Outcome

In both cases the ultimate outcome is the creation of the bucket in the back-end object storage, and the creation of a Kubernetes secret that maintains the credentials for access to the bucket. The existence of the bucket secret is prerequisite to the continuation of the user workspace creation.

"},{"location":"eoepca/workspace/#minio-bucket-api-webhook","title":"Minio Bucket API (Webhook)","text":"

The Minio Bucket API provides an implementation of a Bucket Creation Webhook for a Minio S3 Object Storage backend. This is used as the default in this guide - but should be replaced for a production deployment with an appropriate webhook to integrate with the object storage solution of the deployment environment.

"},{"location":"eoepca/workspace/#helm-chart_1","title":"Helm Chart","text":"

The Minio Bucket API is deployed via the rm-minio-bucket-api helm chart from the EOEPCA Helm Chart Repository - ref. Helm Chart for the Minio Bucket API.

helm install --version 0.0.4 --values minio-bucket-api-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  rm-minio-bucket-api rm-minio-bucket-api\n
"},{"location":"eoepca/workspace/#values_1","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • The URL for the Minio endpoint - minIOServerEndpoint
  • The credentials for admin access to Minio - via the specified secret accessCredentials.secretName (ref. Minio Credentials Secret)

Example minio-bucket-api-values.yaml\u2026 

fullnameOverride: minio-bucket-api\nminIOServerEndpoint: https://minio.192-168-49-2.nip.io\naccessCredentials:\n  secretName: minio-auth\n

"},{"location":"eoepca/workspace/#additional-information_1","title":"Additional Information","text":"

Additional information regarding the Minio Bucket API can be found at:

  • Helm Chart
  • GitHub Repository
"},{"location":"quickstart/application-hub-deployment/","title":"Application Hub Deployment","text":""},{"location":"quickstart/application-hub-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for an \u2018Application Hub\u2019 deployment - that provides the Application Hub integrated with the Identity Service (Keycloak) via OIDC for user authentication.

The script deploy/apphub/apphub achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/apphub/apphub-options.

The Application Hub deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • Identity Service (Keycloak) With test users eric, bob and alice created in Keycloak
    • Application Hub User eric and bob predefined as admin users
  • Other eoepca services not deployed
"},{"location":"quickstart/application-hub-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/apphub/apphub\n

The Identity Service (Keycloak) is accessed at the following endpoints\u2026

  • http://keycloak.192-168-49-2.nip.io/
  • http://identity-api.192-168-49-2.nip.io/docs (Swagger docs for the API)

The Application Hub is accessed at the endpoint - http://applicationhub.192-168-49-2.nip.io/.

"},{"location":"quickstart/application-hub-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/application-hub-deployment/#application-hub-notes","title":"Application Hub Notes","text":""},{"location":"quickstart/application-hub-deployment/#login","title":"Login","text":"

Authentication is made via the Sign in with EOEPCA button on the service home page - which redirects to Keycloak for authentication.

With the out-of-the-box configuration user eric or bob should be used with default password changeme. Users eric and bob are currently predefined within the helm chart as admin users - see https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/files/hub/jupyter_config.py#L171.

"},{"location":"quickstart/application-hub-deployment/#spawning-applications","title":"Spawning Applications","text":"

Once logged in, the service list is presented for spawning of applications. Note that this list of applications is currently defined within the helm chart - see https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/files/hub/config.yml.

From the list, a service is selected and the Start button initiates spawning.

For a clean deployment, the first spawn of each application may take some time whilst the container image representing the application is downloaded to the node. Subsequent invocations (at least on the same node) should be much faster. Once running, the application continues (in the background) until stopped by the user using the Stop Server button on the user\u2019s home screen.

The current JupyterHub configuration assumes a single application service (per user) running at a time - i.e. the current application must be stopped before the next can be started. There is an alternative configuration in which applications can be run in parallel and their lifecycles individually managed.

"},{"location":"quickstart/application-hub-deployment/#returning-to-the-home-screen","title":"Returning to the Home Screen","text":"

The launched applications do not (yet) have a navigation link \u2018out\u2019 of the application back to the home screen.

Therefore, it is necessary to manually modify the url in the browser address bar to /hub/home to navigate to the home screen - from where the current running server can be stopped or re-entered.

"},{"location":"quickstart/application-hub-deployment/#iat-jupyterlab","title":"IAT - JupyterLab","text":"

Following instantiation, the IAT application (Interactive Analysis Tool) defaults to the \u2018Jupyter Notebook\u2019 view (/user/<user>/tree) - rather than the Jupyter Lab view (/user/<user>/lab).

To switch to the Jupyter Lab view it is necessary to manually edit the url path from /user/<user>/tree to /user/<user>/lab. It is intended to update the default to this Jupyter Lab path.

"},{"location":"quickstart/creodias-deployment/","title":"CREODIAS Deployment","text":""},{"location":"quickstart/creodias-deployment/#deployment","title":"Deployment","text":"

Based upon our development experiences on CREODIAS, there is a wrapper script creodias with particular customisations suited to the CREODIAS infrastructure and data offering. The customisations are expressed through environment variables that are captured in the file creodias-options.

These scripts are examples that can be seen as a starting point, from which they can be adapted to your needs.

The CREODIAS deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Protected service endpoints requiring IAM authorization

With reference to the file creodias-options, particular attention is drawn to the following environment variables that require tailoring to your CREODIAS (Cloudferro) environment\u2026

  • Passwords: MINIO_ROOT_PASSWORD, HARBOR_ADMIN_PASSWORD
  • Identity Service credentials - e.g. IDENTITY_SERVICE_DEFAULT_SECRET, IDENTITY_SERVICE_ADMIN_PASSWORD, etc.
  • OpenStack details: see section Openstack Configuration
  • If configuring an external deployment - ref. Public Deployment\u2026
    • public_ip - The public IP address through which the deployment is exposed via the ingress-controller
    • domain - The DNS domain name through which the deployment is accessed - forming the stem for all service hostnames in the ingress rules

Once the file creodias-options has been well populated for your environment, then the deployment is initiated with\u2026 

./deploy/creodias/creodias\n
\u2026noting that this step is a customised version of that described in section Deployment.

"},{"location":"quickstart/creodias-deployment/#harvest-creodias-data","title":"Harvest CREODIAS Data","text":"

The harvester can be deployed with a default configuration file at /config.yaml. As described in the Data Access section, harvesting according to this configuration can be triggered with\u2026 

kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\n

See the Harvester section below for an explanation of this harvester configuration.

"},{"location":"quickstart/creodias-deployment/#data-specification-walkthrough","title":"Data Specification Walkthrough","text":"

The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables.

This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example.

"},{"location":"quickstart/creodias-deployment/#harvester","title":"Harvester","text":"

The harvester configuration specifies datasets with spatial/temporal extents, which is configured into the file /config.yaml of the data-access-harvester deployment.

The harvester is configured as follows\u2026

harvester:\n  replicaCount: 1\n  resources:\n    requests:\n      cpu: 100m\n      memory: 100Mi\n  config:\n    redis:\n      host: data-access-redis-master\n      port: 6379\n    harvesters:\n      - name: Creodias-Opensearch\n        resource:\n          url: https://datahub.creodias.eu/resto/api/collections/Sentinel2/describe.xml\n          type: OpenSearch\n          format_config:\n            type: 'application/json'\n            property_mapping:\n              start_datetime: 'startDate'\n              end_datetime: 'completionDate'\n              productIdentifier: 'productIdentifier'\n          query:\n            time:\n              property: sensed\n              begin: 2019-09-10T00:00:00Z\n              end: 2019-09-11T00:00:00Z\n            collection: null\n            bbox: 14.9,47.7,16.4,48.7\n        filter: {}\n        postprocess:\n          - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor\n        queue: register\n      - name: Creodias-Opensearch-Sentinel1\n        resource:\n          url: https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml\n          type: OpenSearch\n          format_config:\n            type: 'application/json'\n            property_mapping:\n              start_datetime: 'startDate'\n              end_datetime: 'completionDate'\n              productIdentifier: 'productIdentifier'\n          query:\n            time:\n              property: sensed\n              begin: 2019-09-10T00:00:00Z\n              end: 2019-09-11T00:00:00Z\n            collection: null\n            bbox: 14.9,47.7,16.4,48.7\n            extra_params:\n              productType: GRD-COG\n        filter: {}\n        postprocess:\n          - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel1Postprocessor\n        queue: register\n

Based upon this harvester configuration we expect that the following query is made to discover data - i.e. an OpenSearch query, with json response representation, for a defined spatial and temporal extent\u2026

https://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2019-09-10T00:00:00Z&completionDate=2019-09-11T00:00:00Z&box=14.9,47.7,16.4,48.7\n

From the result returned, the path to each product (feature) is obtained from the productIdentifier property, e.g.

{\n  \"type\": \"FeatureCollection\",\n  \"features\": [\n    {\n      \"type\": \"Feature\",\n      \"properties\": {\n        \"productIdentifier\": \"/eodata/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE\"\n        ...\n      }\n      ...\n    }\n    ...\n  ]\n}\n

The harvester is configured with a Sentinel-2/CREODIAS specific post-processor harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor which transforms the product path from /eodata/... to s3://EODATA/....

The harvester post-processor follows this path to the Sentinel-2 scene and uses stactools (with built-in support for Sentinel-2) to establish a STAC item representing the product. This includes enumeration of assets for inspire-metadata and product-metadata - which are used by the registrar pycsw backend to embelesh the product record metadata.

Note

The above description considers Sentinel-2 data. Similar considerations apply for Sentinel-1 that is also detailed in the above harvester configuration.

The harvester outputs the STAC item for each product, which is pushed to the registrar via the register redis queue.

"},{"location":"quickstart/creodias-deployment/#registration","title":"Registration","text":"

The registrar is configured at deployment to have the access details for the CREODIAS data in S3\u2026

global:\n  storage:\n    data:\n      data:\n        type: S3\n        endpoint_url: http://data.cloudferro.com\n        access_key_id: access\n        secret_access_key: access\n        region_name: RegionOne\n        validate_bucket_name: false\n

Using this S3 configuration, the registrar pycsw backend uses the product metadata linked in the STAC item (ref. assets inspire-metadata and product-metadata) to embelesh the metadata. For example, product-metadata in the file\u2026

s3://EODATA/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE/MTD_MSIL1C.xml\n

The registrar uses this information to create the ISO XML metadata that is loaded into the resource-catalogue.

"},{"location":"quickstart/creodias-deployment/#product-type","title":"Product Type","text":"

The registrar recognises the product as Sentinel-2 and so reads its metadata XML files to obtain additional information. From the metadata XML file (e.g. MTD_MSIL1C.xml) the registrar obtains the Product Type for each product from the field <PRODUCT_TYPE>\u2026

<n1:Level-1C_User_Product>\n  <n1:General_Info>\n    <Product_Info>\n      <PRODUCT_TYPE>S2MSI1C</PRODUCT_TYPE>\n      ...\n    </Product_Info>\n    ...\n  </n1:General_Info>\n  ...\n<n1:Level-1C_User_Product>\n
"},{"location":"quickstart/creodias-deployment/#resource-catalogue-collections","title":"Resource Catalogue Collections","text":"

The registrar (eoepca/rm-data-access-core) container image is pre-loaded with two collections at the path /registrar_pycsw/registrar_pycsw/resources, (in the built container the files are at the path /usr/local/lib/python3.8/dist-packages/registrar_pycsw/resources/):

  • S2MSI1C.yml - identifier: S2MSI1C
  • S2MSI2A.yml - identifier: S2MSI2A

The registrar applies these collections into the resource-catalogue during start-up - to create pre-defined out-of-the-box collections in pycsw.

During registration, the PycswBackend of the registrar uses the Product Type to map the product into the collection of the same name - using metadata field parentidentifier.

"},{"location":"quickstart/creodias-deployment/#data-specification","title":"Data Specification","text":"

The data-access service data handling is configured by definition of productTypes, collections and layers\u2026

  • productTypes identify the underlying file assets as WCS coverages and their visual representation
  • collections provide groupings into which products are organised
  • layers specifies the hoe the product visual representations are exposed through the WMS service
"},{"location":"quickstart/creodias-deployment/#producttype","title":"productType","text":"

During registration, products are mapped into a productType via a filter that is applied against the STAC item metadata.

The registrar uses the product_type of each product to determine the collection into which the product should be registered - noting that the name of the product type does not take part in the matching logic (and hence can be any text name)\u2026

  productTypes:\n    - name: S2MSI1C\n      filter:\n        s2:product_type: S2MSI1C\n

In the above example, the field s2:product_type is populated by the stactools that prepares the STAC item from the product metadata.

"},{"location":"quickstart/creodias-deployment/#producttype-coverages","title":"productType - coverages","text":"

coverages defines the coverages for the WCS service. Each coverage links to the assets that are defined within the product STAC item.

"},{"location":"quickstart/creodias-deployment/#producttype-browses","title":"productType - browses","text":"

browses defines the images that are visualised in the View Server Client. Expressions are used to map the product assets into their visual representation.

"},{"location":"quickstart/creodias-deployment/#collections","title":"collections","text":"

Collections are defined by reference to the defined productTypes and coverages.

"},{"location":"quickstart/creodias-deployment/#layers","title":"layers","text":"

layers defines the layers that are presented through the WMS service - each layer being linked to the underlying browse that provides the image source. Layers are defined via their id that relies upon the naming convection <collection>__<browse> to identify the browse and so define the layer.

"},{"location":"quickstart/creodias-deployment/#example-configuration","title":"Example Configuration","text":"

Example configuration for Sentinel-2 L1C and L2A data.

global:\n  layers:\n    - id: S2L1C\n      title: Sentinel-2 Level 1C True Color\n      abstract: Sentinel-2 Level 2A True Color\n      displayColor: '#eb3700'\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L1C__TRUE_COLOR\n      title: Sentinel-2 Level 1C True Color\n      abstract: Sentinel-2 Level 2A True Color\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L1C__masked_clouds\n      title: Sentinel-2 Level 1C True Color with cloud masks\n      abstract: Sentinel-2 Level 1C True Color with cloud masks\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L1C__FALSE_COLOR\n      title: Sentinel-2 Level 1C False Color\n      abstract: Sentinel-2 Level 1C False Color\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L1C__NDVI\n      title: Sentinel-2 Level 21CNDVI\n      abstract: Sentinel-2 Level 1C NDVI\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L2A\n      title: Sentinel-2 Level 2A True Color\n      abstract: Sentinel-2 Level 2A True Color\n      displayColor: '#eb3700'\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n    - id: S2L2A__TRUE_COLOR\n      title: Sentinel-2 Level 2A True Color\n      abstract: Sentinel-2 Level 2A True Color\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n    - id: S2L2A__masked_clouds\n      title: Sentinel-2 Level 2A True Color with cloud masks\n      abstract: Sentinel-2 Level 2A True Color with cloud masks\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n    - id: S2L2A__FALSE_COLOR\n      title: Sentinel-2 Level 2A False Color\n      abstract: Sentinel-2 Level 2A False Color\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n    - id: S2L2A__NDVI\n      title: Sentinel-2 Level 2A NDVI\n      abstract: Sentinel-2 Level 2A NDVI\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n  collections:\n    S2L1C:\n      product_types:\n        - S2MSI1C\n      coverage_types:\n        - S2L1C_B01\n        - S2L1C_B02\n        - S2L1C_B03\n        - S2L1C_B04\n        - S2L1C_B05\n        - S2L1C_B06\n        - S2L1C_B07\n        - S2L1C_B08\n        - S2L1C_B8A\n        - S2L1C_B09\n        - S2L1C_B10\n        - S2L1C_B11\n        - S2L1C_B12\n    S2L2A:\n      product_types:\n        - S2MSI2A\n      product_levels:\n        - Level-2A\n      coverage_types:\n        - S2L2A_B01\n        - S2L2A_B02\n        - S2L2A_B03\n        - S2L2A_B04\n        - S2L2A_B05\n        - S2L2A_B06\n        - S2L2A_B07\n        - S2L2A_B08\n        - S2L2A_B8A\n        - S2L2A_B09\n        - S2L2A_B11\n        - S2L2A_B12\n  productTypes:\n    - name: S2MSI1C\n      filter:\n        s2:product_type: S2MSI1C\n      metadata_assets: []\n      coverages:\n        S2L1C_B01:\n          assets:\n            - B01\n        S2L1C_B02:\n          assets:\n            - B02\n        S2L1C_B03:\n          assets:\n            - B03\n        S2L1C_B04:\n          assets:\n            - B04\n        S2L1C_B05:\n          assets:\n            - B05\n        S2L1C_B06:\n          assets:\n            - B06\n        S2L1C_B07:\n          assets:\n            - B07\n        S2L1C_B08:\n          assets:\n            - B08\n        S2L1C_B8A:\n          assets:\n            - B8A\n        S2L1C_B09:\n          assets:\n            - B09\n        S2L1C_B10:\n          assets:\n            - B10\n        S2L1C_B11:\n          assets:\n            - B11\n        S2L1C_B12:\n          assets:\n            - B12\n      defaultBrowse: TRUE_COLOR\n      browses:\n        TRUE_COLOR:\n          asset: visual\n          red:\n            expression: B04\n            range: [0, 4000]\n            nodata: 0\n          green:\n            expression: B03\n            range: [0, 4000]\n            nodata: 0\n          blue:\n            expression: B02\n            range: [0, 4000]\n            nodata: 0\n        FALSE_COLOR:\n          red:\n            expression: B08\n            range: [0, 4000]\n            nodata: 0\n          green:\n            expression: B04\n            range: [0, 4000]\n            nodata: 0\n          blue:\n            expression: B03\n            range: [0, 4000]\n            nodata: 0\n        NDVI:\n          grey:\n            expression: (B08-B04)/(B08+B04)\n            range: [-1, 1]\n      masks:\n        clouds:\n          validity: false\n    - name: S2MSI2A\n      filter:\n        s2:product_type: S2MSI2A\n      metadata_assets: []\n      coverages:\n        S2L2A_B01:\n          assets:\n            - B01\n        S2L2A_B02:\n          assets:\n            - B02\n        S2L2A_B03:\n          assets:\n            - B03\n        S2L2A_B04:\n          assets:\n            - B04\n        S2L2A_B05:\n          assets:\n            - B05\n        S2L2A_B06:\n          assets:\n            - B06\n        S2L2A_B07:\n          assets:\n            - B07\n        S2L2A_B08:\n          assets:\n            - B08\n        S2L2A_B8A:\n          assets:\n            - B8A\n        S2L2A_B09:\n          assets:\n            - B09\n        S2L2A_B11:\n          assets:\n            - B11\n        S2L2A_B12:\n          assets:\n            - B12\n      default_browse_locator: TCI_10m\n      browses:\n        TRUE_COLOR:\n          asset: visual-10m\n          red:\n            expression: B04\n            range: [0, 4000]\n            nodata: 0\n          green:\n            expression: B03\n            range: [0, 4000]\n            nodata: 0\n          blue:\n            expression: B02\n            range: [0, 4000]\n            nodata: 0\n        FALSE_COLOR:\n          red:\n            expression: B08\n            range: [0, 4000]\n            nodata: 0\n          green:\n            expression: B04\n            range: [0, 4000]\n            nodata: 0\n          blue:\n            expression: B03\n            range: [0, 4000]\n            nodata: 0\n        NDVI:\n          grey:\n            expression: (B08-B04)/(B08+B04)\n            range: [-1, 1]\n      masks:\n        clouds:\n          validity: false\n
"},{"location":"quickstart/data-access-deployment/","title":"Data Access Deployment","text":""},{"location":"quickstart/data-access-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for a \u2018data access\u2019 deployment - that is focused on the Resource Catalogue and Data Access services.

The script deploy/data-access/data-access achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/data-access/data-access-options.

The data-access deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • Resource Catalogue for data discovery
    • Data Access for data visualisation and download
  • Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION
  • Open ingress are enabled for unauthenticated access to resource-catalogue and data-access services
  • Other eoepca services not deployed
"},{"location":"quickstart/data-access-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/data-access/data-access\n

The Resource Catalogue is accessed at the endpoint resource-catalogue-open.192-168-49-2.nip.io - e.g. resource-catalogue-open.192-168-49-2.nip.io.

The Data Access View Server is accessed at the endpoint data-access-open.192-168-49-2.nip.io - e.g. data-access-open.192-168-49-2.nip.io.

"},{"location":"quickstart/data-access-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/data-access-deployment/#data-harvesting","title":"Data Harvesting","text":"

See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.

"},{"location":"quickstart/exploitation-deployment/","title":"Exploitation Deployment","text":""},{"location":"quickstart/exploitation-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for an \u2018exploitation\u2019 deployment - that provides deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services.

The script deploy/exploitation/exploitation achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/exploitation/exploitation-options.

The exploitation deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • ADES for processing
    • Resource Catalogue for data discovery
    • Data Access for data visualisation and download
    • Minio for S3 object storage
  • ADES stage-out to Minio
  • Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION
  • Open ingress are enabled for unauthenticated access to ADES, resource-catalogue and data-access services
  • Other eoepca services not deployed
"},{"location":"quickstart/exploitation-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/exploitation/exploitation\n

The ADES service is accessed at the endpoint ades-open.192-168-49-2.nip.io.

The Resource Catalogue is accessed at the endpoint resource-catalogue-open.192-168-49-2.nip.io.

The Data Access View Server is accessed at the endpoint data-access-open.192-168-49-2.nip.io.

"},{"location":"quickstart/exploitation-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/exploitation-deployment/#example-requests-s-expression-on-creodias","title":"Example Requests - s-expression on CREODIAS","text":"

NOTE that this example processing request requires harvesting data from CREODIAS, which can only be performed if the deployment is made to a CREODIAS VM with access to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION.

Section Processing provides an example of a simple self-contained processing deployment and execution, and access to the processing results.

In addition to the snuggs example, the file deploy/samples/requests/processing/s-expression.http has been prepared to exploit data that has been registered within the Resource Catalogue and Data Access services.

First the input data for processing must be harvested into the resource management services. Sentinel-2 data on 2nd Sept 2020\u2026

./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2020.09.02.yaml\n

Then the s-expression.http file provides sample requests for OGC API Processes operations:

  • List Processes
  • Deploy Process
  • Get Process Details
  • Execute Process
  • Get Job Status
  • Get Job Results

NOTE that the first requests in the file provide optional calls to obtain a user access token (openidConfiguration / authenticate) - to be used in the case that protected (not \u2018open\u2019) endpoints are deployed.

The files snuggs.http and s-expression.http describe the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client. Install in vscode with ext install humao.rest-client.

Various variables, such as to specify the @domain for your deployment, can be configured at the top of the file.

At the completion of successful processing execution, the procesing results are obtained as described in section Processing Results.

"},{"location":"quickstart/exploitation-deployment/#data-harvesting","title":"Data Harvesting","text":"

See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.

"},{"location":"quickstart/processing-deployment/","title":"Processing Deployment","text":""},{"location":"quickstart/processing-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for a \u2018processing\u2019 deployment - that is focused on the ADES and the deployment/execution of processing jobs.

The script deploy/processing/processing achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/processing/processing-options.

The processing deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • ADES for processing
    • Minio for S3 object storage
  • ADES stage-out to Minio
  • Open ingress are enabled for unauthenticated access to ADES service
  • Other eoepca services not deployed
"},{"location":"quickstart/processing-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/processing/processing\n

The ADES service is accessed at the endpoint zoo-open.192-168-49-2.nip.io.

"},{"location":"quickstart/processing-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/processing-deployment/#example-requests","title":"Example Requests","text":"

Some sample requests have been prepared in the subdirectory deploy/samples/requests/processing - for example\u2026

  • convert Provides a \u2018hello world\u2019 processing example that can be used simply to check that the processing capability has been well deployed
  • snuggs Provides a packaged EO exploitation algorithm that perform \u2018real\u2019 work and, as such, is more resource demanding (10GB RAM, 3 CPU) - and so may not be suitable for execution within a local minikube deployment (depending on resource allocations)

These sample http files have been prepared with sample requests for OGC API Processes operations:

  • List Processes
  • Deploy Process
  • Get Process Details
  • Execute Process
  • Get Job Status
  • Get Job Results

Note

  • The first requests in the file provide optional calls to obtain a user ID token (openidConfiguration / authenticate). These are to be used in the case that protected (not \u2018open\u2019) endpoints are deployed.
  • The file describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client. Install in vscode with ext install humao.rest-client.
  • The variables @hostname and @domain can be configured at the top of the file.
"},{"location":"quickstart/processing-deployment/#alternative-curl-commands","title":"Alternative curl Commands","text":"

Alternatively the following curl commands can be used\u2026

List Processes 
curl -k \\\n  --request GET \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n  --header 'accept: application/json'\n
Deploy & Execute (convert) Deploy Process (convert) - By Reference (JSON) 
curl -k \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/json' \\\n  --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/convert/main/convert-url-app.cwl\",\"type\": \"application/cwl\"}}'\n
Deploy Process (convert) - Inline (CWL) 
curl -k \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/cwl+yaml' \\\n  --data '< convert-url-app.cwl'\n
Get Process Details (convert) 
curl -k \\\n  --request GET \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \\\n  --header 'accept: application/json'\n
Execute Process (convert) 
curl -k -v \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url/execution \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/json' \\\n  --header 'prefer: respond-async' \\\n  --data '{\"inputs\": {\"fn\":  \"resize\",\"url\": \"https://eoepca.org/media_portal/images/logo6_med.original.png\", \"size\": \"50%\"},\"response\":\"raw\"}'\n
Undeploy Process (convert) 
curl -k -v \\\n  --request DELETE \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \\\n  --header 'accept: application/json'\n
Deploy & Execute (snuggs) Deploy Process (snuggs) 
curl -k \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/json' \\\n  --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/deployment-guide/eoepca-v1.4/deploy/samples/requests/processing/snuggs.cwl\",\"type\": \"application/cwl\"}}'\n
Get Process Details (snuggs) 
curl -k \\\n  --request GET \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \\\n  --header 'accept: application/json'\n
Execute Process (snuggs) 
curl -k -v \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs/execution \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/json' \\\n  --header 'prefer: respond-async' \\\n  --data '{\"inputs\": {\"input_reference\":  \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\",\"s_expression\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\"},\"response\":\"raw\"}'\n
Undeploy Process (snuggs) 
curl -k -v \\\n  --request DELETE \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \\\n  --header 'accept: application/json'\n
Get Job Status

This request requires the Location header from the response to the execute request. This will be of the form http://zoo-open.192-168-49-2.nip.io/{user}/ogc-api/jobs/{job-id} - e.g. http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e.

curl -k \\\n  --request GET \\\n  --url {location-header} \\\n  --header 'accept: application/json'\n
Get Job Results

This request uses the same URL as Get Job Status, with the additional URL path /results - i.e. /{user}/ogc-api/jobs/{job-id}/results - e.g. /eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e/results

curl -k \\\n  --request GET \\\n  --url {location-header}/results \\\n  --header 'accept: application/json'\n

The response indicates the location of the results, which should be in the minio object storage. See Processing Results.

The response also provides links to log files regarding each step of the Application Package workflow execution - which may be useful for debugging.

List Jobs 
curl -k \\\n  --request GET \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs \\\n  --header 'accept: application/json'\n
"},{"location":"quickstart/processing-deployment/#processing-results","title":"Processing Results","text":"

The outputs are published as a static STAC catalogue to a path that includes the unique job ID.

In the default configuration, the processing results are pushed to the Minio S3 object storage. This can be checked via browser access at the endpoint http://console.minio.192-168-49-2.nip.io/, or using an S3 client such as\u2026

s3cmd -c ./deploy/cluster/s3cfg ls s3://eoepca\n

For the default credentials to connect to Minio see Minio Object Storage Default Credentials.

Note

If the ADES deployment has been configured to stage-out to the user\u2019s Workspace, then the above s3cmd and credentials would have to be adjusted accordingly - for example the bucket s3://ws-eric.

"},{"location":"quickstart/quickstart/","title":"Quick Start","text":"

Note

The deployment of the EOEPCA components and the supporting Kubernetes cluster is described in the sections Prepare Cluster and Deploy EOEPCA Components. These sections should be consulted for more detailed information.

"},{"location":"quickstart/quickstart/#scripted-deployment","title":"Scripted Deployment","text":"

As a companion to these descriptions, we have developed a set of scripts to provide a demonstration of example deployments - see section Scripted Deployment for a detailed description of the scripts and how they are configured and used.

Note

The scripted deployment assumes that installation of the Prerequisite Tooling has been performed

"},{"location":"quickstart/quickstart/#customised-deployments","title":"Customised Deployments","text":"

The Scripted Deployment can be quickly exploited through the following customisations (profiles) for particular use cases:

  • Simple Basic local deployment
  • Processing Deployment focused on processing
  • Data Access Deployment focused on the Resource Catalogue and Data Access services
  • Exploitation Deployment providing deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services
  • User Management Deployment focused on the Identity & Access Management services
  • Application Hub Deployment providing the Application Hub that is pre-integrated via OIDC with the Identity Service
  • CREODIAS Deployment with access to CREODIAS EO data

Each customisation is introduced in their respective sections.

"},{"location":"quickstart/quickstart/#quick-example","title":"Quick Example","text":"

Follow these steps to create a simple local deployment in minikube\u2026

  1. Prerequisite Tooling Follow the steps in section Prerequisite Tooling to install the required tooling.
  2. Clone the repository git clone -b eoepca-v1.4 https://github.com/EOEPCA/deployment-guide
  3. Initiate the deployment 
    cd deployment-guide\n./deploy/simple/simple\n
  4. Wait for deployment ready
    1. List pod status watch kubectl get pod -A
    2. Wait until all pods report either Running or Completed This may take 10-20 mins depending on the capabilities of your platform.
  5. Test the deployment Make the sample requests to the ADES processing service.
"},{"location":"quickstart/scripted-deployment/","title":"Scripted Deployment","text":""},{"location":"quickstart/scripted-deployment/#overview","title":"Overview","text":"

The Scripted Deployment provides a demonstration of an example deployment, and can found in the subdirectory deployment-guide/deploy of the source repository for this guide\u2026

git clone -b eoepca-v1.4 https://github.com/EOEPCA/deployment-guide \\\n&& cd deployment-guide \\\n&& ls deploy\n

The script deploy/eoepca/eoepca.sh acts as an entry-point to the full system deployment. In order to tailor the deployment for your target environment, the script is configured through environment variables and command-line arguments. By default the script assumes deployment to a local minikube.

Note

The scripted deployment assumes that installation of the Prerequisite Tooling has been performed.

The following subsections lead through the steps for a full local deployment. Whilst minikube is assumed, minimal adaptions are required to make the deployment to your existing Kubernetes cluster.

The deployment follows these broad steps:

  • Configuration Tailoring of deployment options.
  • Deployment Creation of cluster and deployment of eoepca services.
  • Manual Steps Manual steps to be performed post-deployment.
"},{"location":"quickstart/scripted-deployment/#configuration","title":"Configuration","text":"

The script deploy/eoepca/eoepca.sh is configured by some environment variables and command-line arguments.

"},{"location":"quickstart/scripted-deployment/#environment-variables","title":"Environment Variables","text":"Environment Variables Variable Description Default REQUIRE_<cluster-component> A set of variables that can be used to control which CLUSTER components are deployed by the script, as follows (with defaults):REQUIRE_MINIKUBE=trueREQUIRE_INGRESS_NGINX=trueREQUIRE_CERT_MANAGER=trueREQUIRE_LETSENCRYPT=trueREQUIRE_SEALED_SECRETS=falseREQUIRE_MINIO=false see description REQUIRE_<eoepca-component> A set of variables that can be used to control which EOEPCA components are deployed by the script, as follows (with defaults):REQUIRE_STORAGE=trueREQUIRE_DUMMY_SERVICE=falseREQUIRE_IDENTITY_SERVICE=trueREQUIRE_ADES=trueREQUIRE_RESOURCE_CATALOGUE=trueREQUIRE_DATA_ACCESS=trueREQUIRE_REGISTRATION_API=trueREQUIRE_WORKSPACE_API=trueREQUIRE_HARBOR=trueREQUIRE_PORTAL=trueREQUIRE_APPLICATION_HUB=true see description REQUIRE_<protection-component> A set of variables that can be used to control which PROTECTION components are deployed by the script, as follows (with defaults):REQUIRE_DUMMY_SERVICE_PROTECTION=falseREQUIRE_ADES_PROTECTION=trueREQUIRE_RESOURCE_CATALOGUE_PROTECTION=trueREQUIRE_DATA_ACCESS_PROTECTION=trueREGISTRATION_API_PROTECTION=trueREQUIRE_WORKSPACE_API_PROTECTION=true see description MINIKUBE_VERSION The Minikube version to be (optionally) installedNote that the EOEPCA development has been conducted using the default stated here. v1.32.0 MINIKUBE_KUBERNETES_VERSION The Kubernetes version to be used by minikubeNote that the EOEPCA development has been conducted primarily using version 1.22.5. v1.22.5 MINIKUBE_MEMORY_AMOUNT Amount of memory to allocate to the docker containers used by minikube to implement the cluster. 12g MINIKUBE_DISK_AMOUNT Amount of disk space to allocate to the docker containers used by minikube to implement the cluster. 20g MINIKUBE_EXTRA_OPTIONS Additional options to pass to minikube start command-line --ports=80:80,443:443 USE_METALLB Enable use of minikube\u2019s built-in load-balancer.The load-balancer can be used to facilitate exposing services publicly. However, the same can be achieved using minikube\u2019s built-in ingress-controller. Therefore, this option is suppressed by default. false USE_INGRESS_NGINX_HELM Install the ingress-nginx controller using the published helm chart, rather than relying upon the version that is built-in to minikube. By default we prefer the version that is built in to minikube. false USE_INGRESS_NGINX_LOADBALANCER Patch the built-in minikube nginx-ingress-controller to offer a service of type LoadBalancer, rather than the default NodePort. It was initially thought that this would be necessary to achieve public access to the ingress services - but was subsequently found that the default NodePort configuration of the ingress-controller was sufficient. This option is left in case it proves useful.Only applicable for USE_INGRESS_NGINX_HELM=false (i.e. when using the minikube built-in ) false OPEN_INGRESS Create \u2018open\u2019 ingress endpoints that are not subject to authorization protection. For a secure system the open endpoints should be disabled (false) and access to resource should be protected via ingress that apply protection false USE_TLS Indicates whether TLS will be configured for service Ingress rules.If not (i.e. USE_TLS=false), then the ingress-controller is configured to disable ssl-redirect, and TLS_CLUSTER_ISSUER=notls is set. true TLS_CLUSTER_ISSUER The name of the ClusterIssuer to satisfy ingress tls certificates.Out-of-the-box ClusterIssuer instances are configured in the file deploy/cluster/letsencrypt.sh. letsencrypt-staging IDENTITY_SERVICE_DEFAULT_SECRET Default secret that is used by exception for other Identity Service credentials changeme IDENTITY_SERVICE_ADMIN_USER The admin user for Keycloak admin IDENTITY_SERVICE_ADMIN_PASSWORD The admin user password for Keycloak ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_SERVICE_ADMIN_CLIENT The Keycloak client to use for admin API tasks during scripted deployment admin-cli IDENTITY_POSTGRES_PASSWORD The password for the Keycloak Postgres service ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_GATEKEEPER_CLIENT_SECRET The secret used for each Keycloak client (one per resource service) created during scripted deployment ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_GATEKEEPER_ENCRYPTION_KEY The encryption key for each Keycloak client (one per resource service) created during scripted deployment changemechangeme IDENTITY_REALM Keycloak realm for Identity Service.This is not explicitly created by the scripted deployment, and so is assumed to exist within the Keycloak instance. Thus, will probably break the deployment if modified. master MINIO_ROOT_USER Name of the \u2018root\u2019 user for the Minio object storage service. eoepca MINIO_ROOT_PASSWORD Password for the \u2018root\u2019 user for the Minio object storage service. changeme HARBOR_ADMIN_PASSWORD Password for the \u2018admin\u2019 user for the Harbor artefact registry service. changeme DEFAULT_STORAGE Storage Class to be used by default for all components requiring dynamic storage provisioning.See variables <component>_STORAGE for per-component overrides. standard <component>_STORAGE A set of variables to control the dynamic provisioning Storage Class for individual components, as follows:MINIO_STORAGEADES_STORAGEAPPLICATION_HUB_STORAGEDATA_ACCESS_STORAGEHARBOR_STORAGERESOURCE_CATALOGUE_STORAGE <DEFAULT_STORAGE> PROCESSING_MAX_RAM Max RAM allocated to an individual processing job 8Gi PROCESSING_MAX_CORES Max number of CPU cores allocated to an individual processing job 4 PROCESSING_ZOO_IMAGE Container image for zoo-dru deployment eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762 STAGEOUT_TARGET Configures the ADES with the destination to which it should push processing results:workspace - via the Workspace APIminio - to minio S3 object storage workspace INSTALL_FLUX The Workspace API relies upon Flux CI/CD, and has the capability to install the required flux components to the cluster. If your deployment already has flux installed then set this value false to suppress the Workspace API flux install true CREODIAS_DATA_SPECIFICATION Apply the data specification to harvest from the CREODIAS data offering into the resource-catalogue and data-access services.Can only be used when running in the CREODIAS (Cloudferro) cloud, with access to the eodata network. false TEMP_FORWARDING_PORT Local port used during the scripted deployment for kubectl port-forward operations 9876"},{"location":"quickstart/scripted-deployment/#command-line-arguments","title":"Command-line Arguments","text":"

The eoepca.sh script is further configured via command-line arguments\u2026

eoepca.sh <action> <cluster-name> <domain> <public-ip>\n
eoepca.sh Command-line Arguments Argument Description Default action Action to perform: apply | delete | template.apply makes the deploymentdelete removes the deploymenttemplate outputs generated kubernetes yaml to stdout apply cluster-name The name of the minikube \u2018profile\u2019 for the created minikube cluster eoepca domain The DNS domain name through which the deployment is accessed. Forms the stem for all service hostnames in the ingress rules - i.e. <service-name>.<domain>.By default, the value is deduced from the assigned cluster minikube IP address, using nip.io to establish a DNS lookup - i.e. <minikube ip>.nip.io. <minikube ip>.nip.io public-ip The public IP address through which the deployment is exposed via the ingress-controller.By default, the value is deduced from the assigned cluster minikube IP address - ref. command minikube ip. <minikube-ip>"},{"location":"quickstart/scripted-deployment/#public-deployment","title":"Public Deployment","text":"

For simplicity, the out-of-the-box scripts assume a \u2018private\u2019 deployment - with no public IP / DNS and hence no use of TLS for service ingress endpoints.

In the case that an external-facing public deployment is desired, then the following configuration selections should be made:

  • domain - set to the domain (as per DNS records) for your deployment Note that the EOEPCA components typically configure their ingress with hostname prefixes applied to this domain. Thus, it is necessary that the DNS record for the domain is established as a wildcard record - i.e. *.<domain>
  • public_ip - set to the public IP address through which the deployment is exposed via the ingress-controller i.e. the IP address that is assigned to the ingress controller service of type LoadBalancer
  • USE_TLS=true - to enable configuration of TLS endpoints in each component service ingress
  • TLS_CLUSTER_ISSUER=<issuer> - should be configured ~ e.g. using the letsencrypt-production or letsencrypt-staging (testing only) Cluster Issuer that are configured by the scripted deployment
"},{"location":"quickstart/scripted-deployment/#deployment","title":"Deployment","text":"

The deployment is initiated by setting the appropriate environment variables and invoking the eoepca.sh script with suitable command-line arguments. You may find it convenient to do so using a wrapper script that customises the environment varaibles according to your cluster, and then invokes the eoepca.sh script.

Customised examples are provided for Simple, CREODIAS and Processing deployments.

NOTE that if a prior deployment has been attempted then, before redeploying, a clean-up should be performed as described in the Clean-up section below. This is particularly important in the case that the minikube none driver is used, as the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed.

Initiate the deployment\u2026 

./deploy/eoepca/eoepca.sh apply \"<cluster-name>\" \"<public-ip>\" \"<domain>\"\n

The deployment takes 10+ minutes - depending on the resources of your host/cluster. The progress can be monitored\u2026 

kubectl get pods -A\n

The deployment is ready once all pods are either Running or Completed.

"},{"location":"quickstart/scripted-deployment/#post-deployment-manual-steps","title":"Post-deployment Manual Steps","text":"

The scripted deployment has been designed, as far as possible, to automate the configuration of the deployed components. However, there remain some steps that must be performed manually after the scripted deployment has completed. See the building block specific pages\u2026

  • Identity Service: Token Lifespans
  • Application Hub: Post-deployment Manual Steps
"},{"location":"quickstart/scripted-deployment/#default-credentials","title":"Default Credentials","text":""},{"location":"quickstart/scripted-deployment/#identity-service","title":"Identity Service","text":"

By default, the Identity Service is accessed at the URL https://keycloak.192-168-49-2.nip.io/ with the credentials\u2026

username: `admin`\npassword: `changeme`\n

\u2026unless the password is overridden via the variable IDENTITY_SERVICE_ADMIN_PASSWORD.

"},{"location":"quickstart/scripted-deployment/#minio-object-storage","title":"Minio Object Storage","text":"

By default, Minio is accessed at the URL https://console.minio.192-168-49-2.nip.io/ with the credentials\u2026

username: `eoepca`\npassword: `changeme`\n

\u2026unless the username/password are overridden via the variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD.

"},{"location":"quickstart/scripted-deployment/#harbor-container-registry","title":"Harbor Container Registry","text":"

By default, Harbor is accessed at the URL https://harbor.192-168-49-2.nip.io/ with the credentials\u2026

username: `admin`\npassword: `changeme`\n

\u2026unless the password is overridden via the variable HARBOR_ADMIN_PASSWORD.

"},{"location":"quickstart/scripted-deployment/#protection","title":"Protection","text":"

The protection of resource server endpoints is applied during the deployment of each service requiring protection. This comprises creating a dedicated Keycloak client for each resource server, and the creation of associated resources and policies that protect the service-specific URLs.

This protection can be disabled via the environment variables REQUIRE_XXX_PROTECTION - e.g. REQUIRE_ADES_PROTECTION=false.

Note

By default, if OPEN_INGRESS is set true then PROTECTION will be disabled (false) unless overridden via the REQUIRE_XXX_PROTECTION variables.

"},{"location":"quickstart/scripted-deployment/#test-users","title":"Test Users","text":"

The deployment creates (in the Keycloak Identity Service) the test users: eric, bob, alice.

Note

This does NOT create the workspace for each of these users - which must be performed via the Workspace API.

"},{"location":"quickstart/scripted-deployment/#user-workspace-creation","title":"User Workspace Creation","text":"

The deployment created the test users eric, bob and alice. For completeness we use the Workspace API to create their user workspaces, which hold their personal resources (data, processing results, etc.) within the platform - see Workspace.

"},{"location":"quickstart/scripted-deployment/#using-workspace-swagger-ui","title":"Using Workspace Swagger UI","text":"

The Workspace API provides a Swagger UI that facilitates interaction with the API - at the URL https://workspace-api.192-168-49-2.nip.io/docs#.

Note

If the Workspace API has been protected (via Gatekeeper with Keycloak), then requests must be supported by an access_token carried in the HTTP header Authorozation: Bearer <token>. This diminishes the utility of the swagger UI.

Access the Workspace Swagger UI at https://workspace-api.192-168-49-2.nip.io/docs. Workspaces are created using POST /workspaces (Create Workspace). Expand the node and select Try it out. Complete the request body, such as\u2026 

{\n  \"preferred_name\": \"eric\",\n  \"default_owner\": \"eric\"\n}\n
\u2026where the default_owner is the ID for the user in Keycloak - thus protecting the created workspace for the identified user.

"},{"location":"quickstart/scripted-deployment/#using-curl","title":"Using curl","text":"

The same can be achieved with a straight http request, for example using curl\u2026

curl -X 'POST' \\\n  'http://workspace-api.192-168-49-2.nip.io/workspaces' \\\n  -H 'Content-Type: application/json' \\\n  -H 'Accept: application/json' \\\n  -H 'Authorization: Bearer <admin-access-token>' \\\n  -d '{\n  \"preferred_name\": \"<workspace-name>\",\n  \"default_owner\": \"<user-id>\"\n}'\n

Values must be provided for:

  • admin-access-token - Access Token for the admin user
  • workspace-name - name of the workspace, typically the username
  • user-id - the ID of the user for which the created workspace will be protected, typically the username

The Access Token for the admin user can be obtained with a call to the token endpoint of the Identity Service - supplying the credentials for the admin user and the pre-registered client\u2026

curl -L -X POST 'https://keycloak.192-168-49-2.nip.io/realms/master/protocol/openid-connect/token' \\\n  -H 'Cache-Control: no-cache' \\\n  -H 'Content-Type: application/x-www-form-urlencoded' \\\n  --data-urlencode 'scope=openid profile email' \\\n  --data-urlencode 'grant_type=password' \\\n  --data-urlencode 'username=admin' \\\n  --data-urlencode 'password=<admin-password>' \\\n  --data-urlencode 'client_id=admin-cli'\n

A json response is returned, in which the field access_token provides the Access Token for the admin user.

"},{"location":"quickstart/scripted-deployment/#using-create-workspace-helper-script","title":"Using create-workspace helper script","text":"

As an aide there is a helper script create-workspace. The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The create-workspace helper script requires some command-line arguments\u2026

$ ./deploy/bin/create-workspace -h\n\nCreate a new User Workspace.\ncreate-workspace -h | -w {workspace_api} -a {auth_server} -r {realm} -c {client} -u {admin-username} -p {admin-password} -O {owner} -W {workspace-name}\n\nwhere:\n    -h  show help message\n    -w  workspace-api service url (default: http://workspace-api.192-168-49-2.nip.io)\n    -a  authorization server url (default: http://keycloak.192-168-49-2.nip.io)\n    -r  realm within Keycloak (default: master)\n    -u  username used for authentication (default: admin)\n    -p  password used for authentication (default: changeme)\n    -c  client id of the bootstrap client used in the create request (default: admin-cli)\n    -O  user ID of the 'owner' of the new workspace (default: workspace(-W))\n    -W  name of the workspace to create (default: owner(-O))\n

Most of the arguments have default values that are aligned to the defaults of the scripted deployment. At minimum either -O owner or -W workspace must be specified.

For example (assuming defaults)\u2026

./deploy/bin/create-workspace -O eric\n

For example (all arguments)\u2026

./deploy/bin/create-workspace \n  -w http://workspace-api.192-168-49-2.nip.io \\\n  -a http://keycloak.192-168-49-2.nip.io \\\n  -r master \\\n  -u admin \\\n  -p changeme \\\n  -c admin-cli \\\n  -O bob \\\n  -W bob\n
"},{"location":"quickstart/scripted-deployment/#eoepca-portal","title":"EOEPCA Portal","text":"

The eoepca-portal is a simple web application that is used as a test aid. It\u2019s main purpose is to provide the ability to login, and so establish a session with appropriate browser cookies - which then allow authenticated access to other EOEPCA services such as the Workspace API, Identity API, etc.

The portal is deployed via a helm chart\u2026

helm install eoepca-portal eoepca-portal -f portal-values.yaml - \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"demo\" --create-namespace \\\n  --version 1.0.11\n

The helm values must be tailored for your deployment. For example\u2026

configMap:\n  identity_url: \"http://keycloak.192-168-49-2.nip.io\"\n  realm: \"master\"\n  client_id: \"eoepca-portal\"\n  identity_api_url: \"http://identity-api.192-168-49-2.nip.io\"\n  ades_url: \"http://zoo.192-168-49-2.nip.io/ogc-api/processes\"\n  resource_catalogue_url: \"http://resource-catalogue.192-168-49-2.nip.io\"\n  data_access_url: \"http://data-access.192-168-49-2.nip.io\"\n  workspace_url: \"http://workspace-api.192-168-49-2.nip.io\"\n  workspace_docs_url: \"http://workspace-api.192-168-49-2.nip.io/docs#\"\n  images_registry_url: \"http://harbor.192-168-49-2.nip.io\"\n  dummy_service_url: \"http://dummy-service.192-168-49-2.nip.io\"\n  access_token_name: \"auth_user_id\"\n  access_token_domain: \".192-168-49-2.nip.io\"\n  refresh_token_name: \"auth_refresh_token\"\n  refresh_token_domain: \".192-168-49-2.nip.io\"\ningress:\n  enabled: true\n  annotations:\n    kubernetes.io/ingress.class: nginx\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  hosts:\n    - host: eoepca-portal.192-168-49-2.nip.io\n      paths:\n        - path: /\n          pathType: Prefix\n  tls:\n    - secretName: eoepca-portal-tls\n      hosts:\n        - eoepca-portal.192-168-49-2.nip.io\n

The setting client_id: eoepca-portal identifies a client that must be created in Keycloak - as described in section create-client Helper Script - noting that the eoepca-portal requires a client that is configured as a Public Client\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=eoepca-portal \\\n  --name=\"EOEPCA Portal\" \\\n  --public \\\n  --description=\"Client to be used by the EOEPCA Portal\"\n
"},{"location":"quickstart/scripted-deployment/#clean-up","title":"Clean-up","text":"

Before initiating a fresh deployment, if a prior deployment has been attempted, then it is necessary to remove any persistent artefacts of the prior deployment. This includes\u2026

  1. Minikube cluster Delete the minikube cluster\u2026 minikube delete If necessary specify the cluster (profile)\u2026 minikube -p <profile> delete

  2. Persistent Data In the case that the minikube none driver is used, the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. In this case, the minikube standard StorageClass is fulfilled by the hostpath provisioner, whose persistence is removed as follows\u2026 sudo rm -rf /tmp/hostpath-provisioner

There is a helper script clean that can be used for step 2 above, (the script does not delete the cluster). 

./deploy/cluster/clean\n

"},{"location":"quickstart/simple-deployment/","title":"Simple Deployment","text":""},{"location":"quickstart/simple-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for a \u2018simple\u2019 deployment - designed to get a core local deployment of the primary servies.

The script deploy/simple/simple achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/simple/simple-options.

The simple deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Configuration of \u2018open\u2019 interfaces - i.e. service/API endpoints that are not protected and can accessed without authentication. This facilitates experimentation with the services
  • Configuration of ADES stage-out to a local instance of minio, to avoid the need to create a Workspace for each user
"},{"location":"quickstart/simple-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/simple/simple\n

See section Deployment for more details regarding the outcome of the scripted deployment.

"},{"location":"quickstart/simple-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/userman-deployment/","title":"User Management Deployment","text":""},{"location":"quickstart/userman-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for a \u2018user management\u2019 deployment - that is focused on the Identity Service (Authorization Server), Identity API and Gatekeeper (Protection Policy Enforcement).

The script deploy/userman/userman achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/userman/userman-options.

The user-management deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • Identity Service
    • Identity API
    • Gatekeeper instance, protecting the Identity API
  • Other eoepca services not deployed
"},{"location":"quickstart/userman-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/userman/userman\n

The Identity Service is accessed at the endpoint keycloak.192-168-49-2.nip.io.

The Identity API is accessed at the endpoint identity-api.192-168-49-2.nip.io.

"},{"location":"quickstart/userman-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Deployment Guide","text":"

Changelog

This current version of the Deployment Guide represents the development tip that goes beyond the latest release verion v1.4.

The following provides a summary of changes since the last release (v1.4)\u2026

  • [FIX] Update Application Hub to chart version 2.0.57 to fix hard-coded ingress namespace
  • [FIX] Correct default value of PROCESSING_MAX_RAM to the integer value in Mi 8192 (was string 8Gi)

The Deployment Guide captures each release of the EOEPCA Reference Implementation, by providing for each version\u2026

  • Description of how each building-block is configured and deployed - see Deploy EOEPCA Components
  • Scripted deployment in which each building-block can be selectively deployed to form a system - see Getting Started

A full system deployment is described, in which components are deployed with complementary configurations that facilitate their integration as a coherent system. Nevertheless, each component can be cherry-picked from this system deployment for individual re-use.

The deployment is organised into the following sections:

  • Getting Started A quickstart guide with associated scripts to facilitate example deployments, which preempt the descriptions that follow later in the document. Scripts are provided in a variety of \u2018profiles\u2019 that deploy different combinations of building-blocks for different notional use cases.
  • Prepare Cluster Establish the Kubernetes cluster and other prerequisites for the deployment of the EOEPCA system.
  • Deploy EOEPCA Components Deployment of the EOEPCA components.
"},{"location":"cluster/cluster-prerequisites/","title":"Cluster Prerequisites","text":""},{"location":"cluster/cluster-prerequisites/#overview","title":"Overview","text":"

The following prerequisite components are assumed to be deployed in the cluster.

Note

The Scripted Deployment automatically deploys most of the components list here - in particular\u2026

  • Nginx Ingress Controller
  • Cert Manager
  • Letsencrypt ClusterIssuers
  • Minio Object Storage

The Sealed Secrets controller is not deployed - but can be added following the instructions below.

"},{"location":"cluster/cluster-prerequisites/#nginx-ingress-controller","title":"Nginx Ingress Controller","text":"
# Install the Nginx Ingress Controller helm chart\nhelm upgrade -i --version='<4.5.0' \\\n  --repo https://kubernetes.github.io/ingress-nginx \\\n  ingress-nginx ingress-nginx \\\n  --wait\n

Note

For Kubernetes version 1.22 and earlier the version of the Nginx Ingress Controller must be before v4.5.0.

To target the Nginx Ingress Controller the kubernetes.io/ingress.class: nginx annotation must be applied to the Ingress resource\u2026 

apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  annotations:\n    kubernetes.io/ingress.class: nginx\n    ...\n

"},{"location":"cluster/cluster-prerequisites/#cert-manager","title":"Cert Manager","text":"
# Install the Cert Manager helm chart\nhelm upgrade -i --namespace cert-manager --create-namespace \\\n  --repo https://charts.jetstack.io \\\n  --set installCRDs=true \\\n  cert-manager cert-manager\n
"},{"location":"cluster/cluster-prerequisites/#letsencrypt-certificates","title":"Letsencrypt Certificates","text":"

Once the Certificate Manager is deployed, then we can establish ClusterIssuer operators in the cluster to support use of TLS with service Ingress endpoints.

For Letsencrypt we can define two ClusterIssuer - for production and for staging.

NOTE that these require the cluster to be publicly accessible, in order for the http01 acme flow to verify the domain ownership. Local development deployments will typically not have public IP/DNS - in which case the system deployment can proceed, but without TLS support for the service endpoints.

"},{"location":"cluster/cluster-prerequisites/#production","title":"Production","text":"
apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n  name: letsencrypt-production\nspec:\n  acme:\n    # You must replace this email address with your own.\n    # Let's Encrypt will use this to contact you about expiring\n    # certificates, and issues related to your account.\n    email: eoepca.systemteam@telespazio.com\n    server: https://acme-v02.api.letsencrypt.org/directory\n    privateKeySecretRef:\n      # Secret resource that will be used to store the account's private key.\n      name: letsencrypt-production-account-key\n    # Add a single challenge solver, HTTP01 using nginx\n    solvers:\n      - http01:\n          ingress:\n            class: nginx\n
"},{"location":"cluster/cluster-prerequisites/#staging","title":"Staging","text":"
apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n  name: letsencrypt-staging\nspec:\n  acme:\n    # You must replace this email address with your own.\n    # Let's Encrypt will use this to contact you about expiring\n    # certificates, and issues related to your account.\n    email: eoepca.systemteam@telespazio.com\n    server: https://acme-staging-v02.api.letsencrypt.org/directory\n    privateKeySecretRef:\n      # Secret resource that will be used to store the account's private key.\n      name: letsencrypt-staging-account-key\n    # Add a single challenge solver, HTTP01 using nginx\n    solvers:\n      - http01:\n          ingress:\n            class: nginx\n

To exploit the specified ClusterIssuer the cert-manager.io/cluster-issuer annotation must be applied to the Ingress resource. For example\u2026 

apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n  annotations:\n    kubernetes.io/ingress.class: nginx\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    ...\n

"},{"location":"cluster/cluster-prerequisites/#sealed-secrets","title":"Sealed Secrets","text":"

The EOEPCA development team maintain their deployment configurations in GitHub - for declarative, reproducible cluster deployments.

Various Secret are relied upon by the system services. Secrets should not be exposed by commit to GitHub.

Instead SealedSecret are committed to GitHub, which are encrypted, and can only be decrypted by the sealed-secret-controller that runs within the cluster. The sealed-secret-controller decrypts the SealedSecret to a regular Secret (of the same name) that can then be consumed by the cluster components.

The sealed-secret-controller is deployed to the cluster using the helm chart\u2026

helm install --version 2.1.8 --create-namespace --namespace infra \\\n  --repo https://bitnami-labs.github.io/sealed-secrets \\\n  eoepca-sealed-secrets sealed-secrets\n

Once the controller is deployed within the cluster, then the kubeseal command can be used to create a SealedSecret from a regular Secret, as follows\u2026

Create example Secret\u2026 

kubectl -n test create secret generic mysecret \\\n  --from-literal=password=changeme \\\n  --dry-run=client -o yaml \\\n  > mysecret.yaml\n

Create SealedSecret from Secret using kubeseal\u2026 

kubeseal -o yaml \\\n  --controller-name eoepca-sealed-secrets \\\n  --controller-namespace infra \\\n  < mysecret.yaml \\\n  > mysecret-sealed.yaml\n

"},{"location":"cluster/cluster-prerequisites/#references","title":"References","text":"
  • Sealed Secrets on GitHub
  • kubeseal Release
"},{"location":"cluster/cluster-prerequisites/#minio-object-storage","title":"MinIO Object Storage","text":"

Various building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service.

As a workaround, in the absence of an existing object storage, it is possible to use MinIO to establish an object storage service within the Kubernetes cluster. We use the minio helm chart provided by the MinIO Project.

# Install the minio helm chart\nhelm upgrade -i -f minio-values.yaml --namespace rm --create-namespace \\\n  --repo https://charts.min.io/ \\\n  minio minio \\\n  --wait\n

Note

The Kubernetes namespace rm is used above as an example, and can be changed according to your deployment preference.

The minio deployment is customised via the values file minio-values.yaml, for example\u2026

existingSecret: minio-auth\nreplicas: 2\n\ningress:\n  enabled: true\n  ingressClassName: nginx\n  annotations:\n    cert-manager.io/cluster-issuer: \"letsencrypt\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n    nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n  path: /\n  hosts:\n    - minio.192-168-49-2.nip.io\n  tls:\n    - secretName: minio-tls\n      hosts:\n        - minio.192-168-49-2.nip.io\n\nconsoleIngress:\n  enabled: true\n  ingressClassName: nginx\n  annotations:\n    cert-manager.io/cluster-issuer: \"letsencrypt\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n    nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n  path: /\n  hosts:\n    - console.minio.192-168-49-2.nip.io\n  tls:\n  - secretName: minio-console-tls\n    hosts:\n      - console.minio.192-168-49-2.nip.io\n\nresources:\n  requests:\n    memory: 1Gi\n\npersistence:\n  storageClass: standard\n\nbuckets:\n  - name: eoepca\n  - name: cache-bucket\n

Note

  • The example values assuming a TLS configuration using letsencrypt certificate provider
  • The admin credentials are provided by the Kubernetes secret named minio-auth - see below
  • The annotation nginx.ingress.kubernetes.io/proxy-body-size was found to be required to allow transfer of large files (such as data products) through the nginx proxy
"},{"location":"cluster/cluster-prerequisites/#minio-credentials-secret","title":"Minio Credentials Secret","text":"

The Minio admin credentials are provided via a Kubernetes secret that is referenced from the Minio helm chart deployment values. For example\u2026

kubectl -n rm create secret generic minio-auth \\\n  --from-literal=rootUser=\"eoepca\" \\\n  --from-literal=rootPassword=\"changeme\"\n

Note

The secret must be created in the same Kubernetes namespace as the Minio service deployment - e.g. rm namespce in the example above.

"},{"location":"cluster/cluster-prerequisites/#s3cmd-configuration","title":"s3cmd Configuration","text":"

The s3cmd can be configured for access to the MinIO deployment. The --configure option can be used to prepare a suitable configuration file for s3cmd\u2026

s3cmd -c mys3cfg --configure\n

In response to the prompts, the following configuration selections are applicable to the above settings\u2026

Access Key: eoepca\nSecret Key: changeme\nDefault Region: us-east-1\nS3 Endpoint: minio.192-168-49-2.nip.io\nDNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io\nEncryption password: \nPath to GPG program: /usr/bin/gpg\nUse HTTPS protocol: True\nHTTP Proxy server name: \nHTTP Proxy server port: 0\n

Save the configuration file, and check access to the S3 object store with\u2026

# Create a bucket\ns3cmd -c mys3cfg mb s3://eoepca\n\n# List buckets\ns3cmd -c mys3cfg ls\n

For example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube\u2026 

s3cmd -c deploy/cluster/s3cfg ls\n

"},{"location":"cluster/cluster-prerequisites/#references_1","title":"References","text":"
  • MinIO Website
  • MinIO Helm Chart
  • MinIO on GitHub
"},{"location":"cluster/helm-repositories/","title":"Helm Repositories","text":"

Note

This section identifies some helm chart repositories that can be referenced (for convenience) via helm add. Nevertheless, all helm commands included in the guide specifically reference the source helm repository via the --repo argument to the helm install command - and thus it is not specifically necessary to add these repositories in advance.

"},{"location":"cluster/helm-repositories/#eoepca-helm-charts","title":"EOEPCA Helm Charts","text":"

The EOEPCA building-blocks are engineered as containers for deployment to a Kubernetes cluster. Each building block defines a Helm Chart to facilitate its deployment.

The EOEPCA Helm Chart Repository is configured with helm as follows\u2026 

helm repo add eoepca https://eoepca.github.io/helm-charts/\n

"},{"location":"cluster/helm-repositories/#third-party-helm-charts","title":"Third-party Helm Charts","text":"

In addition to the EOEPCA Helm Chart Repository, a variety of third party helm repositories are relied upon, as identified below.

"},{"location":"cluster/helm-repositories/#cert-manager","title":"Cert Manager","text":"
helm repo add jetstack https://charts.jetstack.io\n
"},{"location":"cluster/helm-repositories/#nginx-ingress-controller","title":"Nginx Ingress Controller","text":"
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\n
"},{"location":"cluster/helm-repositories/#minio","title":"Minio","text":"
helm repo add minio https://charts.min.io/\n
"},{"location":"cluster/helm-repositories/#sealed-secrets-bitnami","title":"Sealed Secrets (Bitnami)","text":"
helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets\n
"},{"location":"cluster/helm-repositories/#harbor","title":"Harbor","text":"
helm repo add harbor https://helm.goharbor.io\n
"},{"location":"cluster/helm-repositories/#repo-update","title":"Repo Update","text":"

Refresh the local repo cache, after helm repo add\u2026

helm repo update\n
"},{"location":"cluster/kubernetes/","title":"Kubernetes Cluster","text":"

The EOEPCA Reference Implementation has been developed with Kubernetes as its deployment target. The system components have been developed, deployed and tested using a cluster at version v1.22.5.

Note

The Scripted Deployment assumes that minikube is installed, and creates a minikube cluster under the profile eoepca.

"},{"location":"cluster/kubernetes/#rancher-kubernetes-engine-rke","title":"Rancher Kubernetes Engine (RKE)","text":"

The development, integration and test clusters have been established using Rancher Kubernetes Engine (RKE) at version v1.22.5.

An example of the creation of the EOEPCA Kubernetes clusters can be found on the GitHub Kubernetes Setup page. CREODIAS has been used for the development hosting infrastructure - which provides OpenStack infrastructure that is backed by Cloudferro. An example of the Terraform configurations used to automate the creation of the cloud infrastructure that underpins the RKE deployment can be found on the GitHub CREODIAS Setup page.

"},{"location":"cluster/kubernetes/#local-kubernetes","title":"Local Kubernetes","text":"

To make a full deployment of the EOEPCA Reference Implementation requires a multi-node node cluster with suitable resources. For example, the development cluster comprises:

  • 1 Master node (2 vCPU, 8 GB RAM)
  • 5 Worker nodes (4 vCPU, 16 GB RAM)
  • 1 NFS server (2 vCPU, 8 GB RAM)

Limited local deployment can be made using a suitable local single-node kuberbetes deployment using - for example using minikube\u2026

minikube -p eoepca start --cpus max --memory max --kubernetes-version v1.22.5\nminikube profile eoepca\n

With such a deployment it is possible to deploy individual building-blocks for local development, or building-blocks in combination - within the constraints of the local host resources.

"},{"location":"cluster/prerequisite-tooling/","title":"Prerequisite Tooling","text":"

There are some standard tools referenced in this guide. These are detailed in the following subsections.

"},{"location":"cluster/prerequisite-tooling/#docker","title":"docker","text":"

Docker faciliates the creation, management and execution of containers. Whilst not strictly necessary to support deployment to an existing/managed Kubernetes cluster, it can nevertheless be useful to have local access to the docker tooling. For example, if minikube is used to follow this guide using a local k8s cluster, then this is best achieved using minikube\u2019s docker driver.

Docker is most easily installed with\u2026 

curl -fsSL https://get.docker.com | sh\n

For convenience, add your user to the docker group\u2026 

sudo usermod -aG docker ${USER}\n

Logout/in to refresh your session\u2019s group permissions.

"},{"location":"cluster/prerequisite-tooling/#kubectl","title":"kubectl","text":"

Kubectl is the main tool for interaction with a Kubernetes cluster. The latest version can be installed with\u2026 

mkdir -p $HOME/.local/bin \\\n&& curl -fsSLo $HOME/.local/bin/kubectl \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\" \\\n&& chmod +x $HOME/.local/bin/kubectl\n

See the official kubectl installation documentation for more installation options.

"},{"location":"cluster/prerequisite-tooling/#helm","title":"helm","text":"

Helm is the Kubernetes package manager, in which components are deployed to a Kubernetes cluster via helm charts. The helm charts are instantiated for deployment via \u2018values\u2019 that configure the chart templates.

The latest helm version can be installed with\u2026 

export HELM_INSTALL_DIR=\"$HOME/.local/bin\" \\\n&& curl -sfL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash\n

See the official helm installation documentation for more installation options.

"},{"location":"cluster/prerequisite-tooling/#minikube","title":"minikube","text":"

Minikube is a tool that allows to create a local (single-node) Kubernetes cluster for development/testing. It is not designed for production use. In the absence of access to a \u2018full\u2019 Kubernetes cluster, this guide can be followed using minikube.

The latest version of minikube can be installed with\u2026 

mkdir -p $HOME/.local/bin \\\n&& curl -fsSLo $HOME/.local/bin/minikube \"https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\" \\\n&& chmod +x $HOME/.local/bin/minikube\n

See the official minikube installation documentation for more installation options.

"},{"location":"eoepca/ades-zoo/","title":"ADES (Processing)","text":"ADES - Application Deployment & Execution Service"},{"location":"eoepca/ades-zoo/#zoo-project-dru","title":"ZOO-Project DRU","text":"

Note

With EOEPCA release 1.4, the ADES implementation has been significantly reworked and fully aligned with the upstream ZOO-Project (GitHub). This zoo-project-dru version deprecates the previous proc-ades implementation.

With this transition, there are some functional changes to be aware of\u2026

  • Service Endpoint With zoo-project-dru the OGC API Processes endpoint is at the path /<username>/ogc-api/processes compared to the previous /<username>/wps3/processes.
  • Deployed Application Endpoint The endpoint for a deployed Application no longer appends the version of the Application Package. For example, previously the application convert-url at version 0.1.2 would result in the endpoint /<username>/wps3/processes/convert-url_0_1_2. With the new zoo-project-dru this same Application Package deployment will result in the endpoint /<username>/ogc-api/processes/convert-url.
  • Deployed Application Version The version of the deployed application is obtained from the Application Package CWL (ref. s:softwareVersion: 0.1.2), and is maintained within the metadata for the deployed process that is returned from the APIs Get Process Details request. In the case that multiple versions of the same Application Package are required to be simultaneously deployed, then this would have to be handled with different CWL documents in which the version is embedded in the workflow id (or some other technique that establishes uniqueness of id between variants).
DRU - Deploy, Replace, Undeploy - OFC API Processes Part 2

The ADES provides a platform-hosted execution engine through which users can initiate parameterised processing jobs using applications made available within the platform - supporting the efficient execution of the processing \u2018close to the data\u2019. Users can deploy specific \u2018applications\u2019 to the ADES, which may be their own applications, or those published by other platform users.

The ADES provides an implementation of the OGC API Processes - Part 1: Core and Part 2: Deploy, Replace, Undeploy (draft).

"},{"location":"eoepca/ades-zoo/#helm-chart","title":"Helm Chart","text":"

The EOEPCA deployment is aligned with the upstream implementation and so relies upon the upstream helm chart that is hosted at the ZOO-Project Helm Chart Repository - in particular the zoo-project-dru chart variant.

The chart is configured via values that are fully documented in the README for the zoo-project-dru chart.

helm install --version 0.2.6 --values ades-values.yaml \\\n  --repo https://zoo-project.github.io/charts/ \\\n  zoo-project-dru zoo-project-dru\n
"},{"location":"eoepca/ades-zoo/#values","title":"Values","text":"

The deployment must be configured for you environment. Some significant configuration values are elaborated here\u2026

"},{"location":"eoepca/ades-zoo/#cookie-cutter-template","title":"Cookie-cutter Template","text":"

The implementation zoo-project-dru provides the core capabilities for OGC API Processes Parts 1 & 2. The deployemnt of this core must be completed by inetgartion with the \u2018runner\u2019 that executes the processes as Application Packages, and integrates as necessary with other platform services - such as Catalogue, Workspace, etc.

Thus, zoo-project-dru is extensible by design via a \u2018cookie-cutter\u2019 that provides the template \u2018runner\u2019 for each Application Package process as it is deployed to the service.

For the purposes of our EOEPCA \u2018release\u2019 as covered by this guide, we provide eoepca-proc-service-template as a cookie-cutter implemetation that provides:

  • Integration with Kubernetes to run process Application packages, via the Calrissian CWL runner
  • Stage-in of inputs as STAC items, integrated as required with S3 object storage
  • Stage-out of outputs as a STAC Collection, integrated with S3 object storage and (optionally) user Workspace inetgration

The cookie-cutter template is identified in the helm values\u2026

cookiecutter:\n  templateUrl: https://github.com/EOEPCA/eoepca-proc-service-template.git\n  templateBranch: master\n

The function of the cookie-cutter template is supported some other aspects, that are elaborated below, which must be configured in collaboration with the expectations of the template. In particular\u2026

  • Template parameterisation that is passed through the core zoo-project-dru configuration [ref]
  • CWL \u2018wrapper\u2019 files that prepend and append the process Application Package CWL to perform stage-in and stage-out functions [ref]
"},{"location":"eoepca/ades-zoo/#zoo-project-dru-custom-configuration","title":"ZOO-Project DRU custom configuration","text":"

In order support our eoepca-proc-service-template cookie-cutter template, there is a custom zoo-project-dru container image that includes the python dependencies that are required by this template. Thus, the deployment must identify the custom container image via helm values\u2026

zoofpm:\n  image:\n    tag: eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762\nzookernel:\n  image:\n    tag: eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762\n

In addition, we can add values to the ZOO-Project DRU main.cfg configuration file via helm values. In this case we add some eoepca-specific values that match those that we know to be expected by our eoepca-proc-service-template cookie-cutter template. In this way we can effectively use helm values to pass parameters through to the template.

customConfig:\n  main:\n    eoepca: |-\n      domain=192-168-49-2.nip.io\n      workspace_prefix=ws\n

This is manifest in zoo\u2019s main.cfg in INI file configuration syntax\u2026

[eoepca]\ndomain=192-168-49-2.nip.io\nworkspace_prefix=ws\n

The presence or otherwise of the workspace_prefix parameter dicates whether or not the stage-out step will integrate with the user\u2019s Workspace for persistence of the processing results, and registration within the Workspace services.

In the case that workspace_prefix is not set, then the object storage specification in the helm values is relied upon\u2026

workflow:\n  inputs:\n    STAGEOUT_AWS_SERVICEURL: https://minio.192-168-49-2.nip.io\n    STAGEOUT_AWS_ACCESS_KEY_ID: eoepca\n    STAGEOUT_AWS_SECRET_ACCESS_KEY: changeme\n    STAGEOUT_AWS_REGION: RegionOne\n    STAGEOUT_OUTPUT: eoepca\n
"},{"location":"eoepca/ades-zoo/#stage-in-stage-out","title":"Stage-in / Stage-out","text":"

The ADES hosts applications that are deployed and invoked in accordance with the OGC Best Practise for Application Package. Thus, the ADES provides a conformant environment within which the application is integrated for execution. A key part of the ADES\u2019s role in this is to faciltate the provision of input data to the application (stage-in), and the handling of the results output at the conclusion of application execution (stage-out).

The zoo-project-dru helm chart provides a default implementation via the included files - main.yaml, rules.yaml, stagein.yaml and stageout.yaml.

The helm values provides a means through which each of these files can be overriden for reasons of integration with your platform environment\u2026

files:\n  # Directory 'files/cwlwrapper-assets' - assets for ConfigMap 'XXX-cwlwrapper-config'\n  cwlwrapperAssets:\n    main.yaml: |-\n      <override file content here>\n    rules.yaml: |-\n      <override file content here>\n    stagein.yaml: |-\n      <override file content here>\n    stageout.yaml: |-\n      <override file content here>\n

In the most part the default CWL wrapper files provided with the helm chart are suffient. In particular the stagein.yaml implements the stage-in of STAC items that are specified as inputs of type Directory in the Application Package CWL.

E.g. 

    inputs:\n      stac:\n        label: the image to convert as a STAC item\n        doc: the image to convert as a STAC item\n        type: Directory\n

Nevertheless, in this guide we provide an override of the stageout.yaml in order to organise the processing outputs into a STAC Collection that is then pushed to the designated S3 object storage, including support for the user\u2019s workspace storage and resource management services.

The custom stage-out embeds, within the CWL document, the python code required to implement the desired stage-out functionality. This should be regarded as an example that could be adapted for alternative behaviour.

cwlVersion: v1.0\nclass: CommandLineTool\nid: stage-out\ndoc: \"Stage-out the results to S3\"\ninputs:\n  process:\n    type: string\n  collection_id:\n    type: string\n  STAGEOUT_OUTPUT:\n    type: string\n  STAGEOUT_AWS_ACCESS_KEY_ID:\n    type: string\n  STAGEOUT_AWS_SECRET_ACCESS_KEY:\n    type: string\n  STAGEOUT_AWS_REGION:\n    type: string\n  STAGEOUT_AWS_SERVICEURL:\n    type: string\noutputs:\n  StacCatalogUri:\n    outputBinding:\n      outputEval: ${  return \"s3://\" + inputs.STAGEOUT_OUTPUT + \"/\" + inputs.process + \"/catalog.json\"; }\n    type: string\nbaseCommand:\n  - python\n  - stageout.py\narguments:\n  - $( inputs.wf_outputs.path )\n  - $( inputs.STAGEOUT_OUTPUT )\n  - $( inputs.process )\n  - $( inputs.collection_id )\nrequirements:\n  DockerRequirement:\n    dockerPull: ghcr.io/terradue/ogc-eo-application-package-hands-on/stage:1.3.2\n  InlineJavascriptRequirement: {}\n  EnvVarRequirement:\n    envDef:\n      AWS_ACCESS_KEY_ID: $( inputs.STAGEOUT_AWS_ACCESS_KEY_ID )\n      AWS_SECRET_ACCESS_KEY: $( inputs.STAGEOUT_AWS_SECRET_ACCESS_KEY )\n      AWS_REGION: $( inputs.STAGEOUT_AWS_REGION )\n      AWS_S3_ENDPOINT: $( inputs.STAGEOUT_AWS_SERVICEURL )\n  InitialWorkDirRequirement:\n    listing:\n      - entryname: stageout.py\n        entry: |-\n          import sys\n          import shutil\n          import os\n          import pystac\n\n          cat_url = sys.argv[1]\n\n          shutil.copytree(cat_url, \"/tmp/catalog\")\n          cat = pystac.read_file(os.path.join(\"/tmp/catalog\", \"catalog.json\"))\n\n          ...\n

The helm chart values provide the opportunity to pass through additional inputs - to satisfy the input specifications that are specified in the cwlwrapperAssets files\u2026

workflow:\n  inputs:\n    STAGEIN_AWS_SERVICEURL: http://data.cloudferro.com\n    STAGEIN_AWS_ACCESS_KEY_ID: test\n    STAGEIN_AWS_SECRET_ACCESS_KEY: test\n    STAGEIN_AWS_REGION: RegionOne\n    STAGEOUT_AWS_SERVICEURL: https://minio.192-168-49-2.nip.io\n    STAGEOUT_AWS_ACCESS_KEY_ID: eoepca\n    STAGEOUT_AWS_SECRET_ACCESS_KEY: changeme\n    STAGEOUT_AWS_REGION: RegionOne\n    STAGEOUT_OUTPUT: eoepca\n
"},{"location":"eoepca/ades-zoo/#node-selection","title":"Node Selection","text":"

The zoo-project-dru services uses a Node Selector to determine the node(s) upon which the processing execution is run. This is configured as a matching rule in the helm values, and must be tailored to your cluster.

For example, for minikube\u2026

workflow:\n  nodeSelector:\n    minikube.k8s.io/primary: \"true\"\n
"},{"location":"eoepca/ades-zoo/#ingress","title":"Ingress","text":"

Ingress can be enabled and configured to establish (reverse-proxy) external access to the zoo-project-dru services.

Hosturl

In the case that protection is enabled - e.g. via Resource Guard - then it is likely that ingress should be disabled here, since the ingress will instead be handled by the protection.

In this case, the hosturl parameter should be set to reflect the public url through the service will be accessed.

In the case that ingress is enabled then it is not necessary to specify the hosturl, since it will be taken from the ingress.hosts[0].host value.

Ingress disabled\u2026

ingress:\n  enabled: false\n  hosturl: zoo.192-168-49-2.nip.io\n

Ingress enabled\u2026

ingress:\n  enabled: true\n  annotations:\n    kubernetes.io/ingress.class: nginx\n    ingress.kubernetes.io/ssl-redirect: true\n    nginx.ingress.kubernetes.io/ssl-redirect: true\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  hosts:\n  - host: zoo-open.192-168-49-2.nip.io\n    paths:\n    - path: /\n      pathType: ImplementationSpecific\n  tls:\n  - hosts:\n    - zoo-open.192-168-49-2.nip.io\n    secretName: zoo-open-tls\n

The above example assumes that TLS should be enabled via Letsencrypt as certificate provider - see section Letsencrypt Certificates.

"},{"location":"eoepca/ades-zoo/#persistence","title":"Persistence","text":"

Various of the services deployed as part of zoo-project-dru rely upon dynamic provisioning of persistent storage volumes.

A number of helm values are impacted by this setting, which must be configured with the Storage Class appropriate to your cluster. For example, using the minikube standard storage class\u2026

workflow:\n  storageClass: standard\npersistence:\n  procServicesStorageClass: standard\n  storageClass: standard\n  tmpStorageClass: standard\npostgresql:\n  primary:\n    persistence:\n      storageClass: standard\n  readReplicas:\n    persistence:\n      storageClass: standard\nrabbitmq:\n  persistence:\n    storageClass: standard\n
"},{"location":"eoepca/ades-zoo/#built-in-iam","title":"Built-in IAM","text":"

ZOO-Project DRU has a built-in capability for Identity & Access Management (IAM), in which the zoo-project-dru service is configured as an OIDC client of an OIDC Identity Provider service.

This capability is disabled by the default deployment offered by this guide (ingress.enabled: false) - which instead (optionally) applies resource protection using the EOEPCA IAM solution. Nevertheless, the built-in IAM can be enabled and configured through helm values.

For example\u2026

iam: \n  enabled: true\n  openIdConnectUrl: https://keycloak.192-168-49-2.nip.io/realms/master/.well-known/openid-configuration\n  type: openIdConnect\n  name: OpenIDAuth\n  realm: Secured section\n
"},{"location":"eoepca/ades-zoo/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the zoo-project-dru service to provide access authorization decisions

"},{"location":"eoepca/ades-zoo/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install zoo-project-dru-protection identity-gatekeeper -f zoo-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"zoo\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the zoo-project-dru - in particular the specific ingress requirements for the zoo-project-dru-service\u2026

Example zoo-protection-values.yaml\u2026

fullnameOverride: zoo-project-dru-protection\nconfig:\n  client-id: ades\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: zoo.192-168-49-2.nip.io\n  name: zoo-project-dru-service\n  port:\n    number: 80\nsecrets:\n  # Values for secret 'zoo-project-dru-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  serverSnippets:\n    custom: |-\n      # Open access to some endpoints, including Swagger UI\n      location ~ /(ogc-api/api|swagger-ui) {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n
"},{"location":"eoepca/ades-zoo/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: ades above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example, with path protection for test users\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=ades \\\n  --name=\"ADES Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by ADES Gatekeeper\" \\\n  --resource=\"eric\" --uris='/eric/*' --scopes=view --users=\"eric\" \\\n  --resource=\"bob\" --uris='/bob/*' --scopes=view --users=\"bob\" \\\n  --resource=\"alice\" --uris='/alice/*' --scopes=view --users=\"alice\"\n
"},{"location":"eoepca/ades-zoo/#service-urls","title":"Service URLs","text":"

The zoo-project-dru service provides a mutil-user aware set of service interfaces at\u2026

  • OGC API Processes: https://zoo.192-168-49-2.nip.io/<username>/ogc-api/
  • Swagger UI: https://zoo.192-168-49-2.nip.io/swagger-ui/oapip/
"},{"location":"eoepca/ades-zoo/#usage-samples","title":"Usage Samples","text":"

See the Example Requests in the Processing Deployment for sample requests that cans be used to test your deployment, and to learn usage of the OGC API Processes.

"},{"location":"eoepca/ades-zoo/#debugging-tips","title":"Debugging Tips","text":"

This section includes some tips that may be useful in debugging errors with deployed application packages.

For debugging, establish a shell session with the zoofpm pod\u2026

$ kubectl -n zoo exec -it deploy/zoo-project-dru-zoofpm -c zoofpm -- bash\n
"},{"location":"eoepca/ades-zoo/#execution-logs","title":"Execution Logs","text":"

The logs are in the directory /tmp/zTmp\u2026

$ cd /tmp/zTmp/\n

In the log directory, each execution is characterised by a set of files/directories\u2026

  • <appname>_<jobid>_error.log <<START HERE The main log output of the job
  • <appname>_<jobid>.json The output (results) of the job
  • <jobid>_status.json The overall status of the job
  • <jobid>_logs.cfg Index of logs for job workflow steps
  • convert-url-c6637d4a-d561-11ee-bf3b-0242ac11000e (directory) Subdirectory with a dedicated log file for each step of the CWL workflow, including the stage-in and stage-out steps
"},{"location":"eoepca/ades-zoo/#deployed-process-executables","title":"Deployed Process \u2018Executables\u2019","text":"

When the process is deployed from its Application Package, then a representation is created using the configured cookiecutter.templateUrl.

It may be useful to debug the consequent process files, which are located under the path /opt/zooservices_user/<username>, with a dedicated subdirectory for each deployed process - i.e. /opt/zooservices_user/<username>/<appname>/.

For example\u2026

$ cd /opt/zooservices_user/eric/convert-url\n$ ls -l\ntotal 28\n-rw-rw-r-- 1 www-data www-data     0 Feb 27 11:17 __init__.py\ndrwxrwxr-x 2 www-data www-data  4096 Feb 27 11:17 __pycache__\n-rw-rw-r-- 1 www-data www-data  1408 Feb 27 11:17 app-package.cwl\n-rw-rw-r-- 1 www-data www-data 17840 Feb 27 11:17 service.py\n

Note

In the case that the cookie-cutter template is updated, then the process can be re-deployed to force a refresh against the updated template.

"},{"location":"eoepca/ades-zoo/#swagger-ui-openapi","title":"Swagger UI (OpenAPI)","text":"

The zoo-project-dru service includes a Swagger UI interactive representation of its OpenAPI REST interface - available at the URL https://zoo.192-168-49-2.nip.io/swagger-ui/oapip/.

"},{"location":"eoepca/ades-zoo/#application-package-example","title":"Application Package Example","text":"

For a (trivial) example application package see Example Application Package, which provides a description and illustration of the basics of creating an application that integrates with the expectations of the ADES stage-in and stage-out.

For further reference see\u2026

  • Application Packages
    • OGC Best Practise for Application Package
    • Example Application Package
  • Common Workflow Language (CWL)
    • Guide for CWL in Earth Observation
    • CWL Specification
    • CWL User Guide
"},{"location":"eoepca/ades-zoo/#additional-information","title":"Additional Information","text":"

Additional information regarding the ADES can be found at:

  • ZOO-Project DRU\u2026
    • Helm Chart
    • Documentation
  • Git Repositories\u2026
    • ZOO-Project Core OGC API Processes capability
    • eoepca-proc-service-template Cookie-cutter template for Application Package execution in Kubernetes
    • zoo-calrissian-runner Python library used by the eoepca-proc-service-template to aid orchestration of CWL application packages running in Kubernetes via Calrissian
    • pycalrissian Python library used by zoo-calrissian-runner to aid interfacing with Calrissian and Kubernetes
"},{"location":"eoepca/application-hub/","title":"Application Hub","text":"

The Application Hub provides a set of web-based tooling, including JupyterLab for interactive analysis, Code Server for application development, and the capability to add user-defined interactive dashboards.

"},{"location":"eoepca/application-hub/#helm-chart","title":"Helm Chart","text":"

The Application Hub is deployed via the application-hub helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values, which are detailed in the default values file for the chart.

helm install --version 2.0.57 --values application-hub-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  application-hub application-hub\n
"},{"location":"eoepca/application-hub/#values","title":"Values","text":"

The Application Hub supports many values to configure the service - ref. the default values file for the chart.

Typically, values for the following attributes may be specified:

  • The fully-qualified public URL for the service
  • Specification of Ingress for reverse-proxy access to the service
  • Storage class for persistence
  • Node selector rule - required by JupyterHub to spawn container workloads
  • Values for integration with the user workspace
  • Integration of JupyterHub with the Login Service (identity provider) via OpenID Connect configuration
  • OIDC client credentials from a secret

Example application-hub-values.yaml\u2026

ingress:\n  enabled: true\n  annotations: {}\n  hosts:\n    - host: applicationhub.192-168-49-2.nip.io\n      paths:\n        - path: /\n          pathType: ImplementationSpecific\n  tls:\n    - secretName: applicationhub-tls\n      hosts:\n      - applicationhub.192-168-49-2.nip.io\n  clusterIssuer: letsencrypt-production\n\njupyterhub:\n  fullnameOverride: \"application-hub\"\n  hub:\n    existingSecret: application-hub-secrets\n    extraEnv: \n        JUPYTERHUB_ENV: \"dev\"\n        JUPYTERHUB_SINGLE_USER_IMAGE: \"eoepca/pde-container:1.0.3\"\n        OAUTH_CALLBACK_URL: https://applicationhub.192-168-49-2.nip.io/hub/oauth_callback\n        OAUTH2_USERDATA_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/userinfo\n        OAUTH2_TOKEN_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/token\n        OAUTH2_AUTHORIZE_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/authorize\n        OAUTH_LOGOUT_REDIRECT_URL: \"https://applicationhub.192-168-49-2.nip.io\"\n        OAUTH2_USERNAME_KEY: \"preferred_username\"\n        STORAGE_CLASS: \"standard\"\n        RESOURCE_MANAGER_WORKSPACE_PREFIX: \"ws\"\n\n        JUPYTERHUB_CRYPT_KEY:\n          valueFrom:\n            secretKeyRef:\n              name: application-hub-secrets\n              key: JUPYTERHUB_CRYPT_KEY\n\n        OAUTH_CLIENT_ID:\n          valueFrom:\n            secretKeyRef:\n              name: application-hub-secrets\n              key: OAUTH_CLIENT_ID\n\n        OAUTH_CLIENT_SECRET:\n          valueFrom:\n            secretKeyRef:\n              name: application-hub-secrets\n              key: OAUTH_CLIENT_SECRET\n\n    image:\n      # name: eoepca/application-hub\n      # tag: \"1.2.0\"\n      pullPolicy: Always\n      # pullSecrets: []\n\n    db:\n      pvc:\n        storageClassName: standard\n\n  singleuser:\n    image:\n      name: jupyter/minimal-notebook\n      tag: \"2343e33dec46\"\n    profileList: \n    - display_name:  \"Minimal environment\"\n      description: \"To avoid too much bells and whistles: Python.\"\n      default: \"True\"\n    - display_name:  \"EOEPCA profile\"\n      description: \"Sample profile\"\n      kubespawner_override:\n        cpu_limit\": 4\n        mem_limit\": \"8G\"\n\nnodeSelector:\n  key: minikube.k8s.io/primary\n  value: \\\"true\\\"\n
"},{"location":"eoepca/application-hub/#client-and-credentials","title":"Client and Credentials","text":"

The Application Hub requires an OIDC client to be registered with the Identity Service (Keycloak) in order to enable user identity integration - ref. OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=application-hub \\\n  --name=\"Application Hub OIDC Client\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Application Hub for OIDC integration\"\n

Corresponding to this client, a secret application-hub-secrets must be created (ref. value jupyterhub.hub.existingSecret: application-hub-secrets)\u2026

kubectl -n proc create secret generic application-hub-secrets \\\n  --from-literal=JUPYTERHUB_CRYPT_KEY=\"$(openssl rand -hex 32)\" \\\n  --from-literal=OAUTH_CLIENT_ID=\"application-hub\" \\\n  --from-literal=OAUTH_CLIENT_SECRET=\"changeme\"\n
"},{"location":"eoepca/application-hub/#post-deployment-manual-steps","title":"Post-deployment Manual Steps","text":"

The deployment of the Application Hub has been designed, as far as possible, to automate the configuration. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026

  • Configure Groups and Users
"},{"location":"eoepca/application-hub/#groups-and-users","title":"Groups and Users","text":"

The default helm chart has some built-in application launchers whose assignments to example users (eric and bob) assume the existence of some JupyterHub groups - which must be replicated to exploit this configuration.

  • In a browser, navigate to the Application Hub - https://applicationhub.192-168-49-2.nip.io/
  • Login as the user eric (or bob) for admin access
  • Select the Admin menu (top of page)
  • Add groups group-1, group-2, group-3 to ApplicationHub, and add users eric, bob to these groups

This setup corresponds to the \u2018sample\u2019 configuration that is built=in to the help chart - see file config.yaml.

"},{"location":"eoepca/application-hub/#additional-information","title":"Additional Information","text":"

Additional information regarding the Application Hub can be found at:

  • Helm Chart
"},{"location":"eoepca/container-registry/","title":"Container Registry","text":"

To support the development (ref. Application Hub) and deployment/execution (ref. ADES) of user-defined applications, we deploy a container registry to host container images. This is provied by a deployment of the Harbor artefact repository.

"},{"location":"eoepca/container-registry/#helm-chart","title":"Helm Chart","text":"

Harbor is deployed via the harbor helm chart from the Harbor Helm Chart Repository.

helm install --version 1.7.3 --values harbor-values.yaml \\\n   --repo https://helm.goharbor.io \\\n  harbor harbor\n
"},{"location":"eoepca/container-registry/#values","title":"Values","text":"

The chart is configured via values that are fully documented on the Harbor website.

Example\u2026

expose:\n  ingress:\n    annotations:\n      kubernetes.io/ingress.class: nginx\n      cert-manager.io/cluster-issuer: letsencrypt-production\n      nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n\n      # from chart:\n      ingress.kubernetes.io/ssl-redirect: letsencrypt-production\n      ingress.kubernetes.io/proxy-body-size: \"0\"\n      nginx.ingress.kubernetes.io/ssl-redirect: letsencrypt-production\n      nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n\n    hosts:\n      core: harbor.192-168-49-2.nip.io\n    tls:\n      enabled: \"true\"\n      certSource: secret\n      secret:\n        secretName: \"harbor-tls\"\n\npersistence:\n  persistentVolumeClaim:\n    registry:\n      storageClass: standard\n    chartmuseum:\n      storageClass: standard\n    jobservice:\n      storageClass: standard\n    database:\n      storageClass: standard\n    redis:\n      storageClass: standard\n    trivy:\n      storageClass: standard\n\nexternalURL: https://harbor.192-168-49-2.nip.io\n# initial password for logging in with user \"admin\"\nharborAdminPassword: \"changeme\"\n\nchartmuseum:\n  enabled: false\ntrivy:\n  enabled: false\nnotary:\n  enabled: false\n

Note

  • We specify use of \u2018valid\u2019 certificates from Letsencrypt \u2018production\u2019. The Workspace API, which calls the Harbor API, expects valid certificates and will thus fail if presented with TLS certificates that fail validation.
  • The letsencrypt-production Cluster Issuer relies upon the deployment being accessible from the public internet via the expose.ingress.hosts.core DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller. The Workspace API will not like this.
"},{"location":"eoepca/container-registry/#container-registry-usage","title":"Container Registry Usage","text":"

After deployemnt Harbor is accessible via its web interface at https://harbor.192-168-49-2.nip.io/e.g. https://harbor.192-168-49-2.nip.io/.

Login as the admin user with the password specified in the helm values.

"},{"location":"eoepca/container-registry/#additional-information","title":"Additional Information","text":"

Additional information regarding the Container Registry can be found at:

  • Web Site
  • Helm Chart Repository
  • Helm Chart Description
  • Harbor Documentation
"},{"location":"eoepca/data-access/","title":"Data Access","text":"

The Data Access provides standards-based services for access to platform hosted data - including OGC WMS/WMTS for visualisation, and OGC WCS for data retrieval. This component also includes Harvester and Registrar services to discover/watch the existing data holding of the infrastructure data layer and populate/maintain the data access and resource catalogue services accordingly.

"},{"location":"eoepca/data-access/#helm-chart","title":"Helm Chart","text":"

The Data Access is deployed via the data-access helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are supplied with the instantiation of the helm release. The EOEPCA data-access chart provides a thin wrapper around the EOX View Server (vs) helm chart. The documentation for the View Server can be found here:

  • User Guide: https://vs.pages.eox.at/documentation/user/main/
  • Operator Guide: https://vs.pages.eox.at/documentation/operator/main/
helm install --version 1.4.0 --values data-access-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  data-access data-access\n
"},{"location":"eoepca/data-access/#values","title":"Values","text":"

The Data Access supports many values to configure the service. These are documented in full in the View Server - Operator Guide Configuration page.

"},{"location":"eoepca/data-access/#core-configuration","title":"Core Configuration","text":"

Typically, values for the following attributes may be specified to override the chart defaults:

  • The fully-qualified public URL for the service, ref. (global.ingress.hosts.host[0])
  • Metadata describing the service instance
  • Dynamic provisioning StorageClass for persistence
  • Persistent Volume Claims for database and redis components
  • Object storage details for data and cache
  • Container images for renderer and registrar
  • (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Data Access will not be protected by the identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.
global:\n  env:\n    REGISTRAR_REPLACE: \"true\"\n    CPL_VSIL_CURL_ALLOWED_EXTENSIONS: .TIF,.tif,.xml,.jp2,.jpg,.jpeg\n    AWS_ENDPOINT_URL_S3: https://minio.192-168-49-2.nip.io\n    AWS_HTTPS: \"FALSE\"\n    startup_scripts:\n      - /registrar_pycsw/registrar_pycsw/initialize-collections.sh\n  ingress:\n    enabled: true\n    annotations:\n      kubernetes.io/ingress.class: nginx\n      kubernetes.io/tls-acme: \"true\"\n      nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n      nginx.ingress.kubernetes.io/enable-cors: \"true\"\n      cert-manager.io/cluster-issuer: letsencrypt-production\n    hosts:\n      - host: data-access.192-168-49-2.nip.io\n    tls:\n      - hosts:\n          - data-access.192-168-49-2.nip.io\n        secretName: data-access-tls\n  storage:\n    data:\n      data:\n        type: S3\n        endpoint_url: http://data.cloudferro.com\n        access_key_id: access\n        secret_access_key: access\n        region_name: RegionOne\n        validate_bucket_name: false\n    cache:\n      type: S3\n      endpoint_url: \"https://minio.192-168-49-2.nip.io\"\n      host: \"minio.192-168-49-2.nip.io\"\n      access_key_id: xxx\n      secret_access_key: xxx\n      region: us-east-1\n      bucket: cache-bucket\n  metadata:\n    title: EOEPCA Data Access Service developed by EOX\n    abstract: EOEPCA Data Access Service developed by EOX\n    header: \"EOEPCA Data Access View Server (VS) Client powered by <a href=\\\"//eox.at\\\"><img src=\\\"//eox.at/wp-content/uploads/2017/09/EOX_Logo.svg\\\" alt=\\\"EOX\\\" style=\\\"height:25px;margin-left:10px\\\"/></a>\"\n    url: https://data-access.192-168-49-2.nip.io/ows\n  layers:\n    # see section 'Data-layer Configuration'\n  collections:\n    # see section 'Data-layer Configuration'\n  productTypes:\n    # see section 'Data-layer Configuration'\nvs:\n  renderer:\n    replicaCount: 4\n    ingress:\n      enabled: false\n    resources:\n      requests:\n        cpu: 100m\n        memory: 300Mi\n      limits:\n        cpu: 1.5\n        memory: 3Gi\n  registrar:\n    replicaCount: 1\n    config:\n      # see section 'Registrar Routes Configuration'\n    resources:\n      requests:\n        cpu: 100m\n        memory: 100Mi\n  harvester:\n    # see section 'Harvester Configuration'\n    replicaCount: 1\n    resources:\n      requests:\n        cpu: 100m\n        memory: 100Mi\n  client:\n    replicaCount: 1\n    ingress:\n      enabled: false\n  redis:\n    master:\n      persistence:\n        enabled: true\n        storageClass: standard\n  ingestor:\n    replicaCount: 0\n    ingress:\n      enabled: false\n  preprocessor:\n    replicaCount: 0\n  cache:\n    ingress:\n      enabled: false\n  scheduler:\n    resources:\n      requests:\n        cpu: 100m\n        memory: 100Mi\n

Note

The resources: above have been limited for the benefit of a minikube deployment. For a production deployment the values should be tuned (upwards) according to operational needs.

"},{"location":"eoepca/data-access/#registrar-routes-configuration","title":"Registrar Routes Configuration","text":"

The Data Access registrar component supports a number of different resource types. For each a dedicated \u2018backend\u2019 is configured to handle the specific registration of the resource type\u2026

vs:\n  registrar:\n    config:\n      #--------------\n      # Default route\n      #--------------\n      disableDefaultRoute: false\n      # Additional backends for the default route\n      defaultBackends:\n        - path: registrar_pycsw.backend.ItemBackend\n          kwargs:\n            repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n            ows_url: https://data-access.192-168-49-2.nip.io/ows\n      defaultSuccessQueue: seed_queue\n      #----------------\n      # Specific routes\n      #----------------\n      routes:\n        collections:\n          path: registrar.route.stac.CollectionRoute\n          queue: register_collection_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.CollectionBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        ades:\n          path: registrar.route.json.JSONRoute\n          queue: register_ades_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.ADESBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        application:\n          path: registrar.route.json.JSONRoute\n          queue: register_application_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.CWLBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        catalogue:\n          path: registrar.route.json.JSONRoute\n          queue: register_catalogue_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.CatalogueBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        json:\n          path: registrar.route.json.JSONRoute\n          queue: register_json_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.JSONBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n        xml:\n          path: registrar.route.json.JSONRoute\n          queue: register_xml_queue\n          replace: true\n          backends:\n            - path: registrar_pycsw.backend.XMLBackend\n              kwargs:\n                repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n
"},{"location":"eoepca/data-access/#data-layer-configuration","title":"Data-layer Configuration","text":"

Configuration of the service data-layer - as described in the View Server Operator Guide.

The data-access service data handling is configured by definition of productTypes, collections and layers\u2026

  • productTypes - Product Types Identify the underlying file assets as WCS coverages and their visual representation
  • collections - Data Collections Provides groupings into which products are organised
  • layers - Layers Specifies the hoe the product visual representations are exposed through the WMS service

For more information, see the worked example in section Data Specification for the example CREODIAS deployment.

"},{"location":"eoepca/data-access/#harvester","title":"Harvester","text":"

The Data Access service includes a Harvester component. The following subsections describe its configuration and usage.

"},{"location":"eoepca/data-access/#harvester-helm-configuration","title":"Harvester Helm Configuration","text":"

The Harvester can be configured through the helm chart values\u2026

vs:\n  harvester:\n    replicaCount: 1\n    config:\n      redis:\n        host: data-access-redis-master\n        port: 6379\n      harvesters:\n        - name: Creodias-Opensearch\n          resource:\n            url: https://datahub.creodias.eu/resto/api/collections/Sentinel2/describe.xml\n            type: OpenSearch\n            format_config:\n              type: 'application/json'\n              property_mapping:\n                start_datetime: 'startDate'\n                end_datetime: 'completionDate'\n                productIdentifier: 'productIdentifier'\n            query:\n              time:\n                property: sensed\n                begin: 2019-09-10T00:00:00Z\n                end: 2019-09-11T00:00:00Z\n              collection: null\n              bbox: 14.9,47.7,16.4,48.7\n          filter: {}\n          postprocess:\n            - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor\n          queue: register\n        - name: Creodias-Opensearch-Sentinel1\n          resource:\n            url: https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml\n            type: OpenSearch\n            format_config:\n              type: 'application/json'\n              property_mapping:\n                start_datetime: 'startDate'\n                end_datetime: 'completionDate'\n                productIdentifier: 'productIdentifier'\n            query:\n              time:\n                property: sensed\n                begin: 2019-09-10T00:00:00Z\n                end: 2019-09-11T00:00:00Z\n              collection: null\n              bbox: 14.9,47.7,16.4,48.7\n              extra_params:\n                productType: GRD-COG\n          filter: {}\n          postprocess:\n            - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel1Postprocessor\n          queue: register\n

The harvester.config.harvesters list defines a set of pre-defined harvesters which can be invoked in a later stage. The name property must be unique for each harvester and must be unique among all harvesters in the list. Each harvester is associated with a resource, an optional filter or postprocess function, and a queue.

The resource defines where each item is harvested from. This can be a file system, a search service, catalog file or something similar. The example above defines a connection to an OpenSearch service on CREODIAS, with associated default query parameters and a format configuration.

The filter allows to filter elements within the harvester, when the resource does not provide a specific filter. This filter can be supplied using CQL2-JSON.

The postprocess can adjust the harvested results. In this example the harvested items are not complete, and additional metadata must be retrieved from an object storage.

The queue defines where harvested items will be pushed into. Usually this is a registration queue, where the registrar will pick up and start registration according to its configuration.

"},{"location":"eoepca/data-access/#starting-the-harvester","title":"Starting the Harvester","text":"

The harvester can either do one-off harvests via the CLI or listen on a redis queue to run consecutive harvests whenever a harvesting request is received on that queue.

"},{"location":"eoepca/data-access/#one-off-harvests-via-the-cli","title":"One-off harvests via the CLI","text":"

In order to start a harvest from the CLI, the operator first needs to connect to the kubernetes pod of the harvester. Within that pod, the harvest can be executed like this\u2026 

python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\n

This will invoke the Creodias-Opensearch harvester with default arguments. When some values are to be overridden, the \u2013values switch can be used to pass override values. These values must be a JSON string. The following example adjusts the begin and end times of the query parameters\u2026 

python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch --values '{\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}'\n

"},{"location":"eoepca/data-access/#harvests-via-the-harvest-daemon","title":"Harvests via the harvest daemon","text":"

The harvester pod runs a service listening on a redis queue. When a message is read from the queue, it will be read as a JSON string, expecting an object with at least a name property. Optionally, it can also have a values property, working in the same way as with CLI --values.

To send a harvesting request via the redis queue, it is necessary to connect to the redis pod and execute the redis-cli there. Then the following command can be used to achieve the same result as above with CLI harvesting\u2026 

redis-cli LPUSH '{\"name\": \"Creodias-Opensearch\", \"values\": {\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}}'\n

"},{"location":"eoepca/data-access/#results-of-the-harvesting","title":"Results of the harvesting","text":"

The harvester produces a continous stream of STAC Items which are sent down via the configured queue. It is possible that the harvested metadata is not sufficient to create a fully functional STAC Item. In this case the postprocess must transform this intermediate item to a valid STAC Item. In our example, the postprocessor looks up the Sentinel-2 product file referenced by the product identifier which is then accessed on the object storage. From the stored metadata files, the STAC Items to be sent is created.

"},{"location":"eoepca/data-access/#storage","title":"Storage","text":"

Specification of PVCs and access to object storage.

"},{"location":"eoepca/data-access/#persistent-volume-claims","title":"Persistent Volume Claims","text":"

The PVCs specified in the helm chart values must be created.

"},{"location":"eoepca/data-access/#pvc-for-database","title":"PVC for Database","text":"
kind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n  name: data-access-db\n  namespace: rm\n  labels:\n    k8s-app: data-access\n    name: data-access\nspec:\n  storageClassName: standard\n  accessModes:\n    - ReadWriteMany\n  resources:\n    requests:\n      storage: 100Gi\n
"},{"location":"eoepca/data-access/#pvc-for-redis","title":"PVC for Redis","text":"
kind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n  name: data-access-redis\n  namespace: rm\n  labels:\n    k8s-app: data-access\n    name: data-access\nspec:\n  storageClassName: standard\n  accessModes:\n    - ReadWriteMany\n  resources:\n    requests:\n      storage: 1Gi\n
"},{"location":"eoepca/data-access/#object-storage","title":"Object Storage","text":"

The helm chart values expect specification of object storage details for:

  • data: to access the EO data of the underlying infrastructure
  • cache: a dedicated object storage bucket is used to support the cache function of the data access services
"},{"location":"eoepca/data-access/#platform-eo-data","title":"Platform EO Data","text":"

Specifies the details for the infrastructure object storage that provides direct access to the EO product files.

For example, the CREODIAS metadata catalogue provides references to product files in their eodata object storage - the access details for which are configured in the data access services:

global:\n  storage:\n    data:\n      data:\n        type: S3\n        endpoint_url: http://data.cloudferro.com\n        access_key_id: access\n        secret_access_key: access\n        region_name: RegionOne\n        validate_bucket_name: false\n
"},{"location":"eoepca/data-access/#data-access-cache","title":"Data Access Cache","text":"

The Data Access services maintain a cache, which relies on the usage of a dedicate object storage bucket for data persistence. This bucket must be created (manual step) and its access details configured in the data access services. Example based upon CREODIAS:

global:\n  storage:\n    cache:\n      type: S3\n      endpoint_url: \"https://cf2.cloudferro.com:8080/cache-bucket\"\n      host: \"cf2.cloudferro.com:8080\"\n      access_key_id: xxx\n      secret_access_key: xxx\n      region: RegionOne\n      bucket: cache-bucket\n

\u2026where xxx must be replaced with the bucket credentials.

"},{"location":"eoepca/data-access/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the data-access service to provide access authorization decisions

"},{"location":"eoepca/data-access/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install data-access-protection identity-gatekeeper -f data-access-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"rm\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the data-access - in particular the specific ingress requirements for the data-access backend services\u2026

Example data-access-protection-values.yaml\u2026

fullnameOverride: data-access-protection\nconfig:\n  client-id: data-access\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: data-access.192-168-49-2.nip.io\n  name: data-access-renderer\n  port:\n    number: 80\nsecrets:\n  # Values for secret 'data-access-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n    nginx.ingress.kubernetes.io/enable-cors: \"true\"\n    nginx.ingress.kubernetes.io/rewrite-target: /$1\n  serverSnippets:\n    custom: |-\n      # Open access to renderer...\n      location ~ ^/(ows.*|opensearch.*|coverages/metadata.*|admin.*) {\n        proxy_pass http://data-access-renderer.rm.svc.cluster.local:80/$1;\n      }\n      # Open access to cache...\n      location ~ ^/cache/(.*) {\n        proxy_pass http://data-access-cache.rm.svc.cluster.local:80/$1;\n      }\n      # Open access to client...\n      # Note that we use a negative lookahead to avoid matching '/.well-known/*' which\n      # otherwise appears to interfere with the work of cert-manager/letsencrypt.\n      location ~ ^/(?!\\.well-known)(.*) {\n        proxy_pass http://data-access-client.rm.svc.cluster.local:80/$1;\n      }\n
"},{"location":"eoepca/data-access/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: data-access above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=data-access \\\n  --name=\"Data Access Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Data Access Gatekeeper\"\n
"},{"location":"eoepca/data-access/#data-access-usage","title":"Data Access Usage","text":""},{"location":"eoepca/data-access/#default-harvesting","title":"Default Harvesting","text":"

At deployment time the harvester helm values include configuration that populates a default harvester configuration, that is prepared in the file /config.yaml in the harvester pod.

The Data Access and Resource Catalogue services are configured to properly interpret harvested data via these values specified in the instantiation of the helm release. See section Data-layer Configuration.

The harvesting of data can be triggered (post deployment), in accordance with this default configuration, by connecting to the rm/harvester service and executing the command\u2026 

python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\n

"},{"location":"eoepca/data-access/#ad-hoc-harvesting","title":"Ad-hoc Harvesting","text":"

Ad-hoc harvesting can be invoked by provision of a suitable config.yaml into the harvester pod, which can then be invoked as shown above for the default harvester configuration established at deploy time.

The helper script ./deploy/bin/harvest faciltates this\u2026

./deploy/bin/harvest <path-to-config-file>\n

See directory ./deploy/samples/harvester/ that contains some sample harvesting configuration files. For example\u2026

./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2019.09.10.yaml\n
"},{"location":"eoepca/data-access/#registration-of-collections","title":"Registration of Collections","text":"

The helper script ./deploy/bin/register-collection is provided to faciltate the registration of collections that are specfied in STAC Collection format.

./deploy/bin/register-collection <path-to-stac-collection-file>\n

See directory ./deploy/samples/collections/ that contains some same STAC Collection files. For example\u2026

./deploy/bin/register-collection ./deploy/samples/collections/S2MSI2A.json\n
"},{"location":"eoepca/data-access/#additional-information","title":"Additional Information","text":"

Additional information regarding the Data Access can be found at:

  • Helm Chart
  • Documentation:
    • User Guide
    • Operator Guide
  • Git Repository
"},{"location":"eoepca/iam-overview/","title":"IAM Overview","text":"

This guide includes two approaches for Identity & Access Management:

  • Keycloak Solution (NEW)
  • Gluu Solution (deprecated)

Until now, our IAM solution has been based solely upon Gluu.

In the course of the project Keycloak has emerged as a preferred solution across EO platforms.

Thus, we have introduced an IAM approach based upon Keycloak, whilst retaining the Gluu-based approach for reference, which will be deprecated.

"},{"location":"eoepca/identity-service/","title":"Identity Service","text":"

The Identity Service provides the platform Authorization Server for authenticated user identity and request authorization.

Identity Service is composed of:

  • Keycloak IAM Authorization Service - supporting OpenID Connect (OIDC), etc.
  • Postgres DB Relational database used by Keycloak for persistence
  • Identity API Service that provided a convenience API to simplify IAM management interactions with Keycloak. Provides endpoints to create clients and protect resources. Uses a keycloak python client which sends requests to Keycloak API
  • Identity API Gatekeeper Instance of Gatekeeper to \u2018protect\u2019 access requests to the Identity API service. Gatekeeper is a reusable component that provides the Policy Enforcement for requests to individual resource servers. A Gatekeeper instance should be configured and deployed for each application that requires protection by access policies.
"},{"location":"eoepca/identity-service/#helm-chart","title":"Helm Chart","text":"

The Identity Service is deployed via the identity-service helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values - the full set of available values can be tailored according the helm chart defaults, that can be found here\u2026

  • identity-service https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/values.yaml
  • identity-keycloak https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-keycloak/values.yaml
  • identity-postgres https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-postgres/values.yaml
  • identity-api https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-api/values.yaml
helm install --version 1.0.97 --values identity-service-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  identity-service identity-service\n
"},{"location":"eoepca/identity-service/#values","title":"Values","text":"

The deployment must be configured for you environment. Some significant configuration values are elaborated here\u2026

"},{"location":"eoepca/identity-service/#identity-keycloak","title":"identity-keycloak","text":""},{"location":"eoepca/identity-service/#secrets","title":"Secrets","text":"

Keycloak relies upon a secret identity-keycloak that provides\u2026

  • KEYCLOAK_ADMIN_PASSWORD - admin password for Keycloak
  • KC_DB_PASSWORD - password for connecting with Postgres DB This should match the POSTGRES_PASSWORD setting for identity-postgres (see below)

The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026

identity-keycloak:\n  secrets:\n    # Values for secret 'identity-keycloak'\n    # Note - if ommitted, these can instead be set by creating the secret independently.\n    kcDbPassword: \"changeme\"\n    keycloakAdminPassword: \"changeme\"\n
"},{"location":"eoepca/identity-service/#ingress","title":"Ingress","text":"

The details for ingress (reverse-proxy) to the Keycloak service - in particular the hostname and possible TLS - must be specified\u2026

identity-keycloak:\n  ingress:\n    enabled: true\n    className: nginx\n    annotations:\n      ingress.kubernetes.io/ssl-redirect: \"true\"\n      nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n      cert-manager.io/cluster-issuer: letsencrypt-production\n    hosts:\n      - host: keycloak.192-168-49-2.nip.io\n        paths:\n          - path: /\n            pathType: Prefix\n    tls:\n      - secretName: identity-keycloak-tls\n        hosts:\n          - keycloak.192-168-49-2.nip.io\n
"},{"location":"eoepca/identity-service/#identity-postgres","title":"identity-postgres","text":""},{"location":"eoepca/identity-service/#secrets_1","title":"Secrets","text":"

Postgres relies upon a secret identity-postgres that provides\u2026

  • POSTGRES_PASSWORD - superuser password for PostgreSQL
  • PGPASSWORD - password used for client connections to the DB

The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026

identity-postgres:\n  secrets:\n    # Values for secret 'identity-postgres'\n    # Note - if ommitted, these can instead be set by creating the secret independently.\n    postgresPassword: \"changeme\"\n    pgPassword: \"changeme\"\n
"},{"location":"eoepca/identity-service/#persistence","title":"Persistence","text":"

In order to persist data, Postgres requires a Persistent Volume Claim.

This can be specified as an existing volume claim - for example as described in the Persistence section.

identity-postgres:\n  volumeClaim:\n    name: eoepca-userman-pvc\n
"},{"location":"eoepca/identity-service/#identity-api","title":"identity-api","text":""},{"location":"eoepca/identity-service/#secrets_2","title":"Secrets","text":"

The Identity API relies upon a secret identity-api that provides\u2026

  • ADMIN_PASSWORD Admin password for Keycloak This should match the KEYCLOAK_ADMIN_PASSWORD setting for identity-keycloak (see above)

The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026

identity-api:\n  secrets:\n    # Values for secret 'identity-api'\n    # Note - if ommitted, these can instead be set by creating the secret independently\n    # e.g. as a SealedSecret via GitOps.\n    adminPassword: \"changeme\"\n

Note

It is also possible to set the value of ADMIN_PASSWORD directly as an environment variable. In this case it is necessary to set the secret as optional\u2026

identity-api:\n  secrets:\n    optional: true\n
"},{"location":"eoepca/identity-service/#environment-variables","title":"Environment Variables","text":"

The Identity API service can be configured via environment variables as follows\u2026

  • AUTH_SERVER_URL URL of the Keycloak Authorization Server. Can also be set via value configMap.authServerUrl
  • ADMIN_USERNAME Admin user for Keycloak
  • REALM The Keycloak realm
identity-api:\n  deployment:\n    # Config values that can be passed via env vars\n    extraEnv:\n      - name: AUTH_SERVER_URL  # see configMap.authServerUrl instead\n        value: https://keycloak.192-168-49-2.nip.io\n      - name: ADMIN_USERNAME\n        value: admin\n      - name: ADMIN_PASSWORD  # see secrets.adminPassword instead\n        value: changeme\n      - name: REALM\n        value: master\n
"},{"location":"eoepca/identity-service/#identity-api-gatekeeper","title":"identity-api-gatekeeper","text":""},{"location":"eoepca/identity-service/#secrets_3","title":"Secrets","text":"

gatekeeper relies upon a secret identity-api-protection that provides\u2026

  • PROXY_CLIENT_SECRET Password for the Keycloak client configured for use by this Gatekeeper instance - corresponding to config.client-id.
  • PROXY_ENCRYPTION_KEY Encryption Key used by Gatekeeper.

The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026

identity-api-gatekeeper:\n  secrets:\n    # Values for secret 'identity-api-protection'\n    # Note - if ommitted, these can instead be set by creating the secret independently.\n    clientSecret: \"changeme\"\n    encryptionKey: \"changemechangeme\"\n
"},{"location":"eoepca/identity-service/#configuration","title":"Configuration","text":"

Configuration of Gatekeeper via the file config.yaml that is mounted into the deployment\u2026

  • client-id ID of the Keycloak client to be used by this Gatekeeper instance.
  • discovery-url Discovery URL of the Keycloak Authorization Server
  • cookie-domain Domain in which this Gatekeeper instance creates cookies 
identity-api-gatekeeper:\n  config:\n    client-id: identity-api\n    discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n    cookie-domain: 192-168-49-2.nip.io\n
"},{"location":"eoepca/identity-service/#ingress_1","title":"Ingress","text":"

The details for ingress (reverse-proxy) to the Gatekeeper service that protects the Identity API\u2026

identity-api-gatekeeper:\n  targetService:\n    host: identity-api.192-168-49-2.nip.io\n  ingress:\n    annotations:\n      ingress.kubernetes.io/ssl-redirect: \"true\"\n      nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n      cert-manager.io/cluster-issuer: letsencrypt\n
"},{"location":"eoepca/identity-service/#identity-api-client","title":"Identity API Client","text":"

The Identity API is protected via an instance of Gatekeeper - which relies upon a Keycloak client having been created for authorization decision/enforcement flows between Gatekeeper and Keycloak.

As described in the \u2018create-client\u2019 section below, this can be achieved using the create-client helper script.

Note

At time of client creation, the Identity API is not yet protected with an ingress. Therefore, we use a port-forward to interface directly with the Identity API service.

$ kubectl -n um port-forward svc/identity-api \"9876\":http >/dev/null &\n$ portForwardPid=$!\n\n$ ./deploy/bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i http://localhost:9876 \\\n  -r master \\\n  -u admin \\\n  -p changeme \\\n  -c admin-cli \\\n  --id=identity-api \\\n  --name=\"Identity API Gatekeeper\" \\\n  --secret=changeme \\\n  --description=\"Client to be used by Identity API Gatekeeper\" \\\n  --resource=\"admin\" --uris='/*' --scopes=view --users=\"admin\"\n\n$ kill -TERM $portForwardPid\n
"},{"location":"eoepca/identity-service/#create-user-helper-script","title":"create-user Helper Script","text":"

The Keycloak Admin UI can be used to create users interactively.

Alternatvely there is a helper script create-user that can be used.

The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The create-user helper script requires some command-line arguments\u2026

$ ./deploy/bin/create-user -h\n\nCreate a new user.\ncreate-user -h | -a {auth_server} -r {realm} -c {client} -u {admin-username} -p {admin-password} -U {new-username} -P {new-password}\n\nwhere:\n    -h  show help message\n    -a  authorization server url (default: http://keycloak.192-168-49-2.nip.io)\n    -r  realm within Keycloak (default: master)\n    -u  username used for authentication (default: admin)\n    -p  password used for authentication (default: changeme)\n    -c  client id of the bootstrap client used in the create request (default: admin-cli)\n    -U  name of the (new) user to create\n    -P  password for the (new) user to create\n
"},{"location":"eoepca/identity-service/#protection-of-resources","title":"Protection of Resources","text":"

The Identity Service is capable of protecting resources using OpenID-connect/SAML clients, resources (URIs/scopes), policies (user based, role based, etc) and permissions (associations between policies and resources).

Creating and protecting resources can be done in multiple ways, as described in the following sections.

"},{"location":"eoepca/identity-service/#keycloak-admin-ui","title":"Keycloak Admin UI","text":"

To create and protect resources using the keycloak User Interface (UI), do the following steps:

  • (Optional) Create clients. Clients can be created using the keycloak user interface at http://keycloak.192-168-49-2.nip.io. You need to login as admin. To create a client: Login as admin in the keycloak UI > Clients > Create Client > Set a name > Next > Turn Client Authentication and Authorization On > Add the valid redirect URI\u2019s > Save.
  • (Optional) Create Users. Users > Add User. Then set a password for the user. Credentials > Set Password.
  • Select a client.
  • Create a Resource: Select Authorization tab > Resources > Create Resource.
  • Create a Policy: In client details, select Authorization > Policies > Create Policy > Select Policy Type (e.g.: User) > Select users > Save.
  • Create Authorization Scope: In client details, select Authorization > Scopes > Create authorization scope > Save.
  • Create a Permission: In client details, select Authorization > Permissions > Create Permission > Create Resource Based Permission > Select Resources to protect > Select Policies > Save.
"},{"location":"eoepca/identity-service/#create-client-helper-script","title":"create-client Helper Script","text":"

Alternatively, a script was developed to allow simultaneaously create a client, create resources and protect them.

The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The create-client helper script requires some command-line arguments\u2026

$ ./deploy/bin/create-client -h\n\nAdd a client with protected resources.\ncreate-client [-h] [-a] [-i] [-u] [-p] [-c] [-s] [-t | --token t] [-r] --id id [--name name] (--secret secret | --public) [--default] [--authenticated] [--resource name] [--uris u1,u2] [--scopes s1,s2] [--users u1,u2] [--roles r1,r2]\n\nwhere:\n    -h                    show help message\n    -a                    authorization server url - e.g. https://keycloak.192-168-49-2.nip.io\n    -i                    identity-api server url - e.g. https://identity-api.192-168-49-2.nip.io\n    -u                    username used for authentication\n    -p                    password used for authentication\n    -c                    client id (of the bootstrap client used in the create request)\n    -s                    client secret (of the bootstrap client used in the create request)\n    -t or --token         access token used for authentication\n    -r                    realm\n    --id                  client id (of the created client)\n    --name                client name (of the created client)\n    --secret              client secret (of the created client)\n    --public              public client (no client secret)\n    --default             add default resource - /* authenticated\n    --authenticated       allow access to the resource only when authenticated\n    --resource            resource name\n    --uris                resource uris - separated by comma (,)\n    --scopes              resource scopes - separated by comma (,)\n    --users               user names with access to the resource - separated by comma (,)\n    --roles               role names with access to the resource - separated by comma (,)\n

The script interacts with Identity API and therefore requires admin authorization. It accepts basic authentication with username and password with -u and -p parameters, respectively - or a bearer access token with -t parameter.

To generate the access token needed to use the script, you can get it through the login in the eoepca portal, by accessing the cookies in the browser. See section EOEPCA Portal for details regarding deployment/configuration of the eoepca-portal.

Or you can generate an access token using postman oauth2.0, as described in the Postman document Requesting an OAuth 2.0 token.

Script execution examples:

  1. With username/password 

    ./deploy/bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r master \\\n  -u admin \\\n  -p changeme \\\n  -c admin-cli \\\n  --id=myservice-gatekeeper \\\n  --name=\"MyService Gatekeeper\" \\\n  --secret=changeme \\\n  --description=\"Client to be used by MyService Gatekeeper\" \\\n  --resource=\"Eric space\" --uris=/eric/* --users=eric \\\n  --resource=\"Alice space\" --uris=/alice/* --users=alice \\\n  --resource=\"Admin space\" --uris=/admin/* --roles=admin\n

  2. With access token 

    ./deploy/bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r master \\\n  -t eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJXZWFIY2pscThPc1RUYjdlV0s5SjJTTDFBUDIyazZpajdlMGFlVHRNU2xRIn0.eyJleHAiOjE3MDAyNDM4MzgsImlhdCI6MTcwMDI0Mzc3OCwiYXV0aF90aW1lIjoxNzAwMjQxODYyLCJqdGkiOiI2MWI0ZGRhYy1mOWZjLTRmZjktOWQ4Zi01NWU1N2NlNmE5ODgiLCJpc3MiOiJodHRwczovL2lkZW50aXR5LmtleWNsb2FrLmRldmVsb3AuZW9lcGNhLm9yZy9yZWFsbXMvbWFzdGVyIiwiYXVkIjpbImFkZXMtcmVhbG0iLCJkZW1vLXJlYWxtIiwiZHVtbXktc2VydmljZS1yZWFsbSIsIm1hc3Rlci1yZWFsbSIsImFjY291bnQiLCJlb2VwY2EtcmVhbG0iXSwic3ViIjoiZTNkZTMyNGUtMGY0NS00MWUwLTk2YTctNTM1YzkxMTA1NTUyIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiZW9lcGNhLXBvcnRhbCIsIm5vbmNlIjoiMTIwMGJlNzAtZWI1Ni00Nzc2LThjODgtOWRiOWQxMDdiMGY2Iiwic2Vzc2lvbl9zdGF0ZSI6ImVmNGUwOTlmLTFmMDgtNDY3MC04ZmE2LTJiOGI3OGUwNWMzMSIsImFjciI6IjAiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiY3JlYXRlLXJlYWxtIiwiZGVmYXVsdC1yb2xlcy1tYXN0ZXIiLCJvZmZsaW5lX2FjY2VzcyIsImFkbWluIiwidW1hX2F1dGhvcml6YXRpb24iLCJ1c2VyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWRlcy1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LWlkZW50aXR5LXByb3ZpZGVycyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwiZGVtby1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LXJlYWxtIiwidmlldy1pZGVudGl0eS1wcm92aWRlcnMiLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwiZHVtbXktc2VydmljZS1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LXJlYWxtIiwidmlldy1pZGVudGl0eS1wcm92aWRlcnMiLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwibWFzdGVyLXJlYWxtIjp7InJvbGVzIjpbInZpZXctaWRlbnRpdHktcHJvdmlkZXJzIiwidmlldy1yZWFsbSIsIm1hbmFnZS1pZGVudGl0eS1wcm92aWRlcnMiLCJpbXBlcnNvbmF0aW9uIiwiY3JlYXRlLWNsaWVudCIsIm1hbmFnZS11c2VycyIsInF1ZXJ5LXJlYWxtcyIsInZpZXctYXV0aG9yaXphdGlvbiIsInF1ZXJ5LWNsaWVudHMiLCJxdWVyeS11c2VycyIsIm1hbmFnZS1ldmVudHMiLCJtYW5hZ2UtcmVhbG0iLCJ2aWV3LWV2ZW50cyIsInZpZXctdXNlcnMiLCJ2aWV3LWNsaWVudHMiLCJtYW5hZ2UtYXV0aG9yaXphdGlvbiIsIm1hbmFnZS1jbGllbnRzIiwicXVlcnktZ3JvdXBzIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX0sImVvZXBjYS1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LWlkZW50aXR5LXByb3ZpZGVycyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfX0sInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJzaWQiOiJlZjRlMDk5Zi0xZjA4LTQ2NzAtOGZhNi0yYjhiNzhlMDVjMzEiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInByZWZlcnJlZF91c2VybmFtZSI6ImFkbWluIn0.FK6DhVzpCRFmef2acD2Hmc149e1GTOCGz13dZA828crFbG8j4uhpkoNpiZqdyOPmDtMQ-OebNfjTAUaOt2sS1FmEIBgb9IddcpHKNJOquRjdzQNsX09bX8pFUq1haGwKh6_QmABNOBcT-kQNDSZO-aq7-8FoO9PYa0GWvBRcbcx0W_ngyb7xHglaZTElzcDPBcUTW6llVTTTFygn55smwdxTZ7-tEsMVGM5gNuHwJyLB51HI5KDWrwgUm1hqhhRzvcoutDEAB_HSEXGNNeF7fjP9Qx6q04b7fKOTtnIlXsu3oYW4va9y754llMSJ7w8U-y7yI6Tm2UdNMdYqju7hAA \\\n  -c admin-cli \\\n  --id=myservice-gatekeeper \\\n  --name=\"MyService Gatekeeper\" \\\n  --secret=changeme \\\n  --description=\"Client to be used by MyService Gatekeeper\" \\\n  --resource=\"Eric space\" --uris=/eric/* --users=eric \\\n  --resource=\"Alice space\" --uris=/alice/* --users=alice \\\n  --resource=\"Admin space\" --uris=/admin/* --roles=admin\n

"},{"location":"eoepca/identity-service/#using-identity-api","title":"Using Identity API","text":"

Also, an API was developed to interact more easily with the Keycloak API, that allows client, resource, policies and permissions management.

The API documentation can be found in its Swagger UI at the service endpoint - https://identity-api.192-168-49-2.nip.io/docs.

The Identity API is best used in combination with the eoepca-portal test aide, which can be used to establish a login sesssion in the browser to the benefit of the Identity API swagger UI. See section EOEPCA Portal for details regarding deployment/configuration of the eoepca-portal.

"},{"location":"eoepca/identity-service/#token-lifespans","title":"Token Lifespans","text":"

By default the Access Token Lifespan is 1 minute. With the current ADES (zoo-dru) implementation this presents a problem - since the access_token that is provided to the process execute request will (most likely) have expired by the time the ADES attempts to use the access_token in its call to the Workspace API to register the processing outputs. The lifespan of the token must outlive the duration of the processing execution - which we must assume can take a long time.

To avoid this potential problem, the Keycloak Admin web console can be used to increase this token lifespan.

Thus, the following settings are recommended to be updated following deployment\u2026

  • SSO Session Settings /admin/master/console/#/master/realm-settings/sessions
  • SSO Session Idle: 1 day
  • SSO Session Max: 1 day
  • Tokens /admin/master/console/#/master/realm-settings/tokens
  • Access Token Lifespan: 1 day
"},{"location":"eoepca/identity-service/#additional-information","title":"Additional Information","text":"

Additional information regarding the Identity Service can be found at:

  • Helm Chart
  • GitHub Repository
"},{"location":"eoepca/login-service/","title":"Login Service","text":"

The Login Service provides the platform Authorization Server for authenticated user identity and request authorization.

"},{"location":"eoepca/login-service/#helm-chart","title":"Helm Chart","text":"

The Login Service is deployed via the login-service helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the login-service chart.

helm install --version 1.2.8 --values login-service-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  login-service login-service\n
"},{"location":"eoepca/login-service/#values","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io
  • IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2
  • Kubernetes namespace for the login-service components
  • Initial password for the admin user Note that the password must meet the complexity: at least 6 characters and include one uppercase letter, one lowercase letter, one digit, and one special character
  • Name of Persistent Volume Claim for login-service persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.
  • TLS Certificate Provider, e.g. letsencrypt-production

Example login-service-values.yaml\u2026 

global:\n  domain: auth.192-168-49-2.nip.io\n  nginxIp: 192.168.49.2\n  namespace: um\nvolumeClaim:\n  name: eoepca-userman-pvc\n  create: false\nconfig:\n  domain: auth.192-168-49-2.nip.io\n  adminPass: Chang3me!\n  ldapPass: Chang3me!\n  volumeClaim:\n    name: eoepca-userman-pvc\nopendj:\n  volumeClaim:\n    name: eoepca-userman-pvc\n  resources:\n    requests:\n      cpu: 100m\n      memory: 300Mi\noxauth:\n  volumeClaim:\n    name: eoepca-userman-pvc\n  resources:\n    requests:\n      cpu: 100m\n      memory: 1000Mi\noxtrust:\n  volumeClaim:\n    name: eoepca-userman-pvc\n  resources: \n    requests:\n      cpu: 100m\n      memory: 1500Mi\noxpassport:\n  resources:\n    requests:\n      cpu: 100m\n      memory: 100Mi\nnginx:\n  ingress:\n    annotations:\n      cert-manager.io/cluster-issuer: letsencrypt-production\n    hosts:\n      - auth.192-168-49-2.nip.io\n    tls:\n      - hosts:\n          - auth.192-168-49-2.nip.io\n        secretName: login-service-tls\n

Note

The resources: above have been limited for the benefit of a minikube deployment. For a production deployment the values should be tuned (upwards) according to operational needs.

"},{"location":"eoepca/login-service/#post-deployment-manual-steps","title":"Post-deployment Manual Steps","text":"

The deployment of the Login Service has been designed, as far as possible, to automate the configuration. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026

  • Configure UMA Resource Lifetime
  • Configure Operator user
"},{"location":"eoepca/login-service/#uma-resource-lifetime","title":"UMA Resource Lifetime","text":"

The Login Service maintains a background service that \u2018cleans\u2019 UMA resources that are older than aa certain age - by default 30 days (2592000 secs). This lifetime does not fit the approach we are adopting, and so we must update this lifetime value to avoid the unexpected removal of UMA resources that would cause unexpected failures in policy enforcement.

  • In a browser, navigate to the Login Service (Gluu) - https://auth.192-168-49-2.nip.io/ - and login as the admin user
  • Open Configuration -> JSON Configuration -> OxAuth Configuration
  • Search for the setting umaResourceLifetime
  • Update the values of umaResourceLifetime to 2147483647
  • Select to Save Configuration
  • Restart the oxauth deployment\u2026 
    kubectl -n um rollout restart deploy/login-service-oxauth\n
"},{"location":"eoepca/login-service/#configure-operator-user","title":"Configure Operator user","text":"

The default resource protection establishes policy in which \u2018operator\u2019 privilege is required for some services, such as the Workspace API. Thus, we need to configure a user with this privilege. For convenience we add this attribute to the built-in admin user - but alternatively you may choose to create a new user for this role.

  • In a browser, navigate to the Login Service (Gluu) - https://auth.192-168-49-2.nip.io/ - and login as the admin user
  • Select Users -> Manage People and search for user admin
  • For user admin select Available User Claims -> gluuCustomPerson
  • Select Is Operator and ensure the value is set True
  • Select Update to confirm
"},{"location":"eoepca/login-service/#login-service-usage","title":"Login Service Usage","text":"

Once the deployment has been completed successfully, the Login Service is accessed at the endpoint https://auth.192-168-49-2.nip.io/, configured by your domain - e.g. https://auth.192-168-49-2.nip.io/.

Login as the admin user with the credentials configured in the helm values - ref. adminPass / ldapPass.

Typical first actions to undertake through the Gluu web interface include creation of users and clients.

"},{"location":"eoepca/login-service/#additional-information","title":"Additional Information","text":"

Additional information regarding the Login Service can be found at:

  • Helm Chart
  • Wiki
  • GitHub Repository
"},{"location":"eoepca/pdp/","title":"Policy Decision Point","text":"

The Policy Decision Point (PDP) provides the platform policy database and associated service for access policy decision requests.

"},{"location":"eoepca/pdp/#helm-chart","title":"Helm Chart","text":"

The PDP is deployed via the pdp-engine helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the pdp-engine chart.

helm install --version 1.1.12 --values pdp-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  pdp pdp-engine\n
"},{"location":"eoepca/pdp/#values","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io
  • IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2
  • Name of Persistent Volume Claim for pdp-engine persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.

Example pdp-values.yaml\u2026 

global:\n  nginxIp: 192.168.49.2\n  domain: auth.192-168-49-2.nip.io\nvolumeClaim:\n  name: eoepca-userman-pvc\n  create: false\n

"},{"location":"eoepca/pdp/#additional-information","title":"Additional Information","text":"

Additional information regarding the PDP can be found at:

  • Helm Chart
  • Wiki
  • GitHub Repository
"},{"location":"eoepca/persistence/","title":"Persistence","text":""},{"location":"eoepca/persistence/#overview","title":"Overview","text":"

The EOEPCA building-blocks rely upon Kubernetes Persistent Volumes for their component persistence. Components integrate with the storage provided in the cluster by means of configurable Persistent Volume Claims and/or dynamic Storage Class that are specfied as values at time of deployment. Some components require storage of type ReadWriteMany - which, for a multi-node cluster, implies a network-based storage solution.

Note

Local CLuster Storage For the purposes of the Scripted Deployment, the default Storage Class included with the local Kubernetes distribution can be used for all storage concerns - e.g. standard for minikube which provides the ReadWriteMany persistence that is required by the ADES.

"},{"location":"eoepca/persistence/#readwritemany-storage","title":"ReadWriteMany Storage","text":"

For the EOEPCA development deployment, an NFS server has been established to provide the persistence layer for ReadWriteMany storage.

"},{"location":"eoepca/persistence/#pre-defined-persistent-volume-claims","title":"Pre-defined Persistent Volume Claims","text":"

The EOEPCA development deployment establishes the following pre-defined Persistent Volume Claims, to provide a simple storage architecture that is organised around the \u2018domain areas\u2019 into which the Reference Implementation is split.

  • Resource Managment (resman) - persistentvolumeclaim/eoepca-resman-pvc
  • Processing & Chaining (proc) - persistentvolumeclaim/eoepca-proc-pvc
  • User Management (userman) - persistentvolumeclaim/eoepca-userman-pvc

NOTE that this is offered only as an example thay suits the approach of the development team. Each building-block has configuration through which its persistence (PV/PVC) can be configured according the needs of the deployment.

The following Kubernetes yaml provides an example of provisioning such domain-specific PersistentVolumeClaims within the cluster - in this case using the minikube built-in storage-class standard for dynamic provisioning\u2026

---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: eoepca-proc-pvc\n  namespace: proc\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: standard\n  resources:\n    requests:\n      storage: 5Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: eoepca-resman-pvc\n  namespace: rm\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: standard\n  resources:\n    requests:\n      storage: 5Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n  name: eoepca-userman-pvc\n  namespace: um\nspec:\n  accessModes:\n    - ReadWriteMany\n  storageClassName: standard\n  resources:\n    requests:\n      storage: 5Gi\n

Once established, these PersistentVolumeClaims are then referenced within the deployment configurations of the building-blocks.

"},{"location":"eoepca/persistence/#dynamic-readwritemany-storage-provisioning","title":"Dynamic ReadWriteMany Storage Provisioning","text":"

In addition to the pre-defined PV/PVCs, the EOEPCA Reference Implementation also defines NFS-based storage classes for dynamic storage provisioning:

  • managed-nfs-storage With a Reclaim Policy of Delete.
  • managed-nfs-storage-retain With a Reclaim Policy of Retain.

The building-blocks simply reference the required Storage Class in their volume specifications, to receive a Persistent Volume Claim that is dynamically provisioned at deployment time.

This is acheived through the nfs-provisioner helm chart, with the following typical configurations\u2026

Reclaim Policy Delete\u2026 

provisionerName: nfs-storage\nstorageClass:\n  name: managed-nfs-storage\n  create: true\n  reclaimPolicy: Delete\n  archiveOnDelete: false\n  allowVolumeExpansion: true\nnfs:\n  server: \"<your-nfs-ip-address-here>\"\n  path: /data/dynamic  # your NFS server path here\n

Reclaim Policy Retain\u2026 

provisionerName: nfs-storage-retain\nstorageClass:\n  name: managed-nfs-storage-retain\n  create: true\n  reclaimPolicy: Retain\n  allowVolumeExpansion: true\nnfs:\n  server: \"<your-nfs-ip-address-here>\"\n  path: /data/dynamic  # your NFS server path here\n

"},{"location":"eoepca/persistence/#clustered-storage-solutions","title":"Clustered Storage Solutions","text":"

Clustered storage approaches offer an alternative to NFS. Clustered Storage provides a network-attached storage through a set of commodity hosts whose storage is aggregated to form a distributed file-system. Capacity is scaled by adding additional nodes or adding additional storage to the existing nodes. In the context of a multi-node Kubernetes cluster, then it is typical that the same commodity nodes provide both the cluster members and storage resources, i.e. the clustered storage is spread across the Kubernetes worker nodes.

Candidate clustered storage solutions include:

  • GlusterFS GlusterFS is deployed as an operating system service across each node participating in the storage solution. Thus, with GlusterFS, the distributed storage nodes do not need to be one-and-the-same with the compute (cluster) nodes \u2013 although this may preferably be the case.
  • Longhorn Longhorn offers a solution that is similar to that of GlusterFS, except that Longhorn is \u2018cloud-native\u2019 in that its service layer deploys within the Kubernetes cluster itself. Thus, the storage nodes are also the cluster compute nodes by design.

All things being equal, Longhorn is recommended as the best approach for Kubernetes clusters.

"},{"location":"eoepca/registration-api/","title":"Registration API","text":"

The Registration API provides a REST API through which resources can be registered with both the Resource Catalogue and (as applicable) with the Data Access services.

"},{"location":"eoepca/registration-api/#helm-chart","title":"Helm Chart","text":"

The Registration API is deployed via the rm-registration-api helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the rm-registration-api chart.

helm install --version 1.4.0 --values registration-api-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  registration-api rm-registration-api\n
"},{"location":"eoepca/registration-api/#values","title":"Values","text":"

The Registration API supports many values to configure the service - as described in the Values section of the chart README.

Typically, values for the following attributes may be specified:

  • The fully-qualified public URL for the service
  • (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Registration API will not be protected by the identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.
  • Values for integration with the workspace-api and data-access services

Example registration-api-values.yaml\u2026

fullnameOverride: registration-api\n\ningress:\n  enabled: false\n  hosts:\n    - host: registration-api-open.192-168-49-2.nip.io\n      paths: [\"/\"]\n  tls:\n    - hosts:\n        - registration-api-open.192-168-49-2.nip.io\n      secretName: registration-api-tls\n\n# some values for the workspace API\nworkspaceK8sNamespace: rm\nredisServiceName: \"data-access-redis-master\"\n
"},{"location":"eoepca/registration-api/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the registration-api service to provide access authorization decisions

"},{"location":"eoepca/registration-api/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install registration-api-protection identity-gatekeeper -f registration-api-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"rm\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the registration-api - in particular the specific ingress requirements for the registration-api backend service\u2026

Example registration-api-protection-values.yaml\u2026

fullnameOverride: registration-api-protection\nconfig:\n  client-id: registration-api\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: registration-api.192-168-49-2.nip.io\n  name: registration-api\n  port:\n    number: 8080\nsecrets:\n  # Values for secret 'registration-api-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n    nginx.ingress.kubernetes.io/enable-cors: \"true\"\n  serverSnippets:\n    custom: |-\n      # Open access...\n      location ~ ^/ {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n
"},{"location":"eoepca/registration-api/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: registration-api above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=registration-api \\\n  --name=\"Registration API Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Registration API Gatekeeper\"\n
"},{"location":"eoepca/registration-api/#additional-information","title":"Additional Information","text":"

Additional information regarding the Registration API can be found at:

  • Helm Chart
  • GitHub Repository
"},{"location":"eoepca/resource-catalogue/","title":"Resource Catalogue","text":"

The Resource Catalogue provides a standards-based EO metadata catalogue that includes support for OGC CSW / API Records, STAC and OpenSearch.

"},{"location":"eoepca/resource-catalogue/#helm-chart","title":"Helm Chart","text":"

The Resource Catalogue is deployed via the rm-resource-catalogue helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the rm-resource-catalogue chart.

helm install --version 1.4.0 --values resource-catalogue-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  resource-catalogue rm-resource-catalogue\n
"},{"location":"eoepca/resource-catalogue/#values","title":"Values","text":"

The Resource Catalogue supports many values to configure the service - as described in the Values section of the chart README.

Typically, values for the following attributes may be specified:

  • The fully-qualified public URL for the service
  • Dynamic provisioning StorageClass for database persistence
  • (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Resource Catalogue will not be protected by the identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.
  • Metadata describing the Catalogue instance
  • Tuning configuration for PostgreSQL - see values db.config.XXX.

Example resource-catalogue-values.yaml\u2026

global:\n  namespace: rm\n# For protected access disable this ingress, and rely upon the identity-gatekeeper\n# for ingress with protection.\ningress:\n  # Enabled for unprotected 'open' access to the resource-catalogue.\n  enabled: true\n  name: resource-catalogue\n  host: resource-catalogue.192-168-49-2.nip.io\n  tls_host: resource-catalogue.192-168-49-2.nip.io\n  tls_secret_name: resource-catalogue-tls\n  annotations:\n    cert-manager.io/cluster-issuer: letsencrypt-production\ndb:\n  volume_storage_type: standard\n  # config:\n  #   enabled: true\n  #   shared_buffers: 2GB\n  #   effective_cache_size: 6GB\n  #   maintenance_work_mem: 512MB\n  #   checkpoint_completion_target: 0.9\n  #   wal_buffers: 16MB\n  #   default_statistics_target: 100\n  #   random_page_cost: 4\n  #   work_mem: 4MB\n  #   cpu_tuple_cost: 0.4\npycsw:\n  config:\n    server:\n      url: https://resource-catalogue.192-168-49-2.nip.io/\n    manager:\n      transactions: \"true\"\n      allowed_ips: \"*\"\n

Note

The above example values enable transactions (write-access) to the catalogue from any IP address. This is convenient for testing/demonstration of the capability, but should be disbaled or restricted for operational deployments.

"},{"location":"eoepca/resource-catalogue/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the resource-catalogue service to provide access authorization decisions

"},{"location":"eoepca/resource-catalogue/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install resource-catalogue-protection identity-gatekeeper -f resource-catalogue-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"rm\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the resource-catalogue - in particular the specific ingress requirements for the resource-catalogue-service\u2026

Example resource-catalogue-protection-values.yaml\u2026

fullnameOverride: resource-catalogue-protection\nconfig:\n  client-id: resource-catalogue\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: resource-catalogue.192-168-49-2.nip.io\n  name: resource-catalogue-service\n  port:\n    number: 80\nsecrets:\n  # Values for secret 'resource-catalogue-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n    nginx.ingress.kubernetes.io/enable-cors: \"true\"\n  serverSnippets:\n    custom: |-\n      # Open access...\n      location ~ ^/ {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n
"},{"location":"eoepca/resource-catalogue/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: resource-catalogue above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=resource-catalogue \\\n  --name=\"Resource Catalogue Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Resource Catalogue Gatekeeper\"\n
"},{"location":"eoepca/resource-catalogue/#resource-catalogue-usage","title":"Resource Catalogue Usage","text":"

The Resource Catalogue is initially populated during the initialisation of the Data Access service. See section Data-layer Configuration.

The Resource Catalogue is accessed at the endpoint https://resource-catalogue.192-168-49-2.nip.io/, configured by your domain - e.g. https://resource-catalogue.192-168-49-2.nip.io/.

"},{"location":"eoepca/resource-catalogue/#loading-records","title":"Loading Records","text":"

As described in the pycsw documentation, ISO XML records can be loaded into the resource-catalogue using the pycsw-admin.py admin utility\u2026

pycsw-admin.py load_records -c /path/to/cfg -p /path/to/records\n

The /path/to/records can either be a single metadata file, or a directory containing multiple metadata files.

This is most easily achieved via connection to the pycsw pod, which includes the pycsw-admin.py utility and the pycsw configuration file at /etc/pycsw/pycsw.cfg\u2026

kubectl -n rm cp \"<metadata-file-or-directory>\" \"<pycsw-pod-name>\":/tmp/metadata\nkubectl -n rm exec -i \"<pycsw-pod-name>\" -- pycsw-admin.py load-records -c /etc/pycsw/pycsw.cfg -p /tmp/metadata\n

The name of the pycsw pod can be obtained using kubectl\u2026

kubectl -n rm get pod --selector='io.kompose.service=pycsw' --output=jsonpath={.items[0].metadata.name}\n

To facilitate the loading of records via the pycsw pod, a helper script load-records has been provided in the git repository that hosts this document\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n./deploy/bin/load-records \"<metadata-file-or-directory>\"\n

The helper script identifies the pycsw pod, copies the metadata files to the pod, and runs pycsw-admin.py load-records within the pod to load the records.

"},{"location":"eoepca/resource-catalogue/#additional-information","title":"Additional Information","text":"

Additional information regarding the Resource Catalogue can be found at:

  • Helm Chart
  • pycsw Documentation
  • GitHub Repository
"},{"location":"eoepca/resource-protection-gluu/","title":"Resource Protection (Gluu)","text":"

EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes:

  • Login Service (Authorization Server)
  • Policy Decision Point (PDP)
  • Policy Enforcement Point (PEP)

Building Blocks that act as a Resource Server are individually protected by a Policy Enforcement Point (PEP). The PEP enforces the authorization decision in collaboration with the Login Service and Policy Decision Point (PDP).

The PEP expects to interface to a client (user agent, e.g. browser) using User Managed Access (UMA) flows. It is not typical for a client to support UMA flows, and so the PEP can be deployed with a companion UMA User Agent component that interfaces between the client and the PEP, and performs the UMA Flow on behalf of the client.

The Resource Guard is a \u2018convenience\u2019 component that deploys the PEP & UMA User Agent as a cooperating pair.

The Resource Guard \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, the Resource Guard deploys with an Ingress specification that:

  • Configures the auth_request module to defer access authorization to the uma-user-agent service
  • Configures the ingress rules (host/path) for the target Resource Server
"},{"location":"eoepca/resource-protection-gluu/#helm-chart","title":"Helm Chart","text":"

The Resource Guard is deployed via the resource-guard helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the resource-guard chart.

It is expected to deploy multiple instances of the Resource Guard chart, one for each Resource Server to be protected.

helm install --version 1.3.1 --values myservice-guard-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  myservice-guard resource-guard\n
"},{"location":"eoepca/resource-protection-gluu/#values","title":"Values","text":"

The helm chart is deployed with values that are passed through to the subcharts for the pep-engine and uma-user-agent. Typical values to be specified include:

  • Host/domain details for the Login Service and PDP, e.g. auth.192-168-49-2.nip.io
  • IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2
  • Name of Persistent Volume Claim for pep-engine persistence, e.g. myservice-pep-pvc
  • TLS Certificate Provider, e.g. letsencrypt-production
  • Optional specification of default resources with which to initialise the policy database for the component
  • Ingress rules definition for reverse-proxy to the target Resource Server
  • Name of Secret that contains the client credentials used by the uma-user-agent to interface with the Login Service. See section Client Secret below

Example myservice-guard-values.yaml\u2026 

#---------------------------------------------------------------------------\n# Global values\n#---------------------------------------------------------------------------\nglobal:\n  context: myservice\n  domain: 192-168-49-2.nip.io\n  nginxIp: 192.168.49.2\n  certManager:\n    clusterIssuer: letsencrypt-production\n#---------------------------------------------------------------------------\n# PEP values\n#---------------------------------------------------------------------------\npep-engine:\n  configMap:\n    asHostname: auth\n    pdpHostname: auth\n  customDefaultResources:\n  - name: \"Eric's space\"\n    description: \"Protected Access for eric to his space in myservice\"\n    resource_uri: \"/ericspace\"\n    scopes: []\n    default_owner: \"d3688daa-385d-45b0-8e04-2062e3e2cd86\"\n  volumeClaim:\n    name: myservice-pep-pvc\n    create: false\n#---------------------------------------------------------------------------\n# UMA User Agent values\n#---------------------------------------------------------------------------\numa-user-agent:\n  nginxIntegration:\n    enabled: true\n    hosts:\n      - host: myservice\n        paths:\n          - path: /(.*)\n            service:\n              name: myservice\n              port: 80\n          - path: /(doc.*)\n            service:\n              name: myservice-docs\n              port: 80\n    annotations:\n      nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n      nginx.ingress.kubernetes.io/enable-cors: \"true\"\n      nginx.ingress.kubernetes.io/rewrite-target: /$1\n  client:\n    credentialsSecretName: \"myservice-agent\"\n  logging:\n    level: \"debug\"\n  unauthorizedResponse: 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"'\n#---------------------------------------------------------------------------\n# END values\n#---------------------------------------------------------------------------\n

"},{"location":"eoepca/resource-protection-gluu/#client-credentials","title":"Client Credentials","text":"

The uma-user-agent requires Client Credentials for its interactions with the login-service. The uma-user-agent expects to read these credentials from the file client.yaml, in the form\u2026

client-id: <my-client-id>\nclient-secret: <my-secret>\n
"},{"location":"eoepca/resource-protection-gluu/#client-registration","title":"Client Registration","text":"

To obtain the Client Credentials required by the uma-user-agent it is necessary to register a client with the login-service, or use the credentials for an existing client.

A helper script is provided to register a basic client and obtain the required credentials. The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The register-client helper script requires some command-line arguments\u2026

Usage:\n  register_client <authorization-server-hostname> <client-name> [<redirect-uri> [<logout-uri>]]\n

For example\u2026

./deploy/bin/register-client auth.192-168-49-2.nip.io myclient\n\nINFO: Preparing docker image... [done]\nClient successfully registered.\nMake a note of the credentials:\nclient-id: a98ba66e-e876-46e1-8619-5e130a38d1a4\nclient-secret: 73914cfc-c7dd-4b54-8807-ce17c3645558\n

Or to register OIDC redirect URLs\u2026 

./deploy/bin/register-client auth.192-168-49-2.nip.io myclient https://portal.192-168-49-2.nip.io/oidc/callback/ https://portal.192-168-49-2.nip.io/logout\n

The script writes the \u2018client credentials\u2019 to stdout - in the expected YAML configuration file format - which can be redirected to file\u2026 

./deploy/bin/register-client auth.192-168-49-2.nip.io myclient | tee client.yaml\n
\u2026writes the client credentials to the file client.yaml.

NOTE that the register-client helper relies upon docker to build and run the script.

"},{"location":"eoepca/resource-protection-gluu/#client-secret","title":"Client Secret","text":"

The client.yaml configuration file is made available via a Kubernetes Secret\u2026

kubectl -n myservice-ns create secret generic myservice-agent \\\n  --from-file=client.yaml \\\n  --dry-run=client -o yaml \\\n  > myservice-agent-secret.yaml\n
apiVersion: v1\nkind: Secret\nmetadata:\n  name: myservice-agent\n  namespace: myservice-ns\ndata:\n  client.yaml: Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4\n

The resource-guard deployment is configured with the name of the Secret through the helm chart value client.credentialsSecretName.

"},{"location":"eoepca/resource-protection-gluu/#user-id-token","title":"User ID Token","text":"

As described in the README for the Resource Guard, it is necessary for a request to a protected resource to provide the User ID Token in the request header.

"},{"location":"eoepca/resource-protection-gluu/#obtaining-the-user-id-token","title":"Obtaining the User ID Token","text":"

In the simple case of a user with username/password held within the Login Service, the User ID Token can be obtained as follows:

curl --location --request POST 'https://auth.192-168-49-2.nip.io/oxauth/restv1/token' \\\n--header 'Cache-Control: no-cache' \\\n--header 'Content-Type: application/x-www-form-urlencoded' \\\n--data-urlencode 'scope=openid user_name is_operator' \\\n--data-urlencode 'grant_type=password' \\\n--data-urlencode 'username=<username>' \\\n--data-urlencode 'password=<password>' \\\n--data-urlencode 'client_id=<client-id>' \\\n--data-urlencode 'client_secret=<client-password>'\n

The User ID Token is included in the id_token field of the json response.

Alternatively, OAuth/OIDC flows can be followed to authenticate via external identity providers.

"},{"location":"eoepca/resource-protection-gluu/#user-id-token-in-http-requests","title":"User ID Token in HTTP requests","text":"

The Resource Guard protection supports presentation of the User ID Token via the following HTTP request headers (in order of priority)\u2026

  • Authorization header as a bearer token - in the form: Authorization: Bearer <token>
  • X-User-Id header
  • Cookie: auth_user_id=<token>

    Note that the name of the cookie is configurable

"},{"location":"eoepca/resource-protection-gluu/#additional-information","title":"Additional Information","text":"

Additional information regarding the Resource Guard can be found at:

  • Helm Chart
  • README
  • GitHub Repository:
    • pep-engine
    • uma-user-agent
"},{"location":"eoepca/resource-protection-keycloak/","title":"Resource Protection (Keycloak)","text":"

EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes:

  • Keycloak - Identity Service (Authorization Server)
  • Gatekeeper - Policy Enforcement

Building Blocks that act as a Resource Server are individually protected by a dedicated Gatekeeper instance that enforces the authorization decision in collaboration with the Identity Service (Keycloak).

Gatekeeper \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, Gatekeeper deploys with an Ingress specification that:

  • Configures the auth_request module to defer access authorization to the gatekeeper service
  • Configures the ingress rules (host/path) for the target Resource Server
"},{"location":"eoepca/resource-protection-keycloak/#helm-chart","title":"Helm Chart","text":"

Each Gatekeeper is deployed via the identity-gatekeeper helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values - the full set of available values can be seen at https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/values.yaml.

It is expected to deploy multiple instances of the Gatekeeper chart, one for each Resource Server to be protected.

helm install --version 1.0.10 --values myservice-gatekeeper-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  myservice-protection identity-gatekeeper\n
"},{"location":"eoepca/resource-protection-keycloak/#values","title":"Values","text":"

The helm chart is deployed with values that customise the service for the specific needs of the resource-server under protection and the deployment target platform. Typical values to be specified include:

  • Host/domain details for the Keycloak Identity Service, e.g. keycloak.192-168-49-2.nip.io
  • Credentials for the Keycloak client to be used by Gatekeeper (ideally via secret)
  • TLS Certificate Provider, e.g. letsencrypt-production
  • Ingress rules definition for reverse-proxy to the target Resource Server

Example myservice-protection-values.yaml\u2026 

nameOverride: myservice-protection\nconfig:\n  client-id: myservice\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: myservice.192-168-49-2.nip.io\n  name: myservice\n  port:\n    number: 80\nsecrets:\n  # Values for secret 'myservice-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  serverSnippets:\n    custom: |-\n      # Open access to some endpoints, including Swagger UI\n      location ~ ^/(docs|openapi.json|probe) {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n

"},{"location":"eoepca/resource-protection-keycloak/#client-credentials","title":"Client Credentials","text":"

Gatekeeper requires Client Credentials for its interactions with the Keycloak identity-service. These credentials must be supplied by the secret named <myservice>-protection. The secret can be created directly by the helm chart - via the values secrets.clientSecret and secrets.encryptionKey - or perhaps more securely the secret can be created independently (e.g. via a SealedSecret).

"},{"location":"eoepca/resource-protection-keycloak/#client-registration","title":"Client Registration","text":"

The Keycloak client can be created directly in the Keycloak admin console - e.g. via https://keycloak.192-168-49-2.nip.io/admin.

As an aide there is a helper script create-client. The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The create-client helper script requires some command-line arguments\u2026

$ ./deploy/bin/create-client -h\n\nAdd a client with protected resources.\ncreate-client [-h] [-a] [-i] [-u] [-p] [-c] [-s] [-t | --token t] [-r] --id id [--name name] (--secret secret | --public) [--default] [--authenticated] [--resource name] [--uris u1,u2] [--scopes s1,s2] [--users u1,u2] [--roles r1,r2]\n\nwhere:\n    -h                    show help message\n    -a                    authorization server url - e.g. https://keycloak.192-168-49-2.nip.io\n    -i                    identity-api server url - e.g. https://identity-api.192-168-49-2.nip.io\n    -u                    username used for authentication\n    -p                    password used for authentication\n    -c                    client id (of the bootstrap client used in the create request)\n    -s                    client secret (of the bootstrap client used in the create request)\n    -t or --token         access token used for authentication\n    -r                    realm\n    --id                  client id (of the created client)\n    --name                client name (of the created client)\n    --secret              client secret (of the created client)\n    --public              public client (no client secret)\n    --default             add default resource - /* authenticated\n    --authenticated       allow access to the resource only when authenticated\n    --resource            resource name\n    --uris                resource uris - separated by comma (,)\n    --scopes              resource scopes - separated by comma (,)\n    --users               user names with access to the resource - separated by comma (,)\n    --roles               role names with access to the resource - separated by comma (,)\n

For example\u2026

./deploy/bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=myservice \\\n  --name=\"Gatekeeper for myservice\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Gatekeeper for myservice\" \\\n  --resource=\"eric\" --uris='/eric/*' --scopes=view --users=\"eric\" \\\n  --resource=\"bob\" --uris='/bob/*' --scopes=view --users=\"bob\" \\\n  --resource=\"alice\" --uris='/alice/*' --scopes=view --users=\"alice\"\n
"},{"location":"eoepca/resource-protection-keycloak/#user-tokens","title":"User Tokens","text":"

Requests to resource server endpoints that are protected by Gatekeeper must carry an Access Token that has been obtained on behalf of the requesting user. The access_token is carried in the request header\u2026

Authorization: Bearer <access_token>\n

The Access Token for a given user can be obtained with a call to the token endpoint of the Keycloak Identity Service - supplying the credentials for the user and the pre-registered client\u2026

curl -L -X POST 'https://keycloak.192-168-49-2.nip.io/realms/master/protocol/openid-connect/token' \\\n  -H 'Cache-Control: no-cache' \\\n  -H 'Content-Type: application/x-www-form-urlencoded' \\\n  --data-urlencode 'scope=openid profile email' \\\n  --data-urlencode 'grant_type=password' \\\n  --data-urlencode 'username=<username>' \\\n  --data-urlencode 'password=<password>' \\\n  --data-urlencode 'client_id=admin-cli'\n

A json response is returned, in which the field access_token provides the Access Token for the specified <username>.

"},{"location":"eoepca/resource-protection-keycloak/#additional-information","title":"Additional Information","text":"

Additional information regarding the Gatekeeper can be found at:

  • Container Image
  • Helm Chart
  • GitHub Repository
"},{"location":"eoepca/user-profile/","title":"User Profile","text":"

The User Profile represents the user\u2019s \u2018account\u2019 within the platform.

"},{"location":"eoepca/user-profile/#helm-chart","title":"Helm Chart","text":"

The User Profile is deployed via the user-profile helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the user-profile chart.

helm install --version 1.1.12 --values user-profile-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  user-profile user-profile\n
"},{"location":"eoepca/user-profile/#values","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • Public hostname of the Authorization Server, e.g. auth.192-168-49-2.nip.io
  • IP Address of the public facing reverse proxy (Nginx Ingress Controller), e.g. 192.168.49.2
  • Name of Persistent Volume Claim for user-profile persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.

Example user-profile-values.yaml\u2026 

global:\n  domain: auth.192-168-49-2.nip.io\n  nginxIp: 192.168.49.2\nvolumeClaim:\n  name: eoepca-userman-pvc\n  create: false\n

"},{"location":"eoepca/user-profile/#user-profile-usage","title":"User Profile Usage","text":"

The User Profile is accessed through the /web_ui path of the Login Service, e.g. http://auth.kube.guide.eoepca.org/web_ui.

"},{"location":"eoepca/user-profile/#additional-information","title":"Additional Information","text":"

Additional information regarding the User Profile can be found at:

  • Helm Chart
  • Wiki
  • GitHub Repository
"},{"location":"eoepca/workspace/","title":"Workspace","text":"

The Workspace provides protected user resource management that includes dedicated storage and services for resource discovery and access.

"},{"location":"eoepca/workspace/#workspace-api","title":"Workspace API","text":"

The Workspace API provides a REST service through which user workspaces can be created, interrogated, managed and deleted.

"},{"location":"eoepca/workspace/#helm-chart","title":"Helm Chart","text":"

The Workspace API is deployed via the rm-workspace-api helm chart from the EOEPCA Helm Chart Repository.

The chart is configured via values that are fully documented in the README for the um-workspace-api chart.

helm install --version 1.4.2 --values workspace-api-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  workspace-api rm-workspace-api\n
"},{"location":"eoepca/workspace/#values","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • The fully-qualified public URL for the service
  • (optional) Specification of Ingress for reverse-proxy access to the service Note that this is only required in the case that the Workspace API will not be protected by the identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.
  • Prefix for user projects in OpenStack
  • Details for underlying S3 object storage service
  • Identification of secret that provides the client credentials for resource protection
  • Whether flux components should be installed - otherwise they must already be present - Flux Dependency
  • Name of the ConfigMap for user workspace templates - See User Workspace Templates

Example workspace-api-values.yaml\u2026 

fullnameOverride: workspace-api\ningress:\n  enabled: true\n  annotations:\n    cert-manager.io/cluster-issuer: letsencrypt-production\n    kubernetes.io/ingress.class: nginx\n    nginx.ingress.kubernetes.io/enable-cors: \"true\"\n    nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n  hosts:\n    - host: workspace-api-open.192-168-49-2.nip.io\n      paths: [\"/\"]\n  tls:\n    - hosts:\n        - workspace-api-open.192-168-49-2.nip.io\n      secretName: workspace-api-open-tls\nfluxHelmOperator:\n  enabled: true\nprefixForName: \"ws\"\nworkspaceSecretName: \"bucket\"\nnamespaceForBucketResource: \"rm\"\ns3Endpoint: \"https://minio.192-168-49-2.nip.io\"\ns3Region: \"RegionOne\"\nharborUrl: \"https://harbor.192-168-49-2.nip.io\"\nharborUsername: \"admin\"\nharborPasswordSecretName: \"harbor\"\nworkspaceChartsConfigMap: \"workspace-charts\"\nbucketEndpointUrl: \"http://minio-bucket-api:8080/bucket\"\nkeycloakIntegration:\n  enabled: true\n  keycloakUrl: \"https://keycloak.192-168-49-2.nip.io\"\n  realm: \"master\"\n  identityApiUrl: \"https://identity-api.192-168-49-2.nip.io\"\n  workspaceApiIamClientId: \"workspace-api\"\n  defaultIamClientSecret: \"changeme\"\n

Note

  • The Workspace API assumes a deployment of the Harbor Container Regsitry, as configured by the harborXXX values above.See section Container Registry.
  • The password for the harbor admin user must be created as described in the section Harbor admin Password.
  • The keycloakIntegration allows the Workspace API to apply protecion (for the specified workspace owner) to the services within newly created workspaces.
  • The workspace-api initiates the creation of a storage \u2018bucket\u2019 for each workspace - the actual bucket creation being abstracted via a webhook - the URL of which is specified in the value bucketEndpointUrl. See section Bucket Creation Webhook for details.
"},{"location":"eoepca/workspace/#harbor-admin-password","title":"Harbor admin Password","text":"

The password for the harbor admin user is provided to the workspace-api via the specified secret - defined as harbor above.

This secret must be created - for example as follows\u2026

kubectl -n rm create secret generic harbor \\\n  --from-literal=HARBOR_ADMIN_PASSWORD=\"changeme\"\n
"},{"location":"eoepca/workspace/#flux-dependency","title":"Flux Dependency","text":"

Workspaces are created by instantiating the rm-user-workspace helm chart for each user/group. The Workspace API uses Flux CD as a helper to manage these subordinate helm charts - via flux resources of type HelmRelease. Thus, it is necessary to deploy within the cluster the aspects of flux that support this helm chart management - namely the flux helm-controller, source-controller and the Kubernetes Custom Resource Definitions (CRD) for HelmRelease and HelmRepository.

In case you are not already using flux within your clsuter, then the Workspace API helm chart can be configured to deploy the required flux components\u2026 

fluxHelmOperator:\n  enabled: true  # true = install flux for me, false = I already have flux\n

"},{"location":"eoepca/workspace/#user-workspace-templates","title":"User Workspace Templates","text":"

The Workspace API instantiates for each user a set of services, including a Resource Catalogue and Data Access services. These user services are instantiated via helm using templates. The templates are provided to the Workspace API in a ConfigMap that is, by default, named workspace-charts. Each file in the config-map is expected to be of kind HelmRelease. During creation of a new workspace, the Worksapce API applies each file to the cluster in the namespace of the newly created namespace.

The default ConfigMap that is included with this guide contains the following templates for instantiation of user-specific components:

  • Data Access: template-hr-data-access.yaml
  • Resource Catalogue: template-hr-resource-catalogue.yaml
  • Protection: template-hr-resource-protection.yaml

Each of these templates is expressed as a flux HelmRelease object that describes the helm chart and values required to deploy the service.

In addition, ConfigMap templates are included that provide specific details required to access the user-scoped workspace resources, including access to S3 object storage and container registry:

  • S3 client configuration: template-cm-aws-config.yaml
  • S3 client credentials: template-cm-aws-credentials.yaml
  • Container registry configuration: template-cm-docker-config.yaml

These ConfigMaps are designed to be mounted as files into the runtime environments of other components for user workspace integration. In particular the Application Hub makes use of this approach to provide a user experience that integrates with the user\u2019s workspace resources.

"},{"location":"eoepca/workspace/#templates-configmap","title":"Templates ConfigMap","text":"

The templates are provided to the Workspace API as a ConfigMap in the namespace of the Workspace API deployment\u2026

(for full examples see https://github.com/EOEPCA/deployment-guide/tree/eoepca-v1.4/deploy/eoepca/workspace-templates)

apiVersion: v1\nkind: ConfigMap\nmetadata:\n  name: workspace-charts\ndata:\n  template-hr-resource-catalogue.yaml: |\n    apiVersion: helm.toolkit.fluxcd.io/v2beta1\n    kind: HelmRelease\n    metadata:\n      name: rm-resource-catalogue\n    spec:\n      interval: 5m\n      chart:\n        spec:\n          chart: rm-resource-catalogue\n          version: 1.3.1\n          sourceRef:\n            kind: HelmRepository\n            name: eoepca\n            namespace: rm\n      values:\n        ...\n  template-hr-data-access.yaml: |\n    apiVersion: helm.toolkit.fluxcd.io/v2beta1\n    kind: HelmRelease\n    metadata:\n      name: vs\n    spec:\n      interval: 5m\n      chart:\n        spec:\n          chart: data-access\n          version: 1.3.1\n          sourceRef:\n            kind: HelmRepository\n            name: eoepca\n            namespace: rm\n      values:\n        ...\n  template-hr-resource-protection.yaml: |\n    apiVersion: helm.toolkit.fluxcd.io/v2beta1\n    kind: HelmRelease\n    metadata:\n      name: resource-protection\n    spec:\n      interval: 5m\n      chart:\n        spec:\n          chart: identity-gatekeeper\n          version: 1.0.11\n          sourceRef:\n            kind: HelmRepository\n            name: eoepca\n            namespace: ${NAMESPACE}\n      values:\n        ...\n  template-cm-aws-config.yaml: |\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      name: aws-config\n    data:\n      aws-config: |\n        [default]\n        region = {{ s3_region }}\n        s3 =\n          endpoint_url = {{ s3_endpoint_url }}\n        s3api =\n          endpoint_url = {{ s3_endpoint_url }}\n        [plugins]\n        endpoint = awscli_plugin_endpoint\n  template-cm-aws-credentials.yaml: |\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      name: aws-credentials\n    data:\n      aws-credentials: |\n        [default]\n        aws_access_key_id = {{ access_key_id }}\n        aws_secret_access_key = {{ secret_access_key }}\n  template-cm-docker-config.yaml: |\n    apiVersion: v1\n    kind: ConfigMap\n    metadata:\n      name: docker-config\n    data:\n      docker-config: |\n        {\n          \"auths\": {\n            \"{{ container_registry_host }}\": {\n              \"auth\": \"{{ container_registry_credentials }}\"\n            }\n        }\n

Notice the use of workspace template parameters {{ param_name }} that are used at workspace creation time to contextualise the workspace for the owning user. See section Workspace Template Parameters for more information.

"},{"location":"eoepca/workspace/#helmrepositories-for-templates","title":"HelmRepositories for Templates","text":"

As can be seen above, the HelmRelease templates rely upon objects of type HelmRepository that define the hosting helm chart repository. Thus, in support of the workspace templates, appropriate HelmRepository objects must be provisioned within the cluster. For example, in support of the above examples that rely upon the EOEPCA Helm Chart Repository\u2026

apiVersion: source.toolkit.fluxcd.io/v1beta1\nkind: HelmRepository\nmetadata:\n  name: eoepca\n  namespace: rm\nspec:\n  interval: 2m\n  url: https://eoepca.github.io/helm-charts/\n
"},{"location":"eoepca/workspace/#workspace-template-parameters","title":"Workspace Template Parameters","text":"

The Workspace API uses the jinja2 templating engine when applying the resources for a user workspace. The current parameters are currently supported:

  • workspace_name The name of the workspace - {{ workspace_name }} used to ensure unique naming of cluster resources, such as service ingress
  • default_owner The uuid of the owner of the workspace - {{ default_owner }} used to initialise the workspace protection
  • S3 Object Storage details\u2026
    • {{ s3_endpoint_url }}
    • {{ s3_region }}
    • {{ access_key_id }}
    • {{ secret_access_key }}
  • Container Registry details\u2026
    • {{ container_registry_host }}
    • {{ container_registry_credentials }}
"},{"location":"eoepca/workspace/#protection","title":"Protection","text":"

As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the workspace-api service to provide access authorization decisions

"},{"location":"eoepca/workspace/#gatekeeper","title":"Gatekeeper","text":"

Gatekeeper is deployed using its helm chart\u2026

helm install workspace-api-protection identity-gatekeeper -f workspace-api-protection-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"rm\" --create-namespace \\\n  --version 1.0.11\n

The identity-gatekeeper must be configured with the values applicable to the workspace-api - in particular the specific ingress requirements for the workspace-api backend service\u2026

Example workspace-api-protection-values.yaml\u2026

fullnameOverride: workspace-api-protection\nconfig:\n  client-id: workspace-api\n  discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n  cookie-domain: 192-168-49-2.nip.io\ntargetService:\n  host: workspace-api.192-168-49-2.nip.io\n  name: workspace-api\n  port:\n    number: 8080\nsecrets:\n  # Values for secret 'workspace-api-protection'\n  # Note - if ommitted, these can instead be set by creating the secret independently.\n  clientSecret: \"changeme\"\n  encryptionKey: \"changemechangeme\"\ningress:\n  enabled: true\n  className: nginx\n  annotations:\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  serverSnippets:\n    custom: |-\n      # Open access to some endpoints, including Swagger UI\n      location ~ ^/(docs|openapi.json|probe) {\n        proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n      }\n
"},{"location":"eoepca/workspace/#keycloak-client","title":"Keycloak Client","text":"

The Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: workspace-api above.

This can be created with the create-client helper script, as descirbed in section Client Registration.

For example, with path protection for the admin user\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=\"workspace-api\" \\\n  --name=\"Workspace API Gatekeeper\" \\\n  --secret=\"changeme\" \\\n  --description=\"Client to be used by Workspace API Gatekeeper\" \\\n  --resource=\"admin\" --uris='/*' --scopes=view --users=\"admin\"\n
"},{"location":"eoepca/workspace/#workspace-api-usage","title":"Workspace API Usage","text":"

The Workspace API provides a REST interface that is accessed at the endpoint https://workspace-api.192-168-49-2.nip.io/. See the Swagger Docs - /docs.

Note

If the Workspace API has been protected (via Gatekeeper with Keycloak), then requests must be supported by an access_token carried in the HTTP header Authorozation: Bearer <token>. This diminishes the utility of the swagger UI.

"},{"location":"eoepca/workspace/#additional-information","title":"Additional Information","text":"

Additional information regarding the Workspace API can be found at:

  • Helm Chart
  • Wiki
  • GitHub Repository
"},{"location":"eoepca/workspace/#bucket-creation-webhook","title":"Bucket Creation Webhook","text":"

With helm chart version 1.3.1 of the workspace-api the approach to bucket creation has been re-architected to use a webhook approach.

"},{"location":"eoepca/workspace/#approach","title":"Approach","text":"

During workspace creation the workspace-api needs to create an object storage bucket for the user. The method by which the bucket is created is a function of the hosting infrastructure object storage layer - i.e. there is no \u2018common\u2019 approach for the workspace-api to perform the bucket creation.

In order to allow this bucket creation step to be customised by the platform integrator, the workspace-api is configured with a webhook endpoint that is invoked to effect the bucket creation on behalf of the workspace-api.

The workspace-api is configured by the following value in its helm chart deployment, e.g\u2026 

bucketEndpointUrl: \"http://my-bucket-webhook:8080/bucket\"\n

The webhook service must implement the following REST interface\u2026

method: POST content-type: application/json data: 

{\n  bucketName: str\n  secretName: str\n  secretNamespace: str\n}\n

There are two possible approaches to implement this request, distinguished by the response code\u2026

  • 200 The bucket is created and the credentials are included in the response body. In this case only the supplied bucketName is relevant to fulfil the request.
  • 201 The bucket will be created (asychronously) and the outcome is provided by the webhook via a Kubernetes secret, as per the secretName and secretNamespace request parameters

200 Response

In case 200 response, the response body should communicate the credentials with an application/json content-type in the form\u2026 

{\n    \"bucketname\": \"...\",\n    \"access_key\": \"...\",\n    \"access_secret\": \"....\",\n    \"projectid\": \"...\",\n}\n

In this case the workspace-api will create the appropriate bucket secret using the returned credentials.

201 Response

In case 201 response, the secret should be created in the form\u2026 

data:\n  bucketname: \"...\"\n  access: \"...\"\n  secret: \"...\"\n  projectid: \"...\"\n

In this case the workspace-api will wait for the (asynchronous) creation of the specified secret before continuing with the workspace creation.

Overall Outcome

In both cases the ultimate outcome is the creation of the bucket in the back-end object storage, and the creation of a Kubernetes secret that maintains the credentials for access to the bucket. The existence of the bucket secret is prerequisite to the continuation of the user workspace creation.

"},{"location":"eoepca/workspace/#minio-bucket-api-webhook","title":"Minio Bucket API (Webhook)","text":"

The Minio Bucket API provides an implementation of a Bucket Creation Webhook for a Minio S3 Object Storage backend. This is used as the default in this guide - but should be replaced for a production deployment with an appropriate webhook to integrate with the object storage solution of the deployment environment.

"},{"location":"eoepca/workspace/#helm-chart_1","title":"Helm Chart","text":"

The Minio Bucket API is deployed via the rm-minio-bucket-api helm chart from the EOEPCA Helm Chart Repository - ref. Helm Chart for the Minio Bucket API.

helm install --version 0.0.4 --values minio-bucket-api-values.yaml \\\n  --repo https://eoepca.github.io/helm-charts \\\n  rm-minio-bucket-api rm-minio-bucket-api\n
"},{"location":"eoepca/workspace/#values_1","title":"Values","text":"

At minimum, values for the following attributes should be specified:

  • The URL for the Minio endpoint - minIOServerEndpoint
  • The credentials for admin access to Minio - via the specified secret accessCredentials.secretName (ref. Minio Credentials Secret)

Example minio-bucket-api-values.yaml\u2026 

fullnameOverride: minio-bucket-api\nminIOServerEndpoint: https://minio.192-168-49-2.nip.io\naccessCredentials:\n  secretName: minio-auth\n

"},{"location":"eoepca/workspace/#additional-information_1","title":"Additional Information","text":"

Additional information regarding the Minio Bucket API can be found at:

  • Helm Chart
  • GitHub Repository
"},{"location":"quickstart/application-hub-deployment/","title":"Application Hub Deployment","text":""},{"location":"quickstart/application-hub-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for an \u2018Application Hub\u2019 deployment - that provides the Application Hub integrated with the Identity Service (Keycloak) via OIDC for user authentication.

The script deploy/apphub/apphub achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/apphub/apphub-options.

The Application Hub deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • Identity Service (Keycloak) With test users eric, bob and alice created in Keycloak
    • Application Hub User eric and bob predefined as admin users
  • Other eoepca services not deployed
"},{"location":"quickstart/application-hub-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/apphub/apphub\n

The Identity Service (Keycloak) is accessed at the following endpoints\u2026

  • http://keycloak.192-168-49-2.nip.io/
  • http://identity-api.192-168-49-2.nip.io/docs (Swagger docs for the API)

The Application Hub is accessed at the endpoint - http://applicationhub.192-168-49-2.nip.io/.

"},{"location":"quickstart/application-hub-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/application-hub-deployment/#application-hub-notes","title":"Application Hub Notes","text":""},{"location":"quickstart/application-hub-deployment/#login","title":"Login","text":"

Authentication is made via the Sign in with EOEPCA button on the service home page - which redirects to Keycloak for authentication.

With the out-of-the-box configuration user eric or bob should be used with default password changeme. Users eric and bob are currently predefined within the helm chart as admin users - see https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/files/hub/jupyter_config.py#L171.

"},{"location":"quickstart/application-hub-deployment/#spawning-applications","title":"Spawning Applications","text":"

Once logged in, the service list is presented for spawning of applications. Note that this list of applications is currently defined within the helm chart - see https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/files/hub/config.yml.

From the list, a service is selected and the Start button initiates spawning.

For a clean deployment, the first spawn of each application may take some time whilst the container image representing the application is downloaded to the node. Subsequent invocations (at least on the same node) should be much faster. Once running, the application continues (in the background) until stopped by the user using the Stop Server button on the user\u2019s home screen.

The current JupyterHub configuration assumes a single application service (per user) running at a time - i.e. the current application must be stopped before the next can be started. There is an alternative configuration in which applications can be run in parallel and their lifecycles individually managed.

"},{"location":"quickstart/application-hub-deployment/#returning-to-the-home-screen","title":"Returning to the Home Screen","text":"

The launched applications do not (yet) have a navigation link \u2018out\u2019 of the application back to the home screen.

Therefore, it is necessary to manually modify the url in the browser address bar to /hub/home to navigate to the home screen - from where the current running server can be stopped or re-entered.

"},{"location":"quickstart/application-hub-deployment/#iat-jupyterlab","title":"IAT - JupyterLab","text":"

Following instantiation, the IAT application (Interactive Analysis Tool) defaults to the \u2018Jupyter Notebook\u2019 view (/user/<user>/tree) - rather than the Jupyter Lab view (/user/<user>/lab).

To switch to the Jupyter Lab view it is necessary to manually edit the url path from /user/<user>/tree to /user/<user>/lab. It is intended to update the default to this Jupyter Lab path.

"},{"location":"quickstart/creodias-deployment/","title":"CREODIAS Deployment","text":""},{"location":"quickstart/creodias-deployment/#deployment","title":"Deployment","text":"

Based upon our development experiences on CREODIAS, there is a wrapper script creodias with particular customisations suited to the CREODIAS infrastructure and data offering. The customisations are expressed through environment variables that are captured in the file creodias-options.

These scripts are examples that can be seen as a starting point, from which they can be adapted to your needs.

The CREODIAS deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Protected service endpoints requiring IAM authorization

With reference to the file creodias-options, particular attention is drawn to the following environment variables that require tailoring to your CREODIAS (Cloudferro) environment\u2026

  • Passwords: MINIO_ROOT_PASSWORD, HARBOR_ADMIN_PASSWORD
  • Identity Service credentials - e.g. IDENTITY_SERVICE_DEFAULT_SECRET, IDENTITY_SERVICE_ADMIN_PASSWORD, etc.
  • OpenStack details: see section Openstack Configuration
  • If configuring an external deployment - ref. Public Deployment\u2026
    • public_ip - The public IP address through which the deployment is exposed via the ingress-controller
    • domain - The DNS domain name through which the deployment is accessed - forming the stem for all service hostnames in the ingress rules

Once the file creodias-options has been well populated for your environment, then the deployment is initiated with\u2026 

./deploy/creodias/creodias\n
\u2026noting that this step is a customised version of that described in section Deployment.

"},{"location":"quickstart/creodias-deployment/#harvest-creodias-data","title":"Harvest CREODIAS Data","text":"

The harvester can be deployed with a default configuration file at /config.yaml. As described in the Data Access section, harvesting according to this configuration can be triggered with\u2026 

kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\n

See the Harvester section below for an explanation of this harvester configuration.

"},{"location":"quickstart/creodias-deployment/#data-specification-walkthrough","title":"Data Specification Walkthrough","text":"

The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables.

This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example.

"},{"location":"quickstart/creodias-deployment/#harvester","title":"Harvester","text":"

The harvester configuration specifies datasets with spatial/temporal extents, which is configured into the file /config.yaml of the data-access-harvester deployment.

The harvester is configured as follows\u2026

harvester:\n  replicaCount: 1\n  resources:\n    requests:\n      cpu: 100m\n      memory: 100Mi\n  config:\n    redis:\n      host: data-access-redis-master\n      port: 6379\n    harvesters:\n      - name: Creodias-Opensearch\n        resource:\n          url: https://datahub.creodias.eu/resto/api/collections/Sentinel2/describe.xml\n          type: OpenSearch\n          format_config:\n            type: 'application/json'\n            property_mapping:\n              start_datetime: 'startDate'\n              end_datetime: 'completionDate'\n              productIdentifier: 'productIdentifier'\n          query:\n            time:\n              property: sensed\n              begin: 2019-09-10T00:00:00Z\n              end: 2019-09-11T00:00:00Z\n            collection: null\n            bbox: 14.9,47.7,16.4,48.7\n        filter: {}\n        postprocess:\n          - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor\n        queue: register\n      - name: Creodias-Opensearch-Sentinel1\n        resource:\n          url: https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml\n          type: OpenSearch\n          format_config:\n            type: 'application/json'\n            property_mapping:\n              start_datetime: 'startDate'\n              end_datetime: 'completionDate'\n              productIdentifier: 'productIdentifier'\n          query:\n            time:\n              property: sensed\n              begin: 2019-09-10T00:00:00Z\n              end: 2019-09-11T00:00:00Z\n            collection: null\n            bbox: 14.9,47.7,16.4,48.7\n            extra_params:\n              productType: GRD-COG\n        filter: {}\n        postprocess:\n          - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel1Postprocessor\n        queue: register\n

Based upon this harvester configuration we expect that the following query is made to discover data - i.e. an OpenSearch query, with json response representation, for a defined spatial and temporal extent\u2026

https://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2019-09-10T00:00:00Z&completionDate=2019-09-11T00:00:00Z&box=14.9,47.7,16.4,48.7\n

From the result returned, the path to each product (feature) is obtained from the productIdentifier property, e.g.

{\n  \"type\": \"FeatureCollection\",\n  \"features\": [\n    {\n      \"type\": \"Feature\",\n      \"properties\": {\n        \"productIdentifier\": \"/eodata/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE\"\n        ...\n      }\n      ...\n    }\n    ...\n  ]\n}\n

The harvester is configured with a Sentinel-2/CREODIAS specific post-processor harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor which transforms the product path from /eodata/... to s3://EODATA/....

The harvester post-processor follows this path to the Sentinel-2 scene and uses stactools (with built-in support for Sentinel-2) to establish a STAC item representing the product. This includes enumeration of assets for inspire-metadata and product-metadata - which are used by the registrar pycsw backend to embelesh the product record metadata.

Note

The above description considers Sentinel-2 data. Similar considerations apply for Sentinel-1 that is also detailed in the above harvester configuration.

The harvester outputs the STAC item for each product, which is pushed to the registrar via the register redis queue.

"},{"location":"quickstart/creodias-deployment/#registration","title":"Registration","text":"

The registrar is configured at deployment to have the access details for the CREODIAS data in S3\u2026

global:\n  storage:\n    data:\n      data:\n        type: S3\n        endpoint_url: http://data.cloudferro.com\n        access_key_id: access\n        secret_access_key: access\n        region_name: RegionOne\n        validate_bucket_name: false\n

Using this S3 configuration, the registrar pycsw backend uses the product metadata linked in the STAC item (ref. assets inspire-metadata and product-metadata) to embelesh the metadata. For example, product-metadata in the file\u2026

s3://EODATA/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE/MTD_MSIL1C.xml\n

The registrar uses this information to create the ISO XML metadata that is loaded into the resource-catalogue.

"},{"location":"quickstart/creodias-deployment/#product-type","title":"Product Type","text":"

The registrar recognises the product as Sentinel-2 and so reads its metadata XML files to obtain additional information. From the metadata XML file (e.g. MTD_MSIL1C.xml) the registrar obtains the Product Type for each product from the field <PRODUCT_TYPE>\u2026

<n1:Level-1C_User_Product>\n  <n1:General_Info>\n    <Product_Info>\n      <PRODUCT_TYPE>S2MSI1C</PRODUCT_TYPE>\n      ...\n    </Product_Info>\n    ...\n  </n1:General_Info>\n  ...\n<n1:Level-1C_User_Product>\n
"},{"location":"quickstart/creodias-deployment/#resource-catalogue-collections","title":"Resource Catalogue Collections","text":"

The registrar (eoepca/rm-data-access-core) container image is pre-loaded with two collections at the path /registrar_pycsw/registrar_pycsw/resources, (in the built container the files are at the path /usr/local/lib/python3.8/dist-packages/registrar_pycsw/resources/):

  • S2MSI1C.yml - identifier: S2MSI1C
  • S2MSI2A.yml - identifier: S2MSI2A

The registrar applies these collections into the resource-catalogue during start-up - to create pre-defined out-of-the-box collections in pycsw.

During registration, the PycswBackend of the registrar uses the Product Type to map the product into the collection of the same name - using metadata field parentidentifier.

"},{"location":"quickstart/creodias-deployment/#data-specification","title":"Data Specification","text":"

The data-access service data handling is configured by definition of productTypes, collections and layers\u2026

  • productTypes identify the underlying file assets as WCS coverages and their visual representation
  • collections provide groupings into which products are organised
  • layers specifies the hoe the product visual representations are exposed through the WMS service
"},{"location":"quickstart/creodias-deployment/#producttype","title":"productType","text":"

During registration, products are mapped into a productType via a filter that is applied against the STAC item metadata.

The registrar uses the product_type of each product to determine the collection into which the product should be registered - noting that the name of the product type does not take part in the matching logic (and hence can be any text name)\u2026

  productTypes:\n    - name: S2MSI1C\n      filter:\n        s2:product_type: S2MSI1C\n

In the above example, the field s2:product_type is populated by the stactools that prepares the STAC item from the product metadata.

"},{"location":"quickstart/creodias-deployment/#producttype-coverages","title":"productType - coverages","text":"

coverages defines the coverages for the WCS service. Each coverage links to the assets that are defined within the product STAC item.

"},{"location":"quickstart/creodias-deployment/#producttype-browses","title":"productType - browses","text":"

browses defines the images that are visualised in the View Server Client. Expressions are used to map the product assets into their visual representation.

"},{"location":"quickstart/creodias-deployment/#collections","title":"collections","text":"

Collections are defined by reference to the defined productTypes and coverages.

"},{"location":"quickstart/creodias-deployment/#layers","title":"layers","text":"

layers defines the layers that are presented through the WMS service - each layer being linked to the underlying browse that provides the image source. Layers are defined via their id that relies upon the naming convection <collection>__<browse> to identify the browse and so define the layer.

"},{"location":"quickstart/creodias-deployment/#example-configuration","title":"Example Configuration","text":"

Example configuration for Sentinel-2 L1C and L2A data.

global:\n  layers:\n    - id: S2L1C\n      title: Sentinel-2 Level 1C True Color\n      abstract: Sentinel-2 Level 2A True Color\n      displayColor: '#eb3700'\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L1C__TRUE_COLOR\n      title: Sentinel-2 Level 1C True Color\n      abstract: Sentinel-2 Level 2A True Color\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L1C__masked_clouds\n      title: Sentinel-2 Level 1C True Color with cloud masks\n      abstract: Sentinel-2 Level 1C True Color with cloud masks\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L1C__FALSE_COLOR\n      title: Sentinel-2 Level 1C False Color\n      abstract: Sentinel-2 Level 1C False Color\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L1C__NDVI\n      title: Sentinel-2 Level 21CNDVI\n      abstract: Sentinel-2 Level 1C NDVI\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L1C\n    - id: S2L2A\n      title: Sentinel-2 Level 2A True Color\n      abstract: Sentinel-2 Level 2A True Color\n      displayColor: '#eb3700'\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n    - id: S2L2A__TRUE_COLOR\n      title: Sentinel-2 Level 2A True Color\n      abstract: Sentinel-2 Level 2A True Color\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n    - id: S2L2A__masked_clouds\n      title: Sentinel-2 Level 2A True Color with cloud masks\n      abstract: Sentinel-2 Level 2A True Color with cloud masks\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n    - id: S2L2A__FALSE_COLOR\n      title: Sentinel-2 Level 2A False Color\n      abstract: Sentinel-2 Level 2A False Color\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n    - id: S2L2A__NDVI\n      title: Sentinel-2 Level 2A NDVI\n      abstract: Sentinel-2 Level 2A NDVI\n      grids:\n        - name: WGS84\n          zoom: 13\n      parentLayer: S2L2A\n  collections:\n    S2L1C:\n      product_types:\n        - S2MSI1C\n      coverage_types:\n        - S2L1C_B01\n        - S2L1C_B02\n        - S2L1C_B03\n        - S2L1C_B04\n        - S2L1C_B05\n        - S2L1C_B06\n        - S2L1C_B07\n        - S2L1C_B08\n        - S2L1C_B8A\n        - S2L1C_B09\n        - S2L1C_B10\n        - S2L1C_B11\n        - S2L1C_B12\n    S2L2A:\n      product_types:\n        - S2MSI2A\n      product_levels:\n        - Level-2A\n      coverage_types:\n        - S2L2A_B01\n        - S2L2A_B02\n        - S2L2A_B03\n        - S2L2A_B04\n        - S2L2A_B05\n        - S2L2A_B06\n        - S2L2A_B07\n        - S2L2A_B08\n        - S2L2A_B8A\n        - S2L2A_B09\n        - S2L2A_B11\n        - S2L2A_B12\n  productTypes:\n    - name: S2MSI1C\n      filter:\n        s2:product_type: S2MSI1C\n      metadata_assets: []\n      coverages:\n        S2L1C_B01:\n          assets:\n            - B01\n        S2L1C_B02:\n          assets:\n            - B02\n        S2L1C_B03:\n          assets:\n            - B03\n        S2L1C_B04:\n          assets:\n            - B04\n        S2L1C_B05:\n          assets:\n            - B05\n        S2L1C_B06:\n          assets:\n            - B06\n        S2L1C_B07:\n          assets:\n            - B07\n        S2L1C_B08:\n          assets:\n            - B08\n        S2L1C_B8A:\n          assets:\n            - B8A\n        S2L1C_B09:\n          assets:\n            - B09\n        S2L1C_B10:\n          assets:\n            - B10\n        S2L1C_B11:\n          assets:\n            - B11\n        S2L1C_B12:\n          assets:\n            - B12\n      defaultBrowse: TRUE_COLOR\n      browses:\n        TRUE_COLOR:\n          asset: visual\n          red:\n            expression: B04\n            range: [0, 4000]\n            nodata: 0\n          green:\n            expression: B03\n            range: [0, 4000]\n            nodata: 0\n          blue:\n            expression: B02\n            range: [0, 4000]\n            nodata: 0\n        FALSE_COLOR:\n          red:\n            expression: B08\n            range: [0, 4000]\n            nodata: 0\n          green:\n            expression: B04\n            range: [0, 4000]\n            nodata: 0\n          blue:\n            expression: B03\n            range: [0, 4000]\n            nodata: 0\n        NDVI:\n          grey:\n            expression: (B08-B04)/(B08+B04)\n            range: [-1, 1]\n      masks:\n        clouds:\n          validity: false\n    - name: S2MSI2A\n      filter:\n        s2:product_type: S2MSI2A\n      metadata_assets: []\n      coverages:\n        S2L2A_B01:\n          assets:\n            - B01\n        S2L2A_B02:\n          assets:\n            - B02\n        S2L2A_B03:\n          assets:\n            - B03\n        S2L2A_B04:\n          assets:\n            - B04\n        S2L2A_B05:\n          assets:\n            - B05\n        S2L2A_B06:\n          assets:\n            - B06\n        S2L2A_B07:\n          assets:\n            - B07\n        S2L2A_B08:\n          assets:\n            - B08\n        S2L2A_B8A:\n          assets:\n            - B8A\n        S2L2A_B09:\n          assets:\n            - B09\n        S2L2A_B11:\n          assets:\n            - B11\n        S2L2A_B12:\n          assets:\n            - B12\n      default_browse_locator: TCI_10m\n      browses:\n        TRUE_COLOR:\n          asset: visual-10m\n          red:\n            expression: B04\n            range: [0, 4000]\n            nodata: 0\n          green:\n            expression: B03\n            range: [0, 4000]\n            nodata: 0\n          blue:\n            expression: B02\n            range: [0, 4000]\n            nodata: 0\n        FALSE_COLOR:\n          red:\n            expression: B08\n            range: [0, 4000]\n            nodata: 0\n          green:\n            expression: B04\n            range: [0, 4000]\n            nodata: 0\n          blue:\n            expression: B03\n            range: [0, 4000]\n            nodata: 0\n        NDVI:\n          grey:\n            expression: (B08-B04)/(B08+B04)\n            range: [-1, 1]\n      masks:\n        clouds:\n          validity: false\n
"},{"location":"quickstart/data-access-deployment/","title":"Data Access Deployment","text":""},{"location":"quickstart/data-access-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for a \u2018data access\u2019 deployment - that is focused on the Resource Catalogue and Data Access services.

The script deploy/data-access/data-access achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/data-access/data-access-options.

The data-access deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • Resource Catalogue for data discovery
    • Data Access for data visualisation and download
  • Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION
  • Open ingress are enabled for unauthenticated access to resource-catalogue and data-access services
  • Other eoepca services not deployed
"},{"location":"quickstart/data-access-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/data-access/data-access\n

The Resource Catalogue is accessed at the endpoint resource-catalogue-open.192-168-49-2.nip.io - e.g. resource-catalogue-open.192-168-49-2.nip.io.

The Data Access View Server is accessed at the endpoint data-access-open.192-168-49-2.nip.io - e.g. data-access-open.192-168-49-2.nip.io.

"},{"location":"quickstart/data-access-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/data-access-deployment/#data-harvesting","title":"Data Harvesting","text":"

See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.

"},{"location":"quickstart/exploitation-deployment/","title":"Exploitation Deployment","text":""},{"location":"quickstart/exploitation-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for an \u2018exploitation\u2019 deployment - that provides deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services.

The script deploy/exploitation/exploitation achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/exploitation/exploitation-options.

The exploitation deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • ADES for processing
    • Resource Catalogue for data discovery
    • Data Access for data visualisation and download
    • Minio for S3 object storage
  • ADES stage-out to Minio
  • Includes data specification for CREODIAS Sentinel-2, which can be exploited if running in a CREODIAS VM connected to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION
  • Open ingress are enabled for unauthenticated access to ADES, resource-catalogue and data-access services
  • Other eoepca services not deployed
"},{"location":"quickstart/exploitation-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/exploitation/exploitation\n

The ADES service is accessed at the endpoint ades-open.192-168-49-2.nip.io.

The Resource Catalogue is accessed at the endpoint resource-catalogue-open.192-168-49-2.nip.io.

The Data Access View Server is accessed at the endpoint data-access-open.192-168-49-2.nip.io.

"},{"location":"quickstart/exploitation-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/exploitation-deployment/#example-requests-s-expression-on-creodias","title":"Example Requests - s-expression on CREODIAS","text":"

NOTE that this example processing request requires harvesting data from CREODIAS, which can only be performed if the deployment is made to a CREODIAS VM with access to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION.

Section Processing provides an example of a simple self-contained processing deployment and execution, and access to the processing results.

In addition to the snuggs example, the file deploy/samples/requests/processing/s-expression.http has been prepared to exploit data that has been registered within the Resource Catalogue and Data Access services.

First the input data for processing must be harvested into the resource management services. Sentinel-2 data on 2nd Sept 2020\u2026

./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2020.09.02.yaml\n

Then the s-expression.http file provides sample requests for OGC API Processes operations:

  • List Processes
  • Deploy Process
  • Get Process Details
  • Execute Process
  • Get Job Status
  • Get Job Results

NOTE that the first requests in the file provide optional calls to obtain a user access token (openidConfiguration / authenticate) - to be used in the case that protected (not \u2018open\u2019) endpoints are deployed.

The files snuggs.http and s-expression.http describe the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client. Install in vscode with ext install humao.rest-client.

Various variables, such as to specify the @domain for your deployment, can be configured at the top of the file.

At the completion of successful processing execution, the procesing results are obtained as described in section Processing Results.

"},{"location":"quickstart/exploitation-deployment/#data-harvesting","title":"Data Harvesting","text":"

See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.

"},{"location":"quickstart/processing-deployment/","title":"Processing Deployment","text":""},{"location":"quickstart/processing-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for a \u2018processing\u2019 deployment - that is focused on the ADES and the deployment/execution of processing jobs.

The script deploy/processing/processing achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/processing/processing-options.

The processing deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • ADES for processing
    • Minio for S3 object storage
  • ADES stage-out to Minio
  • Open ingress are enabled for unauthenticated access to ADES service
  • Other eoepca services not deployed
"},{"location":"quickstart/processing-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/processing/processing\n

The ADES service is accessed at the endpoint zoo-open.192-168-49-2.nip.io.

"},{"location":"quickstart/processing-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/processing-deployment/#example-requests","title":"Example Requests","text":"

Some sample requests have been prepared in the subdirectory deploy/samples/requests/processing - for example\u2026

  • convert Provides a \u2018hello world\u2019 processing example that can be used simply to check that the processing capability has been well deployed
  • snuggs Provides a packaged EO exploitation algorithm that perform \u2018real\u2019 work and, as such, is more resource demanding (10GB RAM, 3 CPU) - and so may not be suitable for execution within a local minikube deployment (depending on resource allocations)

These sample http files have been prepared with sample requests for OGC API Processes operations:

  • List Processes
  • Deploy Process
  • Get Process Details
  • Execute Process
  • Get Job Status
  • Get Job Results

Note

  • The first requests in the file provide optional calls to obtain a user ID token (openidConfiguration / authenticate). These are to be used in the case that protected (not \u2018open\u2019) endpoints are deployed.
  • The file describes the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client. Install in vscode with ext install humao.rest-client.
  • The variables @hostname and @domain can be configured at the top of the file.
"},{"location":"quickstart/processing-deployment/#alternative-curl-commands","title":"Alternative curl Commands","text":"

Alternatively the following curl commands can be used\u2026

List Processes 
curl -k \\\n  --request GET \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n  --header 'accept: application/json'\n
Deploy & Execute (convert) Deploy Process (convert) - By Reference (JSON) 
curl -k \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/json' \\\n  --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/convert/main/convert-url-app.cwl\",\"type\": \"application/cwl\"}}'\n
Deploy Process (convert) - Inline (CWL) 
curl -k \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/cwl+yaml' \\\n  --data '< convert-url-app.cwl'\n
Get Process Details (convert) 
curl -k \\\n  --request GET \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \\\n  --header 'accept: application/json'\n
Execute Process (convert) 
curl -k -v \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url/execution \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/json' \\\n  --header 'prefer: respond-async' \\\n  --data '{\"inputs\": {\"fn\":  \"resize\",\"url\": \"https://eoepca.org/media_portal/images/logo6_med.original.png\", \"size\": \"50%\"},\"response\":\"raw\"}'\n
Undeploy Process (convert) 
curl -k -v \\\n  --request DELETE \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \\\n  --header 'accept: application/json'\n
Deploy & Execute (snuggs) Deploy Process (snuggs) 
curl -k \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/json' \\\n  --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/deployment-guide/eoepca-v1.4/deploy/samples/requests/processing/snuggs.cwl\",\"type\": \"application/cwl\"}}'\n
Get Process Details (snuggs) 
curl -k \\\n  --request GET \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \\\n  --header 'accept: application/json'\n
Execute Process (snuggs) 
curl -k -v \\\n  --request POST \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs/execution \\\n  --header 'accept: application/json' \\\n  --header 'content-type: application/json' \\\n  --header 'prefer: respond-async' \\\n  --data '{\"inputs\": {\"input_reference\":  \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\",\"s_expression\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\"},\"response\":\"raw\"}'\n
Undeploy Process (snuggs) 
curl -k -v \\\n  --request DELETE \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \\\n  --header 'accept: application/json'\n
Get Job Status

This request requires the Location header from the response to the execute request. This will be of the form http://zoo-open.192-168-49-2.nip.io/{user}/ogc-api/jobs/{job-id} - e.g. http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e.

curl -k \\\n  --request GET \\\n  --url {location-header} \\\n  --header 'accept: application/json'\n
Get Job Results

This request uses the same URL as Get Job Status, with the additional URL path /results - i.e. /{user}/ogc-api/jobs/{job-id}/results - e.g. /eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e/results

curl -k \\\n  --request GET \\\n  --url {location-header}/results \\\n  --header 'accept: application/json'\n

The response indicates the location of the results, which should be in the minio object storage. See Processing Results.

The response also provides links to log files regarding each step of the Application Package workflow execution - which may be useful for debugging.

List Jobs 
curl -k \\\n  --request GET \\\n  --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs \\\n  --header 'accept: application/json'\n
"},{"location":"quickstart/processing-deployment/#processing-results","title":"Processing Results","text":"

The outputs are published as a static STAC catalogue to a path that includes the unique job ID.

In the default configuration, the processing results are pushed to the Minio S3 object storage. This can be checked via browser access at the endpoint http://console.minio.192-168-49-2.nip.io/, or using an S3 client such as\u2026

s3cmd -c ./deploy/cluster/s3cfg ls s3://eoepca\n

For the default credentials to connect to Minio see Minio Object Storage Default Credentials.

Note

If the ADES deployment has been configured to stage-out to the user\u2019s Workspace, then the above s3cmd and credentials would have to be adjusted accordingly - for example the bucket s3://ws-eric.

"},{"location":"quickstart/quickstart/","title":"Quick Start","text":"

Note

The deployment of the EOEPCA components and the supporting Kubernetes cluster is described in the sections Prepare Cluster and Deploy EOEPCA Components. These sections should be consulted for more detailed information.

"},{"location":"quickstart/quickstart/#scripted-deployment","title":"Scripted Deployment","text":"

As a companion to these descriptions, we have developed a set of scripts to provide a demonstration of example deployments - see section Scripted Deployment for a detailed description of the scripts and how they are configured and used.

Note

The scripted deployment assumes that installation of the Prerequisite Tooling has been performed

"},{"location":"quickstart/quickstart/#customised-deployments","title":"Customised Deployments","text":"

The Scripted Deployment can be quickly exploited through the following customisations (profiles) for particular use cases:

  • Simple Basic local deployment
  • Processing Deployment focused on processing
  • Data Access Deployment focused on the Resource Catalogue and Data Access services
  • Exploitation Deployment providing deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services
  • User Management Deployment focused on the Identity & Access Management services
  • Application Hub Deployment providing the Application Hub that is pre-integrated via OIDC with the Identity Service
  • CREODIAS Deployment with access to CREODIAS EO data

Each customisation is introduced in their respective sections.

"},{"location":"quickstart/quickstart/#quick-example","title":"Quick Example","text":"

Follow these steps to create a simple local deployment in minikube\u2026

  1. Prerequisite Tooling Follow the steps in section Prerequisite Tooling to install the required tooling.
  2. Clone the repository git clone -b eoepca-v1.4 https://github.com/EOEPCA/deployment-guide
  3. Initiate the deployment 
    cd deployment-guide\n./deploy/simple/simple\n
  4. Wait for deployment ready
    1. List pod status watch kubectl get pod -A
    2. Wait until all pods report either Running or Completed This may take 10-20 mins depending on the capabilities of your platform.
  5. Test the deployment Make the sample requests to the ADES processing service.
"},{"location":"quickstart/scripted-deployment/","title":"Scripted Deployment","text":""},{"location":"quickstart/scripted-deployment/#overview","title":"Overview","text":"

The Scripted Deployment provides a demonstration of an example deployment, and can found in the subdirectory deployment-guide/deploy of the source repository for this guide\u2026

git clone -b eoepca-v1.4 https://github.com/EOEPCA/deployment-guide \\\n&& cd deployment-guide \\\n&& ls deploy\n

The script deploy/eoepca/eoepca.sh acts as an entry-point to the full system deployment. In order to tailor the deployment for your target environment, the script is configured through environment variables and command-line arguments. By default the script assumes deployment to a local minikube.

Note

The scripted deployment assumes that installation of the Prerequisite Tooling has been performed.

The following subsections lead through the steps for a full local deployment. Whilst minikube is assumed, minimal adaptions are required to make the deployment to your existing Kubernetes cluster.

The deployment follows these broad steps:

  • Configuration Tailoring of deployment options.
  • Deployment Creation of cluster and deployment of eoepca services.
  • Manual Steps Manual steps to be performed post-deployment.
"},{"location":"quickstart/scripted-deployment/#configuration","title":"Configuration","text":"

The script deploy/eoepca/eoepca.sh is configured by some environment variables and command-line arguments.

"},{"location":"quickstart/scripted-deployment/#environment-variables","title":"Environment Variables","text":"Environment Variables Variable Description Default REQUIRE_<cluster-component> A set of variables that can be used to control which CLUSTER components are deployed by the script, as follows (with defaults):REQUIRE_MINIKUBE=trueREQUIRE_INGRESS_NGINX=trueREQUIRE_CERT_MANAGER=trueREQUIRE_LETSENCRYPT=trueREQUIRE_SEALED_SECRETS=falseREQUIRE_MINIO=false see description REQUIRE_<eoepca-component> A set of variables that can be used to control which EOEPCA components are deployed by the script, as follows (with defaults):REQUIRE_STORAGE=trueREQUIRE_DUMMY_SERVICE=falseREQUIRE_IDENTITY_SERVICE=trueREQUIRE_ADES=trueREQUIRE_RESOURCE_CATALOGUE=trueREQUIRE_DATA_ACCESS=trueREQUIRE_REGISTRATION_API=trueREQUIRE_WORKSPACE_API=trueREQUIRE_HARBOR=trueREQUIRE_PORTAL=trueREQUIRE_APPLICATION_HUB=true see description REQUIRE_<protection-component> A set of variables that can be used to control which PROTECTION components are deployed by the script, as follows (with defaults):REQUIRE_DUMMY_SERVICE_PROTECTION=falseREQUIRE_ADES_PROTECTION=trueREQUIRE_RESOURCE_CATALOGUE_PROTECTION=trueREQUIRE_DATA_ACCESS_PROTECTION=trueREGISTRATION_API_PROTECTION=trueREQUIRE_WORKSPACE_API_PROTECTION=true see description MINIKUBE_VERSION The Minikube version to be (optionally) installedNote that the EOEPCA development has been conducted using the default stated here. v1.32.0 MINIKUBE_KUBERNETES_VERSION The Kubernetes version to be used by minikubeNote that the EOEPCA development has been conducted primarily using version 1.22.5. v1.22.5 MINIKUBE_MEMORY_AMOUNT Amount of memory to allocate to the docker containers used by minikube to implement the cluster. 12g MINIKUBE_DISK_AMOUNT Amount of disk space to allocate to the docker containers used by minikube to implement the cluster. 20g MINIKUBE_EXTRA_OPTIONS Additional options to pass to minikube start command-line --ports=80:80,443:443 USE_METALLB Enable use of minikube\u2019s built-in load-balancer.The load-balancer can be used to facilitate exposing services publicly. However, the same can be achieved using minikube\u2019s built-in ingress-controller. Therefore, this option is suppressed by default. false USE_INGRESS_NGINX_HELM Install the ingress-nginx controller using the published helm chart, rather than relying upon the version that is built-in to minikube. By default we prefer the version that is built in to minikube. false USE_INGRESS_NGINX_LOADBALANCER Patch the built-in minikube nginx-ingress-controller to offer a service of type LoadBalancer, rather than the default NodePort. It was initially thought that this would be necessary to achieve public access to the ingress services - but was subsequently found that the default NodePort configuration of the ingress-controller was sufficient. This option is left in case it proves useful.Only applicable for USE_INGRESS_NGINX_HELM=false (i.e. when using the minikube built-in ) false OPEN_INGRESS Create \u2018open\u2019 ingress endpoints that are not subject to authorization protection. For a secure system the open endpoints should be disabled (false) and access to resource should be protected via ingress that apply protection false USE_TLS Indicates whether TLS will be configured for service Ingress rules.If not (i.e. USE_TLS=false), then the ingress-controller is configured to disable ssl-redirect, and TLS_CLUSTER_ISSUER=notls is set. true TLS_CLUSTER_ISSUER The name of the ClusterIssuer to satisfy ingress tls certificates.Out-of-the-box ClusterIssuer instances are configured in the file deploy/cluster/letsencrypt.sh. letsencrypt-staging IDENTITY_SERVICE_DEFAULT_SECRET Default secret that is used by exception for other Identity Service credentials changeme IDENTITY_SERVICE_ADMIN_USER The admin user for Keycloak admin IDENTITY_SERVICE_ADMIN_PASSWORD The admin user password for Keycloak ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_SERVICE_ADMIN_CLIENT The Keycloak client to use for admin API tasks during scripted deployment admin-cli IDENTITY_POSTGRES_PASSWORD The password for the Keycloak Postgres service ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_GATEKEEPER_CLIENT_SECRET The secret used for each Keycloak client (one per resource service) created during scripted deployment ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_GATEKEEPER_ENCRYPTION_KEY The encryption key for each Keycloak client (one per resource service) created during scripted deployment changemechangeme IDENTITY_REALM Keycloak realm for Identity Service.This is not explicitly created by the scripted deployment, and so is assumed to exist within the Keycloak instance. Thus, will probably break the deployment if modified. master MINIO_ROOT_USER Name of the \u2018root\u2019 user for the Minio object storage service. eoepca MINIO_ROOT_PASSWORD Password for the \u2018root\u2019 user for the Minio object storage service. changeme HARBOR_ADMIN_PASSWORD Password for the \u2018admin\u2019 user for the Harbor artefact registry service. changeme DEFAULT_STORAGE Storage Class to be used by default for all components requiring dynamic storage provisioning.See variables <component>_STORAGE for per-component overrides. standard <component>_STORAGE A set of variables to control the dynamic provisioning Storage Class for individual components, as follows:MINIO_STORAGEADES_STORAGEAPPLICATION_HUB_STORAGEDATA_ACCESS_STORAGEHARBOR_STORAGERESOURCE_CATALOGUE_STORAGE <DEFAULT_STORAGE> PROCESSING_MAX_RAM Max RAM allocated to an individual processing job 8Gi PROCESSING_MAX_CORES Max number of CPU cores allocated to an individual processing job 4 PROCESSING_ZOO_IMAGE Container image for zoo-dru deployment eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762 STAGEOUT_TARGET Configures the ADES with the destination to which it should push processing results:workspace - via the Workspace APIminio - to minio S3 object storage workspace INSTALL_FLUX The Workspace API relies upon Flux CI/CD, and has the capability to install the required flux components to the cluster. If your deployment already has flux installed then set this value false to suppress the Workspace API flux install true CREODIAS_DATA_SPECIFICATION Apply the data specification to harvest from the CREODIAS data offering into the resource-catalogue and data-access services.Can only be used when running in the CREODIAS (Cloudferro) cloud, with access to the eodata network. false TEMP_FORWARDING_PORT Local port used during the scripted deployment for kubectl port-forward operations 9876"},{"location":"quickstart/scripted-deployment/#command-line-arguments","title":"Command-line Arguments","text":"

The eoepca.sh script is further configured via command-line arguments\u2026

eoepca.sh <action> <cluster-name> <domain> <public-ip>\n
eoepca.sh Command-line Arguments Argument Description Default action Action to perform: apply | delete | template.apply makes the deploymentdelete removes the deploymenttemplate outputs generated kubernetes yaml to stdout apply cluster-name The name of the minikube \u2018profile\u2019 for the created minikube cluster eoepca domain The DNS domain name through which the deployment is accessed. Forms the stem for all service hostnames in the ingress rules - i.e. <service-name>.<domain>.By default, the value is deduced from the assigned cluster minikube IP address, using nip.io to establish a DNS lookup - i.e. <minikube ip>.nip.io. <minikube ip>.nip.io public-ip The public IP address through which the deployment is exposed via the ingress-controller.By default, the value is deduced from the assigned cluster minikube IP address - ref. command minikube ip. <minikube-ip>"},{"location":"quickstart/scripted-deployment/#public-deployment","title":"Public Deployment","text":"

For simplicity, the out-of-the-box scripts assume a \u2018private\u2019 deployment - with no public IP / DNS and hence no use of TLS for service ingress endpoints.

In the case that an external-facing public deployment is desired, then the following configuration selections should be made:

  • domain - set to the domain (as per DNS records) for your deployment Note that the EOEPCA components typically configure their ingress with hostname prefixes applied to this domain. Thus, it is necessary that the DNS record for the domain is established as a wildcard record - i.e. *.<domain>
  • public_ip - set to the public IP address through which the deployment is exposed via the ingress-controller i.e. the IP address that is assigned to the ingress controller service of type LoadBalancer
  • USE_TLS=true - to enable configuration of TLS endpoints in each component service ingress
  • TLS_CLUSTER_ISSUER=<issuer> - should be configured ~ e.g. using the letsencrypt-production or letsencrypt-staging (testing only) Cluster Issuer that are configured by the scripted deployment
"},{"location":"quickstart/scripted-deployment/#deployment","title":"Deployment","text":"

The deployment is initiated by setting the appropriate environment variables and invoking the eoepca.sh script with suitable command-line arguments. You may find it convenient to do so using a wrapper script that customises the environment varaibles according to your cluster, and then invokes the eoepca.sh script.

Customised examples are provided for Simple, CREODIAS and Processing deployments.

NOTE that if a prior deployment has been attempted then, before redeploying, a clean-up should be performed as described in the Clean-up section below. This is particularly important in the case that the minikube none driver is used, as the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed.

Initiate the deployment\u2026 

./deploy/eoepca/eoepca.sh apply \"<cluster-name>\" \"<public-ip>\" \"<domain>\"\n

The deployment takes 10+ minutes - depending on the resources of your host/cluster. The progress can be monitored\u2026 

kubectl get pods -A\n

The deployment is ready once all pods are either Running or Completed.

"},{"location":"quickstart/scripted-deployment/#post-deployment-manual-steps","title":"Post-deployment Manual Steps","text":"

The scripted deployment has been designed, as far as possible, to automate the configuration of the deployed components. However, there remain some steps that must be performed manually after the scripted deployment has completed. See the building block specific pages\u2026

  • Identity Service: Token Lifespans
  • Application Hub: Post-deployment Manual Steps
"},{"location":"quickstart/scripted-deployment/#default-credentials","title":"Default Credentials","text":""},{"location":"quickstart/scripted-deployment/#identity-service","title":"Identity Service","text":"

By default, the Identity Service is accessed at the URL https://keycloak.192-168-49-2.nip.io/ with the credentials\u2026

username: `admin`\npassword: `changeme`\n

\u2026unless the password is overridden via the variable IDENTITY_SERVICE_ADMIN_PASSWORD.

"},{"location":"quickstart/scripted-deployment/#minio-object-storage","title":"Minio Object Storage","text":"

By default, Minio is accessed at the URL https://console.minio.192-168-49-2.nip.io/ with the credentials\u2026

username: `eoepca`\npassword: `changeme`\n

\u2026unless the username/password are overridden via the variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD.

"},{"location":"quickstart/scripted-deployment/#harbor-container-registry","title":"Harbor Container Registry","text":"

By default, Harbor is accessed at the URL https://harbor.192-168-49-2.nip.io/ with the credentials\u2026

username: `admin`\npassword: `changeme`\n

\u2026unless the password is overridden via the variable HARBOR_ADMIN_PASSWORD.

"},{"location":"quickstart/scripted-deployment/#protection","title":"Protection","text":"

The protection of resource server endpoints is applied during the deployment of each service requiring protection. This comprises creating a dedicated Keycloak client for each resource server, and the creation of associated resources and policies that protect the service-specific URLs.

This protection can be disabled via the environment variables REQUIRE_XXX_PROTECTION - e.g. REQUIRE_ADES_PROTECTION=false.

Note

By default, if OPEN_INGRESS is set true then PROTECTION will be disabled (false) unless overridden via the REQUIRE_XXX_PROTECTION variables.

"},{"location":"quickstart/scripted-deployment/#test-users","title":"Test Users","text":"

The deployment creates (in the Keycloak Identity Service) the test users: eric, bob, alice.

Note

This does NOT create the workspace for each of these users - which must be performed via the Workspace API.

"},{"location":"quickstart/scripted-deployment/#user-workspace-creation","title":"User Workspace Creation","text":"

The deployment created the test users eric, bob and alice. For completeness we use the Workspace API to create their user workspaces, which hold their personal resources (data, processing results, etc.) within the platform - see Workspace.

"},{"location":"quickstart/scripted-deployment/#using-workspace-swagger-ui","title":"Using Workspace Swagger UI","text":"

The Workspace API provides a Swagger UI that facilitates interaction with the API - at the URL https://workspace-api.192-168-49-2.nip.io/docs#.

Note

If the Workspace API has been protected (via Gatekeeper with Keycloak), then requests must be supported by an access_token carried in the HTTP header Authorozation: Bearer <token>. This diminishes the utility of the swagger UI.

Access the Workspace Swagger UI at https://workspace-api.192-168-49-2.nip.io/docs. Workspaces are created using POST /workspaces (Create Workspace). Expand the node and select Try it out. Complete the request body, such as\u2026 

{\n  \"preferred_name\": \"eric\",\n  \"default_owner\": \"eric\"\n}\n
\u2026where the default_owner is the ID for the user in Keycloak - thus protecting the created workspace for the identified user.

"},{"location":"quickstart/scripted-deployment/#using-curl","title":"Using curl","text":"

The same can be achieved with a straight http request, for example using curl\u2026

curl -X 'POST' \\\n  'http://workspace-api.192-168-49-2.nip.io/workspaces' \\\n  -H 'Content-Type: application/json' \\\n  -H 'Accept: application/json' \\\n  -H 'Authorization: Bearer <admin-access-token>' \\\n  -d '{\n  \"preferred_name\": \"<workspace-name>\",\n  \"default_owner\": \"<user-id>\"\n}'\n

Values must be provided for:

  • admin-access-token - Access Token for the admin user
  • workspace-name - name of the workspace, typically the username
  • user-id - the ID of the user for which the created workspace will be protected, typically the username

The Access Token for the admin user can be obtained with a call to the token endpoint of the Identity Service - supplying the credentials for the admin user and the pre-registered client\u2026

curl -L -X POST 'https://keycloak.192-168-49-2.nip.io/realms/master/protocol/openid-connect/token' \\\n  -H 'Cache-Control: no-cache' \\\n  -H 'Content-Type: application/x-www-form-urlencoded' \\\n  --data-urlencode 'scope=openid profile email' \\\n  --data-urlencode 'grant_type=password' \\\n  --data-urlencode 'username=admin' \\\n  --data-urlencode 'password=<admin-password>' \\\n  --data-urlencode 'client_id=admin-cli'\n

A json response is returned, in which the field access_token provides the Access Token for the admin user.

"},{"location":"quickstart/scripted-deployment/#using-create-workspace-helper-script","title":"Using create-workspace helper script","text":"

As an aide there is a helper script create-workspace. The script is available in the deployment-guide repository, and can be obtained as follows\u2026

git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n

The create-workspace helper script requires some command-line arguments\u2026

$ ./deploy/bin/create-workspace -h\n\nCreate a new User Workspace.\ncreate-workspace -h | -w {workspace_api} -a {auth_server} -r {realm} -c {client} -u {admin-username} -p {admin-password} -O {owner} -W {workspace-name}\n\nwhere:\n    -h  show help message\n    -w  workspace-api service url (default: http://workspace-api.192-168-49-2.nip.io)\n    -a  authorization server url (default: http://keycloak.192-168-49-2.nip.io)\n    -r  realm within Keycloak (default: master)\n    -u  username used for authentication (default: admin)\n    -p  password used for authentication (default: changeme)\n    -c  client id of the bootstrap client used in the create request (default: admin-cli)\n    -O  user ID of the 'owner' of the new workspace (default: workspace(-W))\n    -W  name of the workspace to create (default: owner(-O))\n

Most of the arguments have default values that are aligned to the defaults of the scripted deployment. At minimum either -O owner or -W workspace must be specified.

For example (assuming defaults)\u2026

./deploy/bin/create-workspace -O eric\n

For example (all arguments)\u2026

./deploy/bin/create-workspace \n  -w http://workspace-api.192-168-49-2.nip.io \\\n  -a http://keycloak.192-168-49-2.nip.io \\\n  -r master \\\n  -u admin \\\n  -p changeme \\\n  -c admin-cli \\\n  -O bob \\\n  -W bob\n
"},{"location":"quickstart/scripted-deployment/#eoepca-portal","title":"EOEPCA Portal","text":"

The eoepca-portal is a simple web application that is used as a test aid. It\u2019s main purpose is to provide the ability to login, and so establish a session with appropriate browser cookies - which then allow authenticated access to other EOEPCA services such as the Workspace API, Identity API, etc.

The portal is deployed via a helm chart\u2026

helm install eoepca-portal eoepca-portal -f portal-values.yaml - \\\n  --repo https://eoepca.github.io/helm-charts \\\n  --namespace \"demo\" --create-namespace \\\n  --version 1.0.11\n

The helm values must be tailored for your deployment. For example\u2026

configMap:\n  identity_url: \"http://keycloak.192-168-49-2.nip.io\"\n  realm: \"master\"\n  client_id: \"eoepca-portal\"\n  identity_api_url: \"http://identity-api.192-168-49-2.nip.io\"\n  ades_url: \"http://zoo.192-168-49-2.nip.io/ogc-api/processes\"\n  resource_catalogue_url: \"http://resource-catalogue.192-168-49-2.nip.io\"\n  data_access_url: \"http://data-access.192-168-49-2.nip.io\"\n  workspace_url: \"http://workspace-api.192-168-49-2.nip.io\"\n  workspace_docs_url: \"http://workspace-api.192-168-49-2.nip.io/docs#\"\n  images_registry_url: \"http://harbor.192-168-49-2.nip.io\"\n  dummy_service_url: \"http://dummy-service.192-168-49-2.nip.io\"\n  access_token_name: \"auth_user_id\"\n  access_token_domain: \".192-168-49-2.nip.io\"\n  refresh_token_name: \"auth_refresh_token\"\n  refresh_token_domain: \".192-168-49-2.nip.io\"\ningress:\n  enabled: true\n  annotations:\n    kubernetes.io/ingress.class: nginx\n    ingress.kubernetes.io/ssl-redirect: \"true\"\n    nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n    cert-manager.io/cluster-issuer: letsencrypt-production\n  hosts:\n    - host: eoepca-portal.192-168-49-2.nip.io\n      paths:\n        - path: /\n          pathType: Prefix\n  tls:\n    - secretName: eoepca-portal-tls\n      hosts:\n        - eoepca-portal.192-168-49-2.nip.io\n

The setting client_id: eoepca-portal identifies a client that must be created in Keycloak - as described in section create-client Helper Script - noting that the eoepca-portal requires a client that is configured as a Public Client\u2026

../bin/create-client \\\n  -a https://keycloak.192-168-49-2.nip.io \\\n  -i https://identity-api.192-168-49-2.nip.io \\\n  -r \"master\" \\\n  -u \"admin\" \\\n  -p \"changeme\" \\\n  -c \"admin-cli\" \\\n  --id=eoepca-portal \\\n  --name=\"EOEPCA Portal\" \\\n  --public \\\n  --description=\"Client to be used by the EOEPCA Portal\"\n
"},{"location":"quickstart/scripted-deployment/#clean-up","title":"Clean-up","text":"

Before initiating a fresh deployment, if a prior deployment has been attempted, then it is necessary to remove any persistent artefacts of the prior deployment. This includes\u2026

  1. Minikube cluster Delete the minikube cluster\u2026 minikube delete If necessary specify the cluster (profile)\u2026 minikube -p <profile> delete

  2. Persistent Data In the case that the minikube none driver is used, the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. In this case, the minikube standard StorageClass is fulfilled by the hostpath provisioner, whose persistence is removed as follows\u2026 sudo rm -rf /tmp/hostpath-provisioner

There is a helper script clean that can be used for step 2 above, (the script does not delete the cluster). 

./deploy/cluster/clean\n

"},{"location":"quickstart/simple-deployment/","title":"Simple Deployment","text":""},{"location":"quickstart/simple-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for a \u2018simple\u2019 deployment - designed to get a core local deployment of the primary servies.

The script deploy/simple/simple achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/simple/simple-options.

The simple deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Configuration of \u2018open\u2019 interfaces - i.e. service/API endpoints that are not protected and can accessed without authentication. This facilitates experimentation with the services
  • Configuration of ADES stage-out to a local instance of minio, to avoid the need to create a Workspace for each user
"},{"location":"quickstart/simple-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/simple/simple\n

See section Deployment for more details regarding the outcome of the scripted deployment.

"},{"location":"quickstart/simple-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"},{"location":"quickstart/userman-deployment/","title":"User Management Deployment","text":""},{"location":"quickstart/userman-deployment/#overview","title":"Overview","text":"

A deployment wrapper script has been prepared for a \u2018user management\u2019 deployment - that is focused on the Identity Service (Authorization Server), Identity API and Gatekeeper (Protection Policy Enforcement).

The script deploy/userman/userman achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/userman/userman-options.

The user-management deployment applies the following configuration:

  • Assumes a private deployment - i.e. no external-facing IP/ingress, and hence no TLS To configure an external-facing deployment with TLS protection, then see section Public Deployment
  • No TLS for service ingress endpoints
  • Services deployed:
    • Identity Service
    • Identity API
    • Gatekeeper instance, protecting the Identity API
  • Other eoepca services not deployed
"},{"location":"quickstart/userman-deployment/#initiate-deployment","title":"Initiate Deployment","text":"

Deployment is initiated by invoking the script\u2026

./deploy/userman/userman\n

The Identity Service is accessed at the endpoint keycloak.192-168-49-2.nip.io.

The Identity API is accessed at the endpoint identity-api.192-168-49-2.nip.io.

"},{"location":"quickstart/userman-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"

To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.

"}]} \ No newline at end of file diff --git a/current/sitemap.xml b/current/sitemap.xml index b07e72eb..626666ca 100644 --- a/current/sitemap.xml +++ b/current/sitemap.xml @@ -2,147 +2,147 @@ https://deployment-guide.docs.eoepca.org/current/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/cluster/cluster-prerequisites/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/cluster/helm-repositories/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/cluster/kubernetes/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/cluster/prerequisite-tooling/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/ades-zoo/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/application-hub/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/container-registry/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/data-access/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/iam-overview/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/identity-service/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/login-service/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/pdp/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/persistence/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/registration-api/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/resource-catalogue/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/resource-protection-gluu/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/resource-protection-keycloak/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/user-profile/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/eoepca/workspace/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/quickstart/application-hub-deployment/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/quickstart/creodias-deployment/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/quickstart/data-access-deployment/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/quickstart/exploitation-deployment/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/quickstart/processing-deployment/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/quickstart/quickstart/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/quickstart/scripted-deployment/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/quickstart/simple-deployment/ - 2024-02-28 + 2024-03-01 daily https://deployment-guide.docs.eoepca.org/current/quickstart/userman-deployment/ - 2024-02-28 + 2024-03-01 daily \ No newline at end of file diff --git a/current/sitemap.xml.gz b/current/sitemap.xml.gz index 2fccc545ad29d0f088edea0dd8be543e0bbfd5fb..e8a2cfce59fbdb526030c7df08a6e8b2029d1f3c 100644 GIT binary patch literal 489 zcmV?hWB}jGwcl{?V{~)$gWS2 zJb+^>;K5j~WrxtGuYl8*>AKSy)f->1#sB#t%iZJq>@RPS$YHov_jOxcAQ-eR4A<)U z>8tuwJ@$89NuD5IxO1RuHRc?@G|jRsHMtD4j#OhB8ZskSsJS&ZZT-3G?=GA+14n63 zok!DeR*215h#uD&LRLd@7HW%zYCwoa?`BO&iN7{ovkg02(UfPj{X_fkQMDgbd(T~N zZKrN#)M0?v1mAkAonO1=)N;sce(C!wN{ug+A^pr%Ou+vtZkreq$T5PSRf32NfXRtu zn92*J05TWgyn$3k^n5-AVH{13wm|ATqCn*_dS`TIFR8}zA`ie|$l3+=MF|E+Ij!;- ztj=0#V}MAWc&BHDe9^u+SOluKTyc3-v=JDPN8oXAL9YC0qd?JHT*>!WkT~NvKY93k z9i2o1!jcRqzPL2+{{n&;V!{k&UyQ+*@_&#_uo{ngl1Ec$f7alu^8$C(f@z|tMY$!8 z{>?;Lr~GF$s(#y$$zu|rb(+M3oFY{5Ebz`_Tt0T0bQtzj+oJHRSv<=#IpZw_!5+j9M^Vpu769h{RLw{?& ze)+7Pn&ETpR}Imxn;~SnaC}rUb7u)h4sBrpWBa3_i(CdHL#bq zboFGr)e2WL4x*=Jf{;~T&_V0Satj2}n$4^&Db=s~s@GvhD~9t#uDhHsAJqAxE>E?~ zz3tG=4kq;Qn&4aK4Ep7&4=uYq{Y&3pQEGkR4C!aCVuJdwg6qVvK#l?Yq!L7`0F+Q9 z!&qJ*1(3M_#|@-1k+0`d5XRo5jRRKSi3BQ-(W5oFdPy~u7kL1dLN*wxFG|oO=d{RU za3&jNtOaIy;?Ycs>P7p8Fbh=gY6bJ6cqOnPkHAxpL9YDhqCnAWTv_k0AaTa;escHu zCZa?FrjjhEdU2V${|g9ah>0@TZ83UZ%Kt$!!eTv{Q65dU~n!MyJ>=`ht(t&37$&EioGMUnd1CX5-J g_=RAcq6hI9>rFnHe|yBYj3IYF0QG1IsDKs#02}r7JOBUy diff --git a/versions.json b/versions.json index 7355eefb..360d1bee 100644 --- a/versions.json +++ b/versions.json @@ -1 +1,34 @@ -[{"version": "v1.4", "title": "v1.4", "aliases": ["latest"]}, {"version": "v1.3", "title": "v1.3", "aliases": []}, {"version": "v1.2", "title": "v1.2", "aliases": []}, {"version": "v1.1", "title": "v1.1", "aliases": []}, {"version": "v1.0", "title": "v1.0", "aliases": []}, {"version": "current", "title": "current", "aliases": []}] \ No newline at end of file +[ + { + "version": "current", + "title": "current", + "aliases": [] + }, + { + "version": "v1.4", + "title": "v1.4", + "aliases": [ + "latest" + ] + }, + { + "version": "v1.3", + "title": "v1.3", + "aliases": [] + }, + { + "version": "v1.2", + "title": "v1.2", + "aliases": [] + }, + { + "version": "v1.1", + "title": "v1.1", + "aliases": [] + }, + { + "version": "v1.0", + "title": "v1.0", + "aliases": [] + } +]