The following provides a summary of changes since the last release (v1.4)…
1.4.1 to introduce variables to remedy hard-coded harvester values for access to Creodias eodata. Ref. - CREODIAS_EODATA_S3_ENDPOINT, CREODIAS_EODATA_S3_ACCESS_KEY, CREODIAS_EODATA_S3_ACCESS_SECRET and CREODIAS_EODATA_S3_REGION.eoepca-portalCREODIAS_EODATA_S3_ACCESS_KEY and CREODIAS_EODATA_S3_ACCESS_SECRET
+MINIO_ROOT_PASSWORD, HARBOR_ADMIN_PASSWORDIDENTITY_SERVICE_DEFAULT_SECRET, IDENTITY_SERVICE_ADMIN_PASSWORD, etc.kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch
See the Harvester section below for an explanation of this harvester configuration.
+See EOData Catalogue API Manual on CREODIAS for details regarding access to the CREODIAS data offering.
The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables.
The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables. In addition, it may be necessary to set the variable CREODIAS_EODATA_S3_ENDPOINT if different from the default - for example the value http://eodata.cloudferro.com for the WAW3-2 Cloudferro cloud.
This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example.
The harvester configuration specifies datasets with spatial/temporal extents, which is configured into the file /config.yaml of the data-access-harvester deployment.
CREODIAS_DATA_SPECIFICATION
+eodata network - see description of variable CREODIAS_DATA_SPECIFICATIONCREODIAS_EODATA_S3_ENDPOINT if different from the default - for example the value http://eodata.cloudferro.com for the WAW3-2 Cloudferro cloud.CREODIAS_DATA_SPECIFICATION
+eodata network - see description of variable CREODIAS_DATA_SPECIFICATIONCREODIAS_EODATA_S3_ENDPOINT if different from the default - for example the value http://eodata.cloudferro.com for the WAW3-2 Cloudferro cloud.This current version of the Deployment Guide represents the development tip that goes beyond the latest release version v1.4.
The following provides a summary of changes since the last release (v1.4)\u2026
eoepca-portalsnuggs sample app2.0.59 to add support for path-prefix (BASE_URL)2.0.58 to fix hard-coded namespace proc Namespace can now be set via chart environment variable APP_HUB_NAMESPACEPROCESSING_MAX_RAM to the integer value in Mi 1024 (was string 8Gi)The Deployment Guide captures each release of the EOEPCA Reference Implementation, by providing for each version\u2026
A full system deployment is described, in which components are deployed with complementary configurations that facilitate their integration as a coherent system. Nevertheless, each component can be cherry-picked from this system deployment for individual re-use.
The deployment is organised into the following sections:
The following prerequisite components are assumed to be deployed in the cluster.
Note
The Scripted Deployment automatically deploys most of the components list here - in particular\u2026
The Sealed Secrets controller is not deployed - but can be added following the instructions below.
"},{"location":"cluster/cluster-prerequisites/#nginx-ingress-controller","title":"Nginx Ingress Controller","text":"# Install the Nginx Ingress Controller helm chart\nhelm upgrade -i --version='<4.5.0' \\\n --repo https://kubernetes.github.io/ingress-nginx \\\n ingress-nginx ingress-nginx \\\n --wait\nNote
For Kubernetes version 1.22 and earlier the version of the Nginx Ingress Controller must be before v4.5.0.
To target the Nginx Ingress Controller the kubernetes.io/ingress.class: nginx annotation must be applied to the Ingress resource\u2026
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n annotations:\n kubernetes.io/ingress.class: nginx\n ...\n# Install the Cert Manager helm chart\nhelm upgrade -i --namespace cert-manager --create-namespace \\\n --repo https://charts.jetstack.io \\\n --set installCRDs=true \\\n cert-manager cert-manager\nOnce the Certificate Manager is deployed, then we can establish ClusterIssuer operators in the cluster to support use of TLS with service Ingress endpoints.
For Letsencrypt we can define two ClusterIssuer - for production and for staging.
NOTE that these require the cluster to be publicly accessible, in order for the http01 acme flow to verify the domain ownership. Local development deployments will typically not have public IP/DNS - in which case the system deployment can proceed, but without TLS support for the service endpoints.
apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-production\nspec:\n acme:\n # You must replace this email address with your own.\n # Let's Encrypt will use this to contact you about expiring\n # certificates, and issues related to your account.\n email: eoepca.systemteam@telespazio.com\n server: https://acme-v02.api.letsencrypt.org/directory\n privateKeySecretRef:\n # Secret resource that will be used to store the account's private key.\n name: letsencrypt-production-account-key\n # Add a single challenge solver, HTTP01 using nginx\n solvers:\n - http01:\n ingress:\n class: nginx\napiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-staging\nspec:\n acme:\n # You must replace this email address with your own.\n # Let's Encrypt will use this to contact you about expiring\n # certificates, and issues related to your account.\n email: eoepca.systemteam@telespazio.com\n server: https://acme-staging-v02.api.letsencrypt.org/directory\n privateKeySecretRef:\n # Secret resource that will be used to store the account's private key.\n name: letsencrypt-staging-account-key\n # Add a single challenge solver, HTTP01 using nginx\n solvers:\n - http01:\n ingress:\n class: nginx\nTo exploit the specified ClusterIssuer the cert-manager.io/cluster-issuer annotation must be applied to the Ingress resource. For example\u2026
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n annotations:\n kubernetes.io/ingress.class: nginx\n cert-manager.io/cluster-issuer: letsencrypt-production\n ...\nThe EOEPCA development team maintain their deployment configurations in GitHub - for declarative, reproducible cluster deployments.
Various Secret are relied upon by the system services. Secrets should not be exposed by commit to GitHub.
Instead SealedSecret are committed to GitHub, which are encrypted, and can only be decrypted by the sealed-secret-controller that runs within the cluster. The sealed-secret-controller decrypts the SealedSecret to a regular Secret (of the same name) that can then be consumed by the cluster components.
The sealed-secret-controller is deployed to the cluster using the helm chart\u2026
helm install --version 2.1.8 --create-namespace --namespace infra \\\n --repo https://bitnami-labs.github.io/sealed-secrets \\\n eoepca-sealed-secrets sealed-secrets\nOnce the controller is deployed within the cluster, then the kubeseal command can be used to create a SealedSecret from a regular Secret, as follows\u2026
Create example Secret\u2026
kubectl -n test create secret generic mysecret \\\n --from-literal=password=changeme \\\n --dry-run=client -o yaml \\\n > mysecret.yaml\nCreate SealedSecret from Secret using kubeseal\u2026
kubeseal -o yaml \\\n --controller-name eoepca-sealed-secrets \\\n --controller-namespace infra \\\n < mysecret.yaml \\\n > mysecret-sealed.yaml\nkubeseal ReleaseVarious building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service.
As a workaround, in the absence of an existing object storage, it is possible to use MinIO to establish an object storage service within the Kubernetes cluster. We use the minio helm chart provided by the MinIO Project.
# Install the minio helm chart\nhelm upgrade -i -f minio-values.yaml --namespace rm --create-namespace \\\n --repo https://charts.min.io/ \\\n minio minio \\\n --wait\nNote
The Kubernetes namespace rm is used above as an example, and can be changed according to your deployment preference.
The minio deployment is customised via the values file minio-values.yaml, for example\u2026
existingSecret: minio-auth\nreplicas: 2\n\ningress:\n enabled: true\n ingressClassName: nginx\n annotations:\n cert-manager.io/cluster-issuer: \"letsencrypt\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n path: /\n hosts:\n - minio.192-168-49-2.nip.io\n tls:\n - secretName: minio-tls\n hosts:\n - minio.192-168-49-2.nip.io\n\nconsoleIngress:\n enabled: true\n ingressClassName: nginx\n annotations:\n cert-manager.io/cluster-issuer: \"letsencrypt\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n path: /\n hosts:\n - console.minio.192-168-49-2.nip.io\n tls:\n - secretName: minio-console-tls\n hosts:\n - console.minio.192-168-49-2.nip.io\n\nresources:\n requests:\n memory: 1Gi\n\npersistence:\n storageClass: standard\n\nbuckets:\n - name: eoepca\n - name: cache-bucket\nNote
letsencrypt certificate providerminio-auth - see belownginx.ingress.kubernetes.io/proxy-body-size was found to be required to allow transfer of large files (such as data products) through the nginx proxyThe Minio admin credentials are provided via a Kubernetes secret that is referenced from the Minio helm chart deployment values. For example\u2026
kubectl -n rm create secret generic minio-auth \\\n --from-literal=rootUser=\"eoepca\" \\\n --from-literal=rootPassword=\"changeme\"\nNote
The secret must be created in the same Kubernetes namespace as the Minio service deployment - e.g. rm namespce in the example above.
The s3cmd can be configured for access to the MinIO deployment. The --configure option can be used to prepare a suitable configuration file for s3cmd\u2026
s3cmd -c mys3cfg --configure\nIn response to the prompts, the following configuration selections are applicable to the above settings\u2026
Access Key: eoepca\nSecret Key: changeme\nDefault Region: us-east-1\nS3 Endpoint: minio.192-168-49-2.nip.io\nDNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io\nEncryption password: \nPath to GPG program: /usr/bin/gpg\nUse HTTPS protocol: True\nHTTP Proxy server name: \nHTTP Proxy server port: 0\nSave the configuration file, and check access to the S3 object store with\u2026
# Create a bucket\ns3cmd -c mys3cfg mb s3://eoepca\n\n# List buckets\ns3cmd -c mys3cfg ls\nFor example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube\u2026
s3cmd -c deploy/cluster/s3cfg ls\nNote
This section identifies some helm chart repositories that can be referenced (for convenience) via helm add. Nevertheless, all helm commands included in the guide specifically reference the source helm repository via the --repo argument to the helm install command - and thus it is not specifically necessary to add these repositories in advance.
The EOEPCA building-blocks are engineered as containers for deployment to a Kubernetes cluster. Each building block defines a Helm Chart to facilitate its deployment.
The EOEPCA Helm Chart Repository is configured with helm as follows\u2026
helm repo add eoepca https://eoepca.github.io/helm-charts/\nIn addition to the EOEPCA Helm Chart Repository, a variety of third party helm repositories are relied upon, as identified below.
"},{"location":"cluster/helm-repositories/#cert-manager","title":"Cert Manager","text":"helm repo add jetstack https://charts.jetstack.io\nhelm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\nhelm repo add minio https://charts.min.io/\nhelm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets\nhelm repo add harbor https://helm.goharbor.io\nRefresh the local repo cache, after helm repo add\u2026
helm repo update\nThe EOEPCA Reference Implementation has been developed with Kubernetes as its deployment target. The system components have been developed, deployed and tested using a cluster at version v1.22.5.
Note
The Scripted Deployment assumes that minikube is installed, and creates a minikube cluster under the profile eoepca.
The development, integration and test clusters have been established using Rancher Kubernetes Engine (RKE) at version v1.22.5.
An example of the creation of the EOEPCA Kubernetes clusters can be found on the GitHub Kubernetes Setup page. CREODIAS has been used for the development hosting infrastructure - which provides OpenStack infrastructure that is backed by Cloudferro. An example of the Terraform configurations used to automate the creation of the cloud infrastructure that underpins the RKE deployment can be found on the GitHub CREODIAS Setup page.
"},{"location":"cluster/kubernetes/#local-kubernetes","title":"Local Kubernetes","text":"To make a full deployment of the EOEPCA Reference Implementation requires a multi-node node cluster with suitable resources. For example, the development cluster comprises:
Limited local deployment can be made using a suitable local single-node kuberbetes deployment using - for example using minikube\u2026
minikube -p eoepca start --cpus max --memory max --kubernetes-version v1.22.5\nminikube profile eoepca\nWith such a deployment it is possible to deploy individual building-blocks for local development, or building-blocks in combination - within the constraints of the local host resources.
"},{"location":"cluster/prerequisite-tooling/","title":"Prerequisite Tooling","text":"There are some standard tools referenced in this guide. These are detailed in the following subsections.
"},{"location":"cluster/prerequisite-tooling/#docker","title":"docker","text":"Docker faciliates the creation, management and execution of containers. Whilst not strictly necessary to support deployment to an existing/managed Kubernetes cluster, it can nevertheless be useful to have local access to the docker tooling. For example, if minikube is used to follow this guide using a local k8s cluster, then this is best achieved using minikube\u2019s docker driver.
Docker is most easily installed with\u2026
curl -fsSL https://get.docker.com | sh\nFor convenience, add your user to the docker group\u2026
sudo usermod -aG docker ${USER}\nLogout/in to refresh your session\u2019s group permissions.
"},{"location":"cluster/prerequisite-tooling/#kubectl","title":"kubectl","text":"Kubectl is the main tool for interaction with a Kubernetes cluster. The latest version can be installed with\u2026
mkdir -p $HOME/.local/bin \\\n&& curl -fsSLo $HOME/.local/bin/kubectl \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\" \\\n&& chmod +x $HOME/.local/bin/kubectl\nSee the official kubectl installation documentation for more installation options.
"},{"location":"cluster/prerequisite-tooling/#helm","title":"helm","text":"Helm is the Kubernetes package manager, in which components are deployed to a Kubernetes cluster via helm charts. The helm charts are instantiated for deployment via \u2018values\u2019 that configure the chart templates.
The latest helm version can be installed with\u2026
export HELM_INSTALL_DIR=\"$HOME/.local/bin\" \\\n&& curl -sfL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash\nSee the official helm installation documentation for more installation options.
"},{"location":"cluster/prerequisite-tooling/#minikube","title":"minikube","text":"Minikube is a tool that allows to create a local (single-node) Kubernetes cluster for development/testing. It is not designed for production use. In the absence of access to a \u2018full\u2019 Kubernetes cluster, this guide can be followed using minikube.
The latest version of minikube can be installed with\u2026
mkdir -p $HOME/.local/bin \\\n&& curl -fsSLo $HOME/.local/bin/minikube \"https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\" \\\n&& chmod +x $HOME/.local/bin/minikube\nSee the official minikube installation documentation for more installation options.
"},{"location":"eoepca/ades-zoo/","title":"ADES (Processing)","text":"ADES - Application Deployment & Execution Service"},{"location":"eoepca/ades-zoo/#zoo-project-dru","title":"ZOO-Project DRU","text":"Note
With EOEPCA release 1.4, the ADES implementation has been significantly reworked and fully aligned with the upstream ZOO-Project (GitHub). This zoo-project-dru version deprecates the previous proc-ades implementation.
With this transition, there are some functional changes to be aware of\u2026
zoo-project-dru the OGC API Processes endpoint is at the path /<username>/ogc-api/processes compared to the previous /<username>/wps3/processes.convert-url at version 0.1.2 would result in the endpoint /<username>/wps3/processes/convert-url_0_1_2. With the new zoo-project-dru this same Application Package deployment will result in the endpoint /<username>/ogc-api/processes/convert-url.s:softwareVersion: 0.1.2), and is maintained within the metadata for the deployed process that is returned from the APIs Get Process Details request. In the case that multiple versions of the same Application Package are required to be simultaneously deployed, then this would have to be handled with different CWL documents in which the version is embedded in the workflow id (or some other technique that establishes uniqueness of id between variants).The ADES provides a platform-hosted execution engine through which users can initiate parameterised processing jobs using applications made available within the platform - supporting the efficient execution of the processing \u2018close to the data\u2019. Users can deploy specific \u2018applications\u2019 to the ADES, which may be their own applications, or those published by other platform users.
The ADES provides an implementation of the OGC API Processes - Part 1: Core and Part 2: Deploy, Replace, Undeploy (draft).
"},{"location":"eoepca/ades-zoo/#helm-chart","title":"Helm Chart","text":"The EOEPCA deployment is aligned with the upstream implementation and so relies upon the upstream helm chart that is hosted at the ZOO-Project Helm Chart Repository - in particular the zoo-project-dru chart variant.
The chart is configured via values that are fully documented in the README for the zoo-project-dru chart.
helm install --version 0.2.6 --values ades-values.yaml \\\n --repo https://zoo-project.github.io/charts/ \\\n zoo-project-dru zoo-project-dru\nThe deployment must be configured for you environment. Some significant configuration values are elaborated here\u2026
"},{"location":"eoepca/ades-zoo/#cookie-cutter-template","title":"Cookie-cutter Template","text":"The implementation zoo-project-dru provides the core capabilities for OGC API Processes Parts 1 & 2. The deployemnt of this core must be completed by inetgartion with the \u2018runner\u2019 that executes the processes as Application Packages, and integrates as necessary with other platform services - such as Catalogue, Workspace, etc.
Thus, zoo-project-dru is extensible by design via a \u2018cookie-cutter\u2019 that provides the template \u2018runner\u2019 for each Application Package process as it is deployed to the service.
For the purposes of our EOEPCA \u2018release\u2019 as covered by this guide, we provide eoepca-proc-service-template as a cookie-cutter implemetation that provides:
The cookie-cutter template is identified in the helm values\u2026
cookiecutter:\n templateUrl: https://github.com/EOEPCA/eoepca-proc-service-template.git\n templateBranch: master\nThe function of the cookie-cutter template is supported some other aspects, that are elaborated below, which must be configured in collaboration with the expectations of the template. In particular\u2026
zoo-project-dru configuration [ref]In order support our eoepca-proc-service-template cookie-cutter template, there is a custom zoo-project-dru container image that includes the python dependencies that are required by this template. Thus, the deployment must identify the custom container image via helm values\u2026
zoofpm:\n image:\n tag: eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762\nzookernel:\n image:\n tag: eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762\nIn addition, we can add values to the ZOO-Project DRU main.cfg configuration file via helm values. In this case we add some eoepca-specific values that match those that we know to be expected by our eoepca-proc-service-template cookie-cutter template. In this way we can effectively use helm values to pass parameters through to the template.
customConfig:\n main:\n eoepca: |-\n domain=192-168-49-2.nip.io\n workspace_prefix=ws\nThis is manifest in zoo\u2019s main.cfg in INI file configuration syntax\u2026
[eoepca]\ndomain=192-168-49-2.nip.io\nworkspace_prefix=ws\nThe presence or otherwise of the workspace_prefix parameter dicates whether or not the stage-out step will integrate with the user\u2019s Workspace for persistence of the processing results, and registration within the Workspace services.
In the case that workspace_prefix is not set, then the object storage specification in the helm values is relied upon\u2026
workflow:\n inputs:\n STAGEOUT_AWS_SERVICEURL: https://minio.192-168-49-2.nip.io\n STAGEOUT_AWS_ACCESS_KEY_ID: eoepca\n STAGEOUT_AWS_SECRET_ACCESS_KEY: changeme\n STAGEOUT_AWS_REGION: RegionOne\n STAGEOUT_OUTPUT: eoepca\nThe ADES hosts applications that are deployed and invoked in accordance with the OGC Best Practise for Application Package. Thus, the ADES provides a conformant environment within which the application is integrated for execution. A key part of the ADES\u2019s role in this is to faciltate the provision of input data to the application (stage-in), and the handling of the results output at the conclusion of application execution (stage-out).
The zoo-project-dru helm chart provides a default implementation via the included files - main.yaml, rules.yaml, stagein.yaml and stageout.yaml.
The helm values provides a means through which each of these files can be overriden for reasons of integration with your platform environment\u2026
files:\n # Directory 'files/cwlwrapper-assets' - assets for ConfigMap 'XXX-cwlwrapper-config'\n cwlwrapperAssets:\n main.yaml: |-\n <override file content here>\n rules.yaml: |-\n <override file content here>\n stagein.yaml: |-\n <override file content here>\n stageout.yaml: |-\n <override file content here>\nIn the most part the default CWL wrapper files provided with the helm chart are suffient. In particular the stagein.yaml implements the stage-in of STAC items that are specified as inputs of type Directory in the Application Package CWL.
E.g.
inputs:\n stac:\n label: the image to convert as a STAC item\n doc: the image to convert as a STAC item\n type: Directory\nNevertheless, in this guide we provide an override of the stageout.yaml in order to organise the processing outputs into a STAC Collection that is then pushed to the designated S3 object storage, including support for the user\u2019s workspace storage and resource management services.
The custom stage-out embeds, within the CWL document, the python code required to implement the desired stage-out functionality. This should be regarded as an example that could be adapted for alternative behaviour.
cwlVersion: v1.0\nclass: CommandLineTool\nid: stage-out\ndoc: \"Stage-out the results to S3\"\ninputs:\n process:\n type: string\n collection_id:\n type: string\n STAGEOUT_OUTPUT:\n type: string\n STAGEOUT_AWS_ACCESS_KEY_ID:\n type: string\n STAGEOUT_AWS_SECRET_ACCESS_KEY:\n type: string\n STAGEOUT_AWS_REGION:\n type: string\n STAGEOUT_AWS_SERVICEURL:\n type: string\noutputs:\n StacCatalogUri:\n outputBinding:\n outputEval: ${ return \"s3://\" + inputs.STAGEOUT_OUTPUT + \"/\" + inputs.process + \"/catalog.json\"; }\n type: string\nbaseCommand:\n - python\n - stageout.py\narguments:\n - $( inputs.wf_outputs.path )\n - $( inputs.STAGEOUT_OUTPUT )\n - $( inputs.process )\n - $( inputs.collection_id )\nrequirements:\n DockerRequirement:\n dockerPull: ghcr.io/terradue/ogc-eo-application-package-hands-on/stage:1.3.2\n InlineJavascriptRequirement: {}\n EnvVarRequirement:\n envDef:\n AWS_ACCESS_KEY_ID: $( inputs.STAGEOUT_AWS_ACCESS_KEY_ID )\n AWS_SECRET_ACCESS_KEY: $( inputs.STAGEOUT_AWS_SECRET_ACCESS_KEY )\n AWS_REGION: $( inputs.STAGEOUT_AWS_REGION )\n AWS_S3_ENDPOINT: $( inputs.STAGEOUT_AWS_SERVICEURL )\n InitialWorkDirRequirement:\n listing:\n - entryname: stageout.py\n entry: |-\n import sys\n import shutil\n import os\n import pystac\n\n cat_url = sys.argv[1]\n\n shutil.copytree(cat_url, \"/tmp/catalog\")\n cat = pystac.read_file(os.path.join(\"/tmp/catalog\", \"catalog.json\"))\n\n ...\nThe helm chart values provide the opportunity to pass through additional inputs - to satisfy the input specifications that are specified in the cwlwrapperAssets files\u2026
workflow:\n inputs:\n STAGEIN_AWS_SERVICEURL: http://data.cloudferro.com\n STAGEIN_AWS_ACCESS_KEY_ID: test\n STAGEIN_AWS_SECRET_ACCESS_KEY: test\n STAGEIN_AWS_REGION: RegionOne\n STAGEOUT_AWS_SERVICEURL: https://minio.192-168-49-2.nip.io\n STAGEOUT_AWS_ACCESS_KEY_ID: eoepca\n STAGEOUT_AWS_SECRET_ACCESS_KEY: changeme\n STAGEOUT_AWS_REGION: RegionOne\n STAGEOUT_OUTPUT: eoepca\nThe zoo-project-dru services uses a Node Selector to determine the node(s) upon which the processing execution is run. This is configured as a matching rule in the helm values, and must be tailored to your cluster.
For example, for minikube\u2026
workflow:\n nodeSelector:\n minikube.k8s.io/primary: \"true\"\nIngress can be enabled and configured to establish (reverse-proxy) external access to the zoo-project-dru services.
Hosturl
In the case that protection is enabled - e.g. via Resource Guard - then it is likely that ingress should be disabled here, since the ingress will instead be handled by the protection.
In this case, the hosturl parameter should be set to reflect the public url through the service will be accessed.
In the case that ingress is enabled then it is not necessary to specify the hosturl, since it will be taken from the ingress.hosts[0].host value.
Ingress disabled\u2026
ingress:\n enabled: false\n hosturl: zoo.192-168-49-2.nip.io\nIngress enabled\u2026
ingress:\n enabled: true\n annotations:\n kubernetes.io/ingress.class: nginx\n ingress.kubernetes.io/ssl-redirect: true\n nginx.ingress.kubernetes.io/ssl-redirect: true\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - host: zoo-open.192-168-49-2.nip.io\n paths:\n - path: /\n pathType: ImplementationSpecific\n tls:\n - hosts:\n - zoo-open.192-168-49-2.nip.io\n secretName: zoo-open-tls\nThe above example assumes that TLS should be enabled via Letsencrypt as certificate provider - see section Letsencrypt Certificates.
"},{"location":"eoepca/ades-zoo/#persistence","title":"Persistence","text":"Various of the services deployed as part of zoo-project-dru rely upon dynamic provisioning of persistent storage volumes.
A number of helm values are impacted by this setting, which must be configured with the Storage Class appropriate to your cluster. For example, using the minikube standard storage class\u2026
workflow:\n storageClass: standard\npersistence:\n procServicesStorageClass: standard\n storageClass: standard\n tmpStorageClass: standard\npostgresql:\n primary:\n persistence:\n storageClass: standard\n readReplicas:\n persistence:\n storageClass: standard\nrabbitmq:\n persistence:\n storageClass: standard\nZOO-Project DRU has a built-in capability for Identity & Access Management (IAM), in which the zoo-project-dru service is configured as an OIDC client of an OIDC Identity Provider service.
This capability is disabled by the default deployment offered by this guide (ingress.enabled: false) - which instead (optionally) applies resource protection using the EOEPCA IAM solution. Nevertheless, the built-in IAM can be enabled and configured through helm values.
For example\u2026
iam: \n enabled: true\n openIdConnectUrl: https://keycloak.192-168-49-2.nip.io/realms/master/.well-known/openid-configuration\n type: openIdConnect\n name: OpenIDAuth\n realm: Secured section\nAs described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the zoo-project-dru service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install zoo-project-dru-protection identity-gatekeeper -f zoo-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"zoo\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the zoo-project-dru - in particular the specific ingress requirements for the zoo-project-dru-service\u2026
Example zoo-protection-values.yaml\u2026
fullnameOverride: zoo-project-dru-protection\nconfig:\n client-id: ades\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: zoo.192-168-49-2.nip.io\n name: zoo-project-dru-service\n port:\n number: 80\nsecrets:\n # Values for secret 'zoo-project-dru-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n serverSnippets:\n custom: |-\n # Open access to some endpoints, including Swagger UI\n location ~ /(ogc-api/api|swagger-ui) {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: ades above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example, with path protection for test users\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=ades \\\n --name=\"ADES Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by ADES Gatekeeper\" \\\n --resource=\"eric\" --uris='/eric/*' --scopes=view --users=\"eric\" \\\n --resource=\"bob\" --uris='/bob/*' --scopes=view --users=\"bob\" \\\n --resource=\"alice\" --uris='/alice/*' --scopes=view --users=\"alice\"\nThe zoo-project-dru service provides a mutil-user aware set of service interfaces at\u2026
https://zoo.192-168-49-2.nip.io/<username>/ogc-api/https://zoo.192-168-49-2.nip.io/swagger-ui/oapip/See the Example Requests in the Processing Deployment for sample requests that cans be used to test your deployment, and to learn usage of the OGC API Processes.
"},{"location":"eoepca/ades-zoo/#debugging-tips","title":"Debugging Tips","text":"This section includes some tips that may be useful in debugging errors with deployed application packages.
For debugging, establish a shell session with the zoofpm pod\u2026
$ kubectl -n zoo exec -it deploy/zoo-project-dru-zoofpm -c zoofpm -- bash\nThe logs are in the directory /tmp/zTmp\u2026
$ cd /tmp/zTmp/\nIn the log directory, each execution is characterised by a set of files/directories\u2026
<appname>_<jobid>_error.log <<START HERE The main log output of the job<appname>_<jobid>.json The output (results) of the job<jobid>_status.json The overall status of the job<jobid>_logs.cfg Index of logs for job workflow stepsconvert-url-c6637d4a-d561-11ee-bf3b-0242ac11000e (directory) Subdirectory with a dedicated log file for each step of the CWL workflow, including the stage-in and stage-out stepsWhen the process is deployed from its Application Package, then a representation is created using the configured cookiecutter.templateUrl.
It may be useful to debug the consequent process files, which are located under the path /opt/zooservices_user/<username>, with a dedicated subdirectory for each deployed process - i.e. /opt/zooservices_user/<username>/<appname>/.
For example\u2026
$ cd /opt/zooservices_user/eric/convert-url\n$ ls -l\ntotal 28\n-rw-rw-r-- 1 www-data www-data 0 Feb 27 11:17 __init__.py\ndrwxrwxr-x 2 www-data www-data 4096 Feb 27 11:17 __pycache__\n-rw-rw-r-- 1 www-data www-data 1408 Feb 27 11:17 app-package.cwl\n-rw-rw-r-- 1 www-data www-data 17840 Feb 27 11:17 service.py\nNote
In the case that the cookie-cutter template is updated, then the process can be re-deployed to force a refresh against the updated template.
"},{"location":"eoepca/ades-zoo/#swagger-ui-openapi","title":"Swagger UI (OpenAPI)","text":"The zoo-project-dru service includes a Swagger UI interactive representation of its OpenAPI REST interface - available at the URL https://zoo.192-168-49-2.nip.io/swagger-ui/oapip/.
For a (trivial) example application package see Example Application Package, which provides a description and illustration of the basics of creating an application that integrates with the expectations of the ADES stage-in and stage-out.
For further reference see\u2026
Additional information regarding the ADES can be found at:
eoepca-proc-service-template to aid orchestration of CWL application packages running in Kubernetes via Calrissianzoo-calrissian-runner to aid interfacing with Calrissian and KubernetesThe Application Hub provides a set of web-based tooling, including JupyterLab for interactive analysis, Code Server for application development, and the capability to add user-defined interactive dashboards.
"},{"location":"eoepca/application-hub/#helm-chart","title":"Helm Chart","text":"The Application Hub is deployed via the application-hub helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values, which are detailed in the default values file for the chart.
helm install --version 2.0.58 --values application-hub-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n application-hub application-hub\nThe Application Hub supports many values to configure the service - ref. the default values file for the chart.
Typically, values for the following attributes may be specified:
Example application-hub-values.yaml\u2026
ingress:\n enabled: true\n annotations: {}\n hosts:\n - host: applicationhub.192-168-49-2.nip.io\n paths:\n - path: /\n pathType: ImplementationSpecific\n tls:\n - secretName: applicationhub-tls\n hosts:\n - applicationhub.192-168-49-2.nip.io\n clusterIssuer: letsencrypt-production\n\njupyterhub:\n fullnameOverride: \"application-hub\"\n hub:\n existingSecret: application-hub-secrets\n extraEnv: \n JUPYTERHUB_ENV: \"dev\"\n JUPYTERHUB_SINGLE_USER_IMAGE: \"eoepca/pde-container:1.0.3\"\n OAUTH_CALLBACK_URL: https://applicationhub.192-168-49-2.nip.io/hub/oauth_callback\n OAUTH2_USERDATA_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/userinfo\n OAUTH2_TOKEN_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/token\n OAUTH2_AUTHORIZE_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/authorize\n OAUTH_LOGOUT_REDIRECT_URL: \"https://applicationhub.192-168-49-2.nip.io\"\n OAUTH2_USERNAME_KEY: \"preferred_username\"\n APP_HUB_NAMESPACE: \"app-hub\"\n STORAGE_CLASS: \"standard\"\n RESOURCE_MANAGER_WORKSPACE_PREFIX: \"ws\"\n\n JUPYTERHUB_CRYPT_KEY:\n valueFrom:\n secretKeyRef:\n name: application-hub-secrets\n key: JUPYTERHUB_CRYPT_KEY\n\n OAUTH_CLIENT_ID:\n valueFrom:\n secretKeyRef:\n name: application-hub-secrets\n key: OAUTH_CLIENT_ID\n\n OAUTH_CLIENT_SECRET:\n valueFrom:\n secretKeyRef:\n name: application-hub-secrets\n key: OAUTH_CLIENT_SECRET\n\n image:\n # name: eoepca/application-hub\n # tag: \"1.2.0\"\n pullPolicy: Always\n # pullSecrets: []\n\n db:\n pvc:\n storageClassName: standard\n\n singleuser:\n image:\n name: jupyter/minimal-notebook\n tag: \"2343e33dec46\"\n profileList: \n - display_name: \"Minimal environment\"\n description: \"To avoid too much bells and whistles: Python.\"\n default: \"True\"\n - display_name: \"EOEPCA profile\"\n description: \"Sample profile\"\n kubespawner_override:\n cpu_limit\": 4\n mem_limit\": \"8G\"\n\nnodeSelector:\n key: minikube.k8s.io/primary\n value: \\\"true\\\"\nThe Application Hub requires an OIDC client to be registered with the Identity Service (Keycloak) in order to enable user identity integration - ref. OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=application-hub \\\n --name=\"Application Hub OIDC Client\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Application Hub for OIDC integration\"\nCorresponding to this client, a secret application-hub-secrets must be created (ref. value jupyterhub.hub.existingSecret: application-hub-secrets)\u2026
kubectl -n proc create secret generic application-hub-secrets \\\n --from-literal=JUPYTERHUB_CRYPT_KEY=\"$(openssl rand -hex 32)\" \\\n --from-literal=OAUTH_CLIENT_ID=\"application-hub\" \\\n --from-literal=OAUTH_CLIENT_SECRET=\"changeme\"\nThe deployment of the Application Hub has been designed, as far as possible, to automate the configuration. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026
The default helm chart has some built-in application launchers whose assignments to example users (eric and bob) assume the existence of some JupyterHub groups - which must be replicated to exploit this configuration.
Admin menu (top of page)group-1, group-2, group-3 to ApplicationHub, and add users eric, bob to these groupsThis setup corresponds to the \u2018sample\u2019 configuration that is built=in to the help chart - see file config.yaml.
Additional information regarding the Application Hub can be found at:
To support the development (ref. Application Hub) and deployment/execution (ref. ADES) of user-defined applications, we deploy a container registry to host container images. This is provied by a deployment of the Harbor artefact repository.
"},{"location":"eoepca/container-registry/#helm-chart","title":"Helm Chart","text":"Harbor is deployed via the harbor helm chart from the Harbor Helm Chart Repository.
helm install --version 1.7.3 --values harbor-values.yaml \\\n --repo https://helm.goharbor.io \\\n harbor harbor\nThe chart is configured via values that are fully documented on the Harbor website.
Example\u2026
expose:\n ingress:\n annotations:\n kubernetes.io/ingress.class: nginx\n cert-manager.io/cluster-issuer: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n\n # from chart:\n ingress.kubernetes.io/ssl-redirect: letsencrypt-production\n ingress.kubernetes.io/proxy-body-size: \"0\"\n nginx.ingress.kubernetes.io/ssl-redirect: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n\n hosts:\n core: harbor.192-168-49-2.nip.io\n tls:\n enabled: \"true\"\n certSource: secret\n secret:\n secretName: \"harbor-tls\"\n\npersistence:\n persistentVolumeClaim:\n registry:\n storageClass: standard\n chartmuseum:\n storageClass: standard\n jobservice:\n storageClass: standard\n database:\n storageClass: standard\n redis:\n storageClass: standard\n trivy:\n storageClass: standard\n\nexternalURL: https://harbor.192-168-49-2.nip.io\n# initial password for logging in with user \"admin\"\nharborAdminPassword: \"changeme\"\n\nchartmuseum:\n enabled: false\ntrivy:\n enabled: false\nnotary:\n enabled: false\nNote
letsencrypt-production Cluster Issuer relies upon the deployment being accessible from the public internet via the expose.ingress.hosts.core DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller. The Workspace API will not like this.After deployemnt Harbor is accessible via its web interface at https://harbor.192-168-49-2.nip.io/e.g. https://harbor.192-168-49-2.nip.io/.
Login as the admin user with the password specified in the helm values.
"},{"location":"eoepca/container-registry/#additional-information","title":"Additional Information","text":"Additional information regarding the Container Registry can be found at:
The Data Access provides standards-based services for access to platform hosted data - including OGC WMS/WMTS for visualisation, and OGC WCS for data retrieval. This component also includes Harvester and Registrar services to discover/watch the existing data holding of the infrastructure data layer and populate/maintain the data access and resource catalogue services accordingly.
"},{"location":"eoepca/data-access/#helm-chart","title":"Helm Chart","text":"The Data Access is deployed via the data-access helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are supplied with the instantiation of the helm release. The EOEPCA data-access chart provides a thin wrapper around the EOX View Server (vs) helm chart. The documentation for the View Server can be found here:
helm install --version 1.4.0 --values data-access-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n data-access data-access\nThe Data Access supports many values to configure the service. These are documented in full in the View Server - Operator Guide Configuration page.
"},{"location":"eoepca/data-access/#core-configuration","title":"Core Configuration","text":"Typically, values for the following attributes may be specified to override the chart defaults:
global.ingress.hosts.host[0])database and redis componentsdata and cacherenderer and registraridentity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.global:\n env:\n REGISTRAR_REPLACE: \"true\"\n CPL_VSIL_CURL_ALLOWED_EXTENSIONS: .TIF,.tif,.xml,.jp2,.jpg,.jpeg\n AWS_ENDPOINT_URL_S3: https://minio.192-168-49-2.nip.io\n AWS_HTTPS: \"FALSE\"\n startup_scripts:\n - /registrar_pycsw/registrar_pycsw/initialize-collections.sh\n ingress:\n enabled: true\n annotations:\n kubernetes.io/ingress.class: nginx\n kubernetes.io/tls-acme: \"true\"\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - host: data-access.192-168-49-2.nip.io\n tls:\n - hosts:\n - data-access.192-168-49-2.nip.io\n secretName: data-access-tls\n storage:\n data:\n data:\n type: S3\n endpoint_url: http://data.cloudferro.com\n access_key_id: access\n secret_access_key: access\n region_name: RegionOne\n validate_bucket_name: false\n cache:\n type: S3\n endpoint_url: \"https://minio.192-168-49-2.nip.io\"\n host: \"minio.192-168-49-2.nip.io\"\n access_key_id: xxx\n secret_access_key: xxx\n region: us-east-1\n bucket: cache-bucket\n metadata:\n title: EOEPCA Data Access Service developed by EOX\n abstract: EOEPCA Data Access Service developed by EOX\n header: \"EOEPCA Data Access View Server (VS) Client powered by <a href=\\\"//eox.at\\\"><img src=\\\"//eox.at/wp-content/uploads/2017/09/EOX_Logo.svg\\\" alt=\\\"EOX\\\" style=\\\"height:25px;margin-left:10px\\\"/></a>\"\n url: https://data-access.192-168-49-2.nip.io/ows\n layers:\n # see section 'Data-layer Configuration'\n collections:\n # see section 'Data-layer Configuration'\n productTypes:\n # see section 'Data-layer Configuration'\nvs:\n renderer:\n replicaCount: 4\n ingress:\n enabled: false\n resources:\n requests:\n cpu: 100m\n memory: 300Mi\n limits:\n cpu: 1.5\n memory: 3Gi\n registrar:\n replicaCount: 1\n config:\n # see section 'Registrar Routes Configuration'\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\n harvester:\n # see section 'Harvester Configuration'\n replicaCount: 1\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\n client:\n replicaCount: 1\n ingress:\n enabled: false\n redis:\n master:\n persistence:\n enabled: true\n storageClass: standard\n ingestor:\n replicaCount: 0\n ingress:\n enabled: false\n preprocessor:\n replicaCount: 0\n cache:\n ingress:\n enabled: false\n scheduler:\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\nNote
The resources: above have been limited for the benefit of a minikube deployment. For a production deployment the values should be tuned (upwards) according to operational needs.
The Data Access registrar component supports a number of different resource types. For each a dedicated \u2018backend\u2019 is configured to handle the specific registration of the resource type\u2026
vs:\n registrar:\n config:\n #--------------\n # Default route\n #--------------\n disableDefaultRoute: false\n # Additional backends for the default route\n defaultBackends:\n - path: registrar_pycsw.backend.ItemBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n ows_url: https://data-access.192-168-49-2.nip.io/ows\n defaultSuccessQueue: seed_queue\n #----------------\n # Specific routes\n #----------------\n routes:\n collections:\n path: registrar.route.stac.CollectionRoute\n queue: register_collection_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.CollectionBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n ades:\n path: registrar.route.json.JSONRoute\n queue: register_ades_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.ADESBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n application:\n path: registrar.route.json.JSONRoute\n queue: register_application_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.CWLBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n catalogue:\n path: registrar.route.json.JSONRoute\n queue: register_catalogue_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.CatalogueBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n json:\n path: registrar.route.json.JSONRoute\n queue: register_json_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.JSONBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n xml:\n path: registrar.route.json.JSONRoute\n queue: register_xml_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.XMLBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\nConfiguration of the service data-layer - as described in the View Server Operator Guide.
The data-access service data handling is configured by definition of productTypes, collections and layers\u2026
productTypes - Product Types Identify the underlying file assets as WCS coverages and their visual representationcollections - Data Collections Provides groupings into which products are organisedlayers - Layers Specifies the hoe the product visual representations are exposed through the WMS serviceFor more information, see the worked example in section Data Specification for the example CREODIAS deployment.
"},{"location":"eoepca/data-access/#harvester","title":"Harvester","text":"The Data Access service includes a Harvester component. The following subsections describe its configuration and usage.
"},{"location":"eoepca/data-access/#harvester-helm-configuration","title":"Harvester Helm Configuration","text":"The Harvester can be configured through the helm chart values\u2026
vs:\n harvester:\n replicaCount: 1\n config:\n redis:\n host: data-access-redis-master\n port: 6379\n harvesters:\n - name: Creodias-Opensearch\n resource:\n url: https://datahub.creodias.eu/resto/api/collections/Sentinel2/describe.xml\n type: OpenSearch\n format_config:\n type: 'application/json'\n property_mapping:\n start_datetime: 'startDate'\n end_datetime: 'completionDate'\n productIdentifier: 'productIdentifier'\n query:\n time:\n property: sensed\n begin: 2019-09-10T00:00:00Z\n end: 2019-09-11T00:00:00Z\n collection: null\n bbox: 14.9,47.7,16.4,48.7\n filter: {}\n postprocess:\n - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor\n queue: register\n - name: Creodias-Opensearch-Sentinel1\n resource:\n url: https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml\n type: OpenSearch\n format_config:\n type: 'application/json'\n property_mapping:\n start_datetime: 'startDate'\n end_datetime: 'completionDate'\n productIdentifier: 'productIdentifier'\n query:\n time:\n property: sensed\n begin: 2019-09-10T00:00:00Z\n end: 2019-09-11T00:00:00Z\n collection: null\n bbox: 14.9,47.7,16.4,48.7\n extra_params:\n productType: GRD-COG\n filter: {}\n postprocess:\n - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel1Postprocessor\n queue: register\nThe harvester.config.harvesters list defines a set of pre-defined harvesters which can be invoked in a later stage. The name property must be unique for each harvester and must be unique among all harvesters in the list. Each harvester is associated with a resource, an optional filter or postprocess function, and a queue.
The resource defines where each item is harvested from. This can be a file system, a search service, catalog file or something similar. The example above defines a connection to an OpenSearch service on CREODIAS, with associated default query parameters and a format configuration.
The filter allows to filter elements within the harvester, when the resource does not provide a specific filter. This filter can be supplied using CQL2-JSON.
The postprocess can adjust the harvested results. In this example the harvested items are not complete, and additional metadata must be retrieved from an object storage.
The queue defines where harvested items will be pushed into. Usually this is a registration queue, where the registrar will pick up and start registration according to its configuration.
The harvester can either do one-off harvests via the CLI or listen on a redis queue to run consecutive harvests whenever a harvesting request is received on that queue.
"},{"location":"eoepca/data-access/#one-off-harvests-via-the-cli","title":"One-off harvests via the CLI","text":"In order to start a harvest from the CLI, the operator first needs to connect to the kubernetes pod of the harvester. Within that pod, the harvest can be executed like this\u2026
python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\nThis will invoke the Creodias-Opensearch harvester with default arguments. When some values are to be overridden, the \u2013values switch can be used to pass override values. These values must be a JSON string. The following example adjusts the begin and end times of the query parameters\u2026
python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch --values '{\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}'\nThe harvester pod runs a service listening on a redis queue. When a message is read from the queue, it will be read as a JSON string, expecting an object with at least a name property. Optionally, it can also have a values property, working in the same way as with CLI --values.
To send a harvesting request via the redis queue, it is necessary to connect to the redis pod and execute the redis-cli there. Then the following command can be used to achieve the same result as above with CLI harvesting\u2026
redis-cli LPUSH '{\"name\": \"Creodias-Opensearch\", \"values\": {\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}}'\nThe harvester produces a continous stream of STAC Items which are sent down via the configured queue. It is possible that the harvested metadata is not sufficient to create a fully functional STAC Item. In this case the postprocess must transform this intermediate item to a valid STAC Item. In our example, the postprocessor looks up the Sentinel-2 product file referenced by the product identifier which is then accessed on the object storage. From the stored metadata files, the STAC Items to be sent is created.
"},{"location":"eoepca/data-access/#storage","title":"Storage","text":"Specification of PVCs and access to object storage.
"},{"location":"eoepca/data-access/#persistent-volume-claims","title":"Persistent Volume Claims","text":"The PVCs specified in the helm chart values must be created.
"},{"location":"eoepca/data-access/#pvc-for-database","title":"PVC for Database","text":"kind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n name: data-access-db\n namespace: rm\n labels:\n k8s-app: data-access\n name: data-access\nspec:\n storageClassName: standard\n accessModes:\n - ReadWriteMany\n resources:\n requests:\n storage: 100Gi\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n name: data-access-redis\n namespace: rm\n labels:\n k8s-app: data-access\n name: data-access\nspec:\n storageClassName: standard\n accessModes:\n - ReadWriteMany\n resources:\n requests:\n storage: 1Gi\nThe helm chart values expect specification of object storage details for:
data: to access the EO data of the underlying infrastructurecache: a dedicated object storage bucket is used to support the cache function of the data access servicesSpecifies the details for the infrastructure object storage that provides direct access to the EO product files.
For example, the CREODIAS metadata catalogue provides references to product files in their eodata object storage - the access details for which are configured in the data access services:
global:\n storage:\n data:\n data:\n type: S3\n endpoint_url: http://data.cloudferro.com\n access_key_id: access\n secret_access_key: access\n region_name: RegionOne\n validate_bucket_name: false\nThe Data Access services maintain a cache, which relies on the usage of a dedicate object storage bucket for data persistence. This bucket must be created (manual step) and its access details configured in the data access services. Example based upon CREODIAS:
global:\n storage:\n cache:\n type: S3\n endpoint_url: \"https://cf2.cloudferro.com:8080/cache-bucket\"\n host: \"cf2.cloudferro.com:8080\"\n access_key_id: xxx\n secret_access_key: xxx\n region: RegionOne\n bucket: cache-bucket\n\u2026where xxx must be replaced with the bucket credentials.
As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the data-access service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install data-access-protection identity-gatekeeper -f data-access-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"rm\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the data-access - in particular the specific ingress requirements for the data-access backend services\u2026
Example data-access-protection-values.yaml\u2026
fullnameOverride: data-access-protection\nconfig:\n client-id: data-access\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: data-access.192-168-49-2.nip.io\n name: data-access-renderer\n port:\n number: 80\nsecrets:\n # Values for secret 'data-access-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n nginx.ingress.kubernetes.io/rewrite-target: /$1\n serverSnippets:\n custom: |-\n # Open access to renderer...\n location ~ ^/(ows.*|opensearch.*|coverages/metadata.*|admin.*) {\n proxy_pass http://data-access-renderer.rm.svc.cluster.local:80/$1;\n }\n # Open access to cache...\n location ~ ^/cache/(.*) {\n proxy_pass http://data-access-cache.rm.svc.cluster.local:80/$1;\n }\n # Open access to client...\n # Note that we use a negative lookahead to avoid matching '/.well-known/*' which\n # otherwise appears to interfere with the work of cert-manager/letsencrypt.\n location ~ ^/(?!\\.well-known)(.*) {\n proxy_pass http://data-access-client.rm.svc.cluster.local:80/$1;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: data-access above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=data-access \\\n --name=\"Data Access Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Data Access Gatekeeper\"\nAt deployment time the harvester helm values include configuration that populates a default harvester configuration, that is prepared in the file /config.yaml in the harvester pod.
The Data Access and Resource Catalogue services are configured to properly interpret harvested data via these values specified in the instantiation of the helm release. See section Data-layer Configuration.
The harvesting of data can be triggered (post deployment), in accordance with this default configuration, by connecting to the rm/harvester service and executing the command\u2026
python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\nAd-hoc harvesting can be invoked by provision of a suitable config.yaml into the harvester pod, which can then be invoked as shown above for the default harvester configuration established at deploy time.
The helper script ./deploy/bin/harvest faciltates this\u2026
./deploy/bin/harvest <path-to-config-file>\nSee directory ./deploy/samples/harvester/ that contains some sample harvesting configuration files. For example\u2026
./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2019.09.10.yaml\nThe helper script ./deploy/bin/register-collection is provided to faciltate the registration of collections that are specfied in STAC Collection format.
./deploy/bin/register-collection <path-to-stac-collection-file>\nSee directory ./deploy/samples/collections/ that contains some same STAC Collection files. For example\u2026
./deploy/bin/register-collection ./deploy/samples/collections/S2MSI2A.json\nAdditional information regarding the Data Access can be found at:
This guide includes two approaches for Identity & Access Management:
Until now, our IAM solution has been based solely upon Gluu.
In the course of the project Keycloak has emerged as a preferred solution across EO platforms.
Thus, we have introduced an IAM approach based upon Keycloak, whilst retaining the Gluu-based approach for reference, which will be deprecated.
"},{"location":"eoepca/identity-service/","title":"Identity Service","text":"The Identity Service provides the platform Authorization Server for authenticated user identity and request authorization.
Identity Service is composed of:
The Identity Service is deployed via the identity-service helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values - the full set of available values can be tailored according the helm chart defaults, that can be found here\u2026
identity-service https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/values.yamlidentity-keycloak https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-keycloak/values.yamlidentity-postgres https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-postgres/values.yamlidentity-api https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-api/values.yamlhelm install --version 1.0.97 --values identity-service-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n identity-service identity-service\nThe deployment must be configured for you environment. Some significant configuration values are elaborated here\u2026
"},{"location":"eoepca/identity-service/#identity-keycloak","title":"identity-keycloak","text":""},{"location":"eoepca/identity-service/#secrets","title":"Secrets","text":"Keycloak relies upon a secret identity-keycloak that provides\u2026
KEYCLOAK_ADMIN_PASSWORD - admin password for KeycloakKC_DB_PASSWORD - password for connecting with Postgres DB This should match the POSTGRES_PASSWORD setting for identity-postgres (see below)The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026
identity-keycloak:\n secrets:\n # Values for secret 'identity-keycloak'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n kcDbPassword: \"changeme\"\n keycloakAdminPassword: \"changeme\"\nThe details for ingress (reverse-proxy) to the Keycloak service - in particular the hostname and possible TLS - must be specified\u2026
identity-keycloak:\n ingress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - host: keycloak.192-168-49-2.nip.io\n paths:\n - path: /\n pathType: Prefix\n tls:\n - secretName: identity-keycloak-tls\n hosts:\n - keycloak.192-168-49-2.nip.io\nPostgres relies upon a secret identity-postgres that provides\u2026
POSTGRES_PASSWORD - superuser password for PostgreSQLPGPASSWORD - password used for client connections to the DBThe secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026
identity-postgres:\n secrets:\n # Values for secret 'identity-postgres'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n postgresPassword: \"changeme\"\n pgPassword: \"changeme\"\nIn order to persist data, Postgres requires a Persistent Volume Claim.
This can be specified as an existing volume claim - for example as described in the Persistence section.
identity-postgres:\n volumeClaim:\n name: eoepca-userman-pvc\nThe Identity API relies upon a secret identity-api that provides\u2026
ADMIN_PASSWORD Admin password for Keycloak This should match the KEYCLOAK_ADMIN_PASSWORD setting for identity-keycloak (see above)The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026
identity-api:\n secrets:\n # Values for secret 'identity-api'\n # Note - if ommitted, these can instead be set by creating the secret independently\n # e.g. as a SealedSecret via GitOps.\n adminPassword: \"changeme\"\nNote
It is also possible to set the value of ADMIN_PASSWORD directly as an environment variable. In this case it is necessary to set the secret as optional\u2026
identity-api:\n secrets:\n optional: true\nThe Identity API service can be configured via environment variables as follows\u2026
AUTH_SERVER_URL URL of the Keycloak Authorization Server. Can also be set via value configMap.authServerUrlADMIN_USERNAME Admin user for KeycloakREALM The Keycloak realmidentity-api:\n deployment:\n # Config values that can be passed via env vars\n extraEnv:\n - name: AUTH_SERVER_URL # see configMap.authServerUrl instead\n value: https://keycloak.192-168-49-2.nip.io\n - name: ADMIN_USERNAME\n value: admin\n - name: ADMIN_PASSWORD # see secrets.adminPassword instead\n value: changeme\n - name: REALM\n value: master\ngatekeeper relies upon a secret identity-api-protection that provides\u2026
PROXY_CLIENT_SECRET Password for the Keycloak client configured for use by this Gatekeeper instance - corresponding to config.client-id.PROXY_ENCRYPTION_KEY Encryption Key used by Gatekeeper.The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026
identity-api-gatekeeper:\n secrets:\n # Values for secret 'identity-api-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\nConfiguration of Gatekeeper via the file config.yaml that is mounted into the deployment\u2026
client-id ID of the Keycloak client to be used by this Gatekeeper instance.discovery-url Discovery URL of the Keycloak Authorization Servercookie-domain Domain in which this Gatekeeper instance creates cookies identity-api-gatekeeper:\n config:\n client-id: identity-api\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\nThe details for ingress (reverse-proxy) to the Gatekeeper service that protects the Identity API\u2026
identity-api-gatekeeper:\n targetService:\n host: identity-api.192-168-49-2.nip.io\n ingress:\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt\nThe Identity API is protected via an instance of Gatekeeper - which relies upon a Keycloak client having been created for authorization decision/enforcement flows between Gatekeeper and Keycloak.
As described in the \u2018create-client\u2019 section below, this can be achieved using the create-client helper script.
Note
At time of client creation, the Identity API is not yet protected with an ingress. Therefore, we use a port-forward to interface directly with the Identity API service.
$ kubectl -n um port-forward svc/identity-api \"9876\":http >/dev/null &\n$ portForwardPid=$!\n\n$ ./deploy/bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i http://localhost:9876 \\\n -r master \\\n -u admin \\\n -p changeme \\\n -c admin-cli \\\n --id=identity-api \\\n --name=\"Identity API Gatekeeper\" \\\n --secret=changeme \\\n --description=\"Client to be used by Identity API Gatekeeper\" \\\n --resource=\"admin\" --uris='/*' --scopes=view --users=\"admin\"\n\n$ kill -TERM $portForwardPid\ncreate-user Helper Script","text":"The Keycloak Admin UI can be used to create users interactively.
Alternatvely there is a helper script create-user that can be used.
The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe create-user helper script requires some command-line arguments\u2026
$ ./deploy/bin/create-user -h\n\nCreate a new user.\ncreate-user -h | -a {auth_server} -r {realm} -c {client} -u {admin-username} -p {admin-password} -U {new-username} -P {new-password}\n\nwhere:\n -h show help message\n -a authorization server url (default: http://keycloak.192-168-49-2.nip.io)\n -r realm within Keycloak (default: master)\n -u username used for authentication (default: admin)\n -p password used for authentication (default: changeme)\n -c client id of the bootstrap client used in the create request (default: admin-cli)\n -U name of the (new) user to create\n -P password for the (new) user to create\nThe Identity Service is capable of protecting resources using OpenID-connect/SAML clients, resources (URIs/scopes), policies (user based, role based, etc) and permissions (associations between policies and resources).
Creating and protecting resources can be done in multiple ways, as described in the following sections.
"},{"location":"eoepca/identity-service/#keycloak-admin-ui","title":"Keycloak Admin UI","text":"To create and protect resources using the keycloak User Interface (UI), do the following steps:
create-client Helper Script","text":"Alternatively, a script was developed to allow simultaneaously create a client, create resources and protect them.
The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe create-client helper script requires some command-line arguments\u2026
$ ./deploy/bin/create-client -h\n\nAdd a client with protected resources.\ncreate-client [-h] [-a] [-i] [-u] [-p] [-c] [-s] [-t | --token t] [-r] --id id [--name name] (--secret secret | --public) [--default] [--authenticated] [--resource name] [--uris u1,u2] [--scopes s1,s2] [--users u1,u2] [--roles r1,r2]\n\nwhere:\n -h show help message\n -a authorization server url - e.g. https://keycloak.192-168-49-2.nip.io\n -i identity-api server url - e.g. https://identity-api.192-168-49-2.nip.io\n -u username used for authentication\n -p password used for authentication\n -c client id (of the bootstrap client used in the create request)\n -s client secret (of the bootstrap client used in the create request)\n -t or --token access token used for authentication\n -r realm\n --id client id (of the created client)\n --name client name (of the created client)\n --secret client secret (of the created client)\n --public public client (no client secret)\n --default add default resource - /* authenticated\n --authenticated allow access to the resource only when authenticated\n --resource resource name\n --uris resource uris - separated by comma (,)\n --scopes resource scopes - separated by comma (,)\n --users user names with access to the resource - separated by comma (,)\n --roles role names with access to the resource - separated by comma (,)\nThe script interacts with Identity API and therefore requires admin authorization. It accepts basic authentication with username and password with -u and -p parameters, respectively - or a bearer access token with -t parameter.
To generate the access token needed to use the script, you can get it through the login in the eoepca portal, by accessing the cookies in the browser. See section EOEPCA Portal for details regarding deployment/configuration of the eoepca-portal.
Or you can generate an access token using postman oauth2.0, as described in the Postman document Requesting an OAuth 2.0 token.
Script execution examples:
With username/password
./deploy/bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r master \\\n -u admin \\\n -p changeme \\\n -c admin-cli \\\n --id=myservice-gatekeeper \\\n --name=\"MyService Gatekeeper\" \\\n --secret=changeme \\\n --description=\"Client to be used by MyService Gatekeeper\" \\\n --resource=\"Eric space\" --uris=/eric/* --users=eric \\\n --resource=\"Alice space\" --uris=/alice/* --users=alice \\\n --resource=\"Admin space\" --uris=/admin/* --roles=admin\nWith access token
./deploy/bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r master \\\n -t eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJXZWFIY2pscThPc1RUYjdlV0s5SjJTTDFBUDIyazZpajdlMGFlVHRNU2xRIn0.eyJleHAiOjE3MDAyNDM4MzgsImlhdCI6MTcwMDI0Mzc3OCwiYXV0aF90aW1lIjoxNzAwMjQxODYyLCJqdGkiOiI2MWI0ZGRhYy1mOWZjLTRmZjktOWQ4Zi01NWU1N2NlNmE5ODgiLCJpc3MiOiJodHRwczovL2lkZW50aXR5LmtleWNsb2FrLmRldmVsb3AuZW9lcGNhLm9yZy9yZWFsbXMvbWFzdGVyIiwiYXVkIjpbImFkZXMtcmVhbG0iLCJkZW1vLXJlYWxtIiwiZHVtbXktc2VydmljZS1yZWFsbSIsIm1hc3Rlci1yZWFsbSIsImFjY291bnQiLCJlb2VwY2EtcmVhbG0iXSwic3ViIjoiZTNkZTMyNGUtMGY0NS00MWUwLTk2YTctNTM1YzkxMTA1NTUyIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiZW9lcGNhLXBvcnRhbCIsIm5vbmNlIjoiMTIwMGJlNzAtZWI1Ni00Nzc2LThjODgtOWRiOWQxMDdiMGY2Iiwic2Vzc2lvbl9zdGF0ZSI6ImVmNGUwOTlmLTFmMDgtNDY3MC04ZmE2LTJiOGI3OGUwNWMzMSIsImFjciI6IjAiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiY3JlYXRlLXJlYWxtIiwiZGVmYXVsdC1yb2xlcy1tYXN0ZXIiLCJvZmZsaW5lX2FjY2VzcyIsImFkbWluIiwidW1hX2F1dGhvcml6YXRpb24iLCJ1c2VyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWRlcy1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LWlkZW50aXR5LXByb3ZpZGVycyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwiZGVtby1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LXJlYWxtIiwidmlldy1pZGVudGl0eS1wcm92aWRlcnMiLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwiZHVtbXktc2VydmljZS1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LXJlYWxtIiwidmlldy1pZGVudGl0eS1wcm92aWRlcnMiLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwibWFzdGVyLXJlYWxtIjp7InJvbGVzIjpbInZpZXctaWRlbnRpdHktcHJvdmlkZXJzIiwidmlldy1yZWFsbSIsIm1hbmFnZS1pZGVudGl0eS1wcm92aWRlcnMiLCJpbXBlcnNvbmF0aW9uIiwiY3JlYXRlLWNsaWVudCIsIm1hbmFnZS11c2VycyIsInF1ZXJ5LXJlYWxtcyIsInZpZXctYXV0aG9yaXphdGlvbiIsInF1ZXJ5LWNsaWVudHMiLCJxdWVyeS11c2VycyIsIm1hbmFnZS1ldmVudHMiLCJtYW5hZ2UtcmVhbG0iLCJ2aWV3LWV2ZW50cyIsInZpZXctdXNlcnMiLCJ2aWV3LWNsaWVudHMiLCJtYW5hZ2UtYXV0aG9yaXphdGlvbiIsIm1hbmFnZS1jbGllbnRzIiwicXVlcnktZ3JvdXBzIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX0sImVvZXBjYS1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LWlkZW50aXR5LXByb3ZpZGVycyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfX0sInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJzaWQiOiJlZjRlMDk5Zi0xZjA4LTQ2NzAtOGZhNi0yYjhiNzhlMDVjMzEiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInByZWZlcnJlZF91c2VybmFtZSI6ImFkbWluIn0.FK6DhVzpCRFmef2acD2Hmc149e1GTOCGz13dZA828crFbG8j4uhpkoNpiZqdyOPmDtMQ-OebNfjTAUaOt2sS1FmEIBgb9IddcpHKNJOquRjdzQNsX09bX8pFUq1haGwKh6_QmABNOBcT-kQNDSZO-aq7-8FoO9PYa0GWvBRcbcx0W_ngyb7xHglaZTElzcDPBcUTW6llVTTTFygn55smwdxTZ7-tEsMVGM5gNuHwJyLB51HI5KDWrwgUm1hqhhRzvcoutDEAB_HSEXGNNeF7fjP9Qx6q04b7fKOTtnIlXsu3oYW4va9y754llMSJ7w8U-y7yI6Tm2UdNMdYqju7hAA \\\n -c admin-cli \\\n --id=myservice-gatekeeper \\\n --name=\"MyService Gatekeeper\" \\\n --secret=changeme \\\n --description=\"Client to be used by MyService Gatekeeper\" \\\n --resource=\"Eric space\" --uris=/eric/* --users=eric \\\n --resource=\"Alice space\" --uris=/alice/* --users=alice \\\n --resource=\"Admin space\" --uris=/admin/* --roles=admin\nAlso, an API was developed to interact more easily with the Keycloak API, that allows client, resource, policies and permissions management.
The API documentation can be found in its Swagger UI at the service endpoint - https://identity-api.192-168-49-2.nip.io/docs.
The Identity API is best used in combination with the eoepca-portal test aide, which can be used to establish a login sesssion in the browser to the benefit of the Identity API swagger UI. See section EOEPCA Portal for details regarding deployment/configuration of the eoepca-portal.
By default the Access Token Lifespan is 1 minute. With the current ADES (zoo-dru) implementation this presents a problem - since the access_token that is provided to the process execute request will (most likely) have expired by the time the ADES attempts to use the access_token in its call to the Workspace API to register the processing outputs. The lifespan of the token must outlive the duration of the processing execution - which we must assume can take a long time.
To avoid this potential problem, the Keycloak Admin web console can be used to increase this token lifespan.
Thus, the following settings are recommended to be updated following deployment\u2026
1 day1 day1 dayAdditional information regarding the Identity Service can be found at:
The Login Service provides the platform Authorization Server for authenticated user identity and request authorization.
"},{"location":"eoepca/login-service/#helm-chart","title":"Helm Chart","text":"The Login Service is deployed via the login-service helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the login-service chart.
helm install --version 1.2.8 --values login-service-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n login-service login-service\nAt minimum, values for the following attributes should be specified:
auth.192-168-49-2.nip.io192.168.49.2namespace for the login-service componentslogin-service persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.letsencrypt-productionExample login-service-values.yaml\u2026
global:\n domain: auth.192-168-49-2.nip.io\n nginxIp: 192.168.49.2\n namespace: um\nvolumeClaim:\n name: eoepca-userman-pvc\n create: false\nconfig:\n domain: auth.192-168-49-2.nip.io\n adminPass: Chang3me!\n ldapPass: Chang3me!\n volumeClaim:\n name: eoepca-userman-pvc\nopendj:\n volumeClaim:\n name: eoepca-userman-pvc\n resources:\n requests:\n cpu: 100m\n memory: 300Mi\noxauth:\n volumeClaim:\n name: eoepca-userman-pvc\n resources:\n requests:\n cpu: 100m\n memory: 1000Mi\noxtrust:\n volumeClaim:\n name: eoepca-userman-pvc\n resources: \n requests:\n cpu: 100m\n memory: 1500Mi\noxpassport:\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\nnginx:\n ingress:\n annotations:\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - auth.192-168-49-2.nip.io\n tls:\n - hosts:\n - auth.192-168-49-2.nip.io\n secretName: login-service-tls\nNote
The resources: above have been limited for the benefit of a minikube deployment. For a production deployment the values should be tuned (upwards) according to operational needs.
The deployment of the Login Service has been designed, as far as possible, to automate the configuration. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026
UMA Resource LifetimeOperator userThe Login Service maintains a background service that \u2018cleans\u2019 UMA resources that are older than aa certain age - by default 30 days (2592000 secs). This lifetime does not fit the approach we are adopting, and so we must update this lifetime value to avoid the unexpected removal of UMA resources that would cause unexpected failures in policy enforcement.
https://auth.192-168-49-2.nip.io/ - and login as the admin userConfiguration -> JSON Configuration -> OxAuth ConfigurationumaResourceLifetimeumaResourceLifetime to 2147483647Save Configurationoxauth deployment\u2026 kubectl -n um rollout restart deploy/login-service-oxauth\nOperator user","text":"The default resource protection establishes policy in which \u2018operator\u2019 privilege is required for some services, such as the Workspace API. Thus, we need to configure a user with this privilege. For convenience we add this attribute to the built-in admin user - but alternatively you may choose to create a new user for this role.
https://auth.192-168-49-2.nip.io/ - and login as the admin userUsers -> Manage People and search for user adminadmin select Available User Claims -> gluuCustomPersonIs Operator and ensure the value is set TrueUpdate to confirmOnce the deployment has been completed successfully, the Login Service is accessed at the endpoint https://auth.192-168-49-2.nip.io/, configured by your domain - e.g. https://auth.192-168-49-2.nip.io/.
Login as the admin user with the credentials configured in the helm values - ref. adminPass / ldapPass.
Typical first actions to undertake through the Gluu web interface include creation of users and clients.
"},{"location":"eoepca/login-service/#additional-information","title":"Additional Information","text":"Additional information regarding the Login Service can be found at:
The Policy Decision Point (PDP) provides the platform policy database and associated service for access policy decision requests.
"},{"location":"eoepca/pdp/#helm-chart","title":"Helm Chart","text":"The PDP is deployed via the pdp-engine helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the pdp-engine chart.
helm install --version 1.1.12 --values pdp-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n pdp pdp-engine\nAt minimum, values for the following attributes should be specified:
auth.192-168-49-2.nip.io192.168.49.2pdp-engine persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.Example pdp-values.yaml\u2026
global:\n nginxIp: 192.168.49.2\n domain: auth.192-168-49-2.nip.io\nvolumeClaim:\n name: eoepca-userman-pvc\n create: false\nAdditional information regarding the PDP can be found at:
The EOEPCA building-blocks rely upon Kubernetes Persistent Volumes for their component persistence. Components integrate with the storage provided in the cluster by means of configurable Persistent Volume Claims and/or dynamic Storage Class that are specfied as values at time of deployment. Some components require storage of type ReadWriteMany - which, for a multi-node cluster, implies a network-based storage solution.
Note
Local CLuster Storage For the purposes of the Scripted Deployment, the default Storage Class included with the local Kubernetes distribution can be used for all storage concerns - e.g. standard for minikube which provides the ReadWriteMany persistence that is required by the ADES.
For the EOEPCA development deployment, an NFS server has been established to provide the persistence layer for ReadWriteMany storage.
The EOEPCA development deployment establishes the following pre-defined Persistent Volume Claims, to provide a simple storage architecture that is organised around the \u2018domain areas\u2019 into which the Reference Implementation is split.
resman) - persistentvolumeclaim/eoepca-resman-pvcproc) - persistentvolumeclaim/eoepca-proc-pvcuserman) - persistentvolumeclaim/eoepca-userman-pvcNOTE that this is offered only as an example thay suits the approach of the development team. Each building-block has configuration through which its persistence (PV/PVC) can be configured according the needs of the deployment.
The following Kubernetes yaml provides an example of provisioning such domain-specific PersistentVolumeClaims within the cluster - in this case using the minikube built-in storage-class standard for dynamic provisioning\u2026
---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: eoepca-proc-pvc\n namespace: proc\nspec:\n accessModes:\n - ReadWriteMany\n storageClassName: standard\n resources:\n requests:\n storage: 5Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: eoepca-resman-pvc\n namespace: rm\nspec:\n accessModes:\n - ReadWriteMany\n storageClassName: standard\n resources:\n requests:\n storage: 5Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: eoepca-userman-pvc\n namespace: um\nspec:\n accessModes:\n - ReadWriteMany\n storageClassName: standard\n resources:\n requests:\n storage: 5Gi\nOnce established, these PersistentVolumeClaims are then referenced within the deployment configurations of the building-blocks.
"},{"location":"eoepca/persistence/#dynamic-readwritemany-storage-provisioning","title":"DynamicReadWriteMany Storage Provisioning","text":"In addition to the pre-defined PV/PVCs, the EOEPCA Reference Implementation also defines NFS-based storage classes for dynamic storage provisioning:
managed-nfs-storage With a Reclaim Policy of Delete.managed-nfs-storage-retain With a Reclaim Policy of Retain.The building-blocks simply reference the required Storage Class in their volume specifications, to receive a Persistent Volume Claim that is dynamically provisioned at deployment time.
This is acheived through the nfs-provisioner helm chart, with the following typical configurations\u2026
Reclaim Policy Delete\u2026
provisionerName: nfs-storage\nstorageClass:\n name: managed-nfs-storage\n create: true\n reclaimPolicy: Delete\n archiveOnDelete: false\n allowVolumeExpansion: true\nnfs:\n server: \"<your-nfs-ip-address-here>\"\n path: /data/dynamic # your NFS server path here\nReclaim Policy Retain\u2026
provisionerName: nfs-storage-retain\nstorageClass:\n name: managed-nfs-storage-retain\n create: true\n reclaimPolicy: Retain\n allowVolumeExpansion: true\nnfs:\n server: \"<your-nfs-ip-address-here>\"\n path: /data/dynamic # your NFS server path here\nClustered storage approaches offer an alternative to NFS. Clustered Storage provides a network-attached storage through a set of commodity hosts whose storage is aggregated to form a distributed file-system. Capacity is scaled by adding additional nodes or adding additional storage to the existing nodes. In the context of a multi-node Kubernetes cluster, then it is typical that the same commodity nodes provide both the cluster members and storage resources, i.e. the clustered storage is spread across the Kubernetes worker nodes.
Candidate clustered storage solutions include:
All things being equal, Longhorn is recommended as the best approach for Kubernetes clusters.
"},{"location":"eoepca/registration-api/","title":"Registration API","text":"The Registration API provides a REST API through which resources can be registered with both the Resource Catalogue and (as applicable) with the Data Access services.
"},{"location":"eoepca/registration-api/#helm-chart","title":"Helm Chart","text":"The Registration API is deployed via the rm-registration-api helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the rm-registration-api chart.
helm install --version 1.4.0 --values registration-api-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n registration-api rm-registration-api\nThe Registration API supports many values to configure the service - as described in the Values section of the chart README.
Typically, values for the following attributes may be specified:
identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.Example registration-api-values.yaml\u2026
fullnameOverride: registration-api\n\ningress:\n enabled: false\n hosts:\n - host: registration-api-open.192-168-49-2.nip.io\n paths: [\"/\"]\n tls:\n - hosts:\n - registration-api-open.192-168-49-2.nip.io\n secretName: registration-api-tls\n\n# some values for the workspace API\nworkspaceK8sNamespace: rm\nredisServiceName: \"data-access-redis-master\"\nAs described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the registration-api service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install registration-api-protection identity-gatekeeper -f registration-api-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"rm\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the registration-api - in particular the specific ingress requirements for the registration-api backend service\u2026
Example registration-api-protection-values.yaml\u2026
fullnameOverride: registration-api-protection\nconfig:\n client-id: registration-api\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: registration-api.192-168-49-2.nip.io\n name: registration-api\n port:\n number: 8080\nsecrets:\n # Values for secret 'registration-api-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n serverSnippets:\n custom: |-\n # Open access...\n location ~ ^/ {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: registration-api above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=registration-api \\\n --name=\"Registration API Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Registration API Gatekeeper\"\nAdditional information regarding the Registration API can be found at:
The Resource Catalogue provides a standards-based EO metadata catalogue that includes support for OGC CSW / API Records, STAC and OpenSearch.
"},{"location":"eoepca/resource-catalogue/#helm-chart","title":"Helm Chart","text":"The Resource Catalogue is deployed via the rm-resource-catalogue helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the rm-resource-catalogue chart.
helm install --version 1.4.0 --values resource-catalogue-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n resource-catalogue rm-resource-catalogue\nThe Resource Catalogue supports many values to configure the service - as described in the Values section of the chart README.
Typically, values for the following attributes may be specified:
identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.db.config.XXX.Example resource-catalogue-values.yaml\u2026
global:\n namespace: rm\n# For protected access disable this ingress, and rely upon the identity-gatekeeper\n# for ingress with protection.\ningress:\n # Enabled for unprotected 'open' access to the resource-catalogue.\n enabled: true\n name: resource-catalogue\n host: resource-catalogue.192-168-49-2.nip.io\n tls_host: resource-catalogue.192-168-49-2.nip.io\n tls_secret_name: resource-catalogue-tls\n annotations:\n cert-manager.io/cluster-issuer: letsencrypt-production\ndb:\n volume_storage_type: standard\n # config:\n # enabled: true\n # shared_buffers: 2GB\n # effective_cache_size: 6GB\n # maintenance_work_mem: 512MB\n # checkpoint_completion_target: 0.9\n # wal_buffers: 16MB\n # default_statistics_target: 100\n # random_page_cost: 4\n # work_mem: 4MB\n # cpu_tuple_cost: 0.4\npycsw:\n config:\n server:\n url: https://resource-catalogue.192-168-49-2.nip.io/\n manager:\n transactions: \"true\"\n allowed_ips: \"*\"\nNote
The above example values enable transactions (write-access) to the catalogue from any IP address. This is convenient for testing/demonstration of the capability, but should be disbaled or restricted for operational deployments.
"},{"location":"eoepca/resource-catalogue/#protection","title":"Protection","text":"As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the resource-catalogue service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install resource-catalogue-protection identity-gatekeeper -f resource-catalogue-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"rm\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the resource-catalogue - in particular the specific ingress requirements for the resource-catalogue-service\u2026
Example resource-catalogue-protection-values.yaml\u2026
fullnameOverride: resource-catalogue-protection\nconfig:\n client-id: resource-catalogue\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: resource-catalogue.192-168-49-2.nip.io\n name: resource-catalogue-service\n port:\n number: 80\nsecrets:\n # Values for secret 'resource-catalogue-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n serverSnippets:\n custom: |-\n # Open access...\n location ~ ^/ {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: resource-catalogue above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=resource-catalogue \\\n --name=\"Resource Catalogue Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Resource Catalogue Gatekeeper\"\nThe Resource Catalogue is initially populated during the initialisation of the Data Access service. See section Data-layer Configuration.
The Resource Catalogue is accessed at the endpoint https://resource-catalogue.192-168-49-2.nip.io/, configured by your domain - e.g. https://resource-catalogue.192-168-49-2.nip.io/.
As described in the pycsw documentation, ISO XML records can be loaded into the resource-catalogue using the pycsw-admin.py admin utility\u2026
pycsw-admin.py load_records -c /path/to/cfg -p /path/to/records\nThe /path/to/records can either be a single metadata file, or a directory containing multiple metadata files.
This is most easily achieved via connection to the pycsw pod, which includes the pycsw-admin.py utility and the pycsw configuration file at /etc/pycsw/pycsw.cfg\u2026
kubectl -n rm cp \"<metadata-file-or-directory>\" \"<pycsw-pod-name>\":/tmp/metadata\nkubectl -n rm exec -i \"<pycsw-pod-name>\" -- pycsw-admin.py load-records -c /etc/pycsw/pycsw.cfg -p /tmp/metadata\nThe name of the pycsw pod can be obtained using kubectl\u2026
kubectl -n rm get pod --selector='io.kompose.service=pycsw' --output=jsonpath={.items[0].metadata.name}\nTo facilitate the loading of records via the pycsw pod, a helper script load-records has been provided in the git repository that hosts this document\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n./deploy/bin/load-records \"<metadata-file-or-directory>\"\nThe helper script identifies the pycsw pod, copies the metadata files to the pod, and runs pycsw-admin.py load-records within the pod to load the records.
Additional information regarding the Resource Catalogue can be found at:
EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes:
Building Blocks that act as a Resource Server are individually protected by a Policy Enforcement Point (PEP). The PEP enforces the authorization decision in collaboration with the Login Service and Policy Decision Point (PDP).
The PEP expects to interface to a client (user agent, e.g. browser) using User Managed Access (UMA) flows. It is not typical for a client to support UMA flows, and so the PEP can be deployed with a companion UMA User Agent component that interfaces between the client and the PEP, and performs the UMA Flow on behalf of the client.
The Resource Guard is a \u2018convenience\u2019 component that deploys the PEP & UMA User Agent as a cooperating pair.
The Resource Guard \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, the Resource Guard deploys with an Ingress specification that:
auth_request module to defer access authorization to the uma-user-agent serviceThe Resource Guard is deployed via the resource-guard helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the resource-guard chart.
It is expected to deploy multiple instances of the Resource Guard chart, one for each Resource Server to be protected.
helm install --version 1.3.1 --values myservice-guard-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n myservice-guard resource-guard\nThe helm chart is deployed with values that are passed through to the subcharts for the pep-engine and uma-user-agent. Typical values to be specified include:
auth.192-168-49-2.nip.io192.168.49.2pep-engine persistence, e.g. myservice-pep-pvcletsencrypt-productionSecret that contains the client credentials used by the uma-user-agent to interface with the Login Service. See section Client Secret belowExample myservice-guard-values.yaml\u2026
#---------------------------------------------------------------------------\n# Global values\n#---------------------------------------------------------------------------\nglobal:\n context: myservice\n domain: 192-168-49-2.nip.io\n nginxIp: 192.168.49.2\n certManager:\n clusterIssuer: letsencrypt-production\n#---------------------------------------------------------------------------\n# PEP values\n#---------------------------------------------------------------------------\npep-engine:\n configMap:\n asHostname: auth\n pdpHostname: auth\n customDefaultResources:\n - name: \"Eric's space\"\n description: \"Protected Access for eric to his space in myservice\"\n resource_uri: \"/ericspace\"\n scopes: []\n default_owner: \"d3688daa-385d-45b0-8e04-2062e3e2cd86\"\n volumeClaim:\n name: myservice-pep-pvc\n create: false\n#---------------------------------------------------------------------------\n# UMA User Agent values\n#---------------------------------------------------------------------------\numa-user-agent:\n nginxIntegration:\n enabled: true\n hosts:\n - host: myservice\n paths:\n - path: /(.*)\n service:\n name: myservice\n port: 80\n - path: /(doc.*)\n service:\n name: myservice-docs\n port: 80\n annotations:\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n nginx.ingress.kubernetes.io/rewrite-target: /$1\n client:\n credentialsSecretName: \"myservice-agent\"\n logging:\n level: \"debug\"\n unauthorizedResponse: 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"'\n#---------------------------------------------------------------------------\n# END values\n#---------------------------------------------------------------------------\nThe uma-user-agent requires Client Credentials for its interactions with the login-service. The uma-user-agent expects to read these credentials from the file client.yaml, in the form\u2026
client-id: <my-client-id>\nclient-secret: <my-secret>\nTo obtain the Client Credentials required by the uma-user-agent it is necessary to register a client with the login-service, or use the credentials for an existing client.
A helper script is provided to register a basic client and obtain the required credentials. The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe register-client helper script requires some command-line arguments\u2026
Usage:\n register_client <authorization-server-hostname> <client-name> [<redirect-uri> [<logout-uri>]]\nFor example\u2026
./deploy/bin/register-client auth.192-168-49-2.nip.io myclient\n\nINFO: Preparing docker image... [done]\nClient successfully registered.\nMake a note of the credentials:\nclient-id: a98ba66e-e876-46e1-8619-5e130a38d1a4\nclient-secret: 73914cfc-c7dd-4b54-8807-ce17c3645558\nOr to register OIDC redirect URLs\u2026
./deploy/bin/register-client auth.192-168-49-2.nip.io myclient https://portal.192-168-49-2.nip.io/oidc/callback/ https://portal.192-168-49-2.nip.io/logout\nThe script writes the \u2018client credentials\u2019 to stdout - in the expected YAML configuration file format - which can be redirected to file\u2026
./deploy/bin/register-client auth.192-168-49-2.nip.io myclient | tee client.yaml\nclient.yaml. NOTE that the register-client helper relies upon docker to build and run the script.
The client.yaml configuration file is made available via a Kubernetes Secret\u2026
kubectl -n myservice-ns create secret generic myservice-agent \\\n --from-file=client.yaml \\\n --dry-run=client -o yaml \\\n > myservice-agent-secret.yaml\napiVersion: v1\nkind: Secret\nmetadata:\n name: myservice-agent\n namespace: myservice-ns\ndata:\n client.yaml: Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4\nThe resource-guard deployment is configured with the name of the Secret through the helm chart value client.credentialsSecretName.
As described in the README for the Resource Guard, it is necessary for a request to a protected resource to provide the User ID Token in the request header.
"},{"location":"eoepca/resource-protection-gluu/#obtaining-the-user-id-token","title":"Obtaining the User ID Token","text":"In the simple case of a user with username/password held within the Login Service, the User ID Token can be obtained as follows:
curl --location --request POST 'https://auth.192-168-49-2.nip.io/oxauth/restv1/token' \\\n--header 'Cache-Control: no-cache' \\\n--header 'Content-Type: application/x-www-form-urlencoded' \\\n--data-urlencode 'scope=openid user_name is_operator' \\\n--data-urlencode 'grant_type=password' \\\n--data-urlencode 'username=<username>' \\\n--data-urlencode 'password=<password>' \\\n--data-urlencode 'client_id=<client-id>' \\\n--data-urlencode 'client_secret=<client-password>'\nThe User ID Token is included in the id_token field of the json response.
Alternatively, OAuth/OIDC flows can be followed to authenticate via external identity providers.
"},{"location":"eoepca/resource-protection-gluu/#user-id-token-in-http-requests","title":"User ID Token in HTTP requests","text":"The Resource Guard protection supports presentation of the User ID Token via the following HTTP request headers (in order of priority)\u2026
Authorization header as a bearer token - in the form: Authorization: Bearer <token>X-User-Id headerCookie: auth_user_id=<token> Note that the name of the cookie is configurable
Additional information regarding the Resource Guard can be found at:
EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes:
Building Blocks that act as a Resource Server are individually protected by a dedicated Gatekeeper instance that enforces the authorization decision in collaboration with the Identity Service (Keycloak).
Gatekeeper \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, Gatekeeper deploys with an Ingress specification that:
auth_request module to defer access authorization to the gatekeeper serviceEach Gatekeeper is deployed via the identity-gatekeeper helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values - the full set of available values can be seen at https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/values.yaml.
It is expected to deploy multiple instances of the Gatekeeper chart, one for each Resource Server to be protected.
helm install --version 1.0.10 --values myservice-gatekeeper-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n myservice-protection identity-gatekeeper\nThe helm chart is deployed with values that customise the service for the specific needs of the resource-server under protection and the deployment target platform. Typical values to be specified include:
keycloak.192-168-49-2.nip.ioletsencrypt-productionExample myservice-protection-values.yaml\u2026
nameOverride: myservice-protection\nconfig:\n client-id: myservice\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: myservice.192-168-49-2.nip.io\n name: myservice\n port:\n number: 80\nsecrets:\n # Values for secret 'myservice-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n serverSnippets:\n custom: |-\n # Open access to some endpoints, including Swagger UI\n location ~ ^/(docs|openapi.json|probe) {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nGatekeeper requires Client Credentials for its interactions with the Keycloak identity-service. These credentials must be supplied by the secret named <myservice>-protection. The secret can be created directly by the helm chart - via the values secrets.clientSecret and secrets.encryptionKey - or perhaps more securely the secret can be created independently (e.g. via a SealedSecret).
The Keycloak client can be created directly in the Keycloak admin console - e.g. via https://keycloak.192-168-49-2.nip.io/admin.
As an aide there is a helper script create-client. The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe create-client helper script requires some command-line arguments\u2026
$ ./deploy/bin/create-client -h\n\nAdd a client with protected resources.\ncreate-client [-h] [-a] [-i] [-u] [-p] [-c] [-s] [-t | --token t] [-r] --id id [--name name] (--secret secret | --public) [--default] [--authenticated] [--resource name] [--uris u1,u2] [--scopes s1,s2] [--users u1,u2] [--roles r1,r2]\n\nwhere:\n -h show help message\n -a authorization server url - e.g. https://keycloak.192-168-49-2.nip.io\n -i identity-api server url - e.g. https://identity-api.192-168-49-2.nip.io\n -u username used for authentication\n -p password used for authentication\n -c client id (of the bootstrap client used in the create request)\n -s client secret (of the bootstrap client used in the create request)\n -t or --token access token used for authentication\n -r realm\n --id client id (of the created client)\n --name client name (of the created client)\n --secret client secret (of the created client)\n --public public client (no client secret)\n --default add default resource - /* authenticated\n --authenticated allow access to the resource only when authenticated\n --resource resource name\n --uris resource uris - separated by comma (,)\n --scopes resource scopes - separated by comma (,)\n --users user names with access to the resource - separated by comma (,)\n --roles role names with access to the resource - separated by comma (,)\nFor example\u2026
./deploy/bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=myservice \\\n --name=\"Gatekeeper for myservice\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Gatekeeper for myservice\" \\\n --resource=\"eric\" --uris='/eric/*' --scopes=view --users=\"eric\" \\\n --resource=\"bob\" --uris='/bob/*' --scopes=view --users=\"bob\" \\\n --resource=\"alice\" --uris='/alice/*' --scopes=view --users=\"alice\"\nRequests to resource server endpoints that are protected by Gatekeeper must carry an Access Token that has been obtained on behalf of the requesting user. The access_token is carried in the request header\u2026
Authorization: Bearer <access_token>\nThe Access Token for a given user can be obtained with a call to the token endpoint of the Keycloak Identity Service - supplying the credentials for the user and the pre-registered client\u2026
curl -L -X POST 'https://keycloak.192-168-49-2.nip.io/realms/master/protocol/openid-connect/token' \\\n -H 'Cache-Control: no-cache' \\\n -H 'Content-Type: application/x-www-form-urlencoded' \\\n --data-urlencode 'scope=openid profile email' \\\n --data-urlencode 'grant_type=password' \\\n --data-urlencode 'username=<username>' \\\n --data-urlencode 'password=<password>' \\\n --data-urlencode 'client_id=admin-cli'\nA json response is returned, in which the field access_token provides the Access Token for the specified <username>.
Additional information regarding the Gatekeeper can be found at:
The User Profile represents the user\u2019s \u2018account\u2019 within the platform.
"},{"location":"eoepca/user-profile/#helm-chart","title":"Helm Chart","text":"The User Profile is deployed via the user-profile helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the user-profile chart.
helm install --version 1.1.12 --values user-profile-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n user-profile user-profile\nAt minimum, values for the following attributes should be specified:
auth.192-168-49-2.nip.io192.168.49.2user-profile persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.Example user-profile-values.yaml\u2026
global:\n domain: auth.192-168-49-2.nip.io\n nginxIp: 192.168.49.2\nvolumeClaim:\n name: eoepca-userman-pvc\n create: false\nThe User Profile is accessed through the /web_ui path of the Login Service, e.g. http://auth.kube.guide.eoepca.org/web_ui.
Additional information regarding the User Profile can be found at:
The Workspace provides protected user resource management that includes dedicated storage and services for resource discovery and access.
"},{"location":"eoepca/workspace/#workspace-api","title":"Workspace API","text":"The Workspace API provides a REST service through which user workspaces can be created, interrogated, managed and deleted.
"},{"location":"eoepca/workspace/#helm-chart","title":"Helm Chart","text":"The Workspace API is deployed via the rm-workspace-api helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the um-workspace-api chart.
helm install --version 1.4.2 --values workspace-api-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n workspace-api rm-workspace-api\nAt minimum, values for the following attributes should be specified:
identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.Example workspace-api-values.yaml\u2026
fullnameOverride: workspace-api\ningress:\n enabled: true\n annotations:\n cert-manager.io/cluster-issuer: letsencrypt-production\n kubernetes.io/ingress.class: nginx\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n hosts:\n - host: workspace-api-open.192-168-49-2.nip.io\n paths: [\"/\"]\n tls:\n - hosts:\n - workspace-api-open.192-168-49-2.nip.io\n secretName: workspace-api-open-tls\nfluxHelmOperator:\n enabled: true\nprefixForName: \"ws\"\nworkspaceSecretName: \"bucket\"\nnamespaceForBucketResource: \"rm\"\ns3Endpoint: \"https://minio.192-168-49-2.nip.io\"\ns3Region: \"RegionOne\"\nharborUrl: \"https://harbor.192-168-49-2.nip.io\"\nharborUsername: \"admin\"\nharborPasswordSecretName: \"harbor\"\nworkspaceChartsConfigMap: \"workspace-charts\"\nbucketEndpointUrl: \"http://minio-bucket-api:8080/bucket\"\nkeycloakIntegration:\n enabled: true\n keycloakUrl: \"https://keycloak.192-168-49-2.nip.io\"\n realm: \"master\"\n identityApiUrl: \"https://identity-api.192-168-49-2.nip.io\"\n workspaceApiIamClientId: \"workspace-api\"\n defaultIamClientSecret: \"changeme\"\nNote
harborXXX values above.See section Container Registry.admin user must be created as described in the section Harbor admin Password.keycloakIntegration allows the Workspace API to apply protecion (for the specified workspace owner) to the services within newly created workspaces.bucketEndpointUrl. See section Bucket Creation Webhook for details.admin Password","text":"The password for the harbor admin user is provided to the workspace-api via the specified secret - defined as harbor above.
This secret must be created - for example as follows\u2026
kubectl -n rm create secret generic harbor \\\n --from-literal=HARBOR_ADMIN_PASSWORD=\"changeme\"\nWorkspaces are created by instantiating the rm-user-workspace helm chart for each user/group. The Workspace API uses Flux CD as a helper to manage these subordinate helm charts - via flux resources of type HelmRelease. Thus, it is necessary to deploy within the cluster the aspects of flux that support this helm chart management - namely the flux helm-controller, source-controller and the Kubernetes Custom Resource Definitions (CRD) for HelmRelease and HelmRepository.
In case you are not already using flux within your clsuter, then the Workspace API helm chart can be configured to deploy the required flux components\u2026
fluxHelmOperator:\n enabled: true # true = install flux for me, false = I already have flux\nThe Workspace API instantiates for each user a set of services, including a Resource Catalogue and Data Access services. These user services are instantiated via helm using templates. The templates are provided to the Workspace API in a ConfigMap that is, by default, named workspace-charts. Each file in the config-map is expected to be of kind HelmRelease. During creation of a new workspace, the Worksapce API applies each file to the cluster in the namespace of the newly created namespace.
The default ConfigMap that is included with this guide contains the following templates for instantiation of user-specific components:
template-hr-data-access.yamltemplate-hr-resource-catalogue.yamltemplate-hr-resource-protection.yamlEach of these templates is expressed as a flux HelmRelease object that describes the helm chart and values required to deploy the service.
In addition, ConfigMap templates are included that provide specific details required to access the user-scoped workspace resources, including access to S3 object storage and container registry:
template-cm-aws-config.yamltemplate-cm-aws-credentials.yamltemplate-cm-docker-config.yamlThese ConfigMaps are designed to be mounted as files into the runtime environments of other components for user workspace integration. In particular the Application Hub makes use of this approach to provide a user experience that integrates with the user\u2019s workspace resources.
"},{"location":"eoepca/workspace/#templates-configmap","title":"Templates ConfigMap","text":"The templates are provided to the Workspace API as a ConfigMap in the namespace of the Workspace API deployment\u2026
(for full examples see https://github.com/EOEPCA/deployment-guide/tree/eoepca-v1.4/deploy/eoepca/workspace-templates)
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: workspace-charts\ndata:\n template-hr-resource-catalogue.yaml: |\n apiVersion: helm.toolkit.fluxcd.io/v2beta1\n kind: HelmRelease\n metadata:\n name: rm-resource-catalogue\n spec:\n interval: 5m\n chart:\n spec:\n chart: rm-resource-catalogue\n version: 1.3.1\n sourceRef:\n kind: HelmRepository\n name: eoepca\n namespace: rm\n values:\n ...\n template-hr-data-access.yaml: |\n apiVersion: helm.toolkit.fluxcd.io/v2beta1\n kind: HelmRelease\n metadata:\n name: vs\n spec:\n interval: 5m\n chart:\n spec:\n chart: data-access\n version: 1.3.1\n sourceRef:\n kind: HelmRepository\n name: eoepca\n namespace: rm\n values:\n ...\n template-hr-resource-protection.yaml: |\n apiVersion: helm.toolkit.fluxcd.io/v2beta1\n kind: HelmRelease\n metadata:\n name: resource-protection\n spec:\n interval: 5m\n chart:\n spec:\n chart: identity-gatekeeper\n version: 1.0.11\n sourceRef:\n kind: HelmRepository\n name: eoepca\n namespace: ${NAMESPACE}\n values:\n ...\n template-cm-aws-config.yaml: |\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: aws-config\n data:\n aws-config: |\n [default]\n region = {{ s3_region }}\n s3 =\n endpoint_url = {{ s3_endpoint_url }}\n s3api =\n endpoint_url = {{ s3_endpoint_url }}\n [plugins]\n endpoint = awscli_plugin_endpoint\n template-cm-aws-credentials.yaml: |\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: aws-credentials\n data:\n aws-credentials: |\n [default]\n aws_access_key_id = {{ access_key_id }}\n aws_secret_access_key = {{ secret_access_key }}\n template-cm-docker-config.yaml: |\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: docker-config\n data:\n docker-config: |\n {\n \"auths\": {\n \"{{ container_registry_host }}\": {\n \"auth\": \"{{ container_registry_credentials }}\"\n }\n }\nNotice the use of workspace template parameters {{ param_name }} that are used at workspace creation time to contextualise the workspace for the owning user. See section Workspace Template Parameters for more information.
As can be seen above, the HelmRelease templates rely upon objects of type HelmRepository that define the hosting helm chart repository. Thus, in support of the workspace templates, appropriate HelmRepository objects must be provisioned within the cluster. For example, in support of the above examples that rely upon the EOEPCA Helm Chart Repository\u2026
apiVersion: source.toolkit.fluxcd.io/v1beta1\nkind: HelmRepository\nmetadata:\n name: eoepca\n namespace: rm\nspec:\n interval: 2m\n url: https://eoepca.github.io/helm-charts/\nThe Workspace API uses the jinja2 templating engine when applying the resources for a user workspace. The current parameters are currently supported:
workspace_name The name of the workspace - {{ workspace_name }} used to ensure unique naming of cluster resources, such as service ingressdefault_owner The uuid of the owner of the workspace - {{ default_owner }} used to initialise the workspace protection{{ s3_endpoint_url }}{{ s3_region }}{{ access_key_id }}{{ secret_access_key }}{{ container_registry_host }}{{ container_registry_credentials }}As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the workspace-api service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install workspace-api-protection identity-gatekeeper -f workspace-api-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"rm\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the workspace-api - in particular the specific ingress requirements for the workspace-api backend service\u2026
Example workspace-api-protection-values.yaml\u2026
fullnameOverride: workspace-api-protection\nconfig:\n client-id: workspace-api\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: workspace-api.192-168-49-2.nip.io\n name: workspace-api\n port:\n number: 8080\nsecrets:\n # Values for secret 'workspace-api-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n serverSnippets:\n custom: |-\n # Open access to some endpoints, including Swagger UI\n location ~ ^/(docs|openapi.json|probe) {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: workspace-api above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example, with path protection for the admin user\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=\"workspace-api\" \\\n --name=\"Workspace API Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Workspace API Gatekeeper\" \\\n --resource=\"admin\" --uris='/*' --scopes=view --users=\"admin\"\nThe Workspace API provides a REST interface that is accessed at the endpoint https://workspace-api.192-168-49-2.nip.io/. See the Swagger Docs - /docs.
Note
If the Workspace API has been protected (via Gatekeeper with Keycloak), then requests must be supported by an access_token carried in the HTTP header Authorozation: Bearer <token>. This diminishes the utility of the swagger UI.
Additional information regarding the Workspace API can be found at:
With helm chart version 1.3.1 of the workspace-api the approach to bucket creation has been re-architected to use a webhook approach.
During workspace creation the workspace-api needs to create an object storage bucket for the user. The method by which the bucket is created is a function of the hosting infrastructure object storage layer - i.e. there is no \u2018common\u2019 approach for the workspace-api to perform the bucket creation.
In order to allow this bucket creation step to be customised by the platform integrator, the workspace-api is configured with a webhook endpoint that is invoked to effect the bucket creation on behalf of the workspace-api.
The workspace-api is configured by the following value in its helm chart deployment, e.g\u2026
bucketEndpointUrl: \"http://my-bucket-webhook:8080/bucket\"\nThe webhook service must implement the following REST interface\u2026
method: POST content-type: application/json data:
{\n bucketName: str\n secretName: str\n secretNamespace: str\n}\nThere are two possible approaches to implement this request, distinguished by the response code\u2026
200 The bucket is created and the credentials are included in the response body. In this case only the supplied bucketName is relevant to fulfil the request.201 The bucket will be created (asychronously) and the outcome is provided by the webhook via a Kubernetes secret, as per the secretName and secretNamespace request parameters200 Response
In case 200 response, the response body should communicate the credentials with an application/json content-type in the form\u2026
{\n \"bucketname\": \"...\",\n \"access_key\": \"...\",\n \"access_secret\": \"....\",\n \"projectid\": \"...\",\n}\nIn this case the workspace-api will create the appropriate bucket secret using the returned credentials.
201 Response
In case 201 response, the secret should be created in the form\u2026
data:\n bucketname: \"...\"\n access: \"...\"\n secret: \"...\"\n projectid: \"...\"\nIn this case the workspace-api will wait for the (asynchronous) creation of the specified secret before continuing with the workspace creation.
Overall Outcome
In both cases the ultimate outcome is the creation of the bucket in the back-end object storage, and the creation of a Kubernetes secret that maintains the credentials for access to the bucket. The existence of the bucket secret is prerequisite to the continuation of the user workspace creation.
"},{"location":"eoepca/workspace/#minio-bucket-api-webhook","title":"Minio Bucket API (Webhook)","text":"The Minio Bucket API provides an implementation of a Bucket Creation Webhook for a Minio S3 Object Storage backend. This is used as the default in this guide - but should be replaced for a production deployment with an appropriate webhook to integrate with the object storage solution of the deployment environment.
"},{"location":"eoepca/workspace/#helm-chart_1","title":"Helm Chart","text":"The Minio Bucket API is deployed via the rm-minio-bucket-api helm chart from the EOEPCA Helm Chart Repository - ref. Helm Chart for the Minio Bucket API.
helm install --version 0.0.4 --values minio-bucket-api-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n rm-minio-bucket-api rm-minio-bucket-api\nAt minimum, values for the following attributes should be specified:
minIOServerEndpointaccessCredentials.secretName (ref. Minio Credentials Secret)Example minio-bucket-api-values.yaml\u2026
fullnameOverride: minio-bucket-api\nminIOServerEndpoint: https://minio.192-168-49-2.nip.io\naccessCredentials:\n secretName: minio-auth\nAdditional information regarding the Minio Bucket API can be found at:
A deployment wrapper script has been prepared for an \u2018Application Hub\u2019 deployment - that provides the Application Hub integrated with the Identity Service (Keycloak) via OIDC for user authentication.
The script deploy/apphub/apphub achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/apphub/apphub-options.
The Application Hub deployment applies the following configuration:
Deployment is initiated by invoking the script\u2026
./deploy/apphub/apphub\nThe Identity Service (Keycloak) is accessed at the following endpoints\u2026
The Application Hub is accessed at the endpoint - http://applicationhub.192-168-49-2.nip.io/.
"},{"location":"quickstart/application-hub-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/application-hub-deployment/#application-hub-notes","title":"Application Hub Notes","text":""},{"location":"quickstart/application-hub-deployment/#login","title":"Login","text":"Authentication is made via the Sign in with EOEPCA button on the service home page - which redirects to Keycloak for authentication.
With the out-of-the-box configuration user eric or bob should be used with default password changeme. Users eric and bob are currently predefined within the helm chart as admin users - see https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/files/hub/jupyter_config.py#L171.
Once logged in, the service list is presented for spawning of applications. Note that this list of applications is currently defined within the helm chart - see https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/files/hub/config.yml.
From the list, a service is selected and the Start button initiates spawning.
For a clean deployment, the first spawn of each application may take some time whilst the container image representing the application is downloaded to the node. Subsequent invocations (at least on the same node) should be much faster. Once running, the application continues (in the background) until stopped by the user using the Stop Server button on the user\u2019s home screen.
The current JupyterHub configuration assumes a single application service (per user) running at a time - i.e. the current application must be stopped before the next can be started. There is an alternative configuration in which applications can be run in parallel and their lifecycles individually managed.
"},{"location":"quickstart/application-hub-deployment/#returning-to-the-home-screen","title":"Returning to the Home Screen","text":"The launched applications do not (yet) have a navigation link \u2018out\u2019 of the application back to the home screen.
Therefore, it is necessary to manually modify the url in the browser address bar to /hub/home to navigate to the home screen - from where the current running server can be stopped or re-entered.
Following instantiation, the IAT application (Interactive Analysis Tool) defaults to the \u2018Jupyter Notebook\u2019 view (/user/<user>/tree) - rather than the Jupyter Lab view (/user/<user>/lab).
To switch to the Jupyter Lab view it is necessary to manually edit the url path from /user/<user>/tree to /user/<user>/lab. It is intended to update the default to this Jupyter Lab path.
Based upon our development experiences on CREODIAS, there is a wrapper script creodias with particular customisations suited to the CREODIAS infrastructure and data offering. The customisations are expressed through environment variables that are captured in the file creodias-options.
These scripts are examples that can be seen as a starting point, from which they can be adapted to your needs.
The CREODIAS deployment applies the following configuration:
With reference to the file creodias-options, particular attention is drawn to the following environment variables that require tailoring to your CREODIAS (Cloudferro) environment\u2026
MINIO_ROOT_PASSWORD, HARBOR_ADMIN_PASSWORDIDENTITY_SERVICE_DEFAULT_SECRET, IDENTITY_SERVICE_ADMIN_PASSWORD, etc.public_ip - The public IP address through which the deployment is exposed via the ingress-controllerdomain - The DNS domain name through which the deployment is accessed - forming the stem for all service hostnames in the ingress rulesOnce the file creodias-options has been well populated for your environment, then the deployment is initiated with\u2026
./deploy/creodias/creodias\nThe harvester can be deployed with a default configuration file at /config.yaml. As described in the Data Access section, harvesting according to this configuration can be triggered with\u2026
kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\nSee the Harvester section below for an explanation of this harvester configuration.
"},{"location":"quickstart/creodias-deployment/#data-specification-walkthrough","title":"Data Specification Walkthrough","text":"The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables.
This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example.
"},{"location":"quickstart/creodias-deployment/#harvester","title":"Harvester","text":"The harvester configuration specifies datasets with spatial/temporal extents, which is configured into the file /config.yaml of the data-access-harvester deployment.
The harvester is configured as follows\u2026
harvester:\n replicaCount: 1\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\n config:\n redis:\n host: data-access-redis-master\n port: 6379\n harvesters:\n - name: Creodias-Opensearch\n resource:\n url: https://datahub.creodias.eu/resto/api/collections/Sentinel2/describe.xml\n type: OpenSearch\n format_config:\n type: 'application/json'\n property_mapping:\n start_datetime: 'startDate'\n end_datetime: 'completionDate'\n productIdentifier: 'productIdentifier'\n query:\n time:\n property: sensed\n begin: 2019-09-10T00:00:00Z\n end: 2019-09-11T00:00:00Z\n collection: null\n bbox: 14.9,47.7,16.4,48.7\n filter: {}\n postprocess:\n - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor\n queue: register\n - name: Creodias-Opensearch-Sentinel1\n resource:\n url: https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml\n type: OpenSearch\n format_config:\n type: 'application/json'\n property_mapping:\n start_datetime: 'startDate'\n end_datetime: 'completionDate'\n productIdentifier: 'productIdentifier'\n query:\n time:\n property: sensed\n begin: 2019-09-10T00:00:00Z\n end: 2019-09-11T00:00:00Z\n collection: null\n bbox: 14.9,47.7,16.4,48.7\n extra_params:\n productType: GRD-COG\n filter: {}\n postprocess:\n - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel1Postprocessor\n queue: register\nBased upon this harvester configuration we expect that the following query is made to discover data - i.e. an OpenSearch query, with json response representation, for a defined spatial and temporal extent\u2026
https://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2019-09-10T00:00:00Z&completionDate=2019-09-11T00:00:00Z&box=14.9,47.7,16.4,48.7\nFrom the result returned, the path to each product (feature) is obtained from the productIdentifier property, e.g.
{\n \"type\": \"FeatureCollection\",\n \"features\": [\n {\n \"type\": \"Feature\",\n \"properties\": {\n \"productIdentifier\": \"/eodata/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE\"\n ...\n }\n ...\n }\n ...\n ]\n}\nThe harvester is configured with a Sentinel-2/CREODIAS specific post-processor harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor which transforms the product path from /eodata/... to s3://EODATA/....
The harvester post-processor follows this path to the Sentinel-2 scene and uses stactools (with built-in support for Sentinel-2) to establish a STAC item representing the product. This includes enumeration of assets for inspire-metadata and product-metadata - which are used by the registrar pycsw backend to embelesh the product record metadata.
Note
The above description considers Sentinel-2 data. Similar considerations apply for Sentinel-1 that is also detailed in the above harvester configuration.
The harvester outputs the STAC item for each product, which is pushed to the registrar via the register redis queue.
The registrar is configured at deployment to have the access details for the CREODIAS data in S3\u2026
global:\n storage:\n data:\n data:\n type: S3\n endpoint_url: http://data.cloudferro.com\n access_key_id: access\n secret_access_key: access\n region_name: RegionOne\n validate_bucket_name: false\nUsing this S3 configuration, the registrar pycsw backend uses the product metadata linked in the STAC item (ref. assets inspire-metadata and product-metadata) to embelesh the metadata. For example, product-metadata in the file\u2026
s3://EODATA/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE/MTD_MSIL1C.xml\nThe registrar uses this information to create the ISO XML metadata that is loaded into the resource-catalogue.
"},{"location":"quickstart/creodias-deployment/#product-type","title":"Product Type","text":"The registrar recognises the product as Sentinel-2 and so reads its metadata XML files to obtain additional information. From the metadata XML file (e.g. MTD_MSIL1C.xml) the registrar obtains the Product Type for each product from the field <PRODUCT_TYPE>\u2026
<n1:Level-1C_User_Product>\n <n1:General_Info>\n <Product_Info>\n <PRODUCT_TYPE>S2MSI1C</PRODUCT_TYPE>\n ...\n </Product_Info>\n ...\n </n1:General_Info>\n ...\n<n1:Level-1C_User_Product>\nThe registrar (eoepca/rm-data-access-core) container image is pre-loaded with two collections at the path /registrar_pycsw/registrar_pycsw/resources, (in the built container the files are at the path /usr/local/lib/python3.8/dist-packages/registrar_pycsw/resources/):
S2MSI1CS2MSI2AThe registrar applies these collections into the resource-catalogue during start-up - to create pre-defined out-of-the-box collections in pycsw.
During registration, the PycswBackend of the registrar uses the Product Type to map the product into the collection of the same name - using metadata field parentidentifier.
The data-access service data handling is configured by definition of productTypes, collections and layers\u2026
productTypes identify the underlying file assets as WCS coverages and their visual representationcollections provide groupings into which products are organisedlayers specifies the hoe the product visual representations are exposed through the WMS serviceproductType","text":"During registration, products are mapped into a productType via a filter that is applied against the STAC item metadata.
The registrar uses the product_type of each product to determine the collection into which the product should be registered - noting that the name of the product type does not take part in the matching logic (and hence can be any text name)\u2026
productTypes:\n - name: S2MSI1C\n filter:\n s2:product_type: S2MSI1C\nIn the above example, the field s2:product_type is populated by the stactools that prepares the STAC item from the product metadata.
productType - coverages","text":"coverages defines the coverages for the WCS service. Each coverage links to the assets that are defined within the product STAC item.
productType - browses","text":"browses defines the images that are visualised in the View Server Client. Expressions are used to map the product assets into their visual representation.
collections","text":"Collections are defined by reference to the defined productTypes and coverages.
layers","text":"layers defines the layers that are presented through the WMS service - each layer being linked to the underlying browse that provides the image source. Layers are defined via their id that relies upon the naming convection <collection>__<browse> to identify the browse and so define the layer.
Example configuration for Sentinel-2 L1C and L2A data.
global:\n layers:\n - id: S2L1C\n title: Sentinel-2 Level 1C True Color\n abstract: Sentinel-2 Level 2A True Color\n displayColor: '#eb3700'\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L1C__TRUE_COLOR\n title: Sentinel-2 Level 1C True Color\n abstract: Sentinel-2 Level 2A True Color\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L1C__masked_clouds\n title: Sentinel-2 Level 1C True Color with cloud masks\n abstract: Sentinel-2 Level 1C True Color with cloud masks\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L1C__FALSE_COLOR\n title: Sentinel-2 Level 1C False Color\n abstract: Sentinel-2 Level 1C False Color\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L1C__NDVI\n title: Sentinel-2 Level 21CNDVI\n abstract: Sentinel-2 Level 1C NDVI\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L2A\n title: Sentinel-2 Level 2A True Color\n abstract: Sentinel-2 Level 2A True Color\n displayColor: '#eb3700'\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n - id: S2L2A__TRUE_COLOR\n title: Sentinel-2 Level 2A True Color\n abstract: Sentinel-2 Level 2A True Color\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n - id: S2L2A__masked_clouds\n title: Sentinel-2 Level 2A True Color with cloud masks\n abstract: Sentinel-2 Level 2A True Color with cloud masks\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n - id: S2L2A__FALSE_COLOR\n title: Sentinel-2 Level 2A False Color\n abstract: Sentinel-2 Level 2A False Color\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n - id: S2L2A__NDVI\n title: Sentinel-2 Level 2A NDVI\n abstract: Sentinel-2 Level 2A NDVI\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n collections:\n S2L1C:\n product_types:\n - S2MSI1C\n coverage_types:\n - S2L1C_B01\n - S2L1C_B02\n - S2L1C_B03\n - S2L1C_B04\n - S2L1C_B05\n - S2L1C_B06\n - S2L1C_B07\n - S2L1C_B08\n - S2L1C_B8A\n - S2L1C_B09\n - S2L1C_B10\n - S2L1C_B11\n - S2L1C_B12\n S2L2A:\n product_types:\n - S2MSI2A\n product_levels:\n - Level-2A\n coverage_types:\n - S2L2A_B01\n - S2L2A_B02\n - S2L2A_B03\n - S2L2A_B04\n - S2L2A_B05\n - S2L2A_B06\n - S2L2A_B07\n - S2L2A_B08\n - S2L2A_B8A\n - S2L2A_B09\n - S2L2A_B11\n - S2L2A_B12\n productTypes:\n - name: S2MSI1C\n filter:\n s2:product_type: S2MSI1C\n metadata_assets: []\n coverages:\n S2L1C_B01:\n assets:\n - B01\n S2L1C_B02:\n assets:\n - B02\n S2L1C_B03:\n assets:\n - B03\n S2L1C_B04:\n assets:\n - B04\n S2L1C_B05:\n assets:\n - B05\n S2L1C_B06:\n assets:\n - B06\n S2L1C_B07:\n assets:\n - B07\n S2L1C_B08:\n assets:\n - B08\n S2L1C_B8A:\n assets:\n - B8A\n S2L1C_B09:\n assets:\n - B09\n S2L1C_B10:\n assets:\n - B10\n S2L1C_B11:\n assets:\n - B11\n S2L1C_B12:\n assets:\n - B12\n defaultBrowse: TRUE_COLOR\n browses:\n TRUE_COLOR:\n asset: visual\n red:\n expression: B04\n range: [0, 4000]\n nodata: 0\n green:\n expression: B03\n range: [0, 4000]\n nodata: 0\n blue:\n expression: B02\n range: [0, 4000]\n nodata: 0\n FALSE_COLOR:\n red:\n expression: B08\n range: [0, 4000]\n nodata: 0\n green:\n expression: B04\n range: [0, 4000]\n nodata: 0\n blue:\n expression: B03\n range: [0, 4000]\n nodata: 0\n NDVI:\n grey:\n expression: (B08-B04)/(B08+B04)\n range: [-1, 1]\n masks:\n clouds:\n validity: false\n - name: S2MSI2A\n filter:\n s2:product_type: S2MSI2A\n metadata_assets: []\n coverages:\n S2L2A_B01:\n assets:\n - B01\n S2L2A_B02:\n assets:\n - B02\n S2L2A_B03:\n assets:\n - B03\n S2L2A_B04:\n assets:\n - B04\n S2L2A_B05:\n assets:\n - B05\n S2L2A_B06:\n assets:\n - B06\n S2L2A_B07:\n assets:\n - B07\n S2L2A_B08:\n assets:\n - B08\n S2L2A_B8A:\n assets:\n - B8A\n S2L2A_B09:\n assets:\n - B09\n S2L2A_B11:\n assets:\n - B11\n S2L2A_B12:\n assets:\n - B12\n default_browse_locator: TCI_10m\n browses:\n TRUE_COLOR:\n asset: visual-10m\n red:\n expression: B04\n range: [0, 4000]\n nodata: 0\n green:\n expression: B03\n range: [0, 4000]\n nodata: 0\n blue:\n expression: B02\n range: [0, 4000]\n nodata: 0\n FALSE_COLOR:\n red:\n expression: B08\n range: [0, 4000]\n nodata: 0\n green:\n expression: B04\n range: [0, 4000]\n nodata: 0\n blue:\n expression: B03\n range: [0, 4000]\n nodata: 0\n NDVI:\n grey:\n expression: (B08-B04)/(B08+B04)\n range: [-1, 1]\n masks:\n clouds:\n validity: false\nA deployment wrapper script has been prepared for a \u2018data access\u2019 deployment - that is focused on the Resource Catalogue and Data Access services.
The script deploy/data-access/data-access achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/data-access/data-access-options.
The data-access deployment applies the following configuration:
eodata network - see description of variable CREODIAS_DATA_SPECIFICATIONDeployment is initiated by invoking the script\u2026
./deploy/data-access/data-access\nThe Resource Catalogue is accessed at the endpoint resource-catalogue-open.192-168-49-2.nip.io - e.g. resource-catalogue-open.192-168-49-2.nip.io.
The Data Access View Server is accessed at the endpoint data-access-open.192-168-49-2.nip.io - e.g. data-access-open.192-168-49-2.nip.io.
To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/data-access-deployment/#data-harvesting","title":"Data Harvesting","text":"See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.
"},{"location":"quickstart/exploitation-deployment/","title":"Exploitation Deployment","text":""},{"location":"quickstart/exploitation-deployment/#overview","title":"Overview","text":"A deployment wrapper script has been prepared for an \u2018exploitation\u2019 deployment - that provides deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services.
The script deploy/exploitation/exploitation achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/exploitation/exploitation-options.
The exploitation deployment applies the following configuration:
eodata network - see description of variable CREODIAS_DATA_SPECIFICATIONDeployment is initiated by invoking the script\u2026
./deploy/exploitation/exploitation\nThe ADES service is accessed at the endpoint ades-open.192-168-49-2.nip.io.
The Resource Catalogue is accessed at the endpoint resource-catalogue-open.192-168-49-2.nip.io.
The Data Access View Server is accessed at the endpoint data-access-open.192-168-49-2.nip.io.
To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/exploitation-deployment/#example-requests-s-expression-on-creodias","title":"Example Requests -s-expression on CREODIAS","text":"NOTE that this example processing request requires harvesting data from CREODIAS, which can only be performed if the deployment is made to a CREODIAS VM with access to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION.
Section Processing provides an example of a simple self-contained processing deployment and execution, and access to the processing results.
In addition to the snuggs example, the file deploy/samples/requests/processing/s-expression.http has been prepared to exploit data that has been registered within the Resource Catalogue and Data Access services.
First the input data for processing must be harvested into the resource management services. Sentinel-2 data on 2nd Sept 2020\u2026
./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2020.09.02.yaml\nThen the s-expression.http file provides sample requests for OGC API Processes operations:
NOTE that the first requests in the file provide optional calls to obtain a user access token (openidConfiguration / authenticate) - to be used in the case that protected (not \u2018open\u2019) endpoints are deployed.
The files snuggs.http and s-expression.http describe the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client. Install in vscode with ext install humao.rest-client.
Various variables, such as to specify the @domain for your deployment, can be configured at the top of the file.
At the completion of successful processing execution, the procesing results are obtained as described in section Processing Results.
"},{"location":"quickstart/exploitation-deployment/#data-harvesting","title":"Data Harvesting","text":"See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.
"},{"location":"quickstart/processing-deployment/","title":"Processing Deployment","text":""},{"location":"quickstart/processing-deployment/#overview","title":"Overview","text":"A deployment wrapper script has been prepared for a \u2018processing\u2019 deployment - that is focused on the ADES and the deployment/execution of processing jobs.
The script deploy/processing/processing achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/processing/processing-options.
The processing deployment applies the following configuration:
Deployment is initiated by invoking the script\u2026
./deploy/processing/processing\nThe ADES service is accessed at the endpoint zoo-open.192-168-49-2.nip.io.
To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/processing-deployment/#example-requests","title":"Example Requests","text":"Some sample requests have been prepared in the subdirectory deploy/samples/requests/processing - for example\u2026
convert Provides a \u2018hello world\u2019 processing example that can be used simply to check that the processing capability has been well deployedsnuggs Provides a packaged EO exploitation algorithm that perform \u2018real\u2019 work and, as such, is more resource demanding (10GB RAM, 3 CPU) - and so may not be suitable for execution within a local minikube deployment (depending on resource allocations)These sample http files have been prepared with sample requests for OGC API Processes operations:
Note
openidConfiguration / authenticate). These are to be used in the case that protected (not \u2018open\u2019) endpoints are deployed.ext install humao.rest-client.@hostname and @domain can be configured at the top of the file.curl Commands","text":"Alternatively the following curl commands can be used\u2026
curl -k \\\n --request GET \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n --header 'accept: application/json'\nconvert) Deploy Process (convert) - By Reference (JSON) curl -k \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n --header 'accept: application/json' \\\n --header 'content-type: application/json' \\\n --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/convert/main/convert-url-app.cwl\",\"type\": \"application/cwl\"}}'\nconvert) - Inline (CWL) curl -k \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n --header 'accept: application/json' \\\n --header 'content-type: application/cwl+yaml' \\\n --data '< convert-url-app.cwl'\nconvert) curl -k \\\n --request GET \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \\\n --header 'accept: application/json'\nconvert) curl -k -v \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url/execution \\\n --header 'accept: application/json' \\\n --header 'content-type: application/json' \\\n --header 'prefer: respond-async' \\\n --data '{\"inputs\": {\"fn\": \"resize\",\"url\": \"https://eoepca.org/media_portal/images/logo6_med.original.png\", \"size\": \"50%\"},\"response\":\"raw\"}'\nconvert) curl -k -v \\\n --request DELETE \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \\\n --header 'accept: application/json'\nsnuggs) Deploy Process (snuggs) curl -k \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n --header 'accept: application/json' \\\n --header 'content-type: application/json' \\\n --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/deployment-guide/eoepca-v1.4/deploy/samples/requests/processing/snuggs.cwl\",\"type\": \"application/cwl\"}}'\nsnuggs) curl -k \\\n --request GET \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \\\n --header 'accept: application/json'\nsnuggs) curl -k -v \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs/execution \\\n --header 'accept: application/json' \\\n --header 'content-type: application/json' \\\n --header 'prefer: respond-async' \\\n --data '{\"inputs\": {\"input_reference\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\",\"s_expression\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\"},\"response\":\"raw\"}'\nsnuggs) curl -k -v \\\n --request DELETE \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \\\n --header 'accept: application/json'\nThis request requires the Location header from the response to the execute request. This will be of the form http://zoo-open.192-168-49-2.nip.io/{user}/ogc-api/jobs/{job-id} - e.g. http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e.
curl -k \\\n --request GET \\\n --url {location-header} \\\n --header 'accept: application/json'\nThis request uses the same URL as Get Job Status, with the additional URL path /results - i.e. /{user}/ogc-api/jobs/{job-id}/results - e.g. /eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e/results
curl -k \\\n --request GET \\\n --url {location-header}/results \\\n --header 'accept: application/json'\nThe response indicates the location of the results, which should be in the minio object storage. See Processing Results.
The response also provides links to log files regarding each step of the Application Package workflow execution - which may be useful for debugging.
List Jobscurl -k \\\n --request GET \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs \\\n --header 'accept: application/json'\nThe outputs are published as a static STAC catalogue to a path that includes the unique job ID.
In the default configuration, the processing results are pushed to the Minio S3 object storage. This can be checked via browser access at the endpoint http://console.minio.192-168-49-2.nip.io/, or using an S3 client such as\u2026
s3cmd -c ./deploy/cluster/s3cfg ls s3://eoepca\nFor the default credentials to connect to Minio see Minio Object Storage Default Credentials.
Note
If the ADES deployment has been configured to stage-out to the user\u2019s Workspace, then the above s3cmd and credentials would have to be adjusted accordingly - for example the bucket s3://ws-eric.
Note
The deployment of the EOEPCA components and the supporting Kubernetes cluster is described in the sections Prepare Cluster and Deploy EOEPCA Components. These sections should be consulted for more detailed information.
"},{"location":"quickstart/quickstart/#scripted-deployment","title":"Scripted Deployment","text":"As a companion to these descriptions, we have developed a set of scripts to provide a demonstration of example deployments - see section Scripted Deployment for a detailed description of the scripts and how they are configured and used.
Note
The scripted deployment assumes that installation of the Prerequisite Tooling has been performed
"},{"location":"quickstart/quickstart/#customised-deployments","title":"Customised Deployments","text":"The Scripted Deployment can be quickly exploited through the following customisations (profiles) for particular use cases:
Each customisation is introduced in their respective sections.
"},{"location":"quickstart/quickstart/#quick-example","title":"Quick Example","text":"Follow these steps to create a simple local deployment in minikube\u2026
git clone -b eoepca-v1.4 https://github.com/EOEPCA/deployment-guidecd deployment-guide\n./deploy/simple/simple\nwatch kubectl get pod -ARunning or Completed This may take 10-20 mins depending on the capabilities of your platform.The Scripted Deployment provides a demonstration of an example deployment, and can found in the subdirectory deployment-guide/deploy of the source repository for this guide\u2026
git clone -b eoepca-v1.4 https://github.com/EOEPCA/deployment-guide \\\n&& cd deployment-guide \\\n&& ls deploy\nThe script deploy/eoepca/eoepca.sh acts as an entry-point to the full system deployment. In order to tailor the deployment for your target environment, the script is configured through environment variables and command-line arguments. By default the script assumes deployment to a local minikube.
Note
The scripted deployment assumes that installation of the Prerequisite Tooling has been performed.
The following subsections lead through the steps for a full local deployment. Whilst minikube is assumed, minimal adaptions are required to make the deployment to your existing Kubernetes cluster.
The deployment follows these broad steps:
The script deploy/eoepca/eoepca.sh is configured by some environment variables and command-line arguments.
REQUIRE_MINIKUBE=trueREQUIRE_INGRESS_NGINX=trueREQUIRE_CERT_MANAGER=trueREQUIRE_LETSENCRYPT=trueREQUIRE_SEALED_SECRETS=falseREQUIRE_MINIO=false see description REQUIRE_<eoepca-component> A set of variables that can be used to control which EOEPCA components are deployed by the script, as follows (with defaults):REQUIRE_STORAGE=trueREQUIRE_DUMMY_SERVICE=falseREQUIRE_IDENTITY_SERVICE=trueREQUIRE_ADES=trueREQUIRE_RESOURCE_CATALOGUE=trueREQUIRE_DATA_ACCESS=trueREQUIRE_REGISTRATION_API=trueREQUIRE_WORKSPACE_API=trueREQUIRE_HARBOR=trueREQUIRE_PORTAL=trueREQUIRE_APPLICATION_HUB=true see description REQUIRE_<protection-component> A set of variables that can be used to control which PROTECTION components are deployed by the script, as follows (with defaults):REQUIRE_DUMMY_SERVICE_PROTECTION=falseREQUIRE_ADES_PROTECTION=trueREQUIRE_RESOURCE_CATALOGUE_PROTECTION=trueREQUIRE_DATA_ACCESS_PROTECTION=trueREGISTRATION_API_PROTECTION=trueREQUIRE_WORKSPACE_API_PROTECTION=true see description MINIKUBE_VERSION The Minikube version to be (optionally) installedNote that the EOEPCA development has been conducted using the default stated here. v1.32.0 MINIKUBE_KUBERNETES_VERSION The Kubernetes version to be used by minikubeNote that the EOEPCA development has been conducted primarily using version 1.22.5. v1.22.5 MINIKUBE_MEMORY_AMOUNT Amount of memory to allocate to the docker containers used by minikube to implement the cluster. 12g MINIKUBE_DISK_AMOUNT Amount of disk space to allocate to the docker containers used by minikube to implement the cluster. 20g MINIKUBE_EXTRA_OPTIONS Additional options to pass to minikube start command-line --ports=80:80,443:443 USE_METALLB Enable use of minikube\u2019s built-in load-balancer.The load-balancer can be used to facilitate exposing services publicly. However, the same can be achieved using minikube\u2019s built-in ingress-controller. Therefore, this option is suppressed by default. false USE_INGRESS_NGINX_HELM Install the ingress-nginx controller using the published helm chart, rather than relying upon the version that is built-in to minikube. By default we prefer the version that is built in to minikube. false USE_INGRESS_NGINX_LOADBALANCER Patch the built-in minikube nginx-ingress-controller to offer a service of type LoadBalancer, rather than the default NodePort. It was initially thought that this would be necessary to achieve public access to the ingress services - but was subsequently found that the default NodePort configuration of the ingress-controller was sufficient. This option is left in case it proves useful.Only applicable for USE_INGRESS_NGINX_HELM=false (i.e. when using the minikube built-in ) false OPEN_INGRESS Create \u2018open\u2019 ingress endpoints that are not subject to authorization protection. For a secure system the open endpoints should be disabled (false) and access to resource should be protected via ingress that apply protection false USE_TLS Indicates whether TLS will be configured for service Ingress rules.If not (i.e. USE_TLS=false), then the ingress-controller is configured to disable ssl-redirect, and TLS_CLUSTER_ISSUER=notls is set. true TLS_CLUSTER_ISSUER The name of the ClusterIssuer to satisfy ingress tls certificates.Out-of-the-box ClusterIssuer instances are configured in the file deploy/cluster/letsencrypt.sh. letsencrypt-staging IDENTITY_SERVICE_DEFAULT_SECRET Default secret that is used by exception for other Identity Service credentials changeme IDENTITY_SERVICE_ADMIN_USER The admin user for Keycloak admin IDENTITY_SERVICE_ADMIN_PASSWORD The admin user password for Keycloak ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_SERVICE_ADMIN_CLIENT The Keycloak client to use for admin API tasks during scripted deployment admin-cli IDENTITY_POSTGRES_PASSWORD The password for the Keycloak Postgres service ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_GATEKEEPER_CLIENT_SECRET The secret used for each Keycloak client (one per resource service) created during scripted deployment ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_GATEKEEPER_ENCRYPTION_KEY The encryption key for each Keycloak client (one per resource service) created during scripted deploymentNOTE that this must be either 16 or 32 characters long changemechangeme IDENTITY_REALM Keycloak realm for Identity Service.This is not explicitly created by the scripted deployment, and so is assumed to exist within the Keycloak instance. Thus, will probably break the deployment if modified. master MINIO_ROOT_USER Name of the \u2018root\u2019 user for the Minio object storage service. eoepca MINIO_ROOT_PASSWORD Password for the \u2018root\u2019 user for the Minio object storage service. changeme HARBOR_ADMIN_PASSWORD Password for the \u2018admin\u2019 user for the Harbor artefact registry service. changeme DEFAULT_STORAGE Storage Class to be used by default for all components requiring dynamic storage provisioning.See variables <component>_STORAGE for per-component overrides. standard <component>_STORAGE A set of variables to control the dynamic provisioning Storage Class for individual components, as follows:MINIO_STORAGEADES_STORAGEAPPLICATION_HUB_STORAGEDATA_ACCESS_STORAGEHARBOR_STORAGERESOURCE_CATALOGUE_STORAGE <DEFAULT_STORAGE> PROCESSING_MAX_RAM Max RAM allocated to an individual processing job 8Gi PROCESSING_MAX_CORES Max number of CPU cores allocated to an individual processing job 4 PROCESSING_ZOO_IMAGE Container image for zoo-dru deployment eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762 STAGEOUT_TARGET Configures the ADES with the destination to which it should push processing results:workspace - via the Workspace APIminio - to minio S3 object storage workspace INSTALL_FLUX The Workspace API relies upon Flux CI/CD, and has the capability to install the required flux components to the cluster. If your deployment already has flux installed then set this value false to suppress the Workspace API flux install true CREODIAS_DATA_SPECIFICATION Apply the data specification to harvest from the CREODIAS data offering into the resource-catalogue and data-access services.Can only be used when running in the CREODIAS (Cloudferro) cloud, with access to the eodata network. false TEMP_FORWARDING_PORT Local port used during the scripted deployment for kubectl port-forward operations 9876"},{"location":"quickstart/scripted-deployment/#command-line-arguments","title":"Command-line Arguments","text":"The eoepca.sh script is further configured via command-line arguments\u2026
eoepca.sh <action> <cluster-name> <domain> <public-ip>\neoepca.sh Command-line Arguments Argument Description Default action Action to perform: apply | delete | template.apply makes the deploymentdelete removes the deploymenttemplate outputs generated kubernetes yaml to stdout apply cluster-name The name of the minikube \u2018profile\u2019 for the created minikube cluster eoepca domain The DNS domain name through which the deployment is accessed. Forms the stem for all service hostnames in the ingress rules - i.e. <service-name>.<domain>.By default, the value is deduced from the assigned cluster minikube IP address, using nip.io to establish a DNS lookup - i.e. <minikube ip>.nip.io. <minikube ip>.nip.io public-ip The public IP address through which the deployment is exposed via the ingress-controller.By default, the value is deduced from the assigned cluster minikube IP address - ref. command minikube ip. <minikube-ip>"},{"location":"quickstart/scripted-deployment/#public-deployment","title":"Public Deployment","text":"For simplicity, the out-of-the-box scripts assume a \u2018private\u2019 deployment - with no public IP / DNS and hence no use of TLS for service ingress endpoints.
In the case that an external-facing public deployment is desired, then the following configuration selections should be made:
domain - set to the domain (as per DNS records) for your deployment Note that the EOEPCA components typically configure their ingress with hostname prefixes applied to this domain. Thus, it is necessary that the DNS record for the domain is established as a wildcard record - i.e. *.<domain>public_ip - set to the public IP address through which the deployment is exposed via the ingress-controller i.e. the IP address that is assigned to the ingress controller service of type LoadBalancerUSE_TLS=true - to enable configuration of TLS endpoints in each component service ingressTLS_CLUSTER_ISSUER=<issuer> - should be configured ~ e.g. using the letsencrypt-production or letsencrypt-staging (testing only) Cluster Issuer that are configured by the scripted deploymentThe deployment is initiated by setting the appropriate environment variables and invoking the eoepca.sh script with suitable command-line arguments. You may find it convenient to do so using a wrapper script that customises the environment varaibles according to your cluster, and then invokes the eoepca.sh script.
Customised examples are provided for Simple, CREODIAS and Processing deployments.
NOTE that if a prior deployment has been attempted then, before redeploying, a clean-up should be performed as described in the Clean-up section below. This is particularly important in the case that the minikube none driver is used, as the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed.
Initiate the deployment\u2026
./deploy/eoepca/eoepca.sh apply \"<cluster-name>\" \"<public-ip>\" \"<domain>\"\nThe deployment takes 10+ minutes - depending on the resources of your host/cluster. The progress can be monitored\u2026
kubectl get pods -A\nThe deployment is ready once all pods are either Running or Completed.
The scripted deployment has been designed, as far as possible, to automate the configuration of the deployed components. However, there remain some steps that must be performed manually after the scripted deployment has completed. See the building block specific pages\u2026
By default, the Identity Service is accessed at the URL https://keycloak.192-168-49-2.nip.io/ with the credentials\u2026
username: `admin`\npassword: `changeme`\n\u2026unless the password is overridden via the variable IDENTITY_SERVICE_ADMIN_PASSWORD.
By default, Minio is accessed at the URL https://console.minio.192-168-49-2.nip.io/ with the credentials\u2026
username: `eoepca`\npassword: `changeme`\n\u2026unless the username/password are overridden via the variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD.
By default, Harbor is accessed at the URL https://harbor.192-168-49-2.nip.io/ with the credentials\u2026
username: `admin`\npassword: `changeme`\n\u2026unless the password is overridden via the variable HARBOR_ADMIN_PASSWORD.
The protection of resource server endpoints is applied during the deployment of each service requiring protection. This comprises creating a dedicated Keycloak client for each resource server, and the creation of associated resources and policies that protect the service-specific URLs.
This protection can be disabled via the environment variables REQUIRE_XXX_PROTECTION - e.g. REQUIRE_ADES_PROTECTION=false.
Note
By default, if OPEN_INGRESS is set true then PROTECTION will be disabled (false) unless overridden via the REQUIRE_XXX_PROTECTION variables.
The deployment creates (in the Keycloak Identity Service) the test users: eric, bob, alice.
Note
This does NOT create the workspace for each of these users - which must be performed via the Workspace API.
"},{"location":"quickstart/scripted-deployment/#user-workspace-creation","title":"User Workspace Creation","text":"The deployment created the test users eric, bob and alice. For completeness we use the Workspace API to create their user workspaces, which hold their personal resources (data, processing results, etc.) within the platform - see Workspace.
The Workspace API provides a Swagger UI that facilitates interaction with the API - at the URL https://workspace-api.192-168-49-2.nip.io/docs#.
Note
If the Workspace API has been protected (via Gatekeeper with Keycloak), then requests must be supported by an access_token carried in the HTTP header Authorozation: Bearer <token>. This diminishes the utility of the swagger UI.
Access the Workspace Swagger UI at https://workspace-api.192-168-49-2.nip.io/docs. Workspaces are created using POST /workspaces (Create Workspace). Expand the node and select Try it out. Complete the request body, such as\u2026
{\n \"preferred_name\": \"eric\",\n \"default_owner\": \"eric\"\n}\ndefault_owner is the ID for the user in Keycloak - thus protecting the created workspace for the identified user."},{"location":"quickstart/scripted-deployment/#using-curl","title":"Using curl","text":"The same can be achieved with a straight http request, for example using curl\u2026
curl -X 'POST' \\\n 'http://workspace-api.192-168-49-2.nip.io/workspaces' \\\n -H 'Content-Type: application/json' \\\n -H 'Accept: application/json' \\\n -H 'Authorization: Bearer <admin-access-token>' \\\n -d '{\n \"preferred_name\": \"<workspace-name>\",\n \"default_owner\": \"<user-id>\"\n}'\nValues must be provided for:
admin-access-token - Access Token for the admin userworkspace-name - name of the workspace, typically the usernameuser-id - the ID of the user for which the created workspace will be protected, typically the usernameThe Access Token for the admin user can be obtained with a call to the token endpoint of the Identity Service - supplying the credentials for the admin user and the pre-registered client\u2026
curl -L -X POST 'https://keycloak.192-168-49-2.nip.io/realms/master/protocol/openid-connect/token' \\\n -H 'Cache-Control: no-cache' \\\n -H 'Content-Type: application/x-www-form-urlencoded' \\\n --data-urlencode 'scope=openid profile email' \\\n --data-urlencode 'grant_type=password' \\\n --data-urlencode 'username=admin' \\\n --data-urlencode 'password=<admin-password>' \\\n --data-urlencode 'client_id=admin-cli'\nA json response is returned, in which the field access_token provides the Access Token for the admin user.
create-workspace helper script","text":"As an aide there is a helper script create-workspace. The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe create-workspace helper script requires some command-line arguments\u2026
$ ./deploy/bin/create-workspace -h\n\nCreate a new User Workspace.\ncreate-workspace -h | -w {workspace_api} -a {auth_server} -r {realm} -c {client} -u {admin-username} -p {admin-password} -O {owner} -W {workspace-name}\n\nwhere:\n -h show help message\n -w workspace-api service url (default: http://workspace-api.192-168-49-2.nip.io)\n -a authorization server url (default: http://keycloak.192-168-49-2.nip.io)\n -r realm within Keycloak (default: master)\n -u username used for authentication (default: admin)\n -p password used for authentication (default: changeme)\n -c client id of the bootstrap client used in the create request (default: admin-cli)\n -O user ID of the 'owner' of the new workspace (default: workspace(-W))\n -W name of the workspace to create (default: owner(-O))\nMost of the arguments have default values that are aligned to the defaults of the scripted deployment. At minimum either -O owner or -W workspace must be specified.
For example (assuming defaults)\u2026
./deploy/bin/create-workspace -O eric\nFor example (all arguments)\u2026
./deploy/bin/create-workspace \n -w http://workspace-api.192-168-49-2.nip.io \\\n -a http://keycloak.192-168-49-2.nip.io \\\n -r master \\\n -u admin \\\n -p changeme \\\n -c admin-cli \\\n -O bob \\\n -W bob\nThe eoepca-portal is a simple web application that is used as a test aid. It\u2019s main purpose is to provide the ability to login, and so establish a session with appropriate browser cookies - which then allow authenticated access to other EOEPCA services such as the Workspace API, Identity API, etc.
The portal is deployed via a helm chart\u2026
helm install eoepca-portal eoepca-portal -f portal-values.yaml - \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"demo\" --create-namespace \\\n --version 1.0.11\nThe helm values must be tailored for your deployment. For example\u2026
configMap:\n identity_url: \"http://keycloak.192-168-49-2.nip.io\"\n realm: \"master\"\n client_id: \"eoepca-portal\"\n identity_api_url: \"http://identity-api.192-168-49-2.nip.io\"\n ades_url: \"http://zoo.192-168-49-2.nip.io/ogc-api/processes\"\n resource_catalogue_url: \"http://resource-catalogue.192-168-49-2.nip.io\"\n data_access_url: \"http://data-access.192-168-49-2.nip.io\"\n workspace_url: \"http://workspace-api.192-168-49-2.nip.io\"\n workspace_docs_url: \"http://workspace-api.192-168-49-2.nip.io/docs#\"\n images_registry_url: \"http://harbor.192-168-49-2.nip.io\"\n dummy_service_url: \"http://dummy-service.192-168-49-2.nip.io\"\n access_token_name: \"auth_user_id\"\n access_token_domain: \".192-168-49-2.nip.io\"\n refresh_token_name: \"auth_refresh_token\"\n refresh_token_domain: \".192-168-49-2.nip.io\"\ningress:\n enabled: true\n annotations:\n kubernetes.io/ingress.class: nginx\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - host: eoepca-portal.192-168-49-2.nip.io\n paths:\n - path: /\n pathType: Prefix\n tls:\n - secretName: eoepca-portal-tls\n hosts:\n - eoepca-portal.192-168-49-2.nip.io\nThe setting client_id: eoepca-portal identifies a client that must be created in Keycloak - as described in section create-client Helper Script - noting that the eoepca-portal requires a client that is configured as a Public Client\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=eoepca-portal \\\n --name=\"EOEPCA Portal\" \\\n --public \\\n --description=\"Client to be used by the EOEPCA Portal\"\nBefore initiating a fresh deployment, if a prior deployment has been attempted, then it is necessary to remove any persistent artefacts of the prior deployment. This includes\u2026
Minikube cluster Delete the minikube cluster\u2026 minikube delete If necessary specify the cluster (profile)\u2026 minikube -p <profile> delete
Persistent Data In the case that the minikube none driver is used, the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. In this case, the minikube standard StorageClass is fulfilled by the hostpath provisioner, whose persistence is removed as follows\u2026 sudo rm -rf /tmp/hostpath-provisioner
There is a helper script clean that can be used for step 2 above, (the script does not delete the cluster).
./deploy/cluster/clean\nA deployment wrapper script has been prepared for a \u2018simple\u2019 deployment - designed to get a core local deployment of the primary servies.
The script deploy/simple/simple achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/simple/simple-options.
The simple deployment applies the following configuration:
minio, to avoid the need to create a Workspace for each userDeployment is initiated by invoking the script\u2026
./deploy/simple/simple\nSee section Deployment for more details regarding the outcome of the scripted deployment.
"},{"location":"quickstart/simple-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/userman-deployment/","title":"User Management Deployment","text":""},{"location":"quickstart/userman-deployment/#overview","title":"Overview","text":"A deployment wrapper script has been prepared for a \u2018user management\u2019 deployment - that is focused on the Identity Service (Authorization Server), Identity API and Gatekeeper (Protection Policy Enforcement).
The script deploy/userman/userman achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/userman/userman-options.
The user-management deployment applies the following configuration:
Deployment is initiated by invoking the script\u2026
./deploy/userman/userman\nThe Identity Service is accessed at the endpoint keycloak.192-168-49-2.nip.io.
The Identity API is accessed at the endpoint identity-api.192-168-49-2.nip.io.
To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Deployment Guide","text":"Changelog
This current version of the Deployment Guide represents the development tip that goes beyond the latest release version v1.4.
The following provides a summary of changes since the last release (v1.4)\u2026
1.4.1 to introduce variables to remedy hard-coded harvester values for access to Creodias eodata. Ref. - CREODIAS_EODATA_S3_ENDPOINT, CREODIAS_EODATA_S3_ACCESS_KEY, CREODIAS_EODATA_S3_ACCESS_SECRET and CREODIAS_EODATA_S3_REGION.eoepca-portalsnuggs sample app2.0.59 to add support for path-prefix (BASE_URL)2.0.58 to fix hard-coded namespace proc Namespace can now be set via chart environment variable APP_HUB_NAMESPACEPROCESSING_MAX_RAM to the integer value in Mi 1024 (was string 8Gi)The Deployment Guide captures each release of the EOEPCA Reference Implementation, by providing for each version\u2026
A full system deployment is described, in which components are deployed with complementary configurations that facilitate their integration as a coherent system. Nevertheless, each component can be cherry-picked from this system deployment for individual re-use.
The deployment is organised into the following sections:
The following prerequisite components are assumed to be deployed in the cluster.
Note
The Scripted Deployment automatically deploys most of the components list here - in particular\u2026
The Sealed Secrets controller is not deployed - but can be added following the instructions below.
"},{"location":"cluster/cluster-prerequisites/#nginx-ingress-controller","title":"Nginx Ingress Controller","text":"# Install the Nginx Ingress Controller helm chart\nhelm upgrade -i --version='<4.5.0' \\\n --repo https://kubernetes.github.io/ingress-nginx \\\n ingress-nginx ingress-nginx \\\n --wait\nNote
For Kubernetes version 1.22 and earlier the version of the Nginx Ingress Controller must be before v4.5.0.
To target the Nginx Ingress Controller the kubernetes.io/ingress.class: nginx annotation must be applied to the Ingress resource\u2026
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n annotations:\n kubernetes.io/ingress.class: nginx\n ...\n# Install the Cert Manager helm chart\nhelm upgrade -i --namespace cert-manager --create-namespace \\\n --repo https://charts.jetstack.io \\\n --set installCRDs=true \\\n cert-manager cert-manager\nOnce the Certificate Manager is deployed, then we can establish ClusterIssuer operators in the cluster to support use of TLS with service Ingress endpoints.
For Letsencrypt we can define two ClusterIssuer - for production and for staging.
NOTE that these require the cluster to be publicly accessible, in order for the http01 acme flow to verify the domain ownership. Local development deployments will typically not have public IP/DNS - in which case the system deployment can proceed, but without TLS support for the service endpoints.
apiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-production\nspec:\n acme:\n # You must replace this email address with your own.\n # Let's Encrypt will use this to contact you about expiring\n # certificates, and issues related to your account.\n email: eoepca.systemteam@telespazio.com\n server: https://acme-v02.api.letsencrypt.org/directory\n privateKeySecretRef:\n # Secret resource that will be used to store the account's private key.\n name: letsencrypt-production-account-key\n # Add a single challenge solver, HTTP01 using nginx\n solvers:\n - http01:\n ingress:\n class: nginx\napiVersion: cert-manager.io/v1\nkind: ClusterIssuer\nmetadata:\n name: letsencrypt-staging\nspec:\n acme:\n # You must replace this email address with your own.\n # Let's Encrypt will use this to contact you about expiring\n # certificates, and issues related to your account.\n email: eoepca.systemteam@telespazio.com\n server: https://acme-staging-v02.api.letsencrypt.org/directory\n privateKeySecretRef:\n # Secret resource that will be used to store the account's private key.\n name: letsencrypt-staging-account-key\n # Add a single challenge solver, HTTP01 using nginx\n solvers:\n - http01:\n ingress:\n class: nginx\nTo exploit the specified ClusterIssuer the cert-manager.io/cluster-issuer annotation must be applied to the Ingress resource. For example\u2026
apiVersion: networking.k8s.io/v1\nkind: Ingress\nmetadata:\n annotations:\n kubernetes.io/ingress.class: nginx\n cert-manager.io/cluster-issuer: letsencrypt-production\n ...\nThe EOEPCA development team maintain their deployment configurations in GitHub - for declarative, reproducible cluster deployments.
Various Secret are relied upon by the system services. Secrets should not be exposed by commit to GitHub.
Instead SealedSecret are committed to GitHub, which are encrypted, and can only be decrypted by the sealed-secret-controller that runs within the cluster. The sealed-secret-controller decrypts the SealedSecret to a regular Secret (of the same name) that can then be consumed by the cluster components.
The sealed-secret-controller is deployed to the cluster using the helm chart\u2026
helm install --version 2.1.8 --create-namespace --namespace infra \\\n --repo https://bitnami-labs.github.io/sealed-secrets \\\n eoepca-sealed-secrets sealed-secrets\nOnce the controller is deployed within the cluster, then the kubeseal command can be used to create a SealedSecret from a regular Secret, as follows\u2026
Create example Secret\u2026
kubectl -n test create secret generic mysecret \\\n --from-literal=password=changeme \\\n --dry-run=client -o yaml \\\n > mysecret.yaml\nCreate SealedSecret from Secret using kubeseal\u2026
kubeseal -o yaml \\\n --controller-name eoepca-sealed-secrets \\\n --controller-namespace infra \\\n < mysecret.yaml \\\n > mysecret-sealed.yaml\nkubeseal ReleaseVarious building blocks require access to an S3-compatible object storage service. In particular the ADES processing service expects to stage-out its processing results to S3 object storage. Ideally the cloud provider for your deployment will make available a suitable object storage service.
As a workaround, in the absence of an existing object storage, it is possible to use MinIO to establish an object storage service within the Kubernetes cluster. We use the minio helm chart provided by the MinIO Project.
# Install the minio helm chart\nhelm upgrade -i -f minio-values.yaml --namespace rm --create-namespace \\\n --repo https://charts.min.io/ \\\n minio minio \\\n --wait\nNote
The Kubernetes namespace rm is used above as an example, and can be changed according to your deployment preference.
The minio deployment is customised via the values file minio-values.yaml, for example\u2026
existingSecret: minio-auth\nreplicas: 2\n\ningress:\n enabled: true\n ingressClassName: nginx\n annotations:\n cert-manager.io/cluster-issuer: \"letsencrypt\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n path: /\n hosts:\n - minio.192-168-49-2.nip.io\n tls:\n - secretName: minio-tls\n hosts:\n - minio.192-168-49-2.nip.io\n\nconsoleIngress:\n enabled: true\n ingressClassName: nginx\n annotations:\n cert-manager.io/cluster-issuer: \"letsencrypt\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n path: /\n hosts:\n - console.minio.192-168-49-2.nip.io\n tls:\n - secretName: minio-console-tls\n hosts:\n - console.minio.192-168-49-2.nip.io\n\nresources:\n requests:\n memory: 1Gi\n\npersistence:\n storageClass: standard\n\nbuckets:\n - name: eoepca\n - name: cache-bucket\nNote
letsencrypt certificate providerminio-auth - see belownginx.ingress.kubernetes.io/proxy-body-size was found to be required to allow transfer of large files (such as data products) through the nginx proxyThe Minio admin credentials are provided via a Kubernetes secret that is referenced from the Minio helm chart deployment values. For example\u2026
kubectl -n rm create secret generic minio-auth \\\n --from-literal=rootUser=\"eoepca\" \\\n --from-literal=rootPassword=\"changeme\"\nNote
The secret must be created in the same Kubernetes namespace as the Minio service deployment - e.g. rm namespce in the example above.
The s3cmd can be configured for access to the MinIO deployment. The --configure option can be used to prepare a suitable configuration file for s3cmd\u2026
s3cmd -c mys3cfg --configure\nIn response to the prompts, the following configuration selections are applicable to the above settings\u2026
Access Key: eoepca\nSecret Key: changeme\nDefault Region: us-east-1\nS3 Endpoint: minio.192-168-49-2.nip.io\nDNS-style bucket+hostname:port template for accessing a bucket: minio.192-168-49-2.nip.io\nEncryption password: \nPath to GPG program: /usr/bin/gpg\nUse HTTPS protocol: True\nHTTP Proxy server name: \nHTTP Proxy server port: 0\nSave the configuration file, and check access to the S3 object store with\u2026
# Create a bucket\ns3cmd -c mys3cfg mb s3://eoepca\n\n# List buckets\ns3cmd -c mys3cfg ls\nFor example, using our sample deployment, the following can be used to interface with the MinIO service deployed in minikube\u2026
s3cmd -c deploy/cluster/s3cfg ls\nNote
This section identifies some helm chart repositories that can be referenced (for convenience) via helm add. Nevertheless, all helm commands included in the guide specifically reference the source helm repository via the --repo argument to the helm install command - and thus it is not specifically necessary to add these repositories in advance.
The EOEPCA building-blocks are engineered as containers for deployment to a Kubernetes cluster. Each building block defines a Helm Chart to facilitate its deployment.
The EOEPCA Helm Chart Repository is configured with helm as follows\u2026
helm repo add eoepca https://eoepca.github.io/helm-charts/\nIn addition to the EOEPCA Helm Chart Repository, a variety of third party helm repositories are relied upon, as identified below.
"},{"location":"cluster/helm-repositories/#cert-manager","title":"Cert Manager","text":"helm repo add jetstack https://charts.jetstack.io\nhelm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx\nhelm repo add minio https://charts.min.io/\nhelm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets\nhelm repo add harbor https://helm.goharbor.io\nRefresh the local repo cache, after helm repo add\u2026
helm repo update\nThe EOEPCA Reference Implementation has been developed with Kubernetes as its deployment target. The system components have been developed, deployed and tested using a cluster at version v1.22.5.
Note
The Scripted Deployment assumes that minikube is installed, and creates a minikube cluster under the profile eoepca.
The development, integration and test clusters have been established using Rancher Kubernetes Engine (RKE) at version v1.22.5.
An example of the creation of the EOEPCA Kubernetes clusters can be found on the GitHub Kubernetes Setup page. CREODIAS has been used for the development hosting infrastructure - which provides OpenStack infrastructure that is backed by Cloudferro. An example of the Terraform configurations used to automate the creation of the cloud infrastructure that underpins the RKE deployment can be found on the GitHub CREODIAS Setup page.
"},{"location":"cluster/kubernetes/#local-kubernetes","title":"Local Kubernetes","text":"To make a full deployment of the EOEPCA Reference Implementation requires a multi-node node cluster with suitable resources. For example, the development cluster comprises:
Limited local deployment can be made using a suitable local single-node kuberbetes deployment using - for example using minikube\u2026
minikube -p eoepca start --cpus max --memory max --kubernetes-version v1.22.5\nminikube profile eoepca\nWith such a deployment it is possible to deploy individual building-blocks for local development, or building-blocks in combination - within the constraints of the local host resources.
"},{"location":"cluster/prerequisite-tooling/","title":"Prerequisite Tooling","text":"There are some standard tools referenced in this guide. These are detailed in the following subsections.
"},{"location":"cluster/prerequisite-tooling/#docker","title":"docker","text":"Docker faciliates the creation, management and execution of containers. Whilst not strictly necessary to support deployment to an existing/managed Kubernetes cluster, it can nevertheless be useful to have local access to the docker tooling. For example, if minikube is used to follow this guide using a local k8s cluster, then this is best achieved using minikube\u2019s docker driver.
Docker is most easily installed with\u2026
curl -fsSL https://get.docker.com | sh\nFor convenience, add your user to the docker group\u2026
sudo usermod -aG docker ${USER}\nLogout/in to refresh your session\u2019s group permissions.
"},{"location":"cluster/prerequisite-tooling/#kubectl","title":"kubectl","text":"Kubectl is the main tool for interaction with a Kubernetes cluster. The latest version can be installed with\u2026
mkdir -p $HOME/.local/bin \\\n&& curl -fsSLo $HOME/.local/bin/kubectl \"https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl\" \\\n&& chmod +x $HOME/.local/bin/kubectl\nSee the official kubectl installation documentation for more installation options.
"},{"location":"cluster/prerequisite-tooling/#helm","title":"helm","text":"Helm is the Kubernetes package manager, in which components are deployed to a Kubernetes cluster via helm charts. The helm charts are instantiated for deployment via \u2018values\u2019 that configure the chart templates.
The latest helm version can be installed with\u2026
export HELM_INSTALL_DIR=\"$HOME/.local/bin\" \\\n&& curl -sfL https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash\nSee the official helm installation documentation for more installation options.
"},{"location":"cluster/prerequisite-tooling/#minikube","title":"minikube","text":"Minikube is a tool that allows to create a local (single-node) Kubernetes cluster for development/testing. It is not designed for production use. In the absence of access to a \u2018full\u2019 Kubernetes cluster, this guide can be followed using minikube.
The latest version of minikube can be installed with\u2026
mkdir -p $HOME/.local/bin \\\n&& curl -fsSLo $HOME/.local/bin/minikube \"https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64\" \\\n&& chmod +x $HOME/.local/bin/minikube\nSee the official minikube installation documentation for more installation options.
"},{"location":"eoepca/ades-zoo/","title":"ADES (Processing)","text":"ADES - Application Deployment & Execution Service"},{"location":"eoepca/ades-zoo/#zoo-project-dru","title":"ZOO-Project DRU","text":"Note
With EOEPCA release 1.4, the ADES implementation has been significantly reworked and fully aligned with the upstream ZOO-Project (GitHub). This zoo-project-dru version deprecates the previous proc-ades implementation.
With this transition, there are some functional changes to be aware of\u2026
zoo-project-dru the OGC API Processes endpoint is at the path /<username>/ogc-api/processes compared to the previous /<username>/wps3/processes.convert-url at version 0.1.2 would result in the endpoint /<username>/wps3/processes/convert-url_0_1_2. With the new zoo-project-dru this same Application Package deployment will result in the endpoint /<username>/ogc-api/processes/convert-url.s:softwareVersion: 0.1.2), and is maintained within the metadata for the deployed process that is returned from the APIs Get Process Details request. In the case that multiple versions of the same Application Package are required to be simultaneously deployed, then this would have to be handled with different CWL documents in which the version is embedded in the workflow id (or some other technique that establishes uniqueness of id between variants).The ADES provides a platform-hosted execution engine through which users can initiate parameterised processing jobs using applications made available within the platform - supporting the efficient execution of the processing \u2018close to the data\u2019. Users can deploy specific \u2018applications\u2019 to the ADES, which may be their own applications, or those published by other platform users.
The ADES provides an implementation of the OGC API Processes - Part 1: Core and Part 2: Deploy, Replace, Undeploy (draft).
"},{"location":"eoepca/ades-zoo/#helm-chart","title":"Helm Chart","text":"The EOEPCA deployment is aligned with the upstream implementation and so relies upon the upstream helm chart that is hosted at the ZOO-Project Helm Chart Repository - in particular the zoo-project-dru chart variant.
The chart is configured via values that are fully documented in the README for the zoo-project-dru chart.
helm install --version 0.2.6 --values ades-values.yaml \\\n --repo https://zoo-project.github.io/charts/ \\\n zoo-project-dru zoo-project-dru\nThe deployment must be configured for you environment. Some significant configuration values are elaborated here\u2026
"},{"location":"eoepca/ades-zoo/#cookie-cutter-template","title":"Cookie-cutter Template","text":"The implementation zoo-project-dru provides the core capabilities for OGC API Processes Parts 1 & 2. The deployemnt of this core must be completed by inetgartion with the \u2018runner\u2019 that executes the processes as Application Packages, and integrates as necessary with other platform services - such as Catalogue, Workspace, etc.
Thus, zoo-project-dru is extensible by design via a \u2018cookie-cutter\u2019 that provides the template \u2018runner\u2019 for each Application Package process as it is deployed to the service.
For the purposes of our EOEPCA \u2018release\u2019 as covered by this guide, we provide eoepca-proc-service-template as a cookie-cutter implemetation that provides:
The cookie-cutter template is identified in the helm values\u2026
cookiecutter:\n templateUrl: https://github.com/EOEPCA/eoepca-proc-service-template.git\n templateBranch: master\nThe function of the cookie-cutter template is supported some other aspects, that are elaborated below, which must be configured in collaboration with the expectations of the template. In particular\u2026
zoo-project-dru configuration [ref]In order support our eoepca-proc-service-template cookie-cutter template, there is a custom zoo-project-dru container image that includes the python dependencies that are required by this template. Thus, the deployment must identify the custom container image via helm values\u2026
zoofpm:\n image:\n tag: eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762\nzookernel:\n image:\n tag: eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762\nIn addition, we can add values to the ZOO-Project DRU main.cfg configuration file via helm values. In this case we add some eoepca-specific values that match those that we know to be expected by our eoepca-proc-service-template cookie-cutter template. In this way we can effectively use helm values to pass parameters through to the template.
customConfig:\n main:\n eoepca: |-\n domain=192-168-49-2.nip.io\n workspace_prefix=ws\nThis is manifest in zoo\u2019s main.cfg in INI file configuration syntax\u2026
[eoepca]\ndomain=192-168-49-2.nip.io\nworkspace_prefix=ws\nThe presence or otherwise of the workspace_prefix parameter dicates whether or not the stage-out step will integrate with the user\u2019s Workspace for persistence of the processing results, and registration within the Workspace services.
In the case that workspace_prefix is not set, then the object storage specification in the helm values is relied upon\u2026
workflow:\n inputs:\n STAGEOUT_AWS_SERVICEURL: https://minio.192-168-49-2.nip.io\n STAGEOUT_AWS_ACCESS_KEY_ID: eoepca\n STAGEOUT_AWS_SECRET_ACCESS_KEY: changeme\n STAGEOUT_AWS_REGION: RegionOne\n STAGEOUT_OUTPUT: eoepca\nThe ADES hosts applications that are deployed and invoked in accordance with the OGC Best Practise for Application Package. Thus, the ADES provides a conformant environment within which the application is integrated for execution. A key part of the ADES\u2019s role in this is to faciltate the provision of input data to the application (stage-in), and the handling of the results output at the conclusion of application execution (stage-out).
The zoo-project-dru helm chart provides a default implementation via the included files - main.yaml, rules.yaml, stagein.yaml and stageout.yaml.
The helm values provides a means through which each of these files can be overriden for reasons of integration with your platform environment\u2026
files:\n # Directory 'files/cwlwrapper-assets' - assets for ConfigMap 'XXX-cwlwrapper-config'\n cwlwrapperAssets:\n main.yaml: |-\n <override file content here>\n rules.yaml: |-\n <override file content here>\n stagein.yaml: |-\n <override file content here>\n stageout.yaml: |-\n <override file content here>\nIn the most part the default CWL wrapper files provided with the helm chart are suffient. In particular the stagein.yaml implements the stage-in of STAC items that are specified as inputs of type Directory in the Application Package CWL.
E.g.
inputs:\n stac:\n label: the image to convert as a STAC item\n doc: the image to convert as a STAC item\n type: Directory\nNevertheless, in this guide we provide an override of the stageout.yaml in order to organise the processing outputs into a STAC Collection that is then pushed to the designated S3 object storage, including support for the user\u2019s workspace storage and resource management services.
The custom stage-out embeds, within the CWL document, the python code required to implement the desired stage-out functionality. This should be regarded as an example that could be adapted for alternative behaviour.
cwlVersion: v1.0\nclass: CommandLineTool\nid: stage-out\ndoc: \"Stage-out the results to S3\"\ninputs:\n process:\n type: string\n collection_id:\n type: string\n STAGEOUT_OUTPUT:\n type: string\n STAGEOUT_AWS_ACCESS_KEY_ID:\n type: string\n STAGEOUT_AWS_SECRET_ACCESS_KEY:\n type: string\n STAGEOUT_AWS_REGION:\n type: string\n STAGEOUT_AWS_SERVICEURL:\n type: string\noutputs:\n StacCatalogUri:\n outputBinding:\n outputEval: ${ return \"s3://\" + inputs.STAGEOUT_OUTPUT + \"/\" + inputs.process + \"/catalog.json\"; }\n type: string\nbaseCommand:\n - python\n - stageout.py\narguments:\n - $( inputs.wf_outputs.path )\n - $( inputs.STAGEOUT_OUTPUT )\n - $( inputs.process )\n - $( inputs.collection_id )\nrequirements:\n DockerRequirement:\n dockerPull: ghcr.io/terradue/ogc-eo-application-package-hands-on/stage:1.3.2\n InlineJavascriptRequirement: {}\n EnvVarRequirement:\n envDef:\n AWS_ACCESS_KEY_ID: $( inputs.STAGEOUT_AWS_ACCESS_KEY_ID )\n AWS_SECRET_ACCESS_KEY: $( inputs.STAGEOUT_AWS_SECRET_ACCESS_KEY )\n AWS_REGION: $( inputs.STAGEOUT_AWS_REGION )\n AWS_S3_ENDPOINT: $( inputs.STAGEOUT_AWS_SERVICEURL )\n InitialWorkDirRequirement:\n listing:\n - entryname: stageout.py\n entry: |-\n import sys\n import shutil\n import os\n import pystac\n\n cat_url = sys.argv[1]\n\n shutil.copytree(cat_url, \"/tmp/catalog\")\n cat = pystac.read_file(os.path.join(\"/tmp/catalog\", \"catalog.json\"))\n\n ...\nThe helm chart values provide the opportunity to pass through additional inputs - to satisfy the input specifications that are specified in the cwlwrapperAssets files\u2026
workflow:\n inputs:\n STAGEIN_AWS_SERVICEURL: http://data.cloudferro.com\n STAGEIN_AWS_ACCESS_KEY_ID: test\n STAGEIN_AWS_SECRET_ACCESS_KEY: test\n STAGEIN_AWS_REGION: RegionOne\n STAGEOUT_AWS_SERVICEURL: https://minio.192-168-49-2.nip.io\n STAGEOUT_AWS_ACCESS_KEY_ID: eoepca\n STAGEOUT_AWS_SECRET_ACCESS_KEY: changeme\n STAGEOUT_AWS_REGION: RegionOne\n STAGEOUT_OUTPUT: eoepca\nThe zoo-project-dru services uses a Node Selector to determine the node(s) upon which the processing execution is run. This is configured as a matching rule in the helm values, and must be tailored to your cluster.
For example, for minikube\u2026
workflow:\n nodeSelector:\n minikube.k8s.io/primary: \"true\"\nIngress can be enabled and configured to establish (reverse-proxy) external access to the zoo-project-dru services.
Hosturl
In the case that protection is enabled - e.g. via Resource Guard - then it is likely that ingress should be disabled here, since the ingress will instead be handled by the protection.
In this case, the hosturl parameter should be set to reflect the public url through the service will be accessed.
In the case that ingress is enabled then it is not necessary to specify the hosturl, since it will be taken from the ingress.hosts[0].host value.
Ingress disabled\u2026
ingress:\n enabled: false\n hosturl: zoo.192-168-49-2.nip.io\nIngress enabled\u2026
ingress:\n enabled: true\n annotations:\n kubernetes.io/ingress.class: nginx\n ingress.kubernetes.io/ssl-redirect: true\n nginx.ingress.kubernetes.io/ssl-redirect: true\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - host: zoo-open.192-168-49-2.nip.io\n paths:\n - path: /\n pathType: ImplementationSpecific\n tls:\n - hosts:\n - zoo-open.192-168-49-2.nip.io\n secretName: zoo-open-tls\nThe above example assumes that TLS should be enabled via Letsencrypt as certificate provider - see section Letsencrypt Certificates.
"},{"location":"eoepca/ades-zoo/#persistence","title":"Persistence","text":"Various of the services deployed as part of zoo-project-dru rely upon dynamic provisioning of persistent storage volumes.
A number of helm values are impacted by this setting, which must be configured with the Storage Class appropriate to your cluster. For example, using the minikube standard storage class\u2026
workflow:\n storageClass: standard\npersistence:\n procServicesStorageClass: standard\n storageClass: standard\n tmpStorageClass: standard\npostgresql:\n primary:\n persistence:\n storageClass: standard\n readReplicas:\n persistence:\n storageClass: standard\nrabbitmq:\n persistence:\n storageClass: standard\nZOO-Project DRU has a built-in capability for Identity & Access Management (IAM), in which the zoo-project-dru service is configured as an OIDC client of an OIDC Identity Provider service.
This capability is disabled by the default deployment offered by this guide (ingress.enabled: false) - which instead (optionally) applies resource protection using the EOEPCA IAM solution. Nevertheless, the built-in IAM can be enabled and configured through helm values.
For example\u2026
iam: \n enabled: true\n openIdConnectUrl: https://keycloak.192-168-49-2.nip.io/realms/master/.well-known/openid-configuration\n type: openIdConnect\n name: OpenIDAuth\n realm: Secured section\nAs described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the zoo-project-dru service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install zoo-project-dru-protection identity-gatekeeper -f zoo-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"zoo\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the zoo-project-dru - in particular the specific ingress requirements for the zoo-project-dru-service\u2026
Example zoo-protection-values.yaml\u2026
fullnameOverride: zoo-project-dru-protection\nconfig:\n client-id: ades\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: zoo.192-168-49-2.nip.io\n name: zoo-project-dru-service\n port:\n number: 80\nsecrets:\n # Values for secret 'zoo-project-dru-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n serverSnippets:\n custom: |-\n # Open access to some endpoints, including Swagger UI\n location ~ /(ogc-api/api|swagger-ui) {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: ades above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example, with path protection for test users\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=ades \\\n --name=\"ADES Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by ADES Gatekeeper\" \\\n --resource=\"eric\" --uris='/eric/*' --scopes=view --users=\"eric\" \\\n --resource=\"bob\" --uris='/bob/*' --scopes=view --users=\"bob\" \\\n --resource=\"alice\" --uris='/alice/*' --scopes=view --users=\"alice\"\nThe zoo-project-dru service provides a mutil-user aware set of service interfaces at\u2026
https://zoo.192-168-49-2.nip.io/<username>/ogc-api/https://zoo.192-168-49-2.nip.io/swagger-ui/oapip/See the Example Requests in the Processing Deployment for sample requests that cans be used to test your deployment, and to learn usage of the OGC API Processes.
"},{"location":"eoepca/ades-zoo/#debugging-tips","title":"Debugging Tips","text":"This section includes some tips that may be useful in debugging errors with deployed application packages.
For debugging, establish a shell session with the zoofpm pod\u2026
$ kubectl -n zoo exec -it deploy/zoo-project-dru-zoofpm -c zoofpm -- bash\nThe logs are in the directory /tmp/zTmp\u2026
$ cd /tmp/zTmp/\nIn the log directory, each execution is characterised by a set of files/directories\u2026
<appname>_<jobid>_error.log <<START HERE The main log output of the job<appname>_<jobid>.json The output (results) of the job<jobid>_status.json The overall status of the job<jobid>_logs.cfg Index of logs for job workflow stepsconvert-url-c6637d4a-d561-11ee-bf3b-0242ac11000e (directory) Subdirectory with a dedicated log file for each step of the CWL workflow, including the stage-in and stage-out stepsWhen the process is deployed from its Application Package, then a representation is created using the configured cookiecutter.templateUrl.
It may be useful to debug the consequent process files, which are located under the path /opt/zooservices_user/<username>, with a dedicated subdirectory for each deployed process - i.e. /opt/zooservices_user/<username>/<appname>/.
For example\u2026
$ cd /opt/zooservices_user/eric/convert-url\n$ ls -l\ntotal 28\n-rw-rw-r-- 1 www-data www-data 0 Feb 27 11:17 __init__.py\ndrwxrwxr-x 2 www-data www-data 4096 Feb 27 11:17 __pycache__\n-rw-rw-r-- 1 www-data www-data 1408 Feb 27 11:17 app-package.cwl\n-rw-rw-r-- 1 www-data www-data 17840 Feb 27 11:17 service.py\nNote
In the case that the cookie-cutter template is updated, then the process can be re-deployed to force a refresh against the updated template.
"},{"location":"eoepca/ades-zoo/#swagger-ui-openapi","title":"Swagger UI (OpenAPI)","text":"The zoo-project-dru service includes a Swagger UI interactive representation of its OpenAPI REST interface - available at the URL https://zoo.192-168-49-2.nip.io/swagger-ui/oapip/.
For a (trivial) example application package see Example Application Package, which provides a description and illustration of the basics of creating an application that integrates with the expectations of the ADES stage-in and stage-out.
For further reference see\u2026
Additional information regarding the ADES can be found at:
eoepca-proc-service-template to aid orchestration of CWL application packages running in Kubernetes via Calrissianzoo-calrissian-runner to aid interfacing with Calrissian and KubernetesThe Application Hub provides a set of web-based tooling, including JupyterLab for interactive analysis, Code Server for application development, and the capability to add user-defined interactive dashboards.
"},{"location":"eoepca/application-hub/#helm-chart","title":"Helm Chart","text":"The Application Hub is deployed via the application-hub helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values, which are detailed in the default values file for the chart.
helm install --version 2.0.58 --values application-hub-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n application-hub application-hub\nThe Application Hub supports many values to configure the service - ref. the default values file for the chart.
Typically, values for the following attributes may be specified:
Example application-hub-values.yaml\u2026
ingress:\n enabled: true\n annotations: {}\n hosts:\n - host: applicationhub.192-168-49-2.nip.io\n paths:\n - path: /\n pathType: ImplementationSpecific\n tls:\n - secretName: applicationhub-tls\n hosts:\n - applicationhub.192-168-49-2.nip.io\n clusterIssuer: letsencrypt-production\n\njupyterhub:\n fullnameOverride: \"application-hub\"\n hub:\n existingSecret: application-hub-secrets\n extraEnv: \n JUPYTERHUB_ENV: \"dev\"\n JUPYTERHUB_SINGLE_USER_IMAGE: \"eoepca/pde-container:1.0.3\"\n OAUTH_CALLBACK_URL: https://applicationhub.192-168-49-2.nip.io/hub/oauth_callback\n OAUTH2_USERDATA_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/userinfo\n OAUTH2_TOKEN_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/token\n OAUTH2_AUTHORIZE_URL: https://keycloak.192-168-49-2.nip.io/oxauth/restv1/authorize\n OAUTH_LOGOUT_REDIRECT_URL: \"https://applicationhub.192-168-49-2.nip.io\"\n OAUTH2_USERNAME_KEY: \"preferred_username\"\n APP_HUB_NAMESPACE: \"app-hub\"\n STORAGE_CLASS: \"standard\"\n RESOURCE_MANAGER_WORKSPACE_PREFIX: \"ws\"\n\n JUPYTERHUB_CRYPT_KEY:\n valueFrom:\n secretKeyRef:\n name: application-hub-secrets\n key: JUPYTERHUB_CRYPT_KEY\n\n OAUTH_CLIENT_ID:\n valueFrom:\n secretKeyRef:\n name: application-hub-secrets\n key: OAUTH_CLIENT_ID\n\n OAUTH_CLIENT_SECRET:\n valueFrom:\n secretKeyRef:\n name: application-hub-secrets\n key: OAUTH_CLIENT_SECRET\n\n image:\n # name: eoepca/application-hub\n # tag: \"1.2.0\"\n pullPolicy: Always\n # pullSecrets: []\n\n db:\n pvc:\n storageClassName: standard\n\n singleuser:\n image:\n name: jupyter/minimal-notebook\n tag: \"2343e33dec46\"\n profileList: \n - display_name: \"Minimal environment\"\n description: \"To avoid too much bells and whistles: Python.\"\n default: \"True\"\n - display_name: \"EOEPCA profile\"\n description: \"Sample profile\"\n kubespawner_override:\n cpu_limit\": 4\n mem_limit\": \"8G\"\n\nnodeSelector:\n key: minikube.k8s.io/primary\n value: \\\"true\\\"\nThe Application Hub requires an OIDC client to be registered with the Identity Service (Keycloak) in order to enable user identity integration - ref. OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=application-hub \\\n --name=\"Application Hub OIDC Client\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Application Hub for OIDC integration\"\nCorresponding to this client, a secret application-hub-secrets must be created (ref. value jupyterhub.hub.existingSecret: application-hub-secrets)\u2026
kubectl -n proc create secret generic application-hub-secrets \\\n --from-literal=JUPYTERHUB_CRYPT_KEY=\"$(openssl rand -hex 32)\" \\\n --from-literal=OAUTH_CLIENT_ID=\"application-hub\" \\\n --from-literal=OAUTH_CLIENT_SECRET=\"changeme\"\nThe deployment of the Application Hub has been designed, as far as possible, to automate the configuration. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026
The default helm chart has some built-in application launchers whose assignments to example users (eric and bob) assume the existence of some JupyterHub groups - which must be replicated to exploit this configuration.
Admin menu (top of page)group-1, group-2, group-3 to ApplicationHub, and add users eric, bob to these groupsThis setup corresponds to the \u2018sample\u2019 configuration that is built=in to the help chart - see file config.yaml.
Additional information regarding the Application Hub can be found at:
To support the development (ref. Application Hub) and deployment/execution (ref. ADES) of user-defined applications, we deploy a container registry to host container images. This is provied by a deployment of the Harbor artefact repository.
"},{"location":"eoepca/container-registry/#helm-chart","title":"Helm Chart","text":"Harbor is deployed via the harbor helm chart from the Harbor Helm Chart Repository.
helm install --version 1.7.3 --values harbor-values.yaml \\\n --repo https://helm.goharbor.io \\\n harbor harbor\nThe chart is configured via values that are fully documented on the Harbor website.
Example\u2026
expose:\n ingress:\n annotations:\n kubernetes.io/ingress.class: nginx\n cert-manager.io/cluster-issuer: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-read-timeout: '600'\n\n # from chart:\n ingress.kubernetes.io/ssl-redirect: letsencrypt-production\n ingress.kubernetes.io/proxy-body-size: \"0\"\n nginx.ingress.kubernetes.io/ssl-redirect: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-body-size: \"0\"\n\n hosts:\n core: harbor.192-168-49-2.nip.io\n tls:\n enabled: \"true\"\n certSource: secret\n secret:\n secretName: \"harbor-tls\"\n\npersistence:\n persistentVolumeClaim:\n registry:\n storageClass: standard\n chartmuseum:\n storageClass: standard\n jobservice:\n storageClass: standard\n database:\n storageClass: standard\n redis:\n storageClass: standard\n trivy:\n storageClass: standard\n\nexternalURL: https://harbor.192-168-49-2.nip.io\n# initial password for logging in with user \"admin\"\nharborAdminPassword: \"changeme\"\n\nchartmuseum:\n enabled: false\ntrivy:\n enabled: false\nnotary:\n enabled: false\nNote
letsencrypt-production Cluster Issuer relies upon the deployment being accessible from the public internet via the expose.ingress.hosts.core DNS name. If this is not the case, e.g. for a local minikube deployment in which this is unlikely to be so. In this case the TLS will fall-back to the self-signed certificate built-in to the nginx ingress controller. The Workspace API will not like this.After deployemnt Harbor is accessible via its web interface at https://harbor.192-168-49-2.nip.io/e.g. https://harbor.192-168-49-2.nip.io/.
Login as the admin user with the password specified in the helm values.
"},{"location":"eoepca/container-registry/#additional-information","title":"Additional Information","text":"Additional information regarding the Container Registry can be found at:
The Data Access provides standards-based services for access to platform hosted data - including OGC WMS/WMTS for visualisation, and OGC WCS for data retrieval. This component also includes Harvester and Registrar services to discover/watch the existing data holding of the infrastructure data layer and populate/maintain the data access and resource catalogue services accordingly.
"},{"location":"eoepca/data-access/#helm-chart","title":"Helm Chart","text":"The Data Access is deployed via the data-access helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are supplied with the instantiation of the helm release. The EOEPCA data-access chart provides a thin wrapper around the EOX View Server (vs) helm chart. The documentation for the View Server can be found here:
helm install --version 1.4.0 --values data-access-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n data-access data-access\nThe Data Access supports many values to configure the service. These are documented in full in the View Server - Operator Guide Configuration page.
"},{"location":"eoepca/data-access/#core-configuration","title":"Core Configuration","text":"Typically, values for the following attributes may be specified to override the chart defaults:
global.ingress.hosts.host[0])database and redis componentsdata and cacherenderer and registraridentity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.global:\n env:\n REGISTRAR_REPLACE: \"true\"\n CPL_VSIL_CURL_ALLOWED_EXTENSIONS: .TIF,.tif,.xml,.jp2,.jpg,.jpeg\n AWS_ENDPOINT_URL_S3: https://minio.192-168-49-2.nip.io\n AWS_HTTPS: \"FALSE\"\n startup_scripts:\n - /registrar_pycsw/registrar_pycsw/initialize-collections.sh\n ingress:\n enabled: true\n annotations:\n kubernetes.io/ingress.class: nginx\n kubernetes.io/tls-acme: \"true\"\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - host: data-access.192-168-49-2.nip.io\n tls:\n - hosts:\n - data-access.192-168-49-2.nip.io\n secretName: data-access-tls\n storage:\n data:\n data:\n type: S3\n endpoint_url: http://data.cloudferro.com\n access_key_id: access\n secret_access_key: access\n region_name: RegionOne\n validate_bucket_name: false\n cache:\n type: S3\n endpoint_url: \"https://minio.192-168-49-2.nip.io\"\n host: \"minio.192-168-49-2.nip.io\"\n access_key_id: xxx\n secret_access_key: xxx\n region: us-east-1\n bucket: cache-bucket\n metadata:\n title: EOEPCA Data Access Service developed by EOX\n abstract: EOEPCA Data Access Service developed by EOX\n header: \"EOEPCA Data Access View Server (VS) Client powered by <a href=\\\"//eox.at\\\"><img src=\\\"//eox.at/wp-content/uploads/2017/09/EOX_Logo.svg\\\" alt=\\\"EOX\\\" style=\\\"height:25px;margin-left:10px\\\"/></a>\"\n url: https://data-access.192-168-49-2.nip.io/ows\n layers:\n # see section 'Data-layer Configuration'\n collections:\n # see section 'Data-layer Configuration'\n productTypes:\n # see section 'Data-layer Configuration'\nvs:\n renderer:\n replicaCount: 4\n ingress:\n enabled: false\n resources:\n requests:\n cpu: 100m\n memory: 300Mi\n limits:\n cpu: 1.5\n memory: 3Gi\n registrar:\n replicaCount: 1\n config:\n # see section 'Registrar Routes Configuration'\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\n harvester:\n # see section 'Harvester Configuration'\n replicaCount: 1\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\n client:\n replicaCount: 1\n ingress:\n enabled: false\n redis:\n master:\n persistence:\n enabled: true\n storageClass: standard\n ingestor:\n replicaCount: 0\n ingress:\n enabled: false\n preprocessor:\n replicaCount: 0\n cache:\n ingress:\n enabled: false\n scheduler:\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\nNote
The resources: above have been limited for the benefit of a minikube deployment. For a production deployment the values should be tuned (upwards) according to operational needs.
The Data Access registrar component supports a number of different resource types. For each a dedicated \u2018backend\u2019 is configured to handle the specific registration of the resource type\u2026
vs:\n registrar:\n config:\n #--------------\n # Default route\n #--------------\n disableDefaultRoute: false\n # Additional backends for the default route\n defaultBackends:\n - path: registrar_pycsw.backend.ItemBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n ows_url: https://data-access.192-168-49-2.nip.io/ows\n defaultSuccessQueue: seed_queue\n #----------------\n # Specific routes\n #----------------\n routes:\n collections:\n path: registrar.route.stac.CollectionRoute\n queue: register_collection_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.CollectionBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n ades:\n path: registrar.route.json.JSONRoute\n queue: register_ades_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.ADESBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n application:\n path: registrar.route.json.JSONRoute\n queue: register_application_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.CWLBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n catalogue:\n path: registrar.route.json.JSONRoute\n queue: register_catalogue_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.CatalogueBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n json:\n path: registrar.route.json.JSONRoute\n queue: register_json_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.JSONBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\n xml:\n path: registrar.route.json.JSONRoute\n queue: register_xml_queue\n replace: true\n backends:\n - path: registrar_pycsw.backend.XMLBackend\n kwargs:\n repository_database_uri: postgresql://postgres:mypass@resource-catalogue-db/pycsw\nConfiguration of the service data-layer - as described in the View Server Operator Guide.
The data-access service data handling is configured by definition of productTypes, collections and layers\u2026
productTypes - Product Types Identify the underlying file assets as WCS coverages and their visual representationcollections - Data Collections Provides groupings into which products are organisedlayers - Layers Specifies the hoe the product visual representations are exposed through the WMS serviceFor more information, see the worked example in section Data Specification for the example CREODIAS deployment.
"},{"location":"eoepca/data-access/#harvester","title":"Harvester","text":"The Data Access service includes a Harvester component. The following subsections describe its configuration and usage.
"},{"location":"eoepca/data-access/#harvester-helm-configuration","title":"Harvester Helm Configuration","text":"The Harvester can be configured through the helm chart values\u2026
vs:\n harvester:\n replicaCount: 1\n config:\n redis:\n host: data-access-redis-master\n port: 6379\n harvesters:\n - name: Creodias-Opensearch\n resource:\n url: https://datahub.creodias.eu/resto/api/collections/Sentinel2/describe.xml\n type: OpenSearch\n format_config:\n type: 'application/json'\n property_mapping:\n start_datetime: 'startDate'\n end_datetime: 'completionDate'\n productIdentifier: 'productIdentifier'\n query:\n time:\n property: sensed\n begin: 2019-09-10T00:00:00Z\n end: 2019-09-11T00:00:00Z\n collection: null\n bbox: 14.9,47.7,16.4,48.7\n filter: {}\n postprocess:\n - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor\n queue: register\n - name: Creodias-Opensearch-Sentinel1\n resource:\n url: https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml\n type: OpenSearch\n format_config:\n type: 'application/json'\n property_mapping:\n start_datetime: 'startDate'\n end_datetime: 'completionDate'\n productIdentifier: 'productIdentifier'\n query:\n time:\n property: sensed\n begin: 2019-09-10T00:00:00Z\n end: 2019-09-11T00:00:00Z\n collection: null\n bbox: 14.9,47.7,16.4,48.7\n extra_params:\n productType: GRD-COG\n filter: {}\n postprocess:\n - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel1Postprocessor\n queue: register\nThe harvester.config.harvesters list defines a set of pre-defined harvesters which can be invoked in a later stage. The name property must be unique for each harvester and must be unique among all harvesters in the list. Each harvester is associated with a resource, an optional filter or postprocess function, and a queue.
The resource defines where each item is harvested from. This can be a file system, a search service, catalog file or something similar. The example above defines a connection to an OpenSearch service on CREODIAS, with associated default query parameters and a format configuration.
The filter allows to filter elements within the harvester, when the resource does not provide a specific filter. This filter can be supplied using CQL2-JSON.
The postprocess can adjust the harvested results. In this example the harvested items are not complete, and additional metadata must be retrieved from an object storage.
The queue defines where harvested items will be pushed into. Usually this is a registration queue, where the registrar will pick up and start registration according to its configuration.
The harvester can either do one-off harvests via the CLI or listen on a redis queue to run consecutive harvests whenever a harvesting request is received on that queue.
"},{"location":"eoepca/data-access/#one-off-harvests-via-the-cli","title":"One-off harvests via the CLI","text":"In order to start a harvest from the CLI, the operator first needs to connect to the kubernetes pod of the harvester. Within that pod, the harvest can be executed like this\u2026
python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\nThis will invoke the Creodias-Opensearch harvester with default arguments. When some values are to be overridden, the \u2013values switch can be used to pass override values. These values must be a JSON string. The following example adjusts the begin and end times of the query parameters\u2026
python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch --values '{\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}'\nThe harvester pod runs a service listening on a redis queue. When a message is read from the queue, it will be read as a JSON string, expecting an object with at least a name property. Optionally, it can also have a values property, working in the same way as with CLI --values.
To send a harvesting request via the redis queue, it is necessary to connect to the redis pod and execute the redis-cli there. Then the following command can be used to achieve the same result as above with CLI harvesting\u2026
redis-cli LPUSH '{\"name\": \"Creodias-Opensearch\", \"values\": {\"resource\": {\"query\": {\"time\": {\"begin\": \"2020-09-10T00:00:00Z\", \"end\": \"2020-09-11T00:00:00Z\"}}}}}'\nThe harvester produces a continous stream of STAC Items which are sent down via the configured queue. It is possible that the harvested metadata is not sufficient to create a fully functional STAC Item. In this case the postprocess must transform this intermediate item to a valid STAC Item. In our example, the postprocessor looks up the Sentinel-2 product file referenced by the product identifier which is then accessed on the object storage. From the stored metadata files, the STAC Items to be sent is created.
"},{"location":"eoepca/data-access/#storage","title":"Storage","text":"Specification of PVCs and access to object storage.
"},{"location":"eoepca/data-access/#persistent-volume-claims","title":"Persistent Volume Claims","text":"The PVCs specified in the helm chart values must be created.
"},{"location":"eoepca/data-access/#pvc-for-database","title":"PVC for Database","text":"kind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n name: data-access-db\n namespace: rm\n labels:\n k8s-app: data-access\n name: data-access\nspec:\n storageClassName: standard\n accessModes:\n - ReadWriteMany\n resources:\n requests:\n storage: 100Gi\nkind: PersistentVolumeClaim\napiVersion: v1\nmetadata:\n name: data-access-redis\n namespace: rm\n labels:\n k8s-app: data-access\n name: data-access\nspec:\n storageClassName: standard\n accessModes:\n - ReadWriteMany\n resources:\n requests:\n storage: 1Gi\nThe helm chart values expect specification of object storage details for:
data: to access the EO data of the underlying infrastructurecache: a dedicated object storage bucket is used to support the cache function of the data access servicesSpecifies the details for the infrastructure object storage that provides direct access to the EO product files.
For example, the CREODIAS metadata catalogue provides references to product files in their eodata object storage - the access details for which are configured in the data access services:
global:\n storage:\n data:\n data:\n type: S3\n endpoint_url: http://data.cloudferro.com\n access_key_id: access\n secret_access_key: access\n region_name: RegionOne\n validate_bucket_name: false\nThe Data Access services maintain a cache, which relies on the usage of a dedicate object storage bucket for data persistence. This bucket must be created (manual step) and its access details configured in the data access services. Example based upon CREODIAS:
global:\n storage:\n cache:\n type: S3\n endpoint_url: \"https://cf2.cloudferro.com:8080/cache-bucket\"\n host: \"cf2.cloudferro.com:8080\"\n access_key_id: xxx\n secret_access_key: xxx\n region: RegionOne\n bucket: cache-bucket\n\u2026where xxx must be replaced with the bucket credentials.
As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the data-access service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install data-access-protection identity-gatekeeper -f data-access-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"rm\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the data-access - in particular the specific ingress requirements for the data-access backend services\u2026
Example data-access-protection-values.yaml\u2026
fullnameOverride: data-access-protection\nconfig:\n client-id: data-access\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: data-access.192-168-49-2.nip.io\n name: data-access-renderer\n port:\n number: 80\nsecrets:\n # Values for secret 'data-access-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n nginx.ingress.kubernetes.io/rewrite-target: /$1\n serverSnippets:\n custom: |-\n # Open access to renderer...\n location ~ ^/(ows.*|opensearch.*|coverages/metadata.*|admin.*) {\n proxy_pass http://data-access-renderer.rm.svc.cluster.local:80/$1;\n }\n # Open access to cache...\n location ~ ^/cache/(.*) {\n proxy_pass http://data-access-cache.rm.svc.cluster.local:80/$1;\n }\n # Open access to client...\n # Note that we use a negative lookahead to avoid matching '/.well-known/*' which\n # otherwise appears to interfere with the work of cert-manager/letsencrypt.\n location ~ ^/(?!\\.well-known)(.*) {\n proxy_pass http://data-access-client.rm.svc.cluster.local:80/$1;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: data-access above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=data-access \\\n --name=\"Data Access Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Data Access Gatekeeper\"\nAt deployment time the harvester helm values include configuration that populates a default harvester configuration, that is prepared in the file /config.yaml in the harvester pod.
The Data Access and Resource Catalogue services are configured to properly interpret harvested data via these values specified in the instantiation of the helm release. See section Data-layer Configuration.
The harvesting of data can be triggered (post deployment), in accordance with this default configuration, by connecting to the rm/harvester service and executing the command\u2026
python3 -m harvester harvest --config-file /config-run.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\nAd-hoc harvesting can be invoked by provision of a suitable config.yaml into the harvester pod, which can then be invoked as shown above for the default harvester configuration established at deploy time.
The helper script ./deploy/bin/harvest faciltates this\u2026
./deploy/bin/harvest <path-to-config-file>\nSee directory ./deploy/samples/harvester/ that contains some sample harvesting configuration files. For example\u2026
./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2019.09.10.yaml\nThe helper script ./deploy/bin/register-collection is provided to faciltate the registration of collections that are specfied in STAC Collection format.
./deploy/bin/register-collection <path-to-stac-collection-file>\nSee directory ./deploy/samples/collections/ that contains some same STAC Collection files. For example\u2026
./deploy/bin/register-collection ./deploy/samples/collections/S2MSI2A.json\nAdditional information regarding the Data Access can be found at:
This guide includes two approaches for Identity & Access Management:
Until now, our IAM solution has been based solely upon Gluu.
In the course of the project Keycloak has emerged as a preferred solution across EO platforms.
Thus, we have introduced an IAM approach based upon Keycloak, whilst retaining the Gluu-based approach for reference, which will be deprecated.
"},{"location":"eoepca/identity-service/","title":"Identity Service","text":"The Identity Service provides the platform Authorization Server for authenticated user identity and request authorization.
Identity Service is composed of:
The Identity Service is deployed via the identity-service helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values - the full set of available values can be tailored according the helm chart defaults, that can be found here\u2026
identity-service https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/values.yamlidentity-keycloak https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-keycloak/values.yamlidentity-postgres https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-postgres/values.yamlidentity-api https://github.com/EOEPCA/helm-charts/blob/main/charts/identity-service/charts/identity-api/values.yamlhelm install --version 1.0.97 --values identity-service-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n identity-service identity-service\nThe deployment must be configured for you environment. Some significant configuration values are elaborated here\u2026
"},{"location":"eoepca/identity-service/#identity-keycloak","title":"identity-keycloak","text":""},{"location":"eoepca/identity-service/#secrets","title":"Secrets","text":"Keycloak relies upon a secret identity-keycloak that provides\u2026
KEYCLOAK_ADMIN_PASSWORD - admin password for KeycloakKC_DB_PASSWORD - password for connecting with Postgres DB This should match the POSTGRES_PASSWORD setting for identity-postgres (see below)The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026
identity-keycloak:\n secrets:\n # Values for secret 'identity-keycloak'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n kcDbPassword: \"changeme\"\n keycloakAdminPassword: \"changeme\"\nThe details for ingress (reverse-proxy) to the Keycloak service - in particular the hostname and possible TLS - must be specified\u2026
identity-keycloak:\n ingress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - host: keycloak.192-168-49-2.nip.io\n paths:\n - path: /\n pathType: Prefix\n tls:\n - secretName: identity-keycloak-tls\n hosts:\n - keycloak.192-168-49-2.nip.io\nPostgres relies upon a secret identity-postgres that provides\u2026
POSTGRES_PASSWORD - superuser password for PostgreSQLPGPASSWORD - password used for client connections to the DBThe secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026
identity-postgres:\n secrets:\n # Values for secret 'identity-postgres'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n postgresPassword: \"changeme\"\n pgPassword: \"changeme\"\nIn order to persist data, Postgres requires a Persistent Volume Claim.
This can be specified as an existing volume claim - for example as described in the Persistence section.
identity-postgres:\n volumeClaim:\n name: eoepca-userman-pvc\nThe Identity API relies upon a secret identity-api that provides\u2026
ADMIN_PASSWORD Admin password for Keycloak This should match the KEYCLOAK_ADMIN_PASSWORD setting for identity-keycloak (see above)The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026
identity-api:\n secrets:\n # Values for secret 'identity-api'\n # Note - if ommitted, these can instead be set by creating the secret independently\n # e.g. as a SealedSecret via GitOps.\n adminPassword: \"changeme\"\nNote
It is also possible to set the value of ADMIN_PASSWORD directly as an environment variable. In this case it is necessary to set the secret as optional\u2026
identity-api:\n secrets:\n optional: true\nThe Identity API service can be configured via environment variables as follows\u2026
AUTH_SERVER_URL URL of the Keycloak Authorization Server. Can also be set via value configMap.authServerUrlADMIN_USERNAME Admin user for KeycloakREALM The Keycloak realmidentity-api:\n deployment:\n # Config values that can be passed via env vars\n extraEnv:\n - name: AUTH_SERVER_URL # see configMap.authServerUrl instead\n value: https://keycloak.192-168-49-2.nip.io\n - name: ADMIN_USERNAME\n value: admin\n - name: ADMIN_PASSWORD # see secrets.adminPassword instead\n value: changeme\n - name: REALM\n value: master\ngatekeeper relies upon a secret identity-api-protection that provides\u2026
PROXY_CLIENT_SECRET Password for the Keycloak client configured for use by this Gatekeeper instance - corresponding to config.client-id.PROXY_ENCRYPTION_KEY Encryption Key used by Gatekeeper.The secret can either be created directly within the cluster, or can be created by the helm chart via values\u2026
identity-api-gatekeeper:\n secrets:\n # Values for secret 'identity-api-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\nConfiguration of Gatekeeper via the file config.yaml that is mounted into the deployment\u2026
client-id ID of the Keycloak client to be used by this Gatekeeper instance.discovery-url Discovery URL of the Keycloak Authorization Servercookie-domain Domain in which this Gatekeeper instance creates cookies identity-api-gatekeeper:\n config:\n client-id: identity-api\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\nThe details for ingress (reverse-proxy) to the Gatekeeper service that protects the Identity API\u2026
identity-api-gatekeeper:\n targetService:\n host: identity-api.192-168-49-2.nip.io\n ingress:\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt\nThe Identity API is protected via an instance of Gatekeeper - which relies upon a Keycloak client having been created for authorization decision/enforcement flows between Gatekeeper and Keycloak.
As described in the \u2018create-client\u2019 section below, this can be achieved using the create-client helper script.
Note
At time of client creation, the Identity API is not yet protected with an ingress. Therefore, we use a port-forward to interface directly with the Identity API service.
$ kubectl -n um port-forward svc/identity-api \"9876\":http >/dev/null &\n$ portForwardPid=$!\n\n$ ./deploy/bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i http://localhost:9876 \\\n -r master \\\n -u admin \\\n -p changeme \\\n -c admin-cli \\\n --id=identity-api \\\n --name=\"Identity API Gatekeeper\" \\\n --secret=changeme \\\n --description=\"Client to be used by Identity API Gatekeeper\" \\\n --resource=\"admin\" --uris='/*' --scopes=view --users=\"admin\"\n\n$ kill -TERM $portForwardPid\ncreate-user Helper Script","text":"The Keycloak Admin UI can be used to create users interactively.
Alternatvely there is a helper script create-user that can be used.
The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe create-user helper script requires some command-line arguments\u2026
$ ./deploy/bin/create-user -h\n\nCreate a new user.\ncreate-user -h | -a {auth_server} -r {realm} -c {client} -u {admin-username} -p {admin-password} -U {new-username} -P {new-password}\n\nwhere:\n -h show help message\n -a authorization server url (default: http://keycloak.192-168-49-2.nip.io)\n -r realm within Keycloak (default: master)\n -u username used for authentication (default: admin)\n -p password used for authentication (default: changeme)\n -c client id of the bootstrap client used in the create request (default: admin-cli)\n -U name of the (new) user to create\n -P password for the (new) user to create\nThe Identity Service is capable of protecting resources using OpenID-connect/SAML clients, resources (URIs/scopes), policies (user based, role based, etc) and permissions (associations between policies and resources).
Creating and protecting resources can be done in multiple ways, as described in the following sections.
"},{"location":"eoepca/identity-service/#keycloak-admin-ui","title":"Keycloak Admin UI","text":"To create and protect resources using the keycloak User Interface (UI), do the following steps:
create-client Helper Script","text":"Alternatively, a script was developed to allow simultaneaously create a client, create resources and protect them.
The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe create-client helper script requires some command-line arguments\u2026
$ ./deploy/bin/create-client -h\n\nAdd a client with protected resources.\ncreate-client [-h] [-a] [-i] [-u] [-p] [-c] [-s] [-t | --token t] [-r] --id id [--name name] (--secret secret | --public) [--default] [--authenticated] [--resource name] [--uris u1,u2] [--scopes s1,s2] [--users u1,u2] [--roles r1,r2]\n\nwhere:\n -h show help message\n -a authorization server url - e.g. https://keycloak.192-168-49-2.nip.io\n -i identity-api server url - e.g. https://identity-api.192-168-49-2.nip.io\n -u username used for authentication\n -p password used for authentication\n -c client id (of the bootstrap client used in the create request)\n -s client secret (of the bootstrap client used in the create request)\n -t or --token access token used for authentication\n -r realm\n --id client id (of the created client)\n --name client name (of the created client)\n --secret client secret (of the created client)\n --public public client (no client secret)\n --default add default resource - /* authenticated\n --authenticated allow access to the resource only when authenticated\n --resource resource name\n --uris resource uris - separated by comma (,)\n --scopes resource scopes - separated by comma (,)\n --users user names with access to the resource - separated by comma (,)\n --roles role names with access to the resource - separated by comma (,)\nThe script interacts with Identity API and therefore requires admin authorization. It accepts basic authentication with username and password with -u and -p parameters, respectively - or a bearer access token with -t parameter.
To generate the access token needed to use the script, you can get it through the login in the eoepca portal, by accessing the cookies in the browser. See section EOEPCA Portal for details regarding deployment/configuration of the eoepca-portal.
Or you can generate an access token using postman oauth2.0, as described in the Postman document Requesting an OAuth 2.0 token.
Script execution examples:
With username/password
./deploy/bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r master \\\n -u admin \\\n -p changeme \\\n -c admin-cli \\\n --id=myservice-gatekeeper \\\n --name=\"MyService Gatekeeper\" \\\n --secret=changeme \\\n --description=\"Client to be used by MyService Gatekeeper\" \\\n --resource=\"Eric space\" --uris=/eric/* --users=eric \\\n --resource=\"Alice space\" --uris=/alice/* --users=alice \\\n --resource=\"Admin space\" --uris=/admin/* --roles=admin\nWith access token
./deploy/bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r master \\\n -t eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJXZWFIY2pscThPc1RUYjdlV0s5SjJTTDFBUDIyazZpajdlMGFlVHRNU2xRIn0.eyJleHAiOjE3MDAyNDM4MzgsImlhdCI6MTcwMDI0Mzc3OCwiYXV0aF90aW1lIjoxNzAwMjQxODYyLCJqdGkiOiI2MWI0ZGRhYy1mOWZjLTRmZjktOWQ4Zi01NWU1N2NlNmE5ODgiLCJpc3MiOiJodHRwczovL2lkZW50aXR5LmtleWNsb2FrLmRldmVsb3AuZW9lcGNhLm9yZy9yZWFsbXMvbWFzdGVyIiwiYXVkIjpbImFkZXMtcmVhbG0iLCJkZW1vLXJlYWxtIiwiZHVtbXktc2VydmljZS1yZWFsbSIsIm1hc3Rlci1yZWFsbSIsImFjY291bnQiLCJlb2VwY2EtcmVhbG0iXSwic3ViIjoiZTNkZTMyNGUtMGY0NS00MWUwLTk2YTctNTM1YzkxMTA1NTUyIiwidHlwIjoiQmVhcmVyIiwiYXpwIjoiZW9lcGNhLXBvcnRhbCIsIm5vbmNlIjoiMTIwMGJlNzAtZWI1Ni00Nzc2LThjODgtOWRiOWQxMDdiMGY2Iiwic2Vzc2lvbl9zdGF0ZSI6ImVmNGUwOTlmLTFmMDgtNDY3MC04ZmE2LTJiOGI3OGUwNWMzMSIsImFjciI6IjAiLCJhbGxvd2VkLW9yaWdpbnMiOlsiKiJdLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsiY3JlYXRlLXJlYWxtIiwiZGVmYXVsdC1yb2xlcy1tYXN0ZXIiLCJvZmZsaW5lX2FjY2VzcyIsImFkbWluIiwidW1hX2F1dGhvcml6YXRpb24iLCJ1c2VyIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWRlcy1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LWlkZW50aXR5LXByb3ZpZGVycyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwiZGVtby1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LXJlYWxtIiwidmlldy1pZGVudGl0eS1wcm92aWRlcnMiLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwiZHVtbXktc2VydmljZS1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LXJlYWxtIiwidmlldy1pZGVudGl0eS1wcm92aWRlcnMiLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfSwibWFzdGVyLXJlYWxtIjp7InJvbGVzIjpbInZpZXctaWRlbnRpdHktcHJvdmlkZXJzIiwidmlldy1yZWFsbSIsIm1hbmFnZS1pZGVudGl0eS1wcm92aWRlcnMiLCJpbXBlcnNvbmF0aW9uIiwiY3JlYXRlLWNsaWVudCIsIm1hbmFnZS11c2VycyIsInF1ZXJ5LXJlYWxtcyIsInZpZXctYXV0aG9yaXphdGlvbiIsInF1ZXJ5LWNsaWVudHMiLCJxdWVyeS11c2VycyIsIm1hbmFnZS1ldmVudHMiLCJtYW5hZ2UtcmVhbG0iLCJ2aWV3LWV2ZW50cyIsInZpZXctdXNlcnMiLCJ2aWV3LWNsaWVudHMiLCJtYW5hZ2UtYXV0aG9yaXphdGlvbiIsIm1hbmFnZS1jbGllbnRzIiwicXVlcnktZ3JvdXBzIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX0sImVvZXBjYS1yZWFsbSI6eyJyb2xlcyI6WyJ2aWV3LWlkZW50aXR5LXByb3ZpZGVycyIsInZpZXctcmVhbG0iLCJtYW5hZ2UtaWRlbnRpdHktcHJvdmlkZXJzIiwiaW1wZXJzb25hdGlvbiIsImNyZWF0ZS1jbGllbnQiLCJtYW5hZ2UtdXNlcnMiLCJxdWVyeS1yZWFsbXMiLCJ2aWV3LWF1dGhvcml6YXRpb24iLCJxdWVyeS1jbGllbnRzIiwicXVlcnktdXNlcnMiLCJtYW5hZ2UtZXZlbnRzIiwibWFuYWdlLXJlYWxtIiwidmlldy1ldmVudHMiLCJ2aWV3LXVzZXJzIiwidmlldy1jbGllbnRzIiwibWFuYWdlLWF1dGhvcml6YXRpb24iLCJtYW5hZ2UtY2xpZW50cyIsInF1ZXJ5LWdyb3VwcyJdfX0sInNjb3BlIjoib3BlbmlkIGVtYWlsIHByb2ZpbGUiLCJzaWQiOiJlZjRlMDk5Zi0xZjA4LTQ2NzAtOGZhNi0yYjhiNzhlMDVjMzEiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsInByZWZlcnJlZF91c2VybmFtZSI6ImFkbWluIn0.FK6DhVzpCRFmef2acD2Hmc149e1GTOCGz13dZA828crFbG8j4uhpkoNpiZqdyOPmDtMQ-OebNfjTAUaOt2sS1FmEIBgb9IddcpHKNJOquRjdzQNsX09bX8pFUq1haGwKh6_QmABNOBcT-kQNDSZO-aq7-8FoO9PYa0GWvBRcbcx0W_ngyb7xHglaZTElzcDPBcUTW6llVTTTFygn55smwdxTZ7-tEsMVGM5gNuHwJyLB51HI5KDWrwgUm1hqhhRzvcoutDEAB_HSEXGNNeF7fjP9Qx6q04b7fKOTtnIlXsu3oYW4va9y754llMSJ7w8U-y7yI6Tm2UdNMdYqju7hAA \\\n -c admin-cli \\\n --id=myservice-gatekeeper \\\n --name=\"MyService Gatekeeper\" \\\n --secret=changeme \\\n --description=\"Client to be used by MyService Gatekeeper\" \\\n --resource=\"Eric space\" --uris=/eric/* --users=eric \\\n --resource=\"Alice space\" --uris=/alice/* --users=alice \\\n --resource=\"Admin space\" --uris=/admin/* --roles=admin\nAlso, an API was developed to interact more easily with the Keycloak API, that allows client, resource, policies and permissions management.
The API documentation can be found in its Swagger UI at the service endpoint - https://identity-api.192-168-49-2.nip.io/docs.
The Identity API is best used in combination with the eoepca-portal test aide, which can be used to establish a login sesssion in the browser to the benefit of the Identity API swagger UI. See section EOEPCA Portal for details regarding deployment/configuration of the eoepca-portal.
By default the Access Token Lifespan is 1 minute. With the current ADES (zoo-dru) implementation this presents a problem - since the access_token that is provided to the process execute request will (most likely) have expired by the time the ADES attempts to use the access_token in its call to the Workspace API to register the processing outputs. The lifespan of the token must outlive the duration of the processing execution - which we must assume can take a long time.
To avoid this potential problem, the Keycloak Admin web console can be used to increase this token lifespan.
Thus, the following settings are recommended to be updated following deployment\u2026
1 day1 day1 dayAdditional information regarding the Identity Service can be found at:
The Login Service provides the platform Authorization Server for authenticated user identity and request authorization.
"},{"location":"eoepca/login-service/#helm-chart","title":"Helm Chart","text":"The Login Service is deployed via the login-service helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the login-service chart.
helm install --version 1.2.8 --values login-service-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n login-service login-service\nAt minimum, values for the following attributes should be specified:
auth.192-168-49-2.nip.io192.168.49.2namespace for the login-service componentslogin-service persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.letsencrypt-productionExample login-service-values.yaml\u2026
global:\n domain: auth.192-168-49-2.nip.io\n nginxIp: 192.168.49.2\n namespace: um\nvolumeClaim:\n name: eoepca-userman-pvc\n create: false\nconfig:\n domain: auth.192-168-49-2.nip.io\n adminPass: Chang3me!\n ldapPass: Chang3me!\n volumeClaim:\n name: eoepca-userman-pvc\nopendj:\n volumeClaim:\n name: eoepca-userman-pvc\n resources:\n requests:\n cpu: 100m\n memory: 300Mi\noxauth:\n volumeClaim:\n name: eoepca-userman-pvc\n resources:\n requests:\n cpu: 100m\n memory: 1000Mi\noxtrust:\n volumeClaim:\n name: eoepca-userman-pvc\n resources: \n requests:\n cpu: 100m\n memory: 1500Mi\noxpassport:\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\nnginx:\n ingress:\n annotations:\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - auth.192-168-49-2.nip.io\n tls:\n - hosts:\n - auth.192-168-49-2.nip.io\n secretName: login-service-tls\nNote
The resources: above have been limited for the benefit of a minikube deployment. For a production deployment the values should be tuned (upwards) according to operational needs.
The deployment of the Login Service has been designed, as far as possible, to automate the configuration. However, there remain some steps that must be performed manually after the scripted deployment has completed\u2026
UMA Resource LifetimeOperator userThe Login Service maintains a background service that \u2018cleans\u2019 UMA resources that are older than aa certain age - by default 30 days (2592000 secs). This lifetime does not fit the approach we are adopting, and so we must update this lifetime value to avoid the unexpected removal of UMA resources that would cause unexpected failures in policy enforcement.
https://auth.192-168-49-2.nip.io/ - and login as the admin userConfiguration -> JSON Configuration -> OxAuth ConfigurationumaResourceLifetimeumaResourceLifetime to 2147483647Save Configurationoxauth deployment\u2026 kubectl -n um rollout restart deploy/login-service-oxauth\nOperator user","text":"The default resource protection establishes policy in which \u2018operator\u2019 privilege is required for some services, such as the Workspace API. Thus, we need to configure a user with this privilege. For convenience we add this attribute to the built-in admin user - but alternatively you may choose to create a new user for this role.
https://auth.192-168-49-2.nip.io/ - and login as the admin userUsers -> Manage People and search for user adminadmin select Available User Claims -> gluuCustomPersonIs Operator and ensure the value is set TrueUpdate to confirmOnce the deployment has been completed successfully, the Login Service is accessed at the endpoint https://auth.192-168-49-2.nip.io/, configured by your domain - e.g. https://auth.192-168-49-2.nip.io/.
Login as the admin user with the credentials configured in the helm values - ref. adminPass / ldapPass.
Typical first actions to undertake through the Gluu web interface include creation of users and clients.
"},{"location":"eoepca/login-service/#additional-information","title":"Additional Information","text":"Additional information regarding the Login Service can be found at:
The Policy Decision Point (PDP) provides the platform policy database and associated service for access policy decision requests.
"},{"location":"eoepca/pdp/#helm-chart","title":"Helm Chart","text":"The PDP is deployed via the pdp-engine helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the pdp-engine chart.
helm install --version 1.1.12 --values pdp-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n pdp pdp-engine\nAt minimum, values for the following attributes should be specified:
auth.192-168-49-2.nip.io192.168.49.2pdp-engine persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.Example pdp-values.yaml\u2026
global:\n nginxIp: 192.168.49.2\n domain: auth.192-168-49-2.nip.io\nvolumeClaim:\n name: eoepca-userman-pvc\n create: false\nAdditional information regarding the PDP can be found at:
The EOEPCA building-blocks rely upon Kubernetes Persistent Volumes for their component persistence. Components integrate with the storage provided in the cluster by means of configurable Persistent Volume Claims and/or dynamic Storage Class that are specfied as values at time of deployment. Some components require storage of type ReadWriteMany - which, for a multi-node cluster, implies a network-based storage solution.
Note
Local CLuster Storage For the purposes of the Scripted Deployment, the default Storage Class included with the local Kubernetes distribution can be used for all storage concerns - e.g. standard for minikube which provides the ReadWriteMany persistence that is required by the ADES.
For the EOEPCA development deployment, an NFS server has been established to provide the persistence layer for ReadWriteMany storage.
The EOEPCA development deployment establishes the following pre-defined Persistent Volume Claims, to provide a simple storage architecture that is organised around the \u2018domain areas\u2019 into which the Reference Implementation is split.
resman) - persistentvolumeclaim/eoepca-resman-pvcproc) - persistentvolumeclaim/eoepca-proc-pvcuserman) - persistentvolumeclaim/eoepca-userman-pvcNOTE that this is offered only as an example thay suits the approach of the development team. Each building-block has configuration through which its persistence (PV/PVC) can be configured according the needs of the deployment.
The following Kubernetes yaml provides an example of provisioning such domain-specific PersistentVolumeClaims within the cluster - in this case using the minikube built-in storage-class standard for dynamic provisioning\u2026
---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: eoepca-proc-pvc\n namespace: proc\nspec:\n accessModes:\n - ReadWriteMany\n storageClassName: standard\n resources:\n requests:\n storage: 5Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: eoepca-resman-pvc\n namespace: rm\nspec:\n accessModes:\n - ReadWriteMany\n storageClassName: standard\n resources:\n requests:\n storage: 5Gi\n---\napiVersion: v1\nkind: PersistentVolumeClaim\nmetadata:\n name: eoepca-userman-pvc\n namespace: um\nspec:\n accessModes:\n - ReadWriteMany\n storageClassName: standard\n resources:\n requests:\n storage: 5Gi\nOnce established, these PersistentVolumeClaims are then referenced within the deployment configurations of the building-blocks.
"},{"location":"eoepca/persistence/#dynamic-readwritemany-storage-provisioning","title":"DynamicReadWriteMany Storage Provisioning","text":"In addition to the pre-defined PV/PVCs, the EOEPCA Reference Implementation also defines NFS-based storage classes for dynamic storage provisioning:
managed-nfs-storage With a Reclaim Policy of Delete.managed-nfs-storage-retain With a Reclaim Policy of Retain.The building-blocks simply reference the required Storage Class in their volume specifications, to receive a Persistent Volume Claim that is dynamically provisioned at deployment time.
This is acheived through the nfs-provisioner helm chart, with the following typical configurations\u2026
Reclaim Policy Delete\u2026
provisionerName: nfs-storage\nstorageClass:\n name: managed-nfs-storage\n create: true\n reclaimPolicy: Delete\n archiveOnDelete: false\n allowVolumeExpansion: true\nnfs:\n server: \"<your-nfs-ip-address-here>\"\n path: /data/dynamic # your NFS server path here\nReclaim Policy Retain\u2026
provisionerName: nfs-storage-retain\nstorageClass:\n name: managed-nfs-storage-retain\n create: true\n reclaimPolicy: Retain\n allowVolumeExpansion: true\nnfs:\n server: \"<your-nfs-ip-address-here>\"\n path: /data/dynamic # your NFS server path here\nClustered storage approaches offer an alternative to NFS. Clustered Storage provides a network-attached storage through a set of commodity hosts whose storage is aggregated to form a distributed file-system. Capacity is scaled by adding additional nodes or adding additional storage to the existing nodes. In the context of a multi-node Kubernetes cluster, then it is typical that the same commodity nodes provide both the cluster members and storage resources, i.e. the clustered storage is spread across the Kubernetes worker nodes.
Candidate clustered storage solutions include:
All things being equal, Longhorn is recommended as the best approach for Kubernetes clusters.
"},{"location":"eoepca/registration-api/","title":"Registration API","text":"The Registration API provides a REST API through which resources can be registered with both the Resource Catalogue and (as applicable) with the Data Access services.
"},{"location":"eoepca/registration-api/#helm-chart","title":"Helm Chart","text":"The Registration API is deployed via the rm-registration-api helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the rm-registration-api chart.
helm install --version 1.4.0 --values registration-api-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n registration-api rm-registration-api\nThe Registration API supports many values to configure the service - as described in the Values section of the chart README.
Typically, values for the following attributes may be specified:
identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.Example registration-api-values.yaml\u2026
fullnameOverride: registration-api\n\ningress:\n enabled: false\n hosts:\n - host: registration-api-open.192-168-49-2.nip.io\n paths: [\"/\"]\n tls:\n - hosts:\n - registration-api-open.192-168-49-2.nip.io\n secretName: registration-api-tls\n\n# some values for the workspace API\nworkspaceK8sNamespace: rm\nredisServiceName: \"data-access-redis-master\"\nAs described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the registration-api service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install registration-api-protection identity-gatekeeper -f registration-api-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"rm\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the registration-api - in particular the specific ingress requirements for the registration-api backend service\u2026
Example registration-api-protection-values.yaml\u2026
fullnameOverride: registration-api-protection\nconfig:\n client-id: registration-api\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: registration-api.192-168-49-2.nip.io\n name: registration-api\n port:\n number: 8080\nsecrets:\n # Values for secret 'registration-api-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n serverSnippets:\n custom: |-\n # Open access...\n location ~ ^/ {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: registration-api above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=registration-api \\\n --name=\"Registration API Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Registration API Gatekeeper\"\nAdditional information regarding the Registration API can be found at:
The Resource Catalogue provides a standards-based EO metadata catalogue that includes support for OGC CSW / API Records, STAC and OpenSearch.
"},{"location":"eoepca/resource-catalogue/#helm-chart","title":"Helm Chart","text":"The Resource Catalogue is deployed via the rm-resource-catalogue helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the rm-resource-catalogue chart.
helm install --version 1.4.0 --values resource-catalogue-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n resource-catalogue rm-resource-catalogue\nThe Resource Catalogue supports many values to configure the service - as described in the Values section of the chart README.
Typically, values for the following attributes may be specified:
identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.db.config.XXX.Example resource-catalogue-values.yaml\u2026
global:\n namespace: rm\n# For protected access disable this ingress, and rely upon the identity-gatekeeper\n# for ingress with protection.\ningress:\n # Enabled for unprotected 'open' access to the resource-catalogue.\n enabled: true\n name: resource-catalogue\n host: resource-catalogue.192-168-49-2.nip.io\n tls_host: resource-catalogue.192-168-49-2.nip.io\n tls_secret_name: resource-catalogue-tls\n annotations:\n cert-manager.io/cluster-issuer: letsencrypt-production\ndb:\n volume_storage_type: standard\n # config:\n # enabled: true\n # shared_buffers: 2GB\n # effective_cache_size: 6GB\n # maintenance_work_mem: 512MB\n # checkpoint_completion_target: 0.9\n # wal_buffers: 16MB\n # default_statistics_target: 100\n # random_page_cost: 4\n # work_mem: 4MB\n # cpu_tuple_cost: 0.4\npycsw:\n config:\n server:\n url: https://resource-catalogue.192-168-49-2.nip.io/\n manager:\n transactions: \"true\"\n allowed_ips: \"*\"\nNote
The above example values enable transactions (write-access) to the catalogue from any IP address. This is convenient for testing/demonstration of the capability, but should be disbaled or restricted for operational deployments.
"},{"location":"eoepca/resource-catalogue/#protection","title":"Protection","text":"As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the resource-catalogue service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install resource-catalogue-protection identity-gatekeeper -f resource-catalogue-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"rm\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the resource-catalogue - in particular the specific ingress requirements for the resource-catalogue-service\u2026
Example resource-catalogue-protection-values.yaml\u2026
fullnameOverride: resource-catalogue-protection\nconfig:\n client-id: resource-catalogue\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: resource-catalogue.192-168-49-2.nip.io\n name: resource-catalogue-service\n port:\n number: 80\nsecrets:\n # Values for secret 'resource-catalogue-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n serverSnippets:\n custom: |-\n # Open access...\n location ~ ^/ {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: resource-catalogue above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=resource-catalogue \\\n --name=\"Resource Catalogue Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Resource Catalogue Gatekeeper\"\nThe Resource Catalogue is initially populated during the initialisation of the Data Access service. See section Data-layer Configuration.
The Resource Catalogue is accessed at the endpoint https://resource-catalogue.192-168-49-2.nip.io/, configured by your domain - e.g. https://resource-catalogue.192-168-49-2.nip.io/.
As described in the pycsw documentation, ISO XML records can be loaded into the resource-catalogue using the pycsw-admin.py admin utility\u2026
pycsw-admin.py load_records -c /path/to/cfg -p /path/to/records\nThe /path/to/records can either be a single metadata file, or a directory containing multiple metadata files.
This is most easily achieved via connection to the pycsw pod, which includes the pycsw-admin.py utility and the pycsw configuration file at /etc/pycsw/pycsw.cfg\u2026
kubectl -n rm cp \"<metadata-file-or-directory>\" \"<pycsw-pod-name>\":/tmp/metadata\nkubectl -n rm exec -i \"<pycsw-pod-name>\" -- pycsw-admin.py load-records -c /etc/pycsw/pycsw.cfg -p /tmp/metadata\nThe name of the pycsw pod can be obtained using kubectl\u2026
kubectl -n rm get pod --selector='io.kompose.service=pycsw' --output=jsonpath={.items[0].metadata.name}\nTo facilitate the loading of records via the pycsw pod, a helper script load-records has been provided in the git repository that hosts this document\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\n./deploy/bin/load-records \"<metadata-file-or-directory>\"\nThe helper script identifies the pycsw pod, copies the metadata files to the pod, and runs pycsw-admin.py load-records within the pod to load the records.
Additional information regarding the Resource Catalogue can be found at:
EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes:
Building Blocks that act as a Resource Server are individually protected by a Policy Enforcement Point (PEP). The PEP enforces the authorization decision in collaboration with the Login Service and Policy Decision Point (PDP).
The PEP expects to interface to a client (user agent, e.g. browser) using User Managed Access (UMA) flows. It is not typical for a client to support UMA flows, and so the PEP can be deployed with a companion UMA User Agent component that interfaces between the client and the PEP, and performs the UMA Flow on behalf of the client.
The Resource Guard is a \u2018convenience\u2019 component that deploys the PEP & UMA User Agent as a cooperating pair.
The Resource Guard \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, the Resource Guard deploys with an Ingress specification that:
auth_request module to defer access authorization to the uma-user-agent serviceThe Resource Guard is deployed via the resource-guard helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the resource-guard chart.
It is expected to deploy multiple instances of the Resource Guard chart, one for each Resource Server to be protected.
helm install --version 1.3.1 --values myservice-guard-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n myservice-guard resource-guard\nThe helm chart is deployed with values that are passed through to the subcharts for the pep-engine and uma-user-agent. Typical values to be specified include:
auth.192-168-49-2.nip.io192.168.49.2pep-engine persistence, e.g. myservice-pep-pvcletsencrypt-productionSecret that contains the client credentials used by the uma-user-agent to interface with the Login Service. See section Client Secret belowExample myservice-guard-values.yaml\u2026
#---------------------------------------------------------------------------\n# Global values\n#---------------------------------------------------------------------------\nglobal:\n context: myservice\n domain: 192-168-49-2.nip.io\n nginxIp: 192.168.49.2\n certManager:\n clusterIssuer: letsencrypt-production\n#---------------------------------------------------------------------------\n# PEP values\n#---------------------------------------------------------------------------\npep-engine:\n configMap:\n asHostname: auth\n pdpHostname: auth\n customDefaultResources:\n - name: \"Eric's space\"\n description: \"Protected Access for eric to his space in myservice\"\n resource_uri: \"/ericspace\"\n scopes: []\n default_owner: \"d3688daa-385d-45b0-8e04-2062e3e2cd86\"\n volumeClaim:\n name: myservice-pep-pvc\n create: false\n#---------------------------------------------------------------------------\n# UMA User Agent values\n#---------------------------------------------------------------------------\numa-user-agent:\n nginxIntegration:\n enabled: true\n hosts:\n - host: myservice\n paths:\n - path: /(.*)\n service:\n name: myservice\n port: 80\n - path: /(doc.*)\n service:\n name: myservice-docs\n port: 80\n annotations:\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n nginx.ingress.kubernetes.io/rewrite-target: /$1\n client:\n credentialsSecretName: \"myservice-agent\"\n logging:\n level: \"debug\"\n unauthorizedResponse: 'Bearer realm=\"https://portal.192-168-49-2.nip.io/oidc/authenticate/\"'\n#---------------------------------------------------------------------------\n# END values\n#---------------------------------------------------------------------------\nThe uma-user-agent requires Client Credentials for its interactions with the login-service. The uma-user-agent expects to read these credentials from the file client.yaml, in the form\u2026
client-id: <my-client-id>\nclient-secret: <my-secret>\nTo obtain the Client Credentials required by the uma-user-agent it is necessary to register a client with the login-service, or use the credentials for an existing client.
A helper script is provided to register a basic client and obtain the required credentials. The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe register-client helper script requires some command-line arguments\u2026
Usage:\n register_client <authorization-server-hostname> <client-name> [<redirect-uri> [<logout-uri>]]\nFor example\u2026
./deploy/bin/register-client auth.192-168-49-2.nip.io myclient\n\nINFO: Preparing docker image... [done]\nClient successfully registered.\nMake a note of the credentials:\nclient-id: a98ba66e-e876-46e1-8619-5e130a38d1a4\nclient-secret: 73914cfc-c7dd-4b54-8807-ce17c3645558\nOr to register OIDC redirect URLs\u2026
./deploy/bin/register-client auth.192-168-49-2.nip.io myclient https://portal.192-168-49-2.nip.io/oidc/callback/ https://portal.192-168-49-2.nip.io/logout\nThe script writes the \u2018client credentials\u2019 to stdout - in the expected YAML configuration file format - which can be redirected to file\u2026
./deploy/bin/register-client auth.192-168-49-2.nip.io myclient | tee client.yaml\nclient.yaml. NOTE that the register-client helper relies upon docker to build and run the script.
The client.yaml configuration file is made available via a Kubernetes Secret\u2026
kubectl -n myservice-ns create secret generic myservice-agent \\\n --from-file=client.yaml \\\n --dry-run=client -o yaml \\\n > myservice-agent-secret.yaml\napiVersion: v1\nkind: Secret\nmetadata:\n name: myservice-agent\n namespace: myservice-ns\ndata:\n client.yaml: Y2xpZW50LWlkOiBhOThiYTY2ZS1lODc2LTQ2ZTEtODYxOS01ZTEzMGEzOGQxYTQKY2xpZW50LXNlY3JldDogNzM5MTRjZmMtYzdkZC00YjU0LTg4MDctY2UxN2MzNjQ1NTU4\nThe resource-guard deployment is configured with the name of the Secret through the helm chart value client.credentialsSecretName.
As described in the README for the Resource Guard, it is necessary for a request to a protected resource to provide the User ID Token in the request header.
"},{"location":"eoepca/resource-protection-gluu/#obtaining-the-user-id-token","title":"Obtaining the User ID Token","text":"In the simple case of a user with username/password held within the Login Service, the User ID Token can be obtained as follows:
curl --location --request POST 'https://auth.192-168-49-2.nip.io/oxauth/restv1/token' \\\n--header 'Cache-Control: no-cache' \\\n--header 'Content-Type: application/x-www-form-urlencoded' \\\n--data-urlencode 'scope=openid user_name is_operator' \\\n--data-urlencode 'grant_type=password' \\\n--data-urlencode 'username=<username>' \\\n--data-urlencode 'password=<password>' \\\n--data-urlencode 'client_id=<client-id>' \\\n--data-urlencode 'client_secret=<client-password>'\nThe User ID Token is included in the id_token field of the json response.
Alternatively, OAuth/OIDC flows can be followed to authenticate via external identity providers.
"},{"location":"eoepca/resource-protection-gluu/#user-id-token-in-http-requests","title":"User ID Token in HTTP requests","text":"The Resource Guard protection supports presentation of the User ID Token via the following HTTP request headers (in order of priority)\u2026
Authorization header as a bearer token - in the form: Authorization: Bearer <token>X-User-Id headerCookie: auth_user_id=<token> Note that the name of the cookie is configurable
Additional information regarding the Resource Guard can be found at:
EOEPCA defines Building Blocks within a micro-service architecture. The services are subject to protection within an Identity and Access Management (IAM) approach that includes:
Building Blocks that act as a Resource Server are individually protected by a dedicated Gatekeeper instance that enforces the authorization decision in collaboration with the Identity Service (Keycloak).
Gatekeeper \u2018inserts itself\u2019 into the request path of the target Resource Server using the auth_request facility offered by Nginx. Thus, Gatekeeper deploys with an Ingress specification that:
auth_request module to defer access authorization to the gatekeeper serviceEach Gatekeeper is deployed via the identity-gatekeeper helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values - the full set of available values can be seen at https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/values.yaml.
It is expected to deploy multiple instances of the Gatekeeper chart, one for each Resource Server to be protected.
helm install --version 1.0.10 --values myservice-gatekeeper-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n myservice-protection identity-gatekeeper\nThe helm chart is deployed with values that customise the service for the specific needs of the resource-server under protection and the deployment target platform. Typical values to be specified include:
keycloak.192-168-49-2.nip.ioletsencrypt-productionExample myservice-protection-values.yaml\u2026
nameOverride: myservice-protection\nconfig:\n client-id: myservice\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: myservice.192-168-49-2.nip.io\n name: myservice\n port:\n number: 80\nsecrets:\n # Values for secret 'myservice-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n serverSnippets:\n custom: |-\n # Open access to some endpoints, including Swagger UI\n location ~ ^/(docs|openapi.json|probe) {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nGatekeeper requires Client Credentials for its interactions with the Keycloak identity-service. These credentials must be supplied by the secret named <myservice>-protection. The secret can be created directly by the helm chart - via the values secrets.clientSecret and secrets.encryptionKey - or perhaps more securely the secret can be created independently (e.g. via a SealedSecret).
The Keycloak client can be created directly in the Keycloak admin console - e.g. via https://keycloak.192-168-49-2.nip.io/admin.
As an aide there is a helper script create-client. The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe create-client helper script requires some command-line arguments\u2026
$ ./deploy/bin/create-client -h\n\nAdd a client with protected resources.\ncreate-client [-h] [-a] [-i] [-u] [-p] [-c] [-s] [-t | --token t] [-r] --id id [--name name] (--secret secret | --public) [--default] [--authenticated] [--resource name] [--uris u1,u2] [--scopes s1,s2] [--users u1,u2] [--roles r1,r2]\n\nwhere:\n -h show help message\n -a authorization server url - e.g. https://keycloak.192-168-49-2.nip.io\n -i identity-api server url - e.g. https://identity-api.192-168-49-2.nip.io\n -u username used for authentication\n -p password used for authentication\n -c client id (of the bootstrap client used in the create request)\n -s client secret (of the bootstrap client used in the create request)\n -t or --token access token used for authentication\n -r realm\n --id client id (of the created client)\n --name client name (of the created client)\n --secret client secret (of the created client)\n --public public client (no client secret)\n --default add default resource - /* authenticated\n --authenticated allow access to the resource only when authenticated\n --resource resource name\n --uris resource uris - separated by comma (,)\n --scopes resource scopes - separated by comma (,)\n --users user names with access to the resource - separated by comma (,)\n --roles role names with access to the resource - separated by comma (,)\nFor example\u2026
./deploy/bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=myservice \\\n --name=\"Gatekeeper for myservice\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Gatekeeper for myservice\" \\\n --resource=\"eric\" --uris='/eric/*' --scopes=view --users=\"eric\" \\\n --resource=\"bob\" --uris='/bob/*' --scopes=view --users=\"bob\" \\\n --resource=\"alice\" --uris='/alice/*' --scopes=view --users=\"alice\"\nRequests to resource server endpoints that are protected by Gatekeeper must carry an Access Token that has been obtained on behalf of the requesting user. The access_token is carried in the request header\u2026
Authorization: Bearer <access_token>\nThe Access Token for a given user can be obtained with a call to the token endpoint of the Keycloak Identity Service - supplying the credentials for the user and the pre-registered client\u2026
curl -L -X POST 'https://keycloak.192-168-49-2.nip.io/realms/master/protocol/openid-connect/token' \\\n -H 'Cache-Control: no-cache' \\\n -H 'Content-Type: application/x-www-form-urlencoded' \\\n --data-urlencode 'scope=openid profile email' \\\n --data-urlencode 'grant_type=password' \\\n --data-urlencode 'username=<username>' \\\n --data-urlencode 'password=<password>' \\\n --data-urlencode 'client_id=admin-cli'\nA json response is returned, in which the field access_token provides the Access Token for the specified <username>.
Additional information regarding the Gatekeeper can be found at:
The User Profile represents the user\u2019s \u2018account\u2019 within the platform.
"},{"location":"eoepca/user-profile/#helm-chart","title":"Helm Chart","text":"The User Profile is deployed via the user-profile helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the user-profile chart.
helm install --version 1.1.12 --values user-profile-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n user-profile user-profile\nAt minimum, values for the following attributes should be specified:
auth.192-168-49-2.nip.io192.168.49.2user-profile persistence, e.g. eoepca-userman-pvc The boolen value volumeClaim.create can be used for the PVC to be created by the helm release. This creates a volume of type host-path and, hence, is only useful for single-node development usage.Example user-profile-values.yaml\u2026
global:\n domain: auth.192-168-49-2.nip.io\n nginxIp: 192.168.49.2\nvolumeClaim:\n name: eoepca-userman-pvc\n create: false\nThe User Profile is accessed through the /web_ui path of the Login Service, e.g. http://auth.kube.guide.eoepca.org/web_ui.
Additional information regarding the User Profile can be found at:
The Workspace provides protected user resource management that includes dedicated storage and services for resource discovery and access.
"},{"location":"eoepca/workspace/#workspace-api","title":"Workspace API","text":"The Workspace API provides a REST service through which user workspaces can be created, interrogated, managed and deleted.
"},{"location":"eoepca/workspace/#helm-chart","title":"Helm Chart","text":"The Workspace API is deployed via the rm-workspace-api helm chart from the EOEPCA Helm Chart Repository.
The chart is configured via values that are fully documented in the README for the um-workspace-api chart.
helm install --version 1.4.2 --values workspace-api-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n workspace-api rm-workspace-api\nAt minimum, values for the following attributes should be specified:
identity-gatekeeper component - ref. Resource Protection. Otherwise the ingress will be handled by the identity-gatekeeper - use ingress.enabled: false.Example workspace-api-values.yaml\u2026
fullnameOverride: workspace-api\ningress:\n enabled: true\n annotations:\n cert-manager.io/cluster-issuer: letsencrypt-production\n kubernetes.io/ingress.class: nginx\n nginx.ingress.kubernetes.io/enable-cors: \"true\"\n nginx.ingress.kubernetes.io/proxy-read-timeout: \"600\"\n hosts:\n - host: workspace-api-open.192-168-49-2.nip.io\n paths: [\"/\"]\n tls:\n - hosts:\n - workspace-api-open.192-168-49-2.nip.io\n secretName: workspace-api-open-tls\nfluxHelmOperator:\n enabled: true\nprefixForName: \"ws\"\nworkspaceSecretName: \"bucket\"\nnamespaceForBucketResource: \"rm\"\ns3Endpoint: \"https://minio.192-168-49-2.nip.io\"\ns3Region: \"RegionOne\"\nharborUrl: \"https://harbor.192-168-49-2.nip.io\"\nharborUsername: \"admin\"\nharborPasswordSecretName: \"harbor\"\nworkspaceChartsConfigMap: \"workspace-charts\"\nbucketEndpointUrl: \"http://minio-bucket-api:8080/bucket\"\nkeycloakIntegration:\n enabled: true\n keycloakUrl: \"https://keycloak.192-168-49-2.nip.io\"\n realm: \"master\"\n identityApiUrl: \"https://identity-api.192-168-49-2.nip.io\"\n workspaceApiIamClientId: \"workspace-api\"\n defaultIamClientSecret: \"changeme\"\nNote
harborXXX values above.See section Container Registry.admin user must be created as described in the section Harbor admin Password.keycloakIntegration allows the Workspace API to apply protecion (for the specified workspace owner) to the services within newly created workspaces.bucketEndpointUrl. See section Bucket Creation Webhook for details.admin Password","text":"The password for the harbor admin user is provided to the workspace-api via the specified secret - defined as harbor above.
This secret must be created - for example as follows\u2026
kubectl -n rm create secret generic harbor \\\n --from-literal=HARBOR_ADMIN_PASSWORD=\"changeme\"\nWorkspaces are created by instantiating the rm-user-workspace helm chart for each user/group. The Workspace API uses Flux CD as a helper to manage these subordinate helm charts - via flux resources of type HelmRelease. Thus, it is necessary to deploy within the cluster the aspects of flux that support this helm chart management - namely the flux helm-controller, source-controller and the Kubernetes Custom Resource Definitions (CRD) for HelmRelease and HelmRepository.
In case you are not already using flux within your clsuter, then the Workspace API helm chart can be configured to deploy the required flux components\u2026
fluxHelmOperator:\n enabled: true # true = install flux for me, false = I already have flux\nThe Workspace API instantiates for each user a set of services, including a Resource Catalogue and Data Access services. These user services are instantiated via helm using templates. The templates are provided to the Workspace API in a ConfigMap that is, by default, named workspace-charts. Each file in the config-map is expected to be of kind HelmRelease. During creation of a new workspace, the Worksapce API applies each file to the cluster in the namespace of the newly created namespace.
The default ConfigMap that is included with this guide contains the following templates for instantiation of user-specific components:
template-hr-data-access.yamltemplate-hr-resource-catalogue.yamltemplate-hr-resource-protection.yamlEach of these templates is expressed as a flux HelmRelease object that describes the helm chart and values required to deploy the service.
In addition, ConfigMap templates are included that provide specific details required to access the user-scoped workspace resources, including access to S3 object storage and container registry:
template-cm-aws-config.yamltemplate-cm-aws-credentials.yamltemplate-cm-docker-config.yamlThese ConfigMaps are designed to be mounted as files into the runtime environments of other components for user workspace integration. In particular the Application Hub makes use of this approach to provide a user experience that integrates with the user\u2019s workspace resources.
"},{"location":"eoepca/workspace/#templates-configmap","title":"Templates ConfigMap","text":"The templates are provided to the Workspace API as a ConfigMap in the namespace of the Workspace API deployment\u2026
(for full examples see https://github.com/EOEPCA/deployment-guide/tree/eoepca-v1.4/deploy/eoepca/workspace-templates)
apiVersion: v1\nkind: ConfigMap\nmetadata:\n name: workspace-charts\ndata:\n template-hr-resource-catalogue.yaml: |\n apiVersion: helm.toolkit.fluxcd.io/v2beta1\n kind: HelmRelease\n metadata:\n name: rm-resource-catalogue\n spec:\n interval: 5m\n chart:\n spec:\n chart: rm-resource-catalogue\n version: 1.3.1\n sourceRef:\n kind: HelmRepository\n name: eoepca\n namespace: rm\n values:\n ...\n template-hr-data-access.yaml: |\n apiVersion: helm.toolkit.fluxcd.io/v2beta1\n kind: HelmRelease\n metadata:\n name: vs\n spec:\n interval: 5m\n chart:\n spec:\n chart: data-access\n version: 1.3.1\n sourceRef:\n kind: HelmRepository\n name: eoepca\n namespace: rm\n values:\n ...\n template-hr-resource-protection.yaml: |\n apiVersion: helm.toolkit.fluxcd.io/v2beta1\n kind: HelmRelease\n metadata:\n name: resource-protection\n spec:\n interval: 5m\n chart:\n spec:\n chart: identity-gatekeeper\n version: 1.0.11\n sourceRef:\n kind: HelmRepository\n name: eoepca\n namespace: ${NAMESPACE}\n values:\n ...\n template-cm-aws-config.yaml: |\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: aws-config\n data:\n aws-config: |\n [default]\n region = {{ s3_region }}\n s3 =\n endpoint_url = {{ s3_endpoint_url }}\n s3api =\n endpoint_url = {{ s3_endpoint_url }}\n [plugins]\n endpoint = awscli_plugin_endpoint\n template-cm-aws-credentials.yaml: |\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: aws-credentials\n data:\n aws-credentials: |\n [default]\n aws_access_key_id = {{ access_key_id }}\n aws_secret_access_key = {{ secret_access_key }}\n template-cm-docker-config.yaml: |\n apiVersion: v1\n kind: ConfigMap\n metadata:\n name: docker-config\n data:\n docker-config: |\n {\n \"auths\": {\n \"{{ container_registry_host }}\": {\n \"auth\": \"{{ container_registry_credentials }}\"\n }\n }\nNotice the use of workspace template parameters {{ param_name }} that are used at workspace creation time to contextualise the workspace for the owning user. See section Workspace Template Parameters for more information.
As can be seen above, the HelmRelease templates rely upon objects of type HelmRepository that define the hosting helm chart repository. Thus, in support of the workspace templates, appropriate HelmRepository objects must be provisioned within the cluster. For example, in support of the above examples that rely upon the EOEPCA Helm Chart Repository\u2026
apiVersion: source.toolkit.fluxcd.io/v1beta1\nkind: HelmRepository\nmetadata:\n name: eoepca\n namespace: rm\nspec:\n interval: 2m\n url: https://eoepca.github.io/helm-charts/\nThe Workspace API uses the jinja2 templating engine when applying the resources for a user workspace. The current parameters are currently supported:
workspace_name The name of the workspace - {{ workspace_name }} used to ensure unique naming of cluster resources, such as service ingressdefault_owner The uuid of the owner of the workspace - {{ default_owner }} used to initialise the workspace protection{{ s3_endpoint_url }}{{ s3_region }}{{ access_key_id }}{{ secret_access_key }}{{ container_registry_host }}{{ container_registry_credentials }}As described in section Resource Protection (Keycloak), the identity-gatekeeper component can be inserted into the request path of the workspace-api service to provide access authorization decisions
Gatekeeper is deployed using its helm chart\u2026
helm install workspace-api-protection identity-gatekeeper -f workspace-api-protection-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"rm\" --create-namespace \\\n --version 1.0.11\nThe identity-gatekeeper must be configured with the values applicable to the workspace-api - in particular the specific ingress requirements for the workspace-api backend service\u2026
Example workspace-api-protection-values.yaml\u2026
fullnameOverride: workspace-api-protection\nconfig:\n client-id: workspace-api\n discovery-url: https://keycloak.192-168-49-2.nip.io/realms/master\n cookie-domain: 192-168-49-2.nip.io\ntargetService:\n host: workspace-api.192-168-49-2.nip.io\n name: workspace-api\n port:\n number: 8080\nsecrets:\n # Values for secret 'workspace-api-protection'\n # Note - if ommitted, these can instead be set by creating the secret independently.\n clientSecret: \"changeme\"\n encryptionKey: \"changemechangeme\"\ningress:\n enabled: true\n className: nginx\n annotations:\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n serverSnippets:\n custom: |-\n # Open access to some endpoints, including Swagger UI\n location ~ ^/(docs|openapi.json|probe) {\n proxy_pass {{ include \"identity-gatekeeper.targetUrl\" . }}$request_uri;\n }\nThe Gatekeeper instance relies upon an associated client configured within Keycloak - ref. client-id: workspace-api above.
This can be created with the create-client helper script, as descirbed in section Client Registration.
For example, with path protection for the admin user\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=\"workspace-api\" \\\n --name=\"Workspace API Gatekeeper\" \\\n --secret=\"changeme\" \\\n --description=\"Client to be used by Workspace API Gatekeeper\" \\\n --resource=\"admin\" --uris='/*' --scopes=view --users=\"admin\"\nThe Workspace API provides a REST interface that is accessed at the endpoint https://workspace-api.192-168-49-2.nip.io/. See the Swagger Docs - /docs.
Note
If the Workspace API has been protected (via Gatekeeper with Keycloak), then requests must be supported by an access_token carried in the HTTP header Authorozation: Bearer <token>. This diminishes the utility of the swagger UI.
Additional information regarding the Workspace API can be found at:
With helm chart version 1.3.1 of the workspace-api the approach to bucket creation has been re-architected to use a webhook approach.
During workspace creation the workspace-api needs to create an object storage bucket for the user. The method by which the bucket is created is a function of the hosting infrastructure object storage layer - i.e. there is no \u2018common\u2019 approach for the workspace-api to perform the bucket creation.
In order to allow this bucket creation step to be customised by the platform integrator, the workspace-api is configured with a webhook endpoint that is invoked to effect the bucket creation on behalf of the workspace-api.
The workspace-api is configured by the following value in its helm chart deployment, e.g\u2026
bucketEndpointUrl: \"http://my-bucket-webhook:8080/bucket\"\nThe webhook service must implement the following REST interface\u2026
method: POST content-type: application/json data:
{\n bucketName: str\n secretName: str\n secretNamespace: str\n}\nThere are two possible approaches to implement this request, distinguished by the response code\u2026
200 The bucket is created and the credentials are included in the response body. In this case only the supplied bucketName is relevant to fulfil the request.201 The bucket will be created (asychronously) and the outcome is provided by the webhook via a Kubernetes secret, as per the secretName and secretNamespace request parameters200 Response
In case 200 response, the response body should communicate the credentials with an application/json content-type in the form\u2026
{\n \"bucketname\": \"...\",\n \"access_key\": \"...\",\n \"access_secret\": \"....\",\n \"projectid\": \"...\",\n}\nIn this case the workspace-api will create the appropriate bucket secret using the returned credentials.
201 Response
In case 201 response, the secret should be created in the form\u2026
data:\n bucketname: \"...\"\n access: \"...\"\n secret: \"...\"\n projectid: \"...\"\nIn this case the workspace-api will wait for the (asynchronous) creation of the specified secret before continuing with the workspace creation.
Overall Outcome
In both cases the ultimate outcome is the creation of the bucket in the back-end object storage, and the creation of a Kubernetes secret that maintains the credentials for access to the bucket. The existence of the bucket secret is prerequisite to the continuation of the user workspace creation.
"},{"location":"eoepca/workspace/#minio-bucket-api-webhook","title":"Minio Bucket API (Webhook)","text":"The Minio Bucket API provides an implementation of a Bucket Creation Webhook for a Minio S3 Object Storage backend. This is used as the default in this guide - but should be replaced for a production deployment with an appropriate webhook to integrate with the object storage solution of the deployment environment.
"},{"location":"eoepca/workspace/#helm-chart_1","title":"Helm Chart","text":"The Minio Bucket API is deployed via the rm-minio-bucket-api helm chart from the EOEPCA Helm Chart Repository - ref. Helm Chart for the Minio Bucket API.
helm install --version 0.0.4 --values minio-bucket-api-values.yaml \\\n --repo https://eoepca.github.io/helm-charts \\\n rm-minio-bucket-api rm-minio-bucket-api\nAt minimum, values for the following attributes should be specified:
minIOServerEndpointaccessCredentials.secretName (ref. Minio Credentials Secret)Example minio-bucket-api-values.yaml\u2026
fullnameOverride: minio-bucket-api\nminIOServerEndpoint: https://minio.192-168-49-2.nip.io\naccessCredentials:\n secretName: minio-auth\nAdditional information regarding the Minio Bucket API can be found at:
A deployment wrapper script has been prepared for an \u2018Application Hub\u2019 deployment - that provides the Application Hub integrated with the Identity Service (Keycloak) via OIDC for user authentication.
The script deploy/apphub/apphub achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/apphub/apphub-options.
The Application Hub deployment applies the following configuration:
Deployment is initiated by invoking the script\u2026
./deploy/apphub/apphub\nThe Identity Service (Keycloak) is accessed at the following endpoints\u2026
The Application Hub is accessed at the endpoint - http://applicationhub.192-168-49-2.nip.io/.
"},{"location":"quickstart/application-hub-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/application-hub-deployment/#application-hub-notes","title":"Application Hub Notes","text":""},{"location":"quickstart/application-hub-deployment/#login","title":"Login","text":"Authentication is made via the Sign in with EOEPCA button on the service home page - which redirects to Keycloak for authentication.
With the out-of-the-box configuration user eric or bob should be used with default password changeme. Users eric and bob are currently predefined within the helm chart as admin users - see https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/files/hub/jupyter_config.py#L171.
Once logged in, the service list is presented for spawning of applications. Note that this list of applications is currently defined within the helm chart - see https://github.com/EOEPCA/helm-charts/blob/main/charts/application-hub/files/hub/config.yml.
From the list, a service is selected and the Start button initiates spawning.
For a clean deployment, the first spawn of each application may take some time whilst the container image representing the application is downloaded to the node. Subsequent invocations (at least on the same node) should be much faster. Once running, the application continues (in the background) until stopped by the user using the Stop Server button on the user\u2019s home screen.
The current JupyterHub configuration assumes a single application service (per user) running at a time - i.e. the current application must be stopped before the next can be started. There is an alternative configuration in which applications can be run in parallel and their lifecycles individually managed.
"},{"location":"quickstart/application-hub-deployment/#returning-to-the-home-screen","title":"Returning to the Home Screen","text":"The launched applications do not (yet) have a navigation link \u2018out\u2019 of the application back to the home screen.
Therefore, it is necessary to manually modify the url in the browser address bar to /hub/home to navigate to the home screen - from where the current running server can be stopped or re-entered.
Following instantiation, the IAT application (Interactive Analysis Tool) defaults to the \u2018Jupyter Notebook\u2019 view (/user/<user>/tree) - rather than the Jupyter Lab view (/user/<user>/lab).
To switch to the Jupyter Lab view it is necessary to manually edit the url path from /user/<user>/tree to /user/<user>/lab. It is intended to update the default to this Jupyter Lab path.
Based upon our development experiences on CREODIAS, there is a wrapper script creodias with particular customisations suited to the CREODIAS infrastructure and data offering. The customisations are expressed through environment variables that are captured in the file creodias-options.
These scripts are examples that can be seen as a starting point, from which they can be adapted to your needs.
The CREODIAS deployment applies the following configuration:
With reference to the file creodias-options, particular attention is drawn to the following environment variables that require tailoring to your CREODIAS (Cloudferro) environment\u2026
http://data.cloudferro.comWAW3-2 and FRA1-2 Credentials must be created at - https://eodata-keymanager.creodias.eu/panel/s3-credentials - and set into the variables CREODIAS_EODATA_S3_ACCESS_KEY and CREODIAS_EODATA_S3_ACCESS_SECRETMINIO_ROOT_PASSWORD, HARBOR_ADMIN_PASSWORDIDENTITY_SERVICE_DEFAULT_SECRET, IDENTITY_SERVICE_ADMIN_PASSWORD, etc.public_ip - The public IP address through which the deployment is exposed via the ingress-controllerdomain - The DNS domain name through which the deployment is accessed - forming the stem for all service hostnames in the ingress rulesOnce the file creodias-options has been well populated for your environment, then the deployment is initiated with\u2026
./deploy/creodias/creodias\nThe harvester can be deployed with a default configuration file at /config.yaml. As described in the Data Access section, harvesting according to this configuration can be triggered with\u2026
kubectl -n rm exec -it deployment.apps/data-access-harvester -- python3 -m harvester harvest --config-file /config.yaml --host data-access-redis-master --port 6379 Creodias-Opensearch\nSee the Harvester section below for an explanation of this harvester configuration.
See EOData Catalogue API Manual on CREODIAS for details regarding access to the CREODIAS data offering.
"},{"location":"quickstart/creodias-deployment/#data-specification-walkthrough","title":"Data Specification Walkthrough","text":"The example scripts include optional specifcation of data-access/harvesting configuration that is tailored for the CREODIAS data offering. This is controlled via the option CREODIAS_DATA_SPECIFICATION=true - see Environment Variables. In addition, it may be necessary to set the variable CREODIAS_EODATA_S3_ENDPOINT if different from the default - for example the value http://eodata.cloudferro.com for the WAW3-2 Cloudferro cloud.
This section provides a walkthrough of this configuration for CREODIAS - to act as an aid to understanding by way of a worked example.
"},{"location":"quickstart/creodias-deployment/#harvester","title":"Harvester","text":"The harvester configuration specifies datasets with spatial/temporal extents, which is configured into the file /config.yaml of the data-access-harvester deployment.
The harvester is configured as follows\u2026
harvester:\n replicaCount: 1\n resources:\n requests:\n cpu: 100m\n memory: 100Mi\n config:\n redis:\n host: data-access-redis-master\n port: 6379\n harvesters:\n - name: Creodias-Opensearch\n resource:\n url: https://datahub.creodias.eu/resto/api/collections/Sentinel2/describe.xml\n type: OpenSearch\n format_config:\n type: 'application/json'\n property_mapping:\n start_datetime: 'startDate'\n end_datetime: 'completionDate'\n productIdentifier: 'productIdentifier'\n query:\n time:\n property: sensed\n begin: 2019-09-10T00:00:00Z\n end: 2019-09-11T00:00:00Z\n collection: null\n bbox: 14.9,47.7,16.4,48.7\n filter: {}\n postprocess:\n - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor\n queue: register\n - name: Creodias-Opensearch-Sentinel1\n resource:\n url: https://datahub.creodias.eu/resto/api/collections/Sentinel1/describe.xml\n type: OpenSearch\n format_config:\n type: 'application/json'\n property_mapping:\n start_datetime: 'startDate'\n end_datetime: 'completionDate'\n productIdentifier: 'productIdentifier'\n query:\n time:\n property: sensed\n begin: 2019-09-10T00:00:00Z\n end: 2019-09-11T00:00:00Z\n collection: null\n bbox: 14.9,47.7,16.4,48.7\n extra_params:\n productType: GRD-COG\n filter: {}\n postprocess:\n - type: harvester_eoepca.postprocess.CREODIASOpenSearchSentinel1Postprocessor\n queue: register\nBased upon this harvester configuration we expect that the following query is made to discover data - i.e. an OpenSearch query, with json response representation, for a defined spatial and temporal extent\u2026
https://datahub.creodias.eu/resto/api/collections/Sentinel2/search.json?startDate=2019-09-10T00:00:00Z&completionDate=2019-09-11T00:00:00Z&box=14.9,47.7,16.4,48.7\nFrom the result returned, the path to each product (feature) is obtained from the productIdentifier property, e.g.
{\n \"type\": \"FeatureCollection\",\n \"features\": [\n {\n \"type\": \"Feature\",\n \"properties\": {\n \"productIdentifier\": \"/eodata/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE\"\n ...\n }\n ...\n }\n ...\n ]\n}\nThe harvester is configured with a Sentinel-2/CREODIAS specific post-processor harvester_eoepca.postprocess.CREODIASOpenSearchSentinel2Postprocessor which transforms the product path from /eodata/... to s3://EODATA/....
The harvester post-processor follows this path to the Sentinel-2 scene and uses stactools (with built-in support for Sentinel-2) to establish a STAC item representing the product. This includes enumeration of assets for inspire-metadata and product-metadata - which are used by the registrar pycsw backend to embelesh the product record metadata.
Note
The above description considers Sentinel-2 data. Similar considerations apply for Sentinel-1 that is also detailed in the above harvester configuration.
The harvester outputs the STAC item for each product, which is pushed to the registrar via the register redis queue.
The registrar is configured at deployment to have the access details for the CREODIAS data in S3\u2026
global:\n storage:\n data:\n data:\n type: S3\n endpoint_url: http://data.cloudferro.com\n access_key_id: access\n secret_access_key: access\n region_name: RegionOne\n validate_bucket_name: false\nUsing this S3 configuration, the registrar pycsw backend uses the product metadata linked in the STAC item (ref. assets inspire-metadata and product-metadata) to embelesh the metadata. For example, product-metadata in the file\u2026
s3://EODATA/Sentinel-2/MSI/L1C/2019/09/10/S2B_MSIL1C_20190910T095029_N0208_R079_T33TXN_20190910T120910.SAFE/MTD_MSIL1C.xml\nThe registrar uses this information to create the ISO XML metadata that is loaded into the resource-catalogue.
"},{"location":"quickstart/creodias-deployment/#product-type","title":"Product Type","text":"The registrar recognises the product as Sentinel-2 and so reads its metadata XML files to obtain additional information. From the metadata XML file (e.g. MTD_MSIL1C.xml) the registrar obtains the Product Type for each product from the field <PRODUCT_TYPE>\u2026
<n1:Level-1C_User_Product>\n <n1:General_Info>\n <Product_Info>\n <PRODUCT_TYPE>S2MSI1C</PRODUCT_TYPE>\n ...\n </Product_Info>\n ...\n </n1:General_Info>\n ...\n<n1:Level-1C_User_Product>\nThe registrar (eoepca/rm-data-access-core) container image is pre-loaded with two collections at the path /registrar_pycsw/registrar_pycsw/resources, (in the built container the files are at the path /usr/local/lib/python3.8/dist-packages/registrar_pycsw/resources/):
S2MSI1CS2MSI2AThe registrar applies these collections into the resource-catalogue during start-up - to create pre-defined out-of-the-box collections in pycsw.
During registration, the PycswBackend of the registrar uses the Product Type to map the product into the collection of the same name - using metadata field parentidentifier.
The data-access service data handling is configured by definition of productTypes, collections and layers\u2026
productTypes identify the underlying file assets as WCS coverages and their visual representationcollections provide groupings into which products are organisedlayers specifies the hoe the product visual representations are exposed through the WMS serviceproductType","text":"During registration, products are mapped into a productType via a filter that is applied against the STAC item metadata.
The registrar uses the product_type of each product to determine the collection into which the product should be registered - noting that the name of the product type does not take part in the matching logic (and hence can be any text name)\u2026
productTypes:\n - name: S2MSI1C\n filter:\n s2:product_type: S2MSI1C\nIn the above example, the field s2:product_type is populated by the stactools that prepares the STAC item from the product metadata.
productType - coverages","text":"coverages defines the coverages for the WCS service. Each coverage links to the assets that are defined within the product STAC item.
productType - browses","text":"browses defines the images that are visualised in the View Server Client. Expressions are used to map the product assets into their visual representation.
collections","text":"Collections are defined by reference to the defined productTypes and coverages.
layers","text":"layers defines the layers that are presented through the WMS service - each layer being linked to the underlying browse that provides the image source. Layers are defined via their id that relies upon the naming convection <collection>__<browse> to identify the browse and so define the layer.
Example configuration for Sentinel-2 L1C and L2A data.
global:\n layers:\n - id: S2L1C\n title: Sentinel-2 Level 1C True Color\n abstract: Sentinel-2 Level 2A True Color\n displayColor: '#eb3700'\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L1C__TRUE_COLOR\n title: Sentinel-2 Level 1C True Color\n abstract: Sentinel-2 Level 2A True Color\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L1C__masked_clouds\n title: Sentinel-2 Level 1C True Color with cloud masks\n abstract: Sentinel-2 Level 1C True Color with cloud masks\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L1C__FALSE_COLOR\n title: Sentinel-2 Level 1C False Color\n abstract: Sentinel-2 Level 1C False Color\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L1C__NDVI\n title: Sentinel-2 Level 21CNDVI\n abstract: Sentinel-2 Level 1C NDVI\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L1C\n - id: S2L2A\n title: Sentinel-2 Level 2A True Color\n abstract: Sentinel-2 Level 2A True Color\n displayColor: '#eb3700'\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n - id: S2L2A__TRUE_COLOR\n title: Sentinel-2 Level 2A True Color\n abstract: Sentinel-2 Level 2A True Color\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n - id: S2L2A__masked_clouds\n title: Sentinel-2 Level 2A True Color with cloud masks\n abstract: Sentinel-2 Level 2A True Color with cloud masks\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n - id: S2L2A__FALSE_COLOR\n title: Sentinel-2 Level 2A False Color\n abstract: Sentinel-2 Level 2A False Color\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n - id: S2L2A__NDVI\n title: Sentinel-2 Level 2A NDVI\n abstract: Sentinel-2 Level 2A NDVI\n grids:\n - name: WGS84\n zoom: 13\n parentLayer: S2L2A\n collections:\n S2L1C:\n product_types:\n - S2MSI1C\n coverage_types:\n - S2L1C_B01\n - S2L1C_B02\n - S2L1C_B03\n - S2L1C_B04\n - S2L1C_B05\n - S2L1C_B06\n - S2L1C_B07\n - S2L1C_B08\n - S2L1C_B8A\n - S2L1C_B09\n - S2L1C_B10\n - S2L1C_B11\n - S2L1C_B12\n S2L2A:\n product_types:\n - S2MSI2A\n product_levels:\n - Level-2A\n coverage_types:\n - S2L2A_B01\n - S2L2A_B02\n - S2L2A_B03\n - S2L2A_B04\n - S2L2A_B05\n - S2L2A_B06\n - S2L2A_B07\n - S2L2A_B08\n - S2L2A_B8A\n - S2L2A_B09\n - S2L2A_B11\n - S2L2A_B12\n productTypes:\n - name: S2MSI1C\n filter:\n s2:product_type: S2MSI1C\n metadata_assets: []\n coverages:\n S2L1C_B01:\n assets:\n - B01\n S2L1C_B02:\n assets:\n - B02\n S2L1C_B03:\n assets:\n - B03\n S2L1C_B04:\n assets:\n - B04\n S2L1C_B05:\n assets:\n - B05\n S2L1C_B06:\n assets:\n - B06\n S2L1C_B07:\n assets:\n - B07\n S2L1C_B08:\n assets:\n - B08\n S2L1C_B8A:\n assets:\n - B8A\n S2L1C_B09:\n assets:\n - B09\n S2L1C_B10:\n assets:\n - B10\n S2L1C_B11:\n assets:\n - B11\n S2L1C_B12:\n assets:\n - B12\n defaultBrowse: TRUE_COLOR\n browses:\n TRUE_COLOR:\n asset: visual\n red:\n expression: B04\n range: [0, 4000]\n nodata: 0\n green:\n expression: B03\n range: [0, 4000]\n nodata: 0\n blue:\n expression: B02\n range: [0, 4000]\n nodata: 0\n FALSE_COLOR:\n red:\n expression: B08\n range: [0, 4000]\n nodata: 0\n green:\n expression: B04\n range: [0, 4000]\n nodata: 0\n blue:\n expression: B03\n range: [0, 4000]\n nodata: 0\n NDVI:\n grey:\n expression: (B08-B04)/(B08+B04)\n range: [-1, 1]\n masks:\n clouds:\n validity: false\n - name: S2MSI2A\n filter:\n s2:product_type: S2MSI2A\n metadata_assets: []\n coverages:\n S2L2A_B01:\n assets:\n - B01\n S2L2A_B02:\n assets:\n - B02\n S2L2A_B03:\n assets:\n - B03\n S2L2A_B04:\n assets:\n - B04\n S2L2A_B05:\n assets:\n - B05\n S2L2A_B06:\n assets:\n - B06\n S2L2A_B07:\n assets:\n - B07\n S2L2A_B08:\n assets:\n - B08\n S2L2A_B8A:\n assets:\n - B8A\n S2L2A_B09:\n assets:\n - B09\n S2L2A_B11:\n assets:\n - B11\n S2L2A_B12:\n assets:\n - B12\n default_browse_locator: TCI_10m\n browses:\n TRUE_COLOR:\n asset: visual-10m\n red:\n expression: B04\n range: [0, 4000]\n nodata: 0\n green:\n expression: B03\n range: [0, 4000]\n nodata: 0\n blue:\n expression: B02\n range: [0, 4000]\n nodata: 0\n FALSE_COLOR:\n red:\n expression: B08\n range: [0, 4000]\n nodata: 0\n green:\n expression: B04\n range: [0, 4000]\n nodata: 0\n blue:\n expression: B03\n range: [0, 4000]\n nodata: 0\n NDVI:\n grey:\n expression: (B08-B04)/(B08+B04)\n range: [-1, 1]\n masks:\n clouds:\n validity: false\nA deployment wrapper script has been prepared for a \u2018data access\u2019 deployment - that is focused on the Resource Catalogue and Data Access services.
The script deploy/data-access/data-access achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/data-access/data-access-options.
The data-access deployment applies the following configuration:
eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Note that it may be necessary to set the variable CREODIAS_EODATA_S3_ENDPOINT if different from the default - for example the value http://eodata.cloudferro.com for the WAW3-2 Cloudferro cloud.Deployment is initiated by invoking the script\u2026
./deploy/data-access/data-access\nThe Resource Catalogue is accessed at the endpoint resource-catalogue-open.192-168-49-2.nip.io - e.g. resource-catalogue-open.192-168-49-2.nip.io.
The Data Access View Server is accessed at the endpoint data-access-open.192-168-49-2.nip.io - e.g. data-access-open.192-168-49-2.nip.io.
To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/data-access-deployment/#data-harvesting","title":"Data Harvesting","text":"See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.
"},{"location":"quickstart/exploitation-deployment/","title":"Exploitation Deployment","text":""},{"location":"quickstart/exploitation-deployment/#overview","title":"Overview","text":"A deployment wrapper script has been prepared for an \u2018exploitation\u2019 deployment - that provides deployment/execution of processing via the ADES, supported by Resource Catalogue and Data Access services.
The script deploy/exploitation/exploitation achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/exploitation/exploitation-options.
The exploitation deployment applies the following configuration:
eodata network - see description of variable CREODIAS_DATA_SPECIFICATION Note that it may be necessary to set the variable CREODIAS_EODATA_S3_ENDPOINT if different from the default - for example the value http://eodata.cloudferro.com for the WAW3-2 Cloudferro cloud.Deployment is initiated by invoking the script\u2026
./deploy/exploitation/exploitation\nThe ADES service is accessed at the endpoint ades-open.192-168-49-2.nip.io.
The Resource Catalogue is accessed at the endpoint resource-catalogue-open.192-168-49-2.nip.io.
The Data Access View Server is accessed at the endpoint data-access-open.192-168-49-2.nip.io.
To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/exploitation-deployment/#example-requests-s-expression-on-creodias","title":"Example Requests -s-expression on CREODIAS","text":"NOTE that this example processing request requires harvesting data from CREODIAS, which can only be performed if the deployment is made to a CREODIAS VM with access to the eodata network - see description of variable CREODIAS_DATA_SPECIFICATION.
Section Processing provides an example of a simple self-contained processing deployment and execution, and access to the processing results.
In addition to the snuggs example, the file deploy/samples/requests/processing/s-expression.http has been prepared to exploit data that has been registered within the Resource Catalogue and Data Access services.
First the input data for processing must be harvested into the resource management services. Sentinel-2 data on 2nd Sept 2020\u2026
./deploy/bin/harvest ./deploy/samples/harvester/config-Sentinel2-2020.09.02.yaml\nThen the s-expression.http file provides sample requests for OGC API Processes operations:
NOTE that the first requests in the file provide optional calls to obtain a user access token (openidConfiguration / authenticate) - to be used in the case that protected (not \u2018open\u2019) endpoints are deployed.
The files snuggs.http and s-expression.http describe the HTTP requests for the ADES OGC API Processes endpoint, and is designed for use with the Visual Studio Code (vscode) extension REST Client. Install in vscode with ext install humao.rest-client.
Various variables, such as to specify the @domain for your deployment, can be configured at the top of the file.
At the completion of successful processing execution, the procesing results are obtained as described in section Processing Results.
"},{"location":"quickstart/exploitation-deployment/#data-harvesting","title":"Data Harvesting","text":"See section Harvest CREODIAS Data to harvest the default data specification from the CREODIAS data offering.
"},{"location":"quickstart/processing-deployment/","title":"Processing Deployment","text":""},{"location":"quickstart/processing-deployment/#overview","title":"Overview","text":"A deployment wrapper script has been prepared for a \u2018processing\u2019 deployment - that is focused on the ADES and the deployment/execution of processing jobs.
The script deploy/processing/processing achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/processing/processing-options.
The processing deployment applies the following configuration:
Deployment is initiated by invoking the script\u2026
./deploy/processing/processing\nThe ADES service is accessed at the endpoint zoo-open.192-168-49-2.nip.io.
To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/processing-deployment/#example-requests","title":"Example Requests","text":"Some sample requests have been prepared in the subdirectory deploy/samples/requests/processing - for example\u2026
convert Provides a \u2018hello world\u2019 processing example that can be used simply to check that the processing capability has been well deployedsnuggs Provides a packaged EO exploitation algorithm that perform \u2018real\u2019 work and, as such, is more resource demanding (10GB RAM, 3 CPU) - and so may not be suitable for execution within a local minikube deployment (depending on resource allocations)These sample http files have been prepared with sample requests for OGC API Processes operations:
Note
openidConfiguration / authenticate). These are to be used in the case that protected (not \u2018open\u2019) endpoints are deployed.ext install humao.rest-client.@hostname and @domain can be configured at the top of the file.curl Commands","text":"Alternatively the following curl commands can be used\u2026
curl -k \\\n --request GET \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n --header 'accept: application/json'\nconvert) Deploy Process (convert) - By Reference (JSON) curl -k \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n --header 'accept: application/json' \\\n --header 'content-type: application/json' \\\n --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/convert/main/convert-url-app.cwl\",\"type\": \"application/cwl\"}}'\nconvert) - Inline (CWL) curl -k \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n --header 'accept: application/json' \\\n --header 'content-type: application/cwl+yaml' \\\n --data '< convert-url-app.cwl'\nconvert) curl -k \\\n --request GET \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \\\n --header 'accept: application/json'\nconvert) curl -k -v \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url/execution \\\n --header 'accept: application/json' \\\n --header 'content-type: application/json' \\\n --header 'prefer: respond-async' \\\n --data '{\"inputs\": {\"fn\": \"resize\",\"url\": \"https://eoepca.org/media_portal/images/logo6_med.original.png\", \"size\": \"50%\"},\"response\":\"raw\"}'\nconvert) curl -k -v \\\n --request DELETE \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/convert-url \\\n --header 'accept: application/json'\nsnuggs) Deploy Process (snuggs) curl -k \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes \\\n --header 'accept: application/json' \\\n --header 'content-type: application/json' \\\n --data '{\"executionUnit\": {\"href\": \"https://raw.githubusercontent.com/EOEPCA/deployment-guide/eoepca-v1.4/deploy/samples/requests/processing/snuggs.cwl\",\"type\": \"application/cwl\"}}'\nsnuggs) curl -k \\\n --request GET \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \\\n --header 'accept: application/json'\nsnuggs) curl -k -v \\\n --request POST \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs/execution \\\n --header 'accept: application/json' \\\n --header 'content-type: application/json' \\\n --header 'prefer: respond-async' \\\n --data '{\"inputs\": {\"input_reference\": \"https://earth-search.aws.element84.com/v0/collections/sentinel-s2-l2a-cogs/items/S2B_36RTT_20191205_0_L2A\",\"s_expression\": \"ndvi:(/ (- B05 B03) (+ B05 B03))\"},\"response\":\"raw\"}'\nsnuggs) curl -k -v \\\n --request DELETE \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/processes/snuggs \\\n --header 'accept: application/json'\nThis request requires the Location header from the response to the execute request. This will be of the form http://zoo-open.192-168-49-2.nip.io/{user}/ogc-api/jobs/{job-id} - e.g. http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e.
curl -k \\\n --request GET \\\n --url {location-header} \\\n --header 'accept: application/json'\nThis request uses the same URL as Get Job Status, with the additional URL path /results - i.e. /{user}/ogc-api/jobs/{job-id}/results - e.g. /eric/ogc-api/jobs/7b58bc38-64d4-11ed-b962-0242ac11000e/results
curl -k \\\n --request GET \\\n --url {location-header}/results \\\n --header 'accept: application/json'\nThe response indicates the location of the results, which should be in the minio object storage. See Processing Results.
The response also provides links to log files regarding each step of the Application Package workflow execution - which may be useful for debugging.
List Jobscurl -k \\\n --request GET \\\n --url http://zoo-open.192-168-49-2.nip.io/eric/ogc-api/jobs \\\n --header 'accept: application/json'\nThe outputs are published as a static STAC catalogue to a path that includes the unique job ID.
In the default configuration, the processing results are pushed to the Minio S3 object storage. This can be checked via browser access at the endpoint http://console.minio.192-168-49-2.nip.io/, or using an S3 client such as\u2026
s3cmd -c ./deploy/cluster/s3cfg ls s3://eoepca\nFor the default credentials to connect to Minio see Minio Object Storage Default Credentials.
Note
If the ADES deployment has been configured to stage-out to the user\u2019s Workspace, then the above s3cmd and credentials would have to be adjusted accordingly - for example the bucket s3://ws-eric.
Note
The deployment of the EOEPCA components and the supporting Kubernetes cluster is described in the sections Prepare Cluster and Deploy EOEPCA Components. These sections should be consulted for more detailed information.
"},{"location":"quickstart/quickstart/#scripted-deployment","title":"Scripted Deployment","text":"As a companion to these descriptions, we have developed a set of scripts to provide a demonstration of example deployments - see section Scripted Deployment for a detailed description of the scripts and how they are configured and used.
Note
The scripted deployment assumes that installation of the Prerequisite Tooling has been performed
"},{"location":"quickstart/quickstart/#customised-deployments","title":"Customised Deployments","text":"The Scripted Deployment can be quickly exploited through the following customisations (profiles) for particular use cases:
Each customisation is introduced in their respective sections.
"},{"location":"quickstart/quickstart/#quick-example","title":"Quick Example","text":"Follow these steps to create a simple local deployment in minikube\u2026
git clone -b eoepca-v1.4 https://github.com/EOEPCA/deployment-guidecd deployment-guide\n./deploy/simple/simple\nwatch kubectl get pod -ARunning or Completed This may take 10-20 mins depending on the capabilities of your platform.The Scripted Deployment provides a demonstration of an example deployment, and can found in the subdirectory deployment-guide/deploy of the source repository for this guide\u2026
git clone -b eoepca-v1.4 https://github.com/EOEPCA/deployment-guide \\\n&& cd deployment-guide \\\n&& ls deploy\nThe script deploy/eoepca/eoepca.sh acts as an entry-point to the full system deployment. In order to tailor the deployment for your target environment, the script is configured through environment variables and command-line arguments. By default the script assumes deployment to a local minikube.
Note
The scripted deployment assumes that installation of the Prerequisite Tooling has been performed.
The following subsections lead through the steps for a full local deployment. Whilst minikube is assumed, minimal adaptions are required to make the deployment to your existing Kubernetes cluster.
The deployment follows these broad steps:
The script deploy/eoepca/eoepca.sh is configured by some environment variables and command-line arguments.
REQUIRE_MINIKUBE=trueREQUIRE_INGRESS_NGINX=trueREQUIRE_CERT_MANAGER=trueREQUIRE_LETSENCRYPT=trueREQUIRE_SEALED_SECRETS=falseREQUIRE_MINIO=false see description REQUIRE_<eoepca-component> A set of variables that can be used to control which EOEPCA components are deployed by the script, as follows (with defaults):REQUIRE_STORAGE=trueREQUIRE_DUMMY_SERVICE=falseREQUIRE_IDENTITY_SERVICE=trueREQUIRE_ADES=trueREQUIRE_RESOURCE_CATALOGUE=trueREQUIRE_DATA_ACCESS=trueREQUIRE_REGISTRATION_API=trueREQUIRE_WORKSPACE_API=trueREQUIRE_HARBOR=trueREQUIRE_PORTAL=trueREQUIRE_APPLICATION_HUB=true see description REQUIRE_<protection-component> A set of variables that can be used to control which PROTECTION components are deployed by the script, as follows (with defaults):REQUIRE_DUMMY_SERVICE_PROTECTION=falseREQUIRE_ADES_PROTECTION=trueREQUIRE_RESOURCE_CATALOGUE_PROTECTION=trueREQUIRE_DATA_ACCESS_PROTECTION=trueREGISTRATION_API_PROTECTION=trueREQUIRE_WORKSPACE_API_PROTECTION=true see description MINIKUBE_VERSION The Minikube version to be (optionally) installedNote that the EOEPCA development has been conducted using the default stated here. v1.32.0 MINIKUBE_KUBERNETES_VERSION The Kubernetes version to be used by minikubeNote that the EOEPCA development has been conducted primarily using version 1.22.5. v1.22.5 MINIKUBE_MEMORY_AMOUNT Amount of memory to allocate to the docker containers used by minikube to implement the cluster. 12g MINIKUBE_DISK_AMOUNT Amount of disk space to allocate to the docker containers used by minikube to implement the cluster. 20g MINIKUBE_EXTRA_OPTIONS Additional options to pass to minikube start command-line --ports=80:80,443:443 USE_METALLB Enable use of minikube\u2019s built-in load-balancer.The load-balancer can be used to facilitate exposing services publicly. However, the same can be achieved using minikube\u2019s built-in ingress-controller. Therefore, this option is suppressed by default. false USE_INGRESS_NGINX_HELM Install the ingress-nginx controller using the published helm chart, rather than relying upon the version that is built-in to minikube. By default we prefer the version that is built in to minikube. false USE_INGRESS_NGINX_LOADBALANCER Patch the built-in minikube nginx-ingress-controller to offer a service of type LoadBalancer, rather than the default NodePort. It was initially thought that this would be necessary to achieve public access to the ingress services - but was subsequently found that the default NodePort configuration of the ingress-controller was sufficient. This option is left in case it proves useful.Only applicable for USE_INGRESS_NGINX_HELM=false (i.e. when using the minikube built-in ) false OPEN_INGRESS Create \u2018open\u2019 ingress endpoints that are not subject to authorization protection. For a secure system the open endpoints should be disabled (false) and access to resource should be protected via ingress that apply protection false USE_TLS Indicates whether TLS will be configured for service Ingress rules.If not (i.e. USE_TLS=false), then the ingress-controller is configured to disable ssl-redirect, and TLS_CLUSTER_ISSUER=notls is set. true TLS_CLUSTER_ISSUER The name of the ClusterIssuer to satisfy ingress tls certificates.Out-of-the-box ClusterIssuer instances are configured in the file deploy/cluster/letsencrypt.sh. letsencrypt-staging IDENTITY_SERVICE_DEFAULT_SECRET Default secret that is used by exception for other Identity Service credentials changeme IDENTITY_SERVICE_ADMIN_USER The admin user for Keycloak admin IDENTITY_SERVICE_ADMIN_PASSWORD The admin user password for Keycloak ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_SERVICE_ADMIN_CLIENT The Keycloak client to use for admin API tasks during scripted deployment admin-cli IDENTITY_POSTGRES_PASSWORD The password for the Keycloak Postgres service ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_GATEKEEPER_CLIENT_SECRET The secret used for each Keycloak client (one per resource service) created during scripted deployment ${IDENTITY_SERVICE_DEFAULT_SECRET} IDENTITY_GATEKEEPER_ENCRYPTION_KEY The encryption key for each Keycloak client (one per resource service) created during scripted deploymentNOTE that this must be either 16 or 32 characters long changemechangeme IDENTITY_REALM Keycloak realm for Identity Service.This is not explicitly created by the scripted deployment, and so is assumed to exist within the Keycloak instance. Thus, will probably break the deployment if modified. master MINIO_ROOT_USER Name of the \u2018root\u2019 user for the Minio object storage service. eoepca MINIO_ROOT_PASSWORD Password for the \u2018root\u2019 user for the Minio object storage service. changeme HARBOR_ADMIN_PASSWORD Password for the \u2018admin\u2019 user for the Harbor artefact registry service. changeme DEFAULT_STORAGE Storage Class to be used by default for all components requiring dynamic storage provisioning.See variables <component>_STORAGE for per-component overrides. standard <component>_STORAGE A set of variables to control the dynamic provisioning Storage Class for individual components, as follows:MINIO_STORAGEADES_STORAGEAPPLICATION_HUB_STORAGEDATA_ACCESS_STORAGEHARBOR_STORAGERESOURCE_CATALOGUE_STORAGE <DEFAULT_STORAGE> PROCESSING_MAX_RAM Max RAM allocated to an individual processing job 8Gi PROCESSING_MAX_CORES Max number of CPU cores allocated to an individual processing job 4 PROCESSING_ZOO_IMAGE Container image for zoo-dru deployment eoepca-092ea7a2c6823dba9c6d52c383a73f5ff92d0762 STAGEOUT_TARGET Configures the ADES with the destination to which it should push processing results:workspace - via the Workspace APIminio - to minio S3 object storage workspace INSTALL_FLUX The Workspace API relies upon Flux CI/CD, and has the capability to install the required flux components to the cluster. If your deployment already has flux installed then set this value false to suppress the Workspace API flux install true CREODIAS_DATA_SPECIFICATION Apply the data specification to harvest from the CREODIAS data offering into the resource-catalogue and data-access services.Can only be used when running in the CREODIAS (Cloudferro) cloud, with access to the eodata network. false CREODIAS_EODATA_S3_ENDPOINT URL for the S3 endpoint in CREODIAS http://data.cloudferro.com CREODIAS_EODATA_S3_ACCESS_KEY Access key for CREODIAS S3 endpoint access CREODIAS_EODATA_S3_ACCESS_SECRET Access secret for CREODIAS S3 endpoint access CREODIAS_EODATA_S3_REGION Region for the S3 endpoint in CREODIAS RegionOne TEMP_FORWARDING_PORT Local port used during the scripted deployment for kubectl port-forward operations 9876"},{"location":"quickstart/scripted-deployment/#command-line-arguments","title":"Command-line Arguments","text":"The eoepca.sh script is further configured via command-line arguments\u2026
eoepca.sh <action> <cluster-name> <domain> <public-ip>\neoepca.sh Command-line Arguments Argument Description Default action Action to perform: apply | delete | template.apply makes the deploymentdelete removes the deploymenttemplate outputs generated kubernetes yaml to stdout apply cluster-name The name of the minikube \u2018profile\u2019 for the created minikube cluster eoepca domain The DNS domain name through which the deployment is accessed. Forms the stem for all service hostnames in the ingress rules - i.e. <service-name>.<domain>.By default, the value is deduced from the assigned cluster minikube IP address, using nip.io to establish a DNS lookup - i.e. <minikube ip>.nip.io. <minikube ip>.nip.io public-ip The public IP address through which the deployment is exposed via the ingress-controller.By default, the value is deduced from the assigned cluster minikube IP address - ref. command minikube ip. <minikube-ip>"},{"location":"quickstart/scripted-deployment/#public-deployment","title":"Public Deployment","text":"For simplicity, the out-of-the-box scripts assume a \u2018private\u2019 deployment - with no public IP / DNS and hence no use of TLS for service ingress endpoints.
In the case that an external-facing public deployment is desired, then the following configuration selections should be made:
domain - set to the domain (as per DNS records) for your deployment Note that the EOEPCA components typically configure their ingress with hostname prefixes applied to this domain. Thus, it is necessary that the DNS record for the domain is established as a wildcard record - i.e. *.<domain>public_ip - set to the public IP address through which the deployment is exposed via the ingress-controller i.e. the IP address that is assigned to the ingress controller service of type LoadBalancerUSE_TLS=true - to enable configuration of TLS endpoints in each component service ingressTLS_CLUSTER_ISSUER=<issuer> - should be configured ~ e.g. using the letsencrypt-production or letsencrypt-staging (testing only) Cluster Issuer that are configured by the scripted deploymentThe deployment is initiated by setting the appropriate environment variables and invoking the eoepca.sh script with suitable command-line arguments. You may find it convenient to do so using a wrapper script that customises the environment varaibles according to your cluster, and then invokes the eoepca.sh script.
Customised examples are provided for Simple, CREODIAS and Processing deployments.
NOTE that if a prior deployment has been attempted then, before redeploying, a clean-up should be performed as described in the Clean-up section below. This is particularly important in the case that the minikube none driver is used, as the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed.
Initiate the deployment\u2026
./deploy/eoepca/eoepca.sh apply \"<cluster-name>\" \"<public-ip>\" \"<domain>\"\nThe deployment takes 10+ minutes - depending on the resources of your host/cluster. The progress can be monitored\u2026
kubectl get pods -A\nThe deployment is ready once all pods are either Running or Completed.
The scripted deployment has been designed, as far as possible, to automate the configuration of the deployed components. However, there remain some steps that must be performed manually after the scripted deployment has completed. See the building block specific pages\u2026
By default, the Identity Service is accessed at the URL https://keycloak.192-168-49-2.nip.io/ with the credentials\u2026
username: `admin`\npassword: `changeme`\n\u2026unless the password is overridden via the variable IDENTITY_SERVICE_ADMIN_PASSWORD.
By default, Minio is accessed at the URL https://console.minio.192-168-49-2.nip.io/ with the credentials\u2026
username: `eoepca`\npassword: `changeme`\n\u2026unless the username/password are overridden via the variables MINIO_ROOT_USER and MINIO_ROOT_PASSWORD.
By default, Harbor is accessed at the URL https://harbor.192-168-49-2.nip.io/ with the credentials\u2026
username: `admin`\npassword: `changeme`\n\u2026unless the password is overridden via the variable HARBOR_ADMIN_PASSWORD.
The protection of resource server endpoints is applied during the deployment of each service requiring protection. This comprises creating a dedicated Keycloak client for each resource server, and the creation of associated resources and policies that protect the service-specific URLs.
This protection can be disabled via the environment variables REQUIRE_XXX_PROTECTION - e.g. REQUIRE_ADES_PROTECTION=false.
Note
By default, if OPEN_INGRESS is set true then PROTECTION will be disabled (false) unless overridden via the REQUIRE_XXX_PROTECTION variables.
The deployment creates (in the Keycloak Identity Service) the test users: eric, bob, alice.
Note
This does NOT create the workspace for each of these users - which must be performed via the Workspace API.
"},{"location":"quickstart/scripted-deployment/#user-workspace-creation","title":"User Workspace Creation","text":"The deployment created the test users eric, bob and alice. For completeness we use the Workspace API to create their user workspaces, which hold their personal resources (data, processing results, etc.) within the platform - see Workspace.
The Workspace API provides a Swagger UI that facilitates interaction with the API - at the URL https://workspace-api.192-168-49-2.nip.io/docs#.
Note
If the Workspace API has been protected (via Gatekeeper with Keycloak), then requests must be supported by an access_token carried in the HTTP header Authorozation: Bearer <token>. This diminishes the utility of the swagger UI.
Access the Workspace Swagger UI at https://workspace-api.192-168-49-2.nip.io/docs. Workspaces are created using POST /workspaces (Create Workspace). Expand the node and select Try it out. Complete the request body, such as\u2026
{\n \"preferred_name\": \"eric\",\n \"default_owner\": \"eric\"\n}\ndefault_owner is the ID for the user in Keycloak - thus protecting the created workspace for the identified user."},{"location":"quickstart/scripted-deployment/#using-curl","title":"Using curl","text":"The same can be achieved with a straight http request, for example using curl\u2026
curl -X 'POST' \\\n 'http://workspace-api.192-168-49-2.nip.io/workspaces' \\\n -H 'Content-Type: application/json' \\\n -H 'Accept: application/json' \\\n -H 'Authorization: Bearer <admin-access-token>' \\\n -d '{\n \"preferred_name\": \"<workspace-name>\",\n \"default_owner\": \"<user-id>\"\n}'\nValues must be provided for:
admin-access-token - Access Token for the admin userworkspace-name - name of the workspace, typically the usernameuser-id - the ID of the user for which the created workspace will be protected, typically the usernameThe Access Token for the admin user can be obtained with a call to the token endpoint of the Identity Service - supplying the credentials for the admin user and the pre-registered client\u2026
curl -L -X POST 'https://keycloak.192-168-49-2.nip.io/realms/master/protocol/openid-connect/token' \\\n -H 'Cache-Control: no-cache' \\\n -H 'Content-Type: application/x-www-form-urlencoded' \\\n --data-urlencode 'scope=openid profile email' \\\n --data-urlencode 'grant_type=password' \\\n --data-urlencode 'username=admin' \\\n --data-urlencode 'password=<admin-password>' \\\n --data-urlencode 'client_id=admin-cli'\nA json response is returned, in which the field access_token provides the Access Token for the admin user.
create-workspace helper script","text":"As an aide there is a helper script create-workspace. The script is available in the deployment-guide repository, and can be obtained as follows\u2026
git clone -b eoepca-v1.4 git@github.com:EOEPCA/deployment-guide\ncd deployment-guide\nThe create-workspace helper script requires some command-line arguments\u2026
$ ./deploy/bin/create-workspace -h\n\nCreate a new User Workspace.\ncreate-workspace -h | -w {workspace_api} -a {auth_server} -r {realm} -c {client} -u {admin-username} -p {admin-password} -O {owner} -W {workspace-name}\n\nwhere:\n -h show help message\n -w workspace-api service url (default: http://workspace-api.192-168-49-2.nip.io)\n -a authorization server url (default: http://keycloak.192-168-49-2.nip.io)\n -r realm within Keycloak (default: master)\n -u username used for authentication (default: admin)\n -p password used for authentication (default: changeme)\n -c client id of the bootstrap client used in the create request (default: admin-cli)\n -O user ID of the 'owner' of the new workspace (default: workspace(-W))\n -W name of the workspace to create (default: owner(-O))\nMost of the arguments have default values that are aligned to the defaults of the scripted deployment. At minimum either -O owner or -W workspace must be specified.
For example (assuming defaults)\u2026
./deploy/bin/create-workspace -O eric\nFor example (all arguments)\u2026
./deploy/bin/create-workspace \n -w http://workspace-api.192-168-49-2.nip.io \\\n -a http://keycloak.192-168-49-2.nip.io \\\n -r master \\\n -u admin \\\n -p changeme \\\n -c admin-cli \\\n -O bob \\\n -W bob\nThe eoepca-portal is a simple web application that is used as a test aid. It\u2019s main purpose is to provide the ability to login, and so establish a session with appropriate browser cookies - which then allow authenticated access to other EOEPCA services such as the Workspace API, Identity API, etc.
The portal is deployed via a helm chart\u2026
helm install eoepca-portal eoepca-portal -f portal-values.yaml - \\\n --repo https://eoepca.github.io/helm-charts \\\n --namespace \"demo\" --create-namespace \\\n --version 1.0.11\nThe helm values must be tailored for your deployment. For example\u2026
configMap:\n identity_url: \"http://keycloak.192-168-49-2.nip.io\"\n realm: \"master\"\n client_id: \"eoepca-portal\"\n identity_api_url: \"http://identity-api.192-168-49-2.nip.io\"\n ades_url: \"http://zoo.192-168-49-2.nip.io/ogc-api/processes\"\n resource_catalogue_url: \"http://resource-catalogue.192-168-49-2.nip.io\"\n data_access_url: \"http://data-access.192-168-49-2.nip.io\"\n workspace_url: \"http://workspace-api.192-168-49-2.nip.io\"\n workspace_docs_url: \"http://workspace-api.192-168-49-2.nip.io/docs#\"\n images_registry_url: \"http://harbor.192-168-49-2.nip.io\"\n dummy_service_url: \"http://dummy-service.192-168-49-2.nip.io\"\n access_token_name: \"auth_user_id\"\n access_token_domain: \".192-168-49-2.nip.io\"\n refresh_token_name: \"auth_refresh_token\"\n refresh_token_domain: \".192-168-49-2.nip.io\"\ningress:\n enabled: true\n annotations:\n kubernetes.io/ingress.class: nginx\n ingress.kubernetes.io/ssl-redirect: \"true\"\n nginx.ingress.kubernetes.io/ssl-redirect: \"true\"\n cert-manager.io/cluster-issuer: letsencrypt-production\n hosts:\n - host: eoepca-portal.192-168-49-2.nip.io\n paths:\n - path: /\n pathType: Prefix\n tls:\n - secretName: eoepca-portal-tls\n hosts:\n - eoepca-portal.192-168-49-2.nip.io\nThe setting client_id: eoepca-portal identifies a client that must be created in Keycloak - as described in section create-client Helper Script - noting that the eoepca-portal requires a client that is configured as a Public Client\u2026
../bin/create-client \\\n -a https://keycloak.192-168-49-2.nip.io \\\n -i https://identity-api.192-168-49-2.nip.io \\\n -r \"master\" \\\n -u \"admin\" \\\n -p \"changeme\" \\\n -c \"admin-cli\" \\\n --id=eoepca-portal \\\n --name=\"EOEPCA Portal\" \\\n --public \\\n --description=\"Client to be used by the EOEPCA Portal\"\nBefore initiating a fresh deployment, if a prior deployment has been attempted, then it is necessary to remove any persistent artefacts of the prior deployment. This includes\u2026
Minikube cluster Delete the minikube cluster\u2026 minikube delete If necessary specify the cluster (profile)\u2026 minikube -p <profile> delete
Persistent Data In the case that the minikube none driver is used, the persistence is maintained on the host and so is not naturally removed when the minikube cluster is destroyed. In this case, the minikube standard StorageClass is fulfilled by the hostpath provisioner, whose persistence is removed as follows\u2026 sudo rm -rf /tmp/hostpath-provisioner
There is a helper script clean that can be used for step 2 above, (the script does not delete the cluster).
./deploy/cluster/clean\nA deployment wrapper script has been prepared for a \u2018simple\u2019 deployment - designed to get a core local deployment of the primary servies.
The script deploy/simple/simple achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/simple/simple-options.
The simple deployment applies the following configuration:
minio, to avoid the need to create a Workspace for each userDeployment is initiated by invoking the script\u2026
./deploy/simple/simple\nSee section Deployment for more details regarding the outcome of the scripted deployment.
"},{"location":"quickstart/simple-deployment/#post-deploy-manual-steps","title":"Post-deploy Manual Steps","text":"To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"},{"location":"quickstart/userman-deployment/","title":"User Management Deployment","text":""},{"location":"quickstart/userman-deployment/#overview","title":"Overview","text":"A deployment wrapper script has been prepared for a \u2018user management\u2019 deployment - that is focused on the Identity Service (Authorization Server), Identity API and Gatekeeper (Protection Policy Enforcement).
The script deploy/userman/userman achieves this by appropriate configuration of the environment variables, before launching the eoepca.sh deployment script. The deployment configuration is captured in the file deploy/userman/userman-options.
The user-management deployment applies the following configuration:
Deployment is initiated by invoking the script\u2026
./deploy/userman/userman\nThe Identity Service is accessed at the endpoint keycloak.192-168-49-2.nip.io.
The Identity API is accessed at the endpoint identity-api.192-168-49-2.nip.io.
To complete the deployment, see section Post-deployment Manual Steps of the Scripted Deployment page.
"}]} \ No newline at end of file diff --git a/current/sitemap.xml b/current/sitemap.xml index 5cfe7f26..9621f009 100644 --- a/current/sitemap.xml +++ b/current/sitemap.xml @@ -2,147 +2,147 @@