Patch: update GET job list to API v2

[dapineyro]

Overview

This PR updates the GET job list API call to use v2 endpoint. Old v1 endpoint will be removed soon from the API server.
Additionally, adds the new --archived flag to cloudos job list to be able to retrieve archived jobs list.

Jira

https://lifebit.atlassian.net/browse/LIF-1348

Changes

• Updates GET job list API call to use v2 endpoint.
• Adds --archived option. When used, only archived jobs list are retrieved.
• Removed unused installation from Dockerfile

Note

Getting a single job by ID is still using v1 in the platform so does not need to be updated.

Tests

Using docker image:

docker run --rm -it quay.io/lifebitaiorg/cloudos-cli:v2.13.0

Global variables:

CLOUDOS="https://dev2.sdlc.lifebit.ai"
MY_API_KEY="xxxx"
WORKSPACE_ID="xxxx"

Get job list --help:

cloudos job list --help

CloudOS python package: a package for interacting with CloudOS.

Version: 2.13.0

CloudOS job functionality: run and check jobs in CloudOS.

Usage: cloudos job list [OPTIONS]

  Collect all your jobs from a CloudOS workspace in CSV format.

Options:
  -k, --apikey TEXT           Your CloudOS API key  [required]
  -c, --cloudos-url TEXT      The CloudOS url you are trying to access to.
                              Default=https://cloudos.lifebit.ai.
  --workspace-id TEXT         The specific CloudOS workspace id.  [required]
  --output-basename TEXT      Output file base name to save jobs list.
                              Default=joblist
  --output-format [csv|json]  The desired file format (file extension) for the
                              output. Default=csv.
  --all-fields                Whether to collect all available fields from
                              jobs or just the preconfigured selected fields.
                              Only applicable when --output-format=csv
  --last-n-jobs TEXT          The number of last user's jobs to retrieve. You
                              can use 'all' to retrieve all user's jobs.
                              Default=30.
  --page INTEGER              Response page to retrieve. If --last-n-jobs is
                              set, then --page value corresponds to the first
                              page to retrieve. Default=1.
  --archived                  When this flag is used, only archived jobs list
                              is collected.
  --verbose                   Whether to print information messages or not.
  --disable-ssl-verification  Disable SSL certificate verification. Please,
                              remember that this option is not generally
                              recommended for security reasons.
  --ssl-cert TEXT             Path to your SSL certificate file.
  --help                      Show this message and exit.

Get job list, default options: first page, last 30 jobs:

cloudos job list \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID

Get archived job list, default options: first page, last 30 jobs:

cloudos job list \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --archived \
    --output-basename "archived"

Get job list, options: second page (30 jobs):

cloudos job list \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --output-basename "second_page" \
    --page 2

Note

Each page contains 10 jobs, so with this we are effectively getting page 2, 3, and 4

Get job list, options: json output

cloudos job list \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --output-format json

Get job list, options: --last-n-jobs 100:

cloudos job list \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --output-basename "last_100_jobs" \
    --last-n-jobs 100

Get job list, options: --last-n-jobs all:

cloudos job list \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --output-basename "all_workspace_jobs" \
    --last-n-jobs all

Get archived job list, options: --last-n-jobs all:

cloudos job list \
    --cloudos-url $CLOUDOS \
    --apikey $MY_API_KEY \
    --workspace-id $WORKSPACE_ID \
    --archived \
    --output-basename "all_archived_jobs" \
    --last-n-jobs all

[danielboloc]

LGTM!

Just a few doubts from my side:

1. Page is always size 10 jobs from cloudos-cli side?
2. There is no way to filter by pipeline name or any other filter present in the UI right? These will have to be done manually after downloading the list?

[dapineyro]

LGTM!

Just a few doubts from my side:

1. Page is always size 10 jobs from cloudos-cli side?
2. There is no way to filter by pipeline name or any other filter present in the UI right? These will have to be done manually after downloading the list?

1. Yes, that's the page size the API server returns
2. There are other filters in addition to archived.status. You can filter by user.id, workflow.id, etc. same as in the UI . All these filters are a bit more complicated to implement in cloudos-cli, because the user should know these IDs from the database, and they are not easy to collect. It is something we can definitively add in future releases, but it's out of the scope of this PR

[s0rthak]

Looks good to me!