Merge pull request #140 from seqeralabs/dev

Release v0.4.7

README.md

@@ -1,5 +1,6 @@
 # <img src="https://raw.githubusercontent.com/seqeralabs/seqera-kit/main/assets/seqerakit.svg" width=50 alt="seqerakit logo"> seqerakit
 
+
 `seqerakit` is a Python wrapper for the [Seqera Platform CLI](https://github.com/seqeralabs/tower-cli). It can be leveraged to automate the creation of all of the entities in Seqera Platform via a simple configuration file in YAML format.
 
 The key features are:
@@ -10,7 +11,7 @@ The key features are:
 
 ## Prerequisites
 
-You will need to have an account on Seqera Platform (see [Plans and pricing](https://cloud.tower.nf/pricing/)).
+You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)).
 
 ## Installation
 
@@ -77,7 +78,7 @@ pip show seqerakit
 
 ## Configuration
 
-Create a Seqera Platform access token using the [Seqera Platform](https://tower.nf/) web interface via the **Your Tokens** page in your profile.
+Create a Seqera Platform access token using the [Seqera Platform](https://seqera.io/) web interface via the **Your Tokens** page in your profile.
 
 `seqerakit` reads this token from the environment variable `TOWER_ACCESS_TOKEN`. Please export it into your terminal as shown below:
 
@@ -89,7 +90,7 @@ For Enterprise installations of Seqera Platform, you will also need to configure
 ```bash
 export TOWER_API_ENDPOINT=<Tower API URL>
 ```
-By default, this is set to `https://api.tower.nf` to connect to Seqera Platform Cloud.
+By default, this is set to `https://api.cloud.seqera.io` to connect to Seqera Platform Cloud.
 
 
 ## Usage
@@ -109,6 +110,18 @@ Use `--version` or `-v` to retrieve the current version of your seqerakit instal
 ```bash
 seqerakit --version
 ```
+### Input
+`seqerakit` supports input through either file paths to YAMLs or directly from standard input (stdin).
+
+#### Using File Path
+```bash
+seqerakit /path/to/file.yaml
+```
+#### Using stdin
+```console
+$ cat file.yaml | seqerakit -
+```
+See the [Defining your YAML file using CLI options](#defining-your-yaml-file-using-cli-options) section for guidance on formatting your input YAML file(s).
 
 ### Dryrun
 
@@ -208,7 +221,7 @@ DEBUG:root: Running command: tw organizations add --name $SEQERA_ORGANIZATION_NA
 ### 3. Specifying JSON configuration files with `file-path`
 The Seqera Platform CLI allows export and import of entities through JSON configuration files for pipelines and compute environments. To use these files to add a pipeline or compute environment to a workspace, use the `file-path` key to specify a path to a JSON configuration file.
 
-An example of the `file-path` option is provided in the [compute-envs.yml](templates/compute-envs.yml) template:
+An example of the `file-path` option is provided in the [compute-envs.yml](./templates/compute-envs.yml) template:
 
 ```yaml
 compute-envs:
@@ -225,7 +238,7 @@ compute-envs:
 
 You must provide a YAML file that defines the options for each of the entities you would like to create in Seqera Platform.
 
-You will need to have an account on Seqera Platform (see [Plans and pricing](https://cloud.tower.nf/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.
+You will need to have an account on Seqera Platform (see [Plans and pricing](https://seqera.io/pricing/)). You will also need access to a Workspace and a pre-defined Compute Environment where you can launch a pipeline.
 
 ### Launch via YAML
 
@@ -252,7 +265,7 @@ You will need to have an account on Seqera Platform (see [Plans and pricing](htt
 
 You can also launch the same pipeline via a Python script. This will essentially allow you to extend the functionality on offer within the Seqera Platform CLI by leveraging the flexibility and customisation options available in Python.
 
-1. Download the [`launch_hello_world.py`](https://github.com/seqeralabs/seqera-kit/blob/main/examples/python/launch_hello_world.py) Python script and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required.
+1. Download the [`launch_hello_world.py`](./examples/python/launch_hello_world.py) Python script and customise the `<YOUR_WORKSPACE>` and `<YOUR_COMPUTE_ENVIRONMENT>` entries as required.
 
 2. Launch the pipeline with `seqerakit`:
 
@@ -319,24 +332,25 @@ In this example:
 - Ensure that the indentation and structure of the YAML file are correct - YAML is sensitive to formatting.
 - Use quotes around strings that contain special characters or spaces.
 - When listing multiple values (`labels`, `instance-types`, `allow-buckets`, etc), separate them with commas as shown above.
-- For complex configurations, refer to the [Templates](/templates/) provided in this repository.
+- For complex configurations, refer to the [Templates](./templates/) provided in this repository.
 
 
 ## Templates
 
 We have provided template YAML files for each of the entities that can be created on Seqera Platform. These can be found in the [`templates/`](https://github.com/seqeralabs/blob/main/seqera-kit/templates) directory and should form a good starting point for you to add your own customization:
 
-- [organizations.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/organizations.yml)
-- [teams.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/teams.yml)
-- [workspaces.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/workspaces.yml)
-- [participants.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/participants.yml)
-- [credentials.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/credentials.yml)
-- [secrets.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/secrets.yml)
-- [compute-envs.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/compute-envs.yml)
-- [actions.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/actions.yml)
-- [datasets.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/datasets.yml)
-- [pipelines.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/pipelines.yml)
-- [launch.yml](https://github.com/seqeralabs/seqera-kit/blob/main/templates/launch.yml)
+- [organizations.yml](./templates/organizations.yml)
+- [teams.yml](./templates/teams.yml)
+- [workspaces.yml](./templates/workspaces.yml)
+- [participants.yml](./templates/participants.yml)
+- [credentials.yml](./templates/credentials.yml)
+- [secrets.yml](./templates/secrets.yml)
+- [compute-envs.yml](./templates/compute-envs.yml)
+- [actions.yml](./templates/actions.yml)
+- [datasets.yml](./templates/datasets.yml)
+- [labels.yml](./templates/labels.yml)
+- [pipelines.yml](./templates/pipelines.yml)
+- [launch.yml](./templates/launch.yml)
 
 ## Real world example
 
@@ -358,7 +372,7 @@ $SENTIEON_LICENSE_BASE64
 
 ## Contributions and Support
 
-If you would like to contribute to `seqerakit`, please see the [contributing guidelines](https://github.com/seqeralabs/seqera-kit/blob/main/.github/CONTRIBUTING.md).
+If you would like to contribute to `seqerakit`, please see the [contributing guidelines](./.github/CONTRIBUTING.md).
 
 For further information or help, please don't hesitate to create an [issue](https://github.com/seqeralabs/seqera-kit/issues) in this repository.
 

examples/yaml/e2e/labels.yml

@@ -0,0 +1,5 @@
+labels:
+  - name: 'seqerakit'
+    value: 'test'
+    workspace: 'seqerakit-e2e/showcase'
+    overwrite: True

examples/yaml/e2e/members.yml

@@ -0,0 +1,4 @@
+members:
+   - user: 'laramie.lindsey+owner@seqera.io'
+     organization: 'seqerakit-e2e'
+     overwrite: False

seqerakit/cli.py

@@ -21,7 +21,6 @@
 import logging
 import sys
 
-from pathlib import Path
 from seqerakit import seqeraplatform, helper, overwrite
 from seqerakit.seqeraplatform import ResourceExistsError, ResourceCreationError
 from seqerakit import __version__
@@ -66,7 +65,6 @@ def parse_args(args=None):
     yaml_processing = parser.add_argument_group("YAML Processing Options")
     yaml_processing.add_argument(
         "yaml",
-        type=Path,
         nargs="*",
         help="One or more YAML files with Seqera Platform resource definitions.",
     )
@@ -105,7 +103,7 @@ def __init__(self, sp, list_for_add_method):
         # Create an instance of Overwrite class
         self.overwrite_method = overwrite.Overwrite(self.sp)
 
-    def handle_block(self, block, args, destroy=False):
+    def handle_block(self, block, args, destroy=False, dryrun=False):
         # Check if delete is set to True, and call delete handler
         if destroy:
             logging.debug(" The '--delete' flag has been specified.\n")
@@ -127,12 +125,12 @@ def handle_block(self, block, args, destroy=False):
 
         # Check if overwrite is set to True, and call overwrite handler
         overwrite_option = args.get("overwrite", False)
-        if overwrite_option:
+        if overwrite_option and dryrun is False:
             logging.debug(f" Overwrite is set to 'True' for {block}\n")
             self.overwrite_method.handle_overwrite(
                 block, args["cmd_args"], overwrite_option
             )
-        else:
+        elif dryrun is False:
             self.overwrite_method.handle_overwrite(block, args["cmd_args"])
 
         if block in self.list_for_add_method:
@@ -154,10 +152,14 @@ def main(args=None):
         return
 
     if not options.yaml:
-        logging.error(
-            " No YAML(s) provided. Please provide atleast one YAML configuration file."
-        )
-        sys.exit(1)
+        if sys.stdin.isatty():
+            logging.error(
+                " No YAML(s) provided and no input from stdin. Please provide "
+                "at least one YAML configuration file or pipe input from stdin."
+            )
+            sys.exit(1)
+        else:
+            options.yaml = [sys.stdin]
 
     # Parse CLI arguments into a list
     cli_args_list = options.cli_args.split() if options.cli_args else []
@@ -169,6 +171,8 @@ def main(args=None):
         [
             "organizations",  # all use method.add
             "workspaces",
+            "labels",
+            "members",
             "credentials",
             "secrets",
             "actions",
@@ -184,7 +188,9 @@ def main(args=None):
             for args in args_list:
                 try:
                     # Run the 'tw' methods for each block
-                    block_manager.handle_block(block, args, destroy=options.delete)
+                    block_manager.handle_block(
+                        block, args, destroy=options.delete, dryrun=options.dryrun
+                    )
                 except (ResourceExistsError, ResourceCreationError) as e:
                     logging.error(e)
                     sys.exit(1)

seqerakit/helper.py

@@ -19,6 +19,7 @@
 """
 import yaml
 from seqerakit import utils
+import sys
 
 
 def parse_yaml_block(yaml_data, block_name):
@@ -36,13 +37,14 @@ def parse_yaml_block(yaml_data, block_name):
     name_values = set()
 
     # Iterate over each item in the block.
+    # TODO: fix for resources that can be duplicate named in an org
     for item in block:
         cmd_args = parse_block(block_name, item)
         name = find_name(cmd_args)
         if name in name_values:
             raise ValueError(
-                f" Duplicate 'name' specified in config file"
-                f" for {block_name}: {name}. Please specify a unique name."
+                f" Duplicate name key specified in config file"
+                f" for {block_name}: {name}. Please specify a unique value."
             )
         name_values.add(name)
 
@@ -56,24 +58,48 @@ def parse_all_yaml(file_paths, destroy=False):
     # If multiple yamls, merge them into one dictionary
     merged_data = {}
 
-    for file_path in file_paths:
-        with open(file_path, "r") as f:
-            data = yaml.safe_load(f)
-
-            # Check if the YAML file is empty or contains no valid data
-            if data is None or not data:
-                raise ValueError(
-                    f" The file '{file_path}' is empty or does not contain valid data."
-                )
+    # Special handling for stdin represented by "-"
+    if not file_paths or "-" in file_paths:
+        # Read YAML directly from stdin
+        data = yaml.safe_load(sys.stdin)
+        if not data:
+            raise ValueError(
+                " The input from stdin is empty or does not contain valid YAML data."
+            )
+        merged_data.update(data)
 
-            for key, value in data.items():
+    for file_path in file_paths:
+        if file_path == "-":
+            continue
+        try:
+            with open(file_path, "r") as f:
+                data = yaml.safe_load(f)
+                if not data:
+                    raise ValueError(
+                        f" The file '{file_path}' is empty or "
+                        "does not contain valid data."
+                    )
+
+            for key, new_value in data.items():
                 if key in merged_data:
-                    try:
-                        merged_data[key].extend(value)
-                    except AttributeError:
-                        merged_data[key] = [merged_data[key], value]
+                    if isinstance(new_value, list) and all(
+                        isinstance(i, dict) for i in new_value
+                    ):
+                        # Handle list of dictionaries & merge without duplication
+                        existing_items = {
+                            tuple(sorted(d.items())) for d in merged_data[key]
+                        }
+                        for item in new_value:
+                            if tuple(sorted(item.items())) not in existing_items:
+                                merged_data[key].append(item)
+                    else:
+                        # override if not list of dictionaries
+                        merged_data[key] = new_value
                 else:
-                    merged_data[key] = value
+                    merged_data[key] = new_value
+        except FileNotFoundError:
+            print(f"Error: The file '{file_path}' was not found.")
+            sys.exit(1)
 
     block_names = list(merged_data.keys())
 
@@ -82,6 +108,8 @@ def parse_all_yaml(file_paths, destroy=False):
         "organizations",
         "teams",
         "workspaces",
+        "labels",
+        "members",
         "participants",
         "credentials",
         "compute-envs",
@@ -92,7 +120,7 @@ def parse_all_yaml(file_paths, destroy=False):
         "launch",
     ]
 
-    # Reverse the order of resources if destroy is True
+    # Reverse the order of resources to delete if destroy is True
     if destroy:
         resource_order = resource_order[:-1][::-1]
 
@@ -340,10 +368,29 @@ def handle_pipelines(sp, args):
 
 def find_name(cmd_args):
     """
-    Find and return the value associated with --name in the cmd_args list.
+    Find and return the value associated with --name in cmd_args, where cmd_args
+    can be a list, a tuple of lists, or nested lists/tuples.
+
+    The function searches for the following keys: --name, --user, --email.
+
+    Parameters:
+    - cmd_args: The command arguments (list, tuple, or nested structures).
+
+    Returns:
+    - The value associated with the first key found, or None if none are found.
     """
-    args_list = cmd_args.get("cmd_args", [])
-    for i in range(len(args_list) - 1):
-        if args_list[i] == "--name":
-            return args_list[i + 1]
-    return None
+    # Predefined list of keys to search for
+    keys = {"--name", "--user", "--email"}
+
+    def search(args):
+        it = iter(args)
+        for arg in it:
+            if isinstance(arg, str) and arg in keys:
+                return next(it, None)
+            elif isinstance(arg, (list, tuple)):
+                result = search(arg)
+                if result is not None:
+                    return result
+        return None
+
+    return search(cmd_args.get("cmd_args", []))