Merge remote-tracking branch 'upstream-main/main' into main

.github/workflows/push-pr_workflow.yml

@@ -56,9 +56,9 @@ jobs:
 
     - name: Lint with Black
       run: |
-        python3 -m black --check --line-length $MAX_LINE_LENGTH --target-version py36 merlin
-        python3 -m black --check --line-length $MAX_LINE_LENGTH --target-version py36 tests
-        python3 -m black --check --line-length $MAX_LINE_LENGTH --target-version py36 *.py
+        python3 -m black --check --line-length $MAX_LINE_LENGTH --target-version py38 merlin
+        python3 -m black --check --line-length $MAX_LINE_LENGTH --target-version py38 tests
+        python3 -m black --check --line-length $MAX_LINE_LENGTH --target-version py38 *.py
 
     - name: Lint with PyLint
       run: |
@@ -129,7 +129,7 @@ jobs:
 
     - name: Run pytest over unit test suite
       run: |
-        python3 -m pytest tests/unit/
+        python3 -m pytest -v --order-scope=module tests/unit/
 
     - name: Run integration test suite for local tests
       run: |

.readthedocs.yaml

@@ -5,11 +5,11 @@ build:
   tools:
     python: "3.8"
 
-sphinx:
-  configuration: docs/source/conf.py
-
 python:
   install:
     - requirements: docs/requirements.txt
 
+mkdocs:
+  fail_on_warning: false
+
 formats: [pdf]

CHANGELOG.md

@@ -4,6 +4,83 @@ All notable changes to Merlin will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.12.0]
+### Added
+- A new command `merlin queue-info` that will print the status of your celery queues
+  - By default this will only pull information from active queues
+  - There are options to look for specific queues (`--specific-queues`), queues defined in certain spec files (`--spec`; this is the same functionality as the `merlin status` command prior to this update), and queues attached to certain steps (`--steps`)
+  - Queue info can be dumped to outfiles with `--dump`
+- A new command `merlin detailed-status` that displays task-by-task status information about your study
+  - This has options to filter by return code, task queues, task statuses, and workers
+  - You can set a limit on the number of tasks to display
+  - There are 3 options to modify the output display
+- Docs for all of the monitoring commands
+- New file `merlin/study/status.py` dedicated to work relating to the status command
+  - Contains the Status and DetailedStatus classes
+- New file `merlin/study/status_renderers.py` dedicated to formatting the output for the detailed-status command
+- New file `merlin/common/dumper.py` containing a Dumper object to help dump output to outfiles
+- Study name and parameter info now stored in the DAG and MerlinStep objects
+- Added functions to `merlin/display.py` that help display status information:
+  - `display_task_by_task_status` handles the display for the `merlin detailed-status` command
+  - `display_status_summary` handles the display for the `merlin status` command
+  - `display_progress_bar` generates and displays a progress bar
+- Added new methods to the MerlinSpec class:
+  - get_worker_step_map()
+  - get_queue_step_relationship()
+  - get_tasks_per_step()
+  - get_step_param_map()
+- Added methods to the MerlinStepRecord class to mark status changes for tasks as they run (follows Maestro's StepRecord format mostly)
+- Added methods to the Step class:
+  - establish_params()
+  - name_no_params()
+- Added a property paramater_labels to the MerlinStudy class
+- Added two new utility functions:
+  - dict_deep_merge() that deep merges two dicts into one
+  - ws_time_to_dt() that converts a workspace timestring (YYYYMMDD-HHMMSS) to a datetime object
+- A new celery task `condense_status_files` to be called when sets of samples finish
+- Added a celery config setting `worker_cancel_long_running_tasks_on_connection_loss` since this functionality is about to change in the next version of celery
+- Tests for the Status and DetailedStatus classes
+  - this required adding a decent amount of test files to help with the tests; these can be found under the tests/unit/study/status_test_files directory
+- Pytest fixtures in the `conftest.py` file of the integration test suite
+  - NOTE: an export command `export LC_ALL='C'` had to be added to fix a bug in the WEAVE CI. This can be removed when we resolve this issue for the `merlin server` command
+- Tests for the `celeryadapter.py` module
+- New CeleryTestWorkersManager context to help with starting/stopping workers for tests
+
+### Changed
+- Reformatted the entire `merlin status` command
+  - Now accepts both spec files and workspace directories as arguments
+  - Removed the --steps flag
+  - Replaced the --csv flag with the --dump flag
+  - New functionality:
+    - Shows step_by_step progress bar for tasks
+    - Displays a summary of task statuses below the progress bar
+- Split the `add_chains_to_chord` function in `merlin/common/tasks.py` into two functions:
+  - `get_1d_chain` which converts a 2D list of chains into a 1D list
+  - `launch_chain` which launches the 1D chain
+- Pulled the needs_merlin_expansion() method out of the Step class and made it a function instead
+- Removed `tabulate_info` function; replaced with tabulate from the tabulate library
+- Moved `verify_filepath` and `verify_dirpath` from `merlin/main.py` to `merlin/utils.py`
+- The entire documentation has been ported to MkDocs and re-organized
+  - *Dark Mode*
+  - New "Getting Started" example for a simple setup tutorial
+  - More detail on configuration instructions
+  - There's now a full page on installation instructions
+  - More detail on explaining the spec file
+  - More detail with the CLI page
+  - New "Running Studies" page to explain different ways to run studies, restart them, and accomplish command line substitution
+  - New "Interpreting Output" page to help users understand how the output workspace is generated in more detail
+  - New "Examples" page has been added
+  - Updated "FAQ" page to include more links to helpful locations throughout the documentation
+  - Set up a place to store API docs
+  - New "Contact" page with info on reaching Merlin devs
+- The Merlin tutorial defaults to using Singularity rather than Docker for the OpenFoam example. Minor tutorial fixes have also been applied.
+
+### Fixed
+- The `merlin status` command so that it's consistent in its output whether using redis or rabbitmq as the broker
+- The `merlin monitor` command will now keep an allocation up if the queues are empty and workers are still processing tasks
+- Add the restart keyword to the specification docs
+- Cyclical imports and config imports that could easily cause ci issues
+
 ## [1.11.1]
 ### Fixed
 - Typo in `batch.py` that caused lsf launches to fail (`ALL_SGPUS` changed to `ALL_GPUS`)

Makefile

@@ -6,7 +6,7 @@
 #
 # LLNL-CODE-797170
 # All rights reserved.
-# This file is part of Merlin, Version: 1.11.1.
+# This file is part of Merlin, Version: 1.12.0.
 #
 # For details, see https://github.com/LLNL/merlin.
 #
@@ -87,7 +87,7 @@ install-dev: virtualenv install-merlin install-workflow-deps
 # tests require a valid dev install of merlin
 unit-tests:
 	. $(VENV)/bin/activate; \
-	$(PYTHON) -m pytest $(UNIT); \
+	$(PYTHON) -m pytest -v --order-scope=module $(UNIT); \
 
 
 # run CLI tests - these require an active install of merlin in a venv
@@ -135,9 +135,9 @@ check-flake8:
 
 check-black:
 	. $(VENV)/bin/activate; \
-	$(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version py36 $(MRLN); \
-	$(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version py36 $(TEST); \
-	$(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version py36 *.py; \
+	$(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version py38 $(MRLN); \
+	$(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version py38 $(TEST); \
+	$(PYTHON) -m black --check --line-length $(MAX_LINE_LENGTH) --target-version py38 *.py; \
 
 
 check-isort:
@@ -179,9 +179,9 @@ fix-style:
 	$(PYTHON) -m isort -w $(MAX_LINE_LENGTH) $(MRLN); \
 	$(PYTHON) -m isort -w $(MAX_LINE_LENGTH) $(TEST); \
 	$(PYTHON) -m isort -w $(MAX_LINE_LENGTH) *.py; \
-	$(PYTHON) -m black --target-version py36 -l $(MAX_LINE_LENGTH) $(MRLN); \
-	$(PYTHON) -m black --target-version py36 -l $(MAX_LINE_LENGTH) $(TEST); \
-	$(PYTHON) -m black --target-version py36 -l $(MAX_LINE_LENGTH) *.py; \
+	$(PYTHON) -m black --target-version py38 -l $(MAX_LINE_LENGTH) $(MRLN); \
+	$(PYTHON) -m black --target-version py38 -l $(MAX_LINE_LENGTH) $(TEST); \
+	$(PYTHON) -m black --target-version py38 -l $(MAX_LINE_LENGTH) *.py; \
 
 
 # Increment the Merlin version. USE ONLY ON DEVELOP BEFORE MERGING TO MASTER.

README.md

@@ -62,7 +62,7 @@ HPC batch systems, since it can scale to a very large number of jobs.
 
 The integrated system looks a little something like this:
 
-<img src="docs/images/merlin_arch.png" alt="a typical Merlin workflow">
+![A Typical Merlin Workflow](docs/assets/images/merlin_arch.png)
 
 In this example, here's how it all works:
 

docs/README.md

@@ -0,0 +1,44 @@
+# Guide to Merlin Documentation
+
+Merlin uses [MkDocs](https://www.mkdocs.org/) to generate its documentation and [Read the Docs](https://about.readthedocs.com/?ref=readthedocs.com) to host it. This README will detail important information on handling the documentation.
+
+## How to Build the Documentation
+
+Ensure you're at the root of the Merlin repository:
+
+```bash
+cd /path/to/merlin/
+```
+
+Install the documentation with:
+
+```bash
+pip install -r docs/requirements.txt
+```
+
+Build the documentation with:
+
+```bash
+mkdocs serve
+```
+
+Once up and running, MkDocs should provide a message telling you where your browser is connected (this is typically `http://127.0.0.1:8000/`). If you're using VSCode, you should be able to `ctrl+click` on the address to open the browser window. An example is shown below:
+
+```bash
+(venv_name) [user@machine:merlin]$ mkdocs serve
+INFO    -  Building documentation...
+INFO    -  Cleaning site directory
+WARNING -  Excluding 'README.md' from the site because it conflicts with 'index.md'.
+WARNING -  A relative path to 'api_reference/' is included in the 'nav' configuration, which is not found in the documentation files.
+INFO    -  Documentation built in 3.24 seconds
+INFO    -  [09:16:00] Watching paths for changes: 'docs', 'mkdocs.yml'
+INFO    -  [09:16:00] Serving on http://127.0.0.1:8000/
+```
+
+## Configuring the Documentation
+
+MkDocs relies on an `mkdocs.yml` file for almost everything to do with configuration. See [their Configuration documentation](https://www.mkdocs.org/user-guide/configuration/) for more information.
+
+## How Do API Docs Work?
+
+Coming soon...

docs/api_reference/index.md

@@ -0,0 +1,5 @@
+# Merlin API Reference
+
+Coming soon!
+
+<!-- Welcome to the Application Program Interface (API) Reference Guide for Merlin! This comprehensive guide is designed to provide developers with a detailed understanding of the various modules, classes, and functions available within the Merlin API. -->

docs/assets/javascripts/swap_lp_image.js

@@ -0,0 +1,26 @@
+/*
+ * This script is for swapping the landing page image between
+ * one with black text (for light mode) and one with white
+ * text (for dark mode).
+ */
+
+function swapLandingPageImage() {
+    // Get the current theme (should be "slate" or "default")
+    const colorSchemeValue = document.body.getAttribute("data-md-color-scheme");
+
+    // Get the image element
+    const imageElement = document.getElementById("landing-page-image");
+
+    // Paths for light/dark mode images
+    const lightModeImgPath = "assets/images/merlin_banner.png";
+    const darkModeImgPath = "assets/images/merlin_banner_white.png";
+
+    // Set the image source based on the color scheme
+    imageElement.src = colorSchemeValue == "slate" ? darkModeImgPath : lightModeImgPath;
+}
+
+// Set up an observer to watch for theme changes
+const observer = new MutationObserver(swapLandingPageImage);
+const targetNode = document.body;
+const config = {attributes: true, childList: true};
+observer.observe(targetNode, config);

docs/assets/stylesheets/extra.css

@@ -0,0 +1,8 @@
+[data-md-color-scheme=slate] {
+    --md-default-bg-color: hsla(var(--md-hue),15%,22%,1);
+}
+
+[data-md-color-primary=black] .md-button--primary {
+    background-color: hsla(202, 97%, 34%, 1);  /* This is the same as #036BA9 (the hex color of Merlin's hat logo) */
+    border-color: hsla(202, 97%, 34%, 1);
+}

docs/contact.md

@@ -0,0 +1,21 @@
+---
+hide:
+  - navigation
+---
+
+# Contact Us
+
+!!! question "Have a Question?"
+
+    :material-email: [merlin@llnl.gov](mailto:merlin@llnl.gov)
+
+    :simple-microsoftteams: LC Teams Channels:
+
+    - [Merlin Support Page](https://teams.microsoft.com/l/team/19%3a8efc6658663c46aca4bfca4f11df7eb3%40thread.skype/conversations?groupId=7c7f88bc-2485-460f-90e2-4f6880b5ed3e&tenantId=a722dec9-ae4e-4ae3-9d75-fd66e2680a63)
+
+    - [WEAVE's Merlin Support Page](https://teams.microsoft.com/l/channel/19%3aa677f3ebd1d9463b9128e9609a5b4177%40thread.tacv2/Merlin?groupId=a56ae26c-fe55-4e8e-8773-dd7975f648a4&tenantId=a722dec9-ae4e-4ae3-9d75-fd66e2680a63)
+
+
+!!! note "Submit an Issue"
+
+    :simple-github: [Report an Issue on GitHub](https://github.com/LLNL/merlin/issues)

docs/examples/feature_demo.md

@@ -0,0 +1,3 @@
+# Feature Demo Examples
+
+Coming soon!

docs/examples/flux.md

@@ -0,0 +1,3 @@
+# Flux Examples
+
+Coming soon!