[develop] Add smoke and dust verification

.github/workflows/python_tests.yaml

@@ -50,6 +50,10 @@ jobs:
           micromamba activate srw_app
           export UNIT_TEST=True
           export PYTHONPATH=$(pwd)/ush
+          # Make dummy directories for fix files so generate_FV3LAM_workflow.py tests complete successfully
+          export CI_FIX_FILES=/home/runner/work
+          mkdir -p $CI_FIX_FILES/fix/fix_am
+
           python -m unittest tests/test_python/*.py
 
       - name: Run python functional tests

doc/UsersGuide/BuildingRunningTesting/RunSRW.rst

@@ -630,7 +630,9 @@ a staged forecast (e.g. from another forecasting system) need to add additional
 machine file (``ush/machine/<platform>.yaml``) or their ``config.yaml`` file. Other users may skip
 to the next step (:numref:`Section %s: Generate the SRW App Workflow <GenerateWorkflow>`). 
 
-To use METplus verification,  MET and METplus modules need to be installed on the system.
+To use METplus verification, MET and METplus modules need to be installed on the system. In order
+to ensure all capabilities work as expected, users should use at a minimum the latest release of
+these tools as of the latest SRW release: MET 12.0.1 and METplus 6.0.0.
 
 .. note::
    If users update their METplus installation, they must also update the module load statements in ``ufs-srweather-app/modulefiles/tasks/<machine>/run_vx.local`` to correspond to their system's updated installation:
@@ -796,6 +798,16 @@ fields they include are given in :numref:`Table %s <VXFieldGroupDescsTable>`.
    * - UPA
      - NDAS
      - Various upper-air fields (e.g. at 800 mb, 500 mb, etc)
+   * - AOD
+     - AERONET
+     - Aerosol Optical Depth
+   * - PM25
+     - AIRNOW
+     - Volumetric mass of particulate matter diameter 2.5 microns or less
+   * - PM10
+     - AIRNOW
+     - Volumetric mass of particulate matter diameter 10 microns or less
+
 
 The ``VX_FIELD_GROUPS`` list in the ``verification:`` section of ``config.yaml`` specifies the VX field
 groups for which to run verification. In order to avoid unwanted computation, the Rocoto XML will include 
@@ -820,7 +832,7 @@ summer period for which ``ASNOW`` is not relevant.
 Staging Observation Files
 ``````````````````````````````````
 The taskgroup in ``verify_pre.yaml`` defines a set of workflow tasks named ``get_obs_*``, where the ``*``
-represents any one of the supported obs types: :term:`CCPA`, :term:`NOHRSC`, :term:`MRMS`, and :term:`NDAS`. These ``get_obs_*`` tasks 
+represents any one of the supported obs types: :term:`CCPA`, :term:`NOHRSC`, :term:`MRMS`, :term:`NDAS`, :term:`AERONET`, and :term:`AIRNOW`. These ``get_obs_*`` tasks 
 will first check on disk for the existence of the obs files required for VX using the locations specified
 by the variables ``*_OBS_DIR`` and ``OBS_*_FN_TEMPLATES[1,3,...]`` in the ``verification:`` section of
 ``config.yaml``. The ``*_OBS_DIR``  are the base directories in which the obs files are or should be
@@ -846,14 +858,18 @@ place them in the locations specified by ``{*_OBS_DIR}/{OBS_*_FN_TEMPLATES[1,3,.
 that attempt is successful, the workflow will move on to subsequent tasks.  Thus:
 
    * Users who have the obs files already available (staged) on their system only need to set ``*_OBS_DIR``
-     and ``OBS_*_FN_TEMPLATES[1,3,...]`` in ``config.yaml`` to match those staging locations and file names.  
+     and ``OBS_*_FN_TEMPLATES[1,3,...]`` in ``config.yaml`` to match those staging locations and file names.
 
    * Users who do not have the obs files available on their systems and do not have access to NOAA HPSS
-     need to download :term:`CCPA`, :term:`NOHRSC`, :term:`MRMS`, and/or :term:`NDAS` files manually
-     from collections of publicly available data. 
+     need to download :term:`CCPA`, :term:`NOHRSC`, :term:`MRMS`, :term:`NDAS`, :term:`AERONET`, and/or
+     :term:`AIRNOW` files manually from collections of publicly available data.
      Then, as above, they must set ``*_OBS_DIR`` and ``OBS_*_FN_TEMPLATES[1,3,...]`` to match those
      staging locations and file names.
-
+
+.. note::
+   AIRNOW observations can be retrieved from AWS or HPSS, but retrieving from AWS requires changing some default settings.
+   See ``ush/config_defaults.yaml`` or :numref:`Section %s <GeneralVXParams>` for more details. 
+
    * Users who have access to a data store that hosts the necessary files (e.g. NOAA HPSS) do not need to
      manually stage the obs data because the ``get_obs_*`` tasks will retrieve the necessary obs and place
      them in the locations specified by ``*_OBS_DIR`` and ``OBS_*_FN_TEMPLATES[1,3,...]``.  By default,
@@ -872,12 +888,22 @@ and ``OBS_*_FN_TEMPLATES`` might be set as follows:
       NOHRSC_OBS_DIR: /path/to/UFS_SRW_data/develop/obs_data/nohrsc
       MRMS_OBS_DIR: /path/to/UFS_SRW_data/develop/obs_data/mrms
       NDAS_OBS_DIR: /path/to/UFS_SRW_data/develop/obs_data/ndas
+      AERONET_OBS_DIR: /path/to/UFS_SRW_data/develop/obs_data/aeronet
+      AIRNOW_OBS_DIR: /path/to/UFS_SRW_data/develop/obs_data/airnow
 
       OBS_CCPA_FN_TEMPLATES: [ 'APCP', '{valid?fmt=%Y%m%d}/ccpa.t{valid?fmt=%H}z.01h.hrap.conus.gb2' ]
       OBS_NOHRSC_FN_TEMPLATES: [ 'ASNOW', 'sfav2_CONUS_6h_{valid?fmt=%Y%m%d%H}_grid184.grb2' ]
       OBS_MRMS_FN_TEMPLATES: [ 'REFC', '{valid?fmt=%Y%m%d}/MergedReflectivityQCComposite_00.50_{valid?fmt=%Y%m%d}-{valid?fmt=%H%M%S}.grib2',
                                'RETOP', '{valid?fmt=%Y%m%d}/EchoTop_18_00.50_{valid?fmt=%Y%m%d}-{valid?fmt=%H%M%S}.grib2' ]
       OBS_NDAS_FN_TEMPLATES: [ 'SFC_UPA', 'prepbufr.ndas.{valid?fmt=%Y%m%d%H}' ]
+      OBS_AERONET_FN_TEMPLATES: [ 'AOD', '{valid?fmt=%Y%m%d}/{valid?fmt=%Y%m%d}.lev15' ]
+      OBS_AIRNOW_FN_TEMPLATES: [ 'PM', '{valid?fmt=%Y%m%d}/HourlyAQObs_{valid?fmt=%Y%m%d%H}.dat' ]
+
+.. note::
+   For AIRNOW obs retrieved from AWS (see ``parm/data_locations.yml``, the default value should be
+   replaced with:
+
+   ``OBS_AIRNOW_FN_TEMPLATES: [ 'PM', '{valid?fmt=%Y%m%d}/HourlyData_{valid?fmt=%Y%m%d%H}.dat' ]``
 
 Now further consider the CCPA obs type.  If one of the days encompassed by the forecast(s) is 20240429,
 then the ``get_obs_ccpa`` task associated with this day will check for the existence of the set of obs
@@ -1104,9 +1130,26 @@ interval (for cumulative fields such as accumulated precipitation), and the name
        from a data store (e.g. NOAA :term:`HPSS`) and place them in those locations.  This task is included
        in the workflow only if ``'SFC'`` and/or ``'UPA'`` are included in ``VX_FIELD_GROUPS``.
 
+   * - :bolditalic:`task_get_obs_aeronet` (``verify_pre.yaml``)
+     - Checks for existence of staged :term:`AERONET` obs files at locations specified by ``AERONET_OBS_DIR``
+       and ``OBS_AERONET_FN_TEMPLATES``.  If any files do not exist, it attempts to retrieve all the files
+       from a data store (e.g. NOAA :term:`HPSS`) and place them in those locations.  This task is included
+       in the workflow only if ``'AOD'``is included in ``VX_FIELD_GROUPS``.
+
+   * - :bolditalic:`task_get_obs_airnow` (``verify_pre.yaml``)
+     - Checks for existence of staged :term:`AIRNOW` obs files at locations specified by ``AIRNOW_OBS_DIR``
+       and ``OBS_AIRNOW_FN_TEMPLATES``.  If any files do not exist, it attempts to retrieve all the files
+       from a data store (e.g. NOAA :term:`HPSS`) and place them in those locations.  This task is included
+       in the workflow only if ``'PM25'`` and/or ``'PM10'`` are included in ``VX_FIELD_GROUPS``.
+
    * - :bolditalic:`task_run_MET_Pb2nc_obs_NDAS` (``verify_pre.yaml``)
      - Converts NDAS obs prepbufr files to NetCDF format.
 
+   * - :bolditalic:`metatask_ASCII2nc_obs` (``verify_pre.yaml``)
+     - Set of tasks that convert observations in ASCII text format to NetCDF files that can be processed by
+       :term:`METplus`; these observation types include AERONET and AIRNOW. This metatask is included in the
+       workflow only if ``'AOD'``, ``'PM25'``, or ``'PM10'`` are included in ``VX_FIELD_GROUPS``.
+
    * - :bolditalic:`metatask_PcpCombine_APCP_all_accums_obs_CCPA` (``verify_pre.yaml``)
      - Set of tasks that generate NetCDF files containing observed APCP for the accumulation intervals
        specified in ``VX_APCP_ACCUMS_HRS``.  Files for accumulation intervals larger than the one
@@ -1129,6 +1172,11 @@ interval (for cumulative fields such as accumulated precipitation), and the name
        the task is being run.  This metatask is included in the workflow only if ``'ASNOW'`` is included in
        ``VX_FIELD_GROUPS``.
 
+   * - :bolditalic:`metatask_PcpCombine_fcst_PM_all_mems` (``verify_pre.yaml``)
+     - Set of tasks that convert the raw forecast output of particulate matter into the appropriate bins for
+       PM 2.5 and PM10. This metatask is included in the workflow only if ``'PM25'`` or ``'PM10'`` is included in
+       ``VX_FIELD_GROUPS``.
+
    * - :bolditalic:`metatask_check_post_output_all_mems` (``verify_pre.yaml``)
      - Set of tasks that ensure that the post-processed forecast files required for verification exist in
        the locations specified by ``VX_FCST_INPUT_BASEDIR``, ``FCST_SUBDIR_TEMPLATE``, and ``FCST_FN_TEMPLATE``.

doc/UsersGuide/CustomizingTheWorkflow/ConfigWorkflow.rst

@@ -199,8 +199,8 @@ These directories are used only by the ``run_WE2E_tests.py`` script, so they are
 ``TEST_ALT_EXTRN_MDL_SYSBASEDIR_ICS``, ``TEST_ALT_EXTRN_MDL_SYSBASEDIR_LBCS``: (Default: "")
    These parameters are used by the testing script to test the mechanism that allows users to point to a data stream on disk. They set up a sandbox location that mimics the stream in a more controlled way and test the ability to access :term:`ICS` or :term:`LBCS`, respectively. 
 
-``TEST_CCPA_OBS_DIR``, ``TEST_MRMS_OBS_DIR``, ``TEST_NDAS_OBS_DIR``: (Default: "")
-These parameters are used by the testing script to test the mechanism that allows user to point to data streams on disk for observation data for verification tasks. They test the ability for users to set ``CCPA_OBS_DIR``, ``MRMS_OBS_DIR``, and ``NDAS_OBS_DIR`` respectively.
+``TEST_CCPA_OBS_DIR``, ``TEST_MRMS_OBS_DIR``, ``TEST_NDAS_OBS_DIR``, ``TEST_NOHRSC_OBS_DIR``, ``TEST_AERONET_OBS_DIR``, ``TEST_AIRNOW_OBS_DIR``: (Default: "")
+These parameters are used by the testing script to test the mechanism that allows user to point to data streams on disk for observation data for verification tasks. They test the ability for users to set ``CCPA_OBS_DIR``, ``MRMS_OBS_DIR``, ``NDAS_OBS_DIR``, etc.
 
 ``TEST_VX_FCST_INPUT_BASEDIR``: (Default: "") 
    The path to user-staged forecast files for WE2E testing of verificaton using user-staged forecast files in a known location on a given platform. 
@@ -633,7 +633,7 @@ Pre-Existing Directory Parameter
 
    * **"delete":** The preexisting directory is deleted and a new directory (having the same name as the original preexisting directory) is created.
 
-   * **"rename":** The preexisting directory is renamed and a new directory (having the same name as the original pre-existing directory) is created. The new name of the preexisting directory consists of its original name and the suffix "_old###", where ``###`` is a 3-digit integer chosen to make the new name unique.
+   * **"rename":** The preexisting directory is renamed and a new directory (having the same name as the original pre-existing directory) is created. The new name of the preexisting directory consists of its original name and the suffix "_old_YYYYMMDD_HHmmss", where ``YYYYMMDD_HHmmss`` is the full date and time of the rename
 
    * **"reuse":** This method will keep the preexisting directory intact. However, when the preexisting directory is ``$EXPDIR``, this method will save all old files to a subdirectory ``oldxxx/`` and then populate new files into the ``$EXPDIR`` directory. This is useful to keep ongoing runs uninterrupted; rocotoco ``*db`` files and previous cycles will stay and hence there is no need to manually copy or move ``*db`` files and previous cycles back, and there is no need to manually restart related rocoto tasks failed during the workflow generation process. This method may be best suited for incremental system reuses.
 
@@ -1676,7 +1676,7 @@ METplus-Specific Parameters
 -----------------------------------
 
 ``METPLUS_VERBOSITY_LEVEL``: (Default: ``2``)
-   Logging verbosity level used by METplus verification tools. Valid values: 0 to 5, with 0 quiet and 5 loudest.
+   Logging verbosity level used by METplus verification tools. Valid values: 0 to 9, with 0 having the fewest log messages and 9 having the most. Levels 5 and above can result in very large log files and slower tool execution.
 
 
 VX Parameters for Observations
@@ -1691,11 +1691,13 @@ VX Parameters for Observations
       * NOHRSC (National Operational Hydrologic Remote Sensing Center)
       * MRMS (Multi-Radar Multi-Sensor)
       * NDAS (NAM Data Assimilation System)
+      * AERONET (Aerosol Robotic Network)
+      * AIRNOW (AirNow air quality reports)
 
    The script ``ush/get_obs.py`` contains further details on the files and
    directory structure of each obs type.
 
-``[CCPA|NOHRSC|MRMS|NDAS]_OBS_AVAIL_INTVL_HRS``: (Defaults: [1|6|1|1])
+``[CCPA|NOHRSC|MRMS|NDAS|AERONET|AIRNOW]_OBS_AVAIL_INTVL_HRS``: (Defaults: [1|6|1|1|24|1])
   Time interval (in hours) at which the various types of obs are available
   on NOAA's HPSS. 
 
@@ -1704,8 +1706,8 @@ VX Parameters for Observations
   is the shortest output interval for forecasts, i.e. the forecasts cannot
   (yet) support sub-hourly output.
 
-``[CCPA|NOHRSC|MRMS|NDAS]_OBS_DIR``: (Default: ``"{{ workflow.EXPTDIR }}/obs_data/[ccpa|nohrsc|mrms|ndas]"``)
-   Base directory in which CCPA, NOHRSC, MRMS, or NDAS obs files needed by
+``[CCPA|NOHRSC|MRMS|NDAS|AERONET|AIRNOW]_OBS_DIR``: (Default: ``"{{ workflow.EXPTDIR }}/obs_data/[ccpa|nohrsc|mrms|ndas|aeronet|airnow]"``)
+   Base directory in which CCPA, NOHRSC, MRMS, NDAS, AERONET, or AIRNOW obs files needed by
    the verification tasks are located.  If the files do not exist, they
    will be retrieved and placed under this directory.  Note that:
 
@@ -1718,7 +1720,7 @@ VX Parameters for Observations
      that need to be corrected during obs retrieval.  This is described
      in more detail in the script ``ush/get_obs.py``.
 
-``OBS_[CCPA|NOHRSC|MRMS|NDAS]_FN_TEMPLATES``:
+``OBS_[CCPA|NOHRSC|MRMS|NDAS|AERONET|AIRNOW]_FN_TEMPLATES``:
      **Defaults:**
 
      ``OBS_CCPA_FN_TEMPLATES``:
@@ -1746,6 +1748,16 @@ VX Parameters for Observations
 
            [ 'SFCandUPA', 'prepbufr.ndas.{valid?fmt=%Y%m%d%H}' ]
 
+     ``OBS_AERONET_FN_TEMPLATES``:
+        .. code-block:: console
+
+           [ 'AOD', '{valid?fmt=%Y%m%d}/{valid?fmt=%Y%m%d}.lev15' ]
+
+     ``OBS_AIRNOW_FN_TEMPLATES``:
+        .. code-block:: console
+
+           [ 'PM', '{valid?fmt=%Y%m%d}/HourlyData_{valid?fmt=%Y%m%d%H}.dat' ]
+
    File name templates for various obs types.  These are meant to be used
    in METplus configuration files and thus contain METplus time formatting
    strings.  Each of these variables is a python list containing pairs of
@@ -1854,7 +1866,7 @@ VX Parameters for Observations
       {{- "ccpa.t{valid?fmt=%H}z." ~ obs_avail_intvl_hrs ~ "h.hrap.conus.gb2_a${ACCUM_HH}h.nc" }}
 
    METplus template for the names of the NetCDF files generated by the
-   worfklow verification tasks that call METplus's PcpCombine tool on
+   workflow verification tasks that call METplus's PcpCombine tool on
    CCPA observations.  These files will contain observed accumulated
    precipitation in NetCDF format for various accumulation intervals.
 
@@ -1867,26 +1879,44 @@ VX Parameters for Observations
       {{- "sfav2_CONUS_" ~ obs_avail_intvl_hrs ~ "h_{valid?fmt=%Y%m%d%H}_grid184.grb2_a${ACCUM_HH}h.nc" }}
 
    METplus template for the names of the NetCDF files generated by the
-   worfklow verification tasks that call METplus's PcpCombine tool on
+   workflow verification tasks that call METplus's PcpCombine tool on
    NOHRSC observations.  These files will contain observed accumulated
-   snowfall for various accumulaton intervals.
+   snowfall for various accumulation intervals.
+
+``OBS_AERONET_FN_TEMPLATE_ASCII2NC_OUTPUT``: (Default: ``'hourly_aeronet_obs_{valid?fmt=%Y%m%d}00.nc'``)
+
+   METplus template for the names of the NetCDF files generated by the
+   workflow verification tasks that call METplus's ASCII2NC tool on
+   AERONET observations.
+
+``OBS_AIRNOW_FN_TEMPLATE_ASCII2NC_OUTPUT``: (Default: ``'hourly_airnow_obs_{valid?fmt=%Y%m%d}00.nc'``)
+
+   METplus template for the names of the NetCDF files generated by the
+   workflow verification tasks that call METplus's ASCII2NC tool on
+   AIRNOW observations.  
+
+``AIRNOW_INPUT_FORMAT``: (Default: ``"airnowhourlyaqobs"``)
+   Observation format for ASCII Airnow observations. Observations retrieved from HPSS are in
+   "airnowhourlyaqobs" format, observations retrieved from AWS are generally in "airnowhourly" format.
+   For more information see the
+   `METplus users guide <https://met.readthedocs.io/en/latest/Users_Guide/reformat_point.html#ascii2nc-tool>`_
+
+
+``OBS_DATA_STORE_AIRNOW``: (Default: ``hpss``)
+   Location to retrieve observation data from. Valid values are "aws" and/or "hpss", see
+   ``parm/data_locations.yaml`` for info on these data stores.
 
 ``OBS_NDAS_SFCandUPA_FN_TEMPLATE_PB2NC_OUTPUT``: (Default: ``'${OBS_NDAS_FN_TEMPLATES[1]}.nc'``)
    METplus template for the names of the NetCDF files generated by the
-   worfklow verification tasks that call METplus's Pb2nc tool on the 
+   workflow verification tasks that call METplus's Pb2nc tool on the 
    prepbufr files in NDAS observations.  These files will contain the
    observed surface (SFC) and upper-air (UPA) fields in NetCDF format
    (instead of NDAS's native prepbufr format).
 
-``NUM_MISSING_OBS_FILES_MAX``: (Default: 2)
+``NUM_MISSING_OBS_FILES_MAX``: (Default: 0)
    For verification tasks that need observational data, this specifies
    the maximum number of observation files that may be missing.  If more
    than this number are missing, the verification task will error out.
-   This is a crude way of checking that there are enough obs to conduct
-   verification (crude because this number should probably depend on the
-   field being verified, the time interval between observations, the
-   length of the forecast, etc; an alternative may be to specify the
-   maximum allowed fraction of obs files that can be missing).
 
 
 VX Parameters for Forecasts
@@ -1955,7 +1985,7 @@ VX Parameters for Forecasts
       {{- ".prslev.f{lead?fmt=%HHH}.${POST_OUTPUT_DOMAIN_NAME}_${VAR}_a${ACCUM_HH}h.nc" }}
 
    METplus template for the names of the NetCDF files generated by the
-   worfklow verification tasks that call METplus's PcpCombine tool on
+   workflow verification tasks that call METplus's PcpCombine tool on
    forecast output.  These files will contain forecast accumulated
    precipitation in NetCDF format for various accumulation intervals.