cpld_gridgen - Update regression tests to run in parallel

[GeorgeGayno-NOAA]

DESCRIPTION OF CHANGES:

Currently, each regression test runs in sequence. These updates submit all tests to the batch queue at once, where they run independently. Then, when they are finished, a 'summary' job is triggered to create some final log files, check for errors and create a "summary" log. All other functions, such as options to build the executable (rt.sh -b), create a new baseline (rt.sh -c) and test against the new baseline (rt.sh -m) are retained.

Some specifics:

• The check_results function was moved from rt.sh to ./ush/cpld_gridgen.sh
• New script rt_summary.sh is called from rt.sh. It checks for errors, concatenates the log files from the individual tests to a final log file, checks for errors and creates a summary file.
• Removed some now unnecessary exports from the scripts.

TESTS CONDUCTED:

If there are changes to the build or source code, the tests below must be conducted. Contact a repository manager if you need assistance.

• Compile branch on all Tier 1 machines using Intel (Orion, Jet, Hera, Hercules and WCOSS2).
• Compile branch on Hera using GNU. Done using 9c15d3a.
• Compile branch in 'Debug' mode on WCOSS2. Done using 9c15d3a.
• Compile with Doxygen on any machine with no errors. Done on Hera using 9c15d3a.
• Run unit tests locally on any Tier 1 machine. Done on Hera using 9c15d3a. All tests passed.
• Run cpld_gridgen consistency tests locally on all Tier 1 machines.
• Run cpld_gridgen consistency tests using the -b option on any Tier 1 machine to test the build executable option.
• Run cpld_gridgen consistency tests using the -c option on any Tier 1 machine to create a new baseline. Then run it again using the -m option to compare against the new baseline.

DEPENDENCIES:

None.

DOCUMENTATION:

All new and updated source code must be documented with Doxygen.

• Doxygen is updated.

If this PR is contributing new capabilities that need to be documented, please also include updates to the RST files in the docs/source directory as supporting material.

ISSUE:

Fixes #1016.

[GeorgeGayno-NOAA]

@DeniseWorthen - I would like you to review this. No hurry. The overall functionality is unchanged.

[GeorgeGayno-NOAA]

I still have some final cleanup to do.

[DeniseWorthen]

Oh that's great! I'll take a look when I've got a chance.

[GeorgeGayno-NOAA]

Besides the standard tests listed in the description, other tests were performed to ensure that all functionality is retained.

Using 9c15d3a on Hercules, the executable directory was removed. When rt.sh was invoked, the correct error message was display and processing stopped:

cpld_gridgen exe file is not found in /work2/noaa/da/ggayno/save/UFS_UTILS/exe/. Try -b to build or -h for help.

[GeorgeGayno-NOAA]

Using 9c15d3a, when ./rt.sh is invoked with the -h option, the correct message is displayed:

Usage: rt.sh [-c] [-m] [-h] [-b]

  -b build the executable

  -c create a new baseline

  -m compare against the new baseline

  -h display this help and exit

  Examples

    './rt.sh -b'  build exe file. compare against the existing baseline
    './rt.sh -bc' build exe file. create a new baseline
    './rt.sh -m'  do not build exe file. compare against the new baseline