Add HPSS support and fix memory unsetting for Gaea C5/6

[DavidHuber-NOAA]

Description

This adds HPSS support for the Gaea clusters by utilizing the es cluster's dtn_f5_f6 partition, which has an HPSS connection. A number of small fixes and some refactoring were also introduced in this PR including

• Fixed memory variable unsetting for Gaea C5/6 in config.resources.GAEAC{5,6}
• Refactoring the system-level parameter detection when determining task resources in the setup scripts to make it easier to define multiple partitions, queues, and clusters.
• Adding a DTN partition, queue, and cluster definition.
• Added/renamed missing/miss-named tasks to tasks.py and added a check that the input task is valid.

NOTE: Archiving from the DTNs for files located on the f6 filesystem is excruciatingly slow and can bog down both C5 and C6. Thus, it is recommended to not use HPSS at this time on Gaea/C6. Therefore, the option is disabled by default. According to system admins, there should be new DTNs installed soon that will help alleviate this issue.

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? NO

How has this been tested?

• C48_ATM on C6
• C48_S2SW on C6
• Cycle testing on C6
• CI suite on Hera
• CI suite on WCOSS2

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary

[DavidHuber-NOAA]

This PR is now ready to be reviewed again. I will be adding some documentation for scrontab and providing the results from a 40-cycle test tomorrow, but the technical details are all ready to go and tested.

[DavidHuber-NOAA]

@TerrenceMcGuinness-NOAA @DavidBurrows-NCO @AnilKumar-NOAA For grins, may I go ahead and add a CI label for Gaea C6?

[aerorahul]

I think the Gaea C6 node is named Gaeac6-EMC

[aerorahul]

I don't think @TerrenceMcGuinness-NOAA has updated his draft PR. He only commented on the change.

[DavidHuber-NOAA]

@junwang-noaa @RussTreadon-NOAA @CoryMartin-NOAA I have finished a retro simulation on Gaea C6. Would you mind taking a look at and validate the forecast and analysis product files in /gpfs/f6/drsa-hurr1/world-shared/noscrub/David.Huber/archive/C96C48_hybatmDA for the time period 2021122000 through 2021123100 (with the exception of the GFS cycle 2021122200 products, which I mistakenly failed to generate).

If you would like to look at the logs and/or the COM directories, they can be found in /gpfs/f6/drsa-hurr1/world-shared/noscrub/David.Huber/para/COMROOT/C96C48_hybatmDA.

[DavidHuber-NOAA]

Launching CI on Hera.