JEDI-based ensemble recentering and analysis calculation

[DavidNew-NOAA]

Description

COORDINATED MERGE

This PR implements ensemble recentering and analysis calculation in the Global Workflow, using JEDI-based applications to replace certain GSI utilities when JEDI is turned on in the workflow. If using GSI, then the workflow will remain unchanged. This PR also (finally) implements native-grid DA increments into the worflow.

The gdas_analcalc and enkfgdas_ecen jobs will be replaced by gdas_analcalc_fv3jedi and enkfgdas_ecen_fv3jedi jobs respectively. The enkfgdas_echgres job is eliminated, since changing of resolution of the deterministic backgrounds is done internally in the JEDI-based recentering application.

The design for this PR is based on discussions between the DA team and GW team a few months ago. Explanation of the flow of data through the workflow:

The gdas_analcalc_fv3jedi job dependencies do not change. The native-grid backgrounds andincrements are staged, and then the GDASApp JEDI fv3jedi_add_increments application is run to add them and interpolate to the Gaussian grid. The Gaussian-grid backgrounds are also staged, and then a simple Python function inserts to these analysis variables into the histories, which become the Gaussian analyses. This is done this way to guarantee that the resulting Gaussian analyses are in the exact format required by UPP.

The enkfgdas_ecen_fv3jedi no longer depends on the analysis calc job, since the ensemble-resolution variational analysis is computed/interpolated internally in the JEDI-based recentering application. All other job dependencies remain the same. We no longer need to compute the ensemble mean analysis in this job, since it can be outputted the the JEDI local ensemble DA application in the enkfgdas_atmensanlsol job and just staged for recentering. The variational increment and deterministic backgrounds are also staged to compute the ensemble-resolution variational analysis. The output of this job is no longer the recentered ensemble increments, but rather the "correction increment", which when added to ensemble increments becomes the recentered increments. The prefix for the "correction increment" is catminc.

The enkfgdas_fcst job now stages both the ensemble increments and the correction increment. They are added together with ncbo in forecast_postdet.sh to generate the recentered increment.

All forecast increments, both deterministic and ensemble, are now on the native cubed-sphere grid

Resolves #3248

Type of change

• Bug fix (fixes something broken)
• New feature (adds functionality)
• Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

• Is this a breaking change (a change in existing functionality)? YES
• Does this change require a documentation update? IDK
• Does this change require an update to any of the following submodules? YES
  • EMC verif-global
  • GDAS #1488
  • GFS-utils
  • GSI
  • GSI-monitor
  • GSI-utils
  • UFS-utils
  • UFS-weather-model
  • wxflow

How has this been tested?

• Clone and build on Hera
• Run C96C48_ufs_hybatmDA

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have documented my code, including function, input, and output descriptions
• My changes generate no new warnings
• New and existing tests pass with my changes
• This change is covered by an existing CI test or a new one has been added
• Any new scripts have been added to the .github/CODEOWNERS file with owners
• I have made corresponding changes to the system documentation if necessary