[develop:] Add the RRFS_SaS physics suite to SRW options

[natalie-perlin]

DESCRIPTION OF CHANGES:

RRFS_sas suite is added to the SRW physics suite options.
This scheme was used in RRFSv1 production runs between ~August to December 2024.
Fractional approach in RUC_LSF not yet used (mosaic_lu=0, mosaic_soil=0), and will be turned on when data generation for the fractional vegetation and soil, as part of the UFS_UTILS, is integrated into the SRW workflow. All other schemes that use RUC_LSM, such as FV3_HRRR and FV3_HRRR_gf, could also be configured to use fractional vegetation and soil data after the data generation becomes functional.

This code still uses sfc_data.nc older (v1) format.

Files modified or added (*):

• parm/FV3.input.yml
• parm/diag_table.RRFS_sas (*)
• parm/field_table.RRFS_sas (*)
• scripts/exregional_make_orog.sh
• scripts/exregional_make_ics.sh
• scripts/exregional_make_lbcs.sh
• scripts/exregional_run_fcst.sh
• ush/link_fix.py
• ush/setup.py
• ush/valid_param_vals.yaml
• tests/WE2E/test_configs/grids_extrn_mdls_suites_community/config.grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas.yaml
• tests/WE2E/test_configs/grids_extrn_mdls_suites_community/config.grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_RRFS_sas_plot.yaml
• tests/WE2E/test_configs/grids_extrn_mdls_suites_community/config.grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas.yaml

NB: With the currently used hashes of ufs-srweather-app and FV3atm in this draft PR, an additional namelist file suite_RRFS_sas.xml is to be placed under ./sorc/ufs-weather-model/FV3/./ccpp/suites/suite_RRFS_sas.xml

@MichaelLueken brought changes to the UFS_UTILS branch in NOAA-EPIC repository that allowed preparing input data correctly for the FV3atm model.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

Tested on Hera

New WE2E tests prepared:
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_RRFS_sas_plot
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas

Checking out the PR and testing it, example for Hera:

git clone https://github.com/ufs-community/ufs-srweather-app.git
cd ufs-srweather-app
git pull origin  pull/1201/head:PR1201
git checkout PR1201
./manage_externals/checkout_externals 
./devbuild.sh -v --platform=hera --compiler=intel 2>&1 | tee log.srw.build
module use $PWD/modulefiles
module load wflow_hera
conda activate srw_app
cd ./tests/WE2E ./run_WE2E_tests.py -m=hera -c=intel -a epic -t grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas

To run other tests, replace the last line by:

./run_WE2E_tests.py -m=hera -c=intel -a epic -t grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_RRFS_sas_plot

or

./run_WE2E_tests.py -m=hera -c=intel -a epic -t grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas

TESTS on PLATFORMS:

• derecho.intel
• gaea.intel
• gaea-c6.intel
• hera.gnu
• hera.intel
• hercules.intel
• jet.intel
• orion.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

DOCUMENTATION:

ISSUE:

This closes an issue #1192

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

CONTRIBUTORS (optional):

@MichaelLueken
@ulmononian

Log files:
test_grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas.txt

test_grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_RRFS_sas_plot.txt

test_grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas.txt

[natalie-perlin]

Tested turning off producing the oro_data_ssnc and oro_data_ls.nc files in
exregional_make_orog.sh. The task make_orog then fails when creating links to these oro_data_ss and oro_data_ls files, as done in ./ush/link_fix.py.
When creating links is turned off for RRFS_sas option in ./ush/link_fix.py, the make_orog task finishes OK.
When a run_fcst task is run, however, it expects to create link to these files from the main run directory, so creating links needs to be turned off as well.

The run_fcst then fails because it still expects to find these files. There are possibly other configuration changes that are needed.

Full log file:
test_3km_ilbcs_HRRR_suite_RRFS_sas_run_fcst.log3.txt

Errors, a snippet from the log file:

  sec               :   0.000000000000000E+000
  first_time_step   :  T
  restart           :  F
  lsm_cold_start    :  T

 lightning threat indexes
  lightning_threat  :  T
 lnd_iau_mod_set_control: Warning! EoF (-1) while reading land_iau namelist, lik
 ely because land_iau_nml was not found in input.nml. It will be set to default.
 land_iau_nml
 &LAND_IAU_NML
 DO_LAND_IAU     = F,
 LAND_IAU_DELTHRS        =  0.000000000000000E+000,
 LAND_IAU_INC_FILES      =                                          
...
...
       ,
 LAND_IAU_FHRS   = 7*-1.00000000000000       ,
 LAND_IAU_FILTER_INCREMENTS      = F,
 LSOIL_INCR      =           4,
 LAND_IAU_UPD_STC        = F,
 LAND_IAU_UPD_SLC        = F,
 LAND_IAU_DO_STCSMC_ADJUSTMENT   = F,
 LAND_IAU_MIN_T_INCREMENT        =  1.000000000000000E-004
 /
 in iau_init   30.0000000000000
 nfiles =            0
 do_hailcast =  T
 in fv3atm_diag_register, nrgst_bl=          62  nrgst_nb=         766
  nrgst_vctbl=           3 hist%isco=           1          40 hist%jsco=
           1          40  hist%num_axes_phys=           3
NOTE from PE     0: reading topographic/orographic information from INPUT/oro_data.tile*.nc

FATAL from PE     1: Error with opening file INPUT/oro_data_ls.nc
FATAL from PE     2: Error with opening file INPUT/oro_data_ls.nc
...

In the rrfs_workflow repository, while the oro_data_ss and oro_data_ls appear not to be created when RRFS_sas suite is set, the links to these files are still created in ./ush/link_fix.sh, and in ./scripts/exrrfs_run_fcst.sh.

It is possible that different (additional?) input files might be needed for the initialization of the orographic data, as appears from ./rrfs-workflow/scripts/exrrfs_make_orog.sh:

suites=( "FV3_RAP" "FV3_HRRR" "FV3_HRRR_gf" "FV3_GFS_v15_thompson_mynn_lam3km" "FV3_GFS_v17_p8" )
if [[ ${suites[@]} =~ "${CCPP_PHYS_SUITE}" ]] ; then
  cd ${tmp_orog_data}
  mosaic_fn_gwd="${CRES}${DOT_OR_USCORE}mosaic.halo${NH4}.nc"
  mosaic_fp_gwd="${FIXLAM}/${mosaic_fn_gwd}"
  grid_fn_gwd=$( get_charvar_from_netcdf "${mosaic_fp_gwd}" "gridfiles" )
  export err=$?
  if [ $err -ne 0 ]; then
    err_exit "get_charvar_from_netcdf function failed."
  fi
  grid_fp_gwd="${FIXLAM}/${grid_fn_gwd}"
  ls_fn="geo_em.d01.lat-lon.2.5m.HGT_M.nc"
  ss_fn="HGT.Beljaars_filtered.lat-lon.30s_res.nc"
  create_symlink_to_file target="${grid_fp_gwd}" symlink="${tmp_orog_data}/${grid_fn_gwd}" relative="FALSE"
  create_symlink_to_file target="${FIXam}/${ls_fn}" symlink="${tmp_orog_data}/${ls_fn}" relative="FALSE"
  create_symlink_to_file target="${FIXam}/${ss_fn}" symlink="${tmp_orog_data}/${ss_fn}" relative="FALSE"

  input_redirect_fn="grid_info.dat"
  cat > "${input_redirect_fn}" <<EOF
${TILE_RGNL}
${CRES:1}
${NH4}
EOF

  print_info_msg "Starting orography file generation..."

  export pgm="orog_gsl"
  . prep_step

  ${APRUN} ${EXECdir}/$pgm < "${input_redirect_fn}" >>$pgmout 2>${tmp_dir}/errfile
  export err=$?; err_chk
  mv ${tmp_dir}/errfile ${tmp_dir}/errfile_orog_gsl

  mv "${CRES}${DOT_OR_USCORE}oro_data_ss.tile${TILE_RGNL}.halo${NH0}.nc" \
     "${CRES}${DOT_OR_USCORE}oro_data_ls.tile${TILE_RGNL}.halo${NH0}.nc" \
     "${OROG_DIR}"
fi

I will be looking into this possibility of different input files needed.

[natalie-perlin]

@MichaelLueken -
It looks like oro_data_ls and oro_data_ss are generated from the additional files, as listed in ./scripts/exrrfs_make_orog.sh
ls_fn="geo_em.d01.lat-lon.2.5m.HGT_M.nc"
ss_fn="HGT.Beljaars_filtered.lat-lon.30s_res.nc"

However, it looks from my test that this is not what causes the issue, since these files are two-dimensional (lat,lon). Fractional vegetation and soil data will likely have additional dimension equal to number of vegetation and soil categories. In the code, it is read from a RRFS restart file for the same domain. It is not yet clear how it was initialized in RRFS - probably done once, and then could be read from a restart for the rest of the forecast , as long as the grid remains unchanged.

[natalie-perlin]

Updated weather model hash after recent merges of the PRs in UFS-weather-model PR-2597 and FV3atm PR-926 and FV3atm PR-930.

To test the RRFS_sas suite with the fractional approach is turned off temporarily (mosaic_lu=0, mosaic_soil=0) the WE2E test_3km_bcs_HRRR_suite_RRFS_sas completes successfully.

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
test_3km_bcs_HRRR_suite_RRFS_sas_20250220155010                    COMPLETE              19.57
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              19.57

Detailed summary of each experiment:

----------------------------------------------------------------------------------------------------
Detailed summary of experiment test_3km_bcs_HRRR_suite_RRFS_sas_20250220155010
in directory /scratch2/NCEPDEV/stmp1/Natalie.Perlin/SRW1/expt_dirs/test_3km_bcs_HRRR_suite_RRFS_sas
                                        | Status    | Walltime   | Core hours used
----------------------------------------------------------------------------------------------------
make_grid_202008100000                    SUCCEEDED          43.0           0.29
get_extrn_ics_202008100000                SUCCEEDED          30.0           0.01
get_extrn_lbcs_202008100000               SUCCEEDED          33.0           0.01
make_orog_202008100000                    SUCCEEDED          59.0           0.39
make_sfc_climo_202008100000               SUCCEEDED          67.0           0.89
make_ics_mem000_202008100000              SUCCEEDED          67.0           0.89
make_lbcs_mem000_202008100000             SUCCEEDED          62.0           0.83
run_fcst_mem000_202008100000              SUCCEEDED         410.0          13.67
run_post_mem000_f000_202008100000         SUCCEEDED          49.0           0.65
run_post_mem000_f001_202008100000         SUCCEEDED          43.0           0.57
run_post_mem000_f002_202008100000         SUCCEEDED          45.0           0.60
run_post_mem000_f003_202008100000         SUCCEEDED          48.0           0.64
integration_test_mem000_202008100000      SUCCEEDED          19.0           0.13
----------------------------------------------------------------------------------------------------
Total                                     COMPLETE                         19.57

Log file is attached: test_3km_ilbcs_HRRR_suite_RRFS_sas_run_fcst.log4.txt

[natalie-perlin]

Successful runs on Hera for the thee new tests:

grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_RRFS_sas_plot:

Calculating core-hour usage and printing final summary
----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_RRFS_sas_plot_2025  COMPLETE              22.02
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              22.02

Detailed summary written to /scratch2/NCEPDEV/stmp1/Natalie.Perlin/SRW1/expt_dirs/WE2E_summary_20250221143130.txt

grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas:

Calculating core-hour usage and printing final summary
----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas_2025  COMPLETE              68.94
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              68.94

Detailed summary written to /scratch2/NCEPDEV/stmp1/Natalie.Perlin/SRW1/expt_dirs/WE2E_summary_20250221135358.txt

grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas:

Calculating core-hour usage and printing final summary
----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas_202  COMPLETE              20.91
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              20.91

Detailed summary written to /scratch2/NCEPDEV/stmp1/Natalie.Perlin/SRW1/expt_dirs/WE2E_summary_20250221144003.txt

[MichaelLueken]

@natalie-perlin -

Thanks for providing the changes to allow the use of the RRFS_sas physics suite in the SRW App!

The new WE2E tests were successfully run on Derecho:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas_2025  COMPLETE              69.25
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_RRFS_sas_plot_2025  COMPLETE              24.39
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas_202  COMPLETE              26.19
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             119.83

I have some minor questions related to the grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas WE2E test that are included in this review.

Please add the new WE2E tests to the coverage test suites in tests/WE2E/machine_suites/coverage*, so that each are tested on one platform per PR. It would also be good to include all three in each comprehensive test suite (tests/WE2E/machine_suites/comprehensive*) so that they are included in the comprehensive testing. As part of the next item, please replace grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta with the updated grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas in tests/WE2E/machine_suite/fundamental.

When moving forward with RRFS_sas, we will be removing support for the older, outdated RRFS_v1beta physics suite. Please remove RRFS_v1beta related options and files from the SRW App.

[natalie-perlin]

@MichaelLueken - all done. Please advise if any further changes are needed!

[MichaelLueken]

The three new RRFS_sas WE2E tests continue to pass:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
grid_RRFS_CONUScompact_25km_ics_RRFS_lbcs_RRFS_suite_RRFS_sas_202  COMPLETE              16.08
grid_RRFS_CONUScompact_13km_ics_HRRR_lbcs_RAP_suite_RRFS_sas_2025  COMPLETE              63.50
grid_SUBCONUS_Ind_3km_ics_HRRR_lbcs_HRRR_suite_RRFS_sas_plot_2025  COMPLETE              20.62
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE             100.20

Thanks for adding the new RRFS_sas WE2E tests to the various suites and removing of the now obsolete RRFS_v1beta suite, @natalie-perlin!

@mkavulich has pointed out that the config.specify_DOT_OR_USCORE.yaml WE2E test configuration in tests/WE2E/test_configs/wflow_features is a symlink pointing pointing to the removed config.grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta.yaml test configuration. Please remove config.specify_DOT_OR_USCORE.yaml as well.
I don't see any further issues once this has been addressed. Thanks!

[natalie-perlin]

config.specify_DOT_OR_USCORE.yaml

@MichaelLueken @mkavulich - the linked file has been removed. Thank you for noticing.

[MichaelLueken]

Thank you very much for working with me on these updates for RRFS_sas, @natalie-perlin!

Approving this PR now.

[mkavulich]

I have a few questions, not necessarily requesting changes depending on the answers

[MichaelLueken]

The UFS_FIRE WE2E tests were successfully run on Derecho:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used
----------------------------------------------------------------------------------------------------
UFS_FIRE_one-way-coupled_20250226122753                            COMPLETE              28.69
UFS_FIRE_multifire_two-way-coupled_20250226122754                  COMPLETE              28.45
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE              57.14

The AQM WE2E test was also successfully run on Derecho:

----------------------------------------------------------------------------------------------------
Experiment name                                                  | Status    | Core hours used 
----------------------------------------------------------------------------------------------------
aqm_grid_AQM_NA13km_suite_GFS_v16_20250226133539                   COMPLETE            3366.49
----------------------------------------------------------------------------------------------------
Total                                                              COMPLETE            3366.49