FMS fails to build with oneAPI

[manodeep]

The details for the build failure, when trying to build (as part of building ESM1.6) with spack are captured in ACCESS-NRI/ACCESS-ESM1.6#46 (comment)

The offending lines is here which raises a fatal error when the oneAPI compiler is used. Related to #1

[harshula]

Hi @manodeep ,

Use gitref dev_2025.02.1. e.g.

# spack install access-fms@git.dev_2025.02.1=develop %oneapi@2023.2.4 target=x86_64
...
[+] /opt/release/linux-rocky8-x86_64/oneapi-2023.2.4/netcdf-fortran-4.6.1-yyjsnpxcd62hf2on2zel7g5xa6gvuvv3
==> Waiting for access-fms-git.dev_2025.02.1=develop-ky5zi7i3mf7gul46ylzx5i3qvnz==> Installing access-fms-git.dev_2025.02.1=develop-ky5zi7i3mf7gul46ylzx5i3qvnz2d347 [38/38]
==> No binary for access-fms-git.dev_2025.02.1=develop-ky5zi7i3mf7gul46ylzx5i3qvnz2d347 found: installing from source
==> No patches needed for access-fms
==> access-fms: Executing phase: 'cmake'
==> access-fms: Executing phase: 'build'
==> access-fms: Executing phase: 'install'
==> access-fms: Successfully installed access-fms-git.dev_2025.02.1=develop-ky5zi7i3mf7gul46ylzx5i3qvnz2d347
  Stage: 2.67s.  Cmake: 2.32s.  Build: 1m 50.09s.  Install: 0.11s.  Post-install: 0.15s.  Total: 1m 55.45s
[+] /opt/release/linux-rocky8-x86_64/oneapi-2023.2.4/access-fms-git.dev_2025.02.1_develop-ky5zi7i3mf7gul46ylzx5i3qvnz2d347

gitref dev_2025.02.1 contains the upstream FMS ifx compiler flags:

  set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -fpp -fno-alias -auto -safe-cray-ptr -ftz -assume byterecl -align array64byte -nowarn -traceback")
  set(CMAKE_Fortran_FLAGS_RELEASE "-O2 -debug minimal -nowarn -qoverride-limits -qno-opt-dynamic-align")
  set(CMAKE_Fortran_FLAGS_DEBUG "-g -O0 -check -check noarg_temp_created -check nopointer -warn -warn noerrors -fpe0 -ftrapuv")
  set(CMAKE_Fortran_LINK_FLAGS "")

[manodeep]

Thanks @harshula

[manodeep]

@harshula Would it be possible to add -flto to both the Release and Link flags? That supposedly enables better optimisation passes.

Related: Should the link flags append to any existing link flags - following the pattern of fortran flags?

[harshula]

Hi @manodeep , FYI, the current FMS repository is a new fork of the upstream FMS repository. The mom5 branch is the old repository's master branch and the development branch is the old repository's development branch.

As of last night, the new FMS repository's development branch (including tags dev_2024.12.0, dev_2025.02.1, dev_2025.02.2) now uses the MOM5 Fortran compiler flags for ifx:

elseif(CMAKE_Fortran_COMPILER_ID STREQUAL "IntelLLVM")
...
  # Copied from MOM5/bin/mkmf.template.nci
  set(CMAKE_Fortran_FLAGS "${CMAKE_Fortran_FLAGS} -fno-alias -safe-cray-ptr -fpe0 -ftz -assume byterecl -i4 -r8 -traceback -nowarn -check noarg_temp_created -assume nobuffered_io -convert big_endian -grecord-gcc-switches -align all")
  set(CMAKE_Fortran_FLAGS_RELEASE "-g3 -O2 -xCORE-AVX2 -debug all -check none")
  set(CMAKE_Fortran_FLAGS_DEBUG "-g3 -O0 -debug all -check -check noarg_temp_created -check nopointer -warn -warn noerrors -ftrapuv")

I was able to successfully build access-fms using tag dev_2025.02.2 and using oneapi@2025.0.4.

Please document any new compiler flags you want to use in dev-docs, once that is collated and reviewed, we can start adding them to the relevant packages.

[manodeep]

Thanks @harshula. I am testing out the spack build for ESM1.6 - presumably the other failures will still occur.

[manodeep]

@harshula I still can't get access-fms to compile within the ESM1.6 config - fails with an error about MPI_Fortran

==> [2025-02-26-16:01:49.173935] access-fms: Building access-fms-git.dev_2025.02.1=2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u [CMakePackage]
==> [2025-02-26-16:01:49.173700] No patches needed for access-fms
==> [2025-02-26-16:01:49.307241] access-fms: Executing phase: 'cmake'
==> [2025-02-26-16:02:13.598931] Error: ProcessError: Command exited with status 1:
    '/apps/cmake/3.24.2/bin/cmake' '-G' 'Unix Makefiles' '-DCMAKE_INSTALL_PREFIX:STRING=/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_
64_v4/oneapi-2024.2.1/access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u' '-DCMAKE_BUILD_TYPE:STRING=Release' '-DCMAKE_VERBOSE_MAKEFILE:BOO
L=ON' '-DCMAKE_INTERPROCEDURAL_OPTIMIZATION:BOOL=OFF' '-DCMAKE_POLICY_DEFAULT_CMP0090:STRING=NEW' '-DCMAKE_FIND_USE_PACKAGE_REGISTRY:BOOL=OFF' '-DCMAKE_EXPORT_C
OMPILE_COMMANDS:BOOL=ON' '-DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=ON' '-DCMAKE_INSTALL_RPATH:STRING=/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linu
x-rocky8-x86_64_v4/oneapi-2024.2.1/access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/lib;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/re
lease/linux-rocky8-x86_64_v4/oneapi-2024.2.1/access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/lib64;/g/data/tm70/ms2335/spack/0.22/restri
cted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/intel-oneapi-runtime-2024.2.1-e77bzfloyijfih7lb5gqaoxq73mqjyul/lib;/g/data/tm70/ms2335/spack/0.22/restr
icted/ukmo/release/linux-rocky8-x86_64_v4/gcc-14.1.0/gcc-runtime-14.1.0-tgv3p3aaw6n6r6uw5v2zdwztxxbimasr/lib;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/rele
ase/linux-rocky8-x86_64_v4/oneapi-2024.2.1/netcdf-c-4.7.4-mbj7iwszazqcgf5zpiqnndcicokcvaum/lib;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rock
y8-x86_64_v4/oneapi-2024.2.1/hdf5-1.10.11-ocbyjtxheur5zzayydbgaww2j6in6nwt/lib;/apps/openmpi/4.0.2/lib;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/li
nux-rocky8-x86_64_v4/oneapi-2024.2.1/zlib-ng-2.1.6-al4knrktgnaem2iz6dvobjmuz2t3mygp/lib;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_
64_v4/oneapi-2024.2.1/netcdf-fortran-4.5.2-o4m647gx43thtkyclxs53spajkxkml35/lib' '-DCMAKE_PREFIX_PATH:STRING=/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/rele
ase/linux-rocky8-x86_64_v4/oneapi-2024.2.1/netcdf-fortran-4.5.2-o4m647gx43thtkyclxs53spajkxkml35;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-ro
cky8-x86_64_v4/oneapi-2024.2.1/netcdf-c-4.7.4-mbj7iwszazqcgf5zpiqnndcicokcvaum;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/one
api-2024.2.1/hdf5-1.10.11-ocbyjtxheur5zzayydbgaww2j6in6nwt;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/zlib-ng
-2.1.6-al4knrktgnaem2iz6dvobjmuz2t3mygp;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/gmake-4.4.1-wjn2zyjqr6a4sr
4k4pvwa3mvqg3zsizn;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/intel-oneapi-runtime-2024.2.1-e77bzfloyijfih7lb
5gqaoxq73mqjyul;/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/gcc-14.1.0/gcc-runtime-14.1.0-tgv3p3aaw6n6r6uw5v2zdwztxxbimasr;/ap
ps/openmpi/4.0.2;/apps/cmake/3.24.2' '-DGFS_PHYS:BOOL=ON' '-DLARGEFILE:BOOL=OFF' '-DINTERNAL_FILE_NML:BOOL=ON' '/scratch/tm70/ms2335/tmp/restricted/spack-stage/
spack-stage-access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/spack-src'

1 error found in build log:
     9     -- Detecting C compile features - done
     10    -- Detecting Fortran compiler ABI info
     11    -- Detecting Fortran compiler ABI info - done
     12    -- Check for working Fortran compiler: /g/data/tm70/ms2335/spack/0.22/spack/lib/spack/env/oneapi/ifx - skipped
     13    -- Found MPI_C: /apps/openmpi/4.0.2/lib/libmpi.so (found version "3.1")
     14    -- Could NOT find MPI_Fortran (missing: MPI_Fortran_WORKS)
  >> 15    CMake Error at /apps/cmake/3.24.2/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
     16      Could NOT find MPI (missing: MPI_Fortran_FOUND Fortran) (found version
     17      "3.1")
     18    Call Stack (most recent call first):
     19      /apps/cmake/3.24.2/share/cmake-3.24/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
     20      /apps/cmake/3.24.2/share/cmake-3.24/Modules/FindMPI.cmake:1835 (find_package_handle_standard_args)
     21      CMakeLists.txt:179 (find_package)

See build log for details:
  /scratch/tm70/ms2335/tmp/restricted/spack-stage/spack-stage-access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/spack-build-out.txt

From the cmake error log (here: /scratch/tm70/ms2335/tmp/restricted/spack-stage/spack-stage-access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/spack-build-d7mwrwf/CMakeFiles/CMakeError.log):

The MPI test test_mpi for Fortran in mode F77_HEADER_NOKIND failed to compile with the following output:
Change Dir: /scratch/tm70/ms2335/tmp/restricted/spack-stage/spack-stage-access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/spack-build-d7mwrwf/CMakeFiles/CMakeTmp

Run Build Command(s):/g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/gmake-4.4.1-wjn2zyjqr6a4sr4k4pvwa3mvqg3zsizn/bin/gmake -f Makefile cmTC_2e954/fast && /g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/gmake-4.4.1-wjn2zyjqr6a4sr4k4pvwa3mvqg3zsizn/bin/gmake  -f CMakeFiles/cmTC_2e954.dir/build.make CMakeFiles/cmTC_2e954.dir/build
gmake[1]: Entering directory '/scratch/tm70/ms2335/tmp/restricted/spack-stage/spack-stage-access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/spack-build-d7mwrwf/CMakeFiles/CMakeTmp'
Building Fortran object CMakeFiles/cmTC_2e954.dir/test_mpi.f90.o
/g/data/tm70/ms2335/spack/0.22/spack/lib/spack/env/oneapi/ifx  -isystem /apps/openmpi-mofed5.5-pbs2021.1/4.0.2/include -isystem /apps/openmpi/4.0.2/include/Intel -fno-alias -safe-cray-ptr -fpe0 -ftz -assume byterecl -i4 -r8 -traceback -nowarn -check noarg_temp_created -assume nobuffered_io -convert big_endian -grecord-gcc-switches -align all  -pthread -c /scratch/tm70/ms2335/tmp/restricted/spack-stage/spack-stage-access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/spack-build-d7mwrwf/CMakeFiles/FindMPI/test_mpi.f90 -o CMakeFiles/cmTC_2e954.dir/test_mpi.f90.o
xfortcom: Unknown command line argument '-dwarf-debug-flags --gcc-toolchain=/usr -isystem /g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/netcdf-fortran-4.5.2-o4m647gx43thtkyclxs53spajkxkml35/include -isystem /g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/netcdf-c-4.7.4-mbj7iwszazqcgf5zpiqnndcicokcvaum/include -isystem /g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/hdf5-1.10.11-ocbyjtxheur5zzayydbgaww2j6in6nwt/include -isystem /g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/zlib-ng-2.1.6-al4knrktgnaem2iz6dvobjmuz2t3mygp/include -isystem /apps/openmpi-mofed5.5-pbs2021.1/4.0.2/include -isystem /apps/openmpi/4.0.2/include/Intel -isystem /apps/openmpi/4.0.2/include -fno-alias -safe-cray-ptr -fpe0 -ftz -assume byterecl -i4 -r8 -traceback -nowarn -check noarg_temp_created -assume nobuffered_io -convert big_endian -grecord-gcc-switches -align all -pthread -c -o CMakeFiles/cmTC_2e954.dir/test_mpi.f90.o -isystem/apps/openmpi/4.0.2/include/Intel -isystem/apps/openmpi/4.0.2/include -isystem/opt/Modules/v4.3.0/init/ksh-functions '.  Try: '/apps/intel-tools/.packages/2024.2.1/compiler/2024.2/bin/compiler/xfortcom --help'
xfortcom: Did you mean '--asan-debug-stack=/usr -isystem /g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/netcdf-fortran-4.5.2-o4m647gx43thtkyclxs53spajkxkml35/include -isystem /g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/netcdf-c-4.7.4-mbj7iwszazqcgf5zpiqnndcicokcvaum/include -isystem /g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/hdf5-1.10.11-ocbyjtxheur5zzayydbgaww2j6in6nwt/include -isystem /g/data/tm70/ms2335/spack/0.22/restricted/ukmo/release/linux-rocky8-x86_64_v4/oneapi-2024.2.1/zlib-ng-2.1.6-al4knrktgnaem2iz6dvobjmuz2t3mygp/include -isystem /apps/openmpi-mofed5.5-pbs2021.1/4.0.2/include -isystem /apps/openmpi/4.0.2/include/Intel -isystem /apps/openmpi/4.0.2/include -fno-alias -safe-cray-ptr -fpe0 -ftz -assume byterecl -i4 -r8 -traceback -nowarn -check noarg_temp_created -assume nobuffered_io -convert big_endian -grecord-gcc-switches -align all -pthread -c -o CMakeFiles/cmTC_2e954.dir/test_mpi.f90.o -isystem/apps/openmpi/4.0.2/include/Intel -isystem/apps/openmpi/4.0.2/include -isystem/opt/Modules/v4.3.0/init/ksh-functions '?
compilation aborted for /scratch/tm70/ms2335/tmp/restricted/spack-stage/spack-stage-access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/spack-build-d7mwrwf/CMakeFiles/FindMPI/test_mpi.f90 (code 1)
gmake[1]: *** [CMakeFiles/cmTC_2e954.dir/build.make:78: CMakeFiles/cmTC_2e954.dir/test_mpi.f90.o] Error 1
gmake[1]: Leaving directory '/scratch/tm70/ms2335/tmp/restricted/spack-stage/spack-stage-access-fms-git.dev_2025.02.1_2025.02.2-d7mwrwfbpqt6gin4rkkay7lje5eqch3u/spack-build-d7mwrwf/CMakeFiles/CMakeTmp'
gmake: *** [Makefile:127: cmTC_2e954/fast] Error 2

This may be an openmpi/4.0.2 issue - so checking with openmpi/5.0.5 currently.

Separately, would it be worthwhile to send through a PR to the ESM1.6 repo and have the build-cd pipelines build a spack package on gadi with oneAPI?

[harshula]

Hi @manodeep , I used openmpi@4.1.5 and oneapi@2025.0.4 in a Docker container.

[manodeep]

openmpi/5.0.5did not work either.

@harshula I am building directly on gadi. I will try with openmpi/4.1.5 and report back. If that fails, I will send through the PR to test what's happening, and see if build-cd can deploy

[harshula]

Hi @manodeep , You were able to compile it here: ACCESS-NRI/spack-config#54 (comment)

[manodeep]

Hahahahaha - thanks Harshula :D

It must be that the packages are installable separately but not when part of the ESM1.6 spack.yaml