Skip to content

Latest commit

 

History

History
114 lines (71 loc) · 3.33 KB

fugaku.rst

File metadata and controls

114 lines (71 loc) · 3.33 KB

Fugaku (Riken)

The Fugaku cluster is located at the Riken Center for Computational Science (Japan).

Introduction

If you are new to this system, please see the following resources:

Preparation

Use the following commands to download the WarpX source code and switch to the correct branch:

git clone https://github.com/ECP-WarpX/WarpX.git $HOME/src/warpx

Compiling WarpX on Fugaku is more practical on a compute node. Use the following commands to acquire a compute node for one hour:

pjsub --interact -L "elapse=02:00:00" -L "node=1" --sparam "wait-time=300" --mpi "max-proc-per-node=48" --all-mount-gfscache

We use system software modules, add environment hints and further dependencies via the file $HOME/fugaku_warpx.profile. Create it now, modify it if needed, and source it (it will take few minutes):

cp $HOME/src/warpx/Tools/machines/fugaku-riken/fugaku_warpx.profile.example $HOME/fugaku_warpx.profile
source $HOME/fugaku_warpx.profile
.. dropdown:: Script Details
   :color: light
   :icon: info
   :animate: fade-in-slide-down

   .. literalinclude:: ../../../../Tools/machines/fugaku-riken/fugaku_warpx.profile.example
      :language: bash

Finally, since Fugaku does not yet provide software modules for some of our dependencies, install them once:

bash $HOME/src/warpx/Tools/machines/fugaku-riken/install_dependencies.sh
.. dropdown:: Script Details
   :color: light
   :icon: info
   :animate: fade-in-slide-down

   .. literalinclude:: ../../../../Tools/machines/fugaku-riken/install_dependencies.sh
      :language: bash

Compilation

Use the following :ref:`cmake commands <building-cmake>` to compile the application executable:

cd $HOME/src/warpx
rm -rf build

export CC=$(which mpifcc)
export CXX=$(which mpiFCC)
export CFLAGS="-Nclang"
export CXXFLAGS="-Nclang"

cmake -S . -B build -DWarpX_COMPUTE=OMP \
    -DWarpX_DIMS="1;2;3" \
    -DCMAKE_BUILD_TYPE=Release \
    -DCMAKE_CXX_FLAGS_RELEASE="-Ofast" \
    -DAMReX_DIFFERENT_COMPILER=ON \
    -DWarpX_MPI_THREAD_MULTIPLE=OFF

cmake --build build -j 48

That's it! A 3D WarpX executable is now in build/bin/ and :ref:`can be run <running-cpp-fugaku>` with a :ref:`3D example inputs file <usage-examples>`.

Running

A64FX CPUs

In non-interactive runs, you can use pjsub submit.sh where submit.sh can be adapted from:

.. literalinclude:: ../../../../Tools/machines/fugaku-riken/submit.sh
   :language: bash
   :caption: You can copy this file from ``Tools/machines/fugaku-riken/submit.sh``.

Note: the Boost Eco Mode mode that is set in this example increases the default frequency of the A64FX from 2 GHz to 2.2 GHz, while at the same time switching off one of the two floating-point arithmetic pipelines. Some preliminary tests with WarpX show that this mode achieves performances similar to those of the normal mode but with a reduction of the energy consumption of approximately 20%.