How to launch an AMReX program with MPI?

[adamdarx]

Is there any automatic settings for an AMReX program to run it with MPI? I just write one and it always run with only ONE mpi process. I tried to use mpich/mpirun to run it with more process but it just re-run a same program instead of run it parallel-ly. What settings shall I write in CMakeLists.txt or command line?

[WeiqunZhang]

Maybe you are missing -DAMReX_MPI=ON?

[adamdarx]

Maybe you are missing -DAMReX_MPI=ON?

No, this option is set as YES as default. I wonder how i can run this with more mpi process after compilement.😭

[WeiqunZhang]

You mentioned cmake, so I thought you had a build issue. If that is not the case, then you just run it as any other MPI programs. There is nothing special about an amrex code in terms of how to run it. I am actually not sure what your question is. I am just guessing. Is this what you are looking for ? https://www.mpich.org/static/docs/v3.0.x/www1/mpiexec.html

If you still have questions, could you provide more details of what you do exactly step by step?

[adamdarx]

You mentioned cmake, so I thought you had a build issue. If that is not the case, then you just run it as any other MPI programs. There is nothing special about an amrex code in terms of how to run it. I am actually not sure what your question is. I am just guessing. Is this what you are looking for ? https://www.mpich.org/static/docs/v3.0.x/www1/mpiexec.html

If you still have questions, could you provide more details of what you do exactly step by step?

Thanks for your patience! Actually I'm writing a code about MHD and I notice that AMReX initialize MPI automatically but always with only one MPI process. I tried to run it with mpirun or mpiexec(for example mpiexec -n 8 ...), but every program still initialize only one MPI process and do the same thing.

My question is whether I am allowed to increase the number of MPI processes initialized by AMReX.

[WeiqunZhang]

I tried to run it with mpirun or mpiexec(for example mpiexec -n 8 ...),

But your screenshot shows ./GRMHD, not mpiexec -n 8 ./GRMHD. So what do you see when you run mpiexec -n 8 ./GRMHD? And what do you expect to see? And what do you mean by "do the same thing"?

[adamdarx]

I tried to run it with mpirun or mpiexec(for example mpiexec -n 8 ...),

But your screenshot shows ./GRMHD, not mpiexec -n 8 ./GRMHD. So what do you see when you run mpiexec -n 8 ./GRMHD? And what do you expect to see? And what do you mean by "do the same thing"?

if you run mpiexec -n 8 ./GRMHD, then you see 8 copies of that screenshot by which I mean "do the same thing". I expect to see "MPI initialized with 8 MPI processes" for instance.

[WeiqunZhang]

I suspect you have multiple MPI libraries on you system and you are mixing mpicxx from one version with mpiexec from another. You want to remove either openmpi or mpich and keep only one version, or use update-alternatives to make sure your mpicxx and mpiexec are consistent.

[adamdarx]

I suspect you have multiple MPI libraries on you system and you are mixing mpicxx from one version with mpiexec from another. You want to remove either openmpi or mpich and keep only one version, or use update-alternatives to make sure your mpicxx and mpiexec are consistent.

Thanks for your advice! I'll check if I install two different version of MPI.

[bsrunnels]

@WeiqunZhang I'm encountering the same issue. On Ubuntu 24.04 I have the default mpich installed (4.2.0) and have confirmed that is the only MPI on the machine. Running the following with a fresh clone (noting that USE_MPI=TRUE in the GNUmakefile):

git@github.com:AMReX-Codes/amrex.git
git checkout 25.03
cd amrex/Tests/LinearSolvers/ABecLaplacian_C/
make

Attempting to run in parallel produces the same behavior - N independent jobs each thinking they are running in serial.

> mpiexec -np 4 ./main3d.gnu.MPI.ex

Initializing AMReX (25.03)...
MPI initialized with 1 MPI processes
MPI initialized with thread support level 0
Initializing AMReX (25.03)...
MPI initialized with 1 MPI processes
MPI initialized with thread support level 0
Initializing AMReX (25.03)...
MPI initialized with 1 MPI processes
MPI initialized with thread support level 0
Initializing AMReX (25.03)...
MPI initialized with 1 MPI processes
MPI initialized with thread support level 0

This problem has occurred up on all of my machines that are running ubuntu 24.04.

[WeiqunZhang]

What do you see with $ ls -tlr /etc/alternatives/ | grep mpi?

[WeiqunZhang]

And what do you see with ls -l /usr/bin/mpi*?

[bsrunnels]

@WeiqunZhang output of $ ls -tlr /etc/alternatives/ | grep mpi (eliminating access / size / date for readability)

mpi-x86_64-linux-gnu -> /usr/include/x86_64-linux-gnu/mpich
mpirun -> /usr/bin/mpirun.mpich
mpi.pc-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/pkgconfig/mpich.pc
mpifort -> /usr/bin/mpifort.mpich
mpif90 -> /usr/bin/mpifort.mpich
mpif77 -> /usr/bin/mpifort.mpich
mpiexec -> /usr/bin/mpiexec.mpich
mpicxx -> /usr/bin/mpicxx.mpich
mpiCC -> /usr/bin/mpicxx.mpich
mpic++ -> /usr/bin/mpicxx.mpich
mpi -> /usr/bin/mpicc.mpich
libmpi.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/libmpich.so
libmpi++.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/libmpichcxx.so
libmpifort.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/libmpichfort.so
libmpif90.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/libmpichfort.so
libmpif77.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/libmpichfort.so
libmpicxx.so-x86_64-linux-gnu -> /usr/lib/x86_64-linux-gnu/libmpichcxx.so

and $ ls -l /usr/bin/mpi*

/usr/bin/mpic++ -> /etc/alternatives/mpic++
/usr/bin/mpicc -> /etc/alternatives/mpi
/usr/bin/mpiCC -> /etc/alternatives/mpiCC
/usr/bin/mpicc.mpich
/usr/bin/mpichversion
/usr/bin/mpicxx -> /etc/alternatives/mpicxx
/usr/bin/mpicxx.mpich
/usr/bin/mpiexec -> /etc/alternatives/mpiexec
/usr/bin/mpiexec.hydra
/usr/bin/mpiexec.mpich -> mpiexec.hydra
/usr/bin/mpif77 -> /etc/alternatives/mpif77
/usr/bin/mpif77.mpich -> mpifort.mpich
/usr/bin/mpif90 -> /etc/alternatives/mpif90
/usr/bin/mpif90.mpich -> mpifort.mpich
/usr/bin/mpifort -> /etc/alternatives/mpifort
/usr/bin/mpifort.mpich
/usr/bin/mpirun -> /etc/alternatives/mpirun
/usr/bin/mpirun.mpich -> mpiexec.hydra
/usr/bin/mpivars

[WeiqunZhang]

This looks like an Ubuntu bug.
https://bugs.launchpad.net/ubuntu/+source/mpich/+bug/2072338
https://stackoverflow.com/questions/78815680/mpi-comm-size-always-returns-1-again

Maybe you can install openmpi to work around the issue or wait for the bug fix.

[bsrunnels]

Wow, it is surprising that such a significant bug would make it into the .1 LTS for 2024. That is very unfortunate..

I suppose we'll switch to openmpi then. Oddly I remember encountering this same issue several years back (2017), and basically the fix was to switch from openmpi to mpich...