I am running Ubuntu 18.04.3 with MPICH 3.3 and GFortran 7.4.0 and GCC 7.4.0 on dual AMD EPYC 7742 processors with hyper threading enabled. My codes are written in MPI and Fortran. The dual AMD processors have 128 cores and 256 threads. I want to optimize the runtime for 4 mpi jobs running concurrently with 64 threads each. Some timings are provided here:
- One mpi job with mpiexec.hydra -n 64 myprog => 57.32s
- One mpi job with mpiexec.hydra -bind-to numa -n 64 => 50.52s
- Two mpi jobs with mpiexec.hydra -n 64 myprog => 99.77s
- Two mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 72.23s
- Four mpi jobs with mpiexec.hydra -bind-to numa -n 64 => 159.2s
The option "-bind-to numa" helps, but even so, running four mpi jobs concurrently with 64 threads each is considerably slower than running one mpi job with 64 threads. I can almost run four mpi jobs sequentially and match the time for running four mpi jobs concurrently. How can I improve on the result for running 4 mpi jobs concurrently? Thanks, Doug.
