Performance Test (Single Schwarzschild black hole)

Benchmark which solves the static and non-spin single black hole described in a ExaGRyPE simulation.

All extensions and fancy features are removed for simplicity. This benchmark is specially compacted to be as close as possible to the corresponding application, so reconstruction and AMR are enabled. This benchmark can be used as a starting point to understand the application and examine its performance under various setup and optimization.

Run the benchmark

To run the benchmark, configure and compile Peano.

Then enter this subdirectory and add the python path as

cd ./benchmarks/exahype2/ccz4/performance-testbed/
export PYTHONPATH=../../../../python
export PYTHONPATH=$PYTHONPATH:../../../../applications/exahype2/ccz4

generate the executable and run

python3 performance_testbed.py -et 0.01
export OMP_NUM_THREADS=4
./peano_test

As always, the parameter –help makes the Python script yield further info, while the sections below enlist characteristic choices of the parameters. There are different scripts in the directory. They serve different purposes and support slightly different parameters.

Depending on your system, you might have to load the GSL module separately before you build the application:

module load gsl

and include the installation in the CXX flags in configuration

CXXFLAGS="...... -Icosma/local/gsl/2.4/include ......"

Compiling on COSMA

the suggest loading module (before compiling the Peano kernel) when you use COSMA or DINE are

module purge
module load intel_comp/2023.2.0 compiler mpi 
module load gnu_comp/13.1.0
export FLAVOUR_NOCONFLICT=1
module load gsl
export I_MPI_CXX=icpx

Compiling on Hamilton

the suggest loading module (before compiling the Peano kernel) when you use Hamilton are

module purge
module load intel/2023.2 intelmpi/2021.10
module load python/3.10.8
export I_MPI_CXX=icpx

SLURM script

The default setup for this benchmark takes about 5 minutes to finish on 2 nodes on DINE, using the following job script.

#!/bin/bash -l
 
#SBATCH -o %A.out
#SBATCH -e %A.err
#SBATCH -p bluefield1
#SBATCH -A durham
#SBATCH -t 24:00:00
#SBATCH --ntasks-per-node=1
#SBATCH --nodes=2
#SBATCH -J ccz4
 
echo "My SLURM_ARRAY_TASK_ID: " $SLURM_ARRAY_TASK_ID
 
module purge
module load intel_comp/2023.2.0 compiler mpi 
module load gnu_comp/13.1.0
export FLAVOUR_NOCONFLICT=1
module load gsl
 
export UCX_TLS=ud,sm,self
 
export OMP_NUM_THREADS=32
 
mpirun ./peano4_test

Clean up old and temporary files

The following statements remove all the glue code and autogenerated C++ files. There is no risk in deleting them - if you rerun the Python script, they will all be reinstantiated.

rm *.o *.cmake
rm README-*.md
rm Makefile CMakeLists.txt
rm Abstract*
rm -rf celldata facedata globaldata observers tasks vertexdata repositories

To get rid of the output files, type in

rm *patch-file *vtu *.pvd *.bak *.csv

Visualisation

The output of ExaHyPE is given in peano patch file and one can use the following commands to convert them into normal vtu files, and inspect them in other tools like paraview.

module unload python/2.7.15
module load python/3.6.5 paraview/5.8.1
export PYTHONPATH=/path/to/Peano/python/
pvpython /path/to/Peano/python/peano4/visualisation/render.py path/to/solution.peano-patch-file