Small guide to PETSc

Quoting from their website:
"PETSc, the Portable, Extensible Toolkit for Scientific Computation, pronounced PET-see (/ˈpɛt-siː/), is for the scalable (parallel) solution of scientific applications modeled by partial differential equations (PDEs).
It has bindings for C, Fortran, and Python (via petsc4py). PETSc also contains TAO, the Toolkit for Advanced Optimization, software library. They support MPI, and GPUs through CUDA, HIP, Kokkos, or OpenCL, as well as hybrid MPI-GPU parallelism; they also support the NEC-SX Tsubasa Vector Engine."

It is a library that I'm using in my works during my PhD. Therefore I decided to write a small guide to keep track of some details not well written in their documentation.

Quick installation

You can follow the steps here, but if you want a quick guide, here it is:
  1. In a folder of your choice run git clone -b release https://gitlab.com/petsc/petsc.git petsc
  2. Get in the folder with cd petsc
  3. To force a specific version use git checkout vMAJOR.MINOR.PATCH
  4. My personal configuration is ./configure --with-cc=gcc --with-cxx=g++ --with-fc=gfortran --download-mpich --download-fblaslapack --download-hdf5 --download-cmake --with-mpi=1 --with-debugging=0 --COPTFLAGS=-O3 -march=native -mtune=native -mavx2 --CXXOPTFLAGS=-O3 -march=native -mtune=native -mavx2 --FOPTFLAGS=-O3 -march=native -mtune=native -mavx2 where
    • make, gcc, g++, gfortran are required packages to be installed;
    • mpich, fblaslapack, hdf5 and cmake are required packages downloaded during the configuration (hdf5 is optional, use it if you expect to export data in .h5 format);
    • it configures the mpi mode and turns the debug mode off (to gain performance). I fou want to configure PETSc in debug mode use --with-debugging=1;
    • C, C++ and Fortran files are always compiled with the optimization label "-O3". If you don't want the compiler to alter the arithmetic, use the label "-O2".

Custom profiling

If you need to measure the runtime of your program, the -log_view option will show you the total CPU time. On the other hand, if you need to measure the time taken by a specific routine, you can use the PetscLogStage variable (full guide here).
This is a generic example: PetscLogStage stagenum;
PetscLogStageRegister("Name of the stage", &stagenum); // Register the stage in the system
PetscLogStagePush(stagenum); // Start the stopwatch
/* stage of code here */
PetscLogStagePop(); // Stop the stopwatch
 In this way, -log_view will show the CPU of the stopwatch associated to "stagenum".