Installation

In order to be installed, TOSCA requires a working C/C++ compiler, PETSc (version 3.14.x, 3.15.x), Open MPI (version 4.0.x, 4.1.x), HDF5 and HYPRE (needed by PETSs in order to build some of the matrix solvers we use). TOSCA has been tested with the above version combinations, it could work with other combinations or versions but it has not been tested (especially older versions). We recommend the following versions of the above libraries:

• gcc : 9.2.0 (https://gcc.gnu.org/).

• PETSc : 3.15.5 (https://ftp.mcs.anl.gov/pub/petsc/).

• Open MPI : 4.1.2 (https://www.open-mpi.org/software/ompi/v4.1/).

• HYPRE : 2.20.0 (https://github.com/hypre-space/hypre/tree/master) (check version in /src/CMakeLists.txt).

• HDF5 : 1.12.1 (https://www.hdfgroup.org/downloads/hdf5/).

Prior to install TOSCA, we suggest to create a folder named software inside $HOME, where the PETSc, TOSCA and potentially other libraries directories will be located. In order to compile TOSCA on your system, please follow these steps:

1. Check your compiler version with gcc --version

2. Download PETSc into $HOME/software/

3. Download Open MPI: you can download the binaries or compile from source (the latter is recommended if using environment-modules e.g. on an HPC architecture). If you have only one version of Open MPI installed on your system in the /usr directory (installed using sudo for example), you can omit the --with-mpi-dir='your--path--to--mpicc' at point 6: Open MPI will be found by the library locator.

4. Download HYPRE into $HOME/software/. This can be omitted if you use the --download-hypre option in the PETSc configure step (suggested).

5. Download HDF5 into $HOME/software/. This can be omitted if you use the --download-hdf5 option in the PETSc configure step (suggested).

6. Configure PETSc. We suggest the following configure options, which will automatically compile HYPRE and HDF5.

   ./configure --with-fc=0 --download-f2cblaslapack --download-hypre --download-hdf5 --with-64-bit-indices=1 --with-debugging=0
   

   Note that other options are available. For example, if one wants to specify paths to already installed OpenMPI libraries, the options --with-mpi-dir='your--path--to--mpicc' can be used. We strongly suggest to use the --download-hypre and --download-hdf5 options, as they will download and compile the libraries automatically, making hte installation process easier.

7. Make PETSc (PETSc will suggest a command after the configure, we advice to use that):

   make all
   

8. Check the PETSc installation (PETSc will suggest a command after compilation, we advice to use that):

   make check
   

9. Save an environment variable that will tell TOSCA where PETSc is installed in your .bashrc. For HPC installations it might be a better practice to use the .bash_profile instead.

   echo "export PETSC_DIR=$HOME/your--path--to--petsc--dir" >> $HOME/.bashrc
   

   or, for HPCs

   echo "export PETSC_DIR=$HOME/your--path--to--petsc--dir" >> $HOME/.bash_profile
   

10. Save an environment variable that will tell TOSCA which PETSc architecture is required in your .bashrc. For HPC installations it might be a better practice to use the .bash_profile instead.

echo "export PETSC_ARCH=arch-linux-c-opt" >> $HOME/.bashrc

or, for HPCs

echo "export PETSC_ARCH=arch-linux-c-opt" >> $HOME/.bash_profile

Note

This is the folder within $PETSC_DIR with a name beginning with arch-. In a typical installation, it will be arch-linux-c-opt.

11. Save an environment variable that will tell TOSCA where HDF5 libraries/include files will be located in your .bashrc. For HPC installations it might be a better practice to use the .bash_profile instead. This variable will be used in the TOSCA makefile to compile tosca2PV. If you have installed HDF5 using the --download-hdf5 option in the PETSc configure step, the HDF5 lib and include directories will be located in the $PETSC_DIR/$PETSC_ARCH/ folder, together with HYPRE and PETSc ones.

    echo "export HDF5_DIR=$PETSC_DIR/$PETSC_ARCH" >> $HOME/.bashrc
    

    or, for HPCs

    echo "export HDF5_DIR=$PETSC_DIR/$PETSC_ARCH" >> $HOME/.bash_profile
    

12. Add the PETSc shared libraries to your library path environment variable in your .bashrc. For HPC installations it might be a better practice to use the .bash_profile instead.

    echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PETSC_DIR/$PETSC_ARCH/lib" >> $HOME/.bashrc
    

    or, for HPCs

    echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PETSC_DIR/$PETSC_ARCH/lib" >> $HOME/.bash_profile
    

13. Reload the environment with

    source $HOME/.bashrc
    

    of, for HPCs

    source $HOME/.bash_profile
    

14. Go inside TOSCA/src directory and compile the executables. Make sure to have added the path to custom libraries (i.e. compiled locally by the user) if any, such as OpenMPI and HDF5, to the LD_LIBRARY_PATH as done for the PETSc libraries at step 10.

    make tosca
    make tosca2PV
    

15. Test the installation by copying tosca and tosca2PV in one of the example cases and run the simulation and the post-processing with ./tosca and ./tosca2PV respectively. To run in parallel you have to use mpirun -np 'your-number-of-processors' ./tosca

Tip

If you run using mpirun ./tosca, MPI will use the maximum number of processors available.

Tip

You can add tosca and tosca2PV to the PATH so that they will be found as executables without the need to copy them inside the case directory.

Contribute to the TOSCA Project

The TOSCA repository is open-source, so anyone can download and use the code. If you want to contribute to the project by adding code to TOSCA repository you need to open a pull-request that has to be approved by our team. In order to do so, please use the following steps:

1. Clone the TOSCA package locally on your machine with git clone https://github.com/sebastipa/TOSCA.git

2. Create a new local branch with git checkout -b your-branch-name

3. Make the desired changes to the TOSCA code, then check which files have been modified with git status

4. Add changes to the git stack with git add modified-files

5. Commit the changes using a short but exhaustive comment with git commit -m "your-commit-description"

6. Push your local branch online with git push origin your-branch-name

7. Go to github, select your branch, click on Contribute and open a pull-request describing the motivation of your changes, their effect on the code and the tests you performed.

8. After approval of the pull-request by our team, commits will be added to the main TOSCA version

9. To stay up-to-date, rebase your local master with the your new commits by first checking out in your local master branch with git checkout master and then rebase with git pull --rebase origin master

10. Delete your local branch as it is not useful anymore with git branch --delete -d your-branch-name and use the master until you want to make new changes

Paraview-Catalyst2 OS-Rendering

TOSCA provides full interface with Paraview-catalyst2 through the USE_CATALYST flag in the makefile. If this is the case, the CATALYST environment variable should point to the catalyst2 installation directory. Paraview-catalyst is optional and can be disabled by setting USE_CATALYST=0

Usage

In order to activate off-screen rendering capabilities, -pvCatalyst=1 should be set in the control.dat file. A file called catalystProperties will be required inside the sampling directory. Entries to this file are

ioType	can be set to script or general
outputType	can be set to timeStep or adjustableTime
startTime	model-time at which catalyst actions start
timeInterval	acquisition period in seconds if outputType is set to adjustableTime or iterations if outputType is set to timeStep
scriptName	name of the catalyst actions python script. Only required if ioType=script

If ioType=general, 3D fields of velocity magnitude, pressure and q-criterion are saved inside the catalyst/ folder. If ioType=script, Praview actions defined in the python script are executed and e.g. png images can be saved at runtime.

Installation

In order to be installed, Paraview-catalyst2 requires a working C/C++ compiler, Open MPI (version 4.0.x, 4.1.x), Python3 and cmake. In order for Paraview to work, OpenGL must be available at runtime or mesa libraries are required to mimic some hardware components. These are usually available on supercomputers through the ‘mesa’ module, which should be loaded at runtime. Lastly, paraview and catalyst2 should be manually compiled of the system. As Paraview-5.10 contains a bug in the definition of rectilinear mesh (used by TOSCA), Paraview-5.11 or later is recommended.

Prior to install TOSCA, we suggest to create a folder named Software inside $HOME, where catalyst2 and paraview-5.11 will be located. In order to re-compile TOSCA with Paraview-catalyst2 capabilities on your system, please follow these steps:

1. Install catalyst2:

export LOCATION=$HOME/Software
cd $LOCATION
mkdir catalyst2 && cd catalyst2
git clone https://gitlab.kitware.com/paraview/catalyst.git catalyst-src && cd catalyst-src
mkdir -p build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=$LOCATION/catalyst2/install
make
make install
echo "export CATALYST=$LOCATION" >> $HOME/.bashrc

Add the Catalyst2 shared libraries to your library path environment variable in your .bashrc with echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LOCATION/catalyst2/install/lib" >> $HOME/.bashrc

Note

In some cases you may have to replace lib with lib64

2. Install Paraview-5.11:

cd $LOCATION
mkdir paraview-5.11.0 && cd paraview-5.11.0
wget https://www.paraview.org/files/v5.11/ParaView-v5.11.0.tar.xz
tar -xvf ParaView-v5.11.0.tar.xz
mv ParaView-v5.11.0 paraview-src && cd paraview-src
mkdir -p build && cd build
export CMAKE_PREFIX_PATH=$CMAKE_PREFIX_PATH:$LOCATION/catalyst2/install/lib/cmake/catalyst-2.0

Note

In some cases you may have to replace lib with lib64

FLAGS=(-DCMAKE_INSTALL_PREFIX=$LOCATION/paraview-5.11.0/install -DVTK_OPENGL_HAS_OSMESA=ON -DPARAVIEW_USE_MPI=ON -DBUILD_TESTING=OFF -DVTK_USE_X=OFF -DPARAVIEW_USE_QT=OFF -DPARAVIEW_USE_PYTHON=ON -DPython3_FIND_STRATEGY=LOCATION -DPython3_ROOT_DIR=$EBROOTPYTHON -DPARAVIEW_BUILD_SHARED_LIBS=ON -DPARAVIEW_ENABLE_RAYTRACING=OFF -DPARAVIEW_ENABLE_CATALYST=ON )
cmake .. "${FLAGS[@]}"
make -j8
make install

Add the Paraview shared libraries to your library path environment variable in your .bashrc with echo "export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LOCATION/paraview-5.11.0/install/lib" >> $HOME/.bashrc

Note

In some cases you may have to replace lib with lib64

3. Re-compile TOSCA

   • Reload the environment with source $HOME/.bashrc

   • Go inside TOSCA/src directory and recompile the solver with make tosca, ensuring that -DUSE_CATALYST=1 in the makefile

4. Running

   In order for catalyst2 to find paraview library, the following envronment variables should be set at runtime:

   • export CATALYST_IMPLEMENTATION_PATHS=$LOCATION/paraview-5.11.0/install/lib/catalyst

   • export CATALYST_IMPLEMENTATION_NAME=paraview

Note

In some cases you may have to replace lib with lib64