LBM section - added figure with the diagram of classes in the LBM implementation

Recall that the global array corresponding to the lattice $\mathcal L_{\overline \Omega}$ must be decomposed in a \emph{structured conforming} manner, but each rank can be assigned multiple subdomains and the rank numbering is arbitrary.

The decomposition of the aforementioned arrays in LBM must be consistent.

Hence, we collect all distributed data structures in a class called \ic{LBM_BLOCK} that contains all data associated to one subdomain: local subarrays, subdomain sizes and offsets, indices of neighboring blocks and ranks that own them.

\todo{add figure: diagram of the classes/objects and their relations}

To facilitate computations on GPU accelerators, two objects for each array are needed to represent data in different memory spaces.

One object allocates the array data in the host memory where it can be processed by the CPU, and the other object allocates the array data in the device memory where it can be processed by the GPU accelerator.

Furthermore, additional attribute represented by the \ic{LBM_DATA} class is added to \ic{LBM_BLOCK}, which contains the data that need to be passed to the CUDA kernel when the processing of the block on the GPU starts.

This includes pointers to the arrays allocated in the device memory, sizes of the lattice, and scalar parameters (e.g., viscosity and parameters related to boundary conditions).

The \ic{LBM_BLOCK} objects representing individual subdomains managed by the current process (MPI rank) are aggregated in the class named \ic{LBM}.

\begin{figure}[tb]

    \centering

    \includegraphics[width=\textwidth]{figures/lbm/class_diagram.pdf}

    \caption{Schematic diagram showing the relations between the main \C++ classes and meta-programming components in the LBM implementation.}

    \label{fig:lbm:class diagram}

\end{figure}

The relation between the \ic{LBM_DATA} and \ic{LBM_BLOCK} classes is highlighted in \cref{fig:lbm:class diagram} along with additional classes that will be described later.

The \ic{LBM_BLOCK} objects representing individual subdomains managed by the current process (MPI rank) are aggregated in an \ic{std::vector} container in the class named \ic{LBM}.

The \ic{LBM} class provides member functions to manipulate all managed blocks collectively, as well as additional functionality related to LBM simulations, such as variables for time-stepping, or conversion between physical and lattice units.

From a high level point of view, each MPI rank creates one object of the \ic{State} class, which combines all components needed to run a simulation.

It contains one object of the \ic{LBM} class to represent one set of discrete distribution functions for a Navier--Stokes problem.

The default implementation of \ic{State} contains one object of the \ic{LBM} class to represent one set of discrete distribution functions for a Navier--Stokes problem.

The \ic{State} class is designed to be extended via inheritance, so users can define their own subclass for a particular case and adjust the behavior of the solver by adding attributes and overriding various virtual member functions.

Hence, it is possible to implement more complicated solvers for coupled problems, such as the Navier--Stokes--Fourier problem of fluid flow coupled with thermal transport \cite{Sharma2020}, which can be solved using multiple sets of discrete distribution functions represented by objects of the \ic{LBM} class.

Therefore, our LBM implementation is based on several template meta-programming components that allow to configure the low-level details while preserving the generation of optimal CUDA kernels for each configuration.

The starting element of template meta-programming used in the code is the \ic{LBM_CONFIG} template class that represents a particular configuration for LBM.

It combines the following components that are set via template parameters:

As shown in \cref{fig:lbm:class diagram}, it combines the following components that are set via template parameters:

\begin{itemize}

    \item

        \ic{TRAITS} is a structure that holds the definition of fundamental types used in the solver, such as the floating-point type, index type, permutations of the multidimensional arrays, etc.

The code base provides multiple models for each component with a sensible default behavior.

Users can subclass each model, implement their modifications and pass their type to \ic{LBM_CONFIG}.

All the aforementioned classes \ic{LBM_BLOCK}, \ic{LBM}, and \ic{State} are in fact templates with one parameter, \ic{LBM_CONFIG}.

All the aforementioned classes \ic{LBM_BLOCK}, \ic{LBM}, and \ic{State} are in fact templates with one parameter, \ic{CONFIG}, as shown in \cref{fig:lbm:class diagram}.

The modifications of the components such as \ic{BC} or \ic{MACRO} may need to be designed together with the extensions of \ic{State}, especially when custom attributes are added to the \ic{DATA} component.

\subsection{Optimization remarks}

// fonts used in the thesis

usepackage("libertine", options="tt=false, type1=true");

usepackage("zi4", options="varqu");

usepackage("newtxmath", options="libertine");

import texcolors;

usepackage("xcolor", options="dvipsnames");

// command for highlighting C++ keywords

texpreamble("\newcommand{\kw}[1]{\textbf{\color{OliveGreen} #1}}");

//size(9cm, 10cm, IgnoreAspect);

unitsize(1cm);

pair d = (5.25, 1.5);

real offset_y = 3;

real s = 1.6d.x;

real y = 0.6;

defaultpen(fontsize(10pt));

pen PEN_ATTR = solid + RoyalBlue;

pen PEN_HINT = dotted + RedViolet;

//pen PEN_HINT = linewidth(1.25) + dotted + RedViolet;

picture class(string ttname, string classname, pair z=(0,0))

{

    picture pic;

    draw(pic, box(-d/2, d/2));

    label(pic, minipage(

              "\tt \kw{template}< \kw{typename} " + ttname + " >\\ "

              "\kw{class} " + classname + "\\ "

              "\{\;\ldots\};",

              5cm

          ),

          (0,0));

    return shift(z) * pic;

}

picture class(string s, pair z=(0,0), pair size=d)

{

    picture pic;

    draw(pic, box(-size/2, size/2));

    label(pic, minipage(s, 5cm), (0,0));

    return shift(z) * pic;

}

picture State = class("CONFIG", "State");

picture LBM = class("CONFIG", "LBM", (0, -offset_y));

picture LBM_BLOCK = class("CONFIG", "LBM\_BLOCK", (0, -2*offset_y));

//picture LBM_DATA = class("TRAITS", "LBM\_DATA", (0, -3*offset_y));

picture LBM_DATA = class("\texttt{\kw{typename} CONFIG::DATA}\\ (model of \texttt{LBM\_DATA})", (0, -2.9*offset_y), size=(d.x, 1.1));

add(State);

add(LBM);

add(LBM_BLOCK);

add(LBM_DATA);

// relation between State and LBM

path p1 = point(State,SW)+(0.5,0)--point(LBM,NW)+(0.5,0);

draw(p1, LeftSide, PEN_ATTR, Arrow, PenMargin);

p1 = point(State,SW)+(0.75,0)--point(LBM,NW)+(0.75,0);

draw(p1, LeftSide, PEN_ATTR + dashed, Arrow, PenMargin);

p1 = point(State,SW)+(1,0)--point(LBM,NW)+(1,0);

draw(minipage("\color{RoyalBlue} \flushleft{has attribute \\ \scriptsize (subclasses can add more instances)}", 5cm),

     p1,

     LeftSide,

     PEN_ATTR + dashed,

     Arrow,

     PenMargin

);

// relation between LBM and LBM_BLOCK

path p2 = point(LBM,SW)+(0.5,0)--point(LBM_BLOCK,NW)+(0.5,0);

draw(p2, LeftSide, PEN_ATTR, Arrow, PenMargin);

p2 = point(LBM,SW)+(0.75,0)--point(LBM_BLOCK,NW)+(0.75,0);

draw(p2, LeftSide, PEN_ATTR + dashed, Arrow, PenMargin);

p2 = point(LBM,SW)+(1,0)--point(LBM_BLOCK,NW)+(1,0);

draw(minipage("\color{RoyalBlue} \flushleft{has attributes \\ \scriptsize (in \texttt{std::vector})}", 3cm),

     p2,

     LeftSide,

     PEN_ATTR + dashed,

     Arrow,

     PenMargin

);

// relation between LBM_BLOCK and LBM_DATA

path p3 = point(LBM_BLOCK,SW)+(0.5,0)--point(LBM_DATA,NW)+(0.5,0);

draw(minipage("\color{RoyalBlue} \flushleft{has attribute}", 2.25cm),

     p3,

     LeftSide,

     PEN_ATTR,

     Arrow,

     PenMargin

);

picture pic_tmp;

draw(pic_tmp, box((-3.75cm,-4.75cm), (3.75cm,4.75cm)));

label(pic_tmp, minipage(

"\tt

\kw{template}< \\ 

\phantom{~~} \kw{typename} \_TRAITS, \\ 

\phantom{~~} \kw{typename} \_KERNEL\_STRUCT, \\ 

\phantom{~~} \kw{typename} \_DATA, \\ 

\phantom{~~} \kw{typename} \_COLL, \\ 

\phantom{~~} \kw{typename} \_EQ, \\ 

\phantom{~~} \kw{typename} \_STREAMING, \\ 

\phantom{~~} \kw{typename} \_BC, \\ 

\phantom{~~} \kw{typename} \_MACRO \\ 

> \\ 

\kw{struct} LBM\_CONFIG \\ 

\{ \\ 

\phantom{~~} \kw{using} TRAITS = \_TRAITS; \\ 

\phantom{~~} \kw{using} KERNEL\_STRUCT = \_KERNEL\_STRUCT; \\ 

\phantom{~~} \kw{using} DATA = \_DATA; \\ 

\phantom{~~} \kw{using} COLL = \_COLL; \\ 

\phantom{~~} \kw{using} EQ = \_EQ; \\ 

\phantom{~~} \kw{using} STREAMING = \_STREAMING; \\ 

\phantom{~~} \kw{using} BC = \_BC; \\ 

\phantom{~~} \kw{using} MACRO = \_MACRO; \\ 

\phantom{~~} \ldots \\ 

\};

",

          7cm

      ),

      (0,0));

add(pic_tmp, (8, -4));

label("Data structures:", point(State,NW)+(0,0.5), align=RightSide);

label("Meta-programming components:", point(State,NW)+(6.8,0.5), align=RightSide);

// add arrows for comprehension hints

pair p_config = (4.45,-3.75);

draw(point(State,NE)-(1.25,0.5){down}..(2.75,-1.5){SE}..p_config{right}, PEN_HINT, BeginArrow, PenMargin);

draw(point(LBM,NE)-(1.25,0.5){down}..p_config{right}, PEN_HINT, BeginArrow, PenMargin);

draw(point(LBM_BLOCK,NE)-(1.25,0.05){up}..p_config{right}, PEN_HINT, BeginArrow, PenMargin);

//pair p_traits = (5,-4.6);

//draw(point(LBM_DATA,NE)-(1.25,0.05){up}..(3,-7)..p_traits{right}, PEN_HINT, BeginArrow, PenMargin);

//pair p_data = (5,-5.45);

//draw(point(LBM_DATA,E)-(2.5,-0.05){right}..(3.5,-8)..p_data{right}, PEN_HINT, BeginArrow, PenMargin);

pair p_data = (4.95,-5.45);

draw(point(LBM_DATA,E)-(1.2,-0.275){right}..(3.5,-8)..p_data{right}, PEN_HINT, BeginArrow, PenMargin);