@@ -62,8 +62,40 @@ The TNL project's documentation (\url{https://tnl-project.org/documentation/})\t
In the following subsections, we describe the \emph{distributed} version of the data structure, which is especially useful for the algorithms presented in \cref{chapter:numerical methods}.
In the \emph{distributed} configuration, a global multidimensional array is decomposed into several subarrays and each MPI rank typically stores the data associated to one subarray.
Since a multidimensional array stores structured data, we will consider only \emph{structured conforming} decompositions, where the array is split by hyperplanes perpendicular to one of the axes.
\Cref{fig:ndarray decomposition}\todo{draw the figure} shows a typical two-dimensional decomposition of a two-dimensional array into 9 subarrays.
In order to represent a distributed multidimensional array, each MPI rank needs to have the following variables:
\begin{itemize}
\item\ic{localArray} -- an instance of a multidimensional array which contains data and indexing attributes of the subarray assigned to the rank.
\item\ic{communicator} -- the MPI communicator comprising all ranks that have a subarray of the global multidimensional array.
\item\ic{globalSizes} -- a multi-index determining the sizes of the global multidimensional array in terms of the number of elements in each dimension.
\item\ic{localBegins} and \ic{localEnds} -- two multi-indices determining the beginning and ending position of the local subarray in the global array.
According to the convention used in TNL, the local subarray spans the multidimensional interval $[\ic{localBegins}, \ic{localEnds})$.
\end{itemize}
Although these attributes of a distributed multidimensional array can be created separately and managed manually, TNL provides a convenient data structure \ic{TNL::Containers::DistributedNDArray} that provides a high-level interface to manage these attributes.
The TNL project's documentation (\url{https://tnl-project.org/documentation/})\todo{cite as Online Resource?} provides an overview of the public interface and examples showing how the data structure can be used.\todo{write the documentation for \ic{DistributedNDArray}}
\subsection{Operations}
Algorithms involving a distributed multidimensional array may perform different operations on the data structure.
Common operations may be classified as follows:
\begin{itemize}
\item
\emph{Local operations} are the simplest type of operations that can be performed on each MPI rank independently of the other ranks.
An example is the \ic{setValue} member function of \ic{DistributedNDArray} which sets all elements of the array to a constant value.
\item
\emph{Collective operations} involve some kind of communication among all ranks in the MPI communicator.
An example is the \ic{operator==} function for the comparison between two instances of \ic{DistributedNDArray}, which involves a local operation (comparison of two local arrays) followed by a collective \ic{AND}-reduction of the local results among all ranks.
\item
\emph{Partially-collective operations} involve communication not on the global communicator, but among smaller groups of MPI ranks.
It is often convenient to create MPI sub-communicators using the \ic{MPI_Comm_split} function \todo{describe in \cref{chapter:programming and architectures} and reference it here} to simplify the communication pattern definition in the program.
For example, computing the sums of array elements per row involves the communication only among ranks containing the same rows.
Another example are stencil computations on regular grids, which typically involve data exchange among neighboring ranks.
\end{itemize}
Additionally, complex algorithms may involve data exchange prior to performing the local operation (such as the input vector re-distribution in the distributed matrix--vector multiplication).
General description of such algorithms is out of scope of this work.
In the following subsection, we focus on data exchange in stencil computations, which are common in numerical modeling.
