Halo Exchanges (Halo)

Omega utilizes domain-based parallelism, where the domain is divided into partitions that run in parallel as separate tasks distributed across the resources of the machine. Arrays representing physical quantities that are defined throughout the domain are divided into overlapping chunks and distributed among the tasks. As the model evolves, the values stored in these arrays are updated. Array elements that are needed by a task to evolve the model forward in time, but are defined on and updated by a neighboring task are halo elements. It is necessary for each parallel task to regularly send locally defined array elements that belong to the halos of neighboring tasks and to receive local halo elements owned by neighboring tasks. In Omega, these halo exchanges are perfromed using the Halo class which, once constructed, contains lists of indices to send to and receive from neighboring tasks, and methods to conduct the exchanges. The halo exchanges are carried out using non-blocking MPI (Message Passing Interface) communication.

An instance of the Halo class is dependent on a parallel domain decomposition and the associated parallel machine environment, so those objects need to be constructed first. The default Halo is created with the call:

OMEGA::Halo::init();

This must be done after the default MachEnv and Decomp have been initialized. Once initialized, a pointer to the default Halo can be retrieved with:

OMEGA::Halo *DefHalo = OMEGA::Halo::getDefault();

Additional Halo objects can be constructed for defined MachEnv and Decomp objects by calling the Halo create method with a supplied std::string Name:

OMEGA::Halo::create(Name, NewEnv, NewDecomp);

A pointer to each constructed Halo is stored in a std::map container, which can be retrieved by supplying the Name to OMEGA::Halo::get(Name).

The Halo constructor will save the MPI communicator handle and the MPI task ID of the local task from the MachEnv object, and will extract and save from the Decomp object indices defining the local and neighboring halo elements for the cell, edge, and vertex index spaces. This information is organized into objects of two nested classes of the Halo class: the ExchList class and the Neighbor class. The ExchList class contains the indices of Halo elements to either send to or receive from a single neighboring task for a particular index space, and the Neighbor class contains all the ExchList objects needed to carry out a halo exchange with a single neighboring task in any index space, as well as the buffer memory space to communicate with that neighbor. All member variables and objects and most member methods of the Halo class are declared private as they are only needed by the Halo class methods to execute an exchange.

The main methods of the Halo class which execute an exchange are

  • startReceives: wrapper to call MPI_Irecv for each neighbor

  • packBuffer: packs halo elements into a buffer to send to a neighbor

  • startSend: wrapper to call MPI_Isend to send a buffer to each neighbor

  • unpackBuffer: unpacks halo elements from a received buffer into an array

The packBuffer and unpackBuffer functions are overloaded to support different array types.

The exchangeFullArrayHalo function is the main interface to conduct a halo exchange for any supported array defined in the DataTypes module. The exchangeFullArrayHalo function is a template which passes the array to the Halo methods above which perform the exchange. The index space on which the array is defined must also be passed to this function. This is currently done using an enum defined in the Halo header file.

enum MeshElement{OnCell, OnEdge, OnVertex};

So, the exchangeFullArrayHalo function is called as follows:

MyHalo.exchangeFullArrayHalo(SomeCellBasedArray, OMEGA::OnCell);

for a constructed Halo named MyHalo and any supported array type in the cell index space.