High-level Communication Library

In this section we discuss extra syntax and usage of high-level communication library in HPJava programs. Two characteristic collective communication methods remap() and writeHalo() are described as examples.

Figure 3.3: A general Matrix multiplication in HPJava.

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} public void... ...m += ta [i, k] * tb [k, j]; c[i, j] = sum; } }\end{verbatim}\end{minipage}$$

We discuss more detail information about the general purpose matrix multiplication routine (Figure 3.3). The method has two temporary arrays ta, tb with the desired distributed format. This program is also using information which is defined for any distributed array: grp() to fetch the distribution group and rng() to fetch the index ranges. This example relies on a high-level Adlib communication schedule that deals explicitly with distributed arrays; the remap() method. The remap() operation can be applied to various ranks and type of array. Any section of an array with any allowed distribution format can be used. Supported element types include Java primitive and Object type. A general API for the remap function is

$$\begin{minipage}[t]{\linewidth}\small\begin{verbatim} void remap ... ... void remap (T [[-,-]] dst, T [[-,-]] src) ; ...\end{verbatim}\end{minipage}$$

where T is a Java primitive or Object type. The arguments here are zero-dimensional, one-dimensional, two-dimensional, and so on. We will often summarize these in the shorthand interface:

where the signature T # means any distributed array with elements of type T (This syntax is not supported by the current HPJava compiler, but it supports method signatures of this generic kind in externally implemented libraries--ie. libraries implemented in standard Java. This more concise signature does not incorporate the constraint that dst and src have the same rank--that has to be tested at run-time.)

Figure 3.4: Solution of Laplace equation by Jacobi relaxation.

As another example, Figure 3.4 is a HPJava program for the Laplace program that uses ghost regions. It illustrates the use the library class ExtBlockRange to create arrays with ghost extensions. In this case, the extensions are of width 1 on either side of the locally held ``physical'' segment. Figure 3.5 illustrates this situation.

Figure 3.5: Example of a distributed array with ghost regions.

From the point of view of this dissertation the most important feature of this example is the appearance of the function Adlib.writeHalo(). This is a collective communication operation. This particular one is used to fill the ghost cells or overlap regions surrounding the ``physical segment'' of a distributed array. A call to a collective operation must be invoked simultaneously by all members of some active process group (which may or may not be the entire set of processes executing the program). The effect of writeHalo is to overwrite the ghost region with values from processes holding the corresponding elements in their physical segments. Figure 3.6 illustrates the effect of executing the writeHalo function. More general forms of writeHalo may specify that only a subset of the available ghost area is to be updated, or may select cyclic wraparound for updating ghost cells at the extreme ends of the array. If an array has ghost regions the rule that the subscripts must be simple distributed indices is relaxed; shifted indices, including a positive or negative integer offset, allow access to elements at locations neighboring the one defined by the overall index.

Figure 3.6: Illustration of the effect of executing the writeHalo function.

Besides remap() and writeHalo(), Adlib includes a family of related regular collective communication operations (shifts, skews, and so on). It also incorporates a set of collective gather and scatter operations for more irregular communications, and a set of reduction operations based on the corresponding Fortran 90 array intrinsics. Reduction operations take one or more distributed arrays as input. They combine the elements to produce one or more scalar values, or arrays of lower rank. Currently our collective communication library is built on top of device level communication library called mpjdev. The mpjdev API is designed with the goal that it can be implemented portably on network platforms and efficiently on parallel hardware. This library is developed with HPJava in mind, but it is a standalone library and could be used by other systems. The main purpose of this library is to perform actual point-to-point communications between processes. We will discuss implementation issues of high- and low-level communication libraries in Chapter 4 and Chapter 5.