This section discusses a protocol for sending a logical message on top of message-oriented transport like MPI, and discusses the physical messages that are sent. On top of a stream-oriented transport like TCP, some of these considerations may be less relevant. Figure 5.12 illustrates layout of possible send messages.
When a buffer object is created, a capacity is defined for that buffer. This capacity is the available space for the primary payload buffer (in practise an extra 16 bytes will be allocated to allow for the primary header and the secondary header, but the user need not be concerned with this).
Through write operations on the buffer, the user will initialize the primary payload. These operations may also define a non-empty secondary payload, presumably held in separate storage.
When the time to send the message comes, the primary and secondary headers are written to the buffer. If the secondary payload is empty, a single message is sent containing the primary payload along with the primary header and secondary header, the latter specifying a size of zero.
In this case (when it is known in advance that the secondary payload will be empty--for example because only primitive elements are expected) it is required that the receiver accepts the message into a buffer object with a capacity equal to at least the actual size of the primary payload.
If the secondary payload is not empty, it may either be concatenated to the same message, or sent as a second message (in which case the first message still contains the primary header, the primary buffer and the secondary header). If there are two messages, the receiver allocates an array large enough to hold the secondary payload after the first message (containing the secondary header) has been received.
If the secondary payload is not empty, but the sum of its size and the size of the primary payload are less than the capacity of the buffer object at the sending end, the two may be concatenated together in a single message.
This implies that if it is possible that the secondary payload may not be empty, the receiver should accept the message into a buffer object with a capacity equal to at least the capacity of the buffer object used to send the message. Otherwise a concatenated message might overflow the receive buffer. Note this is a stronger requirement than in the case where all message elements are primitive (and there is no secondary payload).
Typical message-based transports allow the receiver to determine the actual number of bytes received in a message (in MPI, through the MPI_Status object). Moreover, by examining the primary and secondary header contained in the received message, the receiver can compute the expected size of the complete, logical message. If this is greater than the number of bytes in the physical message, a second message must be expected.
The simplest safe policy is probably to try always to accept a message
into a buffer object with identical capacity to the buffer used
to send the message. This policy can be relaxed if the message is
known only to involve primitive elements, but care should be taken.