path: root/qpid/doc/book/src/cpp-broker/Using-message-groups.xml

<?xml version="1.0" encoding="utf-8"?>
<!--
 
 Licensed to the Apache Software Foundation (ASF) under one
 or more contributor license agreements.  See the NOTICE file
 distributed with this work for additional information
 regarding copyright ownership.  The ASF licenses this file
 to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at
 
   http://www.apache.org/licenses/LICENSE-2.0
 
 Unless required by applicable law or agreed to in writing,
 software distributed under the License is distributed on an
 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 KIND, either express or implied.  See the License for the
 specific language governing permissions and limitations
 under the License.
 
-->

<section id="Using-message-groups">
  <title>
    Using Message Groups
  </title>

  <section role="h2" id="usingmessagegroups-Overview">
    <title>
      Overview
    </title>
      <para>
        The broker allows messaging applications to classify a set of related messages as
        belonging to a group.  This allows a message producer to indicate to the consumer
        that a group of messages should be considered a single logical operation with
        respect to the application.
      </para>
      <para>
        The broker can use this group identification to enforce policies controlling how
        messages from a given group can be distributed to consumers.  For instance, the
        broker can be configured to guarantee all the messages from a particular group are
        processed in order across multiple consumers.
      </para>
      <para>
        For example, assume we have a shopping application that manages items in a virtual
        shopping cart.  A user may add an item to their shopping cart, then change their
        mind and remove it.  If the application sends an <emphasis>add</emphasis> message to the broker,
        immediately followed by a <emphasis>remove</emphasis> message, they will be queued in the proper
        order - <emphasis>add</emphasis>, followed by <emphasis>remove</emphasis>.
      </para>
      <para>
        However, if there are multiple consumers, it is possible that once a consumer
        acquires the <emphasis>add</emphasis> message, a different consumer may acquire the
        <emphasis>remove</emphasis> message.  This allows both messages to be processed in parallel,
        which could result in a "race" where the <emphasis>remove</emphasis> operation is incorrectly
        performed before the <emphasis>add</emphasis> operation.
      </para>
    </section>
    <!--h2-->
    <section role="h2" id="usingmessagegroups-GroupingMessages">
      <title>
        Grouping Messages
      </title>
      <para>
        In order to group messages, the application would designate a particular
        message header as containing a message's <emphasis>group identifier</emphasis>.  The group
        identifier stored in that header field would be a string value set by the message
        producer.  Messages from the same group would have the same group identifier
        value. The key that identifies the header must also be known to the message
        consumers.  This allows the consumers to determine a message's assigned group.
      </para>
      <para>
        The header that is used to hold the group identifier, as well as the values used
        as group identifiers, are totally under control of the application.
      </para>
    </section>
    <section role="h2" id="usingmessagegroups-BrokerRole">
      <title>
        The Role of the Broker
      </title>
      <para>
        The broker will apply the following processing on each grouped message:
        <itemizedlist>
          <listitem>Enqueue a received message on the destination queue.</listitem>
          <listitem>Determine the message's group by examining the message's group identifier header.</listitem>
          <listitem>Enforce <emphasis>consumption ordering</emphasis> among messages belonging to the same group.</listitem>
        </itemizedlist>
        <emphasis>Consumption ordering</emphasis> means that the broker will not allow outstanding
        unacknowledged messages to <emphasis>more than one consumer for a given group</emphasis>.
      </para>
      <para>
        This means that only one consumer can be processing messages from a particular
        group at a given time.  When the consumer acknowledges all of its acquired
        messages, then the broker <emphasis>may</emphasis> pass the next pending message
        from that group to a different consumer.
      </para>
      <para>
        Specifically, for any given group the broker allows only the first N messages in
        the group to be delivered to a consumer.  The value of N would be determined by
        the selected consumer's configured prefetch capacity.  The broker blocks access by
        any other consumer to any remaining undelivered messages in that group.  Once the
        receiving consumer has:
        <itemizedlist>
          <listitem>acknowledged,</listitem>
          <listitem>released, or</listitem>
          <listitem>rejected</listitem>
        </itemizedlist>
        all the delivered messages, the broker allows the next messages in the group to be
        delivered.  The next messages <emphasis>may</emphasis> be delivered to a different
        consumer.
      </para>
      <para>
        Note well that distinct message groups would not block each other from delivery.
        For example, assume a queue contains messages from two different message groups -
        say group "A" and group "B" - and they are enqueued such that "A"'s messages are
        in front of "B". If the first message of group "A" is in the process of being
        consumed by a client, then the remaining "A" messages are blocked, but the
        messages of the "B" group are available for consumption by other consumers - even
        though it is "behind" group "A" in the queue.
      </para>
    </section>
    <section role="h2" id="usingmessagegroups-ConsumerGuide">
      <title>
        Well Behaved Consumers
      </title>
      <para>
        The broker can only enforce policy when delivering messages. To guarantee that
        strict message ordering is preserved, the consuming application must adhere to the
        following rules:
        <itemizedlist>
          <listitem>completely process the data in a received message before accepting
          that message</listitem>
          <listitem>acknowledge (or reject) messages in the same order as they are
          received</listitem>
          <listitem>avoid releasing messages (see below)</listitem>
        </itemizedlist>
        The term <emphasis>processed</emphasis> means that the consumer has finished
        updating all application state affected by the message that has been received.
        See section 2.6.2. Transfer of Responsibility, of the AMQP-0.10 specification for
        more detail.
      </para>
      <note>
        <title>Be Advised</title>
        <para>
          If a consumer does not adhere to the above rules, it may affect the ordering of
          grouped messages even when the broker is enforcing consumption order.  This can
          be done by selectively acknowledging and releasing messages from the same group.
        </para>
        <para>
          Assume a consumer has received two messages from group "A", "A-1" and "A-2", in
          that order.  If the consumer releases "A-1" then acknowledges "A-2", "A-1" will
          be put back onto the queue and "A-2" will be removed from the queue.  This
          allows another consumer to acquire and process "A-1" <emphasis>after</emphasis>
          "A-2" has been processed.
        </para>
        <para>
          Under some application-defined circumstances, this may be acceptable behavior.
          However, if order must be preserved, the client should either release
          <emphasis>all</emphasis> currently held messages, or discard the target message
          using reject.
        </para>
      </note>
    </section>
    <!--h2-->
    <section role="h2" id="usingmessagegroups-BrokerConfig">
      <title>
        Broker Configuration
      </title>
      <para>
        In order for the broker to determine a message's group, the key for the header
        that contains the group identifier must be provided to the broker via
        configuration.  This is done on a per-queue basis, when the queue is first
        configured.
      </para>
      <para>
        This means that message group classification is determined by the message's destination
        queue.
      </para>
      <para>
        Specifically, the queue "holds" the header key that is used to find the message's
        group identifier.  All messages arriving at the queue are expected to use the same
        header key for holding the identifer.  Once the message is enqueued, the broker
        looks up the group identifier in the message's header, and classifies the message
        by its group.
      </para>
      <para>
        Message group support can be enabled on a queue using the
        <command>qpid-config</command> command line tool.  The following options should be
        provided when adding a new queue:
        <table>
          <title>qpid-config options for creating message group queues</title>
          <tgroup cols="2">
            <thead>
              <colspec colnum="1" colwidth="1*"/>
              <colspec colnum="2" colwidth="3*"/>
              <row>
                <entry>Option</entry><entry>Description</entry>
              </row>
            </thead>
            <tbody>
              <row>
                <entry>--group-header=<replaceable>header-name</replaceable></entry>
                <entry>Enable message group support for this queue. Specify name of application header that holds the group identifier.</entry>
              </row>
              <row>
                <entry>--shared-groups</entry>
                <entry>Enforce ordered message group consumption across multiple consumers.</entry>
              </row>
            </tbody>
          </tgroup>
        </table>
      </para>
      <para>
        Message group support may also be specified in the
        <command>queue.declare</command> method via the <command>arguments</command>
        parameter map, or using the messaging address syntax.  The following keys must be
        provided in the arguments map to enable message group support on a queue:
      </para>
      <table>
        <title>Queue Declare/Address Syntax Message Group Configuration Arguments</title>
        <tgroup cols="2">
          <thead>
            <row>
              <entry>Key</entry>
              <entry>Value</entry>
            </row>
          </thead>
          <tbody>
            <row>
              <entry>qpid.group_header_key</entry>
              <entry>string - key for message header that holds the group identifier value</entry>
            </row>
            <row>
              <entry>qpid.shared_msg_group</entry>
              <entry>1 - enforce ordering across multiple consumers</entry>
            </row>
          </tbody>
        </tgroup>
      </table>
      <para>
        It is important to note that there is no need to provide the actual group
        identifer values that will be used. The broker learns this values as messages are
        recieved.  Also, there is no practical limit - aside from resource limitations -
        to the number of different groups that the broker can track at run time.
      </para>
      <note>
        <title>Restrictions</title>
        <para>
          Message grouping is not supported on LVQ or Priority queues.
        </para>
      </note>
      <example>
        <title>Creating a message group queue via qpid-config</title>
        <para>
          This example uses the qpid-config tool to create a message group queue called
          "MyMsgQueue".  The message header that contains the group identifier will use
          the key "GROUP_KEY".
        </para>
        <programlisting>
qpid-config add queue MyMsgQueue --group-header="GROUP_KEY" --shared-groups
        </programlisting>
      </example>
      <example>
        <title>Creating a message group queue using address syntax (C++)</title>
        <para>
          This example uses the messaging address syntax to create a message group queue
          with the same configuration as the previous example.
        </para>
        <programlisting>
sender = session.createSender("MyMsgQueue;"
                              " {create:always, delete:receiver,"
                              " node: {x-declare: {arguments:"
                              " {'qpid.group_header_key':'GROUP_KEY',"
                              " 'qpid.shared_msg_group':1}}}}")
        </programlisting>
      </example>
      <section role="h3" id="usingmessagegroups-DefaultGroup">
        <title>
          Default Group
        </title>
        <para>
          Should a message without a group identifier arrive at a queue configured for message grouping, the broker assigns the message to the default group.  Therefore, all such "unidentified" messages are considered by the broker as part of the same group.  The name of the default group is <command>"qpid.no-group"</command>.  This default can be overridden by suppling a different value to the broker configuration item <command>"default-message-group"</command>:
          <example>
            <title>Overriding the default message group identifier for the broker</title>
            <programlisting>
qpidd --default-msg-group "EMPTY-GROUP"
            </programlisting>
          </example>
        </para>
      </section>
    </section>
  </section>