QPID-3858: Updated branch - merged from trunk r.1368650

git-svn-id: https://svn.apache.org/repos/asf/qpid/branches/asyncstore@1368910 13f79535-47bb-0310-9956-ffa450edef68

26 files changed, 1806 insertions, 423 deletions

diff --git a/doc/book/src/cpp-broker/.gitignore b/doc/book/src/cpp-broker/.gitignore
new file mode 100644
index 0000000000..1f57b975cd
--- /dev/null
+++ b/doc/book/src/cpp-broker/.gitignore
@@ -0,0 +1 @@
+/output
diff --git a/doc/book/src/cpp-broker/Active-Passive-Cluster.xml b/doc/book/src/cpp-broker/Active-Passive-Cluster.xml
index 5f5823bdd2..805ceb06e0 100644
--- a/doc/book/src/cpp-broker/Active-Passive-Cluster.xml
+++ b/doc/book/src/cpp-broker/Active-Passive-Cluster.xml
@@ -22,42 +22,124 @@ under the License.
 
 <section id="chap-Messaging_User_Guide-Active_Passive_Cluster">
 
-  <title>Active-passive Messaging Clusters (Preview)</title>
+  <title>Active-passive Messaging Clusters</title>
 
   <section>
     <title>Overview</title>
     <para>
-      This release provides a preview of a new module for High Availability (HA). The new module is
-      not yet complete or ready for production use. It being made available so that users can
-      experiment with the new approach and provide feedback early in the development process.
-      Feedback should go to <ulink url="mailto:user@qpid.apache.org">dev@qpid.apache.org</ulink>.
+
+      The High Availability (HA) module provides
+      <firstterm>active-passive</firstterm>, <firstterm>hot-standby</firstterm>
+      messaging clusters to provide fault tolerant message delivery.
     </para>
     <para>
-      The old cluster module takes an <firstterm>active-active</firstterm> approach, i.e. all the
-      brokers in a cluster are able to handle client requests simultaneously. The new HA module
-      takes an <firstterm>active-passive</firstterm>, <firstterm>hot-standby</firstterm> approach.
+      In an active-passive cluster only one broker, known as the
+      <firstterm>primary</firstterm>, is active and serving clients at a time. The other
+      brokers are standing by as <firstterm>backups</firstterm>. Changes on the primary
+      are replicated to all the backups so they are always up-to-date or "hot". Backup
+      brokers reject client connection attempts, to enforce the requirement that clients
+      only connect to the primary.
     </para>
     <para>
-      In an active-passive cluster only one broker, known as the <firstterm>primary</firstterm>, is
-      active and serving clients at a time. The other brokers are standing by as
-      <firstterm>backups</firstterm>. Changes on the primary are immediately replicated to all the
-      backups so they are always up-to-date or "hot".  If the primary fails, one of the backups is
-      promoted to take over as the new primary. Clients fail-over to the new primary
-      automatically. If there are multiple backups, the backups also fail-over to become backups of
-      the new primary.  Backup brokers reject connection attempts, to enforce the requirement that
-      only the primary be active.
+      If the primary fails, one of the backups is promoted to take over as the new
+      primary. Clients fail-over to the new primary automatically. If there are multiple
+      backups, the other backups also fail-over to become backups of the new primary.
     </para>
     <para>
-      This approach depends on an external <firstterm>cluster resource manager</firstterm> to detect
-      failures and choose the primary. <ulink
+      This approach relies on an external <firstterm>cluster resource manager</firstterm>
+      to detect failures, choose the new primary and handle network partitions. <ulink
       url="https://fedorahosted.org/cluster/wiki/RGManager">Rgmanager</ulink> is supported
       initially, but others may be supported in the future.
     </para>
     <section>
-      <title>Why the new approach?</title>
+      <title>Avoiding message loss</title>
+      <para>
+	In order to avoid message loss, the primary broker <emphasis>delays
+	acknowledgment</emphasis> of messages received from clients until the
+	message has been replicated to and acknowledged by all of the back-up
+	brokers.
+      </para>
+      <para>
+	Clients buffer unacknowledged messages and re-send them in the event of
+	a fail-over.
+	<footnote>
+	  <para>
+	  Clients must use "at-least-once" reliability to enable re-send of unacknowledged
+	  messages. This is the default behavior, no options need be set to enable it. For
+	  details of client addressing options see &#34;Using the Qpid Messaging API&#34;
+	  in <citetitle>Programming in Apache Qpid</citetitle>
+	  </para>
+	</footnote>
+	If the primary crashes before a message is replicated to
+	all the backups, the client will re-send the message when it fails over
+	to the new primary.
+      </para>
+      <para>
+	Note that this means it is possible for messages to be
+	<emphasis>duplicated</emphasis>. In the event of a failure it is
+	possible for a message to be both received by the backup that becomes
+	the new primary <emphasis>and</emphasis> re-sent by the client.
+      </para>
+      <para>
+	When a new primary is promoted after a fail-over it is initially in
+	"recovering" mode. In this mode, it delays acknowledgment of messages
+	on behalf of all the backups that were connected to the previous
+	primary. This protects those messages against a failure of the new
+	primary until the backups have a chance to connect and catch up.
+      </para>
+      <variablelist>
+	<title>Status of a HA broker</title>
+	<varlistentry>
+	  <term>Joining</term>
+	  <listitem>
+	    <para>
+	      Initial status of a new broker that has not yet connected to the primary.
+	    </para>
+	  </listitem>
+	</varlistentry>
+	<varlistentry>
+	  <term>Catch-up</term>
+	  <listitem>
+	    <para>
+	      A backup broker that is connected to the primary and catching up
+	      on queues and messages.
+	    </para>
+	  </listitem>
+	</varlistentry>
+	<varlistentry>
+	  <term>Ready</term>
+	  <listitem>
+	    <para>
+	      A backup broker that is fully caught-up and ready to take over as
+	      primary.
+	    </para>
+	  </listitem>
+	</varlistentry>
+	<varlistentry>
+	  <term>Recovering</term>
+	  <listitem>
+	    <para>
+	      The newly-promoted primary, waiting for backups to connect and catch up.
+	    </para>
+	  </listitem>
+	</varlistentry>
+	<varlistentry>
+	  <term>Active</term>
+	  <listitem>
+	    <para>
+	      The active primary broker with all backups connected and caught-up.
+	    </para>
+	  </listitem>
+	</varlistentry>
+      </variablelist>
+    </section>
+    <section>
+      <title>Replacing the old cluster module</title>
       <para>
-	The new active-passive approach has several advantages compared to the
-	existing active-active cluster module.
+	The High Availability (HA) module replaces the previous
+	<firstterm>active-active</firstterm> cluster module.  The new active-passive
+	approach has several advantages compared to the existing active-active cluster
+	module.
 	<itemizedlist>
 	  <listitem>
 	    It does not depend directly on openais or corosync. It does not use multicast
@@ -77,69 +159,42 @@ under the License.
 	    virtual IP addresses.
 	  </listitem>
 	  <listitem>
-	    Improved performance and scalability due to better use of multiple CPU s
+	    Improved performance and scalability due to better use of multiple CPUs
 	  </listitem>
 	</itemizedlist>
       </para>
     </section>
     <section>
       <title>Limitations</title>
-
-      <para>
-	There are a number of known limitations in the current preview implementation. These
-	will be fixed in the production versions.
-      </para>
-
       <itemizedlist>
 	<listitem>
-	  Transactional changes to queue state are not replicated atomically. If the primary crashes
-	  during a transaction, it is possible that the backup could contain only part of the
-	  changes introduced by a transaction.
-	</listitem>
-	<listitem>
-	  During a fail-over one backup is promoted to primary and any other backups switch to
-	  the new primary. Messages sent to the new primary before all the backups have
-	  switched could be lost if the new primary itself fails before all the backups have
-	  switched.
-	</listitem>
-	<listitem>
-	  Acknowledgments are confirmed to clients before the message has been dequeued
-	  from replicas or indeed from the local store if that is asynchronous.
-	</listitem>
-	<listitem>
-	  When used with a persistent store: if the entire cluster fails, there are no tools to help
-	  identify the most recent store.
+	  Transactional changes to queue state are not replicated atomically. If the
+	  primary crashes during a transaction, it is possible that the backup could
+	  contain only part of the changes introduced by a transaction.
 	</listitem>
 	<listitem>
-	  A persistent broker must have its store erased before joining an existing cluster.
-	  In the production version a persistent broker will be able to load its store and
-	  avoid downloading messages that are in the store from the primary.
+	  Not yet integrated with the persistent store.  A persistent broker must have its
+	  store erased before joining an existing cluster.  If the entire cluster fails,
+	  there are no tools to help identify the most recent store. In the future a
+	  persistent broker will be able to use its stored messages to avoid downloading
+	  messages from the primary when joining a cluster.
 	</listitem>
 	<listitem>
 	  Configuration changes (creating or deleting queues, exchanges and bindings) are
-	  replicated asynchronously. Management tools used to make changes will consider the
-	  change complete when it is complete on the primary, it may not yet be replicated
-	  to all the backups.
-	</listitem>
-	<listitem>
-	  Deletions made immediately after a failure (before all the backups are ready) may
-	  be lost on a backup. Queues, exchange or bindings that were deleted on the primary could
-	  re-appear if that backup is promoted to primary on a subsequent failure.
-	</listitem>
-	<listitem>
-	  Better control is needed over which queues/exchanges are replicated and which are not.
-	</listitem>
-	<listitem>
-	  There are some known issues affecting performance, both the throughput of
-	  replication and the time taken for backups to fail-over. Performance will improve
-	  in the production version.
+	  replicated asynchronously. Management tools used to make changes will consider
+	  the change complete when it is complete on the primary, it may not yet be
+	  replicated to all the backups.
 	</listitem>
 	<listitem>
-	  Federated links from the primary will be lost in fail over, they will not be
-	  re-connected on the new primary. Federation links to the primary can fail over.
+	  Deletions made immediately after a failure (before all the backups are ready)
+	  may be lost on a backup. Queues, exchange or bindings that were deleted on the
+	  primary could re-appear if that backup is promoted to primary on a subsequent
+	  failure.
 	</listitem>
 	<listitem>
-	  Only plain FIFO queues can be replicated. LVQ and ring queues are not yet supported.
+	  Federated links <emphasis>from</emphasis> the primary will be lost in fail over,
+	  they will not be re-connected to the new primary. Federation links
+	  <emphasis>to</emphasis> the primary can fail over.
 	</listitem>
       </itemizedlist>
     </section>
@@ -162,7 +217,7 @@ under the License.
       virtual IP addresses for clients or brokers.
     </para>
   </section>
-  
+
   <section>
     <title>Configuring the Brokers</title>
     <para>
@@ -170,10 +225,10 @@ under the License.
       default. The following broker options are available for the HA module.
     </para>
     <table frame="all" id="ha-broker-options">
-      <title>Options for High Availability Messaging Cluster</title>
+      <title>Broker Options for High Availability Messaging Cluster</title>
       <tgroup align="left" cols="2" colsep="1" rowsep="1">
-	<colspec colname="c1" colwidth="1*"/>
-	<colspec colname="c2" colwidth="3*"/>
+	<colspec colname="c1"/>
+	<colspec colname="c2"/>
 	<thead>
 	  <row>
 	    <entry align="center" nameend="c2" namest="c1">
@@ -184,7 +239,7 @@ under the License.
 	<tbody>
 	  <row>
 	    <entry>
-	      <literal>--ha-cluster <replaceable>yes|no</replaceable></literal>
+	      <literal>ha-cluster <replaceable>yes|no</replaceable></literal>
 	    </entry>
 	    <entry>
 	      Set to "yes" to have the broker join a cluster.
@@ -192,7 +247,7 @@ under the License.
 	  </row>
 	  <row>
 	    <entry>
-	      <literal>--ha-brokers <replaceable>URL</replaceable></literal>
+	      <literal>ha-brokers-url <replaceable>URL</replaceable></literal>
 	    </entry>
 	    <entry>
 	      <para>
@@ -201,54 +256,78 @@ under the License.
 		  <para>
 		  The full format of the URL is given by this grammar:
 		  <programlisting>
-		    url = ["amqp:"][ user ["/" password] "@" ] addr ("," addr)*
-		    addr = tcp_addr / rmda_addr / ssl_addr / ...
-		    tcp_addr = ["tcp:"] host [":" port]
-		    rdma_addr = "rdma:" host [":" port]
-		    ssl_addr = "ssl:" host [":" port]'
+url = ["amqp:"][ user ["/" password] "@" ] addr ("," addr)*
+addr = tcp_addr / rmda_addr / ssl_addr / ...
+tcp_addr = ["tcp:"] host [":" port]
+rdma_addr = "rdma:" host [":" port]
+ssl_addr = "ssl:" host [":" port]'
 		  </programlisting>
 		  </para>
 		</footnote>
-		used by brokers to connect to each other. If you use a virtual IP address
-		then this is a single address, for example
-		<literal>amqp:20.0.20.200</literal>. If you do not use a virtual IP
-		address then the URL must list all the addresses of brokers in the
-		cluster, for example <literal>amqp:node1,node2,node3</literal>
+		used by cluster brokers to connect to each other. The URL can
+		contain a list of all the broker addresses or it can contain a single
+		virtual IP address.  If a list is used it is comma separated, for example
+		<literal>amqp:node1.exaple.com,node2.exaple.com,node3.exaple.com</literal>
 	      </para>
 	    </entry>
 	  </row>
 	  <row>
-	    <entry><literal>--ha-public-brokers <replaceable>URL</replaceable></literal> </entry>
+	    <entry><literal>ha-public-url <replaceable>URL</replaceable></literal> </entry>
 	    <entry>
 	      <para>
-		The URL used by clients to connect to the brokers. This has the same
-		format as the <literal>--ha-brokers</literal> URL above.
+		The URL that is advertised to clients. This defaults to the
+		<literal>ha-brokers-url</literal> URL above, and has the same format.  A
+		virtual IP address is recommended for the public URL as it simplifies
+		deployment and hides changes to the cluster membership from clients.
 	      </para>
 	      <para>
-		If this option is not set, clients will use the same URL as brokers.
 		This option allows you to put client traffic on a different network from
 		broker traffic, which is recommended.
 	      </para>
 	    </entry>
 	  </row>
 	  <row>
-	    <entry><literal>--ha-replicate</literal></entry>
+	    <entry><literal>ha-replicate </literal><replaceable>VALUE</replaceable></entry>
 	    <foo/>
 	    <entry>
-	      <para> 
+	      <para>
 		Specifies whether queues and exchanges are replicated by default.
-		For details see <xref linkend="creating-replicated"/>
+		<replaceable>VALUE</replaceable> is one of: <literal>none</literal>,
+		<literal>configuration</literal>, <literal>all</literal>.
+		For details see <xref linkend="ha-creating-replicated"/>.
 	      </para>
 	    </entry>
 	  </row>
 	  <row>
 	    <entry>
-	      <para><literal>--ha-username <replaceable>USER</replaceable></literal></para>
-	      <para><literal>--ha-password <replaceable>PASS</replaceable></literal></para>
-	      <para><literal>--ha-mechanism <replaceable>MECH</replaceable></literal></para>
+	      <para><literal>ha-username <replaceable>USER</replaceable></literal></para>
+	      <para><literal>ha-password <replaceable>PASS</replaceable></literal></para>
+	      <para><literal>ha-mechanism <replaceable>MECH</replaceable></literal></para>
+	    </entry>
+	    <entry>
+	      Authentication settings used by HA brokers to connect to each other.
+	      If you are using authorization
+	      (<xref linkend="sect-Messaging_User_Guide-Security-Authorization"/>)
+	      then this user must have all permissions.
+	    </entry>
+	  </row>
+	  <row>
+	    <entry><literal>ha-backup-timeout <replaceable>SECONDS</replaceable></literal> </entry>
+	    <entry>
+	      <para>
+		Maximum time that a recovering primary will wait for an expected
+		backup to connect and become ready.
+	      </para>
 	    </entry>
+	  </row>
+	  <row>
+	    <entry><literal>link-maintenance-interval <replaceable>SECONDS</replaceable></literal></entry>
 	    <entry>
-	      Authentication settings used by brokers to connect to each other.
+	      <para>
+		Interval for the broker to check link health and re-connect links if need
+		be. If you want brokers to fail over quickly you can set this to a
+		fraction of a second, for example: 0.1.
+	      </para>
 	    </entry>
 	  </row>
 	</tbody>
@@ -256,7 +335,7 @@ under the License.
     </table>
     <para>
       To configure a HA cluster you must set at least <literal>ha-cluster</literal> and
-      <literal>ha-brokers</literal>.
+      <literal>ha-brokers-url</literal>.
     </para>
   </section>
 
@@ -270,9 +349,9 @@ under the License.
     </para>
     <para>
       The resource manager is responsible for starting the <command>qpidd</command> broker
-      on each node in the cluster. The resource manager <firstterm>promotes</firstterm>
+      on each node in the cluster. The resource manager then <firstterm>promotes</firstterm>
       one of the brokers to be the primary. The other brokers connect to the primary as
-      backups, using the URL provided in the <literal>ha-brokers</literal> configuration
+      backups, using the URL provided in the <literal>ha-brokers-url</literal> configuration
       option.
     </para>
     <para>
@@ -303,7 +382,7 @@ under the License.
       clustered services using <command>cman</command> and
       <command>rgmanager</command>. It will show you how to configure an active-passive,
       hot-standby <command>qpidd</command> HA cluster with <command>rgmanager</command>.
-    </para>
+    </para> 
     <para>
       You must provide a <literal>cluster.conf</literal> file to configure
       <command>cman</command> and <command>rgmanager</command>.  Here is
@@ -398,7 +477,7 @@ NOTE: fencing is not shown, you must configure fencing appropriately for your cl
     <para>
       The <literal>resources</literal> section also defines a pair of virtual IP
       addresses on different sub-nets. One will be used for broker-to-broker
-      communication, the other for client-to-broker. 
+      communication, the other for client-to-broker.
     </para>
     <para>
       To take advantage of the virtual IP addresses, <filename>qpidd.conf</filename>
@@ -406,8 +485,8 @@ NOTE: fencing is not shown, you must configure fencing appropriately for your cl
     </para>
     <programlisting>
       ha-cluster=yes
-      ha-brokers=20.0.20.200
-      ha-public-brokers=20.0.10.200
+      ha-brokers-url=20.0.20.200
+      ha-public-url=20.0.10.200
     </programlisting>
     <para>
       This configuration specifies that backup brokers will use 20.0.20.200
@@ -436,21 +515,22 @@ NOTE: fencing is not shown, you must configure fencing appropriately for your cl
   <section>
     <title>Broker Administration Tools</title>
     <para>
-      Normally, clients are not allowed to connect to a backup broker. However management tools are
-      allowed to connect to a backup brokers. If you use these tools you <emphasis>must
-      not</emphasis> add or remove messages from replicated queues, or delete replicated queues or
-      exchanges as this will corrupt the replication process and may cause message loss.
+      Normally, clients are not allowed to connect to a backup broker. However
+      management tools are allowed to connect to a backup brokers. If you use
+      these tools you <emphasis>must not</emphasis> add or remove messages from
+      replicated queues, nor create or delete replicated queues or exchanges as
+      this will disrupt the replication process and may cause message loss.
     </para>
     <para>
       <command>qpid-ha</command> allows you to view and change HA configuration settings.
     </para>
     <para>
       The tools <command>qpid-config</command>, <command>qpid-route</command> and
-      <command>qpid-stat</command> will connect to a backup if you pass the flag <command>--ha-admin</command> on the
+      <command>qpid-stat</command> will connect to a backup if you pass the flag <command>ha-admin</command> on the
       command line.
     </para>
   </section>
-  
+
   <section id="ha-creating-replicated">
     <title>Creating replicated queues and exchanges</title>
     <para>
@@ -459,7 +539,8 @@ NOTE: fencing is not shown, you must configure fencing appropriately for your cl
       option. It has one of the following values:
       <itemizedlist>
 	<listitem>
-	  <firstterm>all</firstterm>: Replicate everything automatically: queues, exchanges, bindings and messages.
+	  <firstterm>all</firstterm>: Replicate everything automatically: queues,
+	  exchanges, bindings and messages.
 	</listitem>
 	<listitem>
 	  <firstterm>configuration</firstterm>: Replicate the existence of queues,
@@ -541,10 +622,12 @@ NOTE: fencing is not shown, you must configure fencing appropriately for your cl
       membership.
     </para>
     <para>
-      Suppose your cluster has 3 nodes: <literal>node1</literal>, <literal>node2</literal>
-      and <literal>node3</literal> all using the default AMQP port. To connect a client you
-      need to specify the address(es) and set the <literal>reconnect</literal> property to
-      <literal>true</literal>. Here's how to connect each type of client:
+      Suppose your cluster has 3 nodes: <literal>node1</literal>,
+      <literal>node2</literal> and <literal>node3</literal> all using the
+      default AMQP port, and you are not using a virtual IP address. To connect
+      a client you need to specify the address(es) and set the
+      <literal>reconnect</literal> property to <literal>true</literal>. The
+      following sub-sections show how to connect each type of client.
     </para>
     <section>
       <title>C++ clients</title>
@@ -655,6 +738,89 @@ NOTE: fencing is not shown, you must configure fencing appropriately for your cl
       </screen>
     </section>
   </section>
+
+  <section>
+    <title>Security.</title>
+    <para>
+      You can secure your cluster using the authentication and authorization features
+      described in <xref linkend="chap-Messaging_User_Guide-Security"/>.
+    </para>
+    <para>
+      Backup brokers connect to the primary broker and subscribe for management
+      events and queue contents. You can specify the identity used to connect
+      to the primary with the following options:
+    </para>
+    <table frame="all" id="ha-broker-security-options">
+      <title>Security options for High Availability Messaging Cluster</title>
+      <tgroup align="left" cols="2" colsep="1" rowsep="1">
+	<colspec colname="c1" colwidth="1*"/>
+	<colspec colname="c2" colwidth="3*"/>
+	<thead>
+	  <row>
+	    <entry align="center" nameend="c2" namest="c1">
+	      Security options for High Availability Messaging Cluster
+	    </entry>
+	  </row>
+	</thead>
+	<tbody>
+	  <row>
+	    <entry>
+	      <para><literal>ha-username <replaceable>USER</replaceable></literal></para>
+	      <para><literal>ha-password <replaceable>PASS</replaceable></literal></para>
+	      <para><literal>ha-mechanism <replaceable>MECH</replaceable></literal></para>
+	    </entry>
+	    <entry>
+	      Authentication settings used by HA brokers to connect to each other.
+	      If you are using authorization
+	      (<xref linkend="sect-Messaging_User_Guide-Security-Authorization"/>)
+	      then this user must have all permissions.
+	    </entry>
+	  </row>
+	</tbody>
+      </tgroup>
+    </table>
+    <para>
+      This identity is also used to authorize actions taken on the backup broker to replicate
+      from the primary, for example to create queues or exchanges.
+    </para>
+  </section>
+
+  <section>
+    <title>Integrating with other Cluster Resource Managers</title>
+    <para>
+      To integrate with a different resource manager you must configure it to:
+      <itemizedlist>
+	<listitem>Start a qpidd process on each node of the cluster.</listitem>
+	<listitem>Restart qpidd if it crashes.</listitem>
+	<listitem>Promote exactly one of the brokers to primary.</listitem>
+	<listitem>Detect a failure and promote a new primary.</listitem>
+      </itemizedlist>
+    </para>
+    <para>
+      The <command>qpid-ha</command> command allows you to check if a broker is primary,
+      and to promote a backup to primary.
+    </para>
+    <para>
+      To test if a broker is the primary:
+      <programlisting>
+	qpid-ha -b <replaceable>broker-address</replaceable> status --expect=primary
+      </programlisting>
+      This command will return 0 if the broker at <replaceable>broker-address</replaceable>
+      is the primary, non-0 otherwise.
+    </para>
+    <para>
+      To promote a broker to primary:
+      <programlisting>
+	qpid-ha -b <replaceable>broker-address</replaceable> promote
+      </programlisting>
+    </para>
+    <para>
+      <command>qpid-ha --help</command> gives information on other commands and options available.
+      You can also use <command>qpid-ha</command> to manually examine and promote brokers. This
+      can be useful for testing failover scenarios without having to set up a full resource manager,
+      or to simulate a cluster on a single node. For deployment, a resource manager is required.
+    </para>
+  </section>
 </section>
 
 <!-- LocalWords:  scalability rgmanager multicast RGManager mailto LVQ qpidd IP dequeued Transactional username
diff --git a/doc/book/src/cpp-broker/Cheat-Sheet-for-configuring-Queue-Options.xml b/doc/book/src/cpp-broker/Cheat-Sheet-for-configuring-Queue-Options.xml
index d50948e0cc..78272b7ef4 100644
--- a/doc/book/src/cpp-broker/Cheat-Sheet-for-configuring-Queue-Options.xml
+++ b/doc/book/src/cpp-broker/Cheat-Sheet-for-configuring-Queue-Options.xml
@@ -176,7 +176,7 @@
             ignored.
             </para></listitem>
           </itemizedlist><para>
-            A fully worked <xref linkend="LVQ-Examplesource"/> can be found here
+            A fully worked <xref linkend="LVQ-Example"/> can be found here
           </para>
 	  <!--h3--></section>
 	  <section role="h3" id="CheatSheetforconfiguringQueueOptions-Settingadditionalbehaviors"><title>
diff --git a/doc/book/src/java-broker/AMQP-Messaging-Broker-Java-Book.xml b/doc/book/src/java-broker/AMQP-Messaging-Broker-Java-Book.xml
index 0bb40052cd..54c2984d0a 100644
--- a/doc/book/src/java-broker/AMQP-Messaging-Broker-Java-Book.xml
+++ b/doc/book/src/java-broker/AMQP-Messaging-Broker-Java-Book.xml
@@ -1,6 +1,6 @@
 <?xml version="1.0"?>
 <!--
- 
+
  Licensed to the Apache Software Foundation (ASF) under one
  or more contributor license agreements.  See the NOTICE file
  distributed with this work for additional information
@@ -31,7 +31,7 @@
       <listitem><para>Implemented in Java - Fully JMS compliant, runs on any Java platform.</para></listitem>
     </itemizedlist>
 
-    <para>Both AMQP messaging brokers support clients in multiple languages, as long as the messaging client and the messaging broker use the same version of AMQP. See <link linkend="AMQP-Compatibility"/> to see which messaging clients work with each broker.</para>
+    <para>Both AMQP messaging brokers support clients in multiple languages, as long as the messaging client and the messaging broker use the same version of AMQP.</para>
 
     <para>This manual contains information specific to the broker that is implemented in Java.</para>
   </preface>
@@ -44,6 +44,7 @@
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Java-Environment-Variables.xml"/>
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid-Troubleshooting-Guide.xml"/>
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Broker-Configuration-Guide.xml"/>
+        <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="HA-Guide.xml"/>
 </chapter>
 
 <chapter  id="Qpid-Java-Broker-HowTos">
@@ -58,8 +59,8 @@
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Debug-using-log4j.xml"/>
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="How-to-Tune-M3-Java-Broker-Performance.xml"/>
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid-Java-Build-How-To.xml"/>
-        <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Use-Priority-Queues.xml"/>
-        <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="How-to-Use-SlowConsumerDisconnect.xml"/> 
+        <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="OtherQueueTypes.xml"/>
+        <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="How-to-Use-SlowConsumerDisconnect.xml"/>
 </chapter>
 
 
@@ -67,7 +68,6 @@
 
 <chapter id="QpidJavaBroker-ManagementTools">
 <title>Management Tools</title>
-        <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="MessageStore-Tool.xml"/>
         <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Qpid-Java-Broker-Management-CLI.xml"/>
 </chapter>
 </book>
diff --git a/doc/book/src/java-broker/Broker-Configuration-Guide.xml b/doc/book/src/java-broker/Broker-Configuration-Guide.xml
index 28a2e9f2d3..558d17c63c 100644
--- a/doc/book/src/java-broker/Broker-Configuration-Guide.xml
+++ b/doc/book/src/java-broker/Broker-Configuration-Guide.xml
@@ -23,6 +23,6 @@
 <section id="Java-Broker-Configuration-Guide">
     <title>Broker Configuration Guide </title>
 
-    <xi:include xmlns:xi="http://www.w3.org/2001/XInclude"
-            href="Topic-Configuration.xml"/>
+    <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Producer-Flow-Control.xml"/>
+    <xi:include xmlns:xi="http://www.w3.org/2001/XInclude" href="Topic-Configuration.xml"/>
 </section>
diff --git a/doc/book/src/java-broker/HA-Guide.xml b/doc/book/src/java-broker/HA-Guide.xml
new file mode 100644
index 0000000000..429f2f76c2
--- /dev/null
+++ b/doc/book/src/java-broker/HA-Guide.xml
@@ -0,0 +1,990 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE urls [
+<!ENTITY oracleBdbProductOverviewUrl "http://www.oracle.com/technetwork/products/berkeleydb/overview/index-093405.html">
+<!ENTITY oracleBdbProductVersion "5.0.48">
+<!ENTITY oracleBdbRepGuideUrl "http://oracle.com/cd/E17277_02/html/ReplicationGuide/">
+<!ENTITY oracleBdbJavaDocUrl "http://docs.oracle.com/cd/E17277_02/html/java/">
+<!ENTITY oracleJdkDocUrl "http://oracle.com/javase/6/docs/api/">
+]>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+<section>
+  <title>High Availability</title>
+  <section role="h3" id="HAGeneralIntroduction">
+    <title>General Introduction</title>
+    <para>The term High Availability (HA) usually refers to having a number of instances of a service such as a Message Broker
+      available so that should a service unexpectedly fail, or requires to be shutdown for maintenance, users may quickly connect
+      to another instance and continue their work with minimal interuption. HA is one way to make a overall system more resilient
+      by eliminating a single point of failure from a system.</para>
+    <para>HA offerings are usually categorised as <emphasis role="bold">Active/Active</emphasis> or <emphasis role="bold">Active/Passive</emphasis>.
+      An Active/Active system is one where all nodes within the cluster are usuaully available for use by clients all of the time.  In an
+      Active/Passive system, one only node within the cluster is available for use by clients at any one time, whilst the others are in
+      some kind of standby state, awaiting to quickly step-in in the event the active node becomes unavailable.
+    </para>
+  </section>
+  <section role="h3" id="HAOfferingsOfJavaBroker">
+    <title>HA offerings of the Java Broker</title>
+    <para>The Java Broker's HA offering became available at release <emphasis role="bold">0.18</emphasis>.  HA is provided by way of the HA
+      features built into the <ulink url="&oracleBdbProductOverviewUrl;">Java Edition of the Berkley Database (BDB JE)</ulink> and as such
+      is currently only available to Java Broker users who use the optional BDB JE based persistence store. This
+      <emphasis role="bold">optional</emphasis> store requires the use of BDB JE which is licensed under the Sleepycat Licence, which is
+      not compatible with the Apache Licence and thus BDB JE is not distributed with Qpid. Users who elect to use this optional store for
+      the broker have to provide this dependency.</para>
+    <para>HA in the Java Broker provides an <emphasis role="bold">Active/Passive</emphasis> mode of operation with Virtual hosts being
+      the unit of replication.  The Active node (referred to as the <emphasis role="bold">Master</emphasis>) accepts all work from all the clients.
+       The Passive nodes (referred to as <emphasis role="bold">Replicas</emphasis>) are unavailable for work: the only task they must perform is
+       to remain in synch with the Master node by consuming a replication stream containing all data and state.</para>
+    <para>If the Master node fails, a Replica node is elected to become the new Master node.  All clients automatically failover
+      <footnote><para>The automatic failover feature is available only for AMQP connections from the Java client.  Management connections (JMX)
+        do not current offer this feature.</para></footnote> to the new Master and continue their work.</para>
+    <para>The Java Broker HA solution is incompatible with the HA solution offered by the CPP Broker.  It is not possible to co-locate Java and CPP
+       Brokers within the same cluster.</para>
+    <para>HA is not currently available for those using the the <emphasis role="bold">Derby Store</emphasis> or <emphasis role="bold">Memory
+      Message Store</emphasis>.</para>
+  </section>
+  <section role="h3" id="HATwoNodeCluster">
+    <title>Two Node Cluster</title>
+    <section role="h4">
+      <title>Overview</title>
+      <para>In this HA solution, a cluster is formed with two nodes. one node serves as
+        <emphasis role="bold">master</emphasis> and the other is a <emphasis role="bold">replica</emphasis>.
+      </para>
+      <para>All data and state required for the operation of the virtual host is automatically sent from the
+        master to the replica. This is called the replication stream. The master virtual host confirms each
+        message is on the replica before the client transaction completes. The exact way the client awaits
+        for the master and replica is gorverned by the <link linkend="HADurabilityGuarantee">durability</link>
+        configuration, which is discussed later. In this way, the replica remains ready to take over the
+        role of the master if the master becomes unavailable.
+      </para>
+      <para>It is important to note that there is an inherent limitation of two node clusters is that
+        the replica node cannot make itself master automatically in the event of master failure.  This
+        is because the replica has no way to distinguish between a network partition (with potentially
+        the master still alive on the other side of the partition) and the case of genuine master failure.
+        (If the replica were to elect itself as master, the cluster would run the risk of a
+        <ulink url="http://en.wikipedia.org/wiki/Split-brain_(computing)">split-brain</ulink> scenario).
+        In the event of a master failure, a third party must designate the replica as primary.  This process
+        is described in more detail later.
+      </para>
+      <para>Clients connect to the cluster using a <link linkend="HAClientFailover">failover url</link>.
+        This allows the client to maintain a connection to the master in a way that is transparent
+        to the client application.</para>
+    </section>
+    <section role="h4">
+      <title>Depictions of cluster operation</title>
+      <para>In this section, the operation of the cluster is depicted through a series of figures
+        supported by explanatory text.</para>
+      <figure>
+        <title>Key for figures</title>
+        <mediaobject>
+          <imageobject>
+            <imagedata fileref="images/HA-2N-Key.png" format="PNG" scalefit="1"/>
+          </imageobject>
+          <textobject>
+            <phrase>Key to figures</phrase>
+          </textobject>
+        </mediaobject>
+      </figure>
+      <section role="h5" id="HATwoNodeNormalOperation">
+        <title>Normal Operation</title>
+        <para>The figure below illustrates normal operation.  Clients connecting to the cluster by way
+	  of the failover URL achieve a connection to the master. As clients perform work (message
+	  production, consumption, queue creation etc), the master additionally sends this data to the
+	  replica over the network.</para>
+        <figure>
+          <title>Normal operation of a two-node cluster</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-Normal.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Normal operation</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+      <section role="h5" id="HATwoNodeMasterFailure">
+        <title>Master Failure and Recovery</title>
+        <para>The figure below illustrates a sequence of events whereby the master suffers a failure
+	  and the replica is made the master to allow the clients to continue to work. Later the
+	  old master is repaired and comes back on-line in replica role.</para>
+        <para>The item numbers in this list apply to the numbered boxes in the figure below.</para>
+        <orderedlist>
+          <listitem>
+            <para>System operating normally</para>
+          </listitem>
+          <listitem>
+            <para>Master suffers a failure and disconnects all clients. Replica realises that it is no
+	      longer in contact with master. Clients begin to try to reconnect to the cluster, although these
+	      connection attempts will fail at this point.</para>
+          </listitem>
+          <listitem>
+            <para>A third-party (an operator, a script or a combination of the two) verifies that the master has truely
+           failed <emphasis role="bold">and is no longer running</emphasis>. If it has truely failed, the decision is made
+           to designate the replica as primary, allowing it to assume the role of master despite the other node being down.
+           This primary designation is performed using <link linkend="HAJMXAPI">JMX</link>.</para>
+          </listitem>
+          <listitem>
+            <para>Client connections to the new master succeed and the <emphasis role="bold">service is restored
+	      </emphasis>, albeit without a replica.</para>
+          </listitem>
+          <listitem>
+            <para>The old master is repaired and brought back on-line.  It automatically rejoins the cluster
+	       in the <emphasis role="bold">replica</emphasis> role.</para>
+          </listitem>
+        </orderedlist>
+        <figure>
+          <title>Failure of master and recovery sequence</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-MasterFail.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Failure of master and subsequent recovery sequence</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+      <section role="h5" id="HATwoNodeReplicaFailure">
+        <title>Replica Failure and Recovery</title>
+        <para>The figure that follows illustrates a sequence of events whereby the replica suffers a failure
+	   leaving the master to continue processing alone.  Later the replica is repaired and is restarted.
+	   It rejoins the cluster so that it is once again ready to take over in the event of master failure.</para>
+        <para>The behavior of the replica failure case is governed by the <varname>designatedPrimary</varname>
+        configuration item. If set true on the master, the master will continue to operate solo without outside
+        intervention when the replica fails. If false, a third-party must designate the master as primary in order
+        for it to continue solo.</para>
+        <para>The item numbers in this list apply to the numbered boxes in the figure below. This example assumes
+	   that <varname>designatedPrimary</varname> is true on the original master node.</para>
+        <orderedlist>
+          <listitem>
+            <para>System operating normally</para>
+          </listitem>
+          <listitem>
+            <para>Replica suffers a failure. Master realises that replica longer in contact but as
+	      <varname>designatedPrimary</varname> is true, master continues processing solo and thus client
+	      connections are uninterrupted by the loss of the replica. System continues operating normally, albeit
+          with a single node.</para>
+          </listitem>
+          <listitem>
+            <para>Replica is repaired.</para>
+          </listitem>
+          <listitem>
+            <para>After catching up with missed work, replica is once again ready to take over in the event of master failure.</para>
+          </listitem>
+        </orderedlist>
+        <figure>
+          <title>Failure of replica and subsequent recovery sequence</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-ReplicaFail.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Failure of replica and subsequent recovery sequence</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+      <section role="h5" id="HATwoNodeNetworkPartition">
+        <title>Network Partition and Recovery</title>
+        <para>The figure below illustrates the sequence of events that would occur if the network between
+	  master and replica were to suffer a partition, and the nodes were out of contact with one and other.</para>
+        <para>As with <link linkend="HATwoNodeReplicaFailure">Replica Failure and Recovery</link>, the
+	  behaviour is governed by the <varname>designatedPrimary</varname>.
+	  Only if <varname>designatedPrimary</varname> is true on the master, will the master continue solo.</para>
+        <para>The item numbers in this list apply to the numbered boxes in the figure below. This example assumes
+	   that <varname>designatedPrimary</varname> is true on the original master node.</para>
+        <orderedlist>
+          <listitem>
+            <para>System operating normally</para>
+          </listitem>
+          <listitem>
+            <para>Network suffers a failure. Master realises that replica longer in contact but as
+	      <varname>designatedPrimary</varname> is true, master continues processing solo and thus client
+	      connections are uninterrupted by the network partition between master and replica.</para>
+          </listitem>
+          <listitem>
+            <para>Network is repaired.</para>
+          </listitem>
+          <listitem>
+            <para>After catching up with missed work, replica is once again ready to take over in the event of master failure.
+	    System operating normally again.</para>
+          </listitem>
+        </orderedlist>
+        <figure>
+          <title>Partition of the network separating master and replica</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-NetworkPartition.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Network Partition and Recovery</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+      <section role="h5" id="HATwoNodeSplitBrain">
+        <title>Split Brain</title>
+        <para>A <ulink url="http://en.wikipedia.org/wiki/Split-brain_(computing)">split-brain</ulink>
+          is a situation where the two node cluster has two masters. BDB normally strives to prevent
+	  this situation arising by preventing two nodes in a cluster being master at the same time.
+	  However, if the network suffers a partition, and the third-party intervenes incorrectly
+	  and makes the replica a second master a split-brain will be formed and both masters will
+	  proceed to perform work  <emphasis role="bold">independently</emphasis> of one and other.</para>
+        <para>There is no automatic recovery from a split-brain.</para>
+        <para>Manual intervention will be required to choose which store will be retained as master
+	  and which will be discarded.  Manual intervention will be required to identify and repeat the
+          lost business transactions.</para>
+        <para>The item numbers in this list apply to the numbered boxes in the figure below.</para>
+        <orderedlist>
+          <listitem>
+            <para>System operating normally</para>
+          </listitem>
+          <listitem>
+            <para>Network suffers a failure. Master realises that replica longer in contact but as
+	      <varname>designatedPrimary</varname> is true, master continues processing solo.  Client
+	      connections are uninterrupted by the network partition.</para>
+            <para>A third-party <emphasis role="bold">erroneously</emphasis> designates the replica as primary while the
+            original master continues running (now solo).</para>
+          </listitem>
+          <listitem>
+            <para>As the nodes cannot see one and other, both behave as masters. Clients may perform work against
+	      both master nodes.</para>
+          </listitem>
+        </orderedlist>
+        <figure>
+          <title>Split Brain</title>
+          <mediaobject>
+            <imageobject>
+              <imagedata fileref="images/HA-2N-SplitBrain.png" format="PNG" scalefit="1"/>
+            </imageobject>
+            <textobject>
+              <phrase>Split Brain</phrase>
+            </textobject>
+          </mediaobject>
+        </figure>
+      </section>
+    </section>
+  </section>
+  <section role="h3" id="HAMultiNodeCluster">
+    <title>Multi Node Cluster</title>
+    <para>Multi node clusters, that is clusters where the number of nodes is three or more, are not yet
+         ready for use.</para>
+  </section>
+  <section role="h3" id="HAConfiguration">
+    <title>Configuring a Virtual Host to be a node</title>
+    <para>To configure a virtualhost as a cluster node, configure the virtualhost.xml in the following manner:</para>
+    <para>
+      <programlisting language="xml"><![CDATA[
+<virtualhost>
+  <name>myhost</name>
+  <myvhost>
+    <store>
+      <class>org.apache.qpid.server.store.berkeleydb.BDBHAMessageStore</class>
+      <environment-path>${work}/bdbhastore</environment-path>
+      <highAvailability>
+        <groupName>myclustername</groupName>
+        <nodeName>mynode1</nodeName>
+        <nodeHostPort>node1host:port</nodeHostPort>
+        <helperHostPort>node1host:port</helperHostPort>
+        <durability>NO_SYNC\,NO_SYNC\,SIMPLE_MAJORITY</durability>
+        <coalescingSync>true|false</coalescingSync>
+        <designatedPrimary>true|false</designatedPrimary>
+      </highAvailability>
+    </store>
+    ...
+ </myvhost>
+</virtualhost>]]></programlisting>
+    </para>
+    <para>The <varname>groupName</varname> is the name of logical name of the cluster.  All nodes within the
+      cluster must use the same <varname>groupName</varname> in order to be considered part of the cluster.</para>
+    <para>The <varname>nodeName</varname> is the logical name of the node.  All nodes within the cluster must have a
+      unique name.  It is recommended that the node name should be chosen from a different nomenclature from that of
+      the servers on which they are hosted, in case the need arises to move node to a new server in the future.</para>
+    <para>The <varname>nodeHostPort</varname> is the hostname and port number used by this node to communicate with the
+      the other nodes in the cluster. For the hostname, an IP address, hostname or fully qualified hostname may be used.
+      For the port number, any free port can be used.  It is important that this address is stable over time, as BDB
+      records and uses this address internally.</para>
+    <para>The <varname>helperHostPort</varname> is the hostname and port number that new nodes use to discover other
+      nodes within the cluster when they are newly introduced to the cluster.  When configuring the first node, set the
+      <varname>helperHostPort</varname> to its own <varname>nodeHostPort</varname>.  For the second and subsequent nodes,
+      set their <varname>helperHostPort</varname> to that of the first node.</para>
+    <para><varname>durability</varname> controls the <link linkend="HADurabilityGuarantee">durability</link>
+      guarantees made by the cluster. It is important that all nodes use the same value for this property. The default value is
+      NO_SYNC\,NO_SYNC\,SIMPLE_MAJORITY. Owing to the internal use of Apache Commons Config, it is currently necessary
+      to escape the commas within the durability string.</para>
+    <para><varname>coalescingSync</varname> controls the <link linkend="HADurabilityGuarantee_CoalescingSync">coalescing-sync</link>
+      mode of Qpid. It is important that all nodes use the same value. If omitted, it defaults to true.</para>
+    <para>The <varname>designatedPrimary</varname> is applicable only to the <link linkend="HATwoNodeCluster">two-node
+     case.</link>  It governs the behaviour of a node when the other node fails or becomes uncontactable.  If true,
+     the node will be designated as primary at startup and will be able to continue operating as a single node master.
+     If false, the node will transition to an unavailable state until a third-party manually designates the node as
+     primary or the other node is restored. It is suggested that the node that normally fulfils the role of master is
+     set true in config file and the node that is normally replica is set false.  Be aware that setting both nodes to
+     true will lead to a failure to start up, as both cannot be designated at the point of contact. Designating both
+     nodes as primary at runtime (using the JMX interface) will lead to a <link linkend="HATwoNodeSplitBrain">split-brain</link>
+     in the case of network partition and must be avoided.</para>
+     <note><para>Usage of domain names in  <varname>helperHostPort</varname> and <varname>nodeHostPort</varname> is more preferebale
+     over IP addresses due to the tendency of more frequent changes of the last over the former.
+     If server IP address changes but domain name remains the same the HA cluster can continue working as normal
+     in case when domain names are used in cluster configuration. In case when IP addresses are used and they are changed with the time
+     than Qpid <link linkend="HAJMXAPI">JMX API for HA</link> can be used to change the addresses or remove the nodes from the cluster.</para></note>
+
+    <section role="h4" id="HAConfiguration_BDBEnvVars">
+      <title>Passing BDB environment and replication configuration options</title>
+      <para>It is possible to pass BDB <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/EnvironmentConfig.html">
+         environment</ulink> and <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/rep/ReplicationConfig.html">
+         replication</ulink> configuration options from the virtualhost.xml. Environment configuration options are passed using
+         the <varname>envConfig</varname> element, and replication config using <varname>repConfig</varname>.</para>
+      <para>For example, to override the BDB environment configuration options <varname>je.cleaner.threads</varname> and
+        <varname>je.txn.timeout</varname></para>
+      <programlisting language="xml"><![CDATA[
+         ...
+      </highAvailability>
+      <envConfig>
+        <name>je.cleaner.threads</name>
+        <value>2</value>
+      </envConfig>
+      <envConfig>
+        <name>je.txn.timeout</name>
+        <value>15 min</value>
+      </envConfig>
+      ...
+    </store>]]></programlisting>
+      <para>And to override the BDB replication configuration options <varname>je.rep.insufficientReplicasTimeout</varname>.</para>
+      <programlisting language="xml"><![CDATA[
+         ...
+      </highAvailability>
+      ...
+      <repConfig>
+        <name>je.rep.insufficientReplicasTimeout</name>
+        <value>2</value>
+      </envConfig>
+      <envConfig>
+        <name>je.txn.timeout</name>
+        <value>10 s</value>
+      </envConfig>
+      ...
+    </store>]]></programlisting>
+    </section>
+  </section>
+  <section role="h3" id="HADurabilityGuarantee">
+    <title>Durability Guarantees</title>
+    <para>The term <ulink url="http://en.wikipedia.org/wiki/ACID#Durability">durability</ulink> is used to mean that once a
+      transaction is committed, it remains committed regardless of subsequent failures. A highly durable system is one where
+      loss of a committed transaction is extermely unlikely, whereas with a less durable system loss of a transaction is likely
+      in a greater number of scenarios.  Typically, the more highly durable a system the slower and more costly it will be.</para>
+    <para>Qpid exposes the all the
+      <ulink url="&oracleBdbRepGuideUrl;txn-management.html#durabilitycontrols">durability controls</ulink>
+      offered by by BDB JE JA and a Qpid specific optimisation called <emphasis role="bold">coalescing-sync</emphasis> which defaults
+      to enabled.</para>
+    <section role="h4" id="HADurabilityGuarantee_BDBControls">
+      <title>BDB Durability Controls</title>
+      <para>BDB expresses durability as a triplet with the following form:</para>
+      <programlisting><![CDATA[<master sync policy>,<replica sync policy>,<replica acknowledgement policy>]]></programlisting>
+      <para>The sync polices controls whether the thread performing the committing thread awaits the successful completion of the
+        write, or the write and sync before continuing. The master sync policy and replica sync policy need not be the same.</para>
+      <para>For master and replic sync policies, the available values are:
+        <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.SyncPolicy.html#SYNC">SYNC</ulink>,
+        <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.SyncPolicy.html#WRITE_NO_SYNC">WRITE_NO_SYNC</ulink>,
+        <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.SyncPolicy.html#NO_SYNC">NO_SYNC</ulink>. SYNC
+        is offers the highest durability whereas NO_SYNC the lowest.</para>
+      <para>Note: the combination of a master sync policy of SYNC and <link linkend="HADurabilityGuarantee_CoalescingSync">coalescing-sync</link>
+        true would result in poor performance with no corresponding increase in durability guarantee.  It cannot not be used.</para>
+      <para>The acknowledgement policy defines whether when a master commits a transaction, it also awaits for the replica(s) to
+         commit the same transaction before continuing.  For the two-node case, ALL and SIMPLE_MAJORITY are equal.</para>
+      <para>For acknowledgement policy, the available value are:
+         <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.ReplicaAckPolicy.html#ALL">ALL</ulink>,
+         <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.ReplicaAckPolicy.html#SIMPLE_MAJORITY">SIMPLE_MAJORITY</ulink>
+         <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/Durability.ReplicaAckPolicy.html#NONE">NONE</ulink>.</para>
+    </section>
+    <section role="h4" id="HADurabilityGuarantee_CoalescingSync">
+      <title>Coalescing-sync</title>
+      <para>If enabled (the default) Qpid works to reduce the number of separate
+        <ulink url="&oracleJdkDocUrl;java/io/FileDescriptor.html#sync()">file-system sync</ulink> operations
+        performed by the <emphasis role="bold">master</emphasis> on the underlying storage device thus improving performance.  It does
+        this coalescing separate sync operations arising from the different client commits operations occuring at approximately the same time.
+        It does this in such a manner not to reduce the ACID guarantees of the system.</para>
+      <para>Coalescing-sync has no effect on the behaviour of the replicas.</para>
+    </section>
+    <section role="h4" id="HADurabilityGuarantee_Default">
+      <title>Default</title>
+      <para>The default durability guarantee is <constant>NO_SYNC, NO_SYNC, SIMPLE_MAJORITY</constant> with coalescing-sync enabled. The effect
+         of this combination is described in the table below. It offers a good compromise between durability guarantee and performance
+         with writes being guaranteed on the master and the additional guarantee that a majority of replicas have received the
+         transaction.</para>
+    </section>
+    <section role="h4" id="HADurabilityGuarantee_Examples">
+      <title>Examples</title>
+      <para>Here are some examples illustrating the effects of the durability and coalescing-sync settings.</para>
+      <para>
+        <table>
+          <title>Effect of different durability guarantees</title>
+          <tgroup cols="4">
+            <thead>
+              <row>
+                <entry/>
+                <entry>Durability</entry>
+                <entry>Coalescing-sync</entry>
+                <entry>Description</entry>
+              </row>
+            </thead>
+            <tbody>
+              <row>
+                <entry>1</entry>
+                <entry>NO_SYNC, NO_SYNC, SIMPLE_MAJORITY</entry>
+                <entry>true</entry>
+                <entry>Before the commit returns to the client, the transaction will be written/sync'd to the Master's disk (effect of
+                   coalescing-sync) and a majority of the replica(s) will have acknowledged the <emphasis role="bold">receipt</emphasis>
+                   of the transaction.  The replicas will write and sync the transaction to their disk at a point in the future governed by
+                   <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/rep/ReplicationMutableConfig.html#LOG_FLUSH_TASK_INTERVAL">ReplicationMutableConfig#LOG_FLUSH_INTERVAL</ulink>.
+                </entry>
+              </row>
+              <row>
+                <entry>2</entry>
+                <entry>NO_SYNC, WRITE_NO_SYNC, SIMPLE_MAJORITY</entry>
+                <entry>true</entry>
+                <entry>Before the commit returns to the client, the transaction will be written/sync'd to the Master's disk (effect of
+                  coalescing-sync and a majority of the replica(s) will have acknowledged the <emphasis role="bold">write</emphasis> of
+                  the transaction to their disk.  The replicas will sync the transaction to disk at a point in the future with an upper bound governed by
+                  ReplicationMutableConfig#LOG_FLUSH_INTERVAL.</entry>
+              </row>
+              <row>
+                <entry>3</entry>
+                <entry>NO_SYNC, NO_SYNC, NONE</entry>
+                <entry>false</entry>
+                <entry>After the commit returns to the client, the transaction is neither guaranteed to be written to the disk of the master
+                   nor received by any of the replicas. The master and replicas will write and sync the transaction to their disk at a point
+                   in the future with an upper bound governed by ReplicationMutableConfig#LOG_FLUSH_INTERVAL. This offers the weakest durability guarantee.</entry>
+              </row>
+            </tbody>
+          </tgroup>
+        </table>
+      </para>
+    </section>
+  </section>
+  <section id="HAClientFailover">
+    <title>Client failover configuration</title>
+    <para>The details about format of Qpid connection URLs can be found at section
+        <ulink url="../../Programming-In-Apache-Qpid/html/QpidJNDI.html">Connection URLs</ulink>
+        of book <ulink url="../../Programming-In-Apache-Qpid/html/">Programming In Apache Qpid</ulink>.</para>
+    <para>The failover policy option in the connection URL for the HA Cluster should be set to <emphasis>roundrobin</emphasis>.
+      The Master broker should be put into a first place in <emphasis>brokerlist</emphasis> URL option.
+      The recommended value for <emphasis>connectdelay</emphasis> option in broker URL should be set to
+      the value greater than 1000 milliseconds. If it is desired that clients re-connect automatically after a
+      master to replica failure, <varname>cyclecount</varname> should be tuned so that the retry period is longer than
+      the expected length of time to perform the failover.</para>
+    <example><title>Example of connection URL for the HA Cluster</title><![CDATA[
+amqp://guest:guest@clientid/test?brokerlist='tcp://localhost:5672?connectdelay='2000'&retries='3';tcp://localhost:5671?connectdelay='2000'&retries='3';tcp://localhost:5673?connectdelay='2000'&retries='3''&failover='roundrobin?cyclecount='30''
+        ]]></example>
+  </section>
+  <section role="h3" id="HAJMXAPI">
+    <title>Qpid JMX API for HA</title>
+    <para>Qpid exposes the BDB HA store information via its JMX interface and provides APIs to remove a Node from
+     the group, update a Node IP address, and assign a Node as the designated primary.</para>
+    <para>An instance of the <classname>BDBHAMessageStore</classname> MBean is instantiated by the broker for the each virtualhost using the HA store.</para>
+    <para>The reference to this MBean can be obtained via JMX API using an ObjectName like <emphasis>org.apache.qpid:type=BDBHAMessageStore,name=&lt;virtualhost name&gt;</emphasis>
+                 where &lt;virtualhost name&gt; is the name of a specific virtualhost on the broker.</para>
+    <table border="1">
+      <title>Mbean <classname>BDBHAMessageStore</classname> attributes</title>
+      <thead>
+        <tr>
+          <td>Name</td>
+          <td>Type</td>
+          <td>Accessibility</td>
+          <td>Description</td>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td>GroupName</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Name identifying the group</td>
+        </tr>
+        <tr>
+          <td>NodeName</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Unique name identifying the node within the group</td>
+        </tr>
+        <tr>
+          <td>NodeHostPort</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Host/port used to replicate data between this node and others in the group</td>
+        </tr>
+        <tr>
+          <td>HelperHostPort</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Host/port used to allow a new node to discover other group members</td>
+        </tr>
+        <tr>
+          <td>NodeState</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Current state of the node</td>
+        </tr>
+        <tr>
+          <td>ReplicationPolicy</td>
+          <td>String</td>
+          <td>Read only</td>
+          <td>Node replication durability</td>
+        </tr>
+        <tr id="JMXDesignatedPrimary">
+          <td>DesignatedPrimary</td>
+          <td>boolean</td>
+          <td>Read/Write</td>
+          <td>Designated primary flag. Applicable to the two node case.</td>
+        </tr>
+        <tr>
+          <td>CoalescingSync</td>
+          <td>boolean</td>
+          <td>Read only</td>
+          <td>Coalescing sync flag. Applicable to the master sync policies NO_SYNC and WRITE_NO_SYNC only.</td>
+        </tr>
+       <tr>
+          <td>getAllNodesInGroup</td>
+          <td>TabularData</td>
+          <td>Read only</td>
+          <td>Get all nodes within the group, regardless of whether currently attached or not</td>
+        </tr>
+      </tbody>
+    </table>
+    <table border="1">
+      <title>Mbean <classname>BDBHAMessageStore</classname> operations</title>
+      <thead>
+        <tr>
+          <td>Operation</td>
+          <td>Parameters</td>
+          <td>Returns</td>
+          <td>Description</td>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td>removeNodeFromGroup</td>
+          <td>
+            <para><emphasis>nodeName</emphasis>, name of node, string</para>
+          </td>
+          <td>void</td>
+          <td>Remove an existing node from the group</td>
+        </tr>
+        <tr>
+          <td>updateAddress</td>
+          <td>
+            <itemizedlist>
+              <itemizedlist>
+                <para><emphasis>nodeName</emphasis>, name of node, string</para>
+              </itemizedlist>
+              <itemizedlist>
+                <para><emphasis>newHostName</emphasis>, new host name, string</para>
+              </itemizedlist>
+              <itemizedlist>
+                <para><emphasis>newPort</emphasis>, new port number, int</para>
+              </itemizedlist>
+            </itemizedlist>
+          </td>
+          <td>void</td>
+          <td>Update the address of another node. The node must be in a STOPPED state.</td>
+        </tr>
+      </tbody>
+    </table>
+    <figure>
+      <title>BDBHAMessageStore view from jconsole.</title>
+      <graphic fileref="images/HA-BDBHAMessageStore-MBean-jconsole.png"/>
+    </figure>
+    <example>
+      <title>Example of java code to get the node state value</title>
+      <programlisting language="java"><![CDATA[
+Map<String, Object> environment = new HashMap<String, Object>();
+
+// credentials: user name and password
+environment.put(JMXConnector.CREDENTIALS, new String[] {"admin","admin"});
+JMXServiceURL url =  new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:9001/jmxrmi");
+JMXConnector jmxConnector = JMXConnectorFactory.connect(url, environment);
+MBeanServerConnection mbsc =  jmxConnector.getMBeanServerConnection();
+
+ObjectName queueObjectName = new ObjectName("org.apache.qpid:type=BDBHAMessageStore,name=test");
+String state = (String)mbsc.getAttribute(queueObjectName, "NodeState");
+
+System.out.println("Node state:" + state);
+        ]]></programlisting>
+      <para>Example system output:</para>
+      <screen><![CDATA[Node state:MASTER]]></screen>
+    </example>
+  </section>
+  <section id="BDB-HA-Monitoring-cluster">
+    <title>Monitoring cluster</title>
+    <para>In order to discover potential issues with HA Cluster early, all nodes in the Cluster should be monitored on regular basis
+    using the following techniques:</para>
+    <itemizedlist>
+      <listitem>
+        <para>Broker log files scrapping for WARN or ERROR entries and operational log entries like:</para>
+        <itemizedlist>
+          <listitem>
+            <para><emphasis>MST-1007 :</emphasis> Store Passivated. It can indicate that Master virtual host has gone down.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>MST-1006 :</emphasis> Recovery Complete. It can indicate that a former Replica virtual host is up and became the Master.</para>
+          </listitem>
+        </itemizedlist>
+      </listitem>
+      <listitem>
+        <para>Disk space usage and system load using system tools.</para>
+      </listitem>
+      <listitem>
+        <para>Berkeley HA node status using <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/rep/util/DbPing.html"><classname>DbPing</classname></ulink> utility.</para>
+        <example><title>Using <classname>DbPing</classname> utility for monitoring HA nodes.</title><command>
+java -jar je-&oracleBdbProductVersion;.jar DbPing -groupName TestClusterGroup -nodeName Node-5001 -nodeHost localhost:5001 -socketTimeout 10000
+</command><screen>
+Current state of node: Node-5001 from group: TestClusterGroup
+  Current state: MASTER
+  Current master: Node-5001
+  Current JE version: &oracleBdbProductVersion;
+  Current log version: 8
+  Current transaction end (abort or commit) VLSN: 165
+  Current master transaction end (abort or commit) VLSN: 0
+  Current active feeders on node: 0
+  Current system load average: 0.35
+</screen></example>
+        <para>In the example above <classname>DbPing</classname> utility requested status of Cluster node with name
+            <emphasis>Node-5001</emphasis> from replication group <emphasis>TestClusterGroup</emphasis> running on host <emphasis>localhost:5001</emphasis>.
+            The state of the node was reported into a system output.
+            </para>
+      </listitem>
+      <listitem>
+        <para>Using Qpid broker JMX interfaces.</para>
+        <para>Mbean <classname>BDBHAMessageStore</classname> can be used to request the following node information:</para>
+        <itemizedlist>
+          <listitem>
+            <para><emphasis>NodeState</emphasis> indicates whether node is a Master or Replica.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>Durability</emphasis> replication durability.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>DesignatedPrimary</emphasis> indicates whether Master node is designated primary.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>GroupName</emphasis> replication group name.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>NodeName</emphasis> node name.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>NodeHostPort</emphasis> node host and port.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>HelperHostPort</emphasis> helper host and port.</para>
+          </listitem>
+          <listitem>
+            <para><emphasis>AllNodesInGroup</emphasis> lists of all nodes in the replication group including their names, hosts and ports.</para>
+          </listitem>
+        </itemizedlist>
+        <para>For more details about <classname>BDBHAMessageStore</classname> MBean please refer section <link linkend="HAJMXAPI">Qpid JMX API for HA</link></para>
+      </listitem>
+    </itemizedlist>
+  </section>
+  <section id="HADiskSpace">
+    <title>Disk space requirements</title>
+    <para>Disk space is a critical resource for the HA Qpid broker.</para>
+    <para>In case when a Replica goes down (or falls behind the Master in 2 node cluster where the Master is designated primary)
+    and the Master continues running, the non-replicated store files are kept on the Masters disk for the period of time
+    as specified in <emphasis>je.rep.repStreamTimeout</emphasis> JE setting in order to replicate this data later
+    when the Replica is back. This setting is set to 1 hour by default by the broker. The setting can be overridden as described in
+    <xref linkend="HAConfiguration_BDBEnvVars"/>.</para>
+    <para>Depending from the application publishing/consuming rates and message sizes,
+    the disk space might become overfull during this period of time due to preserved logs.
+    Please, make sure to allocate enough space on your disk to avoid this from happening.
+    </para>
+  </section>
+  <section id="BDB-HA-Network-Requirements">
+    <title>Network Requirements</title>
+    <para>The HA Cluster performance depends on the network bandwidth, its use by existing traffic, and quality of service.</para>
+    <para>In order to achieve the best performance it is recommended to use a separate network infrastructure for the Qpid HA Nodes
+     which might include installation of dedicated network hardware on Broker hosts, assigning a higher priority to replication ports,
+     installing a cluster in a separate network not impacted by any other traffic.</para>
+  </section>
+  <section id="BDB-HA-Security">
+    <title>Security</title>
+    <para>At the moment Berkeley replication API supports only TCP/IP protocol to transfer replication data between Master and Replicas.</para>
+    <para>As result, the replicated data is unprotected and can be intercepted by anyone having access to the replication network.</para>
+    <para>Also, anyone who can access to this network can introduce a new node and therefore receive a copy of the data.</para>
+    <para>In order to reduce the security risks the entire HA cluster is recommended to run in a separate network protected from general access.</para>
+  </section>
+  <section id="BDB-HA-Backup">
+    <title>Backups</title>
+    <para>In order to protect the entire cluster from some cataclysms which might destroy all cluster nodes,
+    backups of the Master store should be taken on a regular basis.</para>
+    <para>Qpid Broker distribution includes the "hot" backup utility <emphasis>backup.sh</emphasis> which can be found at broker bin folder.
+         This utility can perform the backup when broker is running.</para>
+    <para><emphasis>backup.sh</emphasis> script invokes <classname>org.apache.qpid.server.store.berkeleydb.BDBBackup</classname> to do the job.</para>
+    <para>You can also run this class from command line like in an example below:</para>
+    <example><title>Performing store backup by using <classname>BDBBackup</classname> class directly</title><command>
+        java -cp qpid-bdbstore-0.18.jar org.apache.qpid.server.store.berkeleydb.BDBBackup -fromdir path/to/store/folder -todir path/to/backup/foldeAr</command>
+    </example>
+    <para>In the example above BDBBackup utility is called from qpid-bdbstore-0.18.jar to backup the store at <emphasis>path/to/store/folder</emphasis> and copy store logs into <emphasis>path/to/backup/folder</emphasis>.</para>
+    <para>Linux and Unix users can take advantage of <emphasis>backup.sh</emphasis> bash script by running this script in a similar way.</para>
+    <example><title>Performing store backup by using <classname>backup.sh</classname> bash script</title>
+        <command>backup.sh -fromdir path/to/store/folder -todir path/to/backup/folder</command>
+    </example>
+    <note>
+      <para>Do not forget to ensure that the Master store is being backed up, in the event the Node elected Master changes during
+      the lifecycle of the cluster.</para>
+    </note>
+  </section>
+  <section id="HAMigrationFromNonHA">
+    <title>Migration of a non-HA store to HA</title>
+    <para>Non HA stores starting from schema version 4 (0.14 Qpid release) can be automatically converted into HA store on broker startup if replication is first enabled with the <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/rep/util/DbEnableReplication.html"><classname>DbEnableReplication</classname></ulink> utility from the BDB JE jar.</para>
+    <para>DbEnableReplication converts a non HA store into an HA store and can be used as follows:</para>
+    <example><title>Enabling replication</title><command>
+java -jar je-&oracleBdbProductVersion;.jar DbEnableReplication -h /path/to/store -groupName MyReplicationGroup -nodeName MyNode1 -nodeHostPort  localhost:5001
+        </command></example>
+    <para>In the examples above, je jar of version &oracleBdbProductVersion; is used to convert store at <emphasis>/path/to/store</emphasis> into HA store having replication group name <emphasis>MyReplicationGroup</emphasis>, node name <emphasis>MyNode1</emphasis> and running on host <emphasis>localhost</emphasis> and port <emphasis>5001</emphasis>.</para>
+    <para>After running DbEnableReplication and updating the virtual host store to configuration to be an HA message store, like in example below,
+    on broker start up the store schema will be upgraded to the most recent version and the broker can be used as normal.</para>
+    <example>
+      <title>Example of XML configuration for HA message store</title>
+      <programlisting language="xml"><![CDATA[
+<store>
+    <class>org.apache.qpid.server.store.berkeleydb.BDBHAMessageStore</class>
+    <environment-path>/path/to/store</environment-path>
+    <highAvailability>
+        <groupName>MyReplicationGroup</groupName>
+        <nodeName>MyNode1</nodeName>
+        <nodeHostPort>localhost:5001</nodeHostPort>
+        <helperHostPort>localhost:5001</helperHostPort>
+    </highAvailability>
+</store>]]></programlisting>
+    </example>
+    <para>The Replica nodes can be started with empty stores. The data will be automatically copied from Master to Replica on Replica start-up.
+      This will take a period of time determined by the size of the Masters store and the network bandwidth between the nodes.</para>
+    <note>
+      <para>Due to existing caveats in Berkeley JE with copying of data from Master into Replica it is recommended to restart the Master node after store schema upgrade is finished before starting the Replica nodes.</para>
+    </note>
+  </section>
+  <section id="HADisasterRecovery">
+    <title>Disaster Recovery</title>
+    <para>This section describes the steps required to restore HA broker cluster from backup.</para>
+    <para>The detailed instructions how to perform backup on replicated environment can be found <link linkend="BDB-HA-Backup">here</link>.</para>
+    <para>At this point we assume that backups are collected on regular basis from Master node.</para>
+    <para>Replication configuration of a cluster is stored internally in HA message store.
+    This information includes IP addresses of the nodes.
+    In case when HA message store needs to be restored on a different host with a different IP address
+    the cluster replication configuration should be reseted in this case</para>
+    <para>Oracle provides a command line utility <ulink url="&oracleBdbJavaDocUrl;com/sleepycat/je/rep/util/DbResetRepGroup.html"><classname>DbResetRepGroup</classname></ulink>
+    to reset the members of a replication group and replace the group with a new group consisting of a single new member
+    as described by the arguments supplied to the utility</para>
+    <para>Cluster can be restored with the following steps:</para>
+    <itemizedlist>
+         <listitem><para>Copy log files into the store folder from backup</para></listitem>
+         <listitem>
+            <para>Use <classname>DbResetRepGroup</classname> to reset an existing environment. See an example below</para>
+            <example>
+                <title>Reseting of replication group with <classname>DbResetRepGroup</classname></title><command>
+java -cp je-&oracleBdbProductVersion;.jar com.sleepycat.je.rep.util.DbResetRepGroup -h ha-work/Node-5001/bdbstore -groupName TestClusterGroup -nodeName Node-5001 -nodeHostPort localhost:5001</command>
+            </example>
+            <para>In the example above <classname>DbResetRepGroup</classname> utility from Berkeley JE of version &oracleBdbProductVersion; is used to reset the store
+            at location <emphasis>ha-work/Node-5001/bdbstore</emphasis> and set a replication group to <emphasis>TestClusterGroup</emphasis>
+            having a node <emphasis>Node-5001</emphasis> which runs at <emphasis>localhost:5001</emphasis>.</para>
+         </listitem>
+         <listitem><para>Start a broker with HA store configured as specified on running of <classname>DbResetRepGroup</classname> utility.</para></listitem>
+         <listitem><para>Start replica nodes having the same replication group and a helper host port pointing to a new master. The store content will be copied into Replicas from Master on their start up.</para></listitem>
+    </itemizedlist>
+  </section>
+
+  <section id="HAPerformance">
+    <title>Performance</title>
+    <para>The aim of this section is not to provide exact performance metrics relating to HA, as this depends heavily on the test
+    environment, but rather showing an impact of HA on Qpid broker performance in comparison with the Non HA case.</para>
+    <para>For testing of impact of HA on a broker performance a special test script was written using Qpid performance test framework.
+    The script opened a number of connections to the Qpid broker, created producers and consumers on separate connections,
+    and published test messages with concurrent producers into a test queue and consumed them with concurrent consumers.
+    The table below shows the number of producers/consumers used in the tests.
+    The overall throughput was collected for each configuration.
+    </para>
+    <table border="1">
+      <title>Number of producers/consumers in performance tests</title>
+      <thead>
+        <tr>
+          <th>Test</th>
+          <th>Number of producers</th>
+          <th>Number of consumers</th>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td>1</td>
+          <td>1</td>
+          <td>1</td>
+        </tr>
+        <tr>
+          <td>2</td>
+          <td>2</td>
+          <td>2</td>
+        </tr>
+        <tr>
+          <td>3</td>
+          <td>4</td>
+          <td>4</td>
+        </tr>
+        <tr>
+          <td>4</td>
+          <td>8</td>
+          <td>8</td>
+        </tr>
+        <tr>
+          <td>5</td>
+          <td>16</td>
+          <td>16</td>
+        </tr>
+        <tr>
+          <td>6</td>
+          <td>32</td>
+          <td>32</td>
+        </tr>
+        <tr>
+          <td>7</td>
+          <td>64</td>
+          <td>64</td>
+        </tr>
+      </tbody>
+    </table>
+    <para>The test was run against the following Qpid Broker configurations</para>
+    <itemizedlist>
+      <listitem>
+        <para>Non HA Broker</para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>SYNC,SYNC,ALL</emphasis></para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>WRITE_NO_SYNC,WRITE_NO_SYNC,ALL</emphasis></para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>WRITE_NO_SYNC,WRITE_NO_SYNC,ALL</emphasis> and <emphasis>coalescing-sync</emphasis> Qpid mode</para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>WRITE_NO_SYNC,NO_SYNC,ALL</emphasis> and <emphasis>coalescing-sync</emphasis> Qpid mode</para>
+      </listitem>
+      <listitem>
+        <para>HA 2 Nodes Cluster with durability <emphasis>NO_SYNC,NO_SYNC,ALL</emphasis> and <emphasis>coalescing-sync</emphasis> Qpid option</para>
+      </listitem>
+    </itemizedlist>
+    <para>The evironment used in testing consisted of 2 servers with 4 CPU cores (2x Intel(r) Xeon(R) CPU 5150@2.66GHz), 4GB of RAM
+        and running under OS Red Hat Enterprise Linux AS release 4 (Nahant Update 4). Network bandwidth was 1Gbit.
+    </para>
+    <para>We ran Master node on the first server and Replica and clients(both consumers and producers) on the second server.</para>
+    <para>In non-HA case Qpid Broker was run on a first server and clients were run on a second server.</para>
+    <para>The table below contains the test results we measured on this environment for different Broker configurations.</para>
+    <para>Each result is represented by throughput value in KB/second and difference in % between HA configuration and non HA case for the same number of clients.</para>
+    <table border="1">
+      <title>Performance Comparison</title>
+      <thead>
+        <tr>
+          <td>Test/Broker</td>
+          <td>No HA</td>
+          <td>SYNC, SYNC, ALL</td>
+          <td>WRITE_NO_SYNC, WRITE_NO_SYNC, ALL</td>
+          <td>WRITE_NO_SYNC, WRITE_NO_SYNC, ALL - coalescing-sync</td>
+          <td>WRITE_NO_SYNC, NO_SYNC,ALL - coalescing-sync</td>
+          <td>NO_SYNC, NO_SYNC, ALL - coalescing-sync</td>
+        </tr>
+      </thead>
+      <tbody>
+        <tr>
+          <td>1 (1/1)</td>
+          <td>0.0%</td>
+          <td>-61.4%</td>
+          <td>117.0%</td>
+          <td>-16.02%</td>
+          <td>-9.58%</td>
+          <td>-25.47%</td>
+        </tr>
+        <tr>
+          <td>2 (2/2)</td>
+          <td>0.0%</td>
+          <td>-75.43%</td>
+          <td>67.87%</td>
+          <td>-66.6%</td>
+          <td>-69.02%</td>
+          <td>-30.43%</td>
+        </tr>
+        <tr>
+          <td>3 (4/4)</td>
+          <td>0.0%</td>
+          <td>-84.89%</td>
+          <td>24.19%</td>
+          <td>-71.02%</td>
+          <td>-69.37%</td>
+          <td>-43.67%</td>
+        </tr>
+        <tr>
+          <td>4 (8/8)</td>
+          <td>0.0%</td>
+          <td>-91.17%</td>
+          <td>-22.97%</td>
+          <td>-82.32%</td>
+          <td>-83.42%</td>
+          <td>-55.5%</td>
+        </tr>
+        <tr>
+          <td>5 (16/16)</td>
+          <td>0.0%</td>
+          <td>-91.16%</td>
+          <td>-21.42%</td>
+          <td>-86.6%</td>
+          <td>-86.37%</td>
+          <td>-46.99%</td>
+        </tr>
+        <tr>
+          <td>6 (32/32)</td>
+          <td>0.0%</td>
+          <td>-94.83%</td>
+          <td>-51.51%</td>
+          <td>-92.15%</td>
+          <td>-92.02%</td>
+          <td>-57.59%</td>
+        </tr>
+        <tr>
+          <td>7 (64/64)</td>
+          <td>0.0%</td>
+          <td>-94.2%</td>
+          <td>-41.84%</td>
+          <td>-89.55%</td>
+          <td>-89.55%</td>
+          <td>-50.54%</td>
+        </tr>
+      </tbody>
+    </table>
+    <para>The figure below depicts the graphs for the performance test results</para>
+    <figure>
+      <title>Test results</title>
+      <graphic fileref="images/HA-perftests-results.png"/>
+    </figure>
+    <para>On using durability <emphasis>SYNC,SYNC,ALL</emphasis> (without coalescing-sync) the performance drops significantly (by 62-95%) in comparison with non HA broker.</para>
+    <para>Whilst, on using durability <emphasis>WRITE_NO_SYNC,WRITE_NO_SYNC,ALL</emphasis> (without coalescing-sync) the performance drops by only half, but with loss of durability guarantee, so is not recommended.</para>
+    <para>In order to have better performance with HA, Qpid Broker comes up with the special mode called <link linkend="HADurabilityGuarantee_CoalescingSync">coalescing-sync</link>,
+    With this mode enabled, Qpid broker batches the concurrent transaction commits and syncs transaction data into Master disk in one go.
+    As result, the HA performance only drops by 25-60% for durability <emphasis>NO_SYNC,NO_SYNC,ALL</emphasis> and by 10-90% for <emphasis>WRITE_NO_SYNC,WRITE_NO_SYNC,ALL</emphasis>.</para>
+  </section>
+</section>
diff --git a/doc/book/src/java-broker/MessageStore-Tool.xml b/doc/book/src/java-broker/MessageStore-Tool.xml
deleted file mode 100644
index fdcb3cd560..0000000000
--- a/doc/book/src/java-broker/MessageStore-Tool.xml
+++ /dev/null
@@ -1,150 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!--
- 
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements.  See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership.  The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License.  You may obtain a copy of the License at
- 
-   http://www.apache.org/licenses/LICENSE-2.0
- 
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied.  See the License for the
- specific language governing permissions and limitations
- under the License.
- 
--->
-
-<section><title>
-      MessageStore Tool
-    </title><section role="h2" id="MessageStoreTool-MessageStoreTool"><title>
-            MessageStore Tool
-          </title>
-
-	 <para>
-            We have a number of implementations of the Qpid MessageStore
-            interface. This tool allows the interrogation of these stores
-            while the broker is offline.
-          </para>
-
-	  <section role="h3" id="MessageStoreTool-MessageStoreImplementations"><title>
-            MessageStore
-            Implementations
-          </title>
-
-	 <itemizedlist>
-            <listitem><para>
-              <xref linkend="qpid_BDBMessageStore--3rd-Party-"/>
-            </para></listitem>
-            <listitem><para>
-              <xref linkend="qpid_JDBCStore"/>
-            </para></listitem>
-            <listitem><para>
-              <xref linkend="qpid_MemoryMessageStore"/>
-            </para></listitem>
-          </itemizedlist>
-<!--h3--></section>
-
-	  <section role="h3" id="MessageStoreTool-Introduction"><title>
-            Introduction
-          </title>
-
-	 <para>
-            Each of the MessageStore implementations provide different back
-            end storage for their messages and so would need a different tool
-            to be able to interrogate their contents at the back end.
-          </para><para>
-            What this tool does is to utilise the Java broker code base to
-            access the contents of the storage providing the user with a
-            consistent means to inspect the storage contents in broker
-            memory. The tool allows the current messages in the store to be
-            inspected and copied/moved between queues. The tool uses the
-            message instance in memory for all its access paths, but changes
-            made will be reflected in the physical store (if one exists).
-          </para>
-<!--h3--></section>
-
-	  <section role="h3" id="MessageStoreTool-Usage"><title>
-            Usage
-          </title>
-
-	 <para>
-            The tools-distribution currently includes a unix shell command
-            'msTool.sh' this script will launch the java tool.
-          </para><para>
-            The tool loads $QPID_HOME/etc/config.xml by default. If an
-            alternative broker configuration is required this should be
-            provided on the command line as would be done for the broker.
-          </para>
-            <programlisting>
-msTool.sh -c &lt;path to different config.xml&gt;
-</programlisting>
-          <para>
-            On startup the user is present with a command prompt
-          </para>
-            <programlisting>
-$ msTool.sh
-MessageStoreTool - for examining Persistent Qpid Broker MessageStore instances
-bdb$ 
-</programlisting>
-<!--h3--></section>
-
-          <section role="h3" id="MessageStoreTool-AvailableCommands"><title>
-            Available
-            Commands
-          </title>
-
-	 <para>
-            The available commands in the tool can be seen through the use of
-            the 'help' command.
-          </para>
-            <programlisting>
-bdb$ help
-+----------------------------------------------------------------+
-|                       Available Commands                       |
-+----------------------------------------------------------------+
-| Command | Description                                          |
-+----------------------------------------------------------------+
-| quit    | Quit the tool.                                       |
-| list    | list available items.                                |
-| dump    | Dump selected message content. Default: show=content |
-| load    | Loads specified broker configuration file.           |
-| clear   | Clears any selection.                                |
-| show    | Shows the messages headers.                          |
-| select  | Perform a selection.                                 |
-| help    | Provides detailed help on commands.                  |
-+----------------------------------------------------------------+
-bdb$
-</programlisting>
-          <para>
-            A brief description is displayed and further usage information is
-            shown with 'help &lt;command&gt;'
-          </para>
-            <programlisting>
-bdb$ help list
-list availble items.
-Usage:list queues [&lt;exchange&gt;] | exchanges | bindings [&lt;exchange&gt;] | all
-bdb$
-</programlisting>
-<!--h3--></section>
-
-
-          <section role="h3" id="MessageStoreTool-FutureWork"><title>
-            Future Work
-          </title>
-
-	 <para>
-            Currently the tool only works whilst the broker is offline i.e.
-            it is up, but not accepting AMQP connections. This requires a
-            stop/start of the broker. If this functionality was incorporated
-            into the broker then a telnet functionality could be provided
-            allowing online management.
-          </para>
-<!--h3--></section>
-<!--h2--></section>
-</section>
diff --git a/doc/book/src/java-broker/OtherQueueTypes.xml b/doc/book/src/java-broker/OtherQueueTypes.xml
new file mode 100644
index 0000000000..d42e4e62cb
--- /dev/null
+++ b/doc/book/src/java-broker/OtherQueueTypes.xml
@@ -0,0 +1,274 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE urls [
+<!ENTITY oracleJeeDocUrl "http://docs.oracle.com/javaee/6/api/">
+]>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+<section id="OtherQueueTypes">
+  <title>Other Queue Types</title>
+
+  <section role="h2" id="OtherQueueTypes-Introduction">
+    <title>Introduction</title>
+    <para> In addition to the standard queue type where messages are delivered in the same order
+      that they were sent, the Java Broker supports three additional queue types which allows for
+      alternative delivery behaviours. These are <link linkend="OtherQueueTypes-Priority"
+        >priority-queues</link>, <link linkend="OtherQueueTypes-Sorted">sorted-queues</link>-, and
+        <link linkend="OtherQueueTypes-LVQ">last-value-queues</link> (LVQs). </para>
+    <para> In the following sections, the semantics of each queue type is described, followed by a
+      description of how instances of these queue can be created via <link
+        linkend="OtherQueueTypes-CreateUsingConfig">configuration</link> or <link
+        linkend="OtherQueueTypes-CreateUsingJmsOrJmx">programmatically</link>. </para>
+    <para>The final section discusses the importance of using a <link
+        linkend="OtherQueueTypes-SetLowPrefetch">low client pre-fetch</link> with these queued.
+    </para>
+  </section>
+
+  <section role="h2" id="OtherQueueTypes-Priority">
+    <title>Priority Queues</title>
+    <para>In a priority queue, messages on the queue are delivered in an order determined by the
+        <ulink url="&oracleJeeDocUrl;javax/jms/Message.html#getJMSPriority()">JMS priority message
+        header</ulink> within the message. By default Qpid supports the 10 priority levels mandated
+      by JMS, with priority value 0 as the lowest priority and 9 as the highest. </para>
+    <para>It is possible to reduce the effective number of priorities if desired.</para>
+    <para>JMS defines the <ulink url="&oracleJeeDocUrl;javax/jms/Message.html#DEFAULT_PRIORITY">
+        default message priority</ulink> as 4. Messages sent without a specified priority use this
+      default. </para>
+  </section>
+  <section role="h2" id="OtherQueueTypes-Sorted">
+    <title>Sorted Queues</title>
+    <para>Sorted queues allow the message delivery order to be determined by value of an arbitrary
+        <ulink url="&oracleJeeDocUrl;javax/jms/Message.html#getStringProperty()">JMS message
+        property</ulink>. Sort order is alpha-numeric and the property value must have a type
+      java.lang.String.</para>
+    <para>Messages sent to a sorted queue without the specified JMS message property will be
+      inserted into the 'last' position in the queue.</para>
+  </section>
+  <section role="h2" id="OtherQueueTypes-LVQ">
+    <title>Last Value Queues (LVQ)</title>
+    <para>LVQs (or conflation queues) are special queues that automatically discard any message when
+      a newer message arrives with the same key value. The key is specified by arbitrary <ulink
+        url="&oracleJeeDocUrl;javax/jms/Message.html#getPropertyNames()">JMS message
+        property</ulink>.</para>
+    <para>An example of an LVQ might be where a queue represents prices on a stock exchange: when
+      you first consume from the queue you get the latest quote for each stock, and then as new
+      prices come in you are sent only these updates. </para>
+    <para>Like other queues, LVQs can either be browsed or consumed from. When browsing an
+      individual subscriber does not remove the message from the queue when receiving it. This
+      allows for many subscriptions to browse the same LVQ (i.e. you do not need to create and bind
+      a separate LVQ for each subscriber who wishes to receive the contents of the LVQ).</para>
+    <para>Messages sent to an LVQ without the specified property will be delivered as normal and
+      will never be "replaced".</para>
+  </section>
+  <section role="h2" id="OtherQueueTypes-Create">
+    <title>Creating a Priority, Sorted or LVQ Queue</title>
+    <para>To create a priority, sorted or LVQ queue, it can be defined in the virtualhost
+      configuration file, or the queue can be created programmtically from a client via AMQP (using
+      an extension to JMS), or using JMX. These methods are described below. </para>
+    <para>Once a queue is created you cannot change its type (without deleting it and re-creating).
+      Also note you cannot currently mix the natures of these queue types, for instance, you cannot
+      define a queue which it both an LVQ and a priority-queue.</para>
+    <section role="h2" id="OtherQueueTypes-CreateUsingConfig">
+      <title>Using configuration</title>
+      <para>To create a priority, sorted or LVQ queue within configuration, add the appropriate xml
+        to the virtualhost.xml configuration file within the <varname>queues</varname>
+        element.</para>
+      <section role="h3" id="OtherQueueTypes-CreateUsingConfig-Priority">
+        <title>Priority</title>
+        <para> To defining a priority queue, add a &lt;priority&gt;true&lt;/priority&gt; element. By
+          default the queue will have 10 distinct priorities. </para>
+        <example>
+          <title>Configuring a priority queue</title>
+          <programlisting><![CDATA[<queue>
+    <name>myqueue</name>
+    <myqueue>
+        <exchange>amq.direct</exchange>
+        <priority>true</priority>
+    </myqueue>
+</queue>]]></programlisting>
+        </example>
+        <para> If you require fewer priorities, it is possible to specify a
+            <varname>priorities</varname> element (whose value is a integer value between 2 and 10
+          inclusive) which will give the queue that number of distinct priorities. When messages are
+          sent to that queue, their effective priority will be calculated by partitioning the
+          priority space. If the number of effective priorities is 2, then messages with priority
+          0-4 are treated the same as "lower priority" and messages with priority 5-9 are treated
+          equivalently as "higher priority". </para>
+        <example>
+          <title>Configuring a priority queue with fewer priorities</title>
+          <programlisting><![CDATA[<queue>
+    <name>myqueue</name>
+    <myqueue>
+        <exchange>amq.direct</exchange>
+        <priority>true</priority>
+        <priorities>4</priorities>
+    </myqueue>
+</queue>]]></programlisting>
+        </example>
+      </section>
+      <section role="h3" id="OtherQueueTypes-CreateUsingConfig-Sorted">
+        <title>Sorted</title>
+        <para> To define a sorted queue, add a <varname>sortKey</varname> element. The value of the
+            <varname>sortKey</varname> element defines the message property to use the value of when
+          sorting the messages put onto the queue. </para>
+        <example>
+          <title>Configuring a sorted queue</title>
+          <programlisting><![CDATA[<queue>
+    <name>myqueue</name>
+    <myqueue>
+        <exchange>amq.direct</exchange>
+        <sortKey>message-property-to-sort-by</sortKey>
+    </myqueue>
+</queue>]]></programlisting>
+        </example>
+      </section>
+      <section role="h3" id="OtherQueueTypes-CreateUsingConfig-LVQ">
+        <title>LVQ</title>
+        <para> To define a LVQ, add a <varname>lvq</varname> element with the value
+            <constant>true</constant>. Without any further configuration this will define an LVQ
+          which uses the JMS message property <constant>qpid.LVQ_key</constant> as the key for
+          replacement. </para>
+        <example>
+          <title>Configuring a LVQ queue</title>
+          <programlisting><![CDATA[<queue>
+    <name>myqueue</name>
+    <myqueue>
+        <exchange>amq.direct</exchange>
+        <lvq>true</lvq>
+    </myqueue>
+</queue>]]></programlisting>
+        </example>
+        <para> If you wish to define your own property then you can do so using the
+            <varname>lvqKey</varname> element.</para>
+        <example>
+          <title>Configuring a LVQ queue with custom message property name</title>
+          <programlisting><![CDATA[<queue>
+    <name>myqueue</name>
+    <myqueue>
+        <exchange>amq.direct</exchange>
+        <lvq>true</lvq>
+        <lvqKey>ISIN</lvqKey>
+    </myqueue>
+</queue>]]></programlisting>
+        </example>
+      </section>
+    </section>
+    <section role="h2" id="OtherQueueTypes-CreateUsingJmsOrJmx">
+      <title>Using JMS or AMQP</title>
+      <para>To create a priority, sorted or LVQ queue programmatically from JMX or using a Qpid
+        extension to JMS, pass the appropriate queue-declare arguments.</para>
+      <table>
+        <tgroup cols="4">
+          <thead>
+            <row>
+              <entry>Queue type</entry>
+              <entry>Argument name</entry>
+              <entry>Argument name</entry>
+              <entry>Argument Description</entry>
+            </row>
+          </thead>
+          <tbody>
+            <row>
+              <entry>priority</entry>
+              <entry>priorities</entry>
+              <entry>java.lang.Integer</entry>
+              <entry>Specifies a priority queue with given number priorities</entry>
+            </row>
+            <row>
+              <entry>sorted</entry>
+              <entry>qpid.queue_sort_key</entry>
+              <entry>java.lang.String</entry>
+              <entry>Specifies sorted queue with given message property used to sort the
+                entries</entry>
+            </row>
+            <row>
+              <entry>lvq</entry>
+              <entry>qpid.last_value_queue_key</entry>
+              <entry>java.lang.String</entry>
+              <entry>Specifies lvq queue with given message property used to conflate the
+                entries</entry>
+            </row>
+          </tbody>
+        </tgroup>
+      </table>
+      <para>The following example illustrates the creation of the a LVQ queue from a
+        javax.jms.Session object. Note that this utilises a Qpid specific extension to JMS and
+        involves casting the session object back to its Qpid base-class.</para>
+      <example>
+        <title>Creation of an LVQ using the Qpid extension to JMS</title>
+        <programlisting><![CDATA[Map<String,Object> arguments = new HashMap<String, Object>();
+arguments.put("qpid.last_value_queue_key","ISIN");
+((AMQSession<?,?>) session).createQueue(queueName, autoDelete, durable, exclusive, arguments);]]></programlisting>
+
+      </example>
+      <para> The following example illustrates the creation of the sorted queue from a the JMX
+        interface using the ManagedBroker interface. </para>
+      <example>
+        <title>Creation of a sorted queue using JMX</title>
+        <programlisting><![CDATA[Map<String, Object> environment = new HashMap<String, Object>();
+environment.put(JMXConnector.CREDENTIALS, new String[] {"admin","password"});
+// Connect to service
+JMXServiceURL url =  new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:8999/jmxrmi");
+JMXConnector jmxConnector = JMXConnectorFactory.connect(url, environment);
+MBeanServerConnection mbsc =  jmxConnector.getMBeanServerConnection();
+// Object name for ManagedBroker for virtualhost myvhost
+ObjectName objectName = new ObjectName("org.apache.qpid:type=VirtualHost.VirtualHostManager,VirtualHost=myvhost");
+// Get the ManagedBroker object
+ManagedBroker managedBroker = JMX.newMBeanProxy(mbsc, objectName, ManagedBroker.class);;
+
+// Create the queue passing arguments
+Map<String,Object> arguments = new HashMap<String, Object>();
+arguments.put("qpid.queue_sort_key","myheader");
+managedBroker.createNewQueue("myqueue", null, true, arguments);]]></programlisting>
+      </example>
+    </section>
+  </section>
+
+  <section role="h2" id="OtherQueueTypes-SetLowPrefetch">
+    <title>Low pre-fetch</title>
+    <para>Qpid clients receive buffered messages in batches, sized according to the pre-fetch value.
+      The current default is 500. </para>
+    <para>However, if you use the default value you will probably <emphasis>not</emphasis> see
+      desirable behaviour when using priority, sorted or lvq queues. Once the broker has sent a
+      message to the client its delivery order is then fixed, regardless of the special behaviour of
+      the queue. </para>
+    <para>For example, if using a priority queue and a prefetch of 100, and 100 messages arrive with
+      priority 2, the broker will send these messages to the client. If then a new message arrives
+      will priority 1, the broker cannot leap frog messages of lower priority. The priority 1 will
+      be delivered at the front of the next batch of messages to be sent to the client.</para>
+    <para> So, you need to set the prefetch values for your client (consumer) to make this sensible.
+      To do this set the Java system property <varname>max_prefetch</varname> on the client
+      environment (using -D) before creating your consumer. </para>
+    <para>A default for all client connections can be set via a system property: </para>
+    <programlisting>
+-Dmax_prefetch=1
+</programlisting>
+    <para> The prefetch can be also be adjusted on a per connection basis by adding a
+      <varname>maxprefetch</varname> value to the <ulink url="../../Programming-In-Apache-Qpid/html/QpidJNDI.html#section-jms-connection-url">Connection URLs</ulink>
+    </para>
+    <programlisting>
+amqp://guest:guest@client1/development?maxprefetch='1'&amp;brokerlist='tcp://localhost:5672'
+</programlisting>
+    <para>Setting the Qpid pre-fetch to 1 will give exact queue-type semantics as perceived by the
+      client however, this brings a performance cost. You could test with a slightly higher
+      pre-fetch to trade-off between throughput and exact semantics.</para>
+  </section>
+</section>
diff --git a/doc/book/src/java-broker/Producer-Flow-Control.xml b/doc/book/src/java-broker/Producer-Flow-Control.xml
new file mode 100644
index 0000000000..262279510e
--- /dev/null
+++ b/doc/book/src/java-broker/Producer-Flow-Control.xml
@@ -0,0 +1,217 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!--
+
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements.  See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership.  The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License.  You may obtain a copy of the License at
+
+   http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied.  See the License for the
+ specific language governing permissions and limitations
+ under the License.
+
+-->
+
+<section id="Qpid-Producer-Flow-Control">
+    <title>Producer Flow Control</title>
+
+    <section role="h2" id="QpidProducerFlowControlGeneralInformation">
+        <title>General Information</title>
+        <para>
+            The Qpid 0.6 release introduced a simplistic producer-side flow control mechanism
+            into the Java Messaging Broker, causing producers to be flow-controlled when they
+            attempt to send messages to an overfull queue. Qpid 0.18 introduced a similar
+            mechanism triggered by an overfull persistent message store on a virtual host.
+        </para>
+    </section>
+    <section role="h2" id="QpidProducerFlowControlServerConfiguration">
+        <title>Server Configuration</title>
+        <section role="h3">
+            <title>Configuring a Queue to use flow control</title>
+            <para>
+                Flow control is enabled on a producer when it sends a message to a Queue
+                which is "overfull". The producer flow control will be rescinded when all
+                Queues on which a producer is blocking become "underfull". A Queue is defined
+                as overfull when the size (in bytes) of the messages on the queue exceeds the
+                "capacity" of the Queue. A Queue becomes "underfull" when its size becomes
+                less than the "flowResumeCapacity".
+
+                <programlisting>
+                <![CDATA[
+<queue>
+    <name>test</name>
+    <test>
+        <exchange>amq.direct</exchange>
+        <capacity>10485760</capacity>                     <!-- set the queue capacity to 10Mb -->
+        <flowResumeCapacity>8388608</flowResumeCapacity>  <!-- set the resume capacity to 8Mb -->
+    </test>
+</queue>
+                ]]>
+                </programlisting>
+
+                The default for all queues on a virtual host can also be set
+
+                <programlisting>
+                <![CDATA[
+<virtualhosts>
+    <virtualhost>
+        <name>localhost</name>
+        <localhost>
+            <capacity>10485760</capacity>                     <!-- set the queue capacity to 10Mb -->
+            <flowResumeCapacity>8388608</flowResumeCapacity>  <!-- set the resume capacity to 8Mb -->
+        </localhost>
+    </virtualhost>
+</virtualhosts>
+                ]]>
+                </programlisting>
+
+                Where no flowResumeCapacity is set, the flowResumeCapacity is set to be equal
+                to the capacity. Where no capacity is set, capacity is defaulted to 0 meaning
+                there is no capacity limit.
+            </para>
+            <section role="h4">
+                <title>Broker Log Messages</title>
+                <para>
+                    There are four new Broker log messages that may occur if flow control through queue capacity limits is enabled.
+                    Firstly, when a capacity limited queue becomes overfull, a log message similar to the following is produced
+                </para>
+                <programlisting>
+MESSAGE [vh(/test)/qu(MyQueue)] [vh(/test)/qu(MyQueue)] QUE-1003 : Overfull : Size : 1,200 bytes, Capacity : 1,000
+                </programlisting>
+                <para>Then for each channel which becomes blocked upon the overful queue a log message similar to the following is produced:</para>
+                <programlisting>
+MESSAGE [con:2(guest@anonymous(713889609)/test)/ch:1] [con:2(guest@anonymous(713889609)/test)/ch:1] CHN-1005 : Flow Control Enforced (Queue MyQueue)
+                </programlisting>
+                <para>When enough messages have been consumed from the queue that it becomes underfull, then the following log is generated: </para>
+                <programlisting>
+MESSAGE [vh(/test)/qu(MyQueue)] [vh(/test)/qu(MyQueue)] QUE-1004 : Underfull : Size : 600 bytes, Resume Capacity : 800
+                </programlisting>
+                <para>And for every channel which becomes unblocked you will see a message similar to: </para>
+                <programlisting>
+MESSAGE [con:2(guest@anonymous(713889609)/test)/ch:1] [con:2(guest@anonymous(713889609)/test)/ch:1] CHN-1006 : Flow Control Removed
+                </programlisting>
+                <para>Obviously the details of connection, virtual host, queue, size, capacity, etc would depend on the configuration in use.</para>
+
+
+            </section><!-- Broker Log Messages -->
+        </section><!-- Configuring a Queue to use flow control -->
+
+        <section role="h3">
+            <title>Disk quota-based flow control</title>
+            <para>
+                Since version 0.18 of Qpid Broker, flow control can be triggered when a
+                configured disk quota is exceeded. This is supported by the BDB and Derby message stores.
+            </para>
+            <para>
+                This functionality blocks all producers on reaching the disk overflow limit. When consumers
+                consume the messages, causing disk space usage to falls below the underflow limit, the
+                producers are unblocked and continue working as normal.
+            </para>
+            <para>
+                Two limits can be configured:
+            </para>
+            <para>
+                overfull limit - the maximum space on disk (in bytes) which can be used by store.
+            </para>
+            <para>
+                underfull limit - when the space on disk drops below this limit, producers are allowed to resume publishing.
+            </para>
+            <para>
+                An example of quota configuration for the BDB message store is provided below.
+            </para>
+            <programlisting>
+            <![CDATA[
+<store>
+   <class>org.apache.qpid.server.store.berkeleydb.BDBMessageStore</class>
+   <environment-path>${work}/bdbstore/test</environment-path>
+   <overfull-size>50000000</overfull-size>
+   <underfull-size>45000000</underfull-size>
+</store>
+            ]]>
+            </programlisting>
+            <para>
+                The disk quota functionality is based on "best effort" principle. This means the broker
+                cannot guarantee that the disk space limit will not be exceeded. If several concurrent
+                transactions are started before the limit is reached, which collectively cause the limit
+                to be exceeded, the broker may allow all of them to be committed.
+            </para>
+
+            <section role="h4">
+                <title>Broker Log Messages for quota flow control</title>
+                <para>
+                    There are 2 new broker log messages that may occur if flow control through disk quota limits is enabled.
+                    When the virtual host is blocked due to exceeding of the disk quota limit the following message
+                    appears in the broker log
+                    <programlisting>
+[vh(/test)/ms(BDBMessageStore)] MST-1008 : Store overfull, flow control will be enforced
+                    </programlisting>
+                    When virtual host is unblocked after cleaning the disk space the following message appears in the broker log
+                    <programlisting>
+[vh(/test)/ms(BDBMessageStore)] MST-1009 : Store overfull condition cleared
+                    </programlisting>
+                </para>
+            </section>
+        </section><!-- Disk quota-based flow control -->
+    </section><!-- Server configuration -->
+
+
+    <section role="h2" id="QpidProducerFlowControlClientImpact">
+        <title>Client impact and configuration</title>
+        <para>
+            If a producer sends to a queue which is overfull, the broker will respond by
+            instructing the client not to send any more messages. The impact of this is
+            that any future attempts to send will block until the broker rescinds the flow control order.
+        </para>
+        <para>
+            While blocking the client will periodically log the fact that it is blocked waiting on flow control.
+        </para>
+        <programlisting>
+WARN   Message send delayed by 5s due to broker enforced flow control
+WARN   Message send delayed by 10s due to broker enforced flow control
+        </programlisting>
+        <para>
+            After a set period the send will timeout and throw a JMSException to the calling code.
+        </para>
+        <para>
+            If such a JMSException is thrown, the message will not be sent to the broker,
+            however the underlying Session may still be active - in particular if the
+            Session is transactional then the current transaction will not be automatically
+            rolled back. Users may choose to either attempt to resend the message, or to
+            roll back any transactional work and close the Session.
+        </para>
+        <para>
+            Both the timeout delay and the periodicity of the warning messages can be set
+            using Java system properties.
+        </para>
+        <para>
+            The amount of time (in milliseconds) to wait before timing out
+            is controlled by the property qpid.flow_control_wait_failure.
+        </para>
+        <para>
+            The frequency at which the log message informing that the producer is flow
+            controlled is sent is controlled by the system property qpid.flow_control_wait_notify_period.
+        </para>
+        <para>
+            Adding the following to the command line to start the client would result in a timeout of one minute,
+            with warning messages every ten seconds:
+        </para>
+        <programlisting>
+-Dqpid.flow_control_wait_failure=60000
+-Dqpid.flow_control_wait_notify_period=10000
+        </programlisting>
+        <section role="h3">
+            <title>Older Clients</title>
+            <para>
+                The flow control feature was first added to the Java broker/client in the 0.6 release. If an older client connects to the broker then the flow control commands will be ignored by it and it will not be blocked. So to fully benefit from this feature both Client and Broker need to be at least version 0.6.
+            </para>
+        </section>
+    </section> <!-- Client impact and configuration -->
+</section>
diff --git a/doc/book/src/java-broker/Qpid-Java-FAQ.xml b/doc/book/src/java-broker/Qpid-Java-FAQ.xml
index 845c343350..2940e58138 100644
--- a/doc/book/src/java-broker/Qpid-Java-FAQ.xml
+++ b/doc/book/src/java-broker/Qpid-Java-FAQ.xml
@@ -742,15 +742,9 @@ amqj.logging.level
           </title>
 
 	  <para>
-            There are two possibilities here:
-          </para><para>
-            1) The management console can be used to interogate an active
+            The management console can be used to interogate an active
             broker and browse the contents of a queue.See the <xref linkend="qpid_Qpid-JMX-Management-Console"/>
             page for further details.
-          </para><para>
-            2) The <xref linkend="qpid_MessageStore-Tool"/> can be used to inspect
-            the contents of a persistent message store. Note: this can
-            currently only be used when the broker is offline.
           </para>
 <!--h3--></section>
 
@@ -773,13 +767,14 @@ amqj.logging.level
           </title>
 
 	  <para>
-            The Java broker does not currently implement producer flow
-            control. Publishes are currently asynchronous, so there is no
-            ability to rate limit this automatically. While this is something
-            which will be addressed in the future, it is currently up to
-            applications to ensure that they do not publish faster than the
-            messages are being consumed for signifcant periods of time.
-          </para>
+            Switch on producer flow control to prevent temporary spikes in
+            message production over-filling the broker.
+
+            Of course, if the long-term rate of message production exceeds
+            the rate of message
+            consumption then that is an architectural problem that can only
+            be temporarily mitigated by producer flow control.
+	  </para>
 <!--h3--></section>
 
 	  <section role="h3" id="QpidJavaFAQ-ThebrokerkeepsthrowinganOutOfMemoryexception-3F"><title>
diff --git a/doc/book/src/java-broker/Use-Priority-Queues.xml b/doc/book/src/java-broker/Use-Priority-Queues.xml
deleted file mode 100644
index 466d958d43..0000000000
--- a/doc/book/src/java-broker/Use-Priority-Queues.xml
+++ /dev/null
@@ -1,138 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?>
-<!--
- 
- Licensed to the Apache Software Foundation (ASF) under one
- or more contributor license agreements.  See the NOTICE file
- distributed with this work for additional information
- regarding copyright ownership.  The ASF licenses this file
- to you under the Apache License, Version 2.0 (the
- "License"); you may not use this file except in compliance
- with the License.  You may obtain a copy of the License at
- 
-   http://www.apache.org/licenses/LICENSE-2.0
- 
- Unless required by applicable law or agreed to in writing,
- software distributed under the License is distributed on an
- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
- KIND, either express or implied.  See the License for the
- specific language governing permissions and limitations
- under the License.
- 
--->
-
-<section id="Priority-Queues"><title>
-      Use Priority Queues
-    </title>
-
-    
-    <section role="h2" id="UsePriorityQueues-GeneralInformation"><title>
-            General
-            Information
-          </title>
-	  <para>
-            The Qpid M3 release introduces priority queues into the Java
-            Messaging Broker, supporting JMS clients who wish to make use of
-            priorities in their messaging implementation.
-          </para><para>
-            There are some key points around the use of priority queues in
-            Qpid, discussed in the sections below.
-          </para>
-<!--h2--></section>
-    <section role="h2" id="UsePriorityQueues-DefiningPriorityQueues"><title>
-            Defining
-            Priority Queues
-          </title>
-	  <para>
-            You must define a priority queue specifically before you start to
-            use it. You cannot subsequently change a queue to/from a priority
-            queue (without deleting it and re-creating).
-          </para><para>
-            You define a queue as a priority queue in the virtualhost
-            configuration file, which the broker loads at startup. When
-            defining the queue, add a &lt;priority&gt;true&lt;/priority&gt;
-            element. This will ensure that the queue has 10 distinct
-            priorities, which is the number supported by JMS.
-          </para><para>
-            If you require fewer priorities, it is possible to specify a
-            &lt;priorities&gt;int&lt;/priorities&gt; element (where int is a
-            valid integer value between 2 and 10 inclusive) which will give
-            the queue that number of distinct priorities. When messages are
-            sent to that queue, their effective priority will be calculated
-            by partitioning the priority space. If the number of effective
-            priorities is 2, then messages with priority 0-4 are treated the
-            same as "lower priority" and messages with priority 5-9 are
-            treated equivalently as "higher priority".
-          </para>
-            <programlisting>
-&lt;queue&gt;
-    &lt;name&gt;test&lt;/name&gt;
-    &lt;test&gt;
-        &lt;exchange&gt;amq.direct&lt;/exchange&gt;
-        &lt;priority&gt;true&lt;/priority&gt;
-    &lt;/test&gt;
-&lt;/queue&gt;
-</programlisting>
-<!--h2--></section>
-          
-    <section role="h2" id="UsePriorityQueues-Clientconfiguration-2Fmessagingmodelforpriorityqueues"><title>
-            Client configuration/messaging model for priority queues
-          </title>
-	  <para>
-            There are some other configuration &amp; paradigm changes which
-            are required in order that priority queues work as expected.
-          </para>
-    <section role="h3" id="UsePriorityQueues-Setlowprefetch"><title>
-            Set low pre-fetch
-          </title>
-	  <para>
-            Qpid clients receive buffered messages in batches, sized
-            according to the pre-fetch value. The current default is 5000.
-          </para><para>
-            However, if you use the default value you will probably
-            <emphasis>not</emphasis> see desirable behaviour with messages of different
-            priority. This is because a message arriving after the pre-fetch
-            buffer has filled will not leap frog messages of lower priority.
-            It will be delivered at the front of the next batch of buffered
-            messages (if that is appropriate), but this is most likely NOT
-            what you need.
-          </para><para>
-            So, you need to set the prefetch values for your client
-            (consumer) to make this sensible. To do this set the java system
-            property max_prefetch on the client environment (using -D) before
-            creating your consumer.
-          </para><para>
-            Setting the Qpid pre-fetch to 1 for your client means that
-            message priority will be honoured by the Qpid broker as it
-            dispatches messages to your client. A default for all client
-            connections can be set via a system property:
-          </para>
-            <programlisting>
--Dmax_prefetch=1
-</programlisting>
-          <para>
-            The prefetch can be also be adjusted on a per connection basis by
-            adding a 'maxprefetch' value to the <xref linkend="Connection-URL-Format"/>
-          </para>
-            <programlisting>
-amqp://guest:guest@client1/development?maxprefetch='1'&amp;brokerlist='tcp://localhost:5672'
-</programlisting>
-          <para>
-            There is a slight performance cost here if using the receive()
-            method and you could test with a slightly higher pre-fetch (up to
-            10) if the trade-off between throughput and prioritisation is
-            weighted towards the former for your application. (If you're
-            using OnMessage() then this is not a concern.)
-          </para>
-<!--h3--></section>
-    <section role="h3" id="UsePriorityQueues-Singleconsumerpersession"><title>
-            Single
-            consumer per session
-          </title>
-	  <para>
-            If you are using the receive() method to consume messages then
-            you should also only use one consumer per session with priority
-            queues. If you're using OnMessage() then this is not a concern.
-          </para>
-<!--h3--></section>
-<!--h2--></section>
-</section>
diff --git a/doc/book/src/java-broker/images/HA-2N-Key.png b/doc/book/src/java-broker/images/HA-2N-Key.png
new file mode 100644
index 0000000000..3c1856d496
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-Key.png
diff --git a/doc/book/src/java-broker/images/HA-2N-Key.svg b/doc/book/src/java-broker/images/HA-2N-Key.svg
new file mode 100644
index 0000000000..9567f385d5
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-Key.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-7 -3 435 195" width="435pt" height="195pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><filter id="Shadow" filterUnits="userSpaceOnUse"><feGaussianBlur in="SourceAlpha" result="blur" stdDeviation="3.488"/><feOffset in="blur" result="offset" dx="0" dy="4"/><feFlood flood-color="black" flood-opacity=".75" result="flood"/><feComposite in="flood" in2="offset" operator="in"/></filter><linearGradient x1="0" x2="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(52.803375 70.5) rotate(90) scale(24)"/><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="522.94922" cap-height="717.28516" ascent="770.01953" descent="-229.98047" font-weight="500"><font-face-src><font-face-name name="Helvetica"/></font-face-src></font-face><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(127.99037 28.999998) rotate(90) scale(25.5938)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(160.86441 28.999998) rotate(90) scale(25.5938)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(52.803375 107.5) rotate(90) scale(24)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_5" xl:href="#Gradient_5" gradientTransform="translate(56.678375 41.796898) scale(37.736065)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><linearGradient x1="0" x2="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b5011d"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_6" gradientTransform="translate(287.5 113.94637) rotate(90) scale(26.035202)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_7" xl:href="#Gradient_7" gradientTransform="translate(280 41) scale(26.907248)"/><font-face font-family="Arial Unicode MS" font-size="36" panose-1="2 11 6 4 2 2 2 2 2 4" units-per-em="1000" underline-position="-100.097656" underline-thickness="49.804688" slope="0" x-height="529.78516" cap-height="728.02734" ascent="1068.84766" descent="-270.9961" font-weight="500"><font-face-src><font-face-name name="ArialUnicodeMS"/></font-face-src></font-face></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 7</title><g><title>Layer 1</title><g><use xl:href="#id240_Graphic" filter="url(#Shadow)"/><use xl:href="#id239_Graphic" filter="url(#Shadow)"/><use xl:href="#id238_Graphic" filter="url(#Shadow)"/><use xl:href="#id237_Graphic" filter="url(#Shadow)"/><use xl:href="#id236_Graphic" filter="url(#Shadow)"/><use xl:href="#id235_Graphic" filter="url(#Shadow)"/><use xl:href="#id229_Graphic" filter="url(#Shadow)"/><use xl:href="#id227_Graphic" filter="url(#Shadow)"/></g><g id="id240_Graphic"><rect x="13.999878" y="13" width="394.00012" height="155" fill="white"/><rect x="13.999878" y="13" width="394.00012" height="155" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><g id="id239_Graphic"><rect x="28.177876" y="70.5" width="49.251003" height="24" fill="url(#Obj_Gradient)"/><rect x="28.177876" y="70.5" width="49.251003" height="24" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(33.177876 75.5)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="1.9536247" y="11" textLength="35.34375">Broker</tspan></text></g><g id="id238_Graphic"><path d="M 115.652176 41.796898 L 121.821274 28.999998 L 134.15947 28.999998 L 140.32857 41.796898 L 134.15947 54.593796 L 121.821274 54.593796 Z" fill="url(#Obj_Gradient_2)"/><path d="M 115.652176 41.796898 L 121.821274 28.999998 L 134.15947 28.999998 L 140.32857 41.796898 L 134.15947 54.593796 L 121.821274 54.593796 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><g id="id237_Graphic"><path d="M 148.526215 41.796898 L 154.69531 28.999998 L 167.03351 28.999998 L 173.20261 41.796898 L 167.03351 54.593796 L 154.69531 54.593796 Z" fill="url(#Obj_Gradient_3)"/><path d="M 148.526215 41.796898 L 154.69531 28.999998 L 167.03351 28.999998 L 173.20261 41.796898 L 167.03351 54.593796 L 154.69531 54.593796 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><g id="id236_Graphic"><path d="M 32.853378 107.5 L 72.753372 107.5 C 77.472977 107.5 81.303375 112.876 81.303375 119.5 C 81.303375 126.124 77.472977 131.5 72.753372 131.5 L 32.853378 131.5 C 28.133776 131.5 24.303375 126.124 24.303375 119.5 C 24.303375 112.876 28.133776 107.5 32.853378 107.5" fill="url(#Obj_Gradient_4)"/><path d="M 32.853378 107.5 L 72.753372 107.5 C 77.472977 107.5 81.303375 112.876 81.303375 119.5 C 81.303375 126.124 77.472977 131.5 72.753372 131.5 L 32.853378 131.5 C 28.133776 131.5 24.303375 126.124 24.303375 119.5 C 24.303375 112.876 28.133776 107.5 32.853378 107.5" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(35.003376 112.5)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="2.4601574" y="11" textLength="30.679688">Client</tspan></text></g><g id="id235_Graphic"><path d="M 30.434645 43.17154 C 18.959625 41.796898 23.535576 30.224663 41.840797 32.199223 C 43.539116 28.35017 64.825623 28.974915 64.686462 32.199223 C 78.033752 28.075294 95.09079 36.298325 83.649857 40.422256 C 97.37842 42.421642 83.476616 53.194073 72.209625 51.394573 C 71.307922 54.39391 51.165936 55.443512 49.398033 51.394573 C 37.992596 55.718647 14.210434 49.070145 30.434645 43.17154 Z" fill="url(#Obj_Gradient_5)"/><path d="M 30.434645 43.17154 C 18.959625 41.796898 23.535576 30.224663 41.840797 32.199223 C 43.539116 28.35017 64.825623 28.974915 64.686462 32.199223 C 78.033752 28.075294 95.09079 36.298325 83.649857 40.422256 C 97.37842 42.421642 83.476616 53.194073 72.209625 51.394573 C 71.307922 54.39391 51.165936 55.443512 49.398033 51.394573 C 37.992596 55.718647 14.210434 49.070145 30.434645 43.17154 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(36.828377 34.796898)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x=".84511757" y="11" textLength="38.009766">Cluster</tspan></text></g><text transform="translate(194.40121 29)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="8.0039062">V</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="7.7929688" y="11" textLength="26.009766">irtual</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="25" textLength="22.68164">host</tspan></text><line x1="115.65217" y1="84" x2="162.7522" y2="84" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><line x1="114.77717" y1="114.406006" x2="155.878815" y2="114.99225" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><text transform="translate(181.40121 70.5)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="59.367188">Replication</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="25" textLength="36.673828">stream</tspan></text><text transform="translate(181.90121 93.906006)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="25" textLength="30.679688">Client</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="39" textLength="58.04297">connection</tspan></text><g id="id229_Graphic"><path d="M 257.6145 123.341385 L 303.64609 123.13183 C 303.64609 123.13183 305.97821 110.84691 310.80124 114.69145 C 315.62408 118.53591 319 121.44636 319 121.44636 L 315.92612 122.44486 L 307.22827 122.069695 L 307.22821 132.69258 L 315.47678 133.06787 C 315.47678 133.06787 315.92618 136.46542 315.92618 136.78604 C 315.92618 137.10628 311.77792 139.65259 309.63474 139.972885 C 307.49097 140.29318 304.06018 131.63016 304.06018 131.63039 C 304.06018 131.63039 257.61444 130.06921 257.48056 130.06921 C 257.34662 130.06921 256.13666 132.15001 256.00693 126.86558 C 255.87709 121.581276 257.6145 123.341385 257.6145 123.341385 Z" fill="url(#Obj_Gradient_6)"/><path d="M 257.6145 123.341385 L 303.64609 123.13183 C 303.64609 123.13183 305.97821 110.84691 310.80124 114.69145 C 315.62408 118.53591 319 121.44636 319 121.44636 L 315.92612 122.44486 L 307.22827 122.069695 L 307.22821 132.69258 L 315.47678 133.06787 C 315.47678 133.06787 315.92618 136.46542 315.92618 136.78604 C 315.92618 137.10628 311.77792 139.65259 309.63474 139.972885 C 307.49097 140.29318 304.06018 131.63016 304.06018 131.63039 C 304.06018 131.63039 257.61444 130.06921 257.48056 130.06921 C 257.34662 130.06921 256.13666 132.15001 256.00693 126.86558 C 255.87709 121.581276 257.6145 123.341385 257.6145 123.341385 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><text transform="translate(334.09879 119.963974)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="35.34961">Repair</tspan></text><g id="id227_Graphic"><path d="M 260 32.310394 L 275.19995 42.241386 L 263.13727 54.738464 L 267.99988 59 L 281.17648 45.481422 L 295.20001 54.655197 L 300 49.68975 L 284.79993 39.137974 L 297.59991 27.965553 L 290.39993 23 L 279.19989 35.413807 L 266.4 25.482777 Z" fill="url(#Obj_Gradient_7)"/><path d="M 260 32.310394 L 275.19995 42.241386 L 263.13727 54.738464 L 267.99988 59 L 281.17648 45.481422 L 295.20001 54.655197 L 300 49.68975 L 284.79993 39.137974 L 297.59991 27.965553 L 290.39993 23 L 279.19989 35.413807 L 266.4 25.482777 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g><text transform="translate(329.02448 34)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="26.677734">Fault</tspan></text><text transform="translate(264 59.5)" fill="#262626"><tspan font-family="Arial Unicode MS" font-size="36" font-weight="500" fill="#262626" x="0" y="38" textLength="36">♛</tspan></text><text transform="translate(320.52448 72)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="11" textLength="54.035156">Designate</tspan><tspan font-family="Helvetica" font-size="12" font-weight="500" x="0" y="25" textLength="41.332031">Primary</tspan></text></g></g></svg>
diff --git a/doc/book/src/java-broker/images/HA-2N-MasterFail.png b/doc/book/src/java-broker/images/HA-2N-MasterFail.png
new file mode 100644
index 0000000000..b839d291b9
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-MasterFail.png
diff --git a/doc/book/src/java-broker/images/HA-2N-MasterFail.svg b/doc/book/src/java-broker/images/HA-2N-MasterFail.svg
new file mode 100644
index 0000000000..35b2c643ff
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-MasterFail.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-20 -16 600 871" width="50pc" height="871pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(200.35501 136.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(212.855 146) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(212.85451 162.5) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(73.18171 111) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(195.355 59.999996) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(195.35451 78) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><radialGradient id="Obj_Gradient_7" xl:href="#Gradient" gradientTransform="translate(470.5 133.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_8" xl:href="#Gradient_2" gradientTransform="translate(488 141.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaa3a7"/></linearGradient><linearGradient id="Obj_Gradient_9" xl:href="#Gradient_6" gradientTransform="translate(487.9995 158) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_10" xl:href="#Gradient_4" gradientTransform="translate(350.21 85) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_11" xl:href="#Gradient_2" gradientTransform="translate(470.5 55.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaaaa3"/></linearGradient><linearGradient id="Obj_Gradient_12" xl:href="#Gradient_7" gradientTransform="translate(470.4995 73.5) rotate(90) scale(36)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_8" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_13" xl:href="#Gradient_8" gradientTransform="translate(488 176) scale(38.183766)"/><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="532.22656" cap-height="719.72656" ascent="770.01953" descent="-229.98047" font-weight="bold"><font-face-src><font-face-name name="Helvetica-Bold"/></font-face-src></font-face><radialGradient id="Obj_Gradient_14" xl:href="#Gradient" gradientTransform="translate(474.08 415.255) scale(145.18695)"/><linearGradient id="Obj_Gradient_15" xl:href="#Gradient_2" gradientTransform="translate(491.58 423.255) rotate(90) scale(69)"/><linearGradient id="Obj_Gradient_16" xl:href="#Gradient_6" gradientTransform="translate(491.5795 439.755) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_17" xl:href="#Gradient_4" gradientTransform="translate(353.79 366.755) rotate(90) scale(24.00003)"/><linearGradient id="Obj_Gradient_18" xl:href="#Gradient_2" gradientTransform="translate(474.08 337.255) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_9" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#21aa1a"/></linearGradient><linearGradient id="Obj_Gradient_19" xl:href="#Gradient_9" gradientTransform="translate(474.0795 355.255) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_20" xl:href="#Gradient" gradientTransform="translate(192.79001 696.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_21" xl:href="#Gradient_2" gradientTransform="translate(210.28999 704.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_10" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa1f43"/></linearGradient><linearGradient id="Obj_Gradient_22" xl:href="#Gradient_10" gradientTransform="translate(210.2895 721) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_23" xl:href="#Gradient_4" gradientTransform="translate(72.5 648) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_24" xl:href="#Gradient_2" gradientTransform="translate(192.78999 618.5) rotate(90) scale(69)"/><linearGradient id="Obj_Gradient_25" xl:href="#Gradient_9" gradientTransform="translate(192.7895 636.5) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_26" xl:href="#Gradient" gradientTransform="translate(166.45009 415.255) scale(145.18695)"/><linearGradient id="Obj_Gradient_27" xl:href="#Gradient_2" gradientTransform="translate(183.95009 423.255) rotate(90) scale(69)"/><linearGradient id="Obj_Gradient_28" xl:href="#Gradient_6" gradientTransform="translate(183.9496 439.755) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_29" xl:href="#Gradient_4" gradientTransform="translate(72.5 372.755) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_30" xl:href="#Gradient_2" gradientTransform="translate(180.08009 343.255) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_11" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#a9a5a6"/></linearGradient><linearGradient id="Obj_Gradient_31" xl:href="#Gradient_11" gradientTransform="translate(180.0796 361.255) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_32" xl:href="#Gradient_8" gradientTransform="translate(183.95009 457.755) scale(38.183766)"/><font-face font-family="Arial Unicode MS" font-size="36" panose-1="2 11 6 4 2 2 2 2 2 4" units-per-em="1000" underline-position="-100.097656" underline-thickness="49.804688" slope="0" x-height="529.78516" cap-height="728.02734" ascent="1068.84766" descent="-270.9961" font-weight="500"><font-face-src><font-face-name name="ArialUnicodeMS"/></font-face-src></font-face></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 2</title><g><title>Layer 1</title><rect x="0" y="0" width="278" height="273" fill="white"/><rect x="0" y="0" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" fill="url(#Obj_Gradient)"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="179.355" y="146" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="179.355" y="146" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" fill="url(#Obj_Gradient_3)"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62.68171 111 L 83.68171 111 C 86.16571 111 88.18171 116.376 88.18171 123 C 88.18171 129.62399 86.16571 135 83.68171 135 L 62.68171 135 C 60.197708 135 58.18171 129.62399 58.18171 123 C 58.18171 116.376 60.197708 111 62.68171 111" fill="url(#Obj_Gradient_4)"/><path d="M 62.68171 111 L 83.68171 111 C 86.16571 111 88.18171 116.376 88.18171 123 C 88.18171 129.62399 86.16571 135 83.68171 135 L 62.68171 135 C 60.197708 135 58.18171 129.62399 58.18171 123 C 58.18171 116.376 60.197708 111 62.68171 111" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="161.855" y="60" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="161.855" y="60" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" fill="url(#Obj_Gradient_6)"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 74.355003 137 C 80.38701 150.765915 72.80871 171.11536 92.452835 178.3019 C 109.12348 184.40063 145.40762 181.02281 176.33522 180.23158" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 222.38084 162.05576 C 227.03842 153.03807 238.59448 143.85422 236.355 135 C 234.66139 128.303986 225.07715 121.79347 216.5538 115.25858" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="281.29001" y="0" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="0" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 416.53403 146.98122 C 392.9375 133.5 402.3472 20.010353 439.98892 39.375 C 443.48123 1.6271057 487.2535 7.754013 486.96735 39.375 C 514.41388 -1.06863403 549.4889 79.575165 525.96246 120.01879 C 554.19305 139.62691 525.60626 245.27281 502.4375 227.625 C 500.5833 257.03967 459.16455 267.33319 455.52917 227.625 C 432.07571 270.03143 383.17157 204.82918 416.53403 146.98122 Z" fill="url(#Obj_Gradient_7)"/><path d="M 416.53403 146.98122 C 392.9375 133.5 402.3472 20.010353 439.98892 39.375 C 443.48123 1.6271057 487.2535 7.754013 486.96735 39.375 C 514.41388 -1.06863403 549.4889 79.575165 525.96246 120.01879 C 554.19305 139.62691 525.60626 245.27281 502.4375 227.625 C 500.5833 257.03967 459.16455 267.33319 455.52917 227.625 C 432.07571 270.03143 383.17157 204.82918 416.53403 146.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="454.5" y="141.5" width="67" height="69" fill="url(#Obj_Gradient_8)"/><rect x="454.5" y="141.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 467.669 176 L 477.83426 158 L 498.16476 158 L 508.33002 176 L 498.16476 194 L 477.83426 194 Z" fill="url(#Obj_Gradient_9)"/><path d="M 467.669 176 L 477.83426 158 L 498.16476 158 L 508.33002 176 L 498.16476 194 L 477.83426 194 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 339.71 85 L 360.71 85 C 363.194 85 365.21 90.376 365.21 97 C 365.21 103.624 363.194 109 360.71 109 L 339.71 109 C 337.22598 109 335.21 103.624 335.21 97 C 335.21 90.376 337.22598 85 339.71 85" fill="url(#Obj_Gradient_10)"/><path d="M 339.71 85 L 360.71 85 C 363.194 85 365.21 90.376 365.21 97 C 365.21 103.624 363.194 109 360.71 109 L 339.71 109 C 337.22598 109 335.21 103.624 335.21 97 C 335.21 90.376 337.22598 85 339.71 85" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="437" y="55.5" width="67" height="69" fill="url(#Obj_Gradient_11)"/><rect x="437" y="55.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 450.169 91.5 L 460.33426 73.5 L 480.66476 73.5 L 490.83002 91.5 L 480.66476 109.5 L 460.33426 109.5 Z" fill="url(#Obj_Gradient_12)"/><path d="M 450.169 91.5 L 460.33426 73.5 L 480.66476 73.5 L 490.83002 91.5 L 480.66476 109.5 L 460.33426 109.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 461 162.96559 L 481.51996 177.86208 L 465.23532 196.6077 L 471.79984 203 L 489.58826 182.72214 L 508.52 196.48279 L 515 189.03462 L 494.4799 173.20695 L 511.7599 156.44833 L 502.0399 149 L 486.91986 167.62071 L 469.64001 152.72417 Z" fill="url(#Obj_Gradient_13)"/><path d="M 461 162.96559 L 481.51996 177.86208 L 465.23532 196.6077 L 471.79984 203 L 489.58826 182.72214 L 508.52 196.48279 L 515 189.03462 L 494.4799 173.20695 L 511.7599 156.44833 L 502.0399 149 L 486.91986 167.62071 L 469.64001 152.72417 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(5.8400302 4)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><rect x="281.29001" y="278.755" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="278.755" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 420.11401 428.7362 C 396.51749 415.255 405.92719 301.76535 443.5689 321.13 C 447.06122 283.38211 490.8335 289.50903 490.54733 321.13 C 517.9939 280.68637 553.0689 361.33017 529.54248 401.7738 C 557.773 421.38193 529.18622 527.02783 506.0175 509.38 C 504.1633 538.79468 462.74454 549.0882 459.10913 509.38 C 435.6557 551.78644 386.75156 486.58417 420.11401 428.7362 Z" fill="url(#Obj_Gradient_14)"/><path d="M 420.11401 428.7362 C 396.51749 415.255 405.92719 301.76535 443.5689 321.13 C 447.06122 283.38211 490.8335 289.50903 490.54733 321.13 C 517.9939 280.68637 553.0689 361.33017 529.54248 401.7738 C 557.773 421.38193 529.18622 527.02783 506.0175 509.38 C 504.1633 538.79468 462.74454 549.0882 459.10913 509.38 C 435.6557 551.78644 386.75156 486.58417 420.11401 428.7362 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="458.08" y="423.255" width="66.99997" height="69" fill="url(#Obj_Gradient_15)"/><rect x="458.08" y="423.255" width="66.99997" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 471.249 457.755 L 481.41425 439.755 L 501.74475 439.755 L 511.91 457.755 L 501.74475 475.755 L 481.41425 475.755 Z" fill="url(#Obj_Gradient_16)"/><path d="M 471.249 457.755 L 481.41425 439.755 L 501.74475 439.755 L 511.91 457.755 L 501.74475 475.755 L 481.41425 475.755 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 343.29 366.755 L 364.29 366.755 C 366.77402 366.755 368.79 372.13101 368.79 378.755 C 368.79 385.379 366.77402 390.755 364.29 390.755 L 343.29 390.755 C 340.806 390.755 338.79 385.379 338.79 378.755 C 338.79 372.13101 340.806 366.755 343.29 366.755" fill="url(#Obj_Gradient_17)"/><path d="M 343.29 366.755 L 364.29 366.755 C 366.77402 366.755 368.79 372.13101 368.79 378.755 C 368.79 385.379 366.77402 390.755 364.29 390.755 L 343.29 390.755 C 340.806 390.755 338.79 385.379 338.79 378.755 C 338.79 372.13101 340.806 366.755 343.29 366.755" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="440.57999" y="337.255" width="67" height="69" fill="url(#Obj_Gradient_18)"/><rect x="440.57999" y="337.255" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 453.749 373.255 L 463.91425 355.255 L 484.24475 355.255 L 494.41 373.255 L 484.24475 391.255 L 463.91425 391.255 Z" fill="url(#Obj_Gradient_19)"/><path d="M 453.749 373.255 L 463.91425 355.255 L 484.24475 355.255 L 494.41 373.255 L 484.24475 391.255 L 463.91425 391.255 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 365.46664 366.5985 C 373.74026 357.98486 375.04666 340.75385 390.29 340.755 C 402.3567 340.75592 423.16241 351.55505 442.3288 360.32996" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><rect x="0" y="558" width="278" height="273" fill="white"/><rect x="0" y="558" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 138.824036 709.9812 C 115.22751 696.5 124.63721 583.01038 162.27893 602.375 C 165.77126 564.62708 209.5435 570.75403 209.25735 602.375 C 236.70389 561.9314 271.77893 642.57513 248.25249 683.0188 C 276.48303 702.6269 247.89624 808.27283 224.72751 790.625 C 222.8733 820.03967 181.45457 830.3332 177.81917 790.625 C 154.36572 833.03143 105.46157 767.82916 138.824036 709.9812 Z" fill="url(#Obj_Gradient_20)"/><path d="M 138.824036 709.9812 C 115.22751 696.5 124.63721 583.01038 162.27893 602.375 C 165.77126 564.62708 209.5435 570.75403 209.25735 602.375 C 236.70389 561.9314 271.77893 642.57513 248.25249 683.0188 C 276.48303 702.6269 247.89624 808.27283 224.72751 790.625 C 222.8733 820.03967 181.45457 830.3332 177.81917 790.625 C 154.36572 833.03143 105.46157 767.82916 138.824036 709.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="176.78999" y="704.5" width="67" height="69" fill="url(#Obj_Gradient_21)"/><rect x="176.78999" y="704.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 189.959 739 L 200.12425 721 L 220.45476 721 L 230.62001 739 L 220.45476 757 L 200.12425 757 Z" fill="url(#Obj_Gradient_22)"/><path d="M 189.959 739 L 200.12425 721 L 220.45476 721 L 230.62001 739 L 220.45476 757 L 200.12425 757 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62 648 L 83 648 C 85.484 648 87.5 653.37598 87.5 660 C 87.5 666.62402 85.484 672 83 672 L 62 672 C 59.516 672 57.5 666.62402 57.5 660 C 57.5 653.37598 59.516 648 62 648" fill="url(#Obj_Gradient_23)"/><path d="M 62 648 L 83 648 C 85.484 648 87.5 653.37598 87.5 660 C 87.5 666.62402 85.484 672 83 672 L 62 672 C 59.516 672 57.5 666.62402 57.5 660 C 57.5 653.37598 59.516 648 62 648" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="159.28999" y="618.5" width="67" height="69" fill="url(#Obj_Gradient_24)"/><rect x="159.28999" y="618.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 172.459 654.5 L 182.62425 636.5 L 202.95476 636.5 L 213.12001 654.5 L 202.95476 672.5 L 182.62425 672.5 Z" fill="url(#Obj_Gradient_25)"/><path d="M 172.459 654.5 L 182.62425 636.5 L 202.95476 636.5 L 213.12001 654.5 L 202.95476 672.5 L 182.62425 672.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 84.176636 647.8435 C 92.450264 639.22986 93.75663 621.99884 109 622 C 121.06667 622.0009 141.87247 632.80005 161.03886 641.575" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 181.93205 672.16388 C 174.38545 684.44135 157.17926 699.40002 159.28999 709 C 160.93364 716.4756 174.29166 720.7052 185.68005 725.4799" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(286 8)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">2</tspan></text><text transform="translate(286 289.62299)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">4</tspan></text><text transform="translate(10.3773594 571.472)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">5</tspan></text><rect x="0" y="278.755" width="278" height="273" fill="white"/><rect x="0" y="278.755" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 112.48411 428.7362 C 88.88759 415.255 98.297287 301.76535 135.93901 321.13 C 139.431335 283.38211 183.20358 289.50903 182.91743 321.13 C 210.36397 280.68637 245.439 361.33017 221.91257 401.7738 C 250.14313 421.38193 221.55634 527.02783 198.38759 509.38 C 196.53339 538.79468 155.11465 549.0882 151.47925 509.38 C 128.02582 551.78644 79.12165 486.58417 112.48411 428.7362 Z" fill="url(#Obj_Gradient_26)"/><path d="M 112.48411 428.7362 C 88.88759 415.255 98.297287 301.76535 135.93901 321.13 C 139.431335 283.38211 183.20358 289.50903 182.91743 321.13 C 210.36397 280.68637 245.439 361.33017 221.91257 401.7738 C 250.14313 421.38193 221.55634 527.02783 198.38759 509.38 C 196.53339 538.79468 155.11465 549.0882 151.47925 509.38 C 128.02582 551.78644 79.12165 486.58417 112.48411 428.7362 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="150.45009" y="423.255" width="67" height="69" fill="url(#Obj_Gradient_27)"/><rect x="150.45009" y="423.255" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 163.61909 457.755 L 173.78435 439.755 L 194.11485 439.755 L 204.2801 457.755 L 194.11485 475.755 L 173.78435 475.755 Z" fill="url(#Obj_Gradient_28)"/><path d="M 163.61909 457.755 L 173.78435 439.755 L 194.11485 439.755 L 204.2801 457.755 L 194.11485 475.755 L 173.78435 475.755 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62 372.755 L 83 372.755 C 85.484 372.755 87.5 378.13101 87.5 384.755 C 87.5 391.379 85.484 396.755 83 396.755 L 62 396.755 C 59.516 396.755 57.5 391.379 57.5 384.755 C 57.5 378.13101 59.516 372.755 62 372.755" fill="url(#Obj_Gradient_29)"/><path d="M 62 372.755 L 83 372.755 C 85.484 372.755 87.5 378.13101 87.5 384.755 C 87.5 391.379 85.484 396.755 83 396.755 L 62 396.755 C 59.516 396.755 57.5 391.379 57.5 384.755 C 57.5 378.13101 59.516 372.755 62 372.755" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="146.58009" y="343.255" width="67" height="69" fill="url(#Obj_Gradient_30)"/><rect x="146.58009" y="343.255" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 159.7491 379.255 L 169.91435 361.255 L 190.24486 361.255 L 200.41011 379.255 L 190.24486 397.255 L 169.91435 397.255 Z" fill="url(#Obj_Gradient_31)"/><path d="M 159.7491 379.255 L 169.91435 361.255 L 190.24486 361.255 L 200.41011 379.255 L 190.24486 397.255 L 169.91435 397.255 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 156.95009 444.72058 L 177.47003 459.6171 L 161.18541 478.3627 L 167.74992 484.755 L 185.53835 464.47714 L 204.47009 478.2378 L 210.95009 470.78964 L 190.42998 454.96198 L 207.70998 438.20334 L 197.98997 430.755 L 182.86996 449.3757 L 165.59009 434.47916 Z" fill="url(#Obj_Gradient_32)"/><path d="M 156.95009 444.72058 L 177.47003 459.6171 L 161.18541 478.3627 L 167.74992 484.755 L 185.53835 464.47714 L 204.47009 478.2378 L 210.95009 470.78964 L 190.42998 454.96198 L 207.70998 438.20334 L 197.98997 430.755 L 182.86996 449.3757 L 165.59009 434.47916 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(210.37738 346.23645)" fill="#262626"><tspan font-family="Arial Unicode MS" font-size="36" font-weight="500" fill="#262626" x="0" y="38" textLength="36">♛</tspan></text><text transform="translate(5.8400302 287.736)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text></g></g></svg>
diff --git a/doc/book/src/java-broker/images/HA-2N-NetworkPartition.png b/doc/book/src/java-broker/images/HA-2N-NetworkPartition.png
new file mode 100644
index 0000000000..cd4a196924
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-NetworkPartition.png
diff --git a/doc/book/src/java-broker/images/HA-2N-NetworkPartition.svg b/doc/book/src/java-broker/images/HA-2N-NetworkPartition.svg
new file mode 100644
index 0000000000..375d88a7db
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-NetworkPartition.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-20 -12 600 596" width="50pc" height="596pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(474.08 425.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(466.66 343.723) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa1f43"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(466.65952 360.223) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(353.79 377) rotate(90) scale(24.00003)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(474.08 438) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#21aa1a"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(474.0795 456) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><radialGradient id="Obj_Gradient_7" xl:href="#Gradient" gradientTransform="translate(470.5 137.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_8" xl:href="#Gradient_2" gradientTransform="translate(488 145.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6ead6a"/></linearGradient><linearGradient id="Obj_Gradient_9" xl:href="#Gradient_6" gradientTransform="translate(487.9995 162) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_10" xl:href="#Gradient_4" gradientTransform="translate(350.21 89) rotate(90) scale(24)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_11" xl:href="#Gradient_7" gradientTransform="translate(487.17532 136.54291) scale(20.071926)"/><linearGradient id="Obj_Gradient_12" xl:href="#Gradient_2" gradientTransform="translate(466.66 55.445) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_8" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b6b4b5"/></linearGradient><linearGradient id="Obj_Gradient_13" xl:href="#Gradient_8" gradientTransform="translate(466.65952 71.945) rotate(90) scale(36)"/><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="532.22656" cap-height="719.72656" ascent="770.01953" descent="-229.98047" font-weight="bold"><font-face-src><font-face-name name="Helvetica-Bold"/></font-face-src></font-face><radialGradient id="Obj_Gradient_14" xl:href="#Gradient" gradientTransform="translate(200.35501 140.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_15" xl:href="#Gradient_2" gradientTransform="translate(212.855 150) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_9" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_16" xl:href="#Gradient_9" gradientTransform="translate(212.85451 166.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_17" xl:href="#Gradient_4" gradientTransform="translate(80.355003 115) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_18" xl:href="#Gradient_2" gradientTransform="translate(195.355 63.999996) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_10" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_19" xl:href="#Gradient_10" gradientTransform="translate(195.35451 82) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_20" xl:href="#Gradient" gradientTransform="translate(180.29001 423.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_21" xl:href="#Gradient_2" gradientTransform="translate(172.37 346) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_11" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaa3a7"/></linearGradient><linearGradient id="Obj_Gradient_22" xl:href="#Gradient_11" gradientTransform="translate(172.36951 360.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_23" xl:href="#Gradient_4" gradientTransform="translate(72.5 379) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_24" xl:href="#Gradient_2" gradientTransform="translate(180.953 435.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_12" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#66ab61"/></linearGradient><linearGradient id="Obj_Gradient_25" xl:href="#Gradient_12" gradientTransform="translate(180.95251 452) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_13" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b5011d"/></linearGradient><linearGradient id="Obj_Gradient_26" xl:href="#Gradient_13" gradientTransform="translate(169.38802 421.22614) rotate(45) scale(21.988586)"/></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 5</title><g><title>Layer 1</title><rect x="281.29001" y="287" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="287" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 420.11401 438.9812 C 396.51749 425.5 405.92719 312.01035 443.5689 331.375 C 447.06122 293.6271 490.8335 299.75403 490.54733 331.375 C 517.9939 290.93137 553.0689 371.57516 529.54248 412.0188 C 557.773 431.62692 529.18622 537.27283 506.0175 519.625 C 504.1633 549.03967 462.74454 559.3332 459.10913 519.625 C 435.6557 562.03143 386.75156 496.82916 420.11401 438.9812 Z" fill="url(#Obj_Gradient)"/><path d="M 420.11401 438.9812 C 396.51749 425.5 405.92719 312.01035 443.5689 331.375 C 447.06122 293.6271 490.8335 299.75403 490.54733 331.375 C 517.9939 290.93137 553.0689 371.57516 529.54248 412.0188 C 557.773 431.62692 529.18622 537.27283 506.0175 519.625 C 504.1633 549.03967 462.74454 559.3332 459.10913 519.625 C 435.6557 562.03143 386.75156 496.82916 420.11401 438.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="433.16" y="343.723" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="433.16" y="343.723" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 446.329 378.223 L 456.49426 360.223 L 476.82477 360.223 L 486.99002 378.223 L 476.82477 396.223 L 456.49426 396.223 Z" fill="url(#Obj_Gradient_3)"/><path d="M 446.329 378.223 L 456.49426 360.223 L 476.82477 360.223 L 486.99002 378.223 L 476.82477 396.223 L 456.49426 396.223 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 343.29 377 L 364.29 377 C 366.77402 377 368.79 382.376 368.79 389 C 368.79 395.624 366.77402 401 364.29 401 L 343.29 401 C 340.806 401 338.79 395.624 338.79 389 C 338.79 382.376 340.806 377 343.29 377" fill="url(#Obj_Gradient_4)"/><path d="M 343.29 377 L 364.29 377 C 366.77402 377 368.79 382.376 368.79 389 C 368.79 395.624 366.77402 401 364.29 401 L 343.29 401 C 340.806 401 338.79 395.624 338.79 389 C 338.79 382.376 340.806 377 343.29 377" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="440.57999" y="438" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="440.57999" y="438" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 453.749 474 L 463.91425 456 L 484.24475 456 L 494.41 474 L 484.24475 492 L 463.91425 492 Z" fill="url(#Obj_Gradient_6)"/><path d="M 453.749 474 L 463.91425 456 L 484.24475 456 L 494.41 474 L 484.24475 492 L 463.91425 492 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 358.11069 401.47244 C 364.4065 419.64648 360.82266 444.49713 377 456 C 390.09668 465.31238 416.1513 465.88107 439.42624 468.3237" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 480.27563 455.52597 C 482.5702 448.68466 488.31772 444.80447 487.16 435 C 486.26978 427.46088 481.2962 416.41544 476.94943 405.90878" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="281.29001" y="4" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="4" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 416.53403 150.98122 C 392.9375 137.5 402.3472 24.010353 439.98892 43.375 C 443.48123 5.6271057 487.2535 11.754013 486.96735 43.375 C 514.41388 2.931366 549.4889 83.575165 525.96246 124.01879 C 554.19305 143.62691 525.60626 249.27281 502.4375 231.625 C 500.5833 261.03967 459.16455 271.33319 455.52917 231.625 C 432.07571 274.03143 383.17157 208.82918 416.53403 150.98122 Z" fill="url(#Obj_Gradient_7)"/><path d="M 416.53403 150.98122 C 392.9375 137.5 402.3472 24.010353 439.98892 43.375 C 443.48123 5.6271057 487.2535 11.754013 486.96735 43.375 C 514.41388 2.931366 549.4889 83.575165 525.96246 124.01879 C 554.19305 143.62691 525.60626 249.27281 502.4375 231.625 C 500.5833 261.03967 459.16455 271.33319 455.52917 231.625 C 432.07571 274.03143 383.17157 208.82918 416.53403 150.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="454.5" y="145.5" width="67" height="69" fill="url(#Obj_Gradient_8)"/><rect x="454.5" y="145.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 467.669 180 L 477.83426 162 L 498.16476 162 L 508.33002 180 L 498.16476 198 L 477.83426 198 Z" fill="url(#Obj_Gradient_9)"/><path d="M 467.669 180 L 477.83426 162 L 498.16476 162 L 508.33002 180 L 498.16476 198 L 477.83426 198 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 339.71 89 L 360.71 89 C 363.194 89 365.21 94.376 365.21 101 C 365.21 107.624 363.194 113 360.71 113 L 339.71 113 C 337.22598 113 335.21 107.624 335.21 101 C 335.21 94.376 337.22598 89 339.71 89" fill="url(#Obj_Gradient_10)"/><path d="M 339.71 89 L 360.71 89 C 363.194 89 365.21 94.376 365.21 101 C 365.21 107.624 363.194 113 360.71 113 L 339.71 113 C 337.22598 113 335.21 107.624 335.21 101 C 335.21 94.376 337.22598 89 339.71 89" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 474.53464 129.01602 L 484.14154 137.61832 L 476.51752 148.44287 L 479.59082 152.134415 L 487.91891 140.42473 L 496.7823 148.37096 L 499.816 144.06992 L 490.20901 134.93007 L 498.2991 125.25254 L 493.74844 120.95141 L 486.66962 131.70415 L 478.57965 123.101974 Z" fill="url(#Obj_Gradient_11)"/><path d="M 474.53464 129.01602 L 484.14154 137.61832 L 476.51752 148.44287 L 479.59082 152.134415 L 487.91891 140.42473 L 496.7823 148.37096 L 499.816 144.06992 L 490.20901 134.93007 L 498.2991 125.25254 L 493.74844 120.95141 L 486.66962 131.70415 L 478.57965 123.101974 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="433.16" y="55.445" width="67" height="69" fill="url(#Obj_Gradient_12)"/><rect x="433.16" y="55.445" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 446.329 89.945 L 456.49426 71.945 L 476.82477 71.945 L 486.99002 89.945 L 476.82477 107.945 L 456.49426 107.945 Z" fill="url(#Obj_Gradient_13)"/><path d="M 446.329 89.945 L 456.49426 71.945 L 476.82477 71.945 L 486.99002 89.945 L 476.82477 107.945 L 456.49426 107.945 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 487.69427 161.50006 C 487.55954 153.3342 489.4468 145.84863 487.29 137 C 485.7971 130.87509 482.3659 124.09454 479.00916 117.33957" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(5.8400302 4)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><text transform="translate(286 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">2</tspan></text><rect x="0" y="4" width="278" height="273" fill="white"/><rect x="0" y="4" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 146.38904 153.98122 C 122.79251 140.5 132.20221 27.010353 169.84393 46.375 C 173.33626 8.6271057 217.1085 14.754013 216.82236 46.375 C 244.26889 5.931366 279.34393 86.575165 255.81749 127.01879 C 284.04803 146.62691 255.46124 252.27281 232.29251 234.625 C 230.43831 264.03967 189.01958 274.33319 185.38417 234.625 C 161.930725 277.03143 113.02657 211.82918 146.38904 153.98122 Z" fill="url(#Obj_Gradient_14)"/><path d="M 146.38904 153.98122 C 122.79251 140.5 132.20221 27.010353 169.84393 46.375 C 173.33626 8.6271057 217.1085 14.754013 216.82236 46.375 C 244.26889 5.931366 279.34393 86.575165 255.81749 127.01879 C 284.04803 146.62691 255.46124 252.27281 232.29251 234.625 C 230.43831 264.03967 189.01958 274.33319 185.38417 234.625 C 161.930725 277.03143 113.02657 211.82918 146.38904 153.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="179.355" y="150" width="67" height="69" fill="url(#Obj_Gradient_15)"/><rect x="179.355" y="150" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 192.524 184.5 L 202.68925 166.5 L 223.01976 166.5 L 233.18501 184.5 L 223.01976 202.5 L 202.68925 202.5 Z" fill="url(#Obj_Gradient_16)"/><path d="M 192.524 184.5 L 202.68925 166.5 L 223.01976 166.5 L 233.18501 184.5 L 223.01976 202.5 L 202.68925 202.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 69.855003 115 L 90.855003 115 C 93.339005 115 95.355003 120.376 95.355003 127 C 95.355003 133.62399 93.339005 139 90.855003 139 L 69.855003 139 C 67.371002 139 65.355003 133.62399 65.355003 127 C 65.355003 120.376 67.371002 115 69.855003 115" fill="url(#Obj_Gradient_17)"/><path d="M 69.855003 115 L 90.855003 115 C 93.339005 115 95.355003 120.376 95.355003 127 C 95.355003 133.62399 93.339005 139 90.855003 139 L 69.855003 139 C 67.371002 139 65.355003 133.62399 65.355003 127 C 65.355003 120.376 67.371002 115 69.855003 115" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="161.855" y="64" width="67" height="69" fill="url(#Obj_Gradient_18)"/><rect x="161.855" y="64" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 175.024 100 L 185.18925 82 L 205.51976 82 L 215.68501 100 L 205.51976 118 L 185.18925 118 Z" fill="url(#Obj_Gradient_19)"/><path d="M 175.024 100 L 185.18925 82 L 205.51976 82 L 215.68501 100 L 205.51976 118 L 185.18925 118 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 88 139 C 101.11702 162.9976 109.51097 202.49907 127.355 211 C 141.45248 217.71606 161.45502 205.08597 180.36313 196.03654" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 222.38084 166.05576 C 227.03842 157.03807 238.59448 147.85422 236.355 139 C 234.66139 132.304 225.07715 125.79347 216.5538 119.25858" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(286 293)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">4</tspan></text><text transform="translate(5.8400302 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><text transform="translate(5.840027 287)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text><rect x="0" y="287" width="278" height="273" fill="white"/><rect x="0" y="287" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 126.32403 436.9812 C 102.72751 423.5 112.13721 310.01035 149.77893 329.375 C 153.27126 291.6271 197.0435 297.75403 196.75735 329.375 C 224.20389 288.93137 259.27893 369.57516 235.75249 410.0188 C 263.98303 429.62692 235.39624 535.27283 212.22751 517.625 C 210.37331 547.03967 168.95457 557.3332 165.31917 517.625 C 141.86572 560.03143 92.96157 494.82916 126.32403 436.9812 Z" fill="url(#Obj_Gradient_20)"/><path d="M 126.32403 436.9812 C 102.72751 423.5 112.13721 310.01035 149.77893 329.375 C 153.27126 291.6271 197.0435 297.75403 196.75735 329.375 C 224.20389 288.93137 259.27893 369.57516 235.75249 410.0188 C 263.98303 429.62692 235.39624 535.27283 212.22751 517.625 C 210.37331 547.03967 168.95457 557.3332 165.31917 517.625 C 141.86572 560.03143 92.96157 494.82916 126.32403 436.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="138.869995" y="346" width="67" height="69" fill="url(#Obj_Gradient_21)"/><rect x="138.869995" y="346" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 152.039 378.5 L 162.20425 360.5 L 182.53476 360.5 L 192.70001 378.5 L 182.53476 396.5 L 162.20425 396.5 Z" fill="url(#Obj_Gradient_22)"/><path d="M 152.039 378.5 L 162.20425 360.5 L 182.53476 360.5 L 192.70001 378.5 L 182.53476 396.5 L 162.20425 396.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62 379 L 83 379 C 85.484 379 87.5 384.376 87.5 391 C 87.5 397.624 85.484 403 83 403 L 62 403 C 59.516 403 57.5 397.624 57.5 391 C 57.5 384.376 59.516 379 62 379" fill="url(#Obj_Gradient_23)"/><path d="M 62 379 L 83 379 C 85.484 379 87.5 384.376 87.5 391 C 87.5 397.624 85.484 403 83 403 L 62 403 C 59.516 403 57.5 397.624 57.5 391 C 57.5 384.376 59.516 379 62 379" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="147.453" y="435.5" width="67" height="69" fill="url(#Obj_Gradient_24)"/><rect x="147.453" y="435.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 160.62201 470 L 170.78726 452 L 191.11777 452 L 201.28302 470 L 191.11777 488 L 170.78726 488 Z" fill="url(#Obj_Gradient_25)"/><path d="M 160.62201 470 L 170.78726 452 L 191.11777 452 L 201.28302 470 L 191.11777 488 L 170.78726 488 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 181.17079 451.50006 C 181.27284 442.8504 182.34718 434.63248 181.47696 425.54852 C 180.89874 419.51263 179.46176 413.09204 178.022 406.67056" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 162.48625 439.34933 L 182.67911 418.90622 C 182.67911 418.90622 176.37192 410.54022 180.79669 410.70734 C 185.22139 410.87448 187.68919 413.07565 187.68919 413.07565 L 183.62592 416.69077 L 189.96997 423.03482 L 193.83488 419.61807 C 193.83488 419.61807 196.0623 421.44876 196.25372 421.64017 C 196.44499 421.83145 196.1347 425.18307 195.37999 426.3204 C 194.62503 427.45789 187.93712 423.79865 187.93724 423.79877 C 187.93724 423.79877 166.50412 443.3672 166.44502 443.4263 C 166.38591 443.4854 165.92685 444.32736 163.88141 442.16357 C 161.83583 439.99976 162.48625 439.34933 162.48625 439.34933 Z" fill="url(#Obj_Gradient_26)"/><path d="M 162.48625 439.34933 L 182.67911 418.90622 C 182.67911 418.90622 176.37192 410.54022 180.79669 410.70734 C 185.22139 410.87448 187.68919 413.07565 187.68919 413.07565 L 183.62592 416.69077 L 189.96997 423.03482 L 193.83488 419.61807 C 193.83488 419.61807 196.0623 421.44876 196.25372 421.64017 C 196.44499 421.83145 196.1347 425.18307 195.37999 426.3204 C 194.62503 427.45789 187.93712 423.79865 187.93724 423.79877 C 187.93724 423.79877 166.50412 443.3672 166.44502 443.4263 C 166.38591 443.4854 165.92685 444.32736 163.88141 442.16357 C 161.83583 439.99976 162.48625 439.34933 162.48625 439.34933 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 75.526443 403.48596 C 80.05397 422.16476 74.80702 448.81583 89.11038 459.52802 C 100.46647 468.0329 124.151474 466.49478 145.480865 466.87387" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><text transform="translate(10 297)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text></g></g></svg>
diff --git a/doc/book/src/java-broker/images/HA-2N-Normal.png b/doc/book/src/java-broker/images/HA-2N-Normal.png
new file mode 100644
index 0000000000..eeaad4f230
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-Normal.png
diff --git a/doc/book/src/java-broker/images/HA-2N-Normal.svg b/doc/book/src/java-broker/images/HA-2N-Normal.svg
new file mode 100644
index 0000000000..c4fac9d37a
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-Normal.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-11 -10 318 313" width="318pt" height="313pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(209.35501 142.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(221.85501 152) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(221.85452 168.5) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(71.35501 115) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(204.35501 66) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(204.35452 84) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 1</title><g><title>Layer 1</title><rect x="9" y="6" width="278" height="273" fill="white"/><rect x="9" y="6" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 155.38904 155.98122 C 131.79251 142.5 141.20221 29.010353 178.84393 48.375 C 182.33626 10.627106 226.1085 16.754013 225.82236 48.375 C 253.26889 7.931366 288.34393 88.575165 264.8175 129.01878 C 293.04803 148.62691 264.46124 254.27281 241.29251 236.625 C 239.43831 266.03967 198.01958 276.33319 194.38417 236.625 C 170.93073 279.03143 122.02657 213.82918 155.38904 155.98122 Z" fill="url(#Obj_Gradient)"/><path d="M 155.38904 155.98122 C 131.79251 142.5 141.20221 29.010353 178.84393 48.375 C 182.33626 10.627106 226.1085 16.754013 225.82236 48.375 C 253.26889 7.931366 288.34393 88.575165 264.8175 129.01878 C 293.04803 148.62691 264.46124 254.27281 241.29251 236.625 C 239.43831 266.03967 198.01958 276.33319 194.38417 236.625 C 170.93073 279.03143 122.02657 213.82918 155.38904 155.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="188.35501" y="152" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="188.35501" y="152" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 201.52402 186.5 L 211.68927 168.5 L 232.01978 168.5 L 242.18503 186.5 L 232.01978 204.5 L 211.68927 204.5 Z" fill="url(#Obj_Gradient_3)"/><path d="M 201.52402 186.5 L 211.68927 168.5 L 232.01978 168.5 L 242.18503 186.5 L 232.01978 204.5 L 211.68927 204.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 60.85501 115 L 81.85501 115 C 84.33901 115 86.35501 120.376 86.35501 127 C 86.35501 133.62399 84.33901 139 81.85501 139 L 60.85501 139 C 58.37101 139 56.35501 133.62399 56.35501 127 C 56.35501 120.376 58.37101 115 60.85501 115" fill="url(#Obj_Gradient_4)"/><path d="M 60.85501 115 L 81.85501 115 C 84.33901 115 86.35501 120.376 86.35501 127 C 86.35501 133.62399 84.33901 139 81.85501 139 L 60.85501 139 C 58.37101 139 56.35501 133.62399 56.35501 127 C 56.35501 120.376 58.37101 115 60.85501 115" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="170.85501" y="66" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="170.85501" y="66" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 184.02402 102 L 194.18927 84 L 214.51978 84 L 224.68503 102 L 214.51978 120 L 194.18927 120 Z" fill="url(#Obj_Gradient_6)"/><path d="M 184.02402 102 L 194.18927 84 L 214.51978 84 L 224.68503 102 L 214.51978 120 L 194.18927 120 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 83.35501 143 C 101.01991 166.33099 116.23704 204.83237 136.35501 213 C 152.19005 219.42882 171.06885 207.06767 189.39609 198.1017" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 231.38086 168.05576 C 236.03844 159.03807 247.59447 149.85422 245.35498 141 C 243.66136 134.304 234.07715 127.79349 225.55383 121.2586" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/></g></g></svg>
diff --git a/doc/book/src/java-broker/images/HA-2N-ReplicaFail.png b/doc/book/src/java-broker/images/HA-2N-ReplicaFail.png
new file mode 100644
index 0000000000..769fc959fc
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-ReplicaFail.png
diff --git a/doc/book/src/java-broker/images/HA-2N-ReplicaFail.svg b/doc/book/src/java-broker/images/HA-2N-ReplicaFail.svg
new file mode 100644
index 0000000000..aa872a30e3
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-ReplicaFail.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-20 -16 600 590" width="50pc" height="590pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(470.5 133.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(488 141.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6cad67"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(487.9995 158) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(350.21 85) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(470.5 55.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaaaa3"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(470.4995 73.5) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><radialGradient cx="0" cy="0" r="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_7" xl:href="#Gradient_6" gradientTransform="translate(472.145 91.5) scale(38.183766)"/><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="532.22656" cap-height="719.72656" ascent="770.01953" descent="-229.98047" font-weight="bold"><font-face-src><font-face-name name="Helvetica-Bold"/></font-face-src></font-face><radialGradient id="Obj_Gradient_8" xl:href="#Gradient" gradientTransform="translate(474.08 415.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_9" xl:href="#Gradient_2" gradientTransform="translate(466.66 333.723) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa1f43"/></linearGradient><linearGradient id="Obj_Gradient_10" xl:href="#Gradient_7" gradientTransform="translate(466.65952 350.223) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_11" xl:href="#Gradient_4" gradientTransform="translate(353.79 367) rotate(90) scale(24.00003)"/><linearGradient id="Obj_Gradient_12" xl:href="#Gradient_2" gradientTransform="translate(474.08 428) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_8" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#21aa1a"/></linearGradient><linearGradient id="Obj_Gradient_13" xl:href="#Gradient_8" gradientTransform="translate(474.0795 446) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><radialGradient id="Obj_Gradient_14" xl:href="#Gradient" gradientTransform="translate(180.29001 409.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_15" xl:href="#Gradient_2" gradientTransform="translate(172.37 332) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_9" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aaa3a7"/></linearGradient><linearGradient id="Obj_Gradient_16" xl:href="#Gradient_9" gradientTransform="translate(172.36951 346.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_17" xl:href="#Gradient_4" gradientTransform="translate(72.5 365) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_18" xl:href="#Gradient_2" gradientTransform="translate(180.953 421.5) rotate(90) scale(69)"/><linearGradient id="Obj_Gradient_19" xl:href="#Gradient_8" gradientTransform="translate(180.9525 438) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_10" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b5011d"/></linearGradient><linearGradient id="Obj_Gradient_20" xl:href="#Gradient_10" gradientTransform="translate(134.69392 361.18848) rotate(45) scale(23.213103)"/><radialGradient id="Obj_Gradient_21" xl:href="#Gradient" gradientTransform="translate(200.35501 136.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_22" xl:href="#Gradient_2" gradientTransform="translate(212.855 146) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_11" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_23" xl:href="#Gradient_11" gradientTransform="translate(212.85451 162.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_24" xl:href="#Gradient_4" gradientTransform="translate(80.355003 111) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_25" xl:href="#Gradient_2" gradientTransform="translate(195.355 59.999996) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_12" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_26" xl:href="#Gradient_12" gradientTransform="translate(195.35451 78) rotate(90) scale(36)"/></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 4</title><g><title>Layer 1</title><rect x="281.29001" y="0" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="0" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 416.53403 146.98122 C 392.9375 133.5 402.3472 20.010353 439.98892 39.375 C 443.48123 1.6271057 487.2535 7.754013 486.96735 39.375 C 514.41388 -1.06863403 549.4889 79.575165 525.96246 120.01879 C 554.19305 139.62691 525.60626 245.27281 502.4375 227.625 C 500.5833 257.03967 459.16455 267.33319 455.52917 227.625 C 432.07571 270.03143 383.17157 204.82918 416.53403 146.98122 Z" fill="url(#Obj_Gradient)"/><path d="M 416.53403 146.98122 C 392.9375 133.5 402.3472 20.010353 439.98892 39.375 C 443.48123 1.6271057 487.2535 7.754013 486.96735 39.375 C 514.41388 -1.06863403 549.4889 79.575165 525.96246 120.01879 C 554.19305 139.62691 525.60626 245.27281 502.4375 227.625 C 500.5833 257.03967 459.16455 267.33319 455.52917 227.625 C 432.07571 270.03143 383.17157 204.82918 416.53403 146.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="454.5" y="141.5" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="454.5" y="141.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 467.669 176 L 477.83426 158 L 498.16476 158 L 508.33002 176 L 498.16476 194 L 477.83426 194 Z" fill="url(#Obj_Gradient_3)"/><path d="M 467.669 176 L 477.83426 158 L 498.16476 158 L 508.33002 176 L 498.16476 194 L 477.83426 194 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 339.71 85 L 360.71 85 C 363.194 85 365.21 90.376 365.21 97 C 365.21 103.624 363.194 109 360.71 109 L 339.71 109 C 337.22598 109 335.21 103.624 335.21 97 C 335.21 90.376 337.22598 85 339.71 85" fill="url(#Obj_Gradient_4)"/><path d="M 339.71 85 L 360.71 85 C 363.194 85 365.21 90.376 365.21 97 C 365.21 103.624 363.194 109 360.71 109 L 339.71 109 C 337.22598 109 335.21 103.624 335.21 97 C 335.21 90.376 337.22598 85 339.71 85" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="437" y="55.5" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="437" y="55.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 450.169 91.5 L 460.33426 73.5 L 480.66476 73.5 L 490.83002 91.5 L 480.66476 109.5 L 460.33426 109.5 Z" fill="url(#Obj_Gradient_6)"/><path d="M 450.169 91.5 L 460.33426 73.5 L 480.66476 73.5 L 490.83002 91.5 L 480.66476 109.5 L 460.33426 109.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 355.0216 109.46646 C 360.75049 124.30949 353.19086 143.50909 372.21 154 C 388.3057 162.87834 423.4454 165.52399 453.4511 169.84836" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 445.145 78.46559 L 465.66495 93.362083 L 449.3803 112.107697 L 455.94482 118.5 L 473.73325 98.22214 L 492.66498 111.982796 L 499.145 104.53463 L 478.62488 88.706955 L 495.90488 71.948326 L 486.18488 64.5 L 471.06485 83.12071 L 453.78497 68.224167 Z" fill="url(#Obj_Gradient_7)"/><path d="M 445.145 78.46559 L 465.66495 93.362083 L 449.3803 112.107697 L 455.94482 118.5 L 473.73325 98.22214 L 492.66498 111.982796 L 499.145 104.53463 L 478.62488 88.706955 L 495.90488 71.948326 L 486.18488 64.5 L 471.06485 83.12071 L 453.78497 68.224167 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(286 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">2</tspan></text><rect x="281.29001" y="277" width="278.00003" height="273" fill="white"/><rect x="281.29001" y="277" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 420.11401 428.9812 C 396.51749 415.5 405.92719 302.01035 443.5689 321.375 C 447.06122 283.6271 490.8335 289.75403 490.54733 321.375 C 517.9939 280.93137 553.0689 361.57516 529.54248 402.0188 C 557.773 421.62692 529.18622 527.27283 506.0175 509.625 C 504.1633 539.03967 462.74454 549.3332 459.10913 509.625 C 435.6557 552.03143 386.75156 486.82916 420.11401 428.9812 Z" fill="url(#Obj_Gradient_8)"/><path d="M 420.11401 428.9812 C 396.51749 415.5 405.92719 302.01035 443.5689 321.375 C 447.06122 283.6271 490.8335 289.75403 490.54733 321.375 C 517.9939 280.93137 553.0689 361.57516 529.54248 402.0188 C 557.773 421.62692 529.18622 527.27283 506.0175 509.625 C 504.1633 539.03967 462.74454 549.3332 459.10913 509.625 C 435.6557 552.03143 386.75156 486.82916 420.11401 428.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="433.16" y="333.723" width="67" height="69" fill="url(#Obj_Gradient_9)"/><rect x="433.16" y="333.723" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 446.329 368.223 L 456.49426 350.223 L 476.82477 350.223 L 486.99002 368.223 L 476.82477 386.223 L 456.49426 386.223 Z" fill="url(#Obj_Gradient_10)"/><path d="M 446.329 368.223 L 456.49426 350.223 L 476.82477 350.223 L 486.99002 368.223 L 476.82477 386.223 L 456.49426 386.223 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 343.29 367 L 364.29 367 C 366.77402 367 368.79 372.376 368.79 379 C 368.79 385.624 366.77402 391 364.29 391 L 343.29 391 C 340.806 391 338.79 385.624 338.79 379 C 338.79 372.376 340.806 367 343.29 367" fill="url(#Obj_Gradient_11)"/><path d="M 343.29 367 L 364.29 367 C 366.77402 367 368.79 372.376 368.79 379 C 368.79 385.624 366.77402 391 364.29 391 L 343.29 391 C 340.806 391 338.79 385.624 338.79 379 C 338.79 372.376 340.806 367 343.29 367" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="440.57999" y="428" width="67" height="69" fill="url(#Obj_Gradient_12)"/><rect x="440.57999" y="428" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 453.749 464 L 463.91425 446 L 484.24475 446 L 494.41 464 L 484.24475 482 L 463.91425 482 Z" fill="url(#Obj_Gradient_13)"/><path d="M 453.749 464 L 463.91425 446 L 484.24475 446 L 494.41 464 L 484.24475 482 L 463.91425 482 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 358.11069 391.47244 C 364.4065 409.64648 360.82266 434.49713 377 446 C 390.09668 455.31238 416.1513 455.88107 439.42624 458.3237" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 480.27567 445.52594 C 482.57022 438.68463 488.31775 434.80447 487.16 425 C 486.26974 417.46088 481.2962 406.41544 476.94943 395.90878" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="0" y="277" width="278" height="273" fill="white"/><rect x="0" y="277" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 126.32403 422.9812 C 102.72751 409.5 112.13721 296.01035 149.77893 315.375 C 153.27126 277.6271 197.0435 283.75403 196.75735 315.375 C 224.20389 274.93137 259.27893 355.57516 235.75249 396.0188 C 263.98303 415.62692 235.39624 521.27283 212.22751 503.625 C 210.37331 533.03967 168.95457 543.3332 165.31917 503.625 C 141.86572 546.03143 92.96157 480.82916 126.32403 422.9812 Z" fill="url(#Obj_Gradient_14)"/><path d="M 126.32403 422.9812 C 102.72751 409.5 112.13721 296.01035 149.77893 315.375 C 153.27126 277.6271 197.0435 283.75403 196.75735 315.375 C 224.20389 274.93137 259.27893 355.57516 235.75249 396.0188 C 263.98303 415.62692 235.39624 521.27283 212.22751 503.625 C 210.37331 533.03967 168.95457 543.3332 165.31917 503.625 C 141.86572 546.03143 92.96157 480.82916 126.32403 422.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="138.869995" y="332" width="67" height="69" fill="url(#Obj_Gradient_15)"/><rect x="138.869995" y="332" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 152.039 364.5 L 162.20425 346.5 L 182.53476 346.5 L 192.70001 364.5 L 182.53476 382.5 L 162.20425 382.5 Z" fill="url(#Obj_Gradient_16)"/><path d="M 152.039 364.5 L 162.20425 346.5 L 182.53476 346.5 L 192.70001 364.5 L 182.53476 382.5 L 162.20425 382.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 62 365 L 83 365 C 85.484 365 87.5 370.376 87.5 377 C 87.5 383.624 85.484 389 83 389 L 62 389 C 59.516 389 57.5 383.624 57.5 377 C 57.5 370.376 59.516 365 62 365" fill="url(#Obj_Gradient_17)"/><path d="M 62 365 L 83 365 C 85.484 365 87.5 370.376 87.5 377 C 87.5 383.624 85.484 389 83 389 L 62 389 C 59.516 389 57.5 383.624 57.5 377 C 57.5 370.376 59.516 365 62 365" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="147.453" y="421.5" width="67" height="69" fill="url(#Obj_Gradient_18)"/><rect x="147.453" y="421.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 160.62199 456 L 170.78725 438 L 191.11775 438 L 201.283 456 L 191.11775 474 L 170.78725 474 Z" fill="url(#Obj_Gradient_19)"/><path d="M 160.62199 456 L 170.78725 438 L 191.11775 438 L 201.283 456 L 191.11775 474 L 170.78725 474 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 75.38166 389.48718 C 79.4207 406.98972 73.112816 431.39597 87.5 442 C 99.01256 450.4853 123.781944 450.13696 145.894165 451.55768" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 118.20667 389.5221 L 154.46504 352.99951 C 154.46504 352.99951 148.56358 343.41068 154.80022 342.02164 C 161.03673 340.63266 163.73999 342.85837 163.73999 342.85837 L 156.62726 349.49799 L 163.3246 356.19534 L 170.08206 349.91098 C 170.08206 349.91098 172.57938 351.69775 172.78146 351.89984 C 172.98338 352.10178 171.3094 356.98645 169.817 358.88278 C 168.32417 360.77942 160.15027 358.03 160.15039 358.03012 C 160.15039 358.03012 122.44828 393.7637 122.342445 393.86954 C 122.23656 393.9754 121.27359 395.3426 119.157745 393.0148 C 117.04174 390.68701 118.20667 389.5221 118.20667 389.5221 Z" fill="url(#Obj_Gradient_20)"/><path d="M 118.20667 389.5221 L 154.46504 352.99951 C 154.46504 352.99951 148.56358 343.41068 154.80022 342.02164 C 161.03673 340.63266 163.73999 342.85837 163.73999 342.85837 L 156.62726 349.49799 L 163.3246 356.19534 L 170.08206 349.91098 C 170.08206 349.91098 172.57938 351.69775 172.78146 351.89984 C 172.98338 352.10178 171.3094 356.98645 169.817 358.88278 C 168.32417 360.77942 160.15027 358.03 160.15039 358.03012 C 160.15039 358.03012 122.44828 393.7637 122.342445 393.86954 C 122.23656 393.9754 121.27359 395.3426 119.157745 393.0148 C 117.04174 390.68701 118.20667 389.5221 118.20667 389.5221 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="0" y="0" width="278" height="273" fill="white"/><rect x="0" y="0" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" fill="url(#Obj_Gradient_21)"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="179.355" y="146" width="67" height="69" fill="url(#Obj_Gradient_22)"/><rect x="179.355" y="146" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" fill="url(#Obj_Gradient_23)"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 69.855003 111 L 90.855003 111 C 93.339005 111 95.355003 116.376 95.355003 123 C 95.355003 129.62399 93.339005 135 90.855003 135 L 69.855003 135 C 67.371002 135 65.355003 129.62399 65.355003 123 C 65.355003 116.376 67.371002 111 69.855003 111" fill="url(#Obj_Gradient_24)"/><path d="M 69.855003 111 L 90.855003 111 C 93.339005 111 95.355003 116.376 95.355003 123 C 95.355003 129.62399 93.339005 135 90.855003 135 L 69.855003 135 C 67.371002 135 65.355003 129.62399 65.355003 123 C 65.355003 116.376 67.371002 111 69.855003 111" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="161.855" y="60" width="67" height="69" fill="url(#Obj_Gradient_25)"/><rect x="161.855" y="60" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" fill="url(#Obj_Gradient_26)"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 74.355003 137 C 92.019905 160.33099 107.23704 198.83237 127.355 207 C 143.19005 213.42882 162.06882 201.06767 180.39609 192.10172" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 222.38084 162.05576 C 227.03842 153.03807 238.59448 143.85422 236.355 135 C 234.66139 128.303986 225.07715 121.79347 216.5538 115.25858" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(5.8400302 4)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><text transform="translate(8 286)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text><text transform="translate(286 286)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">4</tspan></text></g></g></svg>
diff --git a/doc/book/src/java-broker/images/HA-2N-SplitBrain.png b/doc/book/src/java-broker/images/HA-2N-SplitBrain.png
new file mode 100644
index 0000000000..74393648a5
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-SplitBrain.png
diff --git a/doc/book/src/java-broker/images/HA-2N-SplitBrain.svg b/doc/book/src/java-broker/images/HA-2N-SplitBrain.svg
new file mode 100644
index 0000000000..c82cc5065a
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-2N-SplitBrain.svg
@@ -0,0 +1,3 @@
+<?xml version="1.0"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">
+<svg xmlns="http://www.w3.org/2000/svg" xmlns:xl="http://www.w3.org/1999/xlink" version="1.1" viewBox="-20 -16 598 591" width="598pt" height="591pt"><metadata xmlns:dc="http://purl.org/dc/elements/1.1/"><dc:date>2012-05-31 06:19Z</dc:date><!-- Produced by OmniGraffle Professional 5.3.6 --></metadata><defs><radialGradient cx="0" cy="0" r="1" id="Gradient" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#b5d0ea"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient" xl:href="#Gradient" gradientTransform="translate(200.35501 136.5) scale(145.18695)"/><linearGradient x1="0" x2="1" id="Gradient_2" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#2824aa"/></linearGradient><linearGradient id="Obj_Gradient_2" xl:href="#Gradient_2" gradientTransform="translate(212.855 146) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_3" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#0caa25"/></linearGradient><linearGradient id="Obj_Gradient_3" xl:href="#Gradient_3" gradientTransform="translate(212.85451 162.5) rotate(90) scale(36)"/><linearGradient x1="0" x2="1" id="Gradient_4" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6923aa"/></linearGradient><linearGradient id="Obj_Gradient_4" xl:href="#Gradient_4" gradientTransform="translate(80.355003 111) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_5" xl:href="#Gradient_2" gradientTransform="translate(195.355 59.999996) rotate(90) scale(69.000007)"/><linearGradient x1="0" x2="1" id="Gradient_5" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#aa172d"/></linearGradient><linearGradient id="Obj_Gradient_6" xl:href="#Gradient_5" gradientTransform="translate(195.35451 78) rotate(90) scale(36)"/><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="FilledArrow_Marker" viewBox="-1 -3 6 6" markerWidth="6" markerHeight="6" color="black"><g><path d="M 3.7333333 0 L 0 -1.4 L 0 1.4 Z" fill="currentColor" stroke="currentColor" stroke-width="1"/></g></marker><marker orient="auto" overflow="visible" markerUnits="strokeWidth" id="Arrow_Marker" viewBox="-1 -4 10 8" markerWidth="10" markerHeight="8" color="black"><g><path d="M 8 0 L 0 -3 L 0 3 Z" fill="none" stroke="currentColor" stroke-width="1"/></g></marker><font-face font-family="Helvetica" font-size="12" units-per-em="1000" underline-position="-75.683594" underline-thickness="49.316406" slope="0" x-height="532.22656" cap-height="719.72656" ascent="770.01953" descent="-229.98047" font-weight="bold"><font-face-src><font-face-name name="Helvetica-Bold"/></font-face-src></font-face><radialGradient id="Obj_Gradient_7" xl:href="#Gradient" gradientTransform="translate(468.855 133.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_8" xl:href="#Gradient_2" gradientTransform="translate(486.355 141.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_6" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#6ead6a"/></linearGradient><linearGradient id="Obj_Gradient_9" xl:href="#Gradient_6" gradientTransform="translate(486.3545 158) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_10" xl:href="#Gradient_4" gradientTransform="translate(348.565 85) rotate(90) scale(24)"/><radialGradient cx="0" cy="0" r="1" id="Gradient_7" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset=".5" stop-color="#ea061f"/><stop offset="1" stop-color="#aaa"/></radialGradient><radialGradient id="Obj_Gradient_11" xl:href="#Gradient_7" gradientTransform="translate(483.56808 132.68761) scale(20.071926)"/><linearGradient id="Obj_Gradient_12" xl:href="#Gradient_2" gradientTransform="translate(465.01501 51.445) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_8" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#b6b4b5"/></linearGradient><linearGradient id="Obj_Gradient_13" xl:href="#Gradient_8" gradientTransform="translate(465.0145 67.945) rotate(90) scale(36)"/><font-face font-family="Arial Unicode MS" font-size="36" panose-1="2 11 6 4 2 2 2 2 2 4" units-per-em="1000" underline-position="-100.097656" underline-thickness="49.804688" slope="0" x-height="529.78516" cap-height="728.02734" ascent="1068.84766" descent="-270.9961" font-weight="500"><font-face-src><font-face-name name="ArialUnicodeMS"/></font-face-src></font-face><radialGradient id="Obj_Gradient_14" xl:href="#Gradient" gradientTransform="translate(180.29001 414.5) scale(145.18695)"/><linearGradient id="Obj_Gradient_15" xl:href="#Gradient_2" gradientTransform="translate(172.37 337) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_9" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#66ab61"/></linearGradient><linearGradient id="Obj_Gradient_16" xl:href="#Gradient_9" gradientTransform="translate(172.36951 351.5) rotate(90) scale(36)"/><linearGradient id="Obj_Gradient_17" xl:href="#Gradient_4" gradientTransform="translate(53.18649 396) rotate(90) scale(24)"/><linearGradient id="Obj_Gradient_18" xl:href="#Gradient_2" gradientTransform="translate(180.953 426.5) rotate(90) scale(69)"/><linearGradient x1="0" x2="1" id="Gradient_10" gradientUnits="userSpaceOnUse"><stop offset="0" stop-color="white"/><stop offset="1" stop-color="#21aa1a"/></linearGradient><linearGradient id="Obj_Gradient_19" xl:href="#Gradient_10" gradientTransform="translate(180.9525 443) rotate(90) scale(36)"/><radialGradient id="Obj_Gradient_20" xl:href="#Gradient_7" gradientTransform="translate(185.10121 416.23209) scale(20.071926)"/><font-face font-family="Arial Unicode MS" font-size="72" panose-1="2 11 6 4 2 2 2 2 2 4" units-per-em="1000" underline-position="-100.097656" underline-thickness="49.804688" slope="0" x-height="529.78516" cap-height="728.02734" ascent="1068.84766" descent="-270.9961" font-weight="500"><font-face-src><font-face-name name="ArialUnicodeMS"/></font-face-src></font-face></defs><g stroke="none" stroke-opacity="1" stroke-dasharray="none" fill="none" fill-opacity="1"><title>Canvas 6</title><g><title>Layer 1</title><rect x="0" y="0" width="278" height="273" fill="white"/><rect x="0" y="0" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" fill="url(#Obj_Gradient)"/><path d="M 146.38904 149.98122 C 122.79251 136.5 132.20221 23.010353 169.84393 42.375 C 173.33626 4.6271057 217.1085 10.754013 216.82236 42.375 C 244.26889 1.931366 279.34393 82.575165 255.81749 123.01879 C 284.04803 142.62691 255.46124 248.27281 232.29251 230.625 C 230.43831 260.03967 189.01958 270.33319 185.38417 230.625 C 161.930725 273.03143 113.02657 207.82918 146.38904 149.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="179.355" y="146" width="67" height="69" fill="url(#Obj_Gradient_2)"/><rect x="179.355" y="146" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" fill="url(#Obj_Gradient_3)"/><path d="M 192.524 180.5 L 202.68925 162.5 L 223.01976 162.5 L 233.18501 180.5 L 223.01976 198.5 L 202.68925 198.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 69.855003 111 L 90.855003 111 C 93.339005 111 95.355003 116.376 95.355003 123 C 95.355003 129.62399 93.339005 135 90.855003 135 L 69.855003 135 C 67.371002 135 65.355003 129.62399 65.355003 123 C 65.355003 116.376 67.371002 111 69.855003 111" fill="url(#Obj_Gradient_4)"/><path d="M 69.855003 111 L 90.855003 111 C 93.339005 111 95.355003 116.376 95.355003 123 C 95.355003 129.62399 93.339005 135 90.855003 135 L 69.855003 135 C 67.371002 135 65.355003 129.62399 65.355003 123 C 65.355003 116.376 67.371002 111 69.855003 111" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="161.855" y="60" width="67" height="69" fill="url(#Obj_Gradient_5)"/><rect x="161.855" y="60" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" fill="url(#Obj_Gradient_6)"/><path d="M 175.024 96 L 185.18925 78 L 205.51976 78 L 215.68501 96 L 205.51976 114 L 185.18925 114 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 83 135 C 87.728355 147.33211 78.882416 164.66258 97.186493 172 C 112.61268 178.18378 147.33044 177.27293 176.94928 178.23598" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 222.38084 162.05576 C 227.03842 153.03807 238.59448 143.85422 236.355 135 C 234.66139 128.303986 225.07715 121.79347 216.5538 115.25858" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(285 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">2</tspan></text><rect x="279.64499" y="0" width="278.00003" height="273" fill="white"/><rect x="279.64499" y="0" width="278.00003" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 414.88904 146.98122 C 391.2925 133.5 400.7022 20.010353 438.34393 39.375 C 441.83624 1.6271057 485.60852 7.754013 485.32236 39.375 C 512.76892 -1.06863403 547.84393 79.575165 524.3175 120.01879 C 552.54803 139.62691 523.96124 245.27281 500.7925 227.625 C 498.93832 257.03967 457.51956 267.33319 453.88416 227.625 C 430.43073 270.03143 381.52658 204.82918 414.88904 146.98122 Z" fill="url(#Obj_Gradient_7)"/><path d="M 414.88904 146.98122 C 391.2925 133.5 400.7022 20.010353 438.34393 39.375 C 441.83624 1.6271057 485.60852 7.754013 485.32236 39.375 C 512.76892 -1.06863403 547.84393 79.575165 524.3175 120.01879 C 552.54803 139.62691 523.96124 245.27281 500.7925 227.625 C 498.93832 257.03967 457.51956 267.33319 453.88416 227.625 C 430.43073 270.03143 381.52658 204.82918 414.88904 146.98122 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="452.855" y="141.5" width="66.99997" height="69" fill="url(#Obj_Gradient_8)"/><rect x="452.855" y="141.5" width="66.99997" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 466.024 176 L 476.18924 158 L 496.51974 158 L 506.685 176 L 496.51974 194 L 476.18924 194 Z" fill="url(#Obj_Gradient_9)"/><path d="M 466.024 176 L 476.18924 158 L 496.51974 158 L 506.685 176 L 496.51974 194 L 476.18924 194 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 338.065 85 L 359.065 85 C 361.54901 85 363.565 90.376 363.565 97 C 363.565 103.624 361.54901 109 359.065 109 L 338.065 109 C 335.581 109 333.565 103.624 333.565 97 C 333.565 90.376 335.581 85 338.065 85" fill="url(#Obj_Gradient_10)"/><path d="M 338.065 85 L 359.065 85 C 361.54901 85 363.565 90.376 363.565 97 C 363.565 103.624 361.54901 109 359.065 109 L 338.065 109 C 335.581 109 333.565 103.624 333.565 97 C 333.565 90.376 335.581 85 338.065 85" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 353.37662 109.46646 C 359.1055 124.30949 351.54587 143.50909 370.565 154 C 386.66068 162.87834 421.80042 165.52399 451.8061 169.84834" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 470.9274 125.16071 L 480.5343 133.763016 L 472.91028 144.58757 L 475.98358 148.27911 L 484.31168 136.56943 L 493.17505 144.515656 L 496.20877 140.214615 L 486.60178 131.07477 L 494.69186 121.39724 L 490.1412 117.09611 L 483.06238 127.84885 L 474.9724 119.246674 Z" fill="url(#Obj_Gradient_11)"/><path d="M 470.9274 125.16071 L 480.5343 133.763016 L 472.91028 144.58757 L 475.98358 148.27911 L 484.31168 136.56943 L 493.17505 144.515656 L 496.20877 140.214615 L 486.60178 131.07477 L 494.69186 121.39724 L 490.1412 117.09611 L 483.06238 127.84885 L 474.9724 119.246674 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="431.51501" y="51.445" width="67" height="69" fill="url(#Obj_Gradient_12)"/><rect x="431.51501" y="51.445" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 444.684 85.945 L 454.84924 67.945 L 475.17975 67.945 L 485.345 85.945 L 475.17975 103.945 L 454.84924 103.945 Z" fill="url(#Obj_Gradient_13)"/><path d="M 444.684 85.945 L 454.84924 67.945 L 475.17975 67.945 L 485.345 85.945 L 475.17975 103.945 L 454.84924 103.945 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 485.18884 157.50099 C 484.6743 149.33481 485.7887 141.847626 483.645 133 C 482.1737 126.92762 479.1671 120.211914 476.20593 113.515625" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(496 56.5)" fill="#262626"><tspan font-family="Arial Unicode MS" font-size="36" font-weight="500" fill="#262626" x="0" y="38" textLength="36">♛</tspan></text><text transform="translate(6 6)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">1</tspan></text><rect x="0" y="278" width="278" height="273" fill="white"/><rect x="0" y="278" width="278" height="273" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 126.32403 427.9812 C 102.72751 414.5 112.13721 301.01035 149.77893 320.375 C 153.27126 282.6271 197.0435 288.75403 196.75735 320.375 C 224.20389 279.93137 259.27893 360.57516 235.75249 401.0188 C 263.98303 420.62692 235.39624 526.27283 212.22751 508.625 C 210.37331 538.03967 168.95457 548.3332 165.31917 508.625 C 141.86572 551.03143 92.96157 485.82916 126.32403 427.9812 Z" fill="url(#Obj_Gradient_14)"/><path d="M 126.32403 427.9812 C 102.72751 414.5 112.13721 301.01035 149.77893 320.375 C 153.27126 282.6271 197.0435 288.75403 196.75735 320.375 C 224.20389 279.93137 259.27893 360.57516 235.75249 401.0188 C 263.98303 420.62692 235.39624 526.27283 212.22751 508.625 C 210.37331 538.03967 168.95457 548.3332 165.31917 508.625 C 141.86572 551.03143 92.96157 485.82916 126.32403 427.9812 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="138.869995" y="337" width="67" height="69" fill="url(#Obj_Gradient_15)"/><rect x="138.869995" y="337" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 152.039 369.5 L 162.20425 351.5 L 182.53476 351.5 L 192.70001 369.5 L 182.53476 387.5 L 162.20425 387.5 Z" fill="url(#Obj_Gradient_16)"/><path d="M 152.039 369.5 L 162.20425 351.5 L 182.53476 351.5 L 192.70001 369.5 L 182.53476 387.5 L 162.20425 387.5 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 42.68649 396 L 63.68649 396 C 66.170486 396 68.186493 401.376 68.186493 408 C 68.186493 414.624 66.170486 420 63.68649 420 L 42.68649 420 C 40.202488 420 38.18649 414.624 38.18649 408 C 38.18649 401.376 40.202488 396 42.68649 396" fill="url(#Obj_Gradient_17)"/><path d="M 42.68649 396 L 63.68649 396 C 66.170486 396 68.186493 401.376 68.186493 408 C 68.186493 414.624 66.170486 420 63.68649 420 L 42.68649 420 C 40.202488 420 38.18649 414.624 38.18649 408 C 38.18649 401.376 40.202488 396 42.68649 396" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><rect x="147.453" y="426.5" width="67" height="69" fill="url(#Obj_Gradient_18)"/><rect x="147.453" y="426.5" width="67" height="69" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 160.62199 461 L 170.78725 443 L 191.11775 443 L 201.283 461 L 191.11775 479 L 170.78725 479 Z" fill="url(#Obj_Gradient_19)"/><path d="M 160.62199 461 L 170.78725 443 L 191.11775 443 L 201.283 461 L 191.11775 479 L 170.78725 479 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 64.073936 420.37445 C 71.881844 429.24875 71.228394 440.71005 87.5 447 C 100.49186 452.02213 124.27972 453.7499 145.93179 456.12848" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><path d="M 61.40507 395.58307 C 67.269463 386.72293 63.968967 373.3653 79 369 C 90.933815 365.53418 114.428574 367.73508 135.71434 368.85406" marker-end="url(#FilledArrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="3"/><text transform="translate(6 286)" fill="black"><tspan font-family="Helvetica" font-size="12" font-weight="bold" x=".16308594" y="11" textLength="6.673828">3</tspan></text><path d="M 172.46051 408.7052 L 182.06741 417.3075 L 174.4434 428.13205 L 177.51671 431.82358 L 185.8448 420.1139 L 194.70818 428.06015 L 197.74191 423.7591 L 188.1349 414.61926 L 196.22501 404.9417 L 191.67433 400.6406 L 184.59552 411.39334 L 176.50554 402.79114 Z" fill="url(#Obj_Gradient_20)"/><path d="M 172.46051 408.7052 L 182.06741 417.3075 L 174.4434 428.13205 L 177.51671 431.82358 L 185.8448 420.1139 L 194.70818 428.06015 L 197.74191 423.7591 L 188.1349 414.61926 L 196.22501 404.9417 L 191.67433 400.6406 L 184.59552 411.39334 L 176.50554 402.79114 Z" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><path d="M 186.722 441.046 C 186.20738 432.87949 187.32178 425.39178 185.17799 416.544 C 183.70665 410.47153 180.69995 403.75586 177.73871 397.05966" marker-end="url(#Arrow_Marker)" stroke="black" stroke-linecap="round" stroke-linejoin="round" stroke-width="1"/><text transform="translate(19.372978 273)" fill="black"><tspan font-family="Arial Unicode MS" font-size="72" font-weight="500" x="0" y="77" textLength="54">☠</tspan></text></g></g></svg>
diff --git a/doc/book/src/java-broker/images/HA-BDBHAMessageStore-MBean-jconsole.png b/doc/book/src/java-broker/images/HA-BDBHAMessageStore-MBean-jconsole.png
new file mode 100644
index 0000000000..6caaacb1e1
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-BDBHAMessageStore-MBean-jconsole.png
diff --git a/doc/book/src/java-broker/images/HA-perftests-results.png b/doc/book/src/java-broker/images/HA-perftests-results.png
new file mode 100644
index 0000000000..e8dbb3c123
--- /dev/null
+++ b/doc/book/src/java-broker/images/HA-perftests-results.png
diff --git a/doc/book/src/programming/Programming-In-Apache-Qpid-Book.xml b/doc/book/src/programming/Programming-In-Apache-Qpid-Book.xml
index 3052e2acc1..1c6bfafc27 100644
--- a/doc/book/src/programming/Programming-In-Apache-Qpid-Book.xml
+++ b/doc/book/src/programming/Programming-In-Apache-Qpid-Book.xml
@@ -1269,7 +1269,7 @@ spout - -content "$(cat rdu.xml | sed -e 's/70/45/')" xml/weather
 		    unreliable will be used and for durable messages on
 		    durable queues there is the possibility that messages
 		    will be redelivered; if exactly-once is requested,
-		    at-most-once will be used and the application needs to
+		    at-least-once will be used and the application needs to
 		    be able to deal with duplicates.</para></footnote>
 		  </entry>
 		</row>
@@ -3783,6 +3783,16 @@ spout - -content "$(cat rdu.xml | sed -e 's/70/45/')" xml/weather
 		  <para>For compatibility with older clients, the synonym <varname>amqj.receiveBufferSize</varname> is supported.</para>
 		</entry>
 	      </row>
+          <row>
+            <entry>qpid.failover_method_timeout</entry>
+            <entry>long</entry>
+            <entry>60000</entry>
+            <entry>
+              <para>During failover, this is the timeout for each attempt to try to re-establish the connection.
+                    If a reconnection attempt exceeds the timeout, the entire failover process is aborted.</para>
+              <para>It is only applicable for AMQP 0-8/0-9/0-9-1 clients.</para>
+            </entry>
+          </row>
 	    </tbody>
 	  </tgroup>
 	</table>