<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE entities [
<!ENTITY %  entities SYSTEM  "commonEntities.xml">
%entities;
]>
<!--

 Licensed to the Apache Software Foundation (ASF) under one
 or more contributor license agreements.  See the NOTICE file
 distributed with this work for additional information
 regarding copyright ownership.  The ASF licenses this file
 to you under the Apache License, Version 2.0 (the
 "License"); you may not use this file except in compliance
 with the License.  You may obtain a copy of the License at

   http://www.apache.org/licenses/LICENSE-2.0

 Unless required by applicable law or agreed to in writing,
 software distributed under the License is distributed on an
 "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 KIND, either express or implied.  See the License for the
 specific language governing permissions and limitations
 under the License.

-->
<chapter id="Java-Broker-High-Availability">
  <title>High Availability</title>

  <section id="Java-Broker-High-Availability-GeneralIntroduction">
    <title>General Introduction</title>
    <para>The term High Availability (HA) usually refers to having a number of instances of a
      service such as a Message Broker available so that should a service unexpectedly fail, or
      requires to be shutdown for maintenance, users may quickly connect to another instance and
      continue their work with minimal interruption. HA is one way to make a overall system more
      resilient by eliminating a single point of failure from a system.</para>
    <para>HA offerings are usually categorised as <emphasis role="bold">Active/Active</emphasis> or
        <emphasis role="bold">Active/Passive</emphasis>. An Active/Active system is one where all
      nodes within the group are usually available for use by clients all of the time. In an
      Active/Passive system, one only node within the group is available for use by clients at any
      one time, whilst the others are in some kind of standby state, awaiting to quickly step-in in
      the event the active node becomes unavailable. </para>
  </section>

  <section id="Java-Broker-High-Availability-OverviewOfHA">
    <title>Overview of HA within the Java Broker</title>
    <para>The Java Broker provides a HA implementation offering an <emphasis role="bold"
        >Active/Passive</emphasis> mode of operation. When using HA, many instances of the Java
      Broker work together to form an high availability group of two or more nodes.</para>
    <para>The remainder of this section now talks about the specifics of how HA is achieved in terms
      of the <link linkend="Java-Broker-Concepts">concepts</link> introduced earlier in this
      book.</para>
    <para>The <link linkend="Java-Broker-Concepts-Virtualhosts">Virtualhost</link> is the unit of
      replication. This means that any <emphasis>durable</emphasis> queues, exchanges, and bindings
      belonging to that virtualhost, any <emphasis>persistent</emphasis> messages contained within
      the queues and any attribute settings applied to the virtualhost itself are automatically
      replicated to all nodes within the group.<footnote>
        <para>Transient messages and messages on non-durable queues are not replicated.</para>
      </footnote></para>
    <para>It is the <link linkend="Java-Broker-Concepts-Virtualhost-Nodes">Virtualhost Nodes</link>
      (from different Broker instances) that join together to form a group. The virtualhost nodes
      collectively to coordinate the group: they organise replication between the master and
      replicas and conduct elections to determine who becomes the new master in the event of the old
      failing.</para>
    <para>When a virtualhost node is in the <emphasis>master</emphasis> role, the virtualhost
      beneath it is available for messaging work. Any write operations sent to the virtualhost are
      automatically replicated to all other nodes in group.</para>
    <para>When a virtualhost node is in the <emphasis>replica</emphasis> role, the virtualhost
      beneath it is always unavailable for message work. Any attempted connections to a virtualhost
      in this state are automatically turned away, allowing a messaging client to discover where the
      master currently resides. When in replica role, the node sole responsibility is to consume a
      replication stream in order that it remains up to date with the master.</para>
    <para>Messaging clients discover the active virtualhost.This can be achieved using a static
      technique (for instance, a failover url (a feature of a Qpid Java Client)), or a dynamic one
      utilising some kind of proxy or virtual IP (VIP).</para>
    <para>The figure that follows illustrates a group formed of three virtualhost nodes from three
      separate Broker instances. A client is connected to the virtualhost node that is in the master
      role. The two virtualhost nodes <literal>weather1</literal> and <literal>weather3</literal>
      are replicas and are receiving a stream of updates.</para>
    <figure id="Java-Broker-High-Availability-OverviewOfHA-Figure">
      <title>3-node group deployed across three Brokers.</title>
      <mediaobject>
        <imageobject>
          <imagedata fileref="images/HA-Overview.png" format="PNG" scalefit="1"/>
        </imageobject>
        <textobject>
          <phrase>Diagram showing a 3 node group deployed across three Brokers</phrase>
        </textobject>
      </mediaobject>
    </figure>
    <para>Currently, the only virtualhost/virtualhost node type offering HA is BDB HA. Internally,
      this leverages the HA capabilities of the Berkeley DB JE edition. BDB JE is an <link
        linkend="Java-Broker-Miscellaneous-Installing-Oracle-BDB-JE">optional dependency</link> of
      the Broker.</para>
    <note>
      <para>The Java Broker HA solution is incompatible with the HA solution offered by the CPP
        Broker. It is not possible to co-locate Java and CPP Brokers within the same group.</para>
    </note>
  </section>
  <section id="Java-Broker-High-Availability-CreatingGroup">
    <title>Creating a group</title>
    <para>This section describes how to create a group. At a high level, creating a group involves
      first creating the first node standalone, then creating subsequent nodes referencing the first
      node so the nodes can introduce themselves and gradually the group is built up.</para>
    <para>A group is created through either <link
        linkend="Java-Broker-Management-Channel-Web-Console">Web Management</link> or the <link
        linkend="Java-Broker-Management-Channel-REST-API">REST API</link>. These instructions
      presume you are using Web Management. To illustrate the example it builds the group
      illustrated in figure <xref linkend="Java-Broker-High-Availability-OverviewOfHA-Figure"
      /></para>
    <para><orderedlist>
        <listitem>
          <para>Install a Broker on each machine that will be used to host the group. As messaging
            clients will need to be able to connect to and authentication to all Brokers, it usually
            makes sense to choose a common authentication mechanism e.g. Simple LDAP Authentication,
            External with SSL client authentication or Kerberos.</para>
        </listitem>
        <listitem>
          <para>Select one Broker instance to host the first node instance. This choice is an
            arbitrary one. The node is special only whilst creating group. Once creation is
            complete, all nodes will be considered equal.</para>
        </listitem>
        <listitem>
          <para>
            <para>Click the <literal>Add</literal> button on the Virtualhost Panel on the Broker
              tab.</para>
            <orderedlist>
              <listitem>
                <para>Give the Virtualhost node a unique name e.g. <literal>weather1</literal>. The
                  name must be unique within the group and unique to that Broker. It is best if the
                  node names are chosen from a different nomenclature than the machine names
                  themselves.</para>
              </listitem>
              <listitem>
                <para>Choose <literal>BDB_HA</literal> and select <literal>New group</literal>
                </para>
              </listitem>
              <listitem>
                <para>Give the group a name e.g. <literal>weather</literal>. The group name must be
                  unique and will be the name also given to the virtualhost, so this is the name the
                  messaging clients will use in their connection url.</para>
              </listitem>
              <listitem>
                <para>Give the address of this node. This is an address on this node's host that
                  will be used for replication purposes. The hostname <emphasis>must</emphasis> be
                  resolvable by all the other nodes in the group. This is separate from the address
                  used by messaging clients to connect to the Broker. It is usually best to choose a
                  symbolic name, rather than an IP address.</para>
              </listitem>
              <listitem>
                <para>Now add the node addresses of all the other nodes that will form the group. In
                  our example we are building a three node group so we give the node addresses of
                    <literal>chaac:5000</literal> and <literal>indra:5000</literal>.</para>
              </listitem>
              <listitem>
                <para>Click Add to create the node. The virtualhost node will be created with the
                  virtualhost. As there is only one node at this stage, the role will be
                  master.</para>
              </listitem>
            </orderedlist>
            <para>
              <figure>
                <title>Creating 1st node in a group</title>
                <mediaobject>
                  <imageobject>
                    <imagedata fileref="images/HA-Create-1.png" format="PNG" scalefit="1"/>
                  </imageobject>
                  <textobject>
                    <phrase>Creating 1st node in a group</phrase>
                  </textobject>
                </mediaobject>
              </figure>
            </para>
          </para>
        </listitem>
        <listitem>
          <para>
            <para>Now move to the second Broker to be the group. Click the <literal>Add</literal>
              button on the Virtualhost Panel on the Broker tab of the second Broker.</para>
            <orderedlist>
              <listitem>
                <para>Give the Virtualhost node a unique name e.g.
                  <literal>weather2</literal>.</para>
              </listitem>
              <listitem>
                <para>Choose <literal>BDB_HA</literal> and choose <literal>Existing group</literal>
                </para>
              </listitem>
              <listitem>
                <para>Give the details of the <emphasis>existing node</emphasis>. Following our
                  example, specify <literal>weather</literal>, <literal>weather1</literal> and
                    <literal>thor:5000</literal></para>
              </listitem>
              <listitem>
                <para>Give the address of this node.</para>
              </listitem>
              <listitem>
                <para>Click Add to create the node. The node will use the existing details to
                  contact it and introduce itself into the group. At this stage, the group will have
                  two nodes, with the second node in the replica role.</para>
              </listitem>
              <listitem>
                <para>Repeat these steps until you have added all the nodes to the group.</para>
              </listitem>
            </orderedlist>
            <para>
              <figure>
                <title>Adding subsequent nodes to the group</title>
                <mediaobject>
                  <imageobject>
                    <imagedata fileref="images/HA-Create-2.png" format="PNG" scalefit="1"/>
                  </imageobject>
                  <textobject>
                    <phrase>Adding subsequent nodes to the group</phrase>
                  </textobject>
                </mediaobject>
              </figure>
            </para>
          </para>
        </listitem>

      </orderedlist></para>
    <para>The group is now formed and is ready for us. Looking at the virtualhost node of any of the
      nodes shows a complete view of the whole group. <figure>
        <title>View of group from one node</title>
        <mediaobject>
          <imageobject>
            <imagedata fileref="images/HA-Create-3.png" format="PNG" scalefit="1"/>
          </imageobject>
          <textobject>
            <phrase>View of group from one node</phrase>
          </textobject>
        </mediaobject>
      </figure></para>
  </section>

  <section id="Java-Broker-High-Availability-Behaviour">
    <title>Behaviour of the Group</title>
    <para>This section first describes the behaviour of the group in its default configuration. It
      then goes on to talk about the various controls that are available to override it. It
      describes the controls available that affect the <ulink
        url="http://en.wikipedia.org/wiki/ACID#Durability">durability</ulink> of transactions and
      the data consistency between the master and replicas and thus make trade offs between
      performance and reliability.</para>

    <section id="Java-Broker-High-Availability-Behaviour-Default-Behaviour">
      <title>Default Behaviour</title>
      <para>Let's first look at the behaviour of a group in default configuration.</para>
      <para>In the default configuration, for any messaging work to be done, there must be at least
          <emphasis>quorum</emphasis> nodes present. This means for example, in a three node group,
        this means there must be at least two nodes available.</para>
      <para>When a messaging client sends a transaction, it can be assured that, before the control
        returns back to his application after the commit call that the following is true:</para>
      <para><itemizedlist>
          <listitem>
            <para>At the master, the transaction is <emphasis>written to disk and OS level caches
                are flushed</emphasis> meaning the data is on the storage device.</para>
          </listitem>
          <listitem>
            <para>At least quorum minus 1 replicas, <emphasis>acknowledge the receipt of
                transaction</emphasis>. The replicas will write the data to the storage device
              sometime later.</para>
          </listitem>
        </itemizedlist></para>
      <para>If there were to be a master failure immediately after the transaction was committed,
        the transaction would be held by at least quorum minus one replicas. For example, if we had
        a group of three, then we would be assured that at least one replica held the
        transaction.</para>

      <para>In the event of a master failure, if quorum nodes remain, those nodes hold an election.
        The nodes will elect master the node with the most recent transaction. If two or more nodes
        have the most recent transaction the group makes an arbitrary choice. If quorum number of
        nodes does not remain, the nodes cannot elect a new master and will wait until nodes rejoin.
        You will see later that manual controls are available allow service to be restored from
        fewer than quorum nodes and to influence which node gets elected in the event of a
        tie.</para>

      <para>Whenever a group has fewer than quorum nodes present, the virtualhost will be
        unavailable and messaging connections will be refused. If quorum disappears at the very
        moment a messaging client sends a transaction that transaction will fail.</para>

      <para>You will have noticed the difference in the synchronization policies applied the master
        and the replicas. The replicas send the acknowledgement back before the data is written to
        disk. The master synchronously writes the transaction to storage. This is an example of a
        trade off between durability and performance. We will see more about how to control this
        trade off later.</para>
    </section>
    <section id="Java-Broker-High-Availability-Behaviour-SynchronizationPolicy">
      <title>Synchronization Policy</title>
      <para>The <emphasis>synchronization policy</emphasis> dictates what a node must do when it
        receives a transaction before it acknowledges that transaction to the rest of the
        group.</para>
      <para>The following options are available: <itemizedlist>
          <listitem>
            <para><emphasis>SYNC</emphasis>. The node must write the transaction to disk and flush
              any OS level buffers before sending the acknowledgement. SYNC is offers the highest
              durability but offers the least performance.</para>
          </listitem>
          <listitem>
            <para><emphasis>WRITE_NO_SYNC</emphasis>. The node must write the transaction to disk
              before sending the acknowledgement. OS level buffers will be flush as some point
              later. This typically provides an assurance against failure of the application but not
              the operating system or hardware.</para>
          </listitem>
          <listitem>
            <para><emphasis>NO_SYNC</emphasis>. The node immediately sends the acknowledgement. The
              transaction will be written and OS level buffers flushed as some point later. NO_SYNC
              offers the highest performance but the lowest durability level. This synchronization
              policy is sometimes known as <emphasis>commit to the network</emphasis>.</para>
          </listitem>
        </itemizedlist></para>
      <para>It is possible to assign a one policy to the master and a different policy to the
        replicas. These are configured as <link
          linkend="Java-Broker-Management-Managing-Virtualhost-Attributes">attributes on the
          virtualhost</link>. By default the master uses <emphasis>SYNC</emphasis> and replicas use
          <emphasis>NO_SYNC</emphasis>.</para>
    </section>
    <section id="Java-Broker-High-Availability-Behaviour-NodePriority">
      <title>Node Priority</title>
      <para>Node priority can be used to influence the behaviour of the election algorithm. It is
        useful in the case were you want to favour some nodes over others. For instance, if you wish
        to favour nodes located in a particular data centre over those in a remote site. </para>
      <para>The following options are available: <itemizedlist>
          <listitem>
            <para><emphasis>Highest</emphasis>. Nodes with this priority will be more favoured. In
              the event of two or more nodes having the most recent transaction, the node with this
              priority will be elected master. If two or more nodes have this priority the algorithm
              will make an arbitrary choice.</para>
          </listitem>
          <listitem>
            <para><emphasis>High</emphasis>. Nodes with this priority will be favoured but not as
              much so as those with Highest.</para>
          </listitem>
          <listitem>
            <para><emphasis>Normal</emphasis>. This is default election priority.</para>
          </listitem>
          <listitem>
            <para><emphasis>Never</emphasis>. The node will never be elected <emphasis>even if the
                node has the most recent transaction</emphasis>. The node will still keep up to date
              with the replication stream and will still vote itself, but can just never be
              elected.</para>
          </listitem>
        </itemizedlist>
      </para>
      <para>Node priority is configured as an <link
          linkend="Java-Broker-Management-Managing-Virtualhost-Nodes-Attributes">attribute on the
          virtualhost node</link> and can be changed at runtime and is effective immediately.</para>
      <important>
        <para>Use of the Never priority can lead to transaction loss. For example, consider a group
          of three where replica-2 is marked as Never. If a transaction were to arrive and it be
          acknowledged only by Master and Replica-2, the transaction would succeed. Replica 1 is
          running behind for some reason (perhaps a full-GC). If a Master failure were to occur at
          that moment, the replicas would elect Replica-1 even though Replica-2 had the most recent
          transaction.</para>
      </important>
    </section>
    <section id="Java-Broker-High-Availability-Behaviour-MinimumNumberOfNodes">
      <title>Required Minimum Number Of Nodes</title>
      <para>This controls the required minimum number of nodes to complete a transaction and to
        elect a new master. By default, the required number of nodes is set to
          <emphasis>Default</emphasis> (which signifies quorum).</para>
      <para>It is possible to reduce the required minimum number of nodes. The rationale for doing
        this is normally to temporarily restore service from fewer than quorum nodes following an
        extraordinary failure.</para>
      <para>For example, consider a group of three. If one node were to fail, as quorum still
        remained, the system would continue work without any intervention. If the failing node were
        the master, a new master would be elected.</para>
      <para>What if a further node were to fail? Quorum no longer remains, and the remaining node
        would just wait. It cannot elect itself master. What if we wanted to restore service from
        just this one node?</para>
      <para>In this case, Required Number of Nodes can be reduced to 1 on the remain node, allowing
        the node to elect itself and service to be restored from the singleton. Required minimum
        number of nodes is configured as an <link
          linkend="Java-Broker-Management-Managing-Virtualhost-Nodes-Attributes">attribute on the
          virtualhost node</link> and can be changed at runtime and is effective immediately.</para>
      <important>
        <para>The attribute must be used cautiously. Careless use will lead to lost transactions and
          can lead to a <ulink url="http://en.wikipedia.org/wiki/Split-brain_(computing)"
            >split-brain</ulink> in the event of a network partition. If used to temporarily restore
          service from fewer than quorum nodes, it is <emphasis>imperative</emphasis> to revert it
          to the Default value as the failed nodes are restored.</para>
      </important>
    </section>
    <section id="Java-Broker-High-Availability-Behaviour-DesignatedPrimary">
      <title>Designated Primary</title>
      <para>This attribute applies to the groups of two only.</para>
      <para> In a group of two, if a node were to fail then in default configuration work will cease
        as quorum no longer exists. A single node cannot elect itself master. </para>
      <para>The designated primary flag allows a node in a two node group to elect itself master and
        to operate sole. Designated Primary is configured as an <link
          linkend="Java-Broker-Management-Managing-Virtualhost-Nodes-Attributes">attribute on the
          virtualhost node</link> and can be changed at runtime and is effective immediately.</para>
      <para>For example, consider a group of two where the master fails. Service will be interrupted
        as the remaining node cannot elect itself master. To allow it to become master, apply the
        designated primary flag to it. It will elect itself master and work can continue, albeit
        from one node.</para>
      <important>
        <para>It is imperative not to allow designated primary to be set on both nodes at once. To
          do so will mean, in the event of a network partition, a <ulink
            url="http://en.wikipedia.org/wiki/Split-brain_(computing)">split-brain</ulink> will
          occur.</para>
      </important>
    </section>
  </section>
  <section id="Java-Broker-High-Availability-NodeOperations">
    <title>Node Operations</title>
    <section id="Java-Broker-High-Availability-NodeOperations-Lifecycle">
      <title>Lifecycle</title>
      <para>Virtualhost nodes can be stopped, started and deleted.</para>
      <itemizedlist>
        <listitem>
          <para><emphasis>Stop</emphasis></para>
          <para>Stopping a master node will cause the node to temporarily leave the group. Any
            messaging clients will be disconnected and any in-flight transaction rollbacked. The
            remaining nodes will elect a new master if quorum number of nodes still remains.</para>
          <para>Stopping a replica node will cause the node to temporarily leave the group too.
            Providing quorum still exists, the current master will continue without interruption. If
            by leaving the group, quorum no longer exists, all the nodes will begin waiting,
            disconnecting any messaging clients, and the virtualhost will become unavailable.</para>
          <para>A stopped virtualhost node is still considered to be a member of the group.</para>
        </listitem>
        <listitem>
          <para><emphasis>Start</emphasis></para>
          <para>Starting a virtualhost node allows it to rejoin the group.</para>
          <para>If the group already has a master, the node will catch up from the master and then
            become a replica once it has done so.</para>
          <para>If the group did not have quorum and so had no master, but the rejoining of this
            node means quorum now exists, an election will take place. The node with the most up to
            date transaction will become master unless influenced by the priority rules described
            above.</para>
          <note>
            <para>The length of time taken to catch up will depend on how long the node has been
              stopped. The worst case is where the node has been stopped for more than one hour. In
              this case, the master will perform an automated <literal>network restore</literal>.
              This involves streaming all the data held by the master over to the replica. This
              could take considerable time.</para>
          </note>
        </listitem>
        <listitem>
          <para><emphasis>Delete</emphasis></para>
          <para>A virtualhost node can be deleted. Deleting a node permanently removes the node from
            the group. The data stored locally is removed but this does not affect the data held by
            the remainder of the group.</para>
          <note>
            <para>The names of deleted virtualhost node cannot be reused within a group.</para>
          </note>
        </listitem>
      </itemizedlist>
      <para>It is also possible to add nodes to an existing group using the procedure described
        above.</para>
    </section>
    <section id="Java-Broker-High-Availability-NodeOperations-TransferMaster">
      <title>Transfer Master</title>
      <para>This operation allows the mastership to be moved from node to node. This is useful for
        restoring a business as usual state after a failure.</para>
      <para>When using this function, the following occurs. <orderedlist>
          <listitem>
            <para>The system first gives time for the chosen new master to become reasonable up to
              date. </para>
          </listitem>
          <listitem>
            <para>It then suspends transactions on the old master and allows the chosen node to
              become up to date.</para>
          </listitem>
          <listitem>
            <para>The suspended transactions are aborted and any messaging clients connected to the
              old master are disconnected.</para>
          </listitem>
          <listitem>
            <para>The chosen master becomes the new master. The old master becomes a replica.</para>
          </listitem>
          <listitem>
            <para>Messaging clients reconnect the new master.</para>
          </listitem>
        </orderedlist></para>
    </section>
  </section>

  <section id="Java-Broker-High-Availability-ClientFailover">
    <title>Client failover</title>
    <para>As mentioned above, the clients need to be able to find the location of the active
      virtualhost within the group.</para>
    <para>Clients can do this using a static technique, for example , utilising the <ulink
        url="&qpidjmsdocClientConectionUrl;">failover feature of the Qpid connection url</ulink>
      where the client has a list of all the nodes, and tries each node in sequence until it
      discovers the node with the active virtualhost.</para>
    <para>Another possibility is a dynamic technique utilising a proxy or Virtual IP (VIP). These
      require other software and/or hardware and are outside the scope of this document.</para>
  </section>
  <section role="h3" id="Java-Broker-High-Availability-JMXAPI">
    <title>Qpid JMX API for HA</title>
    <para>The Qpid JMX API for HA is now deprecated. New users are recommended to use the <link
        linkend="Java-Broker-Management-Channel-REST-API">REST API.</link></para>
  </section>

  <section id="Java-Broker-High-Availability-DiskSpace">
    <title>Disk space requirements</title>
    <para>In the case where node in a group are down, the master must keep the data they are missing
      for them to allow them to return to the replica role quickly.</para>
    <para>By default, the master will retain up to 1hour of missed transactions. In a busy
      production system, the disk space occupied could be considerable.</para>
    <para>This setting is controlled by virtualhost context variable
        <literal>je.rep.repStreamTimeout</literal>.</para>
  </section>

  <section id="Java-Broker-High-Availability-Network-Requirements">
    <title>Network Requirements</title>
    <para>The HA Cluster performance depends on the network bandwidth, its use by existing traffic,
      and quality of service.</para>
    <para>In order to achieve the best performance it is recommended to use a separate network
      infrastructure for the Qpid HA Nodes which might include installation of dedicated network
      hardware on Broker hosts, assigning a higher priority to replication ports, installing a group
      in a separate network not impacted by any other traffic.</para>
  </section>

  <section id="Java-Broker-High-Availability-Security">
    <title>Security</title>
    <para>The replication stream between the master and the replicas is insecure and can be
      intercepted by anyone having access to the replication network.</para>
    <para>In order to reduce the security risks the entire HA group is recommended to run in a
      separate network protected from general access and/or utilise SSH-tunnels/IPsec.</para>
  </section>

  <section id="Java-Broker-High-Availability-Backup">
    <title>Backups</title>
    <para>It is recommend to use the hot backup script to periodically backup every node in the
      group. <xref linkend="Java-Broker-Backup-And-Recovery-Virtualhost-Node-BDB-HA"/>.</para>
  </section>


  <section id="Java-Broker-High-Availability-Reset-Group-Infomational">
    <title>Reset Group Information</title>
    <para>BDB JE internally stores details of the group within its database. There are some
      circumstances when resetting this information is useful.<itemizedlist>
        <listitem>
          <para>Copying data between environments (e.g. production to UAT)</para>
        </listitem>
        <listitem>
          <para>Some disaster recovery situations where a group must be recreated on new
            hardware</para>
        </listitem>
      </itemizedlist></para>
    <para>This is not an normal operation and is not usually required</para>
    <para>The following command replaces the group table contained within the JE logs files with the
      provided information. </para>
    <example>
      <title>Resetting of replication group with <classname>DbResetRepGroup</classname></title>
      <cmdsynopsis>
        <command>java</command>
        <arg choice="plain">-cp je-&oracleBdbProductVersion;.jar</arg>
        <arg choice="plain">com.sleepycat.je.rep.util.DbResetRepGroup</arg>
        <arg choice="plain">-h <replaceable>path/to/jelogfiles</replaceable></arg>
        <sbr/>
        <arg choice="plain">-groupName <replaceable>newgroupname</replaceable></arg>
        <arg choice="plain">-nodeName <replaceable>newnodename</replaceable></arg>
        <arg choice="plain">-nodeHostPort <replaceable>newhostname:5000</replaceable></arg>
      </cmdsynopsis>
    </example>
    <para>The modified log files can then by copied into
        <literal>${QPID_WORK}/&lt;nodename&gt;/config</literal> directory of a target Broker. Then
      start the Broker, and add a BDB HA Virtualhost node specify the same group name, node name and
      node address. You will then have a group with a single node, ready to start re-adding
      additional nodes as described above. </para>
  </section>

</chapter>
