1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
|
<?xml version="1.0" encoding="utf-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<section id="queue-state-replication">
<title>
Queue State Replication
</title>
<section role="h2" id="queuestatereplication-AsynchronousReplicationofQueueState">
<title>
Asynchronous
Replication of Queue State
</title>
<section role="h3" id="queuestatereplication-Overview">
<title>
Overview
</title>
<para>
There is support in qpidd for selective asynchronous replication
of queue state. This is achieved by:
</para>
<para>
(a) enabling event generation for the queues in question
</para>
<para>
(b) loading a plugin on the 'source' broker to encode those
events as messages on a replication queue (this plugin is
called
replicating_listener.so)
</para>
<para>
(c) loading a custom exchange plugin on the 'backup' broker (this
plugin is called replication_exchange.so)
</para>
<para>
(d) creating an instance of the replication exchange type on the
backup broker
</para>
<para>
(e) establishing a federation bridge between the replication
queue on the source broker and the replication exchange on the
backup broker
</para>
<para>
The bridge established between the source and backup brokers for
replication (step (e) above) should have acknowledgements turned
on (this may be done through the --ack N option to qpid-route).
This ensures that replication events are not lost if the bridge
fails.
</para>
<para>
The replication protocol will also eliminate duplicates to ensure
reliably replicated state. Note though that only one bridge per
replication exchange is supported. If clients try to publish to
the replication exchange or if more than a the single required
bridge from the replication queue on the source broker is
created, replication will be corrupted. (Access control may be
used to restrict access and help prevent this).
</para>
<para>
The replicating event listener plugin (step (b) above) has the
following options:
</para>
<programlisting>
Queue Replication Options:
--replication-queue QUEUE Queue on which events for
other queues are recorded
--replication-listener-name NAME (replicator) name by which to register the
replicating event listener
--create-replication-queue if set, the replication will
be created if it does not
exist
</programlisting>
<para>
The name of the queue is required. It can either point to a
durable queue whose definition has been previously recorded, or
the --create-replication-queue option can be specified in which
case the queue will be created a simple non-durable queue if it
does not already exist.
</para>
<!--h3-->
</section>
<section role="h3" id="queuestatereplication-UsewithClustering">
<title>
Use with
Clustering
</title>
<para>
The source and/or backup brokers may also be clustered brokers.
In this case the federated bridge will be re-established between
replicas should either of the originally connected nodes fail.
There are however the following limitations at present:
</para>
<itemizedlist>
<listitem>
<para>The backup site does not process membership updates after it
establishes the first connection. In order for newly added
members on a source cluster to be eligible as failover targets,
the bridge must be recreated after those members have been added
to the source cluster.
</para>
</listitem>
</itemizedlist>
<itemizedlist>
<listitem>
<para>New members added to a backup cluster will not receive
information about currently established bridges. Therefore in
order to allow the bridge to be re-established from these members
in the event of failure of older nodes, the bridge must be
recreated after the new members have joined.
</para>
</listitem>
</itemizedlist>
<itemizedlist>
<listitem>
<para>Only a single URL can be passed to create the initial link
from backup site to the primary site. this means that at the time
of creating the initial connection the initial node in the
primary site to which the connection is made needs to be running.
Once connected the backup site will receive a membership update
of all the nodes in the primary site, and if the initial
connection node in the primary fails, the link will be
re-established on the next node that was started (time) on the
primary site.
</para>
</listitem>
</itemizedlist>
<para>
Due to the acknowledged transfer of events over the bridge (see
note above) manual recreation of the bridge and automatic
re-establishment of te bridge after connection failure (including
failover where either or both ends are clustered brokers) will
not result in event loss.
</para>
<!--h3-->
</section>
<section role="h3" id="queuestatereplication-OperationsonBackupQueues">
<title>
Operations
on Backup Queues
</title>
<para>
When replicating the state of a queue to a backup broker it is
important to recognise that any other operations performed
directly on the backup queue may break the replication.
</para>
<para>
If the backup queue is to be an active (i.e. accessed by clients
while replication is on) only enqueues should be selected
for
replication. In this mode, any message enqueued on the source
brokers copy of the queue will also be enqueued on the backup
brokers copy. However not attempt will be made to remove messages
from the backup queue in response to removal of messages from the
source queue.
</para>
<!--h3-->
</section>
<section role="h3" id="queuestatereplication-SelectingQueuesforReplication">
<title>
Selecting
Queues for Replication
</title>
<para>
Queues are selected for replication by specifying the types of
events they should generate (it is from these events that the
replicating plugin constructs messages which are then pulled and
processed by the backup site). This is done through options
passed to the initial queue-declare command that creates the
queue and may be done either through qpid-config or similar
tools, or by the application.
</para>
<para>
With qpid-config, the --generate-queue-events options is used:
</para>
<programlisting>
--generate-queue-events N
If set to 1, every enqueue will generate an event that can be processed by
registered listeners (e.g. for replication). If set to 2, events will be
generated for enqueues and dequeues
</programlisting>
<para>
From an application, the arguments field of the queue-declare
AMQP command is used to convey this information. An entry should
be added to the map with key 'qpid.queue_event_generation' and an
integer value of 1 (to replicate only enqueue events) or 2 (to
replicate both enqueue and dequeue events).
</para>
<para>
Applications written using the c++ client API may fine the
qpid::client::QueueOptions class convenient. This has a
enableQueueEvents() method on it that can be used to set the
option (the instance of QueueOptions is then passed as the value
of the arguments field in the queue-declare command. The boolean
option to that method should be set to true if only enequeue
events should be replicated; by default it is false meaning that
both enqueues and dequeues will be replicated. E.g.
</para>
<programlisting>
QueueOptions options;
options.enableQueueEvents(false);
session.queueDeclare(arg::queue="my-queue", arg::arguments=options);
</programlisting>
<!--h3-->
</section>
<section role="h3" id="queuestatereplication-Example">
<title>
Example
</title>
<para>
Lets assume we will run the primary broker on host1 and the
backup on host2, have installed qpidd on both and have the
replicating_listener and replication_exchange plugins in qpidd's
module directory(*1).
</para>
<para>
On host1 we start the source broker and specifcy that a queue
called 'replication' should be used for storing the events until
consumed by the backup. We also request that this queue be
created (as transient) if not already specified:
</para>
<programlisting>
qpidd --replication-queue replication-queue --create-replication-queue true --log-enable info+
</programlisting>
<para>
On host2 we start up the backup broker ensuring that the
replication exchange module is loaded:
</para>
<programlisting>
qpidd
</programlisting>
<para>
We can then create the instance of that replication exchange that
we will use to process the events:
</para>
<programlisting>
qpid-config -a host2 add exchange replication replication-exchange
</programlisting>
<para>
If this fails with the message "Exchange type not implemented:
replication", it means the replication exchange module was
not
loaded. Check that the module is installed on your system and if
necessary provide the full path to the library.
</para>
<para>
We then connect the replication queue on the source broker with
the replication exchange on the backup broker using the
qpid-route command:
</para>
<programlisting>
qpid-route --ack 50 queue add host2 host1 replication-exchange replication-queue
</programlisting>
<para>
The example above configures the bridge to acknowledge messages
in batches of 50.
</para>
<para>
Now create two queues (on both source and backup brokers), one
replicating both enqueues and dequeues (queue-a) and the
other
replicating only dequeues (queue-b):
</para>
<programlisting>
qpid-config -a host1 add queue queue-a --generate-queue-events 2
qpid-config -a host1 add queue queue-b --generate-queue-events 1
qpid-config -a host2 add queue queue-a
qpid-config -a host2 add queue queue-b
</programlisting>
<para>
We are now ready to use the queues and see the replication.
</para>
<para>
Any message enqueued on queue-a will be replicated to the backup
broker. When the message is acknowledged by a client connected to
host1 (and thus dequeued), that message will be removed from the
copy of the queue on host2. The state of queue-a on host2 will
thus mirror that of the equivalent queue on host1, albeit with a
small lag. (Note
however that we must not have clients connected to host2 publish
to-or consume from- queue-a or the state will fail to replicate
correctly due to conflicts).
</para>
<para>
Any message enqueued on queue-b on host1 will also be enqueued on
the equivalent queue on host2. However the acknowledgement and
consequent dequeuing of messages from queue-b on host1 will have
no effect on the state of queue-b on host2.
</para>
<para>
(*1) If not the paths in the above may need to be modified. E.g.
if using modules built from a qpid svn checkout, the following
would be added to the command line used to start qpidd on host1:
</para>
<programlisting>
--load-module <path-to-qpid-dir>/src/.libs/replicating_listener.so
</programlisting>
<para>
and the following for the equivalent command line on host2:
</para>
<programlisting>
--load-module <path-to-qpid-dir>/src/.libs/replication_exchange.so
</programlisting>
<!--h3-->
</section>
<!--h2-->
</section>
</section>
|