1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
|
<?xml version="1.0" encoding="utf-8"?>
<!DOCTYPE entities [
<!ENTITY % entities SYSTEM "commonEntities.xml">
%entities;
]>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<section id="Java-Broker-Queues-OtherTypes">
<title>Other Queue Types</title>
<section role="h2" id="Java-Broker-Queues-OtherTypes-Introduction">
<title>Introduction</title>
<para> In addition to the standard queue type where messages are delivered in the same order
that they were sent, the Java Broker supports four additional queue types which allows for
alternative delivery behaviours. These are
<link linkend="Java-Broker-Queues-OtherTypes-Priority">priority-queues</link>,
<link linkend="Java-Broker-Queues-OtherTypes-Sorted">sorted-queues</link>-,
<link linkend="Java-Broker-Queues-OtherTypes-LVQ">last-value-queues</link> (LVQs), and
<link linkend="Java-Broker-Queues-OtherTypes-Message-Grouping">grouped queues</link>.
</para>
<para> In the following sections, the semantics of each queue type is described, followed by a
description of how instances of these queue can be created via <link
linkend="Java-Broker-Queues-OtherTypes-CreateUsingConfig">configuration</link> or <link
linkend="Java-Broker-Queues-OtherTypes-CreateUsingJmsOrJmx">programmatically</link>. </para>
<para>The final section discusses the importance of using a <link
linkend="Java-Broker-Queues-OtherTypes-SetLowPrefetch">low client pre-fetch</link> with these queued.
</para>
</section>
<section role="h2" id="Java-Broker-Queues-OtherTypes-Priority">
<title>Priority Queues</title>
<para>In a priority queue, messages on the queue are delivered in an order determined by the
<ulink url="&oracleJeeDocUrl;javax/jms/Message.html#getJMSPriority()">JMS priority message
header</ulink> within the message. By default Qpid supports the 10 priority levels mandated
by JMS, with priority value 0 as the lowest priority and 9 as the highest. </para>
<para>It is possible to reduce the effective number of priorities if desired.</para>
<para>JMS defines the <ulink url="&oracleJeeDocUrl;javax/jms/Message.html#DEFAULT_PRIORITY">
default message priority</ulink> as 4. Messages sent without a specified priority use this
default. </para>
</section>
<section role="h2" id="Java-Broker-Queues-OtherTypes-Sorted">
<title>Sorted Queues</title>
<para>Sorted queues allow the message delivery order to be determined by value of an arbitrary
<ulink url="&oracleJeeDocUrl;javax/jms/Message.html#getStringProperty()">JMS message
property</ulink>. Sort order is alpha-numeric and the property value must have a type
java.lang.String.</para>
<para>Messages sent to a sorted queue without the specified JMS message property will be
inserted into the 'last' position in the queue.</para>
</section>
<section role="h2" id="Java-Broker-Queues-OtherTypes-LVQ">
<title>Last Value Queues (LVQ)</title>
<para>LVQs (or conflation queues) are special queues that automatically discard any message when
a newer message arrives with the same key value. The key is specified by arbitrary <ulink
url="&oracleJeeDocUrl;javax/jms/Message.html#getPropertyNames()">JMS message
property</ulink>.</para>
<para>An example of an LVQ might be where a queue represents prices on a stock exchange: when
you first consume from the queue you get the latest quote for each stock, and then as new
prices come in you are sent only these updates. </para>
<para>Like other queues, LVQs can either be browsed or consumed from. When browsing an
individual subscriber does not remove the message from the queue when receiving it. This
allows for many subscriptions to browse the same LVQ (i.e. you do not need to create and bind
a separate LVQ for each subscriber who wishes to receive the contents of the LVQ).</para>
<para>Messages sent to an LVQ without the specified property will be delivered as normal and
will never be "replaced".</para>
</section>
<section role="h2" id="Java-Broker-Queues-OtherTypes-Create">
<title>Creating a Priority, Sorted or LVQ Queue</title>
<para>To create a priority, sorted or LVQ queue, it can be defined in the virtualhost
configuration file, or the queue can be created programmtically from a client via AMQP (using
an extension to JMS), or using JMX. These methods are described below. </para>
<para>Once a queue is created you cannot change its type (without deleting it and re-creating).
Also note you cannot currently mix the natures of these queue types, for instance, you cannot
define a queue which it both an LVQ and a priority-queue.</para>
<section role="h2" id="Java-Broker-Queues-OtherTypes-CreateUsingConfig">
<title>Using configuration</title>
<para>To create a priority, sorted or LVQ queue within configuration, add the appropriate xml
to the virtualhost.xml configuration file within the <varname>queues</varname>
element.</para>
<section role="h3" id="Java-Broker-Queues-OtherTypes-CreateUsingConfig-Priority">
<title>Priority</title>
<para> To defining a priority queue, add a <priority>true</priority> element. By
default the queue will have 10 distinct priorities. </para>
<example>
<title>Configuring a priority queue</title>
<programlisting><![CDATA[<queue>
<name>myqueue</name>
<myqueue>
<exchange>amq.direct</exchange>
<priority>true</priority>
</myqueue>
</queue>]]></programlisting>
</example>
<para> If you require fewer priorities, it is possible to specify a
<varname>priorities</varname> element (whose value is a integer value between 2 and 10
inclusive) which will give the queue that number of distinct priorities. When messages are
sent to that queue, their effective priority will be calculated by partitioning the
priority space. If the number of effective priorities is 2, then messages with priority
0-4 are treated the same as "lower priority" and messages with priority 5-9 are treated
equivalently as "higher priority". </para>
<example>
<title>Configuring a priority queue with fewer priorities</title>
<programlisting><![CDATA[<queue>
<name>myqueue</name>
<myqueue>
<exchange>amq.direct</exchange>
<priority>true</priority>
<priorities>4</priorities>
</myqueue>
</queue>]]></programlisting>
</example>
</section>
<section role="h3" id="Java-Broker-Queues-OtherTypes-CreateUsingConfig-Sorted">
<title>Sorted</title>
<para> To define a sorted queue, add a <varname>sortKey</varname> element. The value of the
<varname>sortKey</varname> element defines the message property to use the value of when
sorting the messages put onto the queue. </para>
<example>
<title>Configuring a sorted queue</title>
<programlisting><![CDATA[<queue>
<name>myqueue</name>
<myqueue>
<exchange>amq.direct</exchange>
<sortKey>message-property-to-sort-by</sortKey>
</myqueue>
</queue>]]></programlisting>
</example>
</section>
<section role="h3" id="Java-Broker-Queues-OtherTypes-CreateUsingConfig-LVQ">
<title>LVQ</title>
<para> To define a LVQ, add a <varname>lvq</varname> element with the value
<constant>true</constant>. Without any further configuration this will define an LVQ
which uses the JMS message property <constant>qpid.LVQ_key</constant> as the key for
replacement. </para>
<example>
<title>Configuring a LVQ queue</title>
<programlisting><![CDATA[<queue>
<name>myqueue</name>
<myqueue>
<exchange>amq.direct</exchange>
<lvq>true</lvq>
</myqueue>
</queue>]]></programlisting>
</example>
<para> If you wish to define your own property then you can do so using the
<varname>lvqKey</varname> element.</para>
<example>
<title>Configuring a LVQ queue with custom message property name</title>
<programlisting><![CDATA[<queue>
<name>myqueue</name>
<myqueue>
<exchange>amq.direct</exchange>
<lvq>true</lvq>
<lvqKey>ISIN</lvqKey>
</myqueue>
</queue>]]></programlisting>
</example>
</section>
</section>
<section role="h2" id="Java-Broker-Queues-OtherTypes-CreateUsingJmsOrJmx">
<title>Using JMX or AMQP</title>
<para>To create a priority, sorted or LVQ queue programmatically from JMX or using a Qpid
extension to JMS, pass the appropriate queue-declare arguments.</para>
<table>
<title>Queue-declare arguments understood for priority, sorted and LVQ queues</title>
<tgroup cols="4">
<thead>
<row>
<entry>Queue type</entry>
<entry>Argument name</entry>
<entry>Argument name</entry>
<entry>Argument Description</entry>
</row>
</thead>
<tbody>
<row>
<entry>priority</entry>
<entry>priorities</entry>
<entry>java.lang.Integer</entry>
<entry>Specifies a priority queue with given number priorities</entry>
</row>
<row>
<entry>sorted</entry>
<entry>qpid.queue_sort_key</entry>
<entry>java.lang.String</entry>
<entry>Specifies sorted queue with given message property used to sort the
entries</entry>
</row>
<row>
<entry>lvq</entry>
<entry>qpid.last_value_queue_key</entry>
<entry>java.lang.String</entry>
<entry>Specifies lvq queue with given message property used to conflate the
entries</entry>
</row>
</tbody>
</tgroup>
</table>
<para>The following example illustrates the creation of the a LVQ queue from a
javax.jms.Session object. Note that this utilises a Qpid specific extension to JMS and
involves casting the session object back to its Qpid base-class.</para>
<example>
<title>Creation of an LVQ using the Qpid extension to JMS</title>
<programlisting><![CDATA[Map<String,Object> arguments = new HashMap<String, Object>();
arguments.put("qpid.last_value_queue_key","ISIN");
((AMQSession<?,?>) session).createQueue(queueName, autoDelete, durable, exclusive, arguments);]]></programlisting>
</example>
<para> The following example illustrates the creation of the sorted queue from a the JMX
interface using the ManagedBroker interface. </para>
<example>
<title>Creation of a sorted queue using JMX</title>
<programlisting><![CDATA[Map<String, Object> environment = new HashMap<String, Object>();
environment.put(JMXConnector.CREDENTIALS, new String[] {"admin","password"});
// Connect to service
JMXServiceURL url = new JMXServiceURL("service:jmx:rmi:///jndi/rmi://localhost:8999/jmxrmi");
JMXConnector jmxConnector = JMXConnectorFactory.connect(url, environment);
MBeanServerConnection mbsc = jmxConnector.getMBeanServerConnection();
// Object name for ManagedBroker for virtualhost myvhost
ObjectName objectName = new ObjectName("org.apache.qpid:type=VirtualHost.VirtualHostManager,VirtualHost=myvhost");
// Get the ManagedBroker object
ManagedBroker managedBroker = JMX.newMBeanProxy(mbsc, objectName, ManagedBroker.class);;
// Create the queue passing arguments
Map<String,Object> arguments = new HashMap<String, Object>();
arguments.put("qpid.queue_sort_key","myheader");
managedBroker.createNewQueue("myqueue", null, true, arguments);]]></programlisting>
</example>
</section>
</section>
<section role="h2" id="Java-Broker-Queues-OtherTypes-Message-Grouping">
<title>
Messaging Grouping
</title>
<para>
The broker allows messaging applications to classify a set of related messages as
belonging to a group. This allows a message producer to indicate to the consumer
that a group of messages should be considered a single logical operation with
respect to the application.
</para>
<para>
The broker can use this group identification to enforce policies controlling how
messages from a given group can be distributed to consumers. For instance, the
broker can be configured to guarantee all the messages from a particular group are
processed in order across multiple consumers.
</para>
<para>
For example, assume we have a shopping application that manages items in a virtual
shopping cart. A user may add an item to their shopping cart, then change their
mind and remove it. If the application sends an <emphasis>add</emphasis> message to the broker,
immediately followed by a <emphasis>remove</emphasis> message, they will be queued in the proper
order - <emphasis>add</emphasis>, followed by <emphasis>remove</emphasis>.
</para>
<para>
However, if there are multiple consumers, it is possible that once a consumer
acquires the <emphasis>add</emphasis> message, a different consumer may acquire the
<emphasis>remove</emphasis> message. This allows both messages to be processed in parallel,
which could result in a "race" where the <emphasis>remove</emphasis> operation is incorrectly
performed before the <emphasis>add</emphasis> operation.
</para>
<section role="h3" id="Java-Broker-Queues-OtherTypes-GroupingMessages">
<title>
Grouping Messages
</title>
<para>
In order to group messages, the application would designate a particular
message header as containing a message's <emphasis>group identifier</emphasis>. The group
identifier stored in that header field would be a string value set by the message
producer. Messages from the same group would have the same group identifier
value. The key that identifies the header must also be known to the message
consumers. This allows the consumers to determine a message's assigned group.
</para>
<para>
The header that is used to hold the group identifier, as well as the values used
as group identifiers, are totally under control of the application.
</para>
</section>
<section role="h3" id="Java-Broker-Queues-OtherTypes-BrokerRole">
<title>
The Role of the Broker in Message Grouping
</title>
<para>
The broker will apply the following processing on each grouped message:
<itemizedlist>
<listitem>Enqueue a received message on the destination queue.</listitem>
<listitem>Determine the message's group by examining the message's group identifier header.</listitem>
<listitem>Enforce <emphasis>consumption ordering</emphasis> among messages belonging
to the same group. <emphasis>Consumption ordering</emphasis> means one of two things
depending on how the queue has been configured.
<itemizedlist>
<listitem>In default mode, each group is assigned to a consumer for
the lifetime of the consumer.</listitem>
<listitem>In C++ compatibility mode (which gives the same behaviour
as the C++ Qpid Broker), the Broker enforces a looser guarantee, nameley that all the
<emphasis>currently unacknowledged messages</emphasis> in a group will be sent to the
same consumer. This means that only one consumer can be processing messages from a particular
group at a given time. When the consumer acknowledges all of its acquired
messages, then the broker <emphasis>may</emphasis> pass the next pending message
from that group to a different consumer.</listitem>
</itemizedlist>
</listitem>
</itemizedlist>
</para>
<para>
The absence of a value in the designated header field for grouping as treated as indicative
of a lack of desire for the message to be grouped. Messages with such a lack of a value will
be distributed to any available consumer.
</para>
<para>
Note that message grouping has no effect on queue browsers.
</para>
<para>
Note well that distinct message groups would not block each other from delivery.
For example, assume a queue contains messages from two different message groups -
say group "A" and group "B" - and they are enqueued such that "A"'s messages are
in front of "B". If the first message of group "A" is in the process of being
consumed by a client, then the remaining "A" messages are blocked, but the
messages of the "B" group are available for consumption by other consumers - even
though it is "behind" group "A" in the queue.
</para>
</section>
<section role="h3" id="Java-Broker-Queues-OtherTypes-BrokerConfig">
<title>
Broker Message Grouping Configuration
</title>
<para>
In order for the broker to determine a message's group, the key for the header
that contains the group identifier must be provided to the broker via
configuration. This is done on a per-queue basis, when the queue is first
configured.
</para>
<para>
This means that message group classification is determined by the message's destination
queue.
</para>
<para>
Specifically, the queue "holds" the header key that is used to find the message's
group identifier. All messages arriving at the queue are expected to use the same
header key for holding the identifer. Once the message is enqueued, the broker
looks up the group identifier in the message's header, and classifies the message
by its group.
</para>
<para>
Message group support is specified by providing one or more of the following settings
in the arguments map that is used when declaring the queue (e.g. when calling
<code>AMQSession.createQueue()</code>).
<table>
<title>Queue Declare Message Group Configuration Arguments</title>
<tgroup cols="2">
<thead>
<row>
<entry>Key</entry>
<entry>Value</entry>
</row>
</thead>
<tbody>
<row>
<entry>qpid.group_header_key</entry>
<entry>The name of the message header that holds the group identifier value.
The values in this header may be of any supported type (i.e. not just strings).
</entry>
</row>
<row>
<entry>qpid.shared_msg_group</entry>
<entry>Provide a value of "1" to switch on
<link linkend="Java-Broker-Queues-OtherTypes-BrokerRole">C++ compatibility mode</link></entry>
</row>
</tbody>
</tgroup>
</table>
</para>
<para>
It is important to note that there is no need to provide the actual group
identifer values that will be used. The broker learns these values as messages are
recieved. Also, there is no practical limit - aside from resource limitations -
to the number of different groups that the broker can track at run time.
</para>
</section>
</section>
<section role="h2" id="Java-Broker-Queues-OtherTypes-SetLowPrefetch">
<title>Using low pre-fetch with special queue types</title>
<para>Qpid clients receive buffered messages in batches, sized according to the pre-fetch value.
The current default is 500. </para>
<para>However, if you use the default value you will probably <emphasis>not</emphasis> see
desirable behaviour when using priority, sorted, lvq or grouped queues. Once the broker has sent a
message to the client its delivery order is then fixed, regardless of the special behaviour of
the queue. </para>
<para>For example, if using a priority queue and a prefetch of 100, and 100 messages arrive with
priority 2, the broker will send these messages to the client. If then a new message arrives
will priority 1, the broker cannot leap frog messages of lower priority. The priority 1 will
be delivered at the front of the next batch of messages to be sent to the client.</para>
<para> So, you need to set the prefetch values for your client (consumer) to make this sensible.
To do this set the Java system property <varname>max_prefetch</varname> on the client
environment (using -D) before creating your consumer. </para>
<para>A default for all client connections can be set via a system property: </para>
<programlisting>
-Dmax_prefetch=1
</programlisting>
<para> The prefetch can be also be adjusted on a per connection basis by adding a
<varname>maxprefetch</varname> value to the <ulink url="../../Programming-In-Apache-Qpid/html/QpidJNDI.html#section-jms-connection-url">Connection URLs</ulink>
</para>
<programlisting>
amqp://guest:guest@client1/development?maxprefetch='1'&brokerlist='tcp://localhost:5672'
</programlisting>
<para>Setting the Qpid pre-fetch to 1 will give exact queue-type semantics as perceived by the
client however, this brings a performance cost. You could test with a slightly higher
pre-fetch to trade-off between throughput and exact semantics.</para>
</section>
</section>
|