1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
|
<?xml version="1.0" encoding="utf-8"?>
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<section id="How-to-Tune-M3-Java-Broker-Performance">
<title>
How to Tune M3 Java Broker Performance
</title>
<section role="h3" id="HowtoTuneM3JavaBrokerPerformance-ProblemStatement">
<title>
Problem
Statement
</title>
<para>
During destructive testing of the Qpid M3 Java Broker, we tested
some tuning techniques and deployment changes to improve the Qpid
M3 Java Broker's capacity to maintain high levels of throughput,
particularly in the case of a slower consumer than produceer
(i.e. a growing backlog).
</para>
<para>
The focus of this page is to detail the results of tuning &
deployment changes trialled.
</para>
<para>
The successful tuning changes are applicable for any deployment
expecting to see bursts of high volume throughput (1000s of
persistent messages in large batches). Any user wishing to use
these options <emphasis>must test them thoroughly in their own
environment with representative volumes</emphasis>.
</para>
<!--h3-->
</section>
<section role="h3" id="HowtoTuneM3JavaBrokerPerformance-SuccessfulTuningOptions">
<title>
Successful
Tuning Options
</title>
<para>
The key scenario being taregetted by these changes is a broker
under heavy load (processing a large batch of persistent
messages)can be seen to perform slowly when filling up with an
influx of high volume transient messages which are queued behind
the persistent backlog. However, the changes suggested will be
equally applicable to general heavy load scenarios.
</para>
<para>
The easiest way to address this is to separate streams of
messages. Thus allowing the separate streams of messages to be
processed, and preventing a backlog behind a particular slow
consumer.
</para>
<para>
These strategies have been successfully tested to mitigate this
problem:
</para>
<table>
<title/>
<tgroup cols="2">
<tbody>
<row>
<entry>
Strategy
</entry>
<entry>
Result
</entry>
</row>
<row>
<entry>
Seperate connections to one broker for separate streams of
messages.
</entry>
<entry>
Messages processed successfully, no problems experienced
</entry>
</row>
<row>
<entry>
Seperate brokers for transient and persistent messages.
</entry>
<entry>
Messages processed successfully, no problems experienced
</entry>
</row>
</tbody>
</tgroup>
</table>
<para>
<emphasis>Separate Connections</emphasis>
Using separate connections effectively means that the two streams
of data are not being processed via the same buffer, and thus the
broker gets & processes the transient messages while
processing the persistent messages. Thus any build up of
unprocessed data is minimal and transitory.
</para>
<para>
<emphasis>Separate Brokers</emphasis>
Using separate brokers may mean more work in terms of client
connection details being changed, and from an operational
perspective. However, it is certainly the most clear cut way of
isolating the two streams of messages and the heaps impacted.
</para>
<section role="h4" id="HowtoTuneM3JavaBrokerPerformance-Additionaltuning">
<title>
Additional
tuning
</title>
<para>
It is worth testing if changing the size of the Qpid read/write
thread pool improves performance (eg. by setting
JAVA_OPTS="-Damqj.read_write_pool_size=32" before running
qpid-server). By default this is equal to the number of CPU
cores, but a higher number may show better performance with some
work loads.
</para>
<para>
It is also important to note that you should give the Qpid broker
plenty of memory - for any serious application at least a -Xmx of
3Gb. If you are deploying on a 64 bit platform, a larger heap is
definitely worth testing with. We will be testing tuning options
around a larger heap shortly.
</para>
<!--h4-->
</section>
<!--h3-->
</section>
<section role="h3" id="HowtoTuneM3JavaBrokerPerformance-NextSteps">
<title>
Next
Steps
</title>
<para>
These two options have been testing using a Qpid test case, and
demonstrated that for a test case with a profile of persistent
heavy load following by constant transient high load traffic they
provide significant improvment.
</para>
<para>
However, the deploying project <emphasis>must</emphasis> complete their own
testing, using the same destructive test cases, representative
message paradigms & volumes, in order to verify the proposed
mitigation options.
</para>
<para>
The using programme should then choose the option most applicable
for their deployment and perform BAU testing before any
implementation into a production or pilot environment.
</para>
<!--h3-->
</section>
</section>
|
