1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
|
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
<TITLE>Running a High-Performance Web Server for BSD</TITLE>
</HEAD>
<!-- Background white, links blue (unvisited), navy (visited), red (active) -->
<BODY
BGCOLOR="#FFFFFF"
TEXT="#000000"
LINK="#0000FF"
VLINK="#000080"
ALINK="#FF0000"
>
<A NAME="initial">
<!--#include virtual="header.html" -->
</A>
<H1 ALIGN="CENTER">Running a High-Performance Web Server for BSD</H1>
Like other OS's, the listen queue is often the <STRONG>first limit
hit</STRONG>. The
following are comments from "Aaron Gifford <agifford@InfoWest.COM>"
on how to fix this on BSDI 1.x, 2.x, and FreeBSD 2.0 (and earlier):
<P>
Edit the following two files:
<BLOCKQUOTE><CODE> /usr/include/sys/socket.h <BR>
/usr/src/sys/sys/socket.h </CODE></BLOCKQUOTE>
In each file, look for the following:
<PRE>
/*
* Maximum queue length specifiable by listen.
*/
#define SOMAXCONN 5
</PRE>
Just change the "5" to whatever appears to work. I bumped the two
machines I was having problems with up to 32 and haven't noticed the
problem since.
<P>
After the edit, recompile the kernel and recompile the Apache server
then reboot.
<P>
FreeBSD 2.1 seems to be perfectly happy, with SOMAXCONN
set to 32 already.
<P>
<A NAME="detail">
<STRONG>Addendum for <EM>very</EM> heavily loaded BSD servers</STRONG><BR>
</A>
from Chuck Murcko <chuck@telebase.com>
<P>
If you're running a really busy BSD Apache server, the following are useful
things to do if the system is acting sluggish:<P>
<UL>
<LI> Run vmstat to check memory usage, page/swap rates, <EM>etc.</EM>
<LI> Run netstat -m to check mbuf usage
<LI> Run fstat to check file descriptor usage
</UL>
These utilities give you an idea what you'll need to tune in your kernel,
and whether it'll help to buy more RAM.
Here are some BSD kernel config parameters (actually BSDI, but pertinent to
FreeBSD and other 4.4-lite derivatives) from a system getting heavy usage.
The tools mentioned above were used, and the system memory was increased to
48 MB before these tuneups. Other system parameters remained unchanged.
<P>
<PRE>
maxusers 256
</PRE>
Maxusers drives a <EM>lot</EM> of other kernel parameters:
<UL>
<LI> Maximum # of processes
<LI> Maximum # of processes per user
<LI> System wide open files limit
<LI> Per-process open files limit
<LI> Maximum # of mbuf clusters
<LI> Proc/pgrp hash table size
</UL>
The actual formulae for these derived parameters are in
<EM>/usr/src/sys/conf/param.c</EM>.
These calculated parameters can also be overridden (in part) by specifying
your own values in the kernel configuration file:
<PRE>
# Network options. NMBCLUSTERS defines the number of mbuf clusters and
# defaults to 256. This machine is a server that handles lots of traffic,
# so we crank that value.
options NMBCLUSTERS=4096 # mbuf clusters at 4096
#
# Misc. options
#
options CHILD_MAX=512 # maximum number of child processes
options OPEN_MAX=512 # maximum fds (breaks RPC svcs)
</PRE>
<P>
In many cases, NMBCLUSTERS must be set much larger than would appear
necessary at first glance. The reason for this is that if the browser
disconnects in mid-transfer, the socket fd associated with that particular
connection ends up in the TIME_WAIT state for several minutes, during
which time its mbufs are not yet freed. Another reason is that, on server
timeouts, some connections end up in FIN_WAIT_2 state forever, because
this state doesn't time out on the server, and the browser never sent
a final FIN. For more details see the
<A HREF="fin_wait_2.html">FIN_WAIT_2</A> page.
<P>
Some more info on mbuf clusters (from sys/mbuf.h):
<PRE>
/*
* Mbufs are of a single size, MSIZE (machine/machparam.h), which
* includes overhead. An mbuf may add a single "mbuf cluster" of size
* MCLBYTES (also in machine/machparam.h), which has no additional overhead
* and is used instead of the internal data area; this is done when
* at least MINCLSIZE of data must be stored.
*/
</PRE>
<P>
CHILD_MAX and OPEN_MAX are set to allow up to 512 child processes (different
than the maximum value for processes per user ID) and file descriptors.
These values may change for your particular configuration (a higher OPEN_MAX
value if you've got modules or CGI scripts opening lots of connections or
files). If you've got a lot of other activity besides httpd on the same
machine, you'll have to set NPROC higher still. In this example, the NPROC
value derived from maxusers proved sufficient for our load.
<P>
To increase the size of the <CODE>listen()</CODE> queue, you need to
adjust the value of SOMAXCONN. SOMAXCONN is not derived from maxusers,
so you'll always need to increase that yourself. We use a value guaranteed
to be larger than Apache's default for the listen() of 128, currently.
The actual value for SOMAXCONN is set in <CODE>sys/socket.h</CODE>.
The best way to adjust this parameter is run-time, rather than changing
it in this header file and thus hardcoding a value in the kernel and
elsewhere. To do this, edit <CODE>/etc/rc.local</CODE> and add the
following line:
<PRE>
/usr/sbin/sysctl -w kern.somaxconn=256
</PRE>
<P>
We used <CODE>256</CODE> but you can tune it for your own setup. In
many cases, however, even the default value of <CODE>128</CODE> (for
later versions of FreeBSD) is OK.
<P>
<STRONG>Caveats</STRONG>
<P>
Be aware that your system may not boot with a kernel that is configured
to use more resources than you have available system RAM.
<STRONG>ALWAYS</STRONG>
have a known bootable kernel available when tuning your system this way,
and use the system tools beforehand to learn if you need to buy more
memory before tuning.
<P>
RPC services will fail when the value of OPEN_MAX is larger than 256.
This is a function of the original implementations of the RPC library,
which used a byte value for holding file descriptors. BSDI has partially
addressed this limit in its 2.1 release, but a real fix may well await
the redesign of RPC itself.
<P>
Finally, there's the hard limit of child processes configured in Apache.
<P>
For versions of Apache later than 1.0.5 you'll need to change the
definition for <STRONG>HARD_SERVER_LIMIT</STRONG> in <EM>httpd.h</EM> and
recompile if you need to run more than the default 150 instances of httpd.
<P>
From conf/httpd.conf-dist:
<PRE>
# Limit on total number of servers running, <EM>i.e.</EM>, limit on the number
# of clients who can simultaneously connect --- if this limit is ever
# reached, clients will be LOCKED OUT, so it should NOT BE SET TOO LOW.
# It is intended mainly as a brake to keep a runaway server from taking
# Unix with it as it spirals down...
MaxClients 150
</PRE>
Know what you're doing if you bump this value up, and make sure you've
done your system monitoring, RAM expansion, and kernel tuning beforehand.
Then you're ready to service some serious hits!
<P>
Thanks to <EM>Tony Sanders</EM> and <EM>Chris Torek</EM> at BSDI for their
helpful suggestions and information.
<P>
"M. Teterin" <mi@ALDAN.ziplink.net> writes:<P>
<BLOCKQUOTE>It really does help if your kernel and frequently used utilities
are fully optimized. Rebuilding the FreeBSD kernel on an AMD-133
(486-class CPU) web-server with<BR>
<CODE> -m486 -fexpensive-optimizations -fomit-frame-pointer -O2</CODE><BR>
helped reduce the number of "unable" errors, because the CPU was
often maxed out.</BLOCKQUOTE>
<P>
<HR>
<H3>More welcome!</H3>
If you have tips to contribute, send mail to
<A HREF="mailto:apache@apache.org">apache@apache.org</A>
<!--#include virtual="footer.html" -->
</BODY></HTML>
|
