CVS-Version: $Id: ChangeLog,v 1.18 2004/04/09 22:08:19 weigon Exp $ Feature Requests: * complex - nested conditionals - config file per host - compressing dynamic content (filters) - limit read-queue to 64k and use traffic-shaping cap - if fastcgi/cgi are not ready ignore incoming FDEVENT_IN for a while -> reduce memory usage - userdir + suphp -> internal config-hash - don't strip // from PATHINFO - TLS * moderate - errorlog piping - access and auth with host, net - fastcgi-spawn - suid spawn - gracefull restart - mod_accesslog_mysql - mod_auth_mysql - IPv6 for FastCGI - man-page for spawn-fcgi - server.base-directory * simple - file-mapping for alias - add header for broken requests if debug.request-header-on-error - IPv6 for FastCGI - hostnames for remote-FastCGI - fix remote-fastcgi on freebsd + kqueue - crash FDEVENT_OUT on FreeBSD - EPERM on unlink - cgi exec via shell - crash in read-post - same config option twice -> aaa, aaa - remove default port from Host: 20.02.2005 16:54 - 1.3.11 - request handling ignore \r\n before request-line in keep-alive requests (Bug #13) - cgi, fastcgi added REMOTE_PORT and SERVER_ADDR fixed check for broken status-code in nph-scripts (Bug #14) - fastcgi fixed passing cmdline options the spawned binary (Bug #16) fixed failed reconnects (Bug #9) - macos x fixed build on max - mod_secdownload parse conditional secure-download.secret (Bug #8) - mod_userdir redirect if username is not followed by / (Bug #15) 13.02.2005 17:33 - mod_auth fix crash if require, realm or method are empty (Bug #5) 13.02.2005 14:52 - network handle EPIPE and ECONNRESET as 'client has closed connection' in writev() (Bug #1) - macosx compile error on MacOS X due to missing environ (patch by Johan Sörensen) (Bug #2) - indexfiles append the detected indexfile only once to uri.path (reported by Thomas Seifert) (Bug #3) 06.02.2005 15:16 - 1.3.10 - fastcgi display a error-message if a hostname if specified in fastcgi.server->host we need an IP here - debug added debug.log-state-handling - spawn-fcgi accept a full commandline for spawning 06.02.2005 12:50 - fastcgi fixed openssl handling - network_freebsd_sendfilev gracefull handling of connections closed on client-side removed debug-message 06.02.2005 01:44 - 1.3.9 - documentation added docs for SSL setup and mod_status - fastcgi fixed config handling on PowerPC for local-spawning 05.02.2005 15:14 - fastcgi added bin-environment to setup the environment of the spawned process added bin-copy-environment to copy only the specified set of options from the old environment added handling of cmd-line options to bin-path - setenv fixed crashed in setenv.add-response-header 04.02.2005 18:09 - configure fixed docs for --with-mysql - fastcgi improved performance of building the header (drop strlen()) 04.02.2005 01:59 - cgi don't send file on error check if cgi-handler exists before executing it added support for nph-... 02.02.2005 21:18 - pre-release - request parsing handle invalid characters in URI 02.02.2005 15:12 - makefiles dropped unused header files from the distribution 02.02.2005 14:18 - fastcgi delete sockets on shutdown - http/1.1 adding option to disable http/1.1 01.02.2005 12:03 - cygwin fixed plugins_load to use the right extensions again removed mmap check added ssl support 01.02.2005 01:49 - configure make check for valgrind.h covered by --with-valgrind - mod_localizer, mod_maps remove both plugins from the distribution - file-not-found handle file not found again 30.01.2005 16:44 - HEAD requests don't send content on dynamic HEAD requests with status 200 30.01.2005 15:16 - 1.3.8 - network-handler remove debug output on writev() if the remote side closed the connection - directory index handle EACCES correctly 29.01.2005 15:16 - pre-release - mod_alias fixed mod_alias + pathinfo handling - mod_accesslog added access-log to syslog patch from allan 28.01.2005 17:30 - directory redirect without Host-header use server-ip instead of client-ip for the Location: - fastcgi + pathinfo if fastcgi-auth redirects to a directory which doesn't exist handle it correctly (bug introduced in 1.3.8) - requesting directories clean physical.path if directory is requested and dir-listing is disabled send 403 again (buf introduced in 1.3.8) 28.01.2005 12:08 - fastcgi ignore FDEVENT_HUP for unix-sockets as a simple read + timeout will do the job anyway 22.01.2005 20:28 - pre-release - fastcgi send content and headers if authorizer mode is used use a new connection if connection is died to fastcgi and we have not used it yet 18.01.2005 21:21 - pre-release - plugins added version-id to plugins to detect plugins which are not up-to-date 16.01.2005 23:11 - fastcgi fixed write-failed after crash of fastcgi-child 16.01.2005 20:43 - setenv fixed setenv.add-environment - fastcgi fixed authorizer + added testcases 16.01.2005 17:40 - pre-release - mod_status beautified mod_status - mod_setenv added setenv.add-environment - timeouts add timeout to read-post 15.01.2005 12:57 - debug added debug options to log - missing files - request header - response header - request handling added a more usefull error message for the status-code changes in the request parser - server announcement set Server: header for dynamic content too - fastcgi fixed double free don't crash on FDEVENT_ERR added a comment for EAGAIN on connect() 08.01.2005 17:45 - ssl report an error if ssl.engine is enable but no ssl support compiled in 08.01.2005 12:23 - mod_status added request time to the output (late changelog) added host and filename to the output (fobax) (late changelog) HTMLalized the output (fobax) 06.01.2005 19:51 - pre-release - error-handler let the error-handler handle 403 requests too make the error-handler setable by a module - error-pages reworked the error-page handling 05.01.2005 13:10 - keep-alive handling made sure that keep-alive is really handled correctly 04.01.2005 17:02 - mod_setenv added a module to added request and response headers on the fly - error-log send error log to syslog() if no errorlogfile is specified (again) 02.01.2005 22:44 - pre-release - response handling cut of body for status 301, 304 and 205 - buffer optimized all _hex functions (Silvan Minghetti) 02.01.2005 20:32 - fastcgi if bin-path is not specified, don't die (bug introduced in the last pre-rel) - auth if userfile is empty don't auth. 02.01.2005 19:06 - mod_compress fixed off by one if cache-dir is not set 02.01.2005 16:10 - conditional config fixed !~ and != - buffer copy empty buffers correctly 31.12.2004 17:45 - ipv6 + pidfile don't complain if we can't remove the pidfile (Silvan Minghetti) remove ipv6 option from the commandline of lighttpd doesn't support ipv6 (Silvan Minghetti) 31.12.2004 15:41 - pre-release - kqueue simplified event handling (adam) - fastcgi fixed div-by-zero bugs in the adaptive process spawning - mysql-vhost added mysql-vhost (Christer Holgersson) 30.12.2004 19:09 - fastcgi added adaptive spawning of FastCGI processes - traffic shaping added traffic shaping per virtual server 28.12.2004 23:26 - traffic shaping added traffic shaping per connection 25.12.2004 22:58 - mod_status fixed status.url again (Timo) 21.12.2004 11:29 - configure added check for signal and select (compile fix for netbsd 1.4 and 1.5) 11.12.2004 12:38 - 1.3.7 - fastcgi + php retry to connect to another PHP child if one of them dies after connect - cgi + multipart don't transform CONTENT_TYPE to HTTP_CONTENT_TYPE - debian more cleanup, updated changelog, added more deps and suggests (Chris Brown) 10.12.2004 22:33 - event handler fixed crashes in kqueue 10.12.2004 13:57 - pre-release - mod_status fixed wraparound in total requests and total traffic - debian updated licence and packaging - security call setgroups() to get rid of all groups - ssl handle SSL_shutdown() == 0 correctly fixed openssl detection in configure fixed handling of chunked encoding - request handling handle Connection: keep-alive correctly (case as not ignored) 21.11.2004 02:39 - windows merged basic native windows port (compiles with mingw) 20.11.2004 18:43 - conditional ported - cgi - secdownload - expire - localizer - usertrack - status - proxy - server-tag Server: ... can now be specified by server.tag = "..." - spawn-fcgi fixed typo in usage text - ssl fixed detection of libs and headers 05.11.2004 16:01 - fastcgi added more usefull error messages 04.11.2004 23:01 - ssi added support for ${...} 03.11.2004 14:51 - 1.3.6 - fastcgi added spawn-fcgi to the distribution added spawn-local-fastcgi yourself ( bin-path ) 03.11.2004 11:22 - accesslog don't cycle accesslogs of external processes are used 02.11.2004 15:34 - fastcgi handle END-OF-REQUEST correctly if chunk-encoding is not used 02.11.2004 10:53 - internal redirects fixed handling of query strings in internal redirects for directories 02.11.2004 09:54 - pre-release - cgi add REMOTE_USER, suppress AUTHORIZATION handle payloads > 4k - mod_alias fixed url checking - follow-symlink fixed config 31.10.2004 11:30 - 1.3.5 - writev fixed seg-fault in debug-message if write() fails and LFS is enabled handle EINTR - sendfile linux handle EINTR 31.10.2004 09:09 - freebsd added missing header in joblist.c fixed test-scripts for zsh 30.10.2004 22:26 - modules added mod_userdir and mod_alias added docs for the new modules 30.10.2004 19:52 - porting added defines for MAP_FAILED for NetBSD 1.3.x 30.10.2004 18:54 - pre-release - pipelining fixed offset calculations - ipv6 IPv6 might be disabled at compile-time - rewrite close mem-leak - auth forgot to reset the global-config handler - symlink add option to disable follow-symlink - ssi added support for exec-cmd 23.10.2004 - 1.3.4 - max-fds set the upper limit of fds only if server.max-fds is set 23.10.2004 13:49 - accesslog use a shell to handle accesslog-pipes 22.10.2004 17:00 - accesslog added logging of user-supplied data via %{...}o and X-LIGHTTPD-* header 22.10.2004 14:57 - pre-release - openwrt fixed configure-checks and Makefile.am's to build cleanly with a cross-compiler builds cleanly for openwrt 22.10.2004 13:03 - out-of-fd improved the out-of-fd handler - cgi, fastcgi set SERVER_NAME to server.name or the value submitted by Host: - error-handler only set old status code if it wasn't set by a handler 21.10.2004 22:36 - pre-release - fastcgi don't crash on out-of-fd condition - out-of-fd try handle the out-of-fd condition in a sane way 21.10.2004 15:03 - mod_auth seperated auth.backend.*.userfile for plain, htpasswd and htdigest added 'digest-auth' against 'plain-backend' added auth.debug for debugging 16.10.2004 10:18 - 1.3.3 - mod_simple_vhost, mod_evhost conditional-ized - mod_rrdtool maintain the request-counter for each conditional-config (adam) 14.10.2004 11:30 - accesslogs cycle all access-logs - mod_rewrite tell the user to install pcre.h if he wants to use mod_rewrite 10.10.2004 10:11 - pre-release - error-handler added a error-handler for status 404 (server.error-handler-404) 09.10.2004 16:28 - pre-release - cgi added support for \n in headers - mod_auth added conditional auth 01.10.2004 09:28 - plugins fixed off by one error in plugin initialization (Mike) related into a segfault on AMD64 30.09.2004 21:44 - 1.3.2 - file-cache disabled the file-cache it was taken the wrong files from the cache 30.09.2004 08:39 - 1.3.1 - file-cache drop a unused file-cache entry after 10 seconds reuse unused entries - request-parser accept IPv6 adresses in Host header - tests modified the scripts to work with zsh (check on Linux, Irix and FreeBSD) 26.09.2004 12:28 - comparission function file-cache has delivering the wrong entry if only the last character of the filename differed and the filesize was the same. - cgi + cygwin cgi need s SYSTEMROOT environment 22.09.2004 08:55 - network detect of file has been shrinked while we are sending it out and terminate the connection if would run over the edge 22.09.2004 07:56 - mod rewrite, fastcgi, ... keep REQUEST_URI after rewrite 21.09.2004 22:49 - fastcgi authorizer fixed cleanup code (matt) 21.09.2004 20:08 - rrdtool rrdtool.db-name is now conditional fixed check if write() failed (adam) 17.09.2004 17:50 - 1.3.0 - rewrite added url.rewrite-final = ... 17.09.2004 15:55 - code cleanup integrated the fixes from cygwin into the main tree - kqueue init kqueue after daemonizing (broken since 12.09.2004 14:02) 16.09.2004 21:00 - cygwin + macosx finished the cygwin port this port seems to fix the problems on macosx too 12.09.2004 14:02 - socket handling added support to handle more than one server socket 11.09.2004 12:23 - 1.2.8 - EINTR handle EINTR for linux-sendfile - configfile ignore an extra comma at the end of the array declaration 11.09.2004 09:46 - mod_proxy pass remote-addr as X-Forwarded-For to the real server behind the proxy - code cleanup moved all cut'n'paste versions of the inet_ntop cache to inet_ntop_cache.c - fcgi don't overwrite the fd in fcgi_establish connection if connect fails. this results in various problem in other places. 05.09.2004 09:46 - file-cache cache the mimetype - last-modified don't complain if the If-Modified-Since contains a valid comment/option like ; length = ... 05.09.2004 09:13 - expires overwrite the Expire if it is set by a previous plugin - conditional config conditional config as disabled in 1.2.7 by accident 04.09.2004 10:02 - 1.2.7 - mod-proxy remove the \0 before the post content - cgi fixed hanging process if cgi-crash terminates to fast (before we read its response) - extented attributes added xattr support, submitted by Ari 29.08.2004 16:00 - rrdtool moved the rrdtool support from mod_status into its own module mod_rrdtool rrdtool.binary = "/usr/bin/rrdtool" rrdtool.db-name = "/var/www/lighttpd.rrd" 29.08.2004 11:00 - pre-release - timeouts server.max-keep-alive-requests = 0 replaces server.use-keep-alive = "disable" added server.max-keep-alive-idle server.max-read-idle server.max-write-idle - docs added a entry for each config-value into configuration.txt added simple docs for rewrite redirect compress cgi simple-vhost 29.08.2004 10:05 - config options complain if no configfile is specified - fastcgi removed stupid allocation bug which might cause a problem in really rare cases 26.08.2004 22:06 - 1.2.6 - optimize use array_strcasecmp() in favour of strcasecmp() as it is slightly faster. apply the case-insentive conversion also on the last character. (adam) sort the checked elements in request.c and filter apply the logic to compare some less fields, if the header is not used. improved the config-patch function to use our internal buffer-compare functions instead of strcmp 22.08.2004 16:09 - pre-release - cgi added missing cleanup code - fastcgi remove double-free added handling of EINTR in some places - leaks fixed some leaks in the new config code - array_strcasecmp fixed alignment in the improved array_strcasecmp function (adam) 20.08.2004 14:46 - pre-release - performance optimized a few useless strlen() away as we either know the length from buffer->used - 1 or by sizeof(str) - 1 if it is constant. optimized the 'find the \r\n\r\n' function. improved the array_strcasecmp() based on another idea from (ralf) - accesslog enabled the strftime cache again 15.08.2004 23:41 - accesslog added apache-like CustomLog handling in accesslog.format accesslog.format = "..." 15.08.2004 21:08 - test-cases remove testdir - configfiles handle escaping of " in strings properly 13.08.2004 12:07 - array improved inner-loop of array_strcasecmp() (ralf) 11.08.2004 14:14 - fcgi socket use SUN_LEN if available - keep-alive disable keep-alive on request server.use-keep-alive = "disable" 10.08.2004 15:59 - 1.2.5 - conditional config mod_fastcgi mod_rewrite mod_redirect mod_access mod_compress mod_accesslog are ported 10.08.2004 13:05 - pipelining fixed very stupid pipelining bug 09.08.2004 22:07 - pre-release - conditional config first code for conditional config 09.08.2004 14:21 - fcgi fixed access to free()'d memory (doesn't create any harm) - isdigit, warings signed -> unsigned for 2 more isdigit() calls (adam) removed some unused var's if pcre is not available (adam) 08.08.2004 20:57 - pre-release - debian added a chmod to /var/log/lighttpd/ (allan) 08.08.2004 12:05 - kqueue use EV_SET() instead of setting the ev-struct by hand (adam) - fcgi fixed the EINPROGRESS handling to use getsockopt (er) fixed a leak of server is disabled (er) - solaris 10 port-api added a skeleton for the sol10 port api 06.08.2004 10:18 - mod_ssi fix DATE_LOCAL so it displays the correct time zone (Jeremy Hinegardner) 04.08.2004 11:43 - openbsd fixes dropped usage of MAX() in buffer.c added prober includes for md5.h if openssl is enabled (brad) - memory usage documented the way how lighttpd caches memory blocks reset the buffers after they have been written by the network-layer - kqueue modify fd-bitmask only if kevent succeeded (adam) 03.08.2004 15:09 - mod_compress compress even if you have no cachedir set 03.08.2004 13:26 - pre-release - Makefile fixed dependencies for parallel build in mod_ssi_expr.c - combo patch * Tinker with kqueue(). Add a reset method so that the kqueue file descriptor can be re-enabled after a fork(). Emulate the devpoll driver in that adds and deletes are sent to the notification mechanism im- mediately, which should cut down on phantom events. Use ev->kq_results as a sliding window. * Change F_SETFD calls to use the preferred FD_CLOEXEC instead of 1. * Remove unnecessary fdevent fcntl handlers. It appears that the only driver that needs one is Linux RT signals. * Quiet compiler warning about unused parameter. * Set the close-on-exec flag for the /dev/poll and epoll_create() file descriptors. * Return failure if /dev/poll could not be opened instead of logging and continuing. * Detect EAGAIN after writev() failures. FreeBSD sendfile() doesn't need protection, as the man page says: When using a socket marked for non-blocking I/O, sendfile() may send fewer bytes than requested. In this case, the number of bytes success- fully written is returned in *sbytes (if specified), and the error EAGAIN is returned. (adam, georg, matt) 02.08.2004 18:08 - mod_ssi check for pcre before compiling the module - fdevents dropped fdevent_fcntl added by the last patch (adam) kqueue: events == FDEVENT_IN -> events & FDEVENT_IN (adam) 31.07.2004 22:07 - 1.2.4 - fdevents * Test at configure time for kqueue() and * Remove various hard-coded constants from event handlers * Move maxfds into the event structure, and out of the fdevent_*_init handlers. Event handlers can use the maxfds member to size arrays. * Various event structure renames to discourage clashes * Remove extra (ignored) call to fdevent_event_next_fdndx() in the main server loop. * Wrestle with kqueue(). The implementation has to deal with phantom events (for fds which have been deleted/closed), similar to the Linux RT signals code. Like the RT code, it maintains a bitmask of active fds. After a successful call to kevent(), the code will compress/overwrite dead events. The other annoyance is that the handler must track the event filter for each fd, even though you cannot support both read and write filters for the same fd in one kqueue. The handler maintains a separate bitmask for fd filters (1 == EVFILT_IN, 0 == EVFILT_OUT). (adam) - server side includes added native server-side includes based on the docs from apache: http://httpd.apache.org/docs/mod/mod_include.html not supported are: - exec - nested virtual - config.errmsg - echo.encoding 24.07.2004 - fdevents added a bitset to figure out if we received a event for an unregistered fd in rt-signal (adam) - kqueue added kqueue support (Matt Levine) 13.07.2004 08:58 - configfile parse keys correctly that contain a digit (Geoff Adams) - fcgi fixed large post uploads (Geoff Adams) fixed uri if docroot is set (Geoff Adams) 03.07.2004 22:50 - 1.2.3 - index-files rewrite uri.path to the index-file instead of keeping it at .../ this fixes index-file handling in FastCGI/CGI docroot is used - close-on-exec enable close-on-exec handling to simplify FD handling in CGI code - cgi keep error-log-fd open to catch the error handling for execve() report error if cgi-exec file doesn't exist - proxy pass page-content on error to the user (E.R.) code cleanup (E.R.) - ssi first skeleton of a plugin for ServerSideIncludes - security limit the headerlength again to 64k max 03.07.2004 14:23 - configure fixed compile-check for libpcre if pcre-config doesn't point to /usr/lib 02.07.2004 18:17 - buffers always allocate a multiply of 64bytes. this should reduce the number of realloc()s and still doesn't has a too high overhead. 02.07.2004 11:07 - fds connect stdin, stdout and stderr to /dev/null instead of just closing it use dup2() instead of dup() - accesslog if accesslog.filename starts with a | spawn a process which will get data in one chunk once in a while 01.07.2004 11:00 - sample config added text/css and text/javascript mimetypes 28.06.2004 12:18 - proxy module added a proxy module (based on the fastcgi module) and added some documentation 25.06.2004 21:41 - last-modified handling replaced %Z by GMT. otherwise the last-modified check will most often fail. 24.06.2004 20:20 - relax http-parser don't reply 400 in case of an empty header-field, just ignore it 23.06.2004 22:10 - file-cache don't cache mmap() for files larger than 64k as we run out of RAM otherwise too fast (check with some 200mb files) - 64bit fixes fixed buffer_equal on sparc64 15.06.2004 19:09 - 1.2.2 - mmap cache fixed mmap-caching in network_write.c and network_writev.c after a direct hint by E.Rodichev - sendfile + linux check at config-time if sendfile() works on Linux 11.06.2004 15:09 - fcgi + unix sockets added support for unix domain sockets (spawn-fcgi 1.2.0 required) 10.06.2004 11:49 - configure use pcre-config to determine the position of the pcre headers 05.06.2004 22:06 - filehandle-cache remove mmap-segment if filecache gets invalidated 30.05.2004 14:13 - lighttpd 1.2.1 - response headers request headers that appear twice are grouped together like expected by the CGI spec (concat with a ", ") response headers behaved the same way but are not grouped anymore. They stay seperated. Actually they are concated by \r\n: which is the same in the end. - file uploads the handling of longer post requests is fixed now. 28.05.2004 09:13 - cgi added support for direct calls to cgi-binary 22.05.2004 21:58 - pipeling the code cleanup is finished successfully. Now all cases of pipelining are handle the right way. POST pipelining was still not working up to now. 22.05.2004 12:55 - code cleanup use the well-tested code from the write-queue as the base for the read-queues and simplify the pipeline handling alot that way. 20.05.2004 15:08 - network backends enabled sendfile support again (__FreeBSD__ instead of __freebsd__) added a mmap cache as part of the filedescriptor cache added AIX specific send_file() support (untested) 20.05.2004 10:40 - segfaults fixed some minor segfaults on startup when no config file is used. 17.05.2004 10:58 - lighttpd 1.2.0 - documentation reformated the documentation the doc/ directory 15.05.2004 14:45 - localizer fixed build of localizer extension 15.05.2004 12:35 - POST requests there is no need to die if we spot a simple POST request on a static file - pipelining fixed HTTP/1.1 pipelining which caused the problems with opera - array handling how did that bug survive such a long time ? a wrong compare function was used in one case, but not the other. 15.05.2004 03:20 - secure and fast downloads added a module which allows secure and fast downloading of files: 1. the application (.php, ...) controls the access to the files 2. the webserver is handling the transfer (and check the app generated tokens) the token is based on - a secret - a timestamp - the filename this means even if the token is is distributed by the user it will get invalid after a given timeout (default 60 seconds) - errorfiles check for errorfiles before using them - code cleanup applied cleanup fixes from adam 14.05.2004 18:47 - fdevent handling added a more generic callback interface to the fdevent structures which simplifies the writing of plugins. this might destabilize lighttpd for a while - cgi fixed header parsing of the header is sent in chunks and the terminator is sent in a single chunk - EINTR fixed some occurences of EINTR which read() 03.05.2004 23:55 - portability E.R.: * portability fixes for Solaris 2.5 02.05.2004 10:15 - Expect: Handling added incomplete support for Expect: 100-continue (RFC 2616 8.2.3) by sending always 417 for every Expect-request (see 14.20) we have been blamed for not supporting it: http://lists.w3.org/Archives/Public/ietf-http-wg/2004JanMar/0059.html 29.04.2004 23:07 - 1.1.9 - usertracking added a very basic usertracking cookie handler 29.04.2004 19:37 - network-writev Adam: * call munmap() on error in write() - docs E.R: * documented 'check-local' - test-env made the shell scripts more portable (checked with zsh, bash and ksh) fixed compilation on NetBSD 28.04.2004 22:22 - FastCGI E.Rodichev: * added "Authorizer" mode for FastCGI 27.04.2004 18:08 - ssl Alexey Slynko: * handle SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE in SSL_write the right way. - FastCGI add 'check-local' instead of the implicit 'if-docroot-is-set' handling implemented at 24.04.2004 14:34 E.Rodichev: * remove useless extra-/ in before uri.path * add 'prefix' notation for FastCGI processes 26.04.2004 16:52 - code cleanup patches from Alexey Slynko: * remove the pidfile if lighttpd terminates the normal way (if not in chroot) * init SSL before getting daemonized 25.04.2004 19.15 - mem-leak fixed mem-leak on broken HTTP-headers - FastCGI patches from E.Rodichev: 1. CONTENT_LENGTH "If no data are attached, then this metavariable is either NULL or not defined". 2. QUERY_STRING If the Script-URI does not include a query component, the QUERY_STRING metavariable MUST be defined as an empty string (""). 3. Doubling of REMOTE_ADDR removed. patches from adam: * fixed bug in the sizeof() patches from yesterday. * some more *_long to *_off_t 24.04.2004 14:34 - FastCGI don't check for localfile if 'docroot' for a FastCGI host is specified. 24.04.2004 12:13 - POST fixed POST request handling - chunk-encoding the generated HEX strings where broken since the 1.1.8 (this effected all HTTP/1.1 requests without Content-Length like FastCGI-PHP) - code cleanup patches from adam: * malloc + memset -> calloc * sizeof(int) -> sizeof() * assign fd_set instead of memcpy() * init fd -> connection fd pointers to -1 16.04.2004 08:48 - 1.1.8 - code cleanup don't reuse buffer > 64k (see settings.h) added server.max-request-size to limit the maximum request-body size (in kBytes) don't accept HTTP-request headers larger then 32kBytes (see settings.h) minor speed improvements in the request-parser More cleanup patches from adam: * change pre-ANSI C/valid C++ syntax for function declarations/definitions from using () to (void). Ex: int foo(); --> int foo(void); * use static linkage as much as possible, to limit possible symbol collisions * whack more unneeded variables * try and prevent any errno clobbering by storing the old errno value before any subsequent system calls, and restoring before function exit. * change printf syntax for unsigned variables from %d to %u 15.04.2004 18:41 - code cleanup handle all int != size_t cases in fcgi.c correctly check headerfields to have a value handle both EINVAL cases of writev() before the can occur limit content-length to SSIZE_MAX disallow negative content-length the usage of ltostr() has been reduced to the minimum in favour of buffer_/append|copy)_(long|off_t) dropped ultostr() and ultohex() in favour of buffer_* 15.04.2004 16:35 - portablity more patches from adam: * remove warnings for unused parameters and variables * remove warnings for mismatched pointer assignments * change "gtime_r" to "gmtime_r" 13.04.2004 20:48 - test-cases made the 'make check' target self-contained In our case we have to call the targets in the following order: $ ./configure ... $ make $ make install $ make check because the path to the plugins is hardcoded in the binary itself using $ ./configure --prefix=/tmp/lighttpd-1.1.x/ ... will help if you don't really want to install before testing. 13.04.2004 00:05 - portability adam sent another patchset: * Wrap PCRE-specific data member access with an #ifdef * Add const to pointer using return value from dlerror() * Explicitly initialize pointer in the lemon parser to 0, in order to catch missing else { } clause * Use a time_t rather than an int for gmtime() call. On some systems (including 32- and 64-bit SPARC) time_t is a long. 12.04.2004 17:00 - 1.1.7 - fastcgi strip WS after HTTP-response headers coming from the FastCGI process added REMOTE_USER to the Server->FastCGI headers removed HTTP_AUTHORIZATION from the Server->FastCGI headers 12.04.2004 10:24 - cgi if we don't get a partial HTTP-response-header send the content out as soon as the cgi script is finished 12.04.2004 01:23 - compression added bzip2 compression (supported by w3m) 12.04.2004 00:12 - configfile add some usefull error messages if the tokenizer or the parser fail to read the configfile 11.04.2004 22:04 - configure added --with-ldap and --disable-lfs to the configure options 11.04.2004 20:28 - 64bit offset size disable linux-sendfile support for linux 2.4.x for now as it don't support 64bit transfers fixed all assignments on the path from the stat() to the Content-Length HTTP-header - head requests set content-length in HEAD requests - accesslog write accesslog entry on network error write the correct amount of byte written to the accesslog 11.04.2004 11:48 - code cleanup moved the config for the cgi-plugin from config.c to the plugin. moved some buffers which were only used by a one or two plugin from the server-structure to the plugins keeping the plugins independent from the server-core is a 'good thing' 10.04.2004 19:06 - configfile parser removed the leaks from the configfile parser 09.04.2004 23:15 - 1.1.6 - stricter http-parser added line-folding although noone really seems to use it. 09.04.2004 18:42 - configfile parser the hand-written configfile parser has been replaced by a LALR(1) one. 'lemon' from the sqlite guys has been used to generate the parser. - by-host, by-url, by-file, by ... $HTTP["url"] =~ "~$" { access.deny = "all" } $HTTP["host"] =~ "." { simple-vhost.server-root = "/home/weigon/wwwroot/servers/" simple-vhost.default-host = "grisu.home.kneschke.de" simple-vhost.document-root = "pages" } $HTTP["host"] == "incremental.home.kneschke.de" { server.docroot = "/hasdasd" } at least the parser can handle it now. Currently there is no real support for this context-based config-option. But the syntax and the parser are done. 09.04.2004 10:58 - ssl support enable ssl support again - mmap enabled mmap+write again 08.04.2004 12:34 - stricter http-parser based on a thread at http://lists.w3.org/Archives/Public/ietf-http-wg/2004JanMar/0050.html the HTTP-parser has been adjusted to be more correct when it comes to request-header fieldnames the whitespace handling around the ':' has been relaxed as requested 07.04.2004 17:06 - sigaction use sigaction instead of signal if possible 07.04.2004 13:55 - accesslog use localtime-timestamps in accesslogs if struct tm has a tm_gmtoff field 07.04.2004 10:41 - 1.1.5 - -D_REENTRANT solaris + localtime_r() needs it 07.04.2004 02:54 - mod_auth + ldap added a ldap backend to the auth plugin. 06.04.2004 13:37 - pidfile fixed the permissions of the pidfile (Matthijs van der Klip) - specfile merge the RedHat and SuSE specfile with Matthijs - 64bit file-offsets moved the FILE_OFFSET_BITS settings from the config.h the Makefile to enable 64bit offsets the right way 06.04.2004 12:32 - mod_expire added an apache compatible mod_expire which adds Expires: headers to the request expire.url = ( "/buggy/" => "access 2 hours" ) 05.04.2004 22:34 - solaris devpoll forgot to provide the infrastructure to actually enable the devpoll event handler. the same has been done for the freebsd-kqueue handler (which doesn't work yet) fixed the devpoll support while testing it on a real solaris box 05.04.2004 09:49 - debian added debian packaging support written by Vincent Wagelaar - solaris Another set of patches for Solaris from Adam * Detect * Detect and include for definition of FIONREAD * Detect and link against the library exporting hstrerror * Correct typo in SENDFILE_LIB * Use__sun instead of __solaris__ for detecting operating system. Please see http://predef.sourceforge.net/preos.html for more detail * Explicitly cast arguments to isalpha() and toupper() to unsigned char. The man page says that the functions support the range of an unsigned char, and EOF. * Include and define UIO_MAXIOV as IOV_MAX for Solaris. 04.04.2004 18:05 - 1.1.4 - pidfile added pidfile writing after deamonizing 04.04.2004 01:05 - fdevent added framework for freebsd_kqueue and solaris_devpoll the solaris_devpoll one might event work (untested) 03.04.2004 16:41 - network added framework for filebased chunks - read-write + mmap-write - linux-sendfile - freebsd-sendfile - solaris-sendfilev (untested) and memorybased chunks - write - writev made TCP_CORK a 'global' flag around the write_chunkqueue-calls the writev() support should improve the performance for all non-static pages. 170 req/s against 158 req/s for the following script if writev() is used instead of write() \n"; flush(); } ?> 28.03.2004 13:42 - cleanup applied some cleanup patches submitted by Adam: * variables modified in signal handlers should be sig_atomic_t * assert statements should not have side effects * STD{IN,OUT,ERR}_FILENO preferred instead of {0,1,2} * dieing --> dying * SEGFAULT calls abort directly, instead of derefencing a NULL pointer - mod_accesslog modified the accesslog format to be CLF compatible set locale for LC_TIME to C 26.03.2004 16:13 - path info enabled the pathinfo code again 25.03.2004 13:30 - 1.1.3 - portability compiles fine now without any patches on IRIX - hostname detection reworked "get the hostname for HTTP/1.0 requests which don't specify a Host: ..." to only query the name in this single case for the server side of the connection - errorlog handling stderr is only used until the errorlog is init'ed if no error-log is specified, syslog() is used if cycling error-log fails, syslog() is used - accesslog cycling don't fall back to stdout anymore - event-handler use poll() as the default event-handler again 24.03.2004 01:37 - 1.1.2a - error messages added some hints to the failing error-messages 22.03.2004 01:58 - 1.1.2 - configure some protability changes to get the 'inline' working with the MIPS CC 21.03.2004 22:00 - mod_rewrite, mod_redirect reading the config for those two plugins was not working - fdevents changed the compile time setting for the event handling into a run-time setting. server.event-handler = "select" # poll, linux-rtsig, linux-sysepoll added sys_epoll() for linux 2.6 select - all systems poll - Linux 2.1.23+, all XPG4-UNIX sigio - linux 2.4.0+ sysepoll - linux 2.5.66+ 1000-4k-nok 1000-100k-nok select 1776.99 296.52 poll 678.02 607.28 sigio 3754.46 1411.23 sysepoll 3817.67 1431.02 21.03.2004 00:10 - configure script rewrote large parts of the lib/header detection of the configure script 20.03.2004 01:39 - fastcgi as the docroot on external hosts might be different than the webserver docroot it can now be specified in the config: fastcgi.server = ( ".php" => ( "grisu" => ( "host" => "192.168.2.41", "docroot" => "/home/jan/servers/", "port" => 1026 ) ) ) a huge internal cleanup in the config handling made the code more readable. some more warnings and error checking should track most of the config errors for the fastcgi plugin 19.03.2004 12:34 - external patches Matthijs van der Klip submitted three nice patches: - turn off writing in mod_status if status.rrd-reports is set to disable - fix for a debug message - get the hostname for HTTP/1.0 requests which don't specify a Host: ... - rc-script for RedHat - documentation added a documentation section about authentification - doc/authentification.txt 19.03.2004 05:11 - optimizations implemented special versions of - strcasecmp (array_strcasecmp) - isdigit, isalpha, isalnum (light_*) added a faster check for a finished header after disabling all modules it is still possible to get 20 kreq/s. 15.03.2004 19:45 - 1.1.1 - localizer server added the localizer-server module to the code - chunked-encoding Apple's Safari or HTTP-Handler doesn't handle chunked-extensions defined by RFC 2616 correctly and doesn't ignore them. Disabled the chunked-extension which were just used for debugging purposes. 07.03.2004 12:20 - optimization moved the checks if a plugins support a given function from the dispatcher (plugins_call) to plugins_call_init() to do the check only once. equalized the plugins functions to only two types: SERVER_FUNC() CONNECTION_FUNC() replaced all handwritten plugin_call-handlers with macros made all plugin-functions 'static' with all plugins loaded we are at 15kreq/s gain without any plugin loaded at 16kreq/s before the change we were at 13kreq/s 06.03.2004 21:13 - compilation fixes fixed some warnings on FreeBSD and NetBSD by adding #include ifdef'ed a pcre-entry in base.h remove #define _XOPEN_SOURCE from http_auth.c for IRIX crypt() on Linux needs _XOPEN_SOURCE 06.03.2004 19:18 - 1.1.0 - authentification added htpasswd and htdigest backends to work against user-files generated by htpasswd and htdigest. for basic auth: - plain - htpasswd (crypt only) - htdigest for digest auth: - plain - htdigest 06.03.2004 17:35 - authentification check the method in the authorization header againt the configured method 06.03.2004 14:54 - hostname parsing added a RFC-2396 conforming "Host:" parser added 17 checks for the parser 06.03.2004 10:25 - configuration added a warning for unknown configuration variables. dropped the 'specific-for.url' ideas for now as it is not known yet how to implement it the right way renamed some config-variables to reflect there actual meaning - server.host -> server.bind - server.virtual-* -> simple-vhost.* - server.userid -> server.username - server.groupid -> server.groupname - server.docroot -> server.document-root 05.03.2004 15:36 - fastcgi internals: moved all mod_fastcgi settings from the global struct to the plugin itself - vhosting got a patch for enhanced virtualhosting from christian kruse -> mod_evhost moved the virtual hosting config (server.virtual-*) to it's own module called mod_simple_vhost 23.02.2004 10:06 - configfile rearragned the config-file structure again to be able to add settings for a given URL, Host, Filename, ... This change affects some config-options directly (access.deny, url.rewrite, url.redirect, virtual-hosting, ...) added 'specific-for.url' for url specific config settings - digest auth FIX: md5-sess type seperate the auth-backend stuff 13.02.2004 22:23 - lighttpd 1.0.3 - content-length + POST FIX: If header and content didn't fit into one single packet the rest of the content was not read correctly 13.02.2004 01:07 - content-length + POST the check for content-length on a POST request vanished somehow in one of the previous releases. - header search FIX: the search for header fields was up to now case-sensitive. Now it is like required by the standard case-in-sensitive. - browser bugs w3m 0.2.5 adds an additional \r\n at the end of the POST requests which is handled now 10.02.2004 10:12 - start script took the suse rc-skeleton which states to be LSB compliant and modified it for lighttpd needs 09.02.2004 11:48 - HEAD requests FIX: HEAD requests for static files delived the content from the GET request. (test case is added) 08.02.2004 15:53 - directory listings FIX: the pathname has not encoded at all before it was transfered to the browser. a proper url and html-encoding has been added. added modification-time and filesize to the output added a basic css for the virtual listings 07.02.2004 22:15 - lighttpd 1.0.2 - sample configfile rearranged the config-file to have all the important options at the top - docs added a mod-rewrite options - mod_accesslog stdout is no longer used a default for the accesslog - error-messages the 404 is now reported in the error-log 07.02.2004 17:30 - configfile handling if a key is used twice like url.rewrite = ( "url1" => "url") url.rewrite = ( "url2" => "url") you get an error now. You have to write: url.rewrite = ( "url1" => "url", "url2" => "url") 31.01.2004 12:22 - lighttpd 1.0.1 - log cycling added a sighup-handler to the plugin interface and fixed the log-cycling for access-logs - portability disabled the interval-time optimization on IRIX - documentation added a lot of new documentation to README 31.01.2004 10:59 - status module added two new options rrd-dir and rrd-reports: rrd-reports = (boolean) # enables RRD-reports rrd-dir = (string) # path for the daily status-files fixed the "status-files in /"-bug this way 22.01.2004 13:38 - lighttpd 1.0.0 - simple docroot fixed handling of docroots if virtual-host is disabled 27.12.2003 11:12 - license handling added the first interface to license handling. 25.12.2003 23:48 - protability Verified again that the code compiles and runs cleanly on Linux, FreeBSD, NetBSD and IRIX compiling with gcc and the option -pedantic works fine compiling with mipspro cc works fine, too - tests added some more tests: 39 tests 25.12.2003 16:01 - protability some compile fixes for FreeBSD have been applied and a new switch has been added to choose between IPv4 and IPv6 on FreeBSD. (cmdline: -6, configfile: server.use-ipv6) - packaging cleaned up the specfile for building RPMs 21.12.2003 01:00 - authorization brought basic and digest auth back to life. this module as the last one which had to be updated after the config-file changes - test harness framework add 3 tests for basic auth 20.12.2003 22:10 - compression added gzip compression (gzip-header + deflate + crc) - test harness framework added a 22 tests to verify the correct behaviour of lighttpd - request parsing GET http://www.yahoo.com/ HTTP/1.0 is handled now as GET / HTTP/1.0 - plugins moved FastCGI and CGI handling into modules which can be loaded at run-time 17.12.2003 13:18 - compression the directory structure is now build automaticly 15.12.2003 01:00 - compression added a compression cache to the compression module (mod_compress) Vary: Accept-Encoding is set now as it influences to delivered stream of bytes. This is important for caches. 10.12.2003 00:24 - config files a new config-file format is ready for the final release of lighttpd. it supports: server.docroot = "string" server.host = integer server.modules = ( "string", "string" ) server.mimetypes = ( "key" => "value" ) server.complex = ( "key" => ( "string", integer ), "string", integer ) the syntax should look familar to all who worked with Perl or PHP. config-file handling has been seperated into a parser and a lexer. Both are currently written by hand and will propably be rewritten into bison + flex if time permits. But that would be a cosmetical change. 05.12.2003 03:07 - status-page improved the status page to display - the current connection-stati - average throughput over 5 seconds (requests/s and output-bound traffic) now you can see what is going on in the server - access-log the accesslog is now a module and can be disabled by just removing the plugin from the list of loaded plugins 04.12.2003 16:18 - chroot-ing how-to: using chroot - chroot to /home/www/ - docroot at /servers//pages/ - defaulthost www.example.org e.g. (external view) /home/www/servers/www.example.org/pages/index.html (in chroot) /servers/www.example.org/pages/index.html config: chroot /home/www/ userid wwwrun groupid nogroup virtual-server-root /servers/ virtual-server-default-host www.example.org virtual-server-docroot /pages/ The FastCGI process is living outside this chroot definition as it is started seperatly. 01.12.2003 02:06 - cleanup in preparation for the first stable release some internals had to be cleaned up. Basicly it was a cleanup of workflow of the creation of the response-header. All modules can use a clean interface for this purpose now. This is espacially usefull for all modules which have to pass some HTTP-headers to the client. 29.11.2003 22:22 - modules finally moved the modules to shared libraries and cleaned up some code path to become more readable. So far we have: mod_rewrite mod_redirect mod_access mod_auth mod_cache mod_chat mod_status mod_maps 28.11.2003 18:16 - redirects as lighttpd supports url-rewriting redirection was a few lines of new code. redirect ^/wishlist/(.+) http://jan.kneschke.de/wishlist/$1 rewrite ^/wishlist/(.+) /new/wishlist/$1 28.11.2003 17:00 - signal stuff setitimer is used to send a event every second to call time() only once a second. another system call which has been remove from the main-loop. sending HUP to lighttpd will close and re-open the logfiles. this is used for cycling logfiles. #! /bin/sh ### # # a simple logfile rotator for lighttpd # DATE=`date +"%Y%m%d-%H%M"` LPID=`pidof lighttpd` mv access.log access.log.${DATE} kill -HUP ${LPID} gzip access.log.${DATE} 27.11.2003 01:07 - native win32 port a first attempt for a native win32 has been done. For now mingw is the base for the development as it provides a basic unix-like framework for building native win32 applications. the most internal files have been ported and the over all progress is going well. At the end this will just be a prove of concept. 26.11.2003 01:17 - access denied added a access-deny filter to block specific urls like access-deny ~ access-deny .inc 17.11.2003 01:06 - bug fixing a fstat() on a opened fd which has changed reports wrong the file-info. Using stat() again helps to solve this problem. the sig-io version doesn't suffer from this problem. the etags are now used for verifing file-cache-entries. - chat enabled the internal login mechanism again added support transfering session-infos over MySQL. 15.11.2003 00:19 - optimizing added a buffer_equal_reverse function which is optimized version of strcmp which is going backwards as pathnames are often the same for in the first bytes. wrote a one-pass parser for the request-header. The combination of strstr(..., "\r\n") to seperate lines and strchr(..., ':') to seperate keys from values more or less touched every byte twice. we are still at 18.000 req/s for 4kb keep-alive requests even with etags and handling all header fields. 14.11.2003 17:26 - fcgi Content-Type wasn't passed correctly to the FastCGI app. HTTP_CONTENT_TYPE was sent instead. - cache provided access to the Session-ID - error-log the timestamp is now written in a human readable form 22.10.2003 00:06 - fcgi, cgi added a special set of array-functions which are optimised for the "insert only once" case. this provides access to the headers which are now forwarded to the external interfaces. Before this change only a limited set of request-headers were forwarded. 21.10.2003 11:58 - modules added a new module-hook after the basic-init of the module for handling config-settings and prepare the overall operartion (like building db-connections, compiling regexes, ...) - cache use turckmm-cache 2.4.3 to get some numbers for the php-latency: /usr/sbin/ab -n 10000 -c 10 http://alba.home.kneschke.de:1025/index.php handling the cache-decision and the cache-hit in php: cache-miss: 100% (-> $version = 0) plain : 108.13 req/s turckmm-cache: 218.39 req/s cache-hit: 100% (-> $version = 1) plain : 164.45 req/s turckmm-cache: 653.98 req/s handling the cache-decision and the cache-hit in the server: cache-hit: 100%, but using index.cml cml: 4918.84 req/s (no keep-alive) cml: 6901.31 req/s (keep-alive) cache-miss: 100%, but using index.cml plain : 108.39 req/s turckmm-cache: 217.84 req/s Conclusion: - there is no loss in the cache-miss case through the cml-handling - the cache-hit case can be improved dramaticly with lighttpd-cache - turckmm-cache improves the cache-miss case alot 20.10.2003 00:40 - cache the first 'real-life' test showed dramatic improvements in the req/s handling. The basic idea was to move the decision if a php-page can be taken from the cache from the php-code to the webserver. See here why this is a good thing: the quite common code which works for http://jan.kneschke.de/ is using templates and is quite static, but depends on 4 external files (the menu-structure, the template, the current content, the class-file). the index-file is always: include_once "jk.inc"; $v = new view(); print $v->get(array(array ("file" => "content.html"))); It is more or less the same for all pages. This basic setup can deliver 100 requests/s. The next step has: - application bases caching as we know that each pages depends on those 4 files, you can check if they have been modified since the last request and deliver the content from the cache otherwise. this increased the throughput to 150 req/s. (cache-hit ratio 100%) The next logic step is to move the decision-making process out of the PHP code as PHP is to slow for the cache-hit path: a CML (Cache-Markup-Language) has been written which describes the whole decision process which has been written in PHP-code before: output.content-type text/html output.include _cache.html trigger.handler index.php trigger.if file.mtime("../lib/php/menu.csv") > file.mtime("_cache.html") trigger.if file.mtime("templates/jk.tmpl") > file.mtime("_cache.html") trigger.if file.mtime("content.html") > file.mtime("_cache.html") if one of the 'trigger.if' statements is true the 'trigger.handler' is called to generate the pages. if none of the them is trigger the files from 'output.include' are sent to the browser with content-type specified in the first line: The result was very 'promissing': 5900 req/s with keep-alive 3800 req/s without keep-alive (both for a cache-hit ratio of 100%) for keep-alive this is factor 59 against the plain un-cached version and still 39 againt the php-cache-version which is doing exactly the same. Time for party. :) 19.10.2003 - 18:55 - cache the handling of functions has been improved. they are now 'plugable'. just the dlopen() stuff is missing. a new datatype has been added: the string this makes it possible to evaluate something like: trigger.if unix.time.now() - file.mtime("head.html") > 30 trigger.if mysql.query("SELECT count(*) " + \ " FROM structure AS struc, session AS sess " + \ " WHERE struct.user = sess.user" + \ " AND sess.id = \"" + mysql.escape("ab\"c") + "\"") you see: - string operations (concat) - handing of escape-sequences - functions - comparisions 18.10.2003 - 13:39 - cvs imported everything into the cvs server which makes the whole revision handler a lot easier. The CVS server was up all the time but not used. Importing required 5 minutes of work which included merge the freebsd and the linux tree. - compilers + platforms on a regular various compilers and platform are check to compile with with the current code base: platform | os | compiler | state ---------+---------------------+--------------+--------- ia32 | Linux 2.4.22 | gcc 2.95.3 | ok ia32 | FreeBSD 5.1-CURRENT | gcc 3.3.1 | ok mips64 | IRIX 6.5 | gcc 3.2.2 | ok misp64 | IRIX 6.5 | MIPSpro 7.41 | ok the mipspro compiler revealed some warning which resulted in a nice code-cleanup that made the code more readable. 18.10.2003 - 03:00 - e-tags and friends in preparation for the php-conference at the begin of november in frankfurt/main the server has the support some more caching/proxy tags like: - E-Tag (14.19) [done] - If-Match (14.24) - If-None-Match (14.26) [done] - If-Range (14.27) Section (13.3.3) binds them together. [RFC 2616] using lxr.kde.org and lxr.mozilla.org revealed that - konqui only uses if-none-match - mozilla uses if-none-match and if-range it looks like it isn't that easy to trigger the if-range case. As ulf suggested the etag is a hash of file-size, inode-number and mtime. - fd-caching ulf just phoned me ask proposed the free the cache more agressivly in case of fd-shortage. increasing the the fd-limit is the better idea. :) 17.10.2003 12:45 - chat finished the mysql-support for storing the sessions 12.10.2003 20:56 - valgrind used valgrind again to verify that the code a free of mem-leaks and found a 'leak generator' in the chunk-api. the last few leaks were just some missing free()'s at the end of the program run which would have been freed anyway. at the end valgrind couldn't find any missing free()'s. 11.10.2003 12:09 - FastCGI reduced the number of system calls for FastCGI to WebServer to 2 calls per fd-event. (ioctl() + read()) this has no direct effect of the performance of the server, but improves the possible througput of the load-balancer. 10.10.2003 21:09 - FastCGI - load-balancing a brown paper bug has been fixed which caused to decreasing throughput if load-balancing was enabled. benchmarking the req/s with load-balancing shows really nice results: server : req/s comment ---------+-------------------------------------------------- ulf : 764.06 (php) lappi : 800.06 (php) ulf+lappi: 1526.95 (2 * php) grisu : 1265.66 (php + ab + lighttpd) all : 1647.72 (3 * php + ab + lighttpd) all(nice): 1904.40 (same as all, but the local php on grisu has been 'nice -20'd) if a php is run on the load-balancer it has to get a lower priority than the load-balancer itself as to handle the work of (here) 3 php-servers. 10.10.2003 15:11 - java ? http://jakarta.apache.org/tomcat/tomcat-4.1-doc/jk2/common/AJPv13.html 08.10.2003 21:08 - gigE Last week 3 RTL8169S gigE cards arrived and were installed in the test-framework. They are very cheap (20 Euros) and are a good start for a low-level benchmark network. First result show us: 48 Mb/s with ab. The webserver and the test-server are equipped with a 2000+ AMD CPU. The system load 95%, user load is 3%, the rest is idle on both plattforms. After some calculations at gets clear that there are various bottlenecks: 1. The PCI-Bus (32bit/33Mhz) can only transfer 133Mb/s - 48 Mb/s TCP-Traffic results in 55Mb/s Ethernet-Traffic (verified with slurm) [outgoing] - there is about 10Mb/s incomming traffic at the PCI bus which the requests - the rest of the devices at the PCI bus are eating the last few mb/s 2. IRQ-Handling - There are reasons why the RTL8169S cards are so cheap - they can't send jumbo-frames - only 8k/64k buffers which results in a interrupt every 3 packets - they are at least handling checksum offloading for ip, udp and tcp In the end there is a need for new hardware to limit the throughput by the Ethernet again. - PCI-X, 64bit-PCI, ... - 'real' gigE-Network cards - ... BTW: 2.4.22 + the r8139 driver are very flacky and resulted in 3 lock-ups for today. In another test the dual-PPro-FreeBSD 5.1-CURRENT machine has been used as server with a gigE interface (re0) at has shown that the CPU is the limit for this combination. The maximum thoughput was 7Mb/s. Another small benchmark: $ ab -n 1000 -c 10 http://192.168.2.41:<port>/lighttpd-20030925.tar port | server | CPU Idle ------+-------­---------+----------- 1025 | lighttpd | 75% 1026 | thttpd 2.23b1 | 75% 1027 | boa-0.94.14rc17 | 69% * 1028 | apache 1.3.x | 77% In all 4 cases the thoughput was 8600 kbytes/s. * boa had 9 failed transfers. - FreeBSD The problems with FreeBSD 5.0-RELEASE vanished after updating to 5.1-CURRENT. 26.09.2003 18:22 - FreeBSD A SMP-machine has been added to the test-farm. It is running FreeBSD 5.1-RELEASE and will help to the improve the scalability. fixed sendfile() handling. - FreeBSD problems FreeBSD-5.1-RELEASE-SMP 2 * Pentium Pro 200 MHz 192.168.2.38 (doubleheart) (webserver [lighttpd at port 1025, thttpd at port 1027]) Linux 2.4.20 1 * AMD 2000+ 192.168.2.10 (grisu) (ab) Problem: Connections are reset by the kernel without any application intervention. Calling 'ab' (apachebench) at grisu with the following paramters: /usr/sbin/ab -n 10000 -c 10 http://192.168.2.38:1025/index.html results in the following output: ... Time taken for tests: 40.610 seconds Complete requests: 10000 Failed requests: 5980 (Connect: 0, Length: 5980, Exceptions: 0) ... This is reproducable and the number of failed requests is always 5980 +/- 50 requests. In other words: after 4000 requests tcpdump shows the following output: tcpdump shows: 00:53:48.923029 192.168.2.10.39774 > 192.168.2.38.1025: S [tcp sum ok] 1013737315:1013737315(0) win 5840 <mss 1460,sackOK,timestamp 5208461 0,nop,wscale0> (DF) (ttl 64, id 7918, len 60) 0x0000 4500 003c 1eee 4000 4006 964d c0a8 020a E..<..@.@..M.... 0x0010 c0a8 0226 9b5e 0401 3c6c 6763 0000 0000 ...&.^..<lgc.... 0x0020 a002 16d0 eeaa 0000 0204 05b4 0402 080a ................ 0x0030 004f 798d 0000 0000 0103 0300 .Oy......... 00:53:48.923330 192.168.2.38.1025 > 192.168.2.10.39774: S [tcp sum ok] 1803860672:1803860672(0) ack 1013737316 win 65535 <mss 1460,nop,wscale 1,nop,nop,timestamp 4459794 5208461> (DF) (ttl 64, id 6821, len 60) 0x0000 4500 003c 1aa5 4000 4006 9a96 c0a8 0226 E..<..@.@......& 0x0010 c0a8 020a 0401 9b5e 6b84 bac0 3c6c 6764 .......^k...<lgd 0x0020 a012 ffff d4ce 0000 0204 05b4 0103 0301 ................ 0x0030 0101 080a 0044 0d12 004f 798d .....D...Oy. 00:53:48.924009 192.168.2.10.39774 > 192.168.2.38.1025: . [tcp sum ok] ack 1 win 5840 <nop,nop,timestamp 5208461 4459794> (DF) (ttl 64, id 7919, len 52) 0x0000 4500 0034 1eef 4000 4006 9654 c0a8 020a E..4..@.@..T.... 0x0010 c0a8 0226 9b5e 0401 3c6c 6764 6b84 bac1 ...&.^..<lgdk... 0x0020 8010 16d0 e9c3 0000 0101 080a 004f 798d .............Oy. 0x0030 0044 0d12 00:53:48.924150 192.168.2.10.39774 > 192.168.2.38.1025: P [tcp sum ok] 1:29(28) ack 1 win 5840 <nop,nop,timestamp 5208461 4459794> (DF) (ttl 64, id 7920, len 80) 0x0000 4500 0050 1ef0 4000 4006 9637 c0a8 020a E..P..@.@..7.... 0x0010 c0a8 0226 9b5e 0401 3c6c 6764 6b84 bac1 ...&.^..<lgdk... 0x0020 8018 16d0 27e4 0000 0101 080a 004f 798d ....'........Oy. 0x0030 0044 0d12 4745 5420 2f69 6e64 6578 2e68 .D..GET./index.h 0x0040 746d 6c20 4854 5450 2f31 2e30 0d0a 0d0a tml.HTTP/1.0.... 00:53:48.924728 192.168.2.38.1025 > 192.168.2.10.39774: R [tcp sum ok] 1803860673:1803860673(0) win 0 (ttl 64, id 6831, len 40) 0x0000 4500 0028 1aaf 0000 4006 daa0 c0a8 0226 E..(....@......& 0x0010 c0a8 020a 0401 9b5e 6b84 bac1 0000 0000 .......^k....... 0x0020 5004 0000 64ba 0000 SYN, SYN+ACK, ACK, DATA, RST strace shows that no connection attempt has been reported to the application which is poll()'ing the server socket. /* the common loop without any block attempts */ accept(3, {sa_family=AF_UNSPEC, sa_data="\0\0\0\0\0\0\0\0\0\0\0\0\0\0"}, [0]) = 5 fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0 read(5, "GET /index.html HTTP/1.0\r\n\r\n", 4095) = 28 stat("/home/jan/lighttpd-0.1.0/servers/", {st_mode=S_IFDIR|0755, st_size=512, ...}) = 0 fstat(6, {st_mode=S_IFREG|0644, st_size=4348, ...}) = 0 write(5, "HTTP/1.0 200 OK\r\nConnection: clo"..., 235) = 235 write(2, "1064875136: (network.c.210) 235 "..., 33) = 33 syscall_393(0x6, 0x5, 0, 0, 0x10fc, 0, 0xbfbff2c0, 0) = 0 close(5) = 0 /* no futher waiting connections */ accept(3, 0xbfbff700, [1852702730]) = -1 EAGAIN (Resource temporarily unavailable) /* enter the main-loop */ gettimeofday({1769235301, 1663069807}, NULL) = 0 poll([{fd=3, events=POLLIN}], 1, 1000) = 0 gettimeofday({4294967295, 65537}, NULL) = 0 poll([{fd=3, events=POLLIN}], 1, 1000) = 0 gettimeofday({4294967295, 65537}, NULL) = 0 poll([{fd=3, events=POLLIN}], 1, 1000) = 0 ... (strace is broken for accept() and gettimeofday(), syscall_393() is sendfile()) after entering the main-loop the webserver doesn't receive any new POLLIN events at all for the next 10-15 seconds. Any connection-attempt within the period is, as you can see in the tcpdump output, accepted by the kernel and the received data is thrown away as the kernel sends a RST. After those 10-15 seconds the application gets a POLLIN event for the server socket and the normal data-transfer taken place for the next 4000 requests. This behaviour is reproducable with thttpd 2.20c and the current lighttpd. 26.09.2003 11:42 - localizer ported the localizer-server application into a module for lighttpd. After 30 minutes the code was ported and funtional. This modules provides access to the localizer-db by a HTTP-Interface and generates the response as HTML, CSV and plain-text. So far, the module-interface looks good and flexible enough. 7000 req/s is good enough too. Generating the HTML and querying the DB needs some time. 22.09.2003 08:40 - modules introduced a simple module interface which allow to hook into the process of handling the requests. Basicly it allows to move the url-rewriter, the auth-sub-system and the cache out of the main-code. The module-interface will be extended to allow the cgi and the fastcgi sub-processes to be moved into a module. 15.09.2003 09:36 - error-handling fastcgi and cgi connections are now closed correctly if the corresponding client-connection has died. 14.09.2003 10:40 - cgi finally streaming works with CGI, too this enable support for cgiirc.sf.net and friends which use streaming as there transport mechanism. streaming has been verified with cgiirc-0.5.2. - fdevents there where some reports that sigio didn't work as expected. It just reported no events at all. Looks like a known bug in the glibc on those platforms. Using poll() instead solve the problems. 27.08.2003 22:12 - rewrite a pcre based rewrite engine has been integrated - cgi the file-based cgi interface has been replace by two pipes. 23.07.2003 13:29 - fdevents the whole fd-event handling has been reworked and several smaller bugs and design-errors have fixed. sigio, poll() and select() are working fine again. On IRIX 6.5 SIGIO could be used, but without further testing poll() is used. - cgi CGI-support is still broken. 21.07.2003 18:46 - dnotify using the F_NOTIFY feautre of the Linux 2.4.x kernel gives anothre nice performance boost as lighttpd can cache the stat()'s the right way, now. 23009.66 @ 5-4k-k which means +10% against 30.06.2003 14:03 for 1000-4k-nok the performace nearly doubles: 3730.23 goes up to 6144.39 17.07.2003 13:21 - FreeBSD a small patch (#include <>) to compile lighttpd on FreeBSD again. 08.07.2003 10:48 - fileinfo-cache A reallife test showed that the cache wasn't perfect at all as it made lighttpd crash. This is fixed now. 30.06.2003 17:40 - lighttpd-bench After some problems with other benchmarking tools for webservers the first version of lighttpd-bench has been written. It a revealed a nasty strange behaviour which was fixed by increasing the listen-backlog from 5 to 1024. 30.06.2003 14:03 - fileinfo-cache the fileinfo-cache has been relaxed a little bit and there are always 2 fstat()'s per file-request now. This isn't that dramatic: 21800.74 req/s is still a very good result (7-4k-k). 29.06.2003 03:29 - fileinfo-cache a fileinfo cache has been added to reduce the number of system-calls to stat and open a file. in the releases before the same file was stat'ed at least 2 times plus a stat on the docroot for each request. Now the stat()'s and the corresponding open() + close() calls are cached and the number of system calls has been reduced to the minimum: $ strace -eopen,stat64,read,write,sendfile,accept,shutdown,close \ -p `pidof lighttpd` /* first connection */ accept(3, {sin_family=AF_INET6, sin6_port=htons(56211), inet_pton(AF_INET6, "::ffff:192.168.2.10", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 5 accept(3, 0xbffff470, [28]) = -1 EAGAIN read(5, "GET /index.html HTTP/1.0\r\nUser-A"..., 4095) = 91 stat64("/home/weigon/projects/lighttpd/servers/grisu.home.kneschke.de:1025/pages/", {st_mode=S_IFDIR|0755, st_size=3656, ...}) = 0 stat64("/home/weigon/projects/lighttpd/servers/grisu.home.kneschke.de:1025/pages//index.html", {st_mode=S_IFREG|0644, st_size=4348, ...}) = 0 open("/home/weigon/projects/lighttpd/servers/grisu.home.kneschke.de:1025/pages//index.html", O_RDONLY) = 6 write(5, "HTTP/1.0 200 OK\r\nConnection: clo"..., 235) = 235 sendfile(5, 6, [0], 4348) = 4348 shutdown(5, 1 /* send */) = 0 close(5) = 0 /* second connection */ accept(3, {sin_family=AF_INET6, sin6_port=htons(56212), inet_pton(AF_INET6, "::ffff:192.168.2.10", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, [28]) = 5 accept(3, 0xbffff470, [28]) = -1 EAGAIN read(5, "GET /index.html HTTP/1.0\r\nUser-A"..., 4095) = 91 write(5, "HTTP/1.0 200 OK\r\nConnection: clo"..., 235) = 235 sendfile(5, 6, [0], 4348) = 4348 shutdown(5, 1 /* send */) = 0 close(5) = 0 In the end we have a new speed record: (ab -c 8 -n 100000 http://192.168.2.10:1025/index.html with poll()) Requests per second: 23435.67 [#/sec] (mean) This is a speed increasement of 20-30% against the last internal benchmark. 28.06.2003 02:30 - caching framework the trigger.if directive is working fine. the basic algebra is done ( +, -, *, / and comperations =, <=, >=, >, < and the boolean logic && and || including braces) two basic function are available: - unix.time.now - file.mtime(...) 25.06.2003 17:33 - caching framework added output.include, output.content-type for a cache-hit and trigger.handler for a cache-miss. the actual decision is made be trigger.if which will be added tomorrow. 15.06.2003 15:33 - sig-io After several other smaller optimisations lighttpd performs better with sig-io under high load: sigio+poll poll c1000-4k-nok : 2635.95 1643.39 c1000-4k-k : 7335.68 6788.87 c1000-100k-nok: 2353.49 1217.73 c1000-100k-k : 3097.89 2080.73 The user-space part has been optimized a lot. Now it is time to optimized the number of context-switches between user and kernel-mode. user 0m0.180s sys 0m1.160s a fileinfo-cache is the way to go. struct { buffer *name; struct stat st; int handler; ... int fd; void *mmap_p; } 11.06.2003 14:57 - sig-io is back again and it works fine. sigio (Realtime-Signals under Linux 2.4.x) sends one signal per event and buffers the rest of the events in a kernel queue. If the is full a SIGIO is sent and poll() gets all events at once for further processing. Currently the behaviour is a little bit strange: - sig-io + poll() is good for non-keep-alive connections - poll() for keep-alive connections c1000-4k-nok: (1000 concurrent request, 4k filesize, no keepalive) poll : 1521.38 sigio+poll: 2124.00 c1000-4k-k: (1000 concurrent request, 4k filesize, keepalive) poll : 5882.35 sigio+poll: 1239.46 Very strange for now. 09.06.2003 23:59 - code-cleanup the event-handling code has been rewritten to handle single events better as they are expected from sig-io. the fallback-mode of sig-io is broken now, but the normal poll() mode got a 10% increasement in speed. This means the we are back the speed level of 20030308-0155 are as fast as zeus again. Especially under higher load the current lighttpd performes better. 09.06.2003 11:51 - sig-io benchmark 20030609-1151 20030608-2110 20030308-0155 -c 10 sigio select() poll() poll() 4k : 7870.92 7937.77 8035.36 9443.76 4k (keep-alive) : 14098.41 14590.02 14275.52 17985.61 100k : 3366.32 3382.03 3261.15 3722.32 100k (keep-alive) : 5544.77 5576.00 5573.20 5975.86 -c 100 4k : 6144.77 5821.40 5714.29 6724.95 4k (keep-alive) : 9097.53 9213.19 8979.08 10833.06 100k : 2549.33 2495.94 2318.95 2607.36 100k (keep-alive) : 4267.67 4283.94 4094.17 4314.06 For -c 100 4k and 100k sig-io gives a small increasment. 09.06.2003 01:00 - sig-io writing large files works now as expected. After removing the limit for the chunks of sendfile the write-buffer-is-empty-again signal is generated. that was missing. 321 of 10000 connections still have the wrong length. After that is fixed it is time to start some benchmarks again. 08.06.2003 21:10 - sig-io first attempts in getting SIGIO support running which were not very successfull yet. - poll()/select() benchmark 20030608-2110 20030525-1623 20030308-0155 -c 10 select() poll() poll() poll() 4k : 7937.77 8035.36 8166.60 9443.76 4k (keep-alive) : 14590.02 14275.52 14781.97 17985.61 100k : 3382.03 3261.15 3176.42 3722.32 100k (keep-alive) : 5576.00 5573.20 5809.56 5975.86 -c 100 4k : 5821.40 5714.29 5669.26 6724.95 4k (keep-alive) : 9213.19 8979.08 8418.22 10833.06 100k : 2495.94 2318.95 2314.28 2607.36 100k (keep-alive) : 4283.94 4094.17 4456.92 4314.06 as the overall processing cycle has been rearranged the overall figures changed in minor ranges. c100-4k-k increased, *-100k-k decreased. At least it still works. 08.06.2003 12:45 - select()/poll() implementate an abstration layer for fd-events (like eevry other webserver). currently we support poll() and select(). This should bring us support for Mac OS X and propably Windows. 04.06.2003 18:35 - configure lighttpd can now be build without ssl-support even if openssl is installed - protability - on IRIX IPv6 is supported now out of the box - for FreeBSD some missing haeders have been added 04.06.2003 14:34 - pipelining adding support for pipeling introduces a problem if the request-header was broken. this has been fixed now. - unneccesary slashes when building pathes from different portions of a string (e.g. docroot + virt-server-docroot + request-uri) slashes where added for security and simplicity. This resulted in up to 5 adjacent slashes which caused no harm but looked strange. (fixed) 04.06.2003 09:57 - start/stop messages the error-log contains the start and end-times of the lighttpd process, now. - configfile the config-file parser has been relaxed to accepts tabs instead of spaces. - better error-handling fixed a bug in the FastCGI-SAPI of PHP if the fastcgi process dies or closes the connection unexpectedly we return 500 now instead of closing the connection to the client. - Location the CGI/1.1-rev-03 specification requires us to send Status 302 if a Location-header is sent by the client and a Status-header is missing. - PATH_TRANSLATED if PHP is compiled without --discard-path PATH_TRANSLATED has to be provided. 27.05.2003 15:54 - directory listings if a directory is requested and the directory doesn't contain a index-file a directory-listing can be displayed. You have to enable directory-listings in the config-file (directory-listings on) - url-decoding up to know urls where not decoded at all (%26 -> . and so on). This has been added. Unicode isn't supported as we use 8-bit chars internally. 26.05.2003 00:44 - pipelining as Sascha required pipelining for his benchmarking tool it has been implemented. pipelining allows sending a bunch of requests at once without waiting for the actual responses. This reduces the network-overhead and the round-trip-time. - non-free()d memory dmalloc helped to close some non-free()d memory. For the normal operation this isn't important as only memory chunks which had to be free()d at the end of the live-time of the lighttpd-process were not de-allocated. - partly-initialized variables lighttpd wasn't initializing the main-structure which resulted in strange behaviour in rare circumstances. 25.05.2003 16:23 - benchmarks after removing some useless internal copies we are more or less at the old speed levels. after adding virtual-hosts 10% of the performance were lost. Using less memcpy() operations might add several other boosts. I've just checked how lighttpd compares to Zeus. lighttpd (current) (old) -c 10 20030525-1623 20030308-0155 Zeus 4_2 4k : 8166.60 9443.76 7278.55 4k (keep-alive) : 14781.97 17985.61 16496.21 100k : 3176.42 3722.32 3156.37 100k (keep-alive) : 5809.56 5975.86 5460.30 -c 100 4k : 5669.26 6724.95 5134.26 4k (keep-alive) : 8418.22 10833.06 8010.25 100k : 2314.28 2607.36 2688.32 100k (keep-alive) : 4456.92 4314.06 4240.70 23.05.2003 14:38 - cgi-variables HTTP_HOST was missing for the cgi-module while the fcgi-module passed it through to the handler. Fixed that. - fcgi-errors the connection to the fcgi was dropped and poll() reported an error, the error wasn't reported to the client the right way. 22.05.2003 23:02 - authorization the first password-storage has been added: [auth] backend plan plain-userfile <filename> require /download/ user=jan|user=anom http-auth /download/ "download archiv" digest groups are prepared but not implemented. basic and digest are working fine. 20.05.2003 17:53 - authentification The auth-methods from RFC 2617 have been added. - auth basic - auth digest The only source for accounts is currently only the config-file. auth-digest needs the plain-text passwort. Are there any source which provide a plain-text password ? 12.05.2003 14:33 - virtual hosts added very basic virtual-host support virtual-server-root /home/weigon/projects/lighttpd/servers/ virtual-server-default-host grisu.home.kneschke.de:1025 virtual-server-docroot /pages/ docroot is - if http-host exists <virtual-server-root> + <http-host> + <virtual-server-docroot> - otherwise <virtual-server-root> + <virtual-server-default-host> + <virtual-server-docroot> - if even virtual-server-default-host does not exist, 500 is sent 12.05.2003 13:02 - code cleanup After two month of development it was time clean-up the internal structures. It looks like every went fine as lighttpd works es expected like before. - deflate the on-the-fly compression has been verified to work fine with opera, konqui, mozilla and the IE. 12.05.2003 02:10 - on-the-fly compression: deflate Why the hell are the defining a "deflate" encoding in the form of _deflate_ The "zlib" format defined in RFC 1950 [31] in combination with the "deflate" compression mechanism described in RFC 1951 [29]. (RFC 2616) and noone implements it that way ? Konqui and Mozilla expect a plain deflate() package without the zlib-header. Konqui is using "inflate2(..., -MAX_WBITS); " which is noted in the zlib source as /* handle undocumented nowrap option (no zlib header or check) */ Funny. Very, very funny. Anyway. We have mimetype-depended compression support now. 11.05.2003 21:56 - logging re-arranged the logfile structure to write CLF + useragent + referrer. 11.05.2003 10:23 - POST file-upload added the missing functionality to send more than a single packet to the FCGI-Server (or Client ? ... the PHP). This gives us the file-upload thing for eg. PHP and large user-forms (>16kb). 05.05.2003 15:21 - PATH_INFO added support for PATH_INFO. PHP is a little strange and doesn't trust the passed PATH_INFO setting. Works fine now. 30.04.2003 15:25 - bug-fixing day While testing the FastCGI interface with the MSIE Björn Schotte discovered that sometimes the output repeats itself from the start in an endless loop. This bug has been fixed. The read-write-fallback for ancient systems seeked the wrong FDs. The FastCGI handler is now separting the HTTP-Header from the response-body what results in a cleaner interface. The "header too long for caching" message is gone now. 28.04.2003 18:18 - chunked transfer-encoding The FastCGI part is now using Chunked-Transfer-Encoding if HTTP/1.1 is used and no content-length is specified. 27.04.2003 23:10 - chunked transfer-encoding added "Transfer-Encoding: chunked" which is currently used in the web-chat for the endless stream. Perhaps it helps some browsers. The FastCGI interface will get a the chunked-support too, as it will enable keep-alive even if no content-length is returned from the FastCGI process. We know the size of the chunks and will report it to the browser. 16.04.2003 12:02 - gigE deatchmatch sascha compared the current lighttpd to his premium thttpd in his gigabit-Ethernet-Network. http://schumann.cx/gbit_deathmatch.txt small comment: lighttpd provides the same performance (req/s and thoughput), but uses more CPU-Time. 10.04.2003 17:22 - works on IRIX Jörg Behrens provided me login to his SGI Origin and after fixing two small typos it worked fine on IRIX. (#if define HAVE... was missing a 'd' and getopt returns a 'int' and not a 'char') IPv6 support is currently disabled for IRIX a gethostbyname2() isn't available. 10.04.2003 15:56 - another bug-fixing day The upper limit of open connections was enforced and a caused a seg-fault. The current limit is set to 4096 parallel connections. As sascha benchmarked lighttpd and his premium thttpd I tried to reproduce his reported results and had to fix 2 flaws in the async-io handling of httpd_load which was used for the testing. If sascha starts another benchmark session I'll put a link to it here. 09.04.2003 00:08 - bug-fixing time Date: and Last-Modified: where sending a timezone != GMT what was invalid. The timestamp itself was correct, just the timezone use the wrong characters. the fcgi-code had an buffer-overflow for larger POST-Requests. The fcgi-code still has problems with POST-Request larger than 16kb as the the write buffer is full. Currently we don't handle this case except from reporting it in the error-log. 02.04.2003 01:17 - cgi is back The CGI Interface wasn't tested for a longer time. It don't really survived the introduction of the config-file handling and the internal changes that were part of it. A small test with #! /usr/bin/perl print "Content-Type: text/html\r\n\r\n"; print time()."\n"; 0; produced 219 req/s. Calling PHP via the CGI interface results in 100 req/s. Same script, same parameters for 'ab' and same PHP result in 1400 req/s if we use the FastCGI interface. Don't forget: these benchmarks only represent figures for scenarios which are not very realistic. They are just usefull for comparisions of the internals. We don't want to benchmark applications. 01.04.2003 23:04 - new benchmarks This time we wanted to see if we can get the fastcgi-Interface a little bit faster. We use this small script for the testing the interface. <?php ob_start(/*"ob_gzhandler"*/); print "12345<br />\n"; header("Content-Length: ".ob_get_length()); ob_end_flush(); ?> It generate a small network load and is the best case for benchmarking the overhead of the call to the fastcgi-php. Using the chunk-API instead auf tmp-files increase the req/s vom 1200req/s to 1600 req/s and the IO-load went down as expected. Waiting for the filesystem and creating 1200 files per second required some time. The CGI-Interface is still using tempfiles. 01.04.2003 19:28 (no april fools joke) - added SSL support I took -lssl and -lcrypto and added SSLv2/SSLv3/TLS support to lighttpd. After some reading I realisized that libcrypto contained some of my code. They have a similar buffer-struct with similar functions and they use the same MD5-code from the RFC. :) Adding basic SSL-support was quite esay: replacing all write/read-calls by SSL_write/read, initializing the ssl-context the right way and telling openssl where to get the data from (SSL_set_fd). It fits really well into lighttpd. - sidenote sooner or later the con->filename handler will vanish in favour of the direct interface to the chunk-API which will simplifiy the design a little bit. 31.03.2003 20:50 - added a web-chat module 1400 lines of C-code are neccesary to add a web-chat to lighttpd. Features: - bb-code for text-layouting - auto-highlight for URLs and Email-adresses - unlimited number of channels - max. 32.000 users per channel - fast as it is directly integrated into the web-server - easy to install (as easy as lighttpd) - IRC-like (/msg, /kick, /me, ...) - works with MS IE, Mozilla, Konqueror, ... - uses as little JavaScript as possible to stay compatible with all browsers - flood-protection (2 Levels: ignore and kick) - uses CSS for customized look-and-feel 25.03.2003 00:36 - finished the config-file support now we have config-sections [fastcgi] fastcgi .php 192.168.2.76 1025 The only this missing is -HUP handling to re-read the config. But that will follow when I'm up again. 20.03.2003 01:04 - adding config-file support the most boring part of the writing programs after writing documentation. The format is quite simple and good enough for lighttpd. 19.03.2003 03:05 - more on sendfile support added support for freebsd's version of sendfile(). Linux: ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count); FreeBSD: int sendfile(int fd, int s, off_t offset, size_t nbytes, struct sf_hdtr *hdtr, off_t *sbytes, int flags); the first 2 params are swap, the next 2 are the same and the last 3 are set to NULL or 0. Solaris 9 is providing a direct interface to the chunk-API with the sendfilev() system call: ssize_t sendfilev(int fildes, const struct sendfilevec *vec, int sfvcnt, size_t *xferred); recovering from a failure is a little bit more complex, but it should be fast. I just need a system to test it. Anyone ? 18.03.2003 17:32 - in the news what a surprise: lighttpd is announced in entwickler.com. http://www.entwickler.com/itr/news/psecom,id,9483,nodeid,82.html this was not planned. - implemented the chunked-API rather writing the content of multi-range'd requests to a temp-file we just register the parts in a chunk-queue. chunk 1 [mem] - part-header chunk 2 [file] - source-file (offset, len) chunk 3 [mem] - part-header chunk 4 [file] - source-file (offset, len) chunk 5 [mem] - part-footer If everything is prepared, the content-length is calculated and the http-header is generated and prepended to the chunk-queue. the handle_write() function which handles the write-process just sends the chunks to the network without only further modifications. the chunk-api even simplifies the 'my-fastcgi-process-needs-some-time-to-create-the-response- and-sends-the-data-in-chunks' case. Just add the chunk to the queue if it is received. 13.03.2003 10:49 - another set of real-life benchmarks the chairman was benchmarked a little bit. Chairman is full-flegded WCMS. As every CMS it needs some time to generate pages. To speed the page generating the static-parts of the page can be cached internally. without caching with caching apache + mod_php : 2.50 10.64 req/s lighttpd + fcgi-php: 2.50 22.74 req/s Looks like lighttpd is the way to go :) Please, don't compare these numbers with the other benchmarks: We used a different test-machine which is a little bit under-powered for this test-case. 12.03.2003 13:43 - another large application works with lighttpd Chairman from thinkphp has been verified to work with lighttpd. Three things are worth to note: 1. ALWAYS use log_error in the php.ini if you use FastCGI otherwise it will put the errormessages directly into fastcgi-socket without any FastCGI Header. This will result in strange errors. (its a PHP bug) 2. DOCUMENT_ROOT has been added to the environment variables 3. a small bug for sending larger output from a fastcgi client has been fixed. 11.03.2003 11:52 - added two neccesary features - redirect _dir_ to _dir_/ - append index-file to _dir_/ if they exist 11.03.2003 00:13 - another target after adding some headers lighttpd compiles without any warnings under cygwin on Windows. A native windows port will take some time as I have to figure out the changes on the winsock.h to the unix headers. 10.03.2003 11:52 - added Solaris 8 and 9 lighttpd has been confirmed to compile and work on - linux - FreeBSD, NetBSD - Solaris 8, 9 Perhaps I can get lcc to compile it on windows. 10.03.2003 01:30 - another platform was confirmed lighttpd works on NetBSD - another webserver benchmark mathopd (http://www.mathopd.org/) Mathopd/1.4 -c 10 4k : 6329.11 [#/sec] 4k (keep-alive) : 10235.41 [#/sec] 100k : 1168.50 [#/sec] 100k (keep-alive) : 268.82 [#/sec] (99% idle) -c 100 4k : ---- (connections dropped) 4k (keep-alive) : ---- 100k : ---- 100k (keep-alive) : ---- Very good values for small files, but the rest ? 10.03.2003 00:06 - added more write-handlers 1. sendfile (linux only) 2. mmap + write 3. read + write This means that lighttpd runs on some more systems. It has been verified to compile and work on linux 2.4.x and FreeBSD. -c 100 sendfile mmap read 4k 6476.68 5698.01 5363.08 [#/sec] 100k 2312.35 841.54 783.09 [#/sec] The mmap() numbers could be better with a fd+mmap-cache. The zero-copy thing is good for our performance. - tested another small httpd - BOA just to get a better view about the performance of lighttpd i've tested another httpd: Boa/0.94.14rc16 (http://www.boa.org/) Boa/0.94.14rc16 -c 10 4k : 5659.63 [#/sec] 4k (keep-alive) : 250.23 [#/sec] 100k : 1104.29 [#/sec] 100k (keep-alive) : 1363.14 [#/sec] -c 100 4k : 4319.65 [#/sec] 4k (keep-alive) : 2490.66 [#/sec] 100k : 815.93 [#/sec] 100k (keep-alive) : 1007.05 [#/sec] The results are a little bit strange. Keep-Alive for small files seems to be broken. It looks like lighttpd has enough power for now. Time the port and add more features. 09.03.2003 13:26 - load-balancing works great I used the framework from http://framework.netuse.de/ as a real-life example again to test the load-balancer. setup: - grisu - AMD athlon XP 2000+ - runs lighttpd + ab + a 'nice -15'ed FastCGI-PHP - generates 153 req/s alone - laptop - Intel PIII 850 - runs a FastCGI-PHP - generates 88 req/s using the internal load-balancer of lighttpd which balances the php-requests over grisu and laptop generate: 221 req/s An apache with mod_php running on grisu gives 117.04 req/s only. 09.03.2003 12:46 - building the connect connection to the fcgi-php is not non-blocking socket() -> connect() -> fcntl(non-blocking) resulted in some problems as the connect() call blocked sometimes for 1 seconds. socket() -> fcntl(non-blocking) -> connect() solves this problem, but addes more overhead. The first connect-attempt seems always to return EINPROGRESS. 08.03.2003 15:06 - the first 'real-life' PHP test showed 2 bugs in the POST handling. 1. the Content-Type header was not forwarded via fastcgi 2. the internal content_length handler wasn't reset after the request now lighttpd + php work with http://framework.netuse.de/ the basic application gives us 145 req/s 08.03.2003 11:06 - use diet to 'test' your code diet gives some use full warnings which should be followed. I took the chance and removed the last few fprintf() and sprintf() from the code. now, we don't need stdio.h anymore. a staticly linked, stripped lighttpd is 42884 bytes large. 08.03.2003 01:55 - buffer_strcat() uses strlen() to get the length of the 'to-be-catted-string' if the string is constant or has a known length you can pass that function which gives use buffer_strcat_len() which passes the string length as a parameter. - using a ramdisk as the base for the tempfiles generated mkstemp() is a very cood idea it reduces the io-load on the system and gives as more cpu-time for load-balancing the php-requests. - cache the output of strftime(), gmtime() and localtime we generate more then 10.000 req/s which results in using the same timestamp over and over again. It is sufficient to generate the timestamp once a second and give us a bunch of CPU-cycles for sending files. the same applies for other timestamps like Last-modified which sends a string version of st.st_mtime. Cache it. - don't try to overoptimize for code for the sake of clean code the range support require some hack in the first versions like setting the write_offset to the start of the range and keeping the rest of the "code 200" send code as usual. Only a few lines added this first range feature. fast, but a hack. It made things more complicated then neccesary to add multi-range support in this scenario. removing the hack doing small clean preprocessing helps to clean up the whole 'write a chunk on the wire' code. At the end we use less code and have a cleaner design. Sascha proposed a chunked layer which just takes the chunks (http-header, part-header, part-message, ... and the last boundary) and passes them to the write-to-the-wire-code. This is far more elegant and will perhaps same us from other problems, too. Currently I think that it is not necessary. First we have to fix some protability issues. Ok, today benches: today last -c 10 4k : 9443.76 7739.94 + 22% 4k (keep-alive) : 17985.61 13885.03 + 29% 100k : 3722.32 3349.97 + 11% 100k (keep-alive) : 5975.86 4965.49 + 20% -c 100 4k : 6724.95 5918.56 + 13% 4k (keep-alive) : 10833.06 8405.48 + 28% 100k : 2607.36 2393.60 + 9% 100k (keep-alive) : 4314.06 4035.35 + 7% Top Transfer Rate: 86365.77 [Kbytes/sec] received Too much for a GigE-link ? so far: - use state-engines a non-forking webserver like described in http://www.kegel.com/c10k.html - don't use fprintf() for logging. writing one line to log file takes more time that sending responsing the a HTTP-request - take care of your memory typedef struct { char *ptr; size_t used; size_t size; } buffer; buffer* buffer_init(); void buffer_free(buffer *b); int buffer_prepare_copy(buffer *b, size_t size); int buffer_prepare_append(buffer *b, size_t size); int buffer_strcpy(buffer *b, const char *s); int buffer_strcpy_len(buffer *b, const char *s, int s_len); int buffer_strcat(buffer *b, const char *s); int buffer_strcat_len(buffer *b, const char *s, int s_len); buffer_init() is only called once per buffer. If you don't need the buffer, set 'used' to zero and reused it afterward. buffer_strcpy() + buffer_strcat() check if the 'used' + the new strlen() fit in to the 'size' of the buffer. If not the realloc() the buffer. buffer_strcat() uses memcpy(ptr + used - 1, s, s_len + 1); which should be faster than the original strcat(). Using those buffers keep the memory usage at 2Mb. malloc() and free() are only called at the beginnig of the programm and at the end. - don't used sprintf() to convert a integer into a string as like sprintf(buf, "%d", l); write your own ltostr function. sprintf() is to general for the case and is slow. Speed: ------ Let's assume that thttpd is (one of) the fasted webservers out there. all httpds were started with: $ ./thttpd -p 1026 -D \ -d /home/weigon/projects/localizer/src/lighttpd/docroot/ \ -l thttpd.access.log (lighttpd uses the same parameters). We used ApacheBench for testing the RPS (requests per second). $ /usr/sbin/ab -dS -c 10 -n 100000 http://192.168.2.10:80/dummy.out lighttpd thttpd/2.21b+php thttpd/2.20c thttpd/2.23b1 (concurrency 10) 4k : 7739.94 6040.84 5078.20 5888.24 [#/sec] 4k (keep-alive): 13885.03 10349.82 5034.49 5853.09 [#/sec] 100k : 3349.97 1176.29[1] 1188.74 1198.29 [#/sec] 100k (keep-a.) : 4965.49 2513.38[2] 1100.65 1130.51 [#/sec] (concurrency 100) 4k : 5918.56 4907.01 4987.53 4886.87 [#/sec] 4k (keep-alive): 8405.48 6379.99 4938.76 4816.26 [#/sec] 100k : 2393.60 972.73 958.27 961.09 [#/sec] 100k (keep-a.) : 4035.35 893.51[3] 970.21 955.05 [#/sec] thttpd/2.21b+php has been patched with the keep-alive + php patches from PHP4 sapi/thttpd/thttpd_patch. [1] ab reported: Failed requests: 1 (Connect: 0, Length: 1, Exceptions: 0) [2] ab reported: Failed requests: 5 (Connect: 0, Length: 5, Exceptions: 0) [3] ab reported: Failed requests: 24 (Connect: 0, Length: 24, Exceptions: 0) Tunning the thttpd: ------------------- - disable symlink checking (chroot() or -nos) - log to /dev/shm/logfile, a ramdisk or disable logging $ ./thttpd -p 1026 -D \ -d /home/weigon/projects/localizer/src/lighttpd/docroot/ \ -l /dev/null