summaryrefslogtreecommitdiff
path: root/src/http_chunk.c
Commit message (Collapse)AuthorAgeFilesLines
* [core] _WIN32 sys-unistd.h to wrap <unistd.h>Glenn Strauss2023-05-031-3/+2
| | | | (selective implementations; not complete)
* [core] chunk_file_pread() to wrap pread()Glenn Strauss2022-02-191-12/+3
|
* [core] use pread() to skip lseek()Glenn Strauss2021-10-011-0/+9
|
* [multiple] internal control for backend read bytesGlenn Strauss2021-09-281-13/+5
| | | | | | | | | | | | | separate internal control for backend max_per_read When not streaming, large reads will be flushed to temp files on disk. When streaming, use a smaller buffer to help reduce memory usage. When not streaming, attempt to read and empty kernel socket bufs. (e.g. MAX_READ_LIMIT 256k) When writing to sockets (or pipes) attempt to fill kernel socket bufs. (e.g. MAX_WRITE_LIMIT 256k)
* [core] clear buffer after backend dechunkGlenn Strauss2021-09-091-0/+2
| | | | | | | | | | (thx flynn) clear buffer after backend dechunk if not sending chunked to client x-ref: "Memory fragmentation with HTTP/2 enabled" https://redmine.lighttpd.net/issues/3084
* [multiple] remove base.h include where not usedGlenn Strauss2021-09-081-1/+1
| | | | | (substitute request.h if file only accesses request_st, and not connection or server structs)
* [core] http_response_append_{buffer,mem}()Glenn Strauss2021-09-081-42/+7
| | | | | | | | | | manage r->resp_body_scratchpad in new funcs http_response_append_buffer() http_response_append_mem() rather than http_chunk_decode_append_buffer() http_chunk_decode_append_mem() which now only decode chunked encoding, more apropos for the func names
* [core] reduce excess cc inlining in http_chunk.cGlenn Strauss2021-08-271-0/+10
|
* [multiple] reduce redundant NULL buffer checksGlenn Strauss2021-08-271-11/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit is a large set of code changes and results in removal of hundreds, perhaps thousands, of CPU instructions, a portion of which are on hot code paths. Most (buffer *) used by lighttpd are not NULL, especially since buffers were inlined into numerous larger structs such as request_st and chunk. In the small number of instances where that is not the case, a NULL check is often performed earlier in a function where that buffer is later used with a buffer_* func. In the handful of cases that remained, a NULL check was added, e.g. with r->http_host and r->conf.server_tag. - check for empty strings at config time and set value to NULL if blank string will be ignored at runtime; at runtime, simple pointer check for NULL can be used to check for a value that has been set and is not blank ("") - use buffer_is_blank() instead of buffer_string_is_empty(), and use buffer_is_unset() instead of buffer_is_empty(), where buffer is known not to be NULL so that NULL check can be skipped - use buffer_clen() instead of buffer_string_length() when buffer is known not to be NULL (to avoid NULL check at runtime) - use buffer_truncate() instead of buffer_string_set_length() to truncate string, and use buffer_extend() to extend Examples where buffer known not to be NULL: - cpv->v.b from config_plugin_values_init is not NULL if T_CONFIG_BOOL (though we might set it to NULL if buffer_is_blank(cpv->v.b)) - address of buffer is arg (&foo) (compiler optimizer detects this in most, but not all, cases) - buffer is checked for NULL earlier in func - buffer is accessed in same scope without a NULL check (e.g. b->ptr) internal behavior change: callers must not pass a NULL buffer to some funcs. - buffer_init_buffer() requires non-null args - buffer_copy_buffer() requires non-null args - buffer_append_string_buffer() requires non-null args - buffer_string_space() requires non-null arg
* [core] remove some (now) unused http_chunk APIsGlenn Strauss2021-05-141-37/+0
| | | | | | | remove http_chunk_append_file() and http_chunk_append_file_range() callers should choose to use stat_cache_entry_open() for caching or should open file and check sizes and ranges
* [core] range chk http_chunk_append_file_ref_rangeGlenn Strauss2021-05-141-1/+6
| | | | | add range sanity check in http_chunk_append_file_ref_range() (before potentially sending HTTP/1.1 chunked header)
* [core] fix chunkqueue_small_resp_optim partial rdGlenn Strauss2021-03-221-2/+2
|
* [core] fix decoding chunked from backend (fixes #3049)Glenn Strauss2020-12-171-50/+49
| | | | | | | | | | | | | (thx flynn) fix decoding chunked from backend truncate response and error out if backend sends excess data after chunked encoding x-ref: "Too much content with HTTP/2.0" https://redmine.lighttpd.net/issues/3049
* [core] track Content-Length from backend (fixes #3046)Glenn Strauss2020-12-161-3/+37
| | | | | | | | track Content-Length from backend in r->resp_body_scratchpad x-ref: "Failure on second request in http proxy backend" https://redmine.lighttpd.net/issues/3046
* [core] track chunked encoding state from backend (fixes #3046)Glenn Strauss2020-12-141-21/+31
| | | | | | | | | | (thx flynn) track chunked encoding state when parsing backend response x-ref: "Failure on second request in http proxy backend" https://redmine.lighttpd.net/issues/3046
* [core] reject excess data after chunked encoding (#3046)Glenn Strauss2020-12-141-5/+17
| | | | | | | | reject excess data after chunked encoding when parsing backend response x-ref: "Failure on second request in http proxy backend" https://redmine.lighttpd.net/issues/3046
* [core] fix chunked decoding from backend (fixes #3044)Glenn Strauss2020-12-141-1/+8
| | | | | | | | (thx flynn) x-ref: "Socket errors after update to version 1.4.56" https://redmine.lighttpd.net/issues/3044
* [core] consolidate chunk size checksGlenn Strauss2020-11-271-6/+1
|
* [core] always lseek() with shared fdGlenn Strauss2020-10-201-1/+1
| | | | always lseek() with shared fd; remove optim to skip with offset = 0
* [core] http_chunk_append_file_ref_range()Glenn Strauss2020-10-201-0/+34
| | | | | http_chunk_append_file_ref() and http_chunk_append_file_ref_range() to take stat_cache_entry ref and append FILE_CHUNK
* [core] alloc optim reading file, sending chunkedGlenn Strauss2020-10-191-1/+1
| | | | | | avoid potential double-copy due to not enough space for final '\0' in http_chunk_append_read_fd_range() if read size is exactly multiple of 8k and sending chunked response
* [core] pass open fd to http_response_parse_rangeGlenn Strauss2020-10-131-1/+1
|
* [multiple] inline chunkqueue where always alloc'dGlenn Strauss2020-10-111-9/+9
| | | | | | inline struct chunkqueue where always allocated in other structs (memory locality)
* [multiple] inline chunkqueue_length()Glenn Strauss2020-10-111-1/+1
|
* [core] defer optimization to read small filesGlenn Strauss2020-10-111-1/+1
| | | | | | | defer optimization to read small files into memory until after response_start hooks have a chance to run, e.g. until after mod_deflate chooses whether or not to serve file from compressed cache, if deflate.cache-dir is configured
* [core] decode Transfer-Encoding: chunked from gwGlenn Strauss2020-08-021-1/+207
| | | | | | | | | | | | decode Transfer-Encoding: chunked from gw (gateway backends) Transfer-Encoding: chunked is a hop-by-hop header. Handling chunked encoding remove a hurdle for mod_proxy to send HTTP/1.1 requests to backends and be able to handle HTTP/1.1 responses. Other backends ought not to send Transfer-Encoding: chunked, but in practice, some implementations do.
* [multiple] add summaries to top of some modulesGlenn Strauss2020-07-081-5/+6
|
* [multiple] split con, request (very large change)Glenn Strauss2020-07-081-54/+54
| | | | | | | | | | | | | | | | NB: r->tmp_buf == srv->tmp_buf (pointer is copied for quicker access) NB: request read and write chunkqueues currently point to connection chunkqueues; per-request and per-connection chunkqueues are not distinct from one another con->read_queue == r->read_queue con->write_queue == r->write_queue NB: in the future, a separate connection config may be needed for connection-level module hooks. Similarly, might need to have per-request chunkqueues separate from per-connection chunkqueues. Should probably also have a request_reset() which is distinct from connection_reset().
* [multiple] stat_cache singletonGlenn Strauss2020-07-081-1/+2
|
* [core] create http chunk header on the stackGlenn Strauss2020-07-081-153/+147
| | | | streamline code in http_chunk.c
* [multiple] prefer (connection *) to (srv *)Glenn Strauss2020-07-081-39/+40
| | | | | | | | convert all log_error_write() to log_error() and pass (log_error_st *) use con->errh in preference to srv->errh (even though currently same) avoid passing (server *) when previously used only for logging (errh)
* [core] stat_cache_path_contains_symlink use errhGlenn Strauss2020-07-081-4/+4
| | | | use log_error() with con->errh
* [core] adjust http_chunk read() retry loopGlenn Strauss2019-05-181-1/+1
|
* [core] chunkqueue perf: read small files into memGlenn Strauss2019-05-131-2/+37
|
* [core] http_chunk_append_file_fd()Glenn Strauss2019-05-041-3/+6
|
* [stat_cache] separate func for symlink policy chkGlenn Strauss2019-04-291-4/+3
| | | | | Note: historical ToC-ToU race condition still exists in implementation server.follow-symlink = "disable" is not recommended (default: "enable")
* [core] pass conf.follow_symlink in more placesGlenn Strauss2019-03-101-1/+1
|
* [core] perf: simple, quick buffer_clear()Glenn Strauss2018-11-231-1/+1
| | | | | | | | | quickly clear buffer instead of buffer_string_set_length(b, 0) or buffer_reset(b). Avoids free() of large buffers about to be reused, or buffers that are module-scoped, persistent, and reused. (buffer_reset() should still be used with buffers in connection *con when the data in the buffers is supplied by external, untrusted source)
* [mod_fastcgi] perf: reduce data copiesGlenn Strauss2018-11-121-40/+94
| | | | http_chunk_transfer_cqlen()
* [core] perf: copy small strings; better buf reuseGlenn Strauss2018-10-271-1/+1
| | | | | copy small strings to write queue for better buffer reuse (instead of swapping with larger buffers in write chunkqueue)
* [core] code reuse with http_response_body_clear()Glenn Strauss2018-09-231-7/+7
| | | | | code reuse with http_response_body_clear() rename con->response.transfer_encoding to con->response.send_chunked
* [core] some header cleanupGlenn Strauss2018-04-081-1/+1
| | | | | | | | | provide standard types in first.h instead of base.h provide lighttpd types in base_decls.h instead of settings.h reduce headers exposed by headers for core data structures do not expose <pcre.h> or <stdlib.h> in headers move stat_cache_entry to stat_cache.h reduce use of "server.h" and "base.h" in headers
* [core] include "fdevent.h" where neededGlenn Strauss2017-03-281-0/+1
| | | | (instead of providing #include "fdevent.h" in base.h)
* [core] remove some unused header includesGlenn Strauss2017-03-281-2/+0
| | | | remove exposure of stdio.h in buffer.h for print_backtrace(), now static
* [core] option to stream response body to client (fixes #949, #760, #1283, #1387)Glenn Strauss2016-06-191-1/+6
| | | | | | | | | | | | | | | | | | | | | Set server.stream-response-body = 1 or server.stream-response-body = 2 to have lighttpd stream response body to client as it arrives from the backend (CGI, FastCGI, SCGI, proxy). default: buffer entire response body before sending response to client. (This preserves existing behavior for now, but may in the future be changed to stream response to client, which is the behavior more commonly expected.) x-ref: "fastcgi, cgi, flush, php5 problem." https://redmine.lighttpd.net/issues/949 "Random crashing on FreeBSD 6.1" https://redmine.lighttpd.net/issues/760 "Memory usage increases when proxy+ssl+large file" https://redmine.lighttpd.net/issues/1283 "lighttpd+fastcgi memory problem" https://redmine.lighttpd.net/issues/1387
* [core] buffer large responses to tempfiles (fixes #758, fixes #760, fixes ↵Glenn Strauss2016-06-121-17/+50
| | | | | | | | | | | | | | | | | | | | | | | | | | #933, fixes #1387, #1283, fixes #2083) This replaces buffering entire response in memory which might lead to huge memory footprint and possibly to memory exhaustion. use tempfiles of fixed size so disk space is freed as each file sent update callers of http_chunk_append_mem() and http_chunk_append_buffer() to handle failures when writing to tempfile. x-ref: "memory fragmentation leads to high memory usage after peaks" https://redmine.lighttpd.net/issues/758 "Random crashing on FreeBSD 6.1" https://redmine.lighttpd.net/issues/760 "lighty should buffer responses (after it grows above certain size) on disk" https://redmine.lighttpd.net/issues/933 "Memory usage increases when proxy+ssl+large file" https://redmine.lighttpd.net/issues/1283 "lighttpd+fastcgi memory problem" https://redmine.lighttpd.net/issues/1387 "Excessive Memory usage with streamed files from PHP" https://redmine.lighttpd.net/issues/2083
* [core] open fd when appending file to cq (fixes #2655)Glenn Strauss2016-04-181-10/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | http_chunk_append_file() opens fd when appending file to chunkqueue. Defers calculation of content length until response is finished. This reduces race conditions pertaining to stat() and then (later) open(), when the result of the stat() was used for Content-Length or to generate chunked headers. Note: this does not change how lighttpd handles files that are modified in-place by another process after having been opened by lighttpd -- don't do that. This *does* improve handling of files that are frequently modified via a temporary file and then atomically renamed into place. mod_fastcgi has been modified to use http_chunk_append_file_range() with X-Sendfile2 and will open the target file multiple times if there are multiple ranges. Note: (future todo) not implemented for chunk.[ch] interfaces used by range requests in mod_staticfile or by mod_ssi. Those uses could lead to too many open fds. For mod_staticfile, limits should be put in place for max number of ranges accepted by mod_staticfile. For mod_ssi, limits would need to be placed on the maximum number of includes, and the primary SSI file split across lots of SSI directives should either copy the pieces or perhaps chunk.h could be extended to allow for an open fd to be shared across multiple chunks. Doing either of these would improve the performance of SSI since they would replace many file opens on the pieces of the SSI file around the SSI directives. x-ref: "Serving a file that is getting updated can cause an empty response or incorrect content-length error" https://redmine.lighttpd.net/issues/2655 github: Closes #49
* consistent inclusion of config.h at top of files (fixes #2073)Glenn Strauss2016-03-191-0/+2
| | | | | | From: Glenn Strauss <gstrauss@gluelogic.com> git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@3113 152afb58-edef-0310-8abb-c4023f1b3aa9
* rename buffer_append_long_hex to buffer_append_uint_hexStefan Bühler2015-02-081-19/+3
| | | | | | | | | * takes uintmax_t now * use in http_chunk_append_len From: Stefan Bühler <stbuehler@web.de> git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2980 152afb58-edef-0310-8abb-c4023f1b3aa9
* Use buffer API to read and modify "used" memberStefan Bühler2015-02-081-3/+2
| | | | | | | | | | | | | | | | | | | | | | | - a lot of code tried to handle manually adding terminating zeroes and keeping track of the correct "used" count. Replaced all "external" usages with simple wrapper functions: * buffer_string_is_empty (used <= 1), buffer_is_empty (used == 0); prefer buffer_string_is_empty * buffer_string_set_length * buffer_string_length * CONST_BUF_LEN() macro - removed "static" buffer hacks (buffers pointing to constant/stack memory instead of malloc()ed data) - buffer_append_strftime(): refactor buffer+strftime uses - li_tohex(): no need for a buffer for binary-to-hex conversion: the output data length is easy to predict - remove "-Winline" from extra warnings: the "inline" keyword just supresses the warning about unused but defined (static) functions; don't care whether it actually gets inlined or not. From: Stefan Bühler <stbuehler@web.de> git-svn-id: svn://svn.lighttpd.net/lighttpd/branches/lighttpd-1.4.x@2979 152afb58-edef-0310-8abb-c4023f1b3aa9
View Lorry Controller Status