summaryrefslogtreecommitdiff
path: root/t
diff options
context:
space:
mode:
authorJeff King <peff@peff.net>2015-05-20 03:37:09 -0400
committerJunio C Hamano <gitster@pobox.com>2015-05-25 20:43:18 -0700
commit6bc0cb5176a5e42ca4a74e3558e8f0790ed09bb1 (patch)
tree29d198dce944845ccc55c292671fa23f4fdfefce /t
parentcc969c8dc1cfd1f60b1dab8ee3117172d9d1bb4f (diff)
downloadgit-jk/http-backend-deadlock-2.2.tar.gz
http-backend: spool ref negotiation requests to bufferjk/http-backend-deadlock-2.2
When http-backend spawns "upload-pack" to do ref negotiation, it streams the http request body to upload-pack, who then streams the http response back to the client as it reads. In theory, git can go full-duplex; the client can consume our response while it is still sending the request. In practice, however, HTTP is a half-duplex protocol. Even if our client is ready to read and write simultaneously, we may have other HTTP infrastructure in the way, including the webserver that spawns our CGI, or any intermediate proxies. In at least one documented case[1], this leads to deadlock when trying a fetch over http. What happens is basically: 1. Apache proxies the request to the CGI, http-backend. 2. http-backend gzip-inflates the data and sends the result to upload-pack. 3. upload-pack acts on the data and generates output over the pipe back to Apache. Apache isn't reading because it's busy writing (step 1). This works fine most of the time, because the upload-pack output ends up in a system pipe buffer, and Apache reads it as soon as it finishes writing. But if both the request and the response exceed the system pipe buffer size, then we deadlock (Apache blocks writing to http-backend, http-backend blocks writing to upload-pack, and upload-pack blocks writing to Apache). We need to break the deadlock by spooling either the input or the output. In this case, it's ideal to spool the input, because Apache does not start reading either stdout _or_ stderr until we have consumed all of the input. So until we do so, we cannot even get an error message out to the client. The solution is fairly straight-forward: we read the request body into an in-memory buffer in http-backend, freeing up Apache, and then feed the data ourselves to upload-pack. But there are a few important things to note: 1. We limit the in-memory buffer to prevent an obvious denial-of-service attack. This is a new hard limit on requests, but it's unlikely to come into play. The default value is 10MB, which covers even the ridiculous 100,000-ref negotation in the included test (that actually caps out just over 5MB). But it's configurable on the off chance that you don't mind spending some extra memory to make even ridiculous requests work. 2. We must take care only to buffer when we have to. For pushes, the incoming packfile may be of arbitrary size, and we should connect the input directly to receive-pack. There's no deadlock problem here, though, because we do not produce any output until the whole packfile has been read. For upload-pack's initial ref advertisement, we similarly do not need to buffer. Even though we may generate a lot of output, there is no request body at all (i.e., it is a GET, not a POST). [1] http://article.gmane.org/gmane.comp.version-control.git/269020 Test-adapted-from: Dennis Kaarsemaker <dennis@kaarsemaker.net> Signed-off-by: Jeff King <peff@peff.net> Signed-off-by: Junio C Hamano <gitster@pobox.com>
Diffstat (limited to 't')
-rwxr-xr-xt/t5551-http-fetch-smart.sh11
1 files changed, 11 insertions, 0 deletions
diff --git a/t/t5551-http-fetch-smart.sh b/t/t5551-http-fetch-smart.sh
index c34e930bb9..2d8e5a0fbe 100755
--- a/t/t5551-http-fetch-smart.sh
+++ b/t/t5551-http-fetch-smart.sh
@@ -253,5 +253,16 @@ test_expect_success EXPENSIVE 'clone the 50,000 tag repo to check OS command lin
)
'
+test_expect_success EXPENSIVE 'http can handle enormous ref negotiation' '
+ git -C too-many-refs fetch -q --tags &&
+ (
+ cd "$HTTPD_DOCUMENT_ROOT_PATH/repo.git" &&
+ create_tags 50001 100000
+ ) &&
+ git -C too-many-refs fetch -q --tags &&
+ git -C too-many-refs for-each-ref refs/tags >tags &&
+ test_line_count = 100000 tags
+'
+
stop_httpd
test_done