Added new document describing the thread pool management.

4 files changed, 140 insertions, 51 deletions

diff --git a/docs/index.txt b/docs/index.txt
index b04440a..c9ecf20 100644
--- a/docs/index.txt
+++ b/docs/index.txt
@@ -82,11 +82,12 @@ Paste for Web Developers
 Documentation
 =============
 
-* `Contributing developer guidelines <DeveloperGuidelines.html>`_
-* `Style guide <StyleGuide.html>`_
 * `Testing applications with Paste <testing-applications.html>`_
 * `URL parsing with WSGI <url-parsing-with-wsgi.html>`_
 * `A Do-It-Yourself Framework <do-it-yourself-framework.html>`_
+* `Paste HTTP Server Thread Pool <paste-httpserver-threadpool.html>`_
+* `Contributing developer guidelines <DeveloperGuidelines.html>`_
+* `Style guide <StyleGuide.html>`_
 
 Components
 ==========
diff --git a/docs/paste-httpserver-threadpool.txt b/docs/paste-httpserver-threadpool.txt
new file mode 100644
index 0000000..fa1ca55
--- /dev/null
+++ b/docs/paste-httpserver-threadpool.txt
@@ -0,0 +1,136 @@
+The Paste HTTP Server Thread Pool
+=================================
+
+This document describes how the thread pool in ``paste.httpserver``
+works, and how it can adapt to problems.
+
+Note all of the configuration parameters listed here are prefixed with
+``threadpool_`` when running through a Paste Deploy configuration.
+
+Error Cases
+-----------
+
+When a WSGI application is called, it's possible that it will block
+indefinitely.  There's two basic ways you can manage threads:
+
+* Start a thread on every request, close it down when the thread stops
+* Start a pool of threads, and reuse those threads for subsequent
+  requests
+
+In both cases things go wrong -- if you start a thread every request
+you will have an explosion of threads, and with it memory and a loss
+of performance.  This can culminate in really high loads, swapping,
+and the whole site grinds to a halt.
+
+If you are using a pool of threads, all the threads can simply be used
+up.  New requests go into a queue to be processed, but since that
+queue never moves forward everyone will just block.  The site
+basically freezes, though memory usage doesn't generally get worse.
+
+Paste Thread Pool
+-----------------
+
+The thread pool in Paste has some options to walk the razor's edge
+between the two techniques, and to try to respond usefully in most
+cases.
+
+The pool tracks all workers threads.  Threads can be in a few states:
+
+* Idle, waiting for a request ("idle")
+* Working on a request
+  - For a reasonable amount of time ("busy")
+  - For an unreasonably long amount of time ("hung")
+* Thread that should die
+  - An exception has been injected that should kill the thread, but it
+    hasn't happened yet ("dying")
+  - An exception has been injected, but the thread has persisted for
+    an unreasonable amount of time ("zombie")
+
+When a request comes in, if there are no idle worker threads waiting
+then the server looks at the workers; all workers are busy or hung.
+If too many are hung, another thread is opened up.  The limit is if
+there are less than ``spawn_if_under`` busy threads.  So if you have
+10 workers, ``spawn_if_under`` is 5, and there are 6 hung threads and
+4 busy threads, another thread will be opened (bringing the number of
+busy threads back to 5).  Later those threads may be collected again
+if some of the threads become un-hung.  A thread is hung if it has
+been working for longer than ``hung_thread_limit`` (default 30
+seconds).
+
+Every so often, the server will check all the threads for error
+conditions.  This happens every ``hung_check_period`` requests
+(default 100).  At this time if there are more than enough threads
+(because of ``spawn_if_under``) some threads may be collected.  If any
+threads have been working for longer than ``kill_thread_limit``
+(default 1800 seconds, i.e., 30 minutes) then the thread will be
+killed.
+
+To kill a thread the ``ctypes`` module must be installed.  This will
+raise an exception (``SystemExit``) in the thread, which should cause
+the thread to stop.  It can take quite a while for this to actually
+take effect, sometimes on the order of several minutes.  This uses a
+non-public API (hence the ``ctypes`` requirement), and so it might not
+work in all cases.  I've tried it in pure Python code and with a hung
+socket, and in both cases it worked.  As soon as the thread is killed
+(before it is actually dead) another worker is added to the pool.
+
+If the killed thread lives longer than ``dying_thread_limit`` (default
+300 seconds, 5 minutes) then it is considered a zombie.
+
+Zombie threads are not handled specially unless you set
+``max_zombies_before_die``.  If you set this and there are more than
+this many zombie threads, then the entire process will be killed.
+This is useful if you are running the server under some process
+monitor, such as ``start-stop-daemon``, ``daemontools``, ``runit``, or
+with ``paster serve --monitor``.  To make the process die, it may run
+``os._exit``, which is considered an impolite way to exit a process
+(akin to ``kill -9``).  It *will* try to run the functions registered
+with ``atexit`` (except for the thread cleanup functions, which are
+the ones which will block so long as there are living threads).
+
+Notification
+------------
+
+If you set ``error_email`` (including setting it globally in a Paste
+Deploy ``[DEFAULT]`` section) then you will be notified of two error
+conditions: when hung threads are killed, and when the process is
+killed due to too many zombie threads.
+
+Missed Cases
+------------
+
+If you have a worker pool size of 10, and 11 slow or hung requests
+come in, the first 10 will get handed off but the server won't know
+yet that they will hang.  The last request will stay stuck in a queue
+until another request comes in.  When a later request comes later
+(after ``hung_thread_limit`` seconds) the server will notice the
+problem and add more threads, and the 11th request will come through.
+
+If a trickle of bad requests keeps coming in, the number of hung
+threads will keep increasing.  At 100 the ``hung_check_period`` may
+not clean them up fast enough.
+
+Killing threads is not something Python really supports.  Corruption
+of the process, memory leaks, or who knows what might occur.  For the
+most part the threads seem to be killed in a fairly simple manner --
+an exception is raised, and ``finally`` blocks do get executed.  But
+this hasn't been tried much in production, so there's not much
+experience with it.
+
+watch_threads
+-------------
+
+If you want to see what's going on in your process, you can install
+the application ``egg:Paste#watch_threads`` (in the
+``paste.debug.watchthreads`` module).  This lets you see requests and
+how long they have been running.  In Python 2.5 you can see tracebacks
+of the running requests; before that you can only see request data
+(URLs, User-Agent, etc).  If you set ``allow_kill = true`` then you
+can also kill threads from the application.  The thread pool is
+intended to run reliably without intervention, but this can help debug
+problems or give you some feeling of what causes problems in the site.
+
+This does open up privacy problems, as it gives you access to all the
+request data in the site, including cookies, IP addresses, etc.  It
+shouldn't be left on in a public setting.
+
diff --git a/docs/template.tmpl b/docs/template.tmpl
deleted file mode 100644
index ed42a38..0000000
--- a/docs/template.tmpl
+++ /dev/null
@@ -1,49 +0,0 @@
-<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
-<html>
-  <head>
-    <title>Paste: $title</title>
-    <link rel="stylesheet" type="text/css" href="/default-site.css">
-    <link rel="stylesheet" type="text/css" href="/style.css">
-    <script type="text/javascript" src="/site.js"></script>
-  </head>
-
-  <body>
-
-<div id="header">
- <h1>Paste &#187;&#187;&#187; $title</h1>
-</div>
-
-<div id="nav">
-<ul>
- <li><a href="/">Home</a></li>
- <li><a href="/download/">Download</a></li>
- <li><a href="/docs/">Documentation</a>
-  <ul>
-   <li><a href="/docs/TodoTutorial.html">To-Do Tutorial</a></li>
-   <li><a href="/deploy/paste-deploy.html">paste.deploy</a></li>
-   <li><a href="/docs/what-is-paste.html">What Is Paste?</a></li>
-   <li><a href="/docs/configuration.html">Configuration</a></li>
-   <li><a href="/docs/testing-applications.html">Testing Applications</a></li>
-   <li><a href="/docs/url-parsing-with-wsgi.html">URL Parsing</a></li>
-   <li><a href="/docs/integration.html">Integrating Frameworks</a></li>
-   <li><a href="/docs/roadmap.html">Roadmap</a></li>
-  </ul></li>
- <li><a href="/community">Community</a>
-  <ul>
-   <li><a href="/community/mailing-list.html">Mailing list</a></li>
-   <li><a href="/trac/">Bug tracker</a></li>
-   <li><a href="/news/">Weblog</a></li>
-   <li><a href="/community/repository.html">Repository</a></li>
-  </ul></li>
-</ul>
-</div>
-
-<div id="body">
-
-$content
-
-</div><!--end body-->
-
-
-  </body>
-</html>
diff --git a/setup.cfg b/setup.cfg
index 2dcef30..84bf3ce 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -14,6 +14,7 @@ docs = docs/index.txt docs/DeveloperGuidelines.txt docs/StyleGuide.txt
        docs/enabled.txt docs/install-example.txt
        docs/related-projects.txt docs/news.txt
        docs/do-it-yourself-framework.txt
+       docs/paste-httpserver-threadpool.txt
 exclude_modules = paste.script paste.deploy paste.webkit 
        paste.util.subprocess24 paste.util.doctest24
        paste.util.string24 paste.util.UserDict24