test_wsgirequest_charset: Use UTF-8 instead of iso-8859-1test_wsgirequest_charset_use_UTF-8_instead_of_iso-8859-1

because it seems that the defacto standard for encoding URIs is to use UTF-8.

I've been reading about url encoding and it seems like perhaps using an
encoding other than UTF-8 is very non-standard and not well-supported (this
test is trying to use `iso-8859-1`).

From http://en.wikipedia.org/wiki/Percent-encoding

> For a non-ASCII character, it is typically converted to its byte sequence in
> UTF-8, and then each byte value is represented as above.

> The generic URI syntax mandates that new URI schemes that provide for the
> representation of character data in a URI must, in effect, represent
> characters from the unreserved set without translation, and should convert
> all other characters to bytes according to UTF-8, and then percent-encode
> those values. This requirement was introduced in January 2005 with the
> publication of RFC 3986

From http://tools.ietf.org/html/rfc3986:

> Non-ASCII characters must first be encoded according to UTF-8 [STD63], and
> then each octet of the corresponding UTF-8 sequence must be percent-encoded
> to be represented as URI characters.  URI producing applications must not use
> percent-encoding in host unless it is used to represent a UTF-8 character
> sequence.

From http://tools.ietf.org/html/rfc3987:

> Conversions from URIs to IRIs MUST NOT use any character encoding other than
> UTF-8 in steps 3 and 4, even if it might be possible to guess from the
> context that another character encoding than UTF-8 was used in the URI.  For
> example, the URI "http://www.example.org/r%E9sum%E9.html" might with some
> guessing be interpreted to contain two e-acute characters encoded as
> iso-8859-1. It must not be converted to an IRI containing these e-acute
> characters.  Otherwise, in the future the IRI will be mapped to
> "http://www.example.org/r%C3%A9sum%C3%A9.html", which is a different URI from
> "http://www.example.org/r%E9sum%E9.html".

See issue #7, which I think this at least partially fixes.

1 files changed, 93 insertions, 0 deletions

diff --git a/paste/debug/testserver.py b/paste/debug/testserver.py
new file mode 100755
index 0000000..8044c7c
--- /dev/null
+++ b/paste/debug/testserver.py
@@ -0,0 +1,93 @@
+# (c) 2005 Clark C. Evans
+# This module is part of the Python Paste Project and is released under
+# the MIT License: http://www.opensource.org/licenses/mit-license.php
+# This code was written with funding by http://prometheusresearch.com
+"""
+WSGI Test Server
+
+This builds upon paste.util.baseserver to customize it for regressions
+where using raw_interactive won't do.
+
+
+"""
+import time
+from paste.httpserver import *
+
+class WSGIRegressionServer(WSGIServer):
+    """
+    A threaded WSGIServer for use in regression testing.  To use this
+    module, call serve(application, regression=True), and then call
+    server.accept() to let it handle one request.  When finished, use
+    server.stop() to shutdown the server. Note that all pending requests
+    are processed before the server shuts down.
+    """
+    defaulttimeout = 10
+    def __init__ (self, *args, **kwargs):
+        WSGIServer.__init__(self, *args, **kwargs)
+        self.stopping = []
+        self.pending = []
+        self.timeout = self.defaulttimeout
+        # this is a local connection, be quick
+        self.socket.settimeout(2)
+    def serve_forever(self):
+        from threading import Thread
+        thread = Thread(target=self.serve_pending)
+        thread.start()
+    def reset_expires(self):
+        if self.timeout:
+            self.expires = time.time() + self.timeout
+    def close_request(self, *args, **kwargs):
+        WSGIServer.close_request(self, *args, **kwargs)
+        self.pending.pop()
+        self.reset_expires()
+    def serve_pending(self):
+        self.reset_expires()
+        while not self.stopping or self.pending:
+            now = time.time()
+            if now > self.expires and self.timeout:
+                # note regression test doesn't handle exceptions in
+                # threads very well; so we just print and exit
+                print("\nWARNING: WSGIRegressionServer timeout exceeded\n")
+                break
+            if self.pending:
+                self.handle_request()
+            time.sleep(.1)
+    def stop(self):
+        """ stop the server (called from tester's thread) """
+        self.stopping.append(True)
+    def accept(self, count = 1):
+        """ accept another request (called from tester's thread) """
+        assert not self.stopping
+        [self.pending.append(True) for x in range(count)]
+
+def serve(application, host=None, port=None, handler=None):
+    server = WSGIRegressionServer(application, host, port, handler)
+    print("serving on %s:%s" % server.server_address)
+    server.serve_forever()
+    return server
+
+if __name__ == '__main__':
+    from six.moves.urllib.request import urlopen
+    from paste.wsgilib import dump_environ
+    server = serve(dump_environ)
+    baseuri = ("http://%s:%s" % server.server_address)
+
+    def fetch(path):
+        # tell the server to humor exactly one more request
+        server.accept(1)
+        # not needed; but this is what you do if the server
+        # may not respond in a resonable time period
+        import socket
+        socket.setdefaulttimeout(5)
+        # build a uri, fetch and return
+        return urlopen(baseuri + path).read()
+
+    assert "PATH_INFO: /foo" in fetch("/foo")
+    assert "PATH_INFO: /womble" in fetch("/womble")
+
+    # ok, let's make one more final request...
+    server.accept(1)
+    # and then schedule a stop()
+    server.stop()
+    # and then... fetch it...
+    urlopen(baseuri)