string literals no longer get encoded before being send to Redis

previously all pieces of a command, including the command name and literal
options to it (such as "WITHSCORES" on ZSET commands) would get encoded.
this works fine on utf-8, but other encodings like utf-16 break.

a new Token class has been introduced that command names and literal options
get wrapped. the encoder falls back to the latin-1 encoding for these
literals as they are all ascii.

fixes #430

1 files changed, 9 insertions, 0 deletions

diff --git a/tests/test_encoding.py b/tests/test_encoding.py
index 2aa95d3..b1df0a5 100644
--- a/tests/test_encoding.py
+++ b/tests/test_encoding.py
@@ -22,3 +22,12 @@ class TestEncoding(object):
         result = [unicode_string, unicode_string, unicode_string]
         r.rpush('a', *result)
         assert r.lrange('a', 0, -1) == result
+
+
+class TestCommandsAndTokensArentEncoded(object):
+    @pytest.fixture()
+    def r(self, request):
+        return _redis_client(request=request, charset='utf-16')
+
+    def test_basic_command(self, r):
+        r.set('hello', 'world')