Add tests and docs

10 files changed, 183 insertions, 50 deletions

diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 86b3dbe..dc9454d 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -31,7 +31,7 @@ To set up for local development (requires [poetry](https://python-poetry.org/doc
 ```bash
 $ git clone https://github.com/reclosedev/requests-cache.git
 $ cd requests-cache
-$ poetry install -E backends
+$ poetry install -v -E all
 ```
 
 ## Pre-commit Hooks
diff --git a/HISTORY.md b/HISTORY.md
index 5a07c3a..c7a80fb 100644
--- a/HISTORY.md
+++ b/HISTORY.md
@@ -1,6 +1,6 @@
 # History
 
-## 0.7.0 (2021-06-TBD)
+## 0.7.0 (2021-07-TBD)
 [See all issues and PRs for 0.7](https://github.com/reclosedev/requests-cache/milestone/2?closed=1)
 
 ### Backends
@@ -29,19 +29,34 @@
 * Add support for bypassing the cache if `expire_after=0`
 * Add support for making a cache whitelist using URL patterns
 
+### Serialization
+* Add data models for all serialized objects
+* Add a BSON serializer
+* Add a JSON serializer
+* Add optional support for `cattrs`
+* Add optional support for `ultrajson`
+
 ### General
 * Add option to manually cache response objects with `BaseCache.save_response()`
 * Add `BaseCache.keys()` and `values()` methods
 * Show summarized response details with `str(CachedResponse)`
 * Add more detailed repr methods for `CachedSession`, `CachedResponse`, and `BaseCache`
 * Add support for caching multipart form uploads
-* Update `BaseCache.urls` to only skip invalid responses, not delete them
+* Update `BaseCache.urls` to only skip invalid responses, not delete them (for better performance)
 * Update `old_data_on_error` option to also handle error response codes
 * Update `ignored_parameters` to also exclude ignored request params, body params, or headers from cached response data (to avoid storing API keys or other credentials)
 * Only log request exceptions if `old_data_on_error` is set
+
+### Compatibility, packaging, and tests
 * Fix some compatibility issues with `requests 2.17` and `2.18`
 * Add minimum `requests` version of `2.17`
 * Run tests for each supported version of `requests`
+* Add some package extras to install optional dependencies (via `pip install`):
+    * `requests-cache[bson]`
+    * `requests-cache[json]`
+    * `requests-cache[dynamodb]`
+    * `requests-cache[mongodb]`
+    * `requests-cache[redis]`
 * Packaging is now handled with Poetry. For users, installation still works the same. For developers,
   see [Contributing Guide](https://requests-cache.readthedocs.io/en/stable/contributing.html) for details
 
@@ -74,6 +89,21 @@ Thanks to [Code Shelter](https://www.codeshelter.co) and
 [contributors](https://requests-cache.readthedocs.io/en/stable/contributors.html)
 for making this release possible!
 
+### Backends
+* SQLite: Allow passing user paths (`~/path-to-cache`) to database file with `db_path` param
+* SQLite: Add `timeout` parameter
+* Make default table names consistent across backends (`'http_cache'`)
+
+### Expiration
+* Cached responses are now stored with an absolute expiration time, so `CachedSession.expire_after`
+  no longer applies retroactively. To revalidate previously cached items with a new expiration time,
+  see below:
+* Add support for overriding original expiration (i.e., revalidating) in `CachedSession.remove_expired_responses()`
+* Add support for setting expiration for individual requests
+* Add support for setting expiration based on URL glob patterns
+* Add support for setting expiration as a `datetime`
+* Add support for explicitly disabling expiration with `-1` (Since `None` may be ambiguous in some cases)
+
 ### Serialization
 **Note:** Due to the following changes, responses cached with previous versions of requests-cache
 will be invalid. These **old responses will be treated as expired**, and will be refreshed the
@@ -91,21 +121,6 @@ next time they are requested. They can also be manually converted or removed, if
 * Add `BaseCache.urls` property to get all URLs persisted in the cache
 * Add optional support for `itsdangerous` for more secure serialization
 
-### Backends
-* SQLite: Allow passing user paths (`~/path-to-cache`) to database file with `db_path` param
-* SQLite: Add `timeout` parameter
-* Make default table names consistent across backends (`'http_cache'`)
-
-### Expiration
-* Cached responses are now stored with an absolute expiration time, so `CachedSession.expire_after`
-  no longer applies retroactively. To revalidate previously cached items with a new expiration time,
-  see below:
-* Add support for overriding original expiration (i.e., revalidating) in `CachedSession.remove_expired_responses()`
-* Add support for setting expiration for individual requests
-* Add support for setting expiration based on URL glob patterns
-* Add support for setting expiration as a `datetime`
-* Add support for explicitly disabling expiration with `-1` (Since `None` may be ambiguous in some cases)
-
 ### Bugfixes
 * Fix caching requests with data specified in `json` parameter
 * Fix caching requests with `verify` parameter
diff --git a/docs/advanced_usage.rst b/docs/advanced_usage.rst
index 8e5ffd6..2f19106 100644
--- a/docs/advanced_usage.rst
+++ b/docs/advanced_usage.rst
@@ -7,22 +7,6 @@ This section covers some more advanced and use-case-specific features.
 .. contents::
     :local:
 
-Custom Response Filtering
--------------------------
-If you need more advanced behavior for determining what to cache, you can provide a custom filtering
-function via the ``filter_fn`` param. This can by any function that takes a :py:class:`requests.Response`
-object and returns a boolean indicating whether or not that response should be cached. It will be applied
-to both new responses (on write) and previously cached responses (on read). Example:
-
-    >>> from sys import getsizeof
-    >>> from requests_cache import CachedSession
-    >>>
-    >>> def filter_by_size(response):
-    >>>     """Don't cache responses with a body over 1 MB"""
-    >>>     return getsizeof(response.content) <= 1024 * 1024
-    >>>
-    >>>    session = CachedSession(filter_fn=filter_by_size)
-
 Cache Inspection
 ----------------
 Here are some ways to get additional information out of the cache session, backend, and responses:
@@ -87,10 +71,28 @@ combined keys and responses.
     >>> print('All cache keys for redirects and responses combined:')
     >>> print(list(session.cache.keys()))
 
+Custom Response Filtering
+-------------------------
+If you need more advanced behavior for determining what to cache, you can provide a custom filtering
+function via the ``filter_fn`` param. This can by any function that takes a :py:class:`requests.Response`
+object and returns a boolean indicating whether or not that response should be cached. It will be applied
+to both new responses (on write) and previously cached responses (on read). Example:
+
+    >>> from sys import getsizeof
+    >>> from requests_cache import CachedSession
+    >>>
+    >>> def filter_by_size(response):
+    >>>     """Don't cache responses with a body over 1 MB"""
+    >>>     return getsizeof(response.content) <= 1024 * 1024
+    >>>
+    >>>    session = CachedSession(filter_fn=filter_by_size)
+
 Custom Backends
 ---------------
 If the built-in :py:mod:`Cache Backends <requests_cache.backends>` don't suit your needs, you can
-create your own by making subclasses of :py:class:`.BaseCache` and :py:class:`.BaseStorage`:
+create your own by making subclasses of :py:class:`.BaseCache` and :py:class:`.BaseStorage`.
+
+Example:
 
     >>> from requests_cache import CachedSession
     >>> from requests_cache.backends import BaseCache, BaseStorage
@@ -131,6 +133,30 @@ You can then use your custom backend in a :py:class:`.CachedSession` with the ``
 
     >>> session = CachedSession(backend=CustomCache())
 
+Custom Serializers
+------------------
+If the built-in :ref:`serializers` don't suit your needs, you can create your own by subclassing
+:py:class:`.BaseSerializer`.
+
+Example using an imaginary ``xson`` module that provides ``dumps`` and ``loads`` functions:
+
+    >>> import xson
+    >>> from requests_cache.serializers import BaseSerializer
+    >>>
+    >>> class CustomSerializer(BaseSerializer):
+    ...     """Serializer that converts responses to XSON"""
+    ...
+    ...     def __init__(self, *args, **kwargs):
+    ...         super().__init__(*args, **kwargs)
+    ...
+    ...     def dumps(self, response: CachedResponse) -> bytes:
+    ...         unstructured_response = super().unstructure(response)
+    ...         return xson.dumps(unstructured_response)
+    ...
+    ...     def loads(self, obj: bytes) -> CachedResponse:
+    ...         unstructured_response = xson.loads(obj)
+    ...         return super().structure(unstructured_response)
+
 Usage with other requests features
 ----------------------------------
 
diff --git a/tests/sample_response.json b/docs/sample_response.json
index 1c8e198..1c8e198 100644
--- a/tests/sample_response.json
+++ b/docs/sample_response.json
diff --git a/docs/user_guide.rst b/docs/user_guide.rst
index d089aeb..47cc01f 100644
--- a/docs/user_guide.rst
+++ b/docs/user_guide.rst
@@ -316,6 +316,53 @@ revalidate the cache with the new expiration time:
 
     >>> session.remove_expired_responses(expire_after=timedelta(days=30))
 
+.. _serializers:
+
+Serializers
+-----------
+By default, responses are serialized using :py:mod:`pickle`. Some other options are also available:
+
+.. note:: These features require python 3.7+ and additional dependencies
+
+JSON Serializer
+~~~~~~~~~~~~~~~
+Storing responses as JSON gives you the benefit of making them human-readable, in exchange for a
+slight reduction in performance. This can be especially useful in combination with the filesystem
+backend.
+
+.. admonition:: Example JSON-serialized Response
+    :class: toggle
+
+    .. literalinclude:: sample_response.json
+        :language: JSON
+
+You can install the extra dependencies for this serializer with:
+
+.. code-block:: bash
+
+    pip install requests-cache[json]
+
+BSON Serializer
+~~~~~~~~~~~~~~~
+`BSON <https://www.mongodb.com/json-and-bson>`_ is a serialization format originally created for
+MongoDB, but it can also be used independently. Compared to JSON, it has better performance
+(although still not as fast as ``pickle``), and adds support for additional data types. It is not
+human-readable, but some tools support reading and editing it directly
+(for example, `bson-converter <https://atom.io/packages/bson-converter>`_ for Atom).
+
+You can install the extra dependencies for this serializer with:
+
+.. code-block:: bash
+
+    pip install requests-cache[mongo]
+
+Or if you would like to use the standalone BSON codec for a different backend, without installing
+MongoDB dependencies:
+
+.. code-block:: bash
+
+    pip install requests-cache[bson]
+
 Error Handling
 --------------
 In some cases, you might cache a response, have it expire, but then encounter an error when
diff --git a/requests_cache/serializers/base.py b/requests_cache/serializers/base.py
index 4de5abd..4e7ab9a 100644
--- a/requests_cache/serializers/base.py
+++ b/requests_cache/serializers/base.py
@@ -9,7 +9,6 @@ from urllib3.response import HTTPHeaderDict
 from ..models import CachedResponse
 
 
-# TODO: Document this more thoroughly
 class BaseSerializer:
     """Base serializer class for :py:class:`.CachedResponse` that optionally does
     pre/post-processing with cattrs. This provides an easy starting point for alternative
diff --git a/tests/integration/base_cache_test.py b/tests/integration/base_cache_test.py
index 9abb2f5..a72c3a4 100644
--- a/tests/integration/base_cache_test.py
+++ b/tests/integration/base_cache_test.py
@@ -12,6 +12,7 @@ import requests
 
 from requests_cache import ALL_METHODS, CachedResponse, CachedSession
 from requests_cache.backends.base import BaseCache
+from requests_cache.serializers import SERIALIZER_CLASSES
 from tests.conftest import (
     CACHE_NAME,
     HTTPBIN_FORMATS,
@@ -34,7 +35,10 @@ class BaseCacheTest:
     init_kwargs: Dict = {}
 
     def init_session(self, clear=True, **kwargs) -> CachedSession:
-        kwargs.update({'allowable_methods': ALL_METHODS, 'suppress_warnings': True})
+        kwargs.update(
+            {'allowable_methods': ALL_METHODS, 'suppress_warnings': True, 'secret_key': 'hunter2'}
+        )
+        kwargs.setdefault('serializer', 'pickle')
         backend = self.backend_class(CACHE_NAME, **self.init_kwargs, **kwargs)
         if clear:
             backend.redirects.clear()
@@ -42,22 +46,24 @@ class BaseCacheTest:
 
         return CachedSession(backend=backend, **self.init_kwargs, **kwargs)
 
+    @pytest.mark.parametrize('serializer', SERIALIZER_CLASSES.keys())
     @pytest.mark.parametrize('method', HTTPBIN_METHODS)
     @pytest.mark.parametrize('field', ['params', 'data', 'json'])
-    def test_all_methods(self, field, method):
-        """Test all relevant combinations of methods and data fields. Requests with different request
-        params, data, or json should be cached under different keys.
+    def test_all_methods(self, field, method, serializer):
+        """Test all relevant combinations of methods X data fields X serializers.
+        Requests with different request params, data, or json should be cached under different keys.
         """
         url = httpbin(method.lower())
-        session = self.init_session()
+        session = self.init_session(serializer=serializer)
         for params in [{'param_1': 1}, {'param_1': 2}, {'param_2': 2}]:
             assert session.request(method, url, **{field: params}).from_cache is False
             assert session.request(method, url, **{field: params}).from_cache is True
 
+    @pytest.mark.parametrize('serializer', SERIALIZER_CLASSES.keys())
     @pytest.mark.parametrize('response_format', HTTPBIN_FORMATS)
-    def test_all_response_formats(self, response_format):
-        """Test that all relevant response formats are cached correctly"""
-        session = self.init_session()
+    def test_all_response_formats(self, response_format, serializer):
+        """Test that all relevant combinations of response formats X serializers are cached correctly"""
+        session = self.init_session(serializer=serializer)
         # Temporary workaround for this issue: https://github.com/kevin1024/pytest-httpbin/issues/60
         if response_format == 'json' and USE_PYTEST_HTTPBIN:
             session.allowable_codes = (200, 404)
diff --git a/tests/integration/base_storage_test.py b/tests/integration/base_storage_test.py
index c0f9ce1..8e871fa 100644
--- a/tests/integration/base_storage_test.py
+++ b/tests/integration/base_storage_test.py
@@ -6,6 +6,7 @@ from requests_cache.backends import BaseStorage
 from tests.conftest import CACHE_NAME
 
 
+# TODO: Parameterize tests for all serializers?
 class BaseStorageTest:
     """Base class for testing cache storage dict-like interfaces"""
 
diff --git a/tests/unit/test_serializers.py b/tests/unit/test_serializers.py
new file mode 100644
index 0000000..a920c7d
--- /dev/null
+++ b/tests/unit/test_serializers.py
@@ -0,0 +1,34 @@
+# Note: Almost all serializer logic is covered by parametrized integration tests.
+# Any additional serializer-specific tests can go here.
+import json
+import pytest
+import sys
+from importlib import reload
+from unittest.mock import patch
+
+# TODO: For some reason this doesn't work on python 3.10
+pytestmark = pytest.mark.skipif(
+    (3, 7) > sys.version_info > (3, 9),
+    reason='Requires python 3.7+ version of cattrs',
+)
+
+
+@patch.dict(sys.modules, {'ujson': None, 'cattr.preconf.ujson': None})
+def test_stdlib_json():
+    import requests_cache.serializers.json_serializer
+
+    reload(requests_cache.serializers.json_serializer)
+    from requests_cache.serializers.json_serializer import json as module_json
+
+    assert module_json is json
+
+
+def test_ujson():
+    import ujson
+
+    import requests_cache.serializers.json_serializer
+
+    reload(requests_cache.serializers.json_serializer)
+    from requests_cache.serializers.json_serializer import json as module_json
+
+    assert module_json is ujson
diff --git a/tests/unit/test_session.py b/tests/unit/test_session.py
index 6d6bbde..96cb62d 100644
--- a/tests/unit/test_session.py
+++ b/tests/unit/test_session.py
@@ -20,8 +20,9 @@ from requests_cache.backends import (
     BaseCache,
     DbDict,
     DbPickleDict,
-    get_placeholder_backend,
+    get_placeholder_class,
 )
+from requests_cache.cache_keys import url_to_key
 from requests_cache.models import CachedResponse
 from requests_cache.serializers import PickleSerializer, SafePickleSerializer
 from tests.conftest import (
@@ -39,7 +40,7 @@ def test_init_unregistered_backend():
         CachedSession(backend='nonexistent')
 
 
-@patch.dict(BACKEND_CLASSES, {'mongo': get_placeholder_backend()})
+@patch.dict(BACKEND_CLASSES, {'mongo': get_placeholder_class()})
 def test_init_missing_backend_dependency():
     """Test that the correct error is thrown when a user does not have a dependency installed"""
     with pytest.raises(ImportError):
@@ -472,13 +473,17 @@ def test_remove_expired_responses__error(mock_session):
     # Start with two cached responses, one of which will raise an error
     mock_session.get(MOCKED_URL)
     mock_session.get(MOCKED_URL_JSON)
-    side_effects = [mock_session.get(MOCKED_URL_JSON), PickleError, PickleError]
 
-    with patch.object(DbPickleDict, '__getitem__', side_effect=side_effects):
+    def error_on_key(key):
+        if key == url_to_key(MOCKED_URL_JSON):
+            raise PickleError
+        return mock_session.get(MOCKED_URL_JSON)
+
+    with patch.object(DbPickleDict, '__getitem__', side_effect=error_on_key):
         mock_session.remove_expired_responses()
     assert len(mock_session.cache.responses) == 1
-    assert mock_session.get(MOCKED_URL).from_cache is False
-    assert mock_session.get(MOCKED_URL_JSON).from_cache is True
+    assert mock_session.get(MOCKED_URL).from_cache is True
+    assert mock_session.get(MOCKED_URL_JSON).from_cache is False
 
 
 def test_remove_expired_responses__extend_expiration(mock_session):