relocated many of the crypto routes inside passlib.utils,

and moved them to a separate passlib.crypto subpackage.

along with this move, made a few api cleanups:

* unified all code that's looking up hashes
  to use new passlib.crypto.lookup_hash() wrapper, which takes
  care of hash name normalization, loading fallback implementations,
  and alg metadata inspection, all hidden behind a memoized function.

* deprecated pbkdf2() in favor of pbkdf2_hmac() -- only real use,
  and new signature matches stdlib function.  additionally,
  this version is a bit faster, due to some assumptions that
  can be made due to the PRF always being HMAC based.

* added compile_hmac() helper which does an even more efficient
  job of pre-compiling a keyed HMAC function; this helped speed up pbkdf2 a bit more.

14 files changed, 91 insertions, 54 deletions

diff --git a/docs/conf.py b/docs/conf.py
index 9ddc460..fbda296 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -33,10 +33,6 @@ options = os.environ.get("PASSLIB_DOCS", "").split(",")
 # (https://bitbucket.org/ecollins/cloud_sptheme)
 import cloud_sptheme as csp
 
-# hack to make autodoc generate documentation from the correct class...
-import passlib.utils.md4 as md4_mod
-md4_mod.md4 = md4_mod._builtin_md4
-
 #=============================================================================
 # General configuration
 #=============================================================================
diff --git a/docs/lib/passlib.crypto.des.rst b/docs/lib/passlib.crypto.des.rst
new file mode 100644
index 0000000..89ad0bd
--- /dev/null
+++ b/docs/lib/passlib.crypto.des.rst
@@ -0,0 +1,26 @@
+==============================================
+:mod:`passlib.crypto.des` - DES routines
+==============================================
+
+.. module:: passlib.crypto.des
+    :synopsis: routines for performing DES encryption
+
+.. versionchanged:: 1.7
+
+    This module was relocated from :mod:`!passlib.utils.des`;
+    the old location will be removed in Passlib 2.0.
+
+.. warning::
+
+    NIST has declared DES to be "inadequate" for cryptographic purposes.
+    These routines, and the password hashes based on them,
+    should not be used in new applications.
+
+This module contains routines for encrypting blocks of data using the DES algorithm.
+Note that these functions do not support multi-block operation or decryption,
+since they are designed primarily for use in password hash algorithms
+(such as :class:`~passlib.hash.des_crypt` and :class:`~passlib.hash.bsdi_crypt`).
+
+.. autofunction:: expand_des_key
+.. autofunction:: des_encrypt_block
+.. autofunction:: des_encrypt_int_block
diff --git a/docs/lib/passlib.crypto.digest.rst b/docs/lib/passlib.crypto.digest.rst
new file mode 100644
index 0000000..fda5e80
--- /dev/null
+++ b/docs/lib/passlib.crypto.digest.rst
@@ -0,0 +1,39 @@
+=============================================================
+:mod:`passlib.crypto.digest` - Hash & Related Helpers
+=============================================================
+
+.. module:: passlib.crypto.digest
+    :synopsis: Internal cryptographic helpers
+
+.. versionadded:: 1.7
+
+This module provides various cryptographic support functions used by Passlib
+to implement the various password hashes it provides, as well as paper over
+some VM & version incompatibilities.
+
+Hash Functions
+==============
+.. autofunction:: norm_hash_name
+.. autofunction:: lookup_hash
+
+.. autoclass:: HashInfo()
+
+.. note::
+
+    :func:`!lookup_hash` supports all hashes available directly in :mod:`hashlib`,
+    as well as offered through :func:`hashlib.new`.
+    It will also fallback to passlib's builtin MD4 implementation if one is not natively available.
+
+HMAC Functions
+==============
+
+.. autofunction:: compile_hmac
+
+PKCS#5 Key Derivation Functions
+===============================
+.. autofunction:: pbkdf1
+.. autofunction:: pbkdf2_hmac
+
+.. note::
+
+    The details of PBKDF1 and PBKDF2 are specified in :rfc:`2898`.
diff --git a/docs/lib/passlib.hash.fshp.rst b/docs/lib/passlib.hash.fshp.rst
index 28efbde..1ebcc6f 100644
--- a/docs/lib/passlib.hash.fshp.rst
+++ b/docs/lib/passlib.hash.fshp.rst
@@ -77,7 +77,7 @@ A example hash (of ``password``) is:
     ``da8a6bcfad78c17da993b5940f6524d526d444012cf67ea8f44b7843c7ab2e82``
 
 FSHP is basically just a wrapper around PBKDF1:
-The checksum is calculated using :func:`~passlib.utils.pbkdf2.pbkdf1`,
+The checksum is calculated using :func:`~passlib.crypto.digest.pbkdf1`,
 passing in the password, the decoded salt string, the number of
 rounds, and hash function specified by the variant identifier.
 FSHP has one quirk in that the password is passed in as the pbkdf1 salt,
diff --git a/docs/lib/passlib.hash.grub_pbkdf2_sha512.rst b/docs/lib/passlib.hash.grub_pbkdf2_sha512.rst
index 891f2ab..686f98d 100644
--- a/docs/lib/passlib.hash.grub_pbkdf2_sha512.rst
+++ b/docs/lib/passlib.hash.grub_pbkdf2_sha512.rst
@@ -43,7 +43,7 @@ and :samp:`{checksum}` is the resulting 64-byte derived key, also
 encoded in upper-case hexadecimal. It can be identified by the prefix ``grub.pdkdf2.sha512.``.
 
 The algorithm used is the same as :class:`pbkdf2_sha1`: the password is encoded into UTF-8 if not already encoded,
-and passed through :func:`~passlib.utils.pbkdf2.pbkdf2`
+and passed through :func:`~passlib.crypto.digest.pbkdf1`
 along with the decoded salt, and the number of rounds.
 The result is then encoded into hexadecimal.
 
diff --git a/docs/lib/passlib.hash.msdcc2.rst b/docs/lib/passlib.hash.msdcc2.rst
index f89edf1..8a100e8 100644
--- a/docs/lib/passlib.hash.msdcc2.rst
+++ b/docs/lib/passlib.hash.msdcc2.rst
@@ -69,7 +69,7 @@ The digest is calculated as follows:
    digest from step 2; and the MD4 digest of the result
    is calculated (The result of this is identical to the
    :class:`~passlib.hash.msdcc` digest).
-5. :func:`PBKDF2-HMAC-SHA1 <passlib.utils.pbkdf2.pbkdf2>` is then invoked,
+5. :func:`PBKDF2-HMAC-SHA1 <passlib.crypto.digest.pbkdf2_hmac>` is then invoked,
    using the result of step 4 as the secret, the username from step 3 as
    the salt, 10240 rounds, and resulting in a 16 byte digest.
 6. The result of step 5 is encoded into hexadecimal;
diff --git a/docs/lib/passlib.hash.nthash.rst b/docs/lib/passlib.hash.nthash.rst
index eda18b0..92cea05 100644
--- a/docs/lib/passlib.hash.nthash.rst
+++ b/docs/lib/passlib.hash.nthash.rst
@@ -8,7 +8,7 @@
 
 .. warning::
 
-    This scheme is very weak, the :mod:`~passlib.utils.md4` digest
+    This scheme is very weak, the MD4 digest
     it is based on has been severely compromised for many years.
     It should be used for compatibility with existing systems;
     **do not use** in new code.
@@ -45,7 +45,7 @@ A nthash consists of 32 hexadecimal digits, which encode the digest.
 An example hash (of ``password``) is ``8846f7eaee8fb117ad06bdd830b7586c``.
 
 The digest is calculated by encoding the secret using ``UTF-16-LE``,
-taking the :mod:`~passlib.utils.md4` digest, and then encoding
+taking the MD4 digest, and then encoding
 that as hexadecimal.
 
 FreeBSD Variant
diff --git a/docs/lib/passlib.hash.pbkdf2_digest.rst b/docs/lib/passlib.hash.pbkdf2_digest.rst
index c829edf..9a2ec95 100644
--- a/docs/lib/passlib.hash.pbkdf2_digest.rst
+++ b/docs/lib/passlib.hash.pbkdf2_digest.rst
@@ -96,7 +96,7 @@ follow the same format, :samp:`$pbkdf2-{digest}${rounds}${salt}${checksum}`.
 
 The algorithm used by all of these schemes is deliberately identical and simple:
 The password is encoded into UTF-8 if not already encoded,
-and run through :func:`~passlib.utils.pbkdf2.pbkdf2`
+and run through :func:`~passlib.crypto.digest.pbkdf2_hmac`
 along with the decoded salt, the number of rounds,
 and a prf built from HMAC + the respective message digest.
 The result is then encoded using :func:`~passlib.utils.ab64_encode`.
diff --git a/docs/lib/passlib.hash.scram.rst b/docs/lib/passlib.hash.scram.rst
index 3adc8c4..bcd98d4 100644
--- a/docs/lib/passlib.hash.scram.rst
+++ b/docs/lib/passlib.hash.scram.rst
@@ -17,7 +17,7 @@ SCRAM.
 To accomplish this, Passlib provides the following
 :ref:`modular-crypt-format`-compatible password hash scheme which uses the
 ``$scram$`` identifier. This format encodes a salt, rounds settings, and one
-or more :func:`~passlib.utils.pbkdf2.pbkdf2` digests... one digest for each
+or more :func:`~passlib.crypto.digest.pbkdf2_hmac` digests... one digest for each
 of the hash algorithms the server wishes to support over SCRAM.
 
 Since this format is PBKDF2-based, it has equivalent security to
diff --git a/docs/lib/passlib.utils.des.rst b/docs/lib/passlib.utils.des.rst
index 67fef1e..cb40e17 100644
--- a/docs/lib/passlib.utils.des.rst
+++ b/docs/lib/passlib.utils.des.rst
@@ -1,15 +1,14 @@
-=============================================
-:mod:`passlib.utils.des` - DES routines
-=============================================
+====================================================
+:mod:`passlib.utils.des` - DES routines [deprecated]
+====================================================
 
 .. module:: passlib.utils.des
     :synopsis: routines for performing DES encryption
 
 .. warning::
-
-    NIST has declared DES to be "inadequate" for cryptographic purposes.
-    These routines, and the password hashes based on them,
-    should not be used in new applications.
+    This module is deprecated as of Passlib 1.7:
+    It has been relocated to :mod:`passlib.crypto.des`;
+    and the aliases here will be removed in Passlib 2.0.
 
 This module contains routines for encrypting blocks of data using the DES algorithm.
 Note that these functions do not support multi-block operation or decryption,
diff --git a/docs/lib/passlib.utils.md4.rst b/docs/lib/passlib.utils.md4.rst
deleted file mode 100644
index 87125b7..0000000
--- a/docs/lib/passlib.utils.md4.rst
+++ /dev/null
@@ -1,26 +0,0 @@
-========================================================
-:mod:`passlib.utils.md4` - MD4 message digest algorithm
-========================================================
-
-.. module:: passlib.utils.md4
-    :synopsis: MD4 message digest algorithm
-
-.. warning::
-
-    This digest is considered **VERY INSECURE**,
-    and not suitable for any new cryptographic activities.
-    Trivial-cost real-world attacks exist for all
-    password algorithms, stream ciphers, etc, that have
-    been based on MD4.
-    Do not use this hash or derived schemes unless you *really* have to.
-
-This module implements the MD4 hash algorithm in pure python,
-based on the `rfc 1320 <http://www.faqs.org/rfcs/rfc1320.html>`_ specification of MD4.
-
-.. autoclass:: md4
-
-.. note::
-
-    If MD4 support is detected in :mod:`!hashlib`, the :class:`!md4` class in this module
-    will be replaced by a function wrapping :mod:`!hashlib`'s implementation,
-    which should be faster, but otherwise behave exactly the same.
diff --git a/docs/lib/passlib.utils.pbkdf2.rst b/docs/lib/passlib.utils.pbkdf2.rst
index 7fa9f98..cfc503a 100644
--- a/docs/lib/passlib.utils.pbkdf2.rst
+++ b/docs/lib/passlib.utils.pbkdf2.rst
@@ -1,10 +1,17 @@
-=============================================================
-:mod:`passlib.utils.pbkdf2` - PBKDF2 key derivation algorithm
-=============================================================
+==========================================================================
+:mod:`passlib.utils.pbkdf2` - PBKDF2 key derivation algorithm [deprecated]
+==========================================================================
 
 .. module:: passlib.utils.pbkdf2
     :synopsis: PBKDF2 and related key derivation algorithms
 
+.. warning::
+
+    This module has been deprecated as of Passlib 1.7,
+    and will be removed in Passlib 2.0.
+    The functions in this module have been replaced by equivalent
+    (but not identical) functions in the :mod:`passlib.crypto` module.
+
 This module provides a couple of key derivation functions,
 as well as supporting utilities.
 Primarily, it offers :func:`pbkdf2`,
@@ -26,11 +33,5 @@ PKCS#5 Key Derivation Functions
 Helper Functions
 ================
 .. autofunction:: norm_hash_name
-.. autofunction:: get_hash_info
 
 .. autofunction:: get_prf
-.. autofunction:: get_keyed_prf
-
-..
-    given how this module is expanding in scope,
-    perhaps it should be renamed "kdf" or "crypto"?
diff --git a/docs/lib/passlib.utils.rst b/docs/lib/passlib.utils.rst
index db2d3f3..5f26dea 100644
--- a/docs/lib/passlib.utils.rst
+++ b/docs/lib/passlib.utils.rst
@@ -184,6 +184,8 @@ There are also a few sub modules which provide additional utility functions:
     passlib.utils.handlers
     passlib.utils.des
     passlib.utils.pbkdf2
+    passlib.crypto.digest
+    passlib.crypto.des
 
 ..
     passlib.utils.compat
diff --git a/docs/password_hash_api.rst b/docs/password_hash_api.rst
index 8c04965..67e2a0b 100644
--- a/docs/password_hash_api.rst
+++ b/docs/password_hash_api.rst
@@ -722,7 +722,7 @@ and the following attributes should be defined:
 
 Choosing the right rounds value
 ===============================
-For hash algorithms with a variable time-cost, 
+For hash algorithms with a variable time-cost,
 Passlib's :attr:`~PasswordHash.default_rounds` values attempt to be secure enough for
 the average [#avgsys]_ system. But the "right" value for a given hash
 is dependant on the server, its cpu, its expected load, and its users.