Bug 1648822 Add stricter validation of DH keys when in FIPS mode.

Update:
FIPS now also requires us to do y^q mod p testing on key generation (always).
We now do that in FIPS mode only, but in all modes we do full DH verification
for DH and ECDH. Because of this, the path has now separated out the prime
checks, which are now only done for the DH operation if we aren't using a known
prime and the subprime value has been provided. I've also learned we can accept
keys that we do full validation on in FIPS mode, so I've added that to
this patch, though we still can't generate those kinds of keys without
adding the subprime at keygen time.

The new FIPS standard is dh operations must use approved primes. Approved
primes are those selected in the tls and ike RFCs. Currently tls and ike
have modes with checks whether the primes are approved, but the check may
not always happen. The safest thing to do in FIPS mode is only allow those
primes. In addition, FIPS requires 1< y < p-1 (or technically 2<=y<=p-2, since
y is an integer those two tests are identical).

While making changes I realized we would want a mode where we can do more strict
checks on the prime while not requiring that the prime be an approved prime. We
already allow for strict checking if q is supplied with the private key, but there
were a couple of issues with that check:

    1. there was no way of actually setting q in the current NSS pk11wrap interfaces.
    2. If the prime was a safe prime, but g was an actual generator, then we would fail the y^q mod p = 1 tests for 50% of the keys, even though those keys are safe.
    3. We weren't checking primality of p and q.

So the old code:

  if (q) {
    check y^q mod p = 1
    if not fail
  }

  check 1 <y < p-1 (done in DH_Derive).

New code:

 if (! p is approved prime) {
   if (FIPS) fail;
   if (q) {
      y_test = y
      if (p,q-> p is a safe prime) {
         y_test = 1
      }
      check prime is prime Fail if not
      check subprime is subprime fail if not
      y_test^q mod p = 1
   }
 }
 check 1 < y < p-1 (done in DH_Derive)

This means:

Existing code non-fips without setting the subprime continues to run as before.
Non-fips code which sets the subprime now runs slower, but p and q are checked
  if p or q where not prime, the derive fails (which it should).
In FIPS mode only approved primes will succeed now.
Non-fips code can now set the subprime to q=(p-1)/2 if it doesn't have an
explicit q value (like in tls). If the derive succeeds, we know that p is a
safe prime. If p is approved, the checks are skipped because we already know
that p is a safe prime. Code can optionally do a test derive on a new p and
remember it's safe so that we know longer need to  check ever call (though if
q is not (p-1)/2, you will need to  continue to do the checks each call
because y could still be a small subgroup).

This patch:

gtests/softoken_gtest

  1. Added New dh tests to softoken_gtests. The tests were added to softoken_gtests
because we need to test both non-FIPS and FIPS mode. Test vectors include a
category, so the same test vectors can be used in FIPS and non-FIPS even though
each class may have different results. Most of the test vectors where created
either by dhparams command in openssl, dsaparams in openssl, and the nss makepqg
command. Each vector includes a label, prime, base, optional subprime, optional
public key, test type, and key class (basically size).
  2. If public key is not supplied, we use a generated public key.
  3. If subPrime is supplied to wet it on the private key after generation.

lib/freebl/dh.c

    add primality tests to KEA_VerifyKey().

lib/softokn/

  1. Allow CKA_SUBPRIME to be set after key generation or import. This affects
how we test for it's existance, since it is now always there on the key, we
check it's length to make sure it's non-zero.
  2. We implement the psuedocode above as real code.
  3. We create two new functions: sftl_VerifyDH_Prime which return SECSuccess if Prime is an approved prime. sftk_IsSafePrime which returns SECSuess of both prime and subprime look reasonable, and sets a Bool to PR_TRUE is subprime -> prime is safe (subprime = (prime-1)/2. These functions are implemented in sftkdhverify.c
  4.Cleanup incorrect nominclature on primes (safe primes are not strong primes).

1 files changed, 4 insertions, 1 deletions

diff --git a/lib/freebl/loader.h b/lib/freebl/loader.h
index dcd22daa0..0b5ee5ef0 100644
--- a/lib/freebl/loader.h
+++ b/lib/freebl/loader.h
@@ -10,7 +10,7 @@
 
 #include "blapi.h"
 
-#define FREEBL_VERSION 0x0323
+#define FREEBL_VERSION 0x0324
 
 struct FREEBLVectorStr {
 
@@ -812,6 +812,9 @@ struct FREEBLVectorStr {
 
     /* Version 3.023 came to here */
 
+    PRBool (*p_KEA_PrimeCheck)(SECItem *prime);
+    /* Version 3.024 came to here */
+
     /* Add new function pointers at the end of this struct and bump
      * FREEBL_VERSION at the beginning of this file. */
 };