Bug 1767883 - Need to add policy control to keys lengths for signatures. r=nss-reviewers

There are three changes in the patch which are related to key length processing:

Change RSA_MIN_MODULUS_BITS in blalpit.h from 128 to 1023. This necessitated changes to the following tests: testcrmf.c: up the generated key for the test from 512 to 1024. pk11_rsapkcs1_unittest.cc (in pk11_gtest): skip the min padding test if the MIN_RSA_MODULUS_BITS is more than 736 (The largest hash we support is 512, which fits in an RSA key less then 736. If we can't generate a key less than 736, we can't test minimum padding, but we can never get into that situation anyway now). tls_subcerts_unittest.cc: set our key size to at least RSA_MIN_MODULUS_BITS, and then make sure the policy had a higher minimum key length so we still trigger the 'weakKey' event. pk11kea.c: use 1024 bits for the transfer key now that smaller keysizes aren't supported by softoken.

Expand the add a new flag to meaning of NSS_XXX_MIN_KEY_SIZE beyond it's use in SSL (add the ability to limit signing and verification to this as well). This allows us to set strict FIPS 140-3 policies, where we can only sign with 2048, but can still verify 1024. This part includes: New utility functions in seckey.c: SECKEY_PrivateKeyStrengthInBits(): The private key equivalent to SECKEY_PublicKeyStrengthInBits(). This function could be exported globally, but isn't in this patch. seckey_EnforceKeySize(). Takes a key type and a length and makes sure that length falls into the range set by policy. secsign.c and secvfy.c: add policy length check where we check the other policy flags. nss.h, nssoptions.c: add NSS_KEY_SIZE_POLICY_FLAGS and define flags for SSL, VERIFY, and SIGN. SSL is set by default (to maintain the current behavior). pk11parse.c: add keywords for the new NSS_KEY_SIZE_POLICY_FLAGS. ssl3con.c: use the flags to decide if the policy lengths are active for SSL. policy.txt: Test that the new policy flags are parsed correctly sslpolicy.txt: Add tests to make sure the policy flags are functioning.

Update fips_algorithms.h to make sure the FIPS indicators are exactly compliant with FIPS 140-3 current guidance (RSA 2028 and above, any key size, Legacy verification allowed for 1024, 1280, 1536, and 1792 [1024-1792, step 256]).

The previous attempt to push failed because the pk11_rsapkcs1_unittest.cc
change was eaten in the merge.

Differential Revision: https://phabricator.services.mozilla.com/D146341

1 files changed, 2 insertions, 0 deletions

diff --git a/lib/nss/nssoptions.h b/lib/nss/nssoptions.h
index 024c1e92f..3f19c61ed 100644
--- a/lib/nss/nssoptions.h
+++ b/lib/nss/nssoptions.h
@@ -18,3 +18,5 @@
  * happens because NSS used to count bit lengths incorrectly. */
 #define SSL_DH_MIN_P_BITS 1023
 #define SSL_DSA_MIN_P_BITS 1023
+/* not really used by SSL, but define it here for consistency */
+#define SSL_ECC_MIN_CURVE_BITS 255