pubkey: avoid spurious audit messages from _gnutls_pubkey_compatible_with_sig()

When checking in _gnutls_pubkey_compatible_with_sig() whether a public
key is compatible with a signature algorithm, run first
pubkey_supports_sig() before performing weaker checks that can accept
the given algorithm but with an audit-log warning. This avoids an issue
when a weaker check would log an audit message for some signature
algorithm that would then be determined as incompatible by the
pubkey_supports_sig() check anyway.

For instance, a GnuTLS server might have a certificate with a SECP384R1
public key and a client can report that it supports
ECDSA-SECP256R1-SHA256 and ECDSA-SECP384R1-SHA384. In such a case, the
GnuTLS server will eventually find that it must use
ECDSA-SECP384R1-SHA384 with this public key. However, the code would
first run _gnutls_pubkey_compatible_with_sig() to check if SECP384R1 is
compatible with ECDSA-SECP256R1-SHA256. The function would report the
audit warning "The hash size used in signature (32) is less than the
expected (48)" but then reject the signature algorithm in
pubkey_supports_sig() as incompatible because it has a different curve.
Since the algorithm gets rejected it is not necessary to inform about
its hash size difference in the audit log.

Signed-off-by: Petr Pavlu <petr.pavlu@suse.com>

1 files changed, 7 insertions, 4 deletions

diff --git a/lib/pubkey.c b/lib/pubkey.c
index de95a04c37..6f9d54f119 100644
--- a/lib/pubkey.c
+++ b/lib/pubkey.c
@@ -2092,10 +2092,16 @@ int _gnutls_pubkey_compatible_with_sig(gnutls_session_t session,
 	unsigned int sig_hash_size;
 	const mac_entry_st *me;
 	const gnutls_sign_entry_st *se;
+	int ret;
 
 	se = _gnutls_sign_to_entry(sign);
-	if (se == NULL && _gnutls_version_has_selectable_sighash(ver))
+	if (se != NULL) {
+		ret = pubkey_supports_sig(pubkey, se);
+		if (ret < 0)
+			return gnutls_assert_val(ret);
+	} else if (_gnutls_version_has_selectable_sighash(ver)) {
 		return gnutls_assert_val(GNUTLS_E_INVALID_REQUEST);
+	}
 
 	if (pubkey->params.algo == GNUTLS_PK_DSA) {
 		me = _gnutls_dsa_q_to_hash(&pubkey->params, &hash_size);
@@ -2158,9 +2164,6 @@ int _gnutls_pubkey_compatible_with_sig(gnutls_session_t session,
 		}
 	}
 
-	if (se != NULL)
-		return pubkey_supports_sig(pubkey, se);
-
 	return 0;
 }