EVP: Check that key methods aren't foreign when exporting

The EVP_PKEY_ASN1_METHOD function export_to() must check that the key
we're trying to export has a known libcrypto method, i.e. is a built
in RSA_METHOD, DSA_METHOD, etc.  Otherwise, the method may be defined
by the calling application, by an engine, by another library, and we
simply cannot know all the quirks hidden behind that method, if we
have access to the key data, or much anything.

Such keys are simply deemed impossible to export to provider keys,
i.e. have export_to() return 0.  This cascades back to functions like
evp_pkey_export_to_provider() and evp_pkey_upgrade_to_provider() and
their callers.  In most cases, this is fine, but if these get mixed in
with provider side keys in any function, that function will fail.

Fixes #11179
Fixes #9915

Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
(Merged from https://github.com/openssl/openssl/pull/11193)

1 files changed, 7 insertions, 0 deletions

diff --git a/crypto/rsa/rsa_ameth.c b/crypto/rsa/rsa_ameth.c
index f34eacf552..3411b734e5 100644
--- a/crypto/rsa/rsa_ameth.c
+++ b/crypto/rsa/rsa_ameth.c
@@ -1092,6 +1092,13 @@ static int rsa_pkey_export_to(const EVP_PKEY *from, void *to_keydata,
     OSSL_PARAM *params = NULL;
     int rv = 0;
 
+    /*
+     * If the RSA method is foreign, then we can't be sure of anything, and
+     * can therefore not export or pretend to export.
+     */
+    if (RSA_get_method(rsa) != RSA_PKCS1_OpenSSL())
+        return 0;
+
     /* Public parameters must always be present */
     if (n == NULL || e == NULL)
         goto err;