[ruby/openssl] pkey: restore support for decoding "openssl ecparam -genkey" output

Scan through the input for a private key, then fallback to generic
decoder.

OpenSSL 3.0's OSSL_DECODER supports encoded key parameters. The PEM
header "-----BEGIN EC PARAMETERS-----" is used by one of such encoding
formats. While this is useful for OpenSSL::PKey::PKey, an edge case has
been discovered.

The openssl CLI command line "openssl ecparam -genkey" prints two PEM
blocks in a row, one for EC parameters and another for the private key.
Feeding the whole output into OSSL_DECODER results in only the first PEM
block, the key parameters, being decoded. Previously, ruby/openssl did
not support decoding key parameters and it would decode the private key
PEM block instead.

While the new behavior is technically correct, "openssl ecparam -genkey"
is so widely used that ruby/openssl does not want to break existing
applications.

Fixes https://github.com/ruby/openssl/pull/535

https://github.com/ruby/openssl/commit/d486c82833

1 files changed, 36 insertions, 0 deletions

diff --git a/ext/openssl/ossl_pkey.c b/ext/openssl/ossl_pkey.c
index 0effb3e96a..ec39e8bd77 100644
--- a/ext/openssl/ossl_pkey.c
+++ b/ext/openssl/ossl_pkey.c
@@ -104,6 +104,42 @@ ossl_pkey_read_generic(BIO *bio, VALUE pass)
     /* Then check PEM; multiple OSSL_DECODER_from_bio() calls may be needed */
     if (OSSL_DECODER_CTX_set_input_type(dctx, "PEM") != 1)
         goto out;
+    /*
+     * First check for private key formats. This is to keep compatibility with
+     * ruby/openssl < 3.0 which decoded the following as a private key.
+     *
+     *     $ openssl ecparam -name prime256v1 -genkey -outform PEM
+     *     -----BEGIN EC PARAMETERS-----
+     *     BggqhkjOPQMBBw==
+     *     -----END EC PARAMETERS-----
+     *     -----BEGIN EC PRIVATE KEY-----
+     *     MHcCAQEEIAG8ugBbA5MHkqnZ9ujQF93OyUfL9tk8sxqM5Wv5tKg5oAoGCCqGSM49
+     *     AwEHoUQDQgAEVcjhJfkwqh5C7kGuhAf8XaAjVuG5ADwb5ayg/cJijCgs+GcXeedj
+     *     86avKpGH84DXUlB23C/kPt+6fXYlitUmXQ==
+     *     -----END EC PRIVATE KEY-----
+     *
+     * While the first PEM block is a proper encoding of ECParameters, thus
+     * OSSL_DECODER_from_bio() would pick it up, ruby/openssl used to return
+     * the latter instead. Existing applications expect this behavior.
+     *
+     * Note that normally, the input is supposed to contain a single decodable
+     * PEM block only, so this special handling should not create a new problem.
+     */
+    OSSL_DECODER_CTX_set_selection(dctx, EVP_PKEY_KEYPAIR);
+    while (1) {
+        if (OSSL_DECODER_from_bio(dctx, bio) == 1)
+            goto out;
+        if (BIO_eof(bio))
+            break;
+        pos2 = BIO_tell(bio);
+        if (pos2 < 0 || pos2 <= pos)
+            break;
+        ossl_clear_error();
+        pos = pos2;
+    }
+
+    OSSL_BIO_reset(bio);
+    OSSL_DECODER_CTX_set_selection(dctx, 0);
     while (1) {
         if (OSSL_DECODER_from_bio(dctx, bio) == 1)
             goto out;