punycode: update to use WPACKET instead of using custom range checking

Add test for `.' overflows, remove the output size argument from
ossl_a2ulabel() since it was never used and greatly complicated the code.
Convert ossl_a2ulabel() to use WPACKET for building the output string.
Update the documentation to match the new definition of ossl_a2ulabel().

x509: let punycode handle the '\0' string termination.  Saves a memset(3)
and some size fiddling.  Also update to deal with the modified parameters.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
(Merged from https://github.com/openssl/openssl/pull/19591)

1 files changed, 5 insertions, 6 deletions

diff --git a/doc/internal/man3/ossl_punycode_decode.pod b/doc/internal/man3/ossl_punycode_decode.pod
index bf6b56edfc..8c9484889b 100644
--- a/doc/internal/man3/ossl_punycode_decode.pod
+++ b/doc/internal/man3/ossl_punycode_decode.pod
@@ -12,7 +12,7 @@ ossl_punycode_decode, ossl_a2ulabel, ossl_a2ucompare
   int ossl_punycode_decode(const char *pEncoded, const size_t enc_len,
                            unsigned int *pDecoded, unsigned int *pout_length);
 
-  int ossl_a2ulabel(const char *in, char *out, size_t *outlen);
+  int ossl_a2ulabel(const char *in, char *out, size_t outlen);
 
   int ossl_a2ucompare(const char *a, const char *u);
 
@@ -23,7 +23,7 @@ representation of hostnames in ASCII-only format. Some specifications,
 such as RFC 8398, require comparison of hostnames encoded in UTF-8 charset.
 
 ossl_a2ulabel() decodes NUL-terminated hostname from PUNYCODE to UTF-8,
-using a provided buffer for output.
+using a provided buffer for output.  The output buffer is NUL-terminated.
 
 ossl_a2ucompare() accepts two NUL-terminated hostnames, decodes the 1st
 from PUNYCODE to UTF-8 and compares it with the 2nd one as is.
@@ -33,12 +33,11 @@ a hostname, with stripped PUNYCODE marker I<xn-->.
 
 =head1 RETURN VALUES
 
-ossl_a2ulabel() returns 1 on success, 0 on not enough buf passed,
--1 on invalid PUNYCODE string passed. When valid string is provided, it sets the
-I<*outlen> to the length of required buffer to perform correct decoding.
+ossl_a2ulabel() returns 1 on success, 0 if the output buffer is too small and
+-1 if an invalid PUNYCODE string is passed or another error occurs.
 
 ossl_a2ucompare() returns 1 on non-equal strings, 0 on equal strings,
--1 when invalid PUNYCODE string passed.
+-1 when an invalid PUNYCODE string is passed or another error occurs.
 
 ossl_punycode_decode() returns 1 on success, 0 on error. On success,
 *pout_length contains the number of codepoints decoded.