HTTP: Implement persistent connections (keep-alive)

Both at API and at CLI level (for the CMP app only, so far)
there is a new parameter/option: keep_alive.
* 0 means HTTP connections are not kept open after
receiving a response, which is the default behavior for HTTP 1.0.
* 1 means that persistent connections are requested.
* 2 means that persistent connections are required, i.e.,
in case the server does not grant them an error occurs.

For the CMP app the default value is 1, which means preferring to keep
the connection open. For all other internal uses of the HTTP client
(fetching an OCSP response, a cert, or a CRL) it does not matter
because these operations just take one round trip.

If the client application requested or required a persistent connection
and this was granted by the server, it can keep the OSSL_HTTP_REQ_CTX *
as long as it wants to send further requests and OSSL_HTTP_is_alive()
returns nonzero,
else it should call OSSL_HTTP_REQ_CTX_free() or OSSL_HTTP_close().
In case the client application keeps the OSSL_HTTP_REQ_CTX *
but the connection then dies for any reason at the server side, it will
notice this obtaining an I/O error when trying to send the next request.

This requires extending the HTTP header parsing and
rearranging the high-level HTTP client API. In particular:
* Split the monolithic OSSL_HTTP_transfer() into OSSL_HTTP_open(),
  OSSL_HTTP_set_request(), a lean OSSL_HTTP_transfer(), and OSSL_HTTP_close().
* Split the timeout functionality accordingly and improve default behavior.
* Extract part of OSSL_HTTP_REQ_CTX_new() to OSSL_HTTP_REQ_CTX_set_expected().
* Extend struct ossl_http_req_ctx_st accordingly.

Use the new feature for the CMP client, which requires extending
related transaction management of CMP client and test server.

Update the documentation and extend the tests accordingly.

Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/15053)

1 files changed, 6 insertions, 5 deletions

diff --git a/include/openssl/http.h b/include/openssl/http.h
index 2140d5d2f8..e3bd9d7579 100644
--- a/include/openssl/http.h
+++ b/include/openssl/http.h
@@ -65,22 +65,23 @@ OSSL_HTTP_REQ_CTX *OSSL_HTTP_open(const char *server, const char *port,
                                   const char *proxy, const char *no_proxy,
                                   int use_ssl, BIO *bio, BIO *rbio,
                                   OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
-                                  int buf_size, int overall_timeout);
+                                  int buf_size, unsigned long max_resp_len,
+                                  int overall_timeout);
 int OSSL_HTTP_proxy_connect(BIO *bio, const char *server, const char *port,
                             const char *proxyuser, const char *proxypass,
                             int timeout, BIO *bio_err, const char *prog);
 int OSSL_HTTP_set_request(OSSL_HTTP_REQ_CTX *rctx, const char *path,
                           const STACK_OF(CONF_VALUE) *headers,
-                          const char *content_type, BIO *req,
+                          const char *content_type, BIO *req_mem,
                           const char *expected_content_type, int expect_asn1,
-                          size_t max_resp_len, int timeout, int keep_alive);
+                          int timeout, int keep_alive);
 BIO *OSSL_HTTP_exchange(OSSL_HTTP_REQ_CTX *rctx, char **redirection_url);
 BIO *OSSL_HTTP_get(const char *url, const char *proxy, const char *no_proxy,
                    BIO *bio, BIO *rbio,
                    OSSL_HTTP_bio_cb_t bio_update_fn, void *arg,
                    int buf_size, const STACK_OF(CONF_VALUE) *headers,
                    const char *expected_content_type, int expect_asn1,
-                   size_t max_resp_len, int timeout);
+                   unsigned long max_resp_len, int timeout);
 BIO *OSSL_HTTP_transfer(OSSL_HTTP_REQ_CTX **prctx,
                         const char *server, const char *port,
                         const char *path, int use_ssl,
@@ -90,7 +91,7 @@ BIO *OSSL_HTTP_transfer(OSSL_HTTP_REQ_CTX **prctx,
                         int buf_size, const STACK_OF(CONF_VALUE) *headers,
                         const char *content_type, BIO *req,
                         const char *expected_content_type, int expect_asn1,
-                        size_t max_resp_len, int timeout, int keep_alive);
+                        unsigned long max_resp_len, int timeout, int keep_alive);
 int OSSL_HTTP_close(OSSL_HTTP_REQ_CTX *rctx, int ok);
 
 /* Auxiliary functions */