sources/azure: ensure retries on IMDS request failure (#1271)

There are two issues with IMDS retries:

1. IMDS_VER_WANT will never be attempted if retries=0, such as
when fetching network metadata with infinite=True.

2. get_imds_data_with_api_fallback() will attempt one request with
IMDS_VER_WANT.  If the connection fails due to a timeout, connection
issue, or error code other than 400, an empty dictionary will be
returned without attempting the requested number of retries.

This PR:

- Updates get_imds_data_with_api_fallback() to invoke
get_metadata_from_imds() with the specified retries and infinite
parameters.

- Updates retry_on_url_exc to take a configurable set of HTTP error
codes and exception types to retry on.

- Add IMDS_RETRY_CODES set to retry with when fetching data from IMDS:

  - 404 not found (yet)
  - 410 gone / unavailable (yet)
  - 429 rate-limited/throttled
  - 500 server error

- Replace default callback with imds_readurl_exception_callback,
which configures retry_on_url_exc() with these error codes and
instances.

- Add new pytests for IMDS to eventually replace the unittest
equivalents and improve existing coverage.

Signed-off-by: Chris Patterson <cpatterson@microsoft.com>

1 files changed, 11 insertions, 5 deletions

diff --git a/cloudinit/url_helper.py b/cloudinit/url_helper.py
index 5b2f2ef9..c577e8da 100644
--- a/cloudinit/url_helper.py
+++ b/cloudinit/url_helper.py
@@ -639,16 +639,22 @@ def oauth_headers(
     return signed_headers
 
 
-def retry_on_url_exc(msg, exc):
-    """readurl exception_cb that will retry on NOT_FOUND and Timeout.
+def retry_on_url_exc(
+    msg, exc, *, retry_codes=(NOT_FOUND,), retry_instances=(requests.Timeout,)
+):
+    """Configurable retry exception callback for readurl().
+
+    :param retry_codes: Codes to retry on. Defaults to 404.
+    :param retry_instances: Exception types to retry on. Defaults to
+      requests.Timeout.
 
-    Returns False to raise the exception from readurl, True to retry.
+    :returns: False to raise the exception from readurl(), True to retry.
     """
     if not isinstance(exc, UrlError):
         return False
-    if exc.code == NOT_FOUND:
+    if exc.code in retry_codes:
         return True
-    if exc.cause and isinstance(exc.cause, requests.Timeout):
+    if exc.cause and isinstance(exc.cause, retry_instances):
         return True
     return False