summaryrefslogtreecommitdiff
path: root/cloudinit/url_helper.py
Commit message (Collapse)AuthorAgeFilesLines
* lint: fix tip-flake8 and tip-mypy (#1896)Alberto Contreras2022-12-071-4/+4
|
* Refactor: Drop inheritance from object (#1728)Alberto Contreras2022-09-141-3/+3
| | | As we do not support python2 anymore, we can remove those 8 chars.
* typing: Type UrlResponse.contents (#1633)Alberto Contreras2022-08-081-1/+3
| | | | | | | | | | | typing: Type UrlResponse.contents As requests.Response.contents returns always an instance of bytes, type UrlResponse.contents accordingly. This change undercovered an incorrect typing in sources/helpers/azure, that was hidden because ElementTree.fromstring accepts str and bytes.
* main: avoid downloading full contents cmdline urls (#1606)Alberto Contreras2022-08-041-3/+21
| | | | | | | | | | | | In the case that the url content does not start with `#cloud-config`, avoid downloading the full content. Add deprecated logs to prefer `cloud-config-url` over `url` on the kernel command line. Restructure and link kernel command line docs to instance-data docs. Add some typing. LP: #1937319
* Drop mypy excluded files (#1454)Alberto Contreras2022-05-231-6/+4
| | | | | | | | | - Add types to let mypy pass. - Add mypy flags: - detect unused ignores - redundant casts - Drop support of `ConfigParser` in Python 2 - Harden DataSourceLXD.network_config - Convert old-style commented types to proper types.
* Fix bug in url_helper/dual_stack() logging (#1426)Brett Holman2022-05-111-3/+4
| | | | All exceptions thrown from futures in dual_stack() should be logged for debugging purposes.
* Add support for dual stack IPv6/IPv4 IMDS to Ec2 (#1160)Brett Holman2022-04-141-74/+240
| | | | | | | | | - add support for parallel http(s) requests to wait_for_url() - implementation based loosely on RFC6555: "Happy Eyeballs" [1] - update ec2 datasource to support dual-stack ipv6/ipv4 - only "nitro" instances have ipv6 IMDS, favor ipv4 initially - replace httpretty with responses for ec2 tests [1] https://datatracker.ietf.org/doc/html/rfc6555
* url_helper: add tuple support for readurl timeout (#1328)Chris Patterson2022-03-141-2/+6
| | | | | | | It may be useful to configure connection timeout and read timeout separately. Update readurl() to accept a tuple that is supported by python requests to configure both. Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
* sources/azure: ensure retries on IMDS request failure (#1271)Chris Patterson2022-02-171-5/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two issues with IMDS retries: 1. IMDS_VER_WANT will never be attempted if retries=0, such as when fetching network metadata with infinite=True. 2. get_imds_data_with_api_fallback() will attempt one request with IMDS_VER_WANT. If the connection fails due to a timeout, connection issue, or error code other than 400, an empty dictionary will be returned without attempting the requested number of retries. This PR: - Updates get_imds_data_with_api_fallback() to invoke get_metadata_from_imds() with the specified retries and infinite parameters. - Updates retry_on_url_exc to take a configurable set of HTTP error codes and exception types to retry on. - Add IMDS_RETRY_CODES set to retry with when fetching data from IMDS: - 404 not found (yet) - 410 gone / unavailable (yet) - 429 rate-limited/throttled - 500 server error - Replace default callback with imds_readurl_exception_callback, which configures retry_on_url_exc() with these error codes and instances. - Add new pytests for IMDS to eventually replace the unittest equivalents and improve existing coverage. Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
* sources/azure: address mypy/pyright typing complaints (#1245)Chris Patterson2022-02-111-4/+3
| | | | | | | Raise runtime errors for unhandled cases which would cause other exceptions. Ignore types for a few cases where a non-trivial refactor would be required to prevent the warning. Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
* Remove distutils usage (#1177)Shreenidhi Shedi2022-01-121-49/+11
| | | | | | | | | | distutils is getting deprecated soon. Let's replace it with suggested alternatives as suggested in: https://www.python.org/dev/peps/pep-0632/ Remove `requests` version check and related code from url_helper.py as the versions specified are old enough to no longer be relevant. Signed-off-by: Shreenidhi Shedi <sshedi@vmware.com>
* Adopt Black and isort (SC-700) (#1157)James Falcon2021-12-151-94/+179
| | | | | Applied Black and isort, fixed any linting issues, updated tox.ini and CI.
* tox: bump the pylint version to 2.6.0 in the default run (#544)Paride Legovini2020-08-251-3/+3
| | | | | | Changes: tox: bump the pylint version to 2.6.0 in the default run Fix pylint 2.6.0 W0707 warnings (raise-missing-from)
* ec2: only redact token request headers in logs, avoid altering request (#230)Chad Smith2020-03-031-7/+8
| | | | | | | | | | | Our header redact logic was redacting both logged request headers and the actual source request. This results in DataSourceEc2 sending the invalid header "X-aws-ec2-metadata-token-ttl-seconds: REDACTED" which gets an HTTP status response of 400. Cloud-init retries this failed token request for 2 minutes before falling back to IMDSv1. LP: #1865882
* ec2: Do not log IMDSv2 token values, instead use REDACTED (#219)Ryan Harper2020-02-191-7/+20
| | | | | | | | | Instead of logging the token values used log the headers and replace the actual values with the string 'REDACTED'. This allows users to examine cloud-init.log and see that the IMDSv2 token header is being used but avoids leaving the value used in the log file itself. LP: #1863943
* Start removing dependency on six (#178)Daniel Watkins2020-01-211-15/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * url_helper: drop six * url_helper: sort imports * log: drop six * log: sort imports * handlers/__init__: drop six * handlers/__init__: sort imports * user_data: drop six * user_data: sort imports * sources/__init__: drop six * sources/__init__: sort imports * DataSourceOVF: drop six * DataSourceOVF: sort imports * sources/helpers/openstack: drop six * sources/helpers/openstack: sort imports * mergers/m_str: drop six This also allowed simplification of the logic, as we will never encounter a non-string text type. * type_utils: drop six * mergers/m_dict: drop six * mergers/m_list: drop six * cmd/query: drop six * mergers/__init__: drop six * net/cmdline: drop six * reporting/handlers: drop six * reporting/handlers: sort imports
* url_helper: read_file_or_url should pass headers param into readurl (#66)Chad Smith2019-12-021-8/+39
| | | | | | | | | | | | | | | | Headers param was accidentally omitted and no longer passed through to readurl due to a previous commit. To avoid this omission of params in the future, drop positional param definitions from read_file_or_url and pass all kwargs through to readurl when we are not operating on a file. In util:read_seeded, correct the case where invalid positional param file_retries was being passed into read_file_or_url. Also drop duplicated file:// prefix addition from read_seeded because read_file_or_url does that work anyway. LP: #1854084
* ec2: Add support for AWS IMDS v2 (session-oriented) (#55)Ryan Harper2019-11-221-6/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * ec2: Add support for AWS IMDS v2 (session-oriented) AWS now supports a new version of fetching Instance Metadata[1]. Update cloud-init's ec2 utility functions and update ec2 derived datasources accordingly. For DataSourceEc2 (versus ec2-look-alikes) cloud-init will issue the PUT request to obtain an API token for the maximum lifetime and then all subsequent interactions with the IMDS will include the token in the header. If the API token endpoint is unreachable on Ec2 platform, log a warning and fallback to using IMDS v1 and which does not use session tokens when communicating with the Instance metadata service. We handle read errors, typically seen if the IMDS is beyond one etwork hop (IMDSv2 responses have a ttl=1), by setting the api token to a disabled value and then using IMDSv1 paths. To support token-based headers, ec2_utils functions were updated to support custom headers_cb and exception_cb callback functions so Ec2 could store, or refresh API tokens in the event of token becoming stale. [1] https://docs.aws.amazon.com/AWSEC2/latest/ \ UserGuide/ec2-instance-metadata.html \ #instance-metadata-v2-how-it-works
* Small typo fixes in code comments.Dominic Schlegel2019-10-161-3/+3
|
* Add support for publishing host keys to GCE guest attributesRick Wright2019-08-091-4/+5
| | | | | | | | | | | | | | | | | | | | | This adds an empty publish_host_keys() method to the default datasource that is called by cc_ssh.py. This feature can be controlled by the 'ssh_publish_hostkeys' config option. It is enabled by default but can be disabled by setting 'enabled' to false. Also, a blacklist of key types is supported. In addition, this change implements ssh_publish_hostkeys() for the GCE datasource, attempting to write the hostkeys to the instance's guest attributes. Using these hostkeys for ssh connections is currently supported by the alpha version of Google's 'gcloud' command-line tool. (On Google Compute Engine, this feature will be enabled by setting the 'enable-guest-attributes' metadata key to 'true' for the project/instance that you would like to use this feature for. When connecting to the instance for the first time using 'gcloud compute ssh' the hostkeys will be read from the guest attributes for the instance and written to the user's local known_hosts file for Google Compute Engine instances.)
* Resolve flake8 comparison and pycodestyle over-ident issuesParide Legovini2019-01-301-1/+1
| | | | | | Fixes: - flake8: use ==/!= to compare str, bytes, and int literals - pycodestyle: E117 over-indented
* azure: retry imds polling on requests.TimeoutChad Smith2018-11-131-0/+14
| | | | | | | | | There is an infrequent race when the booting instance can hit the IMDS service before it is fully available. This results in a requests.ConnectTimeout being raised. Azure's retry_callback logic now retries on either 404s or Timeouts. LP:1800223
* azure: report ready to fabric after reprovision and reduce loggingasakkurr2018-10-311-6/+11
| | | | | | | | | When reusing a preprovisioned VM, report ready to Azure fabric as soon as we get the reprovision data and the goal state so that we are not delayed by the cloud-init stage switch, saving 2-3 seconds. Also reduce logging when polling IMDS for reprovision data. LP: #1799594
* read_file_or_url: move to url_helper, fix bug in its FileResponse.Scott Moser2018-05-171-1/+28
| | | | | | | | | | | | | The result of a read_file_or_url on a file and on a url would differ in behavior. str(UrlResponse) would return UrlResponse.contents.decode('utf-8') while str(FileResponse) would return str(FileResponse.contents) The difference being "b'foo'" versus "foo". As part of the general goal of cleaning util, move read_file_or_url into url_helper.
* pylint: pay attention to unused variable warnings.Scott Moser2018-04-191-1/+1
| | | | | This enables warnings produced by pylint for unused variables (W0612), and fixes the existing errors.
* Reduce AzurePreprovisioning HTTP timeouts.Douglas Jordan2018-03-231-5/+8
| | | | | | | | | | | Reducing timeout to 1 second as IMDS responds within a handful of milliseconds. Also get rid of max_retries to prevent exiting out of polling loop early due to IMDS outage / upgrade. Reduce Azure PreProvisioning HTTP timeouts during polling to avoid waiting an extra minute. LP: #1752977
* Revert the logic of exception_cb in read_url.Kurt Garloff2018-03-231-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In commit e9e8616, there was an inversion of the logic of the exception_cb return value meaning, breaking the (network) OpenStack DataSource, which implemented exception_cb as should_retry_cb, returning True when a retry should be done and False when the retry loop should be broken and the exception reraised again immediately. The OpenStack DS was the only user of this callback at the time and not touched by the commit (nor did the commit message mention an intended change), so this almost certainly happened by mistake. These days, we have a second user of the callback in DataSourceScaleway. It uses the new logic, so it needs change if we fix the meaning of the return value. This patch reverts the meaning of url_helper.read_url() execption_cb to the old semantics. It updates the comment and adjusts the Scaleway datasource. The patch has been tested on Open Telekom Cloud (which uses the OpenStack network Datasource) where previously a missing user_data and network_data.json would be retried 6 times each despite them not being present (they are optional!) and the server repsonding with a correct 404. After the patch, boot times are 10s faster, as we no longer pointlessly retry these files. LP: #1702160 LP: #1298921
* Simplify some comparisions.Rémy Léone2018-03-021-2/+2
| | | | | | | | Just replace a couple things like: if b > a and b < c: with: if a < b < c:
* Azure VM Preprovisioning support.Douglas Jordan2018-01-241-9/+20
| | | | | | | | | | | | | | This change will enable azure vms to report provisioning has completed twice, first to tell the fabric it has completed then a second time to enable customer settings. The datasource for the second provisioning is the Instance Metadata Service (IMDS),and the VM will poll indefinitely for the new ovf-env.xml from IMDS. This branch introduces EphemeralDHCPv4 which encapsulates common logic used by both DataSourceEc2 an DataSourceAzure for temporary DHCP interactions without side-effects. LP: #1734991
* url_helper: dynamically import oauthlib import from inside oauth_headersChad Smith2017-08-301-7/+3
| | | | | | | | | | oauth_headers is the only function which requires oauthlib, move the import and ImportError handling inside this function to only attempt loading at runtime if called. This will allow us to build on platforms that don't have python-oauthlib installed by default. Add simple unittests around the missing oauthlib dependencies to make sure the function performs as intended and raises and NotImplementedError if oauthlib can't be imported.
* url_helper: fail gracefully if oauthlib is not availableLars Kellogg-Stedman2017-08-291-1/+9
| | | | | | | | | We are unable to ship python-oauthlib in RHEL. This commit allows imports of url_helper to succeed even when oauthlib is unavailable and OauthUrlHelper.oauth_headers to raise a NotImplementedException when called. LP: #1713760
* Scaleway: add datasource with user and vendor data for Scaleway.Julien Castets2017-07-171-2/+8
| | | | | | | | | | | | Here we add and enable by default a datasource for Scaleway cloud. The datasource quickly exits unless one of three things: a.) 'Scaleway' found as the system vendor b.) 'scaleway' found on the kernel command line. c.) the directory /var/run/scaleway exists (this is currently created by the scaleway initramfs module). One interesting bit of this particular datasource is that it requires the source port of the http request to be < 1024.
* pylint: fix all logging warningsJoshua Powers2017-04-211-5/+6
| | | | | | | This will change all instances of LOG.warn to LOG.warning as warn is now a deprecated method. It will also make sure any logging uses lazy logging by passing string format arguments as function parameters.
* test: add running of pylintJoshua Powers2017-03-241-1/+1
| | | | | Now tox will run pylint. The .pylintrc file sets pylint to only produce errors, and will ignore certain classes that are known problematic (six).
* LICENSE: Allow dual licensing GPL-3 or Apache 2.0Jon Grimm2016-12-221-19/+9
| | | | | | | | | | | | | | | | | | This has been a recurring ask and we had initially just made the change to the cloud-init 2.0 codebase. As the current thinking is we'll just continue to enhance the current codebase, its desirable to relicense to match what we'd intended as part of the 2.0 plan here. - put a brief description of license in LICENSE file - put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0 - simplify the per-file header to reference LICENSE - tox: ignore H102 (Apache License Header check) Add license header to files that ship. Reformat headers, make sure everything has vi: at end of file. Non-shipping files do not need the copyright header, but at the moment tests/ have it.
* Fix up a ton of flake8 issuesJoshua Harlow2016-05-121-2/+3
|
* Update pep8 runner and fix pep8 issuesRyan Harper2016-03-031-3/+3
|
* MAAS: fix issues with url_helper and oauth moduleScott Moser2015-09-291-2/+6
| | | | | | | This would cause problems in the event that we actually had a bad clock. We add a retry in the main (for test) also, to ensure that the oauth timestamp fix gets in place. LP: #1499869
* readurl: if headers are provided still provide base headersScott Moser2015-08-311-4/+8
| | | | | | we want cloud-init user agent to be present even if the user provided some headers. In the event that they provided User-Agent, this will respect their wishes.
* MAAS: fixes to data source and OauthUrlHelperScott Moser2015-08-251-7/+7
| | | | | | | | | | | the previous version was broken. The vital fixes here are: * adding parsedate and oauth1 imports to url_helper * fix skew_data usage intending to use self.skew_data Additionally: * reorder imports in url_helper * fixes to python3 -m cloudinit.sources.DataSourceMaas LP: #1488507
* fix syntaxScott Moser2015-08-071-1/+1
|
* improvements on skewScott Moser2015-08-071-9/+9
|
* seems functional in testScott Moser2015-08-071-6/+4
|
* tests passScott Moser2015-08-061-5/+137
|
* url_helper.py: fix undefined variableScott Moser2015-03-021-6/+8
| | | | | | | | | | | | python2 scoping is different and running wait_for_url in python3 results in a use of undeclared variable 'e'. $ python3 -c 'from cloudinit import url_helper; \ url_helper.wait_for_url("o", max_wait=3,timeout=1, exception_cb=print)' Traceback (most recent call last): File "<string>", line 1, in <module> File "cloudinit/url_helper.py", line 358, in wait_for_url exception_cb(msg=status_msg, exception=e)
* move towards user-data being binaryScott Moser2015-02-251-1/+1
| | | | | | | | | | | | | | | | UrlResponse: biggest change... make readurl return bytes, making user know what to do with it. util: add load_tfile_or_url for loading text file or url as read_file_or_url now returns bytes ec2_utils: all meta-data is text, remove non-obvious string translations DigitalOcean: adjust for ec2_utils DataSourceGCE, DataSourceMAAS: user-data is binary other fields are text. openstack.py: read paths without decoding to text. This is ok as paths other than user-data are json, and load_json will handle load_file still returns text, and that is what most things use.
* Largely merge lp:~harlowja/cloud-init/py2-3 albeit manually because it seemedBarry Warsaw2015-01-211-7/+15
| | | | | | to be behind trunk. `tox -e py27` passes full test suite. Now to work on replacing mocker.
* further remove evidence of pylint.Scott Moser2014-08-261-7/+6
| | | | | This just removes comments '# pylint:' things and other code remnents of pylint.
* fix(pep8): Fix various pep8 violations and version-lock pep8Jay Faulkner2014-08-261-2/+2
| | | | | | Fixed all complaints from running "make pep8". Also version locked pep8 in test-requirements.txt to ensure that pep8 requirements don't change without an explicit commit.
* wait less for the metadata service (by default)Scott Moser2014-02-131-0/+1
| | | | | | | | | | Waiting around for a metadata service in a given datasource means that if its not there all the subsequent datasources have to wait, and boot is slowed down. As it is right now, EC2 is the only one that has the right to wait. In the past, we had to wait around for the EC2 metadata service. I really do not want to extend that courtesy to other cloud platforms. A network based metadata service should be up as soon as networking is up.
View Lorry Controller Status