summaryrefslogtreecommitdiff
path: root/cloudinit/user_data.py
Commit message (Collapse)AuthorAgeFilesLines
* chore: fix style tip (#2071)Alberto Contreras2023-03-141-1/+1
| | | | - remove too broad exceptions - ignore dynamic base types in templater
* cli: schema also validate vendordata*.Chad Smith2023-02-091-1/+0
| | | | | | | | | | | | | | | | cloud-init schema --annotate now walks any valid user-data, vendor-data and vendor2-data to report whether any of these cloud-config files provide invalid or deprecated schema values. Also, fix a bug in nested mapping annotations to properly report the full nested indexed schema path. The following now can be annotated without KeyErrors: users: - default - lock-passwd: false name: root
* Refactor: Drop inheritance from object (#1728)Alberto Contreras2022-09-141-1/+1
| | | As we do not support python2 anymore, we can remove those 8 chars.
* Adopt Black and isort (SC-700) (#1157)James Falcon2021-12-151-54/+67
| | | | | Applied Black and isort, fixed any linting issues, updated tox.ini and CI.
* user_data: remove unused constant (#566)Daniel Watkins2020-09-151-1/+0
| | | | | This was added in d00126c167fc06d913d99cfc184bf3402cb8cf53, but not removed in ef041fd822a2cf3a4022525e942ce988b1f95180 which removed the one usage of it from the original commit.
* user-data: only verify mime-types for TYPE_NEEDED and x-shellscript (#511)Ryan Harper2020-08-141-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit d00126c167fc06d913d99cfc184bf3402cb8cf53 regressed cloud-init handling in multipart MIME user-data. Specifically, cloud-init would examine the payload of the MIME part to determine what the content type and subsequently which handler to use. This meant that user-data which had shellscript payloads (starts with #!) were always handled as shellscripts, rather than their declared MIME type and affected when the payload was handled. One failing scenario was a MIME part with text/cloud-boothook type declared and a shellscript payload. This was run at shellscript processing time rather than boothook time resulting in an change in behavior from previous cloud-init releases. To continue to support known scenarios where clouds have specifed a MIME type of text/x-shellscript but provided a payload of something other than shellscripts, we're changing the lookup logic to check for the TYPES_NEEDED (text/plain, text/x-not-multipart) and only text/x-shellscript. It is safe to check text/x-shellscript parts as all shellscripts must include the #! marker and will be detected as text/x-shellscript types. If the content is missing the #! marker, it will not be excuted. If the content is detected as something cloud-init supports, such as #cloud-config the appropriate cloud-init handler will be used. This change will fix hanldling for parts which were shellscripts but ran with the wrong handler due to ignoring of the provided mime-type. LP: #1888822
* New feature flag functionality and fix includes failing silently (#367)James Falcon2020-06-041-9/+22
| | | | | | | | | | | | | Build time feature flags are now defined in cloudinit/features.py. Feature flags can be added to toggle configuration options or deprecated features. Feature flag overrides can be placed in cloudinit/feature_overrides.py. Further documentation can be found in HACKING.rst. Additionally, updated default behavior to exit with an exception if #include can't retrieve resources as expected. This behavior can be toggled with a feature flag. LP: #1734939
* Fix cloud-init ignoring some misdeclared mimetypes in user-data.Kurt Garloff2020-03-311-1/+3
| | | | | | | | | | | | On some platforms (old heat on OpenTelekomCloud), the user-data mime part is mislabeled x-shellscript. cloud-init would not accept this unexpected mime-type in multipart user-data. Cloud-init will now run find_ctype() on the content of the mime-part to check if it matches known include types. This patch is from Ryan Harper (inspired by my bug report and a suggestion from Scott Moser) from the discussion on PR #234. Signed-off-by: Kurt Garloff <kurt@garloff.de>
* Start removing dependency on six (#178)Daniel Watkins2020-01-211-7/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * url_helper: drop six * url_helper: sort imports * log: drop six * log: sort imports * handlers/__init__: drop six * handlers/__init__: sort imports * user_data: drop six * user_data: sort imports * sources/__init__: drop six * sources/__init__: sort imports * DataSourceOVF: drop six * DataSourceOVF: sort imports * sources/helpers/openstack: drop six * sources/helpers/openstack: sort imports * mergers/m_str: drop six This also allowed simplification of the logic, as we will never encounter a non-string text type. * type_utils: drop six * mergers/m_dict: drop six * mergers/m_list: drop six * cmd/query: drop six * mergers/__init__: drop six * net/cmdline: drop six * reporting/handlers: drop six * reporting/handlers: sort imports
* url_helper: read_file_or_url should pass headers param into readurl (#66)Chad Smith2019-12-021-1/+1
| | | | | | | | | | | | | | | | Headers param was accidentally omitted and no longer passed through to readurl due to a previous commit. To avoid this omission of params in the future, drop positional param definitions from read_file_or_url and pass all kwargs through to readurl when we are not operating on a file. In util:read_seeded, correct the case where invalid positional param file_retries was being passed into read_file_or_url. Also drop duplicated file:// prefix addition from read_seeded because read_file_or_url does that work anyway. LP: #1854084
* Be more safe on string/bytes when writing multipart user-data to disk.Scott Moser2018-06-131-9/+13
| | | | | | | | | | | When creating the multipart mime message that is written as user-data.txt.i, cloud-init losing data on conversion to some things as a string. LP: #1768600 Author: Scott Moser <smoser@ubuntu.com> Co-Authored-By: Chad Smith <chad.smith@canonical.com>
* read_file_or_url: move to url_helper, fix bug in its FileResponse.Scott Moser2018-05-171-3/+3
| | | | | | | | | | | | | The result of a read_file_or_url on a file and on a url would differ in behavior. str(UrlResponse) would return UrlResponse.contents.decode('utf-8') while str(FileResponse) would return str(FileResponse.contents) The difference being "b'foo'" versus "foo". As part of the general goal of cleaning util, move read_file_or_url into url_helper.
* centos: Provide the failed #include url in error messagesChad Smith2017-11-161-1/+6
| | | | | | | | | On python 2.7 and earlier (CentOS 6 & 7), UrlErrors raised by requests do not report the url which failed. In such cases, append the url if not present in the error message. This fixes nightly CI failures at https://jenkins.ubuntu.com/server/view/cloud-init/.
* Catch UrlError when #include'ing URLsAndrew Jorgensen2017-11-131-10/+18
| | | | | | | | | Without this the entire stage can fail, which will leave an instance unaccessible. Reviewed-by: Tom Kirchner <tjk@amazon.com> Reviewed-by: Matt Nierzwicki <nierzwic@amazon.com> Reviewed-by: Ben Cressey <bcressey@amazon.com>
* pylint: fix all logging warningsJoshua Powers2017-04-211-5/+6
| | | | | | | This will change all instances of LOG.warn to LOG.warning as warn is now a deprecated method. It will also make sure any logging uses lazy logging by passing string format arguments as function parameters.
* LICENSE: Allow dual licensing GPL-3 or Apache 2.0Jon Grimm2016-12-221-19/+9
| | | | | | | | | | | | | | | | | | This has been a recurring ask and we had initially just made the change to the cloud-init 2.0 codebase. As the current thinking is we'll just continue to enhance the current codebase, its desirable to relicense to match what we'd intended as part of the 2.0 plan here. - put a brief description of license in LICENSE file - put full license versions in LICENSE-GPLv3 and LICENSE-Apache2.0 - simplify the per-file header to reference LICENSE - tox: ignore H102 (Apache License Header check) Add license header to files that ship. Reformat headers, make sure everything has vi: at end of file. Non-shipping files do not need the copyright header, but at the moment tests/ have it.
* user_data: fix error when user-data is not utf-8 decodableScott Moser2016-06-211-12/+16
| | | | | when user-data was not decodable, cloud-init would raise exception. LP: #1532072
* Fix up a ton of flake8 issuesJoshua Harlow2016-05-121-1/+1
|
* fix cloud-config-archive handlingScott Moser2015-04-161-2/+7
| | | | | | | | | | | | | | handling of cloud-config-archive input would fail in fully_decoded_payload. part.get_charset() would return a Charset object, but get_charset.input_codec is a string suitable for passing to decode. This handles that correctly, and is more careful about binary data inside input. The test added verifies that cloud-config inside a cloud-config-archive is handled correctly and also that binary data there is ignored without exceptions raised. LP: #1445143
* Add util.message_from_string to wrap email.message_from_string.Daniel Watkins2015-03-041-3/+1
| | | | | This is to work-around the fact that email.message_from_string uses cStringIO in Python 2.6, which can't handle Unicode.
* avoid conversion to string in #includeScott Moser2015-02-261-2/+2
|
* UserDataProcessor: during include, do not convert to stringScott Moser2015-02-251-1/+1
| | | | | an include can include a gzip type also or binary data so avoid conversion here.
* use util.decode_binary rather than str, add tests.Scott Moser2015-02-241-2/+2
| | | | | | | | | | just seems to make more sense to decode here. Add a test showing the previous failure (testBytesInPayload) And one that should pass (testStringInPayload) Also, add a test for unencoded content in the ovf xml (test_userdata_plain) And explicitly set encoding on another test (test_userdata_found).
* Fix for Py2 to Py3 difference: cloud-init user-data mime conversion fails onBen Howard2015-02-201-1/+1
| | | | base64 encoded data.
* Remove debugging turd.Barry Warsaw2015-01-271-1/+0
|
* Respond to review:Barry Warsaw2015-01-271-11/+1
| | | | | | - Refactor "fully" decoding the payload of a text/* part. In Python 3, decode=True only means to decode according to Content-Transfer-Encoding, not according to any charset in the Content-Type header. So do that.
* Respond to review:Barry Warsaw2015-01-271-1/+1
| | | | | | | | - Remove str() wrappers to second argument to write_files() where it is no longer necessary. Also: Fixed a couple of other octal literals which clearly weren't being tested.
* Python 3 tests pass, except for skips.Barry Warsaw2015-01-261-1/+3
|
* More test repairs.Barry Warsaw2015-01-261-0/+9
|
* Largely merge lp:~harlowja/cloud-init/py2-3 albeit manually because it seemedBarry Warsaw2015-01-211-3/+5
| | | | | | to be behind trunk. `tox -e py27` passes full test suite. Now to work on replacing mocker.
* pylint and long line fixes.Scott Moser2014-01-241-1/+2
| | | | | | This fixes up many long lines to be < 80 chars and some other pylint issues. pylint 1.1 (in trusty) is now complaining about the lazy logging, so I'll clean that up when I touch things.
* Significant re-working of the userdata handling and introduction ofBen Howard2014-01-081-1/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | vendordata. Vendordata is a datasource provided userdata-like blob that is parsed similiarly to userdata, execept at the user's pleasure. cloudinit/config/cc_scripts_vendor.py: added vendor script cloud config cloudinit/config/cc_vendor_scripts_per_boot.py: added vendor per boot cloud config cloudinit/config/cc_vendor_scripts_per_instance.py: added vendor per instance vendor cloud config cloudinit/config/cc_vendor_scripts_per_once.py: added per once vendor cloud config script doc/examples/cloud-config-vendor-data.txt: documentation of vendor-data examples doc/vendordata.txt: documentation of vendordata for vendors (RENAMED) tests/unittests/test_userdata.py => tests/unittests/test_userdata.py TO: tests/unittests/test_userdata.py => tests/unittests/test_data.py: userdata test cases are not expanded to confirm superiority over vendor data. bin/cloud-init: change instances of 'consume_userdata' to 'consume_data' cloudinit/handlers/cloud_config.py: Added vendor script handling to default cloud-config modules cloudinit/handlers/shell_script.py: Added ability to change the path key to support vendor provided 'vendor-scripts'. Defaults to 'script'. cloudinit/helpers.py: - Changed ConfigMerger to include handling of vendordata. - Changed helpers to include paths for vendordata. cloudinit/sources/__init__.py: Added functions for helping vendordata - get_vendordata_raw(): returns vendordata unprocessed - get_vendordata(): returns vendordata through userdata processor - has_vendordata(): indicator if vendordata is present - consume_vendordata(): datasource directive for indicating explict user approval of vendordata consumption. Defaults to 'false' cloudinit/stages.py: Re-jiggered for handling of vendordata - _initial_subdirs(): added vendor script definition - update(): added self._store_vendordata() - [ADDED] _store_vendordata(): store vendordata - _get_default_handlers(): modified to allow for filtering which handlers will run against vendordata - [ADDED] _do_handlers(): moved logic from consume_userdata to _do_handlers(). This allows _consume_vendordata() and _consume_userdata() to use the same code path. - [RENAMED] consume_userdata() to _consume_userdata() - [ADDED] _consume_vendordata() for handling vendordata - run after userdata to get user cloud-config - uses ConfigMerger to get the configuration from the instance perspective about whether or not to use vendordata - [ADDED] consume_data() to call _consume_{user,vendor}data cloudinit/util.py: - [ADDED] get_nested_option_as_list() used by cc_vendor* for getting a nested value from a dict and returned as a list - runparts(): added 'exe_prefix' for running exe with a prefix, used by cc_vendor* config/cloud.cfg: Added vendor script execution as default tests/unittests/test_runs/test_merge_run.py: changed consume_userdata() to consume_data() tests/unittests/test_runs/test_simple_run.py: changed consume_userdata() to consume_data()
* Just check the filename existing.Joshua Harlow2013-07-201-3/+2
|
* Unify filename, header replacement.Joshua Harlow2013-07-201-24/+32
|
* Keep filename from original part.Joshua Harlow2013-07-201-2/+6
|
* Ensure we reset the part after decompression.Joshua Harlow2013-07-201-9/+27
|
* Add the ability to decompress MIME gzip.Joshua Harlow2013-07-191-8/+33
| | | | | | | | Instead of being restricted to only gzip compressing the overall mime segment or individual included segments, allow for each mime segment to be gzip compressed. LP: #1203203
* Get tests working and further adjustments.harlowja2013-02-231-3/+5
|
* Update to code on trunk.harlowja2013-02-211-1/+1
|\
| * Fixed typo on dict value referenceAvishai Ish-Shalom2012-10-101-1/+1
| | | | | | LP: #1065116
* | Continue working on integrating requests.harlowja2013-02-191-4/+3
|/
* When a parts content type is found to be different thanJoshua Harlow2012-08-311-0/+8
| | | | | | its original content type said it is, make sure we set the new value, also unsure if the old top level message should have the same header (which will flip-flop).
* Don't look into cloud-archive (after processing) for launchJoshua Harlow2012-08-311-2/+2
| | | | | | indexes (since they will be handled beforehand) and fix the types being checked on the root of the archive format to be a tuple instead of a list (which oddly causes complaints).
* Update so that the content types searched for launch-indexJoshua Harlow2012-08-271-2/+3
| | | | | | variable has a little more meaning and by default look in metadata for 'launch-index' and have ec2 instead look for a different variable (thus allowing more datasources to just work).
* Ensure when an archive is exploded and if it contains theJoshua Harlow2012-08-261-1/+3
| | | | | | 'launch-index' key that we copy that key over to the right header (which will then be used later when assigning the 'real' header when the message is attached)
* Fix tests running and add in a check on the content typeJoshua Harlow2012-08-261-10/+23
| | | | | before we look into the payload as well as make the skip test a function that the datasource module can also use.
* Add the capability to understand and filter onJoshua Harlow2012-08-261-15/+35
| | | | | | | | | | | | | | | | | | | | | | | userdata based on a launch-index (or leave userdata alone if none is provided by the datasource). This works by doing the following. 1. Adjusting the userdata processor to attempt to inject a "Launch-Index" header into the messages headers (by either taking a header that already exists or by looking into the payload to see if it exists there). 2. Adjust the get_userdata ds function to apply a filter on the returned userdata (defaulting to false) that will now use the datasources get_launch_index value to restrict the 'final' message used in consuming user data (the same behavior if not existent). 3. Further down the line processes that use the 'resultant' userdata now will only see the ones for there own launch index (ie cloud-config will be restricted automatically and so on) and are unaffected (although they can now ask the cloud object or the datasource for its launch index via the above new ds method.
* fix pep8 complaints.Scott Moser2012-08-221-2/+2
| | | | make pep8 now is silent on precise's pep8 ( 0.6.1-2ubuntu2).
* Fixes 1012854 by implementing file writing, adjustsJoshua Harlow2012-07-101-1/+1
| | | | | | | other code to have user/group parsing in util instead of in stages.py, renames decomp_str to decomp_gzip since it is more meaningful when named that (as thats all it can decompress).
* 1. Adjust the logging of phone home module when its not enabledharlowja2012-06-221-1/+1
| | | | 2. Fix pylint warning in userdata about unused variable
View Lorry Controller Status