| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix cloud-init schema --system being unable to find merged
userdata stored at /var/lib/cloud/instance/cloud_config.txt.
Init.paths.get_ipath only has visibility to merged cloud config in
/var/lib/cloud/<instance_id>/cloud-config.txt after fetching the
existing cached datasource which provides instance-id from metadata
in order to determine the unique instance-id which represents the
path to the cloud-config.txt.
To support reuse of read_cfg_paths helper function, add an optional
parameter fetch_existing_datasource which indicates whether reading
the existing datasource is necessary for this helper function.
cloud-init schema --system calls read_cfg_paths providing
fetch_existing_datasource="trust" prior to calls to
paths.get_ipath().
|
|
|
|
|
| |
* Fix logger, use instance rather than module function
* add dunder names
|
|
|
|
| |
Use the module level Log instances instead of passing log instances
to the cc_* handlers
|
|
|
|
| |
- remove too broad exceptions
- ignore dynamic base types in templater
|
|
|
|
|
|
|
|
|
| |
At inception[1], dhclient hooks were used to filter environment variables into /run/cloud-init/dhclient.hooks/<interface>.json which was consumed by WALinuxAgentShim. The fallback method was to parse the dhcp client lease file.
Today the Azure datasource directly uses the parsed lease file[2], and loading /run/cloud-init/dhclient.hook/<interface>.json file was removed in 22.2[3]. With no other consumers, we can remove this.
[1] https://github.com/canonical/cloud-init/commit/648dbbf6b090c81e989f1ab70bf99f4de16a6a70
[2] https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/DataSourceAzure.py#L447
[3] https://github.com/canonical/cloud-init/commit/5ad0768a796bc07232476d0d29b5225f1e6e131c
|
|
|
|
|
| |
- Add deprecation log level
- Add deprecation utilities for structured format and messaging
- Update existing deprecation log sites, add deprecated versions
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
With systemd-networkd, the removal of /etc/machine-id can
result in DHCP errors such as:
DHCP4 CLIENT: Failed to set IAID+DUID: No such file or directory
Adapt cloud-init clean --machine-id to be systemd-aware.
systemd v. 237 and later will trigger /etc/machine-id regeneration
on next boot when the value of /etc/machine-id file is
'uninitialized\n'.
On systems without systemd, continue to remove /etc/machine-id when
cloud-init clean --machine-id is provided as there are triggers in
place to regenerate on absence of this file.
LP: #1999680
|
|
|
|
|
|
|
| |
Handles any situation where `status.json` does not yet contain
datasource information, by gracefully fulfilling a
`datasource=None` in `StatusDetails`.
LP: #1997559
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change is moving VMware Guest Customization data transport to VMware datasource,
the goal is to have a single datasource for VMware which is named DatasourceVMware.
Besides Guest Customization data transport(by local file), VMware datasource already
has other data transports, like Guestinfo Keys and environment arguments.
The detailed changes are:
1. Remove Guest Customization data transport from OVF datasource.
2. Refactor Guest Customization data transport code and add them to VMware datasource,
For backward compatibility, Guest Customization data transport is put ahead of the
other existing data transport since OVF is ahead of VMware in datasource searching
list.
3. Add instance-id support in customization configuration file, so that instance-id
can be given from vSphere side, if instance-id is not given, this datasource will
read it from the file /sys/class/dmi/id/product_uuid no matter what transport is.
4. Move Guest Customization detection from dscheck_OVF() to dscheck_VMmare() in
ds_identify script, Guest Customization detection is ahead of the other existing
data transports since OVF is ahead of VMware in datasource searching list.
5. Modify unittests according to above changes.
6. Modify both OVF and VMware datasource documents according to above changes.
7. Guestinfo OVF data transport is left to OVF datasource since it uses OVF which is
same with iso9660 transport.
|
|
|
| |
Retain legacy curtin log and config paths for Bionic, Focal, Jammy.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fix last reported event possibly not being sent
The run of every cloud-init mode is wrapped in a reporting
context manager. The final flush of events before the process exits was
happening within the context manager, however, one final event is sent
when the context manager exits. Since this event isn't subject to
waiting for event flush, cloud-init can exit before this event gets
sent.
This commit fixes this issue and also adds logging of POST data when
POSTING to a URL.
LP: #1993836
|
|
|
|
| |
Apport screens out symlinks, so we should send realpath() normalized
paths instead.
|
|
|
| |
When a NIC appears, check for a cloud-init.network-config and apply it.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
cli/apport: collect-logs include subiquity logs config when present
Add support for both cloud-init collect-logs and apport to
include subiquity live installer artifacts if present to aid in
bug triage.
Apport integration to attach subiquity, curtin and
ubuntu-desktop-installer report keys when logs or
config files are present.
`ubuntu-bug cloud-init` will also automatically tag the
bug as curtin, subiquity or ubuntu-desktop-installer if
related logs are present in the bug report.
Additional collect-logs support to collect
/var/lib/cloud/data in the event that cloud-init is disabled
by systemd generator with /run/cloud-init/disabled flag.
In these situations we want to collect /var/lib/cloud/data
dir for more context on prior cloud-init behavior.
|
| |
|
|
|
| |
Sponsored by: FreeBSD Foundation
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Provide --format option for cloud-init status and cloud-id to
surface machine-readable output and detailed boot_status_codes.
Provide new boot_code_code values in json/yaml output
which will surface unique codes for the current disabled
status reasons which gives visibility to whether cloud-init
may become active on next boot or later in the current boot.
The boot-status-code can be:
- 'unknown': systemd generators and ds-identify haven't run yet to
determine if cloud-init should be run during this boot
- 'disabled-by-marker-file': /etc/cloud/cloud-init.disabled exists
which prevents cloud-init from ever running
- 'disabled-by-generator': systemd generator ran ds-identify and
determined no applicable cloud-init datasources
- 'disabled-by-kernel-cmdline': kernel cmdline contained
cloud-init=disabled
- 'enabled-by-kernel-cmdline': kernel cmdline contained
cloud-init=enabled
- 'enabled-by-generator': ds-identify detected possible cloud-init
datasources
- 'enabled-by-sysvinit': enabled by default in SysV init environment
LP: #1883122
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There have been multiple requests to allow jinja templating in
/etc/cloud configs the same way we allow jinja templating in vendordata
and userdata. This commit allows for templating both
/etc/cloud/cloud.cfg and any file in /etc/cloud/cloud.cfg.d. The same
instance data used for substitution in vendordata and userdata will be
used here.
Note that these configs get loaded multiple times during the lifetime
of cloud-init, and during cloud-init's earlier loads, instance data
is not yet available.
LP: #1913461
|
|
|
|
|
|
|
| |
We have multiple areas of the codebase where we fetch the rundir path
and join it with the filename for instance-data.json or
instance-data-sensitive.json. This commit refactors those calls to use a
'Paths' lookup instead.
|
|
|
|
|
|
| |
Commit af40478 added a regression. When parsing v2 config, matching can
fail when no MAC is available. This commit fixes that behavior.
LP: #1986551
|
|
|
|
|
|
|
| |
Adhere to Netplan Passthrough documented behavior,
not limiting v2 netplan configs to the subset of
props that cloud-init supports.
LP: #1978543
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When metadata update events trigger a new datasource.get_data run
ensure we are syncing the cached obj.pkl to disk so subsequent
boot stages can leverage the updated metadata.
Add write_cache param to persist_instance_data to avoid
persisting instance data when init.ds_restored from cache.
This avoids a race on clouds where network config is updated
per boot in init-local timeframe but init-network uses stale network
metadata from cache because updated metadata was not
persisted.
Migate _pkl_load and _pkl_store out of stages module and into
sources as it really is only applicable to datasource serialization.
|
|
|
|
|
|
|
|
|
|
|
|
| |
In the case that the url content does not start with
`#cloud-config`, avoid downloading the full content.
Add deprecated logs to prefer `cloud-config-url` over
`url` on the kernel command line.
Restructure and link kernel command line docs to
instance-data docs.
Add some typing.
LP: #1937319
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Support overriding network activators in cloud.cfg system_info
on disk.
Default cloud-init activators are used either during hot-plug or
for datasources that are detected during init-network stage when
basic networking has already been setup by the OS.a
Activators are discovered in the following priority order and
determined based on the presence of related network tools:
system_info:
network:
activators: [eni, netplan, network-manager, networkd]
On some systems where multiple network config tools are installed
it is necessary to override the priority order of detection to ensure
one activator is chosen over the other. This is done by providing
the a custom cloud config snippet in a /etc/cloud/cloud.cfg.d/*cfg.
Surface configured Distro.network_activator from system_info
so that the hotplug hook has visibility to the configured
distribution network activator during interface bring_up and
tear_down.
Add typing hints to net.activators search_activator and
select_activator functions.
LP: #1958377
|
|
|
|
|
|
|
|
|
| |
Typically when a cloud-image boots /etc/machine-id should be
auto-generated if /etc/machine-id does not exist. Some cloud images
appropriately generate /etc/machine-id if the image is recognized
as a clone, in most other cases it is a best practice to remove
/etc/machine-id and let DBUS or systemd do the work of generating
a new UUID on next boot so the cloned VMs are represented
uniquely to any monitoring/management software.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
cloud-init clean command provides a run-parts /etc/cloud/clean.d
directory to allow third party apps to deliver supplemental cleanup
scripts to aid in golden image creation.
Any executable scripts in /etc/cloud/clean.d will be run in
lexicographical order via subp.runparts.
This aids customers creating golden images, where the typical process
an image creator performs is to reset cloud-init on that system so
the next boot of this cloned image will be treated as first-boot and
perform all early system configuration.
|
|
|
| |
Add tox target for tracing for resource leaks, fix some leaks
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add machinery to react to the schema annotation tag `deprecated`
consisting of:
- Log deprecation warnings if the entrypoint is `main`
- Print to stdout if the entrypoint is `cloud-init schema`
- Render deprecation info in docs
Activate deprecations:
- cc_ca_certs.ca_certs.remove-defaults
- cc_ca_certs.ca-certs
- cc_package_update_upgrade_install.{apt_update,apt_upgrade,
apt_reboot_if_required}
- cc_power_state_change.power_state.delay with `^\\+?[0-9]+$`
- cc_user_groups.users_groups.user.{groups,lock-passwd}
- distro.add_user.kwargs as group dict
As `jsonschema` does not return the annotations nor provide
more flexible hooks, we had to hook into it via
extending/implementing `validators` and overriding
`Validator.is_valid`.
|
|
|
|
|
|
|
|
|
| |
When schema errors are encountered, the section of userdata in question
gets printed to the cloud-init log. As this could contain sensitive
data, so log a generic warning instead and redirect user to run
cloud-init schema --system as root.
LP: #1978422
CVE: 2022-2084
|
|
|
|
|
|
|
|
|
|
|
| |
Ensure cloud_dir setting is respected rather than hardcoding
"/var/lib/cloud"
- Modules affected: cmd.main, apport, devel.logs (collect-logs),
cc_snap, sources.DataSourceAzure, sources.DataSourceBigstep,
util:fetch_ssl_details.
- testing: Extend and port to pytest unit tests, add integration test.
LP: #1976564
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
- Replace parse_network_config() with _generate_network_config()
instance method and consolidate cache checks into network_config.
- Update _generate_network_config_from_imds_metadata() to take
just network metadata portion of instance metadata and rename to
generate_network_config_from_instance_network_metadata().
- Consolidate relevant unit tests and refactor to pytest.
- Update net-convert to use
generate_network_config_from_instance_network_metadata().
Signed-off-by: Chris Patterson <cpatterson@microsoft.com>
|
|
|
|
|
|
|
|
|
|
|
|
| |
When debugging python's NetworkState intance we cannot use
safeyaml.dumps because that leverages the yaml.SafeDumper which
does not allow rendering python objects.
Use yamls.dump instead.
* Additional fix for networkd renderer to create /etc/systemd/network if
it does not exist
LP: #1975907
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Systemd used to sometimes ignore localhost in /etc/hostnames, and many programs
like cloud-init used this as a workaround to set a default hostname.
From https://github.com/systemd/systemd/commit/d39079fcaa05e23540d2b1f0270fa31c22a7e9f1:
We would sometimes ignore localhost-style names in /etc/hostname. That is
brittle. If the user configured some hostname, it's most likely because they
want to use that as the hostname. If they don't want to use such a hostname,
they should just not create the config. Everything becomes simples if we just
use the configured hostname as-is.
This behaviour seems to have been a workaround for Anaconda installer and other
tools writing out /etc/hostname with the default of "localhost.localdomain".
Anaconda PR to stop doing that: rhinstaller/anaconda#3040.
That might have been useful as a work-around for other programs misbehaving if
/etc/hostname was not present, but nowadays it's not useful because systemd
mostly controls the hostname and it is perfectly happy without that file.
Apart from making things simpler, this allows users to set a hostname like
"localhost" and have it honoured, if such a whim strikes them.
As also suggested by the Anaconda PR, we need to stop writing default "localhost"
in /etc/hostnames, and let the right service (networking, user) do that if they
need to. Otherwise, "localhost" will permanently stay as hostname and will
prevent other tools like NetworkManager from setting the right one.
Signed-off-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
RHBZ: 1980403
|
|
|
|
|
|
|
|
|
| |
- Add types to let mypy pass.
- Add mypy flags:
- detect unused ignores
- redundant casts
- Drop support of `ConfigParser` in Python 2
- Harden DataSourceLXD.network_config
- Convert old-style commented types to proper types.
|
|
|
|
|
|
|
|
|
|
|
|
| |
For non-root users, emit warnings and redact on any
/etc/cloud/cloud.cfg.d files which raise permissions errors.
Add tests covering this behavior for query, status and render
cmds.
Migrate `test_render.py` and `test_status.py` to Pytest.
LP: #1953430
SC-658
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
upstream cloud-init not longer publishes to distribution
versions which support upstart.
Trusty has not been receiving cloud-init updates since
0.7.5 and Xenial stopped getting updates as version 21.1.
Even though Ubuntu Xenial's default init system is systemd
there was still an option for folks to launch amd maintain
images which relied on upstart. Now that Xenial no longer has
updates from upstream cloud-init we can drop all upstart
support.
Old Fedora and RHEL releases which use upstart are also not
receiving cloud-init updates from tip of main.
|
|
|
|
|
|
|
| |
Changes:
- pylint 2.13.8: fix E0601(used-before-assignment)
- pylint 2.13.8: fix E4702(modified-iterating-dict)
- pylint 2.13.8: silence W0402(deprecated-module) on distutils.errors
- tox: bump linters versions
|
|
|
| |
Update cli docs and standardize grammar/spelling/punctuation
|
|
|
|
| |
- move datasource helpers to dedicated directory
- drop unnecessary executable bit on shebangless python files
|
|
|
|
|
|
|
|
| |
It was implemented as a namedtuple, because it was written
when the codebase supported Python 2 (where using an enum would have
introduced a new dependency). As enum is in the stdlib in all our
supported Python releases, we can now use it without that constraint.
LP: #1874875
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously, each cc_* module required a hardcoded 'distro' and
'frequency' variable to control the respective distro to run on and
run frequency. If undefined or invalid, we silently changed this to
a default value.
Instead, this commit will validate the MetaSchema definition and raise
an exception if invalid. This means every module MUST contain a
MetaSchema definition.
Additionally, the Modules class was moved out of stages.py into its
own module, along with some helper functions in config/__init__.py.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Fedora currently relies on sysconfig/ifcfg renderer. This is not too great,
because Fedora (also RHEL since version 8) dropped support for the legacy
network service that uses ifcfg files long ago.
In turn, Fedora ended up patching cloud-init downstream to utilize
NetworkManager's ifcfg compatibility mode [1]. This seems to have worked
for a while, nevertheless the NetworkManager's ifcfg backend is reaching
the end of its useful life too [2].
[1] https://src.fedoraproject.org/rpms/cloud-init/blob/rawhide/f/cloud-init-21.3-nm-controlled.patch
[2] https://fedoraproject.org/wiki/Changes/NoIfcfgFiles
Let's not mangle things downstream and make vanilla cloud-init work great
on Fedora instead.
This also means that the sysconfig compatibility with
Network Manager was removed.
Firstly, this relies upon the fact that you can get ifcfg support by adding
it to NetworkManager.conf. That is not guaranteed and certainly will not
be case in future.
Secondly, cloud-init always generates configuration with
NM_CONTROLLED=no, so the generated ifcfg files are no good for
NetworkManager. Fedora patches around this by just removing those lines
in their cloud-init package.
|
|
|
|
|
|
|
|
|
|
|
| |
Handlers for per-boot/per-instance/per-once multipart MIME
Add handlers for adding scripts to userdata that can be run at various
frequencies. Scripts of type x-shellscript-per-boot,
x-shellscript-per-instance, or x-shellscript-per-once can be added
to a multipart MIME userdata message as part of instance userdata.
These scripts will then be added to the appropriate per-boot,
per-instance, or per-once directory in /var/lib/cloud/scripts/
during processing of userdata.
|
|
|
|
|
|
|
|
|
|
|
|
| |
snapd currrently looks for 'not run' from cloud-init status[1].
Avoid changing this behavior and revert "not-run" value to
"not run".
This avoids having to get snapd to change implementation and
release updates as far back as Bionic to handle a hyphenated
not-run string.
[1]: https://github.com/snapcore/snapd/blob/master/sysconfig/\
cloudinit.go#L802
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Package a single JSON schema file for user-data validation at
cloudinit/config/cloud-init-schema.json.
Perform validate_cloudconfig_schema call to just after the
user-data is consumed. This will allow single validation of all
user-data against the full schema instead of
repetitive validatation calls against each cloud-config module
(cloudinit.config.cc_*) sub-schemas.
This branch defines the simple apt_pipelining schema and
migrates existing cc_apk_configure into cloud-init-schema.json.
The expectation will be additional branches to migrate from legacy
"schema" attributes inside each cloud-config module toward unique
cc_<module_name> definitions in the global shema file under "$defs"
of cloud-init-schema-X.Y..json.
Before legacy sub-schema definitions are migrated the following
funcs grew support to read sub-schemas from both static
cloud-init-schema.json and the individual cloud-config module
"schema" attributes:
- get_schema: source base schema file from cloud-init-schema.json
and supplement with all legacy cloud-config module "schema" defs
- get_meta_doc: optional schema param so cloud-config modules
no longer provide the own local sub-schemas
- _get_property_doc: render only documentation of sub-schema based
on meta['id'] provided
- validate_cloudconfig_schema: allow optional schema param
Additionally, fix two minor bugs in _schemapath_for_cloudconfig:
- `cloud-init devel schema --annotate` which results in a Traceback
if two keys at the same indent level have invalid types.
- exit early on empty cloud-config to avoid a Traceback on the CLI
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fix has two elements:
- cloud-init status will not correctly report 'not-run' prior to systemd
generator running. Only report "disabled" when generator has run
and /run/cloud-init/disabled exists.
- Expose not-run and disabled state in cloud-id responses
- Add unique error codes from cloud-id for error, disabled and not-run.
The new cloud-id exit codes:
0: success
1: error
2: cloud-init is in disabled state
3: cloud-init generator has not run yet
|
|
|
|
|
| |
Applied Black and isort, fixed any linting issues, updated tox.ini
and CI.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Improve schema validation.
This adds strict validation of config module definitions at testing
time, with plumbing included for future runtime validation. This
eliminates a class of bugs resulting from schemas that have definitions
that are incorrect, but get interpreted by jsonschema as
"additionalProperties" that are therefore ignored.
- Add strict meta-schema for jsonschema unit test validation
- Separate schema from module metadata structure
- Improve type annotations for various functions and data types
Cleanup:
- Remove unused jsonschema "required" elements
- Eliminate manual memoization in schema.py:get_schema(),
reference module.__doc__ directly
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This attempts to standardize unit test file location under test/unittests/
such that any source file located at cloudinit/path/to/file.py may have a
corresponding unit test file at test/unittests/path/to/test_file.py.
Noteworthy Comments:
====================
Four different duplicate test files existed:
test_{gpg,util,cc_mounts,cc_resolv_conf}.py
Each of these duplicate file pairs has been merged together. This is a
break in git history for these files.
The test suite appears to have a dependency on test order. Changing test
order causes some tests to fail. This should be rectified, but for now
some tests have been modified in
tests/unittests/config/test_set_passwords.py.
A helper class name starts with "Test" which causes pytest to try
executing it as a test case, which then throws warnings "due to Class
having __init__()". Silence by changing the name of the class.
# helpers.py is imported in many test files, import paths change
cloudinit/tests/helpers.py -> tests/unittests/helpers.py
# Move directories:
cloudinit/distros/tests -> tests/unittests/distros
cloudinit/cmd/devel/tests -> tests/unittests/cmd/devel
cloudinit/cmd/tests -> tests/unittests/cmd/
cloudinit/sources/helpers/tests -> tests/unittests/sources/helpers
cloudinit/sources/tests -> tests/unittests/sources
cloudinit/net/tests -> tests/unittests/net
cloudinit/config/tests -> tests/unittests/config
cloudinit/analyze/tests/ -> tests/unittests/analyze/
# Standardize tests already in tests/unittests/
test_datasource -> sources
test_distros -> distros
test_vmware -> sources/vmware
test_handler -> config # this contains cloudconfig module tests
test_runs -> runs
|