diff options
| -rw-r--r-- | docs/dev/README.md | 8 | ||||
| -rw-r--r-- | docs/dev/design_documents/cookbook_root_aliases.md | 52 | ||||
| -rw-r--r-- | docs/dev/design_documents/data_collector.md | 480 | ||||
| -rw-r--r-- | docs/dev/design_documents/event_handler_recipe_dsl.md | 67 | ||||
| -rw-r--r-- | docs/dev/design_documents/gem_installation_via_metadata.md | 31 | ||||
| -rw-r--r-- | docs/dev/design_documents/resource_load_and_converge_methods.md | 236 |
6 files changed, 859 insertions, 15 deletions
diff --git a/docs/dev/README.md b/docs/dev/README.md index 0f15ee6dee..fba17e7b9f 100644 --- a/docs/dev/README.md +++ b/docs/dev/README.md @@ -6,8 +6,8 @@ A good first start is our [How Chef Infra Is Built](how_chef_is_built.md) and [C ## How-To Guides -- [Building and Installing](./how_to/building_and_installing.md) -- [Branching and Backporting](./how_to/branching_and_backporting.md) +- [Building and Installing Locally](./how_to/building_and_installing.md) +- [Branching and Backporting Changes](./how_to/branching_and_backporting.md) - [Updating Dependencies](./how_to/updating_dependencies.md) - [Bumping Major and Minor Versions](./how_to/bumping_minor_or_major_versions.md) @@ -18,10 +18,14 @@ A good first start is our [How Chef Infra Is Built](how_chef_is_built.md) and [C - [Action Collection](./design_documents/action_collection.md) - [Deprecations Within Resources](./design_documents/deprecations_in_resources.md) - [Resource Guard Interpreters](./design_documents/resource_guard_interpreters.md) +- [Resource Load and Converge Methods](./design_documents/resource_load_and_converge_methods.md) - [Resource Property Validation Messaging](./design_documents/resource_property_validation_messaging.md) - [Self Documenting Resources](./design_documents/self_documenting_resources.md) - [Resource Before Notifications](./design_documents/resource_before_notifications.md) - [Resource File Content Verification](resource_file_content_verification.md) - [Ohai Cookbook Segment](./design_documents/ohai_cookbook_segment.md) +- [Cookbook Root Aliases](./design_documents/cookbook_root_aliases.md) +- [Event Handler Recipe DSL](./design_documents/event_handler_recipe_dsl.md) +- [Gem Installation via Metadata](./design_documents/gem_installation_via_metadata.md) - [Client Exit Codes](./design_documents/client_exit_codes.md) - [Server Enforced Recipes](./design_documents/server_enforced_recipes.md) diff --git a/docs/dev/design_documents/cookbook_root_aliases.md b/docs/dev/design_documents/cookbook_root_aliases.md new file mode 100644 index 0000000000..dbf0b2641d --- /dev/null +++ b/docs/dev/design_documents/cookbook_root_aliases.md @@ -0,0 +1,52 @@ +# Root Aliases in Cookbooks + +There are several common cases when writing Chef cookbooks that result in a +folder containing a single file, usually called `default.rb`. Root aliases +allow using a single file instead of a folder. + +## Motivation + + As a cookbook author, + I want to less complex directory layouts, + so that learning and maintenance is easier. + +## Specification + +There are two common cases where a single-file-in-folder comes up: + +1. `attributes/default.rb` +2. `recipes/default.rb` + +With `attributes` this is common to the point of almost complete irrelevance of +other layouts given that all attribute files are always loaded. Recipes aren't +exclusively singletons, but it is common enough to warrant a special case. + +With this in mind, aliases are available for each: + +1. `attributes.rb` +2. `recipe.rb` + +It is an error for a cookbook to contain both an alias and its target or two +aliases for the same target. + +No aliases are provided for other types as they are generally a more advanced +use case where the worry about learning curve is reduced. + +Aliases are equivalent to their target file for purposes of loading either via +standard cookbook loading or methods like `include_recipe`. + +## Rationale + +This meshes well with RFC017 towards a goal of reducing the file layout +complexity of simple cookbooks. There can be compatibility issues with tools +that parse the cookbook manifest data and presume that all files from a given +segment reside under the previously required folder, however the author knows +of no such tools and given that the manifest format is mostly an internal +representation this is not considered a blocker. Overall the goal of these RFCs +is to remove the frequent use of single-child folders. + +The choice of which aliases to provide and what to name them is mostly driven +by the common cases, but is not exhaustive. `attributes.rb` and `recipe.rb` are +chosen to match their usage grammatically. An additional alias of `recipes.rb` +could be provided to match the folder name, but this is left for a future +improvement based on usage feedback.
\ No newline at end of file diff --git a/docs/dev/design_documents/data_collector.md b/docs/dev/design_documents/data_collector.md index be0a92e7fb..dd0ac487db 100644 --- a/docs/dev/design_documents/data_collector.md +++ b/docs/dev/design_documents/data_collector.md @@ -4,18 +4,472 @@ title: Data Collector # Data Collector Design -The Data Collector design and API is covered in: +## Motivation + + As a Chef user who uses both Chef Client Mode and Chef Solo Mode (including the mode commonly known as "Chef Client Local Mode"), + I want to be able to collect data about my entire fleet regardless of their client operation type, + so that I may better understand the impacts of my changes and may better detect failures. + +### Definitions + +To eliminate ambiguity and confusion, the following terms are used throughout this RFC: + + * **Chef**: the tool used to automate your system. + * **Chef Client Mode**: Chef configured in "client mode" where a Chef Server is used to provide Chef its resources and artifacts + * **Chef Solo Mode**: Chef configured in a mode that utilizes a local Chef Zero server. Formerly known as "Chef Client Local Mode" (run as `chef-client --local-mode`) + * **Chef Solo Legacy Mode**: Chef in the pre 12.10 Solo operational mode (run as `chef-solo`) or Chef run as `chef-solo --legacy-mode` + +### Specification + +Similar to how data is collected and reported for Chef Reporting, we expect to implement a new EventDispatch class/instance that collects data about the Chef run and reports it accordingly. Unlike Chef Reporting, the server that receives this data is **not** running on the Chef Server, allowing users to utilize this function whether they use Chef Server or not. No new data collection methods are expected to be implemented as a result of this change; this change serves to implement a generic way to report the collected data in a "webhook-like" fashion to a non-Chef-Server receiver. + +The implementation must work with Chef running in any mode: + + * Chef Client Mode + * Chef Solo Mode + * Chef Solo Legacy Mode + +#### Protocol and Authentication + +All payloads will be sent to the Data Collector server via HTTP POST to the URL specified in the `data_collector_server_url` configuration parameter. Users should be encouraged to use a TLS-protected endpoint. + +Optionally, payloads may also be written out to multiple HTTP endpoints or JSON files on the local filesystem (of the node running chef-client) by specifying the `data_collector_output_locations` configuration parameter. + +For the initial implementation, transmissions to the Data Collector server can optionally be authenticated with the use of a pre-shared token which will be sent in a HTTP header. Given that the receiver is not the Chef Server, existing methods of using a Chef `client` key to authenticate the request are unavailable. + +#### Configuration + +The configuration required for this new functionality can be placed in the client.rb or any other `Chef::Config`-supported location (such as a client.d or solo.d directory). + +##### Parameters + + * **data\_collector\_server\_url**: required*. The full URL to the data collector server API. All messages will be POST'd to this URL. The Data Collector class will be registered and enabled if this config parameter is specified. * If the `data_collector_output_locations` configuration parameter is specified, this setting may be omitted. + * **data\_collector\_token**: optional. A pre-shared token that, if present, will be passed as an HTTP header named `x-data-collector-token` to the Data Collector server. The server can choose to accept or reject the data posted based on the token or lack thereof. + * **data\_collector\_mode**: The Chef mode in which the Data Collector will be enabled. For example, you may wish to only enable the Data Collector when running in Chef Solo Mode. Must be one of: `:solo`, `:client`, or `:both`. The `:solo` value is used for Chef operating in Chef Solo Mode or Chef Solo Legacy Mode. Default: `:both`. + * **data\_collector\_raise\_on\_failure**: If true, the Chef run will fatally exit if it is unable to successfully POST to the Data Collector server. Default: `false` + * **data\_collector\_output\_locations**: optional. An array of URLs and/or file paths to which data collection payloads will also be written. This may be used without specifying the `data_collector_server_url` configuration parameter + +### Schemas + +For the initial implementation, three JSON schemas will be utilized. + +##### Action Schema + +The Action Schema is used to notify when a Chef object changes. In our case, the primary use will be to update the Data Collector server with the current node object. + +```json +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "description": "Data Collector - action schema", + "properties": { + "entity_name": { + "description": "The name of the entity", + "type": "string" + }, + "entity_type": { + "description": "The type of the entity", + "type": "string", + "enum": [ + "bag", + "client", + "cookbook", + "environment", + "group", + "item", + "node", + "organization", + "permission", + "role", + "user", + "version"] + }, + "entity_uuid": { + "description": "Unique ID identifying this object, which should persist across runs and invocations", + "type": "string", + "pattern": "^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$" + }, + "id": { + "description": "Globally Unique ID for this message", + "type": "string", + "pattern": "^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$" + }, + "message_version": { + "description": "Message Version", + "type": "string", + "enum": [ + "1.1.0" + ] + }, + "message_type": { + "description": "Message Type", + "type": "string", + "enum": ["action"] + }, + "organization_name": { + "description": "It is the name of the org on which the run took place", + "type": ["string", "null"] + }, + "recorded_at": { + "description": "It is the ISO timestamp when the action happened", + "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-5][0-9]:[0-9]{2}Z$", + "type": "string" + }, + "remote_hostname": { + "description": "The remote hostname which initiated the action", + "type": "string" + }, + "requestor_name": { + "description": "The name of the client or user that initiated the action", + "type": "string" + }, + "requestor_type": { + "description": "Was the requestor a client or user?", + "type": "string", + "enum": ["client", "user"] + }, + "run_id": { + "description": "The run ID of the run in which this node object was updated", + "pattern": "^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$", + "type": "string" + }, + "service_hostname": { + "description": "The FQDN of the Chef server, if appropriate", + "type": "string" + }, + "source": { + "description": "The tool / client mode that initiated the action. Note that 'chef_solo' includes Chef Solo Mode and Chef Solo Legacy Mode.", + "type": "string", + "enum": ["chef_solo", "chef_client"] + }, + "task": { + "description": "What action was performed?", + "type": "string", + "enum": ["associate", "create", "delete", "dissociate", "invite", "reject", "update"] + }, + "user_agent": { + "description": "The User-Agent of the requestor", + "type": "string" + }, + "data": { + "description": "The payload containing the entire request data", + "type": "object" + } + }, + "required": [ + "entity_name", + "entity_type", + "entity_uuid", + "id", + "message_type", + "message_version", + "organization_name", + "recorded_at", + "remote_hostname", + "requestor_name", + "requestor_type", + "run_id", + "service_hostname", + "source", + "task", + "user_agent" + ], + "title": "ActionSchema", + "type": "object" +} +``` + +The `data` field will contain the value of the object on which an action took place. + +##### Run Start Schema + +The Run Start Schema will be used by Chef to notify the data collection server at the start of the Chef run. + +```json +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "description": "Data Collector - Runs run_start schema", + "properties": { + "chef_server_fqdn": { + "description": "It is the FQDN of the chef_server against whch current reporting instance runs", + "type": "string" + }, + "entity_uuid": { + "description": "Unique ID identifying this node, which should persist across Chef runs", + "type": "string", + "pattern": "^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$" + }, + "id": { + "description": "It is the internal message id for the run", + "pattern": "^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$", + "type": "string" + }, + "message_version": { + "description": "Message Version", + "type": "string", + "enum": [ + "1.0.0" + ] + }, + "message_type": { + "description": "It defines the type of message being sent", + "type": "string", + "enum": ["run_start"] + }, + "node_name": { + "description": "It is the name of the node on which the run took place", + "type": "string" + }, + "organization_name": { + "description": "It is the name of the org on which the run took place", + "type": "string" + }, + "run_id": { + "description": "It is the runid for the run", + "pattern": "^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$", + "type": "string" + }, + "source": { + "description": "The tool / client mode that initiated the action. Note that 'chef_solo' includes Chef Solo Mode and Chef Solo Legacy Mode.", + "type": "string", + "enum": ["chef_solo", "chef_client"] + }, + "start_time": { + "description": "It is the ISO timestamp of when the run started", + "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$", + "type": "string" + } + }, + "required": [ + "chef_server_fqdn", + "entity_uuid", + "id", + "message_version", + "message_type", + "node_name", + "organization_name", + "run_id", + "source", + "start_time" + ], + "title": "RunStartSchema", + "type": "object" +} +``` + +##### Run End Schema + +The Run End Schema will be used by Chef Client to notify the data collection server at the completion of the Chef Client's converge phase and report data on the Chef Client run, including resources changed and any errors encountered. + +```json +{ + "$schema": "http://json-schema.org/draft-04/schema#", + "description": "Data Collector - Runs run_converge schema", + "properties": { + "chef_server_fqdn": { + "description": "It is the FQDN of the chef_server against whch current reporting instance runs", + "type": "string" + }, + "end_time": { + "description": "It is the ISO timestamp of when the run ended", + "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$", + "type": "string" + }, + "entity_uuid": { + "description": "Unique ID identifying this node, which should persist across Chef Client/Solo runs", + "type": "string", + "pattern": "^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$" + }, + "error": { + "description": "It has the details of the error in the run if any", + "type": "object" + }, + "expanded_run_list": { + "description": "The expanded run list object from the node", + "type": "object" + }, + "id": { + "description": "It is the internal message id for the run", + "pattern": "^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$", + "type": "string" + }, + "message_type": { + "description": "It defines the type of message being sent", + "type": "string", + "enum": ["run_converge"] + }, + "message_version": { + "description": "Message Version", + "type": "string", + "enum": [ + "1.1.0" + ] + }, + "node": { + "description": "The node object after the converge completed", + "type": "object" + }, + "node_name": { + "description": "Node Name", + "type": "string", + "format": "node-name" + }, + "organization_name": { + "description": "Organization Name", + "type": "string" + }, + "resources": { + "description": "This is the list of all resources for the run", + "type": "array", + "items": { + "type": "object", + "properties": { + "after": { + "description": "Final State of the resource", + "type": "object" + }, + "before": { + "description": "Initial State of the resource", + "type": "object" + }, + "cookbook_name": { + "description": "Name of the cookbook that initiated the change", + "type": "string" + }, + "cookbook_version": { + "description": "Version of the cookbook that initiated the change", + "type": "string", + "pattern": "^[0-9]*\\.[0-9]*(\\.[0-9]*)?$" + }, + "delta": { + "description": "Difference between initial and final value of resource", + "type": "string" + }, + "duration": { + "description": "Duration of the run consumed by processing of this resource, in milliseconds", + "type": "string" + }, + "id": { + "description": "Resource ID", + "type": "string" + }, + "ignore_failure": { + "description": "the ignore_failure setting on a resource, indicating if a failure on this resource should be ignored", + "type": "boolean" + }, + "name": { + "description": "Resource Name", + "type": "string" + }, + "result": { + "description": "The action taken on the resource", + "type": "string" + }, + "status": { + "description": "Status indicating how Chef processed the resource", + "type": "string", + "enum": [ + "failed", + "skipped", + "unprocessed", + "up-to-date", + "updated" + ] + }, + "type": { + "description": "Resource Type", + "type": "string" + } + }, + "required": [ + "after", + "before", + "delta", + "duration", + "id", + "ignore_failure", + "name", + "result", + "status", + "type" + ] + } + }, + "run_id": { + "description": "It is the runid for the run", + "pattern": "^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$", + "type": "string" + }, + "run_list": { + "description": "It is the runlist for the run", + "type": "array", + "items": { + "type": "string" + } + }, + "source": { + "description": "The tool / client mode that initiated the action. Note that 'chef_solo' includes Chef Solo Mode and Chef Solo Legacy Mode.", + "type": "string", + "enum": ["chef_solo", "chef_client"] + }, + "start_time": { + "description": "It is the ISO timestamp of when the run started", + "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$", + "type": "string" + }, + "status": { + "description": "It gives the status of the run", + "type": "string", + "enum": [ + "success", + "failure" + ] + }, + "total_resource_count": { + "description": "It is the total number of resources for the run", + "type": "integer", + "minimum": 0 + }, + "updated_resource_count": { + "description": "It is the number of updated resources during the course of the run", + "type": "integer", + "minimum": 0 + } + }, + "required": [ + "chef_server_fqdn", + "entity_uuid", + "id", + "end_time", + "expanded_run_list", + "message_type", + "message_version", + "node", + "node_name", + "organization_name", + "resources", + "run_id", + "run_list", + "source", + "start_time", + "status", + "total_resource_count", + "updated_resource_count" + ], + "title": "RunEndSchema", + "type": "object" +} +``` -https://github.com/chef/chef-rfc/blob/master/rfc077-mode-agnostic-data-collection.md +## Technical Implementation -This document will focus entirely on the nuts and bolts of the Data Collector +The remainder of document will focus entirely on the nuts and bolts of the Data Collector -## Action Collection Integration +### Action Collection Integration Most of the work is done by a separate Action Collection to track the actions of Chef resources. If the Data Collector is not enabled, it never registers with the Action Collection and no work will be done by the Action Collection to track resources. -## Additional Collected Information +### Additional Collected Information The Data Collector also collects: @@ -27,14 +481,14 @@ The Data Collector also collects: Most of this is done through hooking events directly in the Data Collector itself. The ErrorHandlers module is broken out into a module which is directly mixed into the Data Collector to separate that concern out into a different file (it is straightforward with fairly little state, but is just a lot of hooked methods). -## Basic Configuration Modes +### Basic Configuration Modes -### Configured for Automate +#### Configured for Automate Do nothing. The URL is constructed from the base `Chef::Config[:chef_server_url]`, auth is just Chef Server API authentication, and the default behavior is that it is configured. -### Configured to Log to a File +#### Configured to Log to a File Setup a file output location, no token is necessary: @@ -44,7 +498,7 @@ Chef::Config[:data_collector][:output_locations] = { files: [ "/Users/lamont/da Note the fact that you can't assign to `Chef::Config[:data_collector][:output_locations][:files]` and will NoMethodError if you try. -### Configured to Log to a Non-Chef Server Endpoint +#### Configured to Log to a Non-Chef Server Endpoint Setup a server url, requiring a token: @@ -64,7 +518,7 @@ it is **not** possible to use this feature to talk to both a Chef Automate endpo incredibly marginally useful feature and it does not work). But given how hopelessly complicated this is, the recommendation is to use the `server_url` and to avoid using any `url` options in the `output_locations` since that feature is fairly poorly designed at this point in time. -## Resiliency to Failures +### Resiliency to Failures The Data Collector in Chef >= 15.0 is resilient to failures that occur anywhere in the main loop of the `Chef::Client#run` method. In order to do this there is a lot of defensive coding around internal data structures that may be nil (e.g. failures before the node is loaded will result in the node being nil). The spec tests for @@ -76,19 +530,19 @@ in the presence of failures. But some data structures will be entirely empty. When the Data Collector fails extraordinarily early it still sends both a start and an end message. This will happen if it fails so early that it would not normally have sent a start message. -## Decision to Be Enabled +### Decision to Be Enabled This is complicated due to over-design and is encapsulated in the `#should_be_enabled?` method and the ConfigValidation module. The `#should_be_enabled?` message and ConfigValidation should probably be merged into one renamed Config module to isolate the concern of processing the Chef::Config options and doing the correct thing. -## Run Start and Run End Message modules +### Run Start and Run End Message modules These are separated out into their own modules, which are very deliberately not mixed into the main Data Collector. They use the Data Collector and Action Collection public interfaces. They are stateless themselves. This keeps the collaboration between them and the Data Collector very easy to understand. The start message is relatively simple and straightforwards. The complication of the end message is mostly due to walking through the Action Collection and all the collected action records from the entire run, along with a lot of defensive programming to deal with early errors. -## Relevant Event Sequence +### Relevant Event Sequence As it happens in the actual chef-client run: diff --git a/docs/dev/design_documents/event_handler_recipe_dsl.md b/docs/dev/design_documents/event_handler_recipe_dsl.md new file mode 100644 index 0000000000..c8d0d94c1b --- /dev/null +++ b/docs/dev/design_documents/event_handler_recipe_dsl.md @@ -0,0 +1,67 @@ +# Recipe DSL method for event handler hooks + +Allow cookbook authors to easily add custom logic on Chef events. + +## Motivation + +Chef has an extensive event [dispatch mechanism](https://github.com/chef/chef/blob/master/lib/chef/event_dispatch/base.rb). +But incorporating some custom logic against any of the events is an onerous process which involves +subclassing the based event handler and adding it via the config. This RFC +proposes a recipe DSL method to ease this. For new chef users this will reduce +the entry barrier. + +## Specification + +Currently chef client sets up couple of default handlers (doc, base) during +initialization. An additional empty event handler (a subclass +of the base handler without any custom logic) can be added alongside the +existing handlers which will used as a placeholder for user specific hooks. + +A top level (::Chef) method will be introduced (`event_handler`) to wrap the +main event handler DSL (`on`). Users can tap into one of the event types +(as specified in base dispatcher) using this DSL to execute their custom logic. + +The additional top level method(`Chef.event_handler`) will allow the handler +DSL usage in and outside of recipes and also ease writing backward compatible +changes for the `on` method if need be. + +Following is an example of sending hipchat notification on chef run failure. + +```ruby +Chef.event_handler do + on :run_failed do |exception| + hipchat_notify exception.message + end +end +``` + +Following is another example of taking a distributed lock via etcd, to +prevent concurrent chef runs in different nodes + +```ruby +lock_key = "#{node.chef_environment}/#{node.name}" + +Chef.event_handler do + on :converge_start do |run_context| + Etcd.lock_acquire(lock_key) + end +end + +Chef.event_handler do + on :converge_complete do + Etcd.lock_release(lock_key) + end +end +``` + +Following is another example of sending a hipchat alert on a key config change + +```ruby +Chef.event_handler do + on :resource_updated do |resource, action| + if resource.to_s == 'template[/etc/nginx/nginx.conf]' + Helper.hipchat_message("#{resource} was updated by chef") + end + end +end +```
\ No newline at end of file diff --git a/docs/dev/design_documents/gem_installation_via_metadata.md b/docs/dev/design_documents/gem_installation_via_metadata.md new file mode 100644 index 0000000000..6263ab903d --- /dev/null +++ b/docs/dev/design_documents/gem_installation_via_metadata.md @@ -0,0 +1,31 @@ +# Enable gem dependencies in cookbook metadata + +Support a 'gem' DSL method for cookbook metadata to create a dependency on a rubygem. The +gem will be installed via `chef_gem` after all the cookbooks are synchronized but before any +other cookbook loading is done. + +## Motivation + + As a Chef User, + I want to be able to use additional gems in libraries, attributes and resources, + to avoid complex workarounds and double-run converges. + +## Specification + +Allow users to specify additional gem dependencies like: + +```ruby +gem "poise" +gem "chef-sugar" +gem "chef-provisioning" +``` + +In the `Chef::RunContext::CookbookCompiler#compile` method a phase will be added before `compile_libraries` which will install all of the gem declarations from all of the synchronized cookbooks before any other cookbook code is compiled. + +The implementation will use an in-memory bundler Gemfile which is constructed against all gem statements in all cookbooks which are in the `run_list`, solved +at the same time. The syntax of the 'gem' statement will support the bundler gem syntax, with the qualification that since it is compiled into metadata.json +that arbitrary ruby code will be expanded at cookbook upload time. + +The resulting gemset bundle will be installed into the LIBPATH of the running chef-client. This may either be directly into the base ruby libraries (per current `chef_gem` behavior) or into a custom location with the LIBPATH of the chef-client extended to use that location--as an open implementation question. + +The normal Gemfile `requires` tag may be used by users to autoload files out of gems. diff --git a/docs/dev/design_documents/resource_load_and_converge_methods.md b/docs/dev/design_documents/resource_load_and_converge_methods.md new file mode 100644 index 0000000000..e08eddb399 --- /dev/null +++ b/docs/dev/design_documents/resource_load_and_converge_methods.md @@ -0,0 +1,236 @@ +# Easy Resource Load And Converge Methods + +With the introduction of `action` on resources, it becomes useful to have a +blessed way to get the actual value of the resource. This proposal adds +`load_current_value` and `converge_if_changed` to help with this purpose, enabling: + +- Low-ceremony load methods (as easy to write as we can make it) +- A super easy converge model that automatically compares current vs. desired + values and prints green text + +## Motivation + + As a Chef resource writer, + I want to be able to read the current value of my resource at converge time, + so that it is easy to tell the difference between current and desired value. + + As a Chef resource writer, + I want a converge model that compares current and desired values for me, + So that the easiest converge to write is the most correct one. + +## Specification + +### `load_current_value`: in-place resource load + +When using `action`, one needs a way to load the *actual* system value of the resource, so that it can be compared to the desired value and a decision made as to whether to change anything. + +When the resource writer defines `load_current_value` on the resource class, it can be called to load the real system value into the resource. Before any action runs, this will be used by `load_current_resource` to load the resource. `action` will do some important work before calling the new method: + +1. Create a new instance of the resource with the same name. +2. Copy all non-desired-state values from the desired resource into the new instance. +3. Call `load_current_value` on the new instance. + + +```ruby +class File < Chef::Resource + property :path, name_attribute: true + property :mode, default: 0666 + property :content + + load_current_value do + current_value_does_not_exist! unless File.exist?(path) + mode File.stat(path).mode + content IO.read(path) + end + + action :create do + converge_if_changed do + File.chmod(mode, path) + IO.write(path, content) + end + end +end + +file '/x.txt' do + # Before the change, the above code would have modified `mode` to be `0666`. + # After, it leaves `mode` alone. + content 'Hello World' +end +``` + +#### Non-existence + +To appropriately handle actual value loading, the user needs a way to specify that the actual value legitimately does not exist (rather than simply not filling in the object and getting `nil`s in it). If `load_current_value` raises `Chef::Exceptions::ActualValueDoesNotExist`, the new resource will be discarded and `current_resource` becomes `nil`. The `current_value_does_not_exist!` method can be called to raise this. + +NOTE: The alternative was to have users return `false` if the resource does not exist; but I didn't want users to be forced into the ceremony of a trailing `true` line. + +```ruby + load_current_value do + # Check for existence before doing anything else. + current_value_does_not_exist! if !File.exist?(path) + + # Set "mode" on the resource. + mode File.stat(path).mode + end +``` + +The block will also be passed the original (desired) resource as a parameter, in case it is needed. + +#### Inheritance + +`super` in `load_current_value!` will call the superclass's `load_current_value!` method. + +#### Handling Multi-Key Resources + +The new resource is created with all properties copied over *except* desired state properties (properties in `ResourceClass.state_properties`). This means `name`, and properties with `identity: true` or `desired_state: false` are copied over. Normal `property` and `attribute` are not. + +```ruby +class DataBagItem < Chef::Resource + # Copied + attribute :item_name, name_attribute: true + attribute :data_bag_name, identity: true + attribute :recursively_delete, desired_state: false + # Not copied: + attribute :data + def load_current_value! + data Chef::DataBagItem.new(data_bag_name, item_name).data + end +end +``` + +### `converge_if_changed`: automatic test-and-set + +The new `converge_if_changed do ... end` syntax is added to actions, which enables a *lot* of help for resource writers to make safe, effective resources. It performs several key tasks common to nearly every resource (which are often not done correctly): + +- Goes through all attributes on the resource and checks whether the desired + value is different from the current value. +- If any attributes are different, prints appropriate green text. +- Honors why-run (and does not call the `converge_if_changed` block if why-run is enabled). + +```ruby +class File < Chef::Resource + property :path, name_attribute: true + property :content + + load_current_value do + current_value_does_not_exist! unless File.exist?(path) + content IO.read(path) + end + + action :create do + converge_if_changed do + IO.write(path, content) + end + end +end +``` + +#### Side-by-side: new and old + +Here is a sample `converge_if_changed` statement from a hypothetical FooBarBaz resource with properties `foo`, `bar` and `baz`: + +```ruby +converge_if_changed do + if current_resource + FooBarBaz.update(new_resource.id, new_resource.foo, new_resource.bar, new_resource.baz) + else + FooBarBaz.create(new_resource.id, new_resource.foo, new_resource.bar, new_resource.baz) + end +end +``` + +This is what you would have to write to do the equivalent: + +```ruby +if current_resource + # We're updating; look for properties that the user wants to change (do the "test" part of test-and-set) + differences = [] + if (new_resource.property_is_set?(:foo) && new_resource.foo != current_resource.foo) + differences << "foo = #{new_resource.foo}" + end + if (new_resource.property_is_set?(:bar) && new_resource.bar != current_resource.bar) + differences << "bar = #{new_resource.bar}" + end + if (new_resource.property_is_set?(:baz) && new_resource.baz != current_resource.baz) + differences << "baz = #{new_resource.baz}" + end + + if !differences.empty? + converge_by "updating FooBarBaz #{new_resource.id}, setting #{differences.join(", ")}" do + FooBarBaz.create(new_resource.id, new_resource.foo, new_resource.bar, new_resource.baz) + end + end + +else + # If the current resource doesn't exist, we're definitely creating it + converge_by "creating FooBarBaz #{new_resource.id} with foo = #{new_resource.foo}, bar = #{new_resource.bar}, baz = #{new_resource.baz}" do + FooBarBaz.update(new_resource.id, new_resource.foo, new_resource.bar, new_resource.baz) + end +end +``` + +#### Desired value = actual value + +> The easiest way to write a resource must be the most correct one. + +There is a subtle pitfall when updating a resource, where the user has set *some* values, but not all. One can easily end up writing a resource which will overwrite perfectly good system properties with their defaults, which can cause instability. If the user does not specify a property, it is generally preferable to preserve its existing value rather than overwrite it. + +To prevent this, referencing the bare property in an `action` will now yield the *actual* value if load_current_value succeeded, and the *default* value if we are creating a new resource (if `load_current_value` raised `ActualValueDoesNotExist`). + +```ruby +class File < Chef::Resource + property :path, name_attribute: true + property :mode, default: 0666 + property :content + + load_current_value do + current_value_does_not_exist! unless File.exist?(path) + mode File.stat(path).mode + content IO.read(path) + end + + action :create do + converge_if_changed do + File.chmod(mode, path) + IO.write(path, content) + end + end +end + +file '/x.txt' do + # Before the change, the above code would have modified `mode` to be `0666`. + # After, it leaves `mode` alone. + content 'Hello World' +end +``` + +There will be times when the old behavior of overwriting with defaults is desired. The resource writer can still find out whether `mode` was set with `property_is_set?(:mode)`, and can still access the default value with `new_resource.mode` if it is not set. + +There are no backwards-compatibility issues with this because it only applies to `action`, which has not been released yet. + +#### Compound Resource Convergence + +Some resources perform several different (possibly expensive) operations depending on what is set. `converge_if_changed :attribute1, :attribute2, ... do` allows the user to target different groups of changes based on exactly which attributes have changed: + +```ruby +class File < Chef::Resource + property :path, name_attribute: true + property :mode + property :content + + load_current_value do + current_value_does_not_exist! unless File.exist?(path) + mode File.stat(path).mode + content IO.read(path) + end + + action :create do + converge_if_changed :mode do + File.chmod(mode, path) + end + converge_if_changed :content do + IO.write(path, content) + end + end +end +```
\ No newline at end of file |