diff options
| author | Julia Kreger <juliaashleykreger@gmail.com> | 2016-02-02 10:44:06 -0500 |
|---|---|---|
| committer | Julia Kreger <juliaashleykreger@gmail.com> | 2016-06-07 09:13:22 -0400 |
| commit | 832826f64076fa23e7a1648b0dda6dfb89ab08b3 (patch) | |
| tree | 5b5ea42567653f5574be871ecf5758a43e08dbb5 | |
| parent | 8b75becccbf0fa1fbdc8c6e9382d34750b892cba (diff) | |
| download | ironic-832826f64076fa23e7a1648b0dda6dfb89ab08b3.tar.gz | |
Active Node Creation via adopt state
At present the ironic API explicitly sets the new state for
nodes to the beginning step in the ironic workflow.
As part of hardware fleet lifecycle management, an operator
expects to be able to migrate inventory and control systems
for their hardware fleet utilizing their existing inventory
data and allocation records. Ultimately this means that
an imported host MAY already be allocated and unavailable
for immediate allocation.
As such, a mechanism is required to permit users to put nodes
into an ACTIVE state without performing a deployment operation.
This adds a new API provision_state verb to allow users to move
nodes from MANAGEABLE state to ACTIVE state.
Partial-Bug: #1526315
Change-Id: Ib3eadf4172e93add9a9855582f56cbb3707f3d39
Depends-On: Ie114bfaab249d73ea3ca7c0edc314ca1ed0448eb
| -rw-r--r-- | doc/source/deploy/adoption.rst | 183 | ||||
| -rw-r--r-- | doc/source/images/states.svg | 430 | ||||
| -rw-r--r-- | doc/source/index.rst | 1 | ||||
| -rw-r--r-- | doc/source/webapi/v1.rst | 8 | ||||
| -rw-r--r-- | ironic/api/controllers/v1/node.py | 3 | ||||
| -rw-r--r-- | ironic/api/controllers/v1/utils.py | 1 | ||||
| -rw-r--r-- | ironic/api/controllers/v1/versions.py | 4 | ||||
| -rw-r--r-- | ironic/common/states.py | 43 | ||||
| -rw-r--r-- | ironic/conductor/manager.py | 58 | ||||
| -rw-r--r-- | ironic/conductor/utils.py | 19 | ||||
| -rw-r--r-- | ironic/tests/unit/api/v1/test_nodes.py | 87 | ||||
| -rw-r--r-- | ironic/tests/unit/api/v1/test_utils.py | 11 | ||||
| -rw-r--r-- | ironic/tests/unit/conductor/test_manager.py | 150 | ||||
| -rw-r--r-- | ironic/tests/unit/conductor/test_utils.py | 17 | ||||
| -rw-r--r-- | releasenotes/notes/active-node-creation-a41c9869c966c82b.yaml | 14 |
15 files changed, 826 insertions, 203 deletions
diff --git a/doc/source/deploy/adoption.rst b/doc/source/deploy/adoption.rst new file mode 100644 index 000000000..34fab413d --- /dev/null +++ b/doc/source/deploy/adoption.rst @@ -0,0 +1,183 @@ +.. _adoption: + +============= +Node adoption +============= + +Overview +======== +As part of hardware inventory lifecycle management, it is not an +unreasonable need to have the capability to be able to add hardware +that should be considered "in-use" by the Bare Metal service, +that may have been deployed by another Bare Metal service +installation or deployed via other means. + +As such, the node adoption feature allows a user to define a node +as ``active`` while skipping the ``available`` and ``deploying`` +states, which will prevent the node from being seen by the Compute +service as ready for use. + +This feature is leveraged as part of the state machine workflow, +where a node in ``manageable`` can be moved to ``active`` state +via the provision_state verb ``adopt``. To view the state +transition capabilities, please see :ref:`states`. + +How it works +============ + +A node initially enrolled begins in the ``enroll`` state. An operator +must then move the node to ``manageable`` state, which causes the driver's +``power`` interface to be validated. Once in ``manageable`` state, +an operator can then explicitly choose to adopt a node. + +Adoption of a node results in the validation of the driver ``boot`` interface, +and upon success the process leverages what is referred to as the "takeover" +logic. The takeover process is intended for conductors to take over the +management of nodes for a conductor that has failed. + +The takeover process involves the driver deploy ``prepare`` and ``take_over`` +methods being called. These steps take driver specific actions such as +downloading and staging the deployment kernel and ramdisk, ISO image, any +required boot image, or boot ISO image and then places any PXE or virtual +media configuration necessary for the node should it be required. + +The adoption process makes no changes to the physical node, with the +exception of operator supplied configurations where virtual media is +used to boot the node under normal circumstances. An operator should +ensure that any supplied configuration defining the node is sufficient +for the continued operation of the node moving forward. Such as, if the +node is configured to network boot via instance_info/boot_option="netboot", +then appropriate driver specific node configuration should be set to +support this capability. + +Possible Risk +============= + +The main risk with this feature is that supplied configuration may ultimately +be incorrect or invalid which could result in potential operational issues: + +* ``rebuild`` verb - Rebuild is intended to allow a user to re-deploy the node + to a fresh state. The risk with adoption is that the image defined when an + operator adopts the node may not be the valid image for the pre-existing + configuration. + + If this feature is utilized for a migration from one deployment to another, + and pristine original images are loaded and provided, then ultimately the + risk is the same with any normal use of the ``rebuild`` feature, the server + is effectively wiped. + +* When deleting a node, the deletion or cleaning processes may fail if the + incorrect deployment image is supplied in the configuration as the node + may NOT have been deployed with the supplied image and driver or + compatibility issues may exist as a result. + + Operators will need to be cognizant of that possibility and should plan + accordingly to ensure that deployment images are known to be compatible + with the hardware in their environment. + +* Networking - Adoption will assert no new networking configuration to the + newly adopted node as that would be considered modifying the node. + + Operators will need to plan accordingly and have network configuration + such that the nodes will be able to network boot. + +How to use +========== + +.. NOTE:: + The power state that the ironic-conductor observes upon the first + successful power state check, as part of the transition to the + ``manageable`` state will be enforced with a node that has been adopted. + This means a node that is in ``power off`` state will, by default, have + the power state enforced as ``power off`` moving forward, unless an + administrator actively changes the power state using the Bare Metal + service. + +Requirements +------------ + +Requirements for use are essentially the same as to deploy a node: + +* Sufficient driver information to allow for a successful + power management validation. + +* Sufficient instance_info to pass deploy driver validation. + +Each driver may have additional requirements dependent upon the +configuration that is supplied. An example of this would be defining +a node to always boot from the network, which will cause the conductor +to attempt to retrieve the pertinent files. Inability to do so will +result in the adoption failing, and the node being placed in the +``adopt failed`` state. + +agent_ipmitool example +---------------------- + +This is an example to create a new node, named ``testnode``, with +sufficient information to pass basic validation in order to be taken +from the ``manageable`` state to ``active`` state.:: + + # Explicitly set the client API version environment variable to + # 1.17, which introduces the adoption capability. + export IRONIC_API_VERSION=1.17 + + ironic node-create -n testnode \ + -d agent_ipmitool \ + -i ipmi_address=<ip_address> \ + -i ipmi_username=<username> \ + -i ipmi_password=<password> \ + -i deploy_kernel=<deploy_kernel_id_or_url> \ + -i deploy_ramdisk=<deploy_ramdisk_id_or_url> + + ironic port-create --node <node_uuid> -a <node_mac_address> + + ironic node-update testnode add \ + instance_info/image_source="http://localhost:8080/blankimage" + + ironic node-set-provision-state testnode manage + + ironic node-set-provision-state testnode adopt + +.. NOTE:: + In the above example, the image_source setting must reference a valid + image or file, however that image or file can ultimately be empty. + +.. NOTE:: + The above example will naturally fail as a fake image is + defined, and no instance_info/image_checksum is defined so + any actual attempt to write the image out will fail. + +.. NOTE:: + A user may wish to assign an instance_uuid to a node, which could be + used to match an instance in the Compute service. Doing so is not + required for the proper operation of the Bare Metal service. + + ironic node-update <node name or uuid> add instance_uuid=<uuid> + +Troubleshooting +=============== + +Should an adoption operation fail for a node, the error that caused the +failure will be logged in the node's ``last_error`` field when viewing the +node. This error, in the case of node adoption, will largely be due to +failure of a validation step. Validation steps are dependent +upon what driver is selected for the node. + +Any node that is in the ``adopt failed`` state can have the ``adopt`` verb +re-attempted. Example:: + + ironic node-set-provision-state <node name or uuid> adopt + +If a user wishes to abort their attempt at adopting, they can then move +the node back to ``manageable`` from ``adopt failed`` state by issuing the +``manage`` verb. Example:: + + ironic node-set-provision-state <node name or uuid> manage + +If all else fails the hardware node can be removed from the Bare Metal +service. The ``node-delete`` command, which is **not** the same as setting +the provision state to ``delete``, can be used while the node is in +``adopt failed`` state. This will delete the node without cleaning +occurring to preserve the node's current state. Example:: + + ironic node-delete <node name or uuid> diff --git a/doc/source/images/states.svg b/doc/source/images/states.svg index 587892f28..2c8ceccb5 100644 --- a/doc/source/images/states.svg +++ b/doc/source/images/states.svg @@ -1,298 +1,338 @@ <?xml version="1.0" encoding="UTF-8" standalone="no"?> <!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"> -<!-- Generated by graphviz version 2.30.1 (20140821.0912) +<!-- Generated by graphviz version 2.36.0 (20140111.2315) --> <!-- Title: Ironic states Pages: 1 --> -<svg width="1724pt" height="486pt" - viewBox="0.00 0.00 1724.00 486.31" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> -<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 482.309)"> +<svg width="1724pt" height="503pt" + viewBox="0.00 0.00 1724.00 502.55" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink"> +<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 498.55)"> <title>Ironic states</title> -<polygon fill="white" stroke="white" points="-4,5 -4,-482.309 1721,-482.309 1721,5 -4,5"/> +<polygon fill="white" stroke="none" points="-4,4 -4,-498.55 1720,-498.55 1720,4 -4,4"/> <!-- enroll --> <g id="node1" class="node"><title>enroll</title> -<ellipse fill="none" stroke="black" stroke-width="1.7" cx="27" cy="-136.309" rx="27" ry="18"/> -<text text-anchor="middle" x="27" y="-133.509" font-family="Times,serif" font-size="11.00">enroll</text> +<ellipse fill="none" stroke="black" stroke-width="1.7" cx="27" cy="-187.55" rx="27" ry="18"/> +<text text-anchor="middle" x="27" y="-184.75" font-family="Times,serif" font-size="11.00">enroll</text> </g> <!-- verifying --> <g id="node2" class="node"><title>verifying</title> -<ellipse fill="none" stroke="black" cx="210" cy="-136.309" rx="33.8507" ry="18"/> -<text text-anchor="middle" x="210" y="-133.509" font-family="Times,serif" font-size="11.00" fill="gray">verifying</text> +<ellipse fill="none" stroke="black" cx="210" cy="-187.55" rx="33.8507" ry="18"/> +<text text-anchor="middle" x="210" y="-184.75" font-family="Times,serif" font-size="11.00" fill="gray">verifying</text> </g> <!-- enroll->verifying --> <g id="edge1" class="edge"><title>enroll->verifying</title> -<path fill="none" stroke="black" d="M54.319,-136.309C83.5522,-136.309 131.193,-136.309 165.889,-136.309"/> -<polygon fill="black" stroke="black" points="166.207,-139.81 176.207,-136.309 166.207,-132.81 166.207,-139.81"/> -<text text-anchor="middle" x="115" y="-139.709" font-family="Times,serif" font-size="12.00">manage (via API)</text> +<path fill="none" stroke="black" d="M54.319,-187.55C83.5522,-187.55 131.193,-187.55 165.889,-187.55"/> +<polygon fill="black" stroke="black" points="166.207,-191.05 176.207,-187.55 166.207,-184.05 166.207,-191.05"/> +<text text-anchor="middle" x="115" y="-190.95" font-family="Times,serif" font-size="12.00">manage (via API)</text> +</g> +<!-- verifying->enroll --> +<g id="edge13" class="edge"><title>verifying->enroll</title> +<path fill="none" stroke="black" d="M182.674,-176.984C174.862,-174.341 166.199,-171.881 158,-170.55 120.271,-164.428 109.575,-163.544 72,-170.55 67.7433,-171.344 63.368,-172.535 59.1046,-173.925"/> +<polygon fill="black" stroke="black" points="57.6647,-170.725 49.4683,-177.438 60.0622,-177.301 57.6647,-170.725"/> +<text text-anchor="middle" x="115" y="-173.95" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> </g> <!-- manageable --> <g id="node3" class="node"><title>manageable</title> -<ellipse fill="none" stroke="black" stroke-width="1.7" cx="346" cy="-136.309" rx="42.1875" ry="18"/> -<text text-anchor="middle" x="346" y="-133.509" font-family="Times,serif" font-size="11.00">manageable</text> +<ellipse fill="none" stroke="black" stroke-width="1.7" cx="346" cy="-187.55" rx="42.1875" ry="18"/> +<text text-anchor="middle" x="346" y="-184.75" font-family="Times,serif" font-size="11.00">manageable</text> </g> <!-- verifying->manageable --> -<g id="edge11" class="edge"><title>verifying->manageable</title> -<path fill="none" stroke="black" d="M243.78,-136.309C258.666,-136.309 276.633,-136.309 293.273,-136.309"/> -<polygon fill="black" stroke="black" points="293.422,-139.81 303.422,-136.309 293.422,-132.81 293.422,-139.81"/> -<text text-anchor="middle" x="274" y="-139.709" font-family="Times,serif" font-size="12.00" fill="gray">done</text> -</g> -<!-- verifying->enroll --> -<g id="edge12" class="edge"><title>verifying->enroll</title> -<path fill="none" stroke="black" d="M182.674,-125.744C174.862,-123.1 166.199,-120.64 158,-119.309 120.271,-113.187 109.575,-112.303 72,-119.309 67.7433,-120.103 63.368,-121.294 59.1046,-122.685"/> -<polygon fill="black" stroke="black" points="57.6647,-119.484 49.4683,-126.198 60.0622,-126.061 57.6647,-119.484"/> -<text text-anchor="middle" x="115" y="-122.709" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> +<g id="edge12" class="edge"><title>verifying->manageable</title> +<path fill="none" stroke="black" d="M243.78,-187.55C258.666,-187.55 276.633,-187.55 293.273,-187.55"/> +<polygon fill="black" stroke="black" points="293.422,-191.05 303.422,-187.55 293.422,-184.05 293.422,-191.05"/> +<text text-anchor="middle" x="274" y="-190.95" font-family="Times,serif" font-size="12.00" fill="gray">done</text> </g> <!-- cleaning --> <g id="node4" class="node"><title>cleaning</title> -<ellipse fill="none" stroke="black" cx="551" cy="-210.309" rx="32.4445" ry="18"/> -<text text-anchor="middle" x="551" y="-207.509" font-family="Times,serif" font-size="11.00" fill="gray">cleaning</text> +<ellipse fill="none" stroke="black" cx="551" cy="-212.55" rx="32.4445" ry="18"/> +<text text-anchor="middle" x="551" y="-209.75" font-family="Times,serif" font-size="11.00" fill="gray">cleaning</text> </g> <!-- manageable->cleaning --> <g id="edge2" class="edge"><title>manageable->cleaning</title> -<path fill="none" stroke="black" d="M356.809,-153.884C366.852,-169.946 384.075,-192.754 406,-203.309 438.278,-218.849 479.474,-219.201 509.527,-216.49"/> -<polygon fill="black" stroke="black" points="510.082,-219.951 519.66,-215.423 509.349,-212.99 510.082,-219.951"/> -<text text-anchor="middle" x="448" y="-220.709" font-family="Times,serif" font-size="12.00">provide (via API)</text> +<path fill="none" stroke="black" d="M368.881,-202.811C379.556,-209.344 392.898,-216.226 406,-219.55 440.464,-228.294 481.019,-225.179 510.294,-220.673"/> +<polygon fill="black" stroke="black" points="511.247,-224.063 520.537,-218.969 510.098,-217.158 511.247,-224.063"/> +<text text-anchor="middle" x="448" y="-227.95" font-family="Times,serif" font-size="12.00">provide (via API)</text> </g> <!-- manageable->cleaning --> <g id="edge3" class="edge"><title>manageable->cleaning</title> -<path fill="none" stroke="black" d="M371.314,-150.921C381.713,-156.68 394.166,-162.948 406,-167.309 441.915,-180.546 453.45,-174.944 490,-186.309 498.393,-188.919 507.268,-192.179 515.545,-195.447"/> -<polygon fill="black" stroke="black" points="514.338,-198.735 524.92,-199.249 516.968,-192.248 514.338,-198.735"/> -<text text-anchor="middle" x="448" y="-189.709" font-family="Times,serif" font-size="12.00">clean (via API)</text> +<path fill="none" stroke="black" d="M387.452,-191.134C416.131,-193.836 455.493,-197.869 490,-202.55 496.387,-203.416 503.138,-204.438 509.718,-205.492"/> +<polygon fill="black" stroke="black" points="509.403,-208.987 519.838,-207.156 510.539,-202.08 509.403,-208.987"/> +<text text-anchor="middle" x="448" y="-205.95" font-family="Times,serif" font-size="12.00">clean (via API)</text> </g> <!-- inspecting --> <g id="node5" class="node"><title>inspecting</title> -<ellipse fill="none" stroke="black" cx="551" cy="-23.3094" rx="37.0671" ry="18"/> -<text text-anchor="middle" x="551" y="-20.5094" font-family="Times,serif" font-size="11.00" fill="gray">inspecting</text> +<ellipse fill="none" stroke="black" cx="551" cy="-25.55" rx="37.0671" ry="18"/> +<text text-anchor="middle" x="551" y="-22.75" font-family="Times,serif" font-size="11.00" fill="gray">inspecting</text> </g> <!-- manageable->inspecting --> <g id="edge4" class="edge"><title>manageable->inspecting</title> -<path fill="none" stroke="black" d="M354.248,-118.338C363.276,-98.3835 380.665,-66.8151 406,-50.3094 421.906,-39.947 468.219,-32.4895 504.072,-28.0722"/> -<polygon fill="black" stroke="black" points="504.827,-31.5072 514.345,-26.853 504.002,-24.556 504.827,-31.5072"/> -<text text-anchor="middle" x="448" y="-53.7094" font-family="Times,serif" font-size="12.00">inspect (via API)</text> +<path fill="none" stroke="black" d="M349.471,-169.433C354.494,-140.014 368.771,-82.1703 406,-52.55 420.855,-40.731 467.343,-33.5154 503.525,-29.5531"/> +<polygon fill="black" stroke="black" points="504.314,-32.99 513.898,-28.4735 503.589,-26.0276 504.314,-32.99"/> +<text text-anchor="middle" x="448" y="-55.95" font-family="Times,serif" font-size="12.00">inspect (via API)</text> +</g> +<!-- adopting --> +<g id="node6" class="node"><title>adopting</title> +<ellipse fill="none" stroke="black" cx="551" cy="-412.55" rx="32.4445" ry="18"/> +<text text-anchor="middle" x="551" y="-409.75" font-family="Times,serif" font-size="11.00" fill="gray">adopting</text> +</g> +<!-- manageable->adopting --> +<g id="edge5" class="edge"><title>manageable->adopting</title> +<path fill="none" stroke="black" d="M347.291,-205.559C348.896,-242.554 357.935,-327.494 406,-373.55 433.219,-399.631 476.034,-408.662 507.913,-411.6"/> +<polygon fill="black" stroke="black" points="508.033,-415.119 518.267,-412.379 508.557,-408.139 508.033,-415.119"/> +<text text-anchor="middle" x="448" y="-411.95" font-family="Times,serif" font-size="12.00">adopt (via API)</text> +</g> +<!-- cleaning->manageable --> +<g id="edge26" class="edge"><title>cleaning->manageable</title> +<path fill="none" stroke="black" d="M531.764,-198.035C520.454,-189.983 505.219,-180.765 490,-176.55 458.156,-167.73 420.901,-171.01 392.401,-176.215"/> +<polygon fill="black" stroke="black" points="391.458,-172.833 382.324,-178.201 392.812,-179.701 391.458,-172.833"/> +<text text-anchor="middle" x="448" y="-179.95" font-family="Times,serif" font-size="12.00" fill="gray">manage</text> </g> <!-- available --> -<g id="node6" class="node"><title>available</title> -<ellipse fill="none" stroke="black" stroke-width="1.7" cx="763" cy="-319.309" rx="34.054" ry="18"/> -<text text-anchor="middle" x="763" y="-316.509" font-family="Times,serif" font-size="11.00">available</text> +<g id="node7" class="node"><title>available</title> +<ellipse fill="none" stroke="black" stroke-width="1.7" cx="763" cy="-313.55" rx="34.054" ry="18"/> +<text text-anchor="middle" x="763" y="-310.75" font-family="Times,serif" font-size="11.00">available</text> </g> <!-- cleaning->available --> -<g id="edge22" class="edge"><title>cleaning->available</title> -<path fill="none" stroke="black" d="M564.556,-226.687C575.644,-240.05 592.974,-258.437 612,-269.309 646.352,-288.94 660.573,-279.478 698,-292.309 707.652,-295.618 717.915,-299.689 727.33,-303.657"/> -<polygon fill="black" stroke="black" points="726.205,-306.983 736.775,-307.719 728.971,-300.553 726.205,-306.983"/> -<text text-anchor="middle" x="655" y="-295.709" font-family="Times,serif" font-size="12.00" fill="gray">done</text> +<g id="edge23" class="edge"><title>cleaning->available</title> +<path fill="none" stroke="black" d="M566.869,-228.52C578.227,-239.83 594.807,-254.535 612,-263.55 647.041,-281.923 660.573,-273.719 698,-286.55 707.652,-289.859 717.915,-293.93 727.33,-297.898"/> +<polygon fill="black" stroke="black" points="726.205,-301.224 736.775,-301.959 728.971,-294.793 726.205,-301.224"/> +<text text-anchor="middle" x="655" y="-289.95" font-family="Times,serif" font-size="12.00" fill="gray">done</text> </g> <!-- clean failed --> -<g id="node13" class="node"><title>clean failed</title> -<ellipse fill="none" stroke="black" cx="964" cy="-216.309" rx="41.4846" ry="18"/> -<text text-anchor="middle" x="964" y="-213.509" font-family="Times,serif" font-size="11.00" fill="red">clean failed</text> +<g id="node14" class="node"><title>clean failed</title> +<ellipse fill="none" stroke="black" cx="964" cy="-214.55" rx="41.4846" ry="18"/> +<text text-anchor="middle" x="964" y="-211.75" font-family="Times,serif" font-size="11.00" fill="red">clean failed</text> </g> <!-- cleaning->clean failed --> -<g id="edge23" class="edge"><title>cleaning->clean failed</title> -<path fill="none" stroke="black" d="M579.519,-218.962C589.632,-221.792 601.241,-224.636 612,-226.309 740.242,-246.259 775.134,-244.727 904,-229.309 908.516,-228.769 913.194,-228.039 917.851,-227.199"/> -<polygon fill="black" stroke="black" points="918.802,-230.579 927.939,-225.214 917.451,-223.71 918.802,-230.579"/> -<text text-anchor="middle" x="763" y="-243.709" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> +<g id="edge24" class="edge"><title>cleaning->clean failed</title> +<path fill="none" stroke="black" d="M580.467,-220.482C590.368,-222.895 601.598,-225.251 612,-226.55 740.778,-242.631 775.06,-242.281 904,-227.55 908.518,-227.034 913.199,-226.32 917.857,-225.489"/> +<polygon fill="black" stroke="black" points="918.803,-228.87 927.946,-223.517 917.46,-222 918.803,-228.87"/> +<text text-anchor="middle" x="763" y="-241.95" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> </g> <!-- clean wait --> -<g id="node14" class="node"><title>clean wait</title> -<ellipse fill="none" stroke="black" cx="763" cy="-204.309" rx="37.7689" ry="18"/> -<text text-anchor="middle" x="763" y="-201.509" font-family="Times,serif" font-size="11.00" fill="gray">clean wait</text> +<g id="node15" class="node"><title>clean wait</title> +<ellipse fill="none" stroke="black" cx="763" cy="-201.55" rx="37.7689" ry="18"/> +<text text-anchor="middle" x="763" y="-198.75" font-family="Times,serif" font-size="11.00" fill="gray">clean wait</text> </g> <!-- cleaning->clean wait --> -<g id="edge24" class="edge"><title>cleaning->clean wait</title> -<path fill="none" stroke="black" d="M583.619,-209.899C613.196,-209.459 658.599,-208.637 698,-207.309 703.489,-207.124 709.241,-206.899 714.951,-206.656"/> -<polygon fill="black" stroke="black" points="715.283,-210.145 725.118,-206.204 714.972,-203.152 715.283,-210.145"/> -<text text-anchor="middle" x="655" y="-212.709" font-family="Times,serif" font-size="12.00" fill="gray">wait</text> -</g> -<!-- cleaning->manageable --> -<g id="edge25" class="edge"><title>cleaning->manageable</title> -<path fill="none" stroke="black" d="M538.574,-193.664C527.829,-179.491 510.378,-159.94 490,-150.309 461.377,-136.782 425.988,-133.226 397.577,-133.08"/> -<polygon fill="black" stroke="black" points="397.42,-129.581 387.451,-133.168 397.481,-136.581 397.42,-129.581"/> -<text text-anchor="middle" x="448" y="-153.709" font-family="Times,serif" font-size="12.00" fill="gray">manage</text> +<g id="edge25" class="edge"><title>cleaning->clean wait</title> +<path fill="none" stroke="black" d="M583.439,-210.903C618.279,-209.078 674.834,-206.116 714.953,-204.014"/> +<polygon fill="black" stroke="black" points="715.228,-207.505 725.031,-203.486 714.862,-200.514 715.228,-207.505"/> +<text text-anchor="middle" x="655" y="-212.95" font-family="Times,serif" font-size="12.00" fill="gray">wait</text> </g> <!-- inspecting->manageable --> -<g id="edge32" class="edge"><title>inspecting->manageable</title> -<path fill="none" stroke="black" d="M521.615,-12.0674C490.699,-1.80437 440.995,8.71678 406,-13.3094 373.153,-33.9839 358.182,-78.5917 351.661,-108.02"/> -<polygon fill="black" stroke="black" points="348.174,-107.607 349.624,-118.102 355.035,-108.994 348.174,-107.607"/> -<text text-anchor="middle" x="448" y="-16.7094" font-family="Times,serif" font-size="12.00" fill="gray">done</text> +<g id="edge33" class="edge"><title>inspecting->manageable</title> +<path fill="none" stroke="black" d="M522.516,-13.7533C491.262,-2.36256 440.155,9.88802 406,-14.55 359.666,-47.703 349.303,-119.445 347.246,-159.343"/> +<polygon fill="black" stroke="black" points="343.743,-159.336 346.868,-169.459 350.739,-159.597 343.743,-159.336"/> +<text text-anchor="middle" x="448" y="-17.95" font-family="Times,serif" font-size="12.00" fill="gray">done</text> </g> <!-- inspect failed --> -<g id="node15" class="node"><title>inspect failed</title> -<ellipse fill="none" stroke="black" cx="763" cy="-51.3094" rx="46.1069" ry="18"/> -<text text-anchor="middle" x="763" y="-48.5094" font-family="Times,serif" font-size="11.00" fill="red">inspect failed</text> +<g id="node16" class="node"><title>inspect failed</title> +<ellipse fill="none" stroke="black" cx="763" cy="-53.55" rx="46.1069" ry="18"/> +<text text-anchor="middle" x="763" y="-50.75" font-family="Times,serif" font-size="11.00" fill="red">inspect failed</text> </g> <!-- inspecting->inspect failed --> -<g id="edge33" class="edge"><title>inspecting->inspect failed</title> -<path fill="none" stroke="black" d="M582.246,-33.1699C591.683,-35.8963 602.191,-38.5786 612,-40.3094 643.032,-45.7849 678.14,-48.5425 706.585,-49.9279"/> -<polygon fill="black" stroke="black" points="706.598,-53.4318 716.744,-50.3787 706.909,-46.4387 706.598,-53.4318"/> -<text text-anchor="middle" x="655" y="-53.7094" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> +<g id="edge34" class="edge"><title>inspecting->inspect failed</title> +<path fill="none" stroke="black" d="M582.246,-35.4105C591.683,-38.1369 602.191,-40.8192 612,-42.55 643.032,-48.0255 678.14,-50.7831 706.585,-52.1685"/> +<polygon fill="black" stroke="black" points="706.598,-55.6724 716.744,-52.6193 706.909,-48.6793 706.598,-55.6724"/> +<text text-anchor="middle" x="655" y="-55.95" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> +</g> +<!-- active --> +<g id="node9" class="node"><title>active</title> +<ellipse fill="none" stroke="black" stroke-width="1.7" cx="1172" cy="-410.55" rx="27" ry="18"/> +<text text-anchor="middle" x="1172" y="-407.75" font-family="Times,serif" font-size="11.00">active</text> +</g> +<!-- adopting->active --> +<g id="edge37" class="edge"><title>adopting->active</title> +<path fill="none" stroke="black" d="M582.779,-408.379C592.145,-407.267 602.478,-406.195 612,-405.55 658.125,-402.423 669.776,-404.37 716,-403.55 852.885,-401.123 887.098,-399.661 1024,-398.55 1060.44,-398.254 1069.71,-395.228 1106,-398.55 1115.82,-399.449 1126.37,-401.096 1136.05,-402.888"/> +<polygon fill="black" stroke="black" points="1135.66,-406.377 1146.15,-404.861 1137,-399.508 1135.66,-406.377"/> +<text text-anchor="middle" x="866" y="-403.95" font-family="Times,serif" font-size="12.00" fill="gray">done</text> +</g> +<!-- adopt failed --> +<g id="node17" class="node"><title>adopt failed</title> +<ellipse fill="none" stroke="black" cx="763" cy="-430.55" rx="41.4846" ry="18"/> +<text text-anchor="middle" x="763" y="-427.75" font-family="Times,serif" font-size="11.00" fill="red">adopt failed</text> +</g> +<!-- adopting->adopt failed --> +<g id="edge38" class="edge"><title>adopting->adopt failed</title> +<path fill="none" stroke="black" d="M578.165,-422.754C588.531,-426.286 600.662,-429.798 612,-431.55 645.152,-436.674 682.919,-436.388 712.263,-434.839"/> +<polygon fill="black" stroke="black" points="712.495,-438.332 722.27,-434.246 712.081,-431.344 712.495,-438.332"/> +<text text-anchor="middle" x="655" y="-438.95" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> +</g> +<!-- available->manageable --> +<g id="edge7" class="edge"><title>available->manageable</title> +<path fill="none" stroke="black" d="M728.535,-313.4C662.605,-311.739 513.031,-301.112 406,-241.55 392.136,-233.835 378.894,-222.248 368.48,-211.784"/> +<polygon fill="black" stroke="black" points="370.878,-209.227 361.43,-204.434 365.826,-214.073 370.878,-209.227"/> +<text text-anchor="middle" x="551" y="-303.95" font-family="Times,serif" font-size="12.00">manage (via API)</text> </g> <!-- deploying --> -<g id="node7" class="node"><title>deploying</title> -<ellipse fill="none" stroke="black" cx="964" cy="-329.309" rx="35.4579" ry="18"/> -<text text-anchor="middle" x="964" y="-326.509" font-family="Times,serif" font-size="11.00" fill="gray">deploying</text> +<g id="node8" class="node"><title>deploying</title> +<ellipse fill="none" stroke="black" cx="964" cy="-320.55" rx="35.4579" ry="18"/> +<text text-anchor="middle" x="964" y="-317.75" font-family="Times,serif" font-size="11.00" fill="gray">deploying</text> </g> <!-- available->deploying --> -<g id="edge5" class="edge"><title>available->deploying</title> -<path fill="none" stroke="black" d="M797.308,-320.983C830.44,-322.648 881.492,-325.214 918.17,-327.057"/> -<polygon fill="black" stroke="black" points="918.418,-330.573 928.581,-327.58 918.769,-323.582 918.418,-330.573"/> -<text text-anchor="middle" x="866" y="-328.709" font-family="Times,serif" font-size="12.00">active (via API)</text> +<g id="edge6" class="edge"><title>available->deploying</title> +<path fill="none" stroke="black" d="M797.308,-314.722C830.44,-315.887 881.492,-317.683 918.17,-318.973"/> +<polygon fill="black" stroke="black" points="918.464,-322.486 928.581,-319.339 918.71,-315.49 918.464,-322.486"/> +<text text-anchor="middle" x="866" y="-320.95" font-family="Times,serif" font-size="12.00">active (via API)</text> </g> -<!-- available->manageable --> -<g id="edge6" class="edge"><title>available->manageable</title> -<path fill="none" stroke="black" d="M728.715,-320.4C660.462,-320.882 502.935,-312.76 406,-234.309 382.975,-215.675 366.751,-185.486 357.221,-163.534"/> -<polygon fill="black" stroke="black" points="360.373,-161.997 353.312,-154.098 353.905,-164.676 360.373,-161.997"/> -<text text-anchor="middle" x="551" y="-312.709" font-family="Times,serif" font-size="12.00">manage (via API)</text> +<!-- deploying->active --> +<g id="edge16" class="edge"><title>deploying->active</title> +<path fill="none" stroke="black" d="M984.128,-335.45C995.3,-343.648 1009.93,-353.547 1024,-360.55 1060.97,-378.95 1106.33,-393.232 1136.8,-401.737"/> +<polygon fill="black" stroke="black" points="1135.96,-405.136 1146.53,-404.397 1137.81,-398.383 1135.96,-405.136"/> +<text text-anchor="middle" x="1065" y="-393.95" font-family="Times,serif" font-size="12.00" fill="gray">done</text> </g> <!-- deploy failed --> -<g id="node11" class="node"><title>deploy failed</title> -<ellipse fill="none" stroke="black" cx="1317" cy="-286.309" rx="44.498" ry="18"/> -<text text-anchor="middle" x="1317" y="-283.509" font-family="Times,serif" font-size="11.00" fill="red">deploy failed</text> +<g id="node12" class="node"><title>deploy failed</title> +<ellipse fill="none" stroke="black" cx="1317" cy="-283.55" rx="44.498" ry="18"/> +<text text-anchor="middle" x="1317" y="-280.75" font-family="Times,serif" font-size="11.00" fill="red">deploy failed</text> </g> <!-- deploying->deploy failed --> -<g id="edge13" class="edge"><title>deploying->deploy failed</title> -<path fill="none" stroke="black" d="M996.718,-322.089C1005.51,-320.297 1015.1,-318.539 1024,-317.309 1125.54,-303.289 1152.62,-315.417 1254,-300.309 1258.8,-299.595 1263.77,-298.713 1268.72,-297.743"/> -<polygon fill="black" stroke="black" points="1269.71,-301.112 1278.79,-295.651 1268.29,-294.258 1269.71,-301.112"/> -<text text-anchor="middle" x="1172" y="-312.709" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> +<g id="edge14" class="edge"><title>deploying->deploy failed</title> +<path fill="none" stroke="black" d="M996.054,-312.65C1005.02,-310.659 1014.85,-308.748 1024,-307.55 1125.45,-294.27 1152.58,-311.095 1254,-297.55 1258.81,-296.908 1263.79,-296.074 1268.75,-295.133"/> +<polygon fill="black" stroke="black" points="1269.72,-298.506 1278.82,-293.078 1268.32,-291.648 1269.72,-298.506"/> +<text text-anchor="middle" x="1172" y="-305.95" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> </g> <!-- wait call-back --> -<g id="node12" class="node"><title>wait call-back</title> -<ellipse fill="none" stroke="black" cx="1172" cy="-351.309" rx="48.2143" ry="18"/> -<text text-anchor="middle" x="1172" y="-348.509" font-family="Times,serif" font-size="11.00" fill="gray">wait call-back</text> +<g id="node13" class="node"><title>wait call-back</title> +<ellipse fill="none" stroke="black" cx="1172" cy="-348.55" rx="48.2143" ry="18"/> +<text text-anchor="middle" x="1172" y="-345.75" font-family="Times,serif" font-size="11.00" fill="gray">wait call-back</text> </g> <!-- deploying->wait call-back --> -<g id="edge14" class="edge"><title>deploying->wait call-back</title> -<path fill="none" stroke="black" d="M995.727,-337.697C1004.77,-339.867 1014.73,-341.968 1024,-343.309 1053.38,-347.56 1086.37,-349.601 1113.63,-350.561"/> -<polygon fill="black" stroke="black" points="1113.68,-354.064 1123.78,-350.879 1113.9,-347.068 1113.68,-354.064"/> -<text text-anchor="middle" x="1065" y="-352.709" font-family="Times,serif" font-size="12.00" fill="gray">wait</text> -</g> -<!-- active --> -<g id="node8" class="node"><title>active</title> -<ellipse fill="none" stroke="black" stroke-width="1.7" cx="1172" cy="-405.309" rx="27" ry="18"/> -<text text-anchor="middle" x="1172" y="-402.509" font-family="Times,serif" font-size="11.00">active</text> -</g> -<!-- deploying->active --> -<g id="edge15" class="edge"><title>deploying->active</title> -<path fill="none" stroke="black" d="M984.914,-343.913C995.976,-351.47 1010.25,-360.337 1024,-366.309 1060.5,-382.163 1104.98,-392.868 1135.39,-398.991"/> -<polygon fill="black" stroke="black" points="1135.04,-402.488 1145.52,-400.967 1136.38,-395.618 1135.04,-402.488"/> -<text text-anchor="middle" x="1065" y="-394.709" font-family="Times,serif" font-size="12.00" fill="gray">done</text> +<g id="edge15" class="edge"><title>deploying->wait call-back</title> +<path fill="none" stroke="black" d="M995.764,-328.698C1004.81,-330.863 1014.76,-333.021 1024,-334.55 1053.6,-339.448 1086.87,-342.794 1114.25,-344.983"/> +<polygon fill="black" stroke="black" points="1114.2,-348.489 1124.44,-345.766 1114.74,-341.51 1114.2,-348.489"/> +<text text-anchor="middle" x="1065" y="-346.95" font-family="Times,serif" font-size="12.00" fill="gray">wait</text> </g> <!-- active->deploying --> -<g id="edge7" class="edge"><title>active->deploying</title> -<path fill="none" stroke="black" d="M1148.06,-413.944C1117.76,-423.837 1063.61,-436.169 1024,-415.309 1000.6,-402.983 984.689,-376.717 975.345,-356.437"/> -<polygon fill="black" stroke="black" points="978.466,-354.835 971.281,-347.049 972.042,-357.616 978.466,-354.835"/> -<text text-anchor="middle" x="1065" y="-429.709" font-family="Times,serif" font-size="12.00">rebuild (via API)</text> +<g id="edge8" class="edge"><title>active->deploying</title> +<path fill="none" stroke="black" d="M1150.34,-421.324C1120.29,-435.36 1063.77,-455.453 1024,-431.55 994.275,-413.687 978.719,-375.231 971.196,-348.631"/> +<polygon fill="black" stroke="black" points="974.525,-347.53 968.61,-338.74 967.753,-349.3 974.525,-347.53"/> +<text text-anchor="middle" x="1065" y="-445.95" font-family="Times,serif" font-size="12.00">rebuild (via API)</text> </g> <!-- deleting --> -<g id="node9" class="node"><title>deleting</title> -<ellipse fill="none" stroke="black" cx="1512" cy="-351.309" rx="31.0408" ry="18"/> -<text text-anchor="middle" x="1512" y="-348.509" font-family="Times,serif" font-size="11.00" fill="gray">deleting</text> +<g id="node10" class="node"><title>deleting</title> +<ellipse fill="none" stroke="black" cx="1512" cy="-348.55" rx="31.0408" ry="18"/> +<text text-anchor="middle" x="1512" y="-345.75" font-family="Times,serif" font-size="11.00" fill="gray">deleting</text> </g> <!-- active->deleting --> -<g id="edge8" class="edge"><title>active->deleting</title> -<path fill="none" stroke="black" d="M1199.12,-402.939C1250.06,-398.041 1366.1,-385.541 1462,-365.309 1466.09,-364.446 1470.34,-363.425 1474.55,-362.333"/> -<polygon fill="black" stroke="black" points="1475.49,-365.706 1484.21,-359.699 1473.64,-358.953 1475.49,-365.706"/> -<text text-anchor="middle" x="1317" y="-397.709" font-family="Times,serif" font-size="12.00">deleted (via API)</text> +<g id="edge9" class="edge"><title>active->deleting</title> +<path fill="none" stroke="black" d="M1198.7,-407.281C1249.41,-400.613 1365.74,-384.222 1462,-362.55 1466.08,-361.632 1470.32,-360.576 1474.53,-359.464"/> +<polygon fill="black" stroke="black" points="1475.47,-362.834 1484.18,-356.807 1473.61,-356.085 1475.47,-362.834"/> +<text text-anchor="middle" x="1317" y="-399.95" font-family="Times,serif" font-size="12.00">deleted (via API)</text> +</g> +<!-- deleting->cleaning --> +<g id="edge32" class="edge"><title>deleting->cleaning</title> +<path fill="none" stroke="black" d="M1503.74,-331.082C1482.69,-283.573 1416.8,-155.55 1318,-155.55 762,-155.55 762,-155.55 762,-155.55 694.339,-155.55 676.135,-159.993 612,-181.55 601.998,-184.912 591.612,-189.742 582.35,-194.591"/> +<polygon fill="black" stroke="black" points="580.63,-191.542 573.517,-199.394 583.974,-197.692 580.63,-191.542"/> +<text text-anchor="middle" x="1065" y="-158.95" font-family="Times,serif" font-size="12.00" fill="gray">clean</text> </g> <!-- error --> -<g id="node10" class="node"><title>error</title> -<ellipse fill="none" stroke="black" stroke-width="1.7" cx="1689" cy="-387.309" rx="27" ry="18"/> -<text text-anchor="middle" x="1689" y="-384.509" font-family="Times,serif" font-size="11.00" fill="red">error</text> +<g id="node11" class="node"><title>error</title> +<ellipse fill="none" stroke="black" stroke-width="1.7" cx="1689" cy="-384.55" rx="27" ry="18"/> +<text text-anchor="middle" x="1689" y="-381.75" font-family="Times,serif" font-size="11.00" fill="red">error</text> </g> <!-- deleting->error --> -<g id="edge30" class="edge"><title>deleting->error</title> -<path fill="none" stroke="black" d="M1541.12,-344.722C1568.37,-339.691 1610.39,-335.56 1644,-348.309 1652.92,-351.693 1661.19,-357.802 1668.09,-364.196"/> -<polygon fill="black" stroke="black" points="1666,-367.058 1675.53,-371.687 1670.97,-362.125 1666,-367.058"/> -<text text-anchor="middle" x="1603" y="-351.709" font-family="Times,serif" font-size="12.00" fill="gray">error</text> -</g> -<!-- deleting->cleaning --> -<g id="edge31" class="edge"><title>deleting->cleaning</title> -<path fill="none" stroke="black" d="M1503.74,-333.842C1482.69,-286.333 1416.8,-158.309 1318,-158.309 762,-158.309 762,-158.309 762,-158.309 694.485,-158.309 676.546,-162.51 612,-182.309 602.638,-185.181 592.863,-189.242 583.986,-193.383"/> -<polygon fill="black" stroke="black" points="582.238,-190.341 574.769,-197.855 585.293,-196.639 582.238,-190.341"/> -<text text-anchor="middle" x="1065" y="-161.709" font-family="Times,serif" font-size="12.00" fill="gray">clean</text> +<g id="edge31" class="edge"><title>deleting->error</title> +<path fill="none" stroke="black" d="M1541.12,-341.962C1568.37,-336.931 1610.39,-332.8 1644,-345.55 1652.92,-348.933 1661.19,-355.042 1668.09,-361.437"/> +<polygon fill="black" stroke="black" points="1666,-364.298 1675.53,-368.928 1670.97,-359.366 1666,-364.298"/> +<text text-anchor="middle" x="1603" y="-348.95" font-family="Times,serif" font-size="12.00" fill="gray">error</text> </g> <!-- error->deploying --> -<g id="edge9" class="edge"><title>error->deploying</title> -<path fill="none" stroke="black" d="M1670.75,-400.896C1640.6,-423.243 1575.53,-465.309 1513,-465.309 1171,-465.309 1171,-465.309 1171,-465.309 1104.94,-465.309 1077.73,-481.741 1024,-443.309 995.197,-422.708 979.419,-383.758 971.593,-357.176"/> -<polygon fill="black" stroke="black" points="974.909,-356.032 968.886,-347.316 968.159,-357.886 974.909,-356.032"/> -<text text-anchor="middle" x="1317" y="-468.709" font-family="Times,serif" font-size="12.00">rebuild (via API)</text> +<g id="edge10" class="edge"><title>error->deploying</title> +<path fill="none" stroke="black" d="M1673.16,-399.706C1644.78,-426.988 1579.69,-481.55 1513,-481.55 1171,-481.55 1171,-481.55 1171,-481.55 1104.94,-481.55 1076.23,-499.999 1024,-459.55 989.177,-432.581 974.65,-381.143 968.777,-348.968"/> +<polygon fill="black" stroke="black" points="972.19,-348.151 967.101,-338.858 965.284,-349.296 972.19,-348.151"/> +<text text-anchor="middle" x="1317" y="-484.95" font-family="Times,serif" font-size="12.00">rebuild (via API)</text> </g> <!-- error->deleting --> -<g id="edge10" class="edge"><title>error->deleting</title> -<path fill="none" stroke="black" d="M1662.26,-383.629C1636.68,-379.781 1596.41,-373.207 1562,-365.309 1557.93,-364.374 1553.69,-363.308 1549.48,-362.189"/> -<polygon fill="black" stroke="black" points="1550.4,-358.811 1539.83,-359.525 1548.53,-365.559 1550.4,-358.811"/> -<text text-anchor="middle" x="1603" y="-383.709" font-family="Times,serif" font-size="12.00">deleted (via API)</text> +<g id="edge11" class="edge"><title>error->deleting</title> +<path fill="none" stroke="black" d="M1662.26,-380.869C1636.68,-377.021 1596.41,-370.447 1562,-362.55 1557.93,-361.615 1553.69,-360.548 1549.48,-359.43"/> +<polygon fill="black" stroke="black" points="1550.4,-356.052 1539.83,-356.766 1548.53,-362.8 1550.4,-356.052"/> +<text text-anchor="middle" x="1603" y="-380.95" font-family="Times,serif" font-size="12.00">deleted (via API)</text> </g> <!-- deploy failed->deploying --> -<g id="edge19" class="edge"><title>deploy failed->deploying</title> -<path fill="none" stroke="black" d="M1272.94,-283.399C1215.09,-280.765 1109.63,-280.481 1024,-305.309 1016.23,-307.561 1008.09,-310.485 1000.42,-313.513"/> -<polygon fill="black" stroke="black" points="998.946,-310.334 991.018,-317.361 1001.6,-316.812 998.946,-310.334"/> -<text text-anchor="middle" x="1172" y="-289.709" font-family="Times,serif" font-size="12.00">rebuild (via API)</text> +<g id="edge20" class="edge"><title>deploy failed->deploying</title> +<path fill="none" stroke="black" d="M1273.1,-280.157C1215.43,-276.818 1110.18,-275.048 1024,-297.55 1016.3,-299.561 1008.23,-302.276 1000.62,-305.134"/> +<polygon fill="black" stroke="black" points="999.326,-301.882 991.292,-308.789 1001.88,-308.4 999.326,-301.882"/> +<text text-anchor="middle" x="1172" y="-283.95" font-family="Times,serif" font-size="12.00">rebuild (via API)</text> </g> <!-- deploy failed->deploying --> -<g id="edge20" class="edge"><title>deploy failed->deploying</title> -<path fill="none" stroke="black" d="M1280.32,-276.116C1262.38,-271.511 1240.22,-266.596 1220,-264.309 1132.95,-254.462 1103.86,-249.275 1024,-285.309 1011.81,-290.811 999.72,-299.171 989.691,-307.132"/> -<polygon fill="black" stroke="black" points="987.276,-304.587 981.794,-313.653 991.733,-309.985 987.276,-304.587"/> -<text text-anchor="middle" x="1172" y="-267.709" font-family="Times,serif" font-size="12.00">active (via API)</text> +<g id="edge21" class="edge"><title>deploy failed->deploying</title> +<path fill="none" stroke="black" d="M1280.8,-273.016C1262.79,-268.167 1240.42,-262.963 1220,-260.55 1133.13,-250.288 1104.63,-244.616 1024,-278.55 1012.17,-283.53 1000.39,-291.217 990.505,-298.661"/> +<polygon fill="black" stroke="black" points="988.053,-296.136 982.351,-305.066 992.377,-301.641 988.053,-296.136"/> +<text text-anchor="middle" x="1172" y="-263.95" font-family="Times,serif" font-size="12.00">active (via API)</text> </g> <!-- deploy failed->deleting --> -<g id="edge21" class="edge"><title>deploy failed->deleting</title> -<path fill="none" stroke="black" d="M1357.34,-293.952C1386.77,-300.32 1427.67,-310.567 1462,-324.309 1468.42,-326.88 1475.06,-330.111 1481.3,-333.434"/> -<polygon fill="black" stroke="black" points="1479.65,-336.517 1490.09,-338.305 1483.04,-330.395 1479.65,-336.517"/> -<text text-anchor="middle" x="1421" y="-327.709" font-family="Times,serif" font-size="12.00">deleted (via API)</text> +<g id="edge22" class="edge"><title>deploy failed->deleting</title> +<path fill="none" stroke="black" d="M1357.34,-291.193C1386.77,-297.561 1427.67,-307.808 1462,-321.55 1468.42,-324.121 1475.06,-327.351 1481.3,-330.675"/> +<polygon fill="black" stroke="black" points="1479.65,-333.758 1490.09,-335.545 1483.04,-327.636 1479.65,-333.758"/> +<text text-anchor="middle" x="1421" y="-324.95" font-family="Times,serif" font-size="12.00">deleted (via API)</text> </g> <!-- wait call-back->deploying --> -<g id="edge16" class="edge"><title>wait call-back->deploying</title> -<path fill="none" stroke="black" d="M1141.57,-337.365C1130.66,-332.9 1118.01,-328.539 1106,-326.309 1073.77,-320.325 1036.68,-321.421 1008.63,-323.828"/> -<polygon fill="black" stroke="black" points="1007.98,-320.374 998.357,-324.803 1008.64,-327.343 1007.98,-320.374"/> -<text text-anchor="middle" x="1065" y="-329.709" font-family="Times,serif" font-size="12.00" fill="gray">resume</text> -</g> -<!-- wait call-back->deploy failed --> -<g id="edge17" class="edge"><title>wait call-back->deploy failed</title> -<path fill="none" stroke="black" d="M1203.33,-337.525C1224.94,-327.699 1254.18,-314.41 1277.45,-303.832"/> -<polygon fill="black" stroke="black" points="1279.04,-306.954 1286.7,-299.63 1276.14,-300.582 1279.04,-306.954"/> -<text text-anchor="middle" x="1246" y="-323.709" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> +<g id="edge17" class="edge"><title>wait call-back->deploying</title> +<path fill="none" stroke="black" d="M1145.71,-333.256C1134.06,-327.073 1119.78,-320.668 1106,-317.55 1073.78,-310.26 1036.35,-311.488 1008.16,-314.28"/> +<polygon fill="black" stroke="black" points="1007.78,-310.801 998.217,-315.366 1008.54,-317.759 1007.78,-310.801"/> +<text text-anchor="middle" x="1065" y="-320.95" font-family="Times,serif" font-size="12.00" fill="gray">resume</text> </g> <!-- wait call-back->deleting --> -<g id="edge18" class="edge"><title>wait call-back->deleting</title> -<path fill="none" stroke="black" d="M1220.35,-351.309C1286.99,-351.309 1407.31,-351.309 1470.2,-351.309"/> -<polygon fill="black" stroke="black" points="1470.52,-354.81 1480.52,-351.309 1470.52,-347.81 1470.52,-354.81"/> -<text text-anchor="middle" x="1317" y="-354.709" font-family="Times,serif" font-size="12.00">deleted (via API)</text> +<g id="edge19" class="edge"><title>wait call-back->deleting</title> +<path fill="none" stroke="black" d="M1220.35,-348.55C1286.99,-348.55 1407.31,-348.55 1470.2,-348.55"/> +<polygon fill="black" stroke="black" points="1470.52,-352.05 1480.52,-348.55 1470.52,-345.05 1470.52,-352.05"/> +<text text-anchor="middle" x="1317" y="-351.95" font-family="Times,serif" font-size="12.00">deleted (via API)</text> +</g> +<!-- wait call-back->deploy failed --> +<g id="edge18" class="edge"><title>wait call-back->deploy failed</title> +<path fill="none" stroke="black" d="M1203.33,-334.765C1224.94,-324.939 1254.18,-311.651 1277.45,-301.072"/> +<polygon fill="black" stroke="black" points="1279.04,-304.195 1286.7,-296.87 1276.14,-297.822 1279.04,-304.195"/> +<text text-anchor="middle" x="1246" y="-320.95" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> </g> <!-- clean failed->manageable --> -<g id="edge29" class="edge"><title>clean failed->manageable</title> -<path fill="none" stroke="black" d="M940.499,-201.349C929.799,-194.764 916.612,-187.387 904,-182.309 816.505,-147.084 791.626,-142.727 698,-131.309 591.548,-118.328 465.6,-125.555 397.254,-131.343"/> -<polygon fill="black" stroke="black" points="396.853,-127.864 387.195,-132.22 397.461,-134.838 396.853,-127.864"/> -<text text-anchor="middle" x="655" y="-134.709" font-family="Times,serif" font-size="12.00">manage (via API)</text> +<g id="edge30" class="edge"><title>clean failed->manageable</title> +<path fill="none" stroke="black" d="M939.234,-199.96C928.717,-193.956 916.028,-187.308 904,-182.55 815.986,-147.736 791.872,-140.649 698,-128.55 585.203,-114.012 453.168,-150.875 387.862,-172.744"/> +<polygon fill="black" stroke="black" points="386.682,-169.448 378.343,-175.984 388.938,-176.074 386.682,-169.448"/> +<text text-anchor="middle" x="655" y="-131.95" font-family="Times,serif" font-size="12.00">manage (via API)</text> </g> -<!-- clean wait->clean failed --> -<g id="edge26" class="edge"><title>clean wait->clean failed</title> -<path fill="none" stroke="black" d="M800.964,-206.538C832.338,-208.43 877.662,-211.163 912.401,-213.258"/> -<polygon fill="black" stroke="black" points="912.623,-216.778 922.815,-213.886 913.044,-209.791 912.623,-216.778"/> -<text text-anchor="middle" x="866" y="-215.709" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> +<!-- clean wait->cleaning --> +<g id="edge29" class="edge"><title>clean wait->cleaning</title> +<path fill="none" stroke="black" d="M728.957,-193.498C698.382,-187.27 651.785,-180.888 612,-188.55 602.955,-190.292 593.541,-193.385 584.924,-196.783"/> +<polygon fill="black" stroke="black" points="583.465,-193.6 575.596,-200.694 586.172,-200.055 583.465,-193.6"/> +<text text-anchor="middle" x="655" y="-191.95" font-family="Times,serif" font-size="12.00" fill="gray">resume</text> </g> <!-- clean wait->clean failed --> <g id="edge27" class="edge"><title>clean wait->clean failed</title> -<path fill="none" stroke="black" d="M796.273,-195.531C824.845,-189.071 867.549,-182.634 904,-190.309 912.176,-192.031 920.601,-194.959 928.434,-198.244"/> -<polygon fill="black" stroke="black" points="927.082,-201.474 937.639,-202.371 929.945,-195.086 927.082,-201.474"/> -<text text-anchor="middle" x="866" y="-193.709" font-family="Times,serif" font-size="12.00">abort (via API)</text> +<path fill="none" stroke="black" d="M800.964,-203.965C832.338,-206.014 877.662,-208.975 912.401,-211.245"/> +<polygon fill="black" stroke="black" points="912.609,-214.765 922.815,-211.925 913.065,-207.78 912.609,-214.765"/> +<text text-anchor="middle" x="866" y="-213.95" font-family="Times,serif" font-size="12.00" fill="gray">fail</text> </g> -<!-- clean wait->cleaning --> -<g id="edge28" class="edge"><title>clean wait->cleaning</title> -<path fill="none" stroke="black" d="M729.336,-195.745C719.352,-193.512 708.29,-191.422 698,-190.309 659.999,-186.202 649.729,-184.187 612,-190.309 603.755,-191.647 595.112,-193.956 587.036,-196.541"/> -<polygon fill="black" stroke="black" points="585.865,-193.242 577.531,-199.783 588.125,-199.867 585.865,-193.242"/> -<text text-anchor="middle" x="655" y="-193.709" font-family="Times,serif" font-size="12.00" fill="gray">resume</text> +<!-- clean wait->clean failed --> +<g id="edge28" class="edge"><title>clean wait->clean failed</title> +<path fill="none" stroke="black" d="M797.117,-193.449C825.68,-187.71 867.901,-182.218 904,-189.55 911.895,-191.154 920.046,-193.851 927.674,-196.894"/> +<polygon fill="black" stroke="black" points="926.429,-200.168 937,-200.879 929.179,-193.731 926.429,-200.168"/> +<text text-anchor="middle" x="866" y="-192.95" font-family="Times,serif" font-size="12.00">abort (via API)</text> </g> <!-- inspect failed->manageable --> -<g id="edge34" class="edge"><title>inspect failed->manageable</title> -<path fill="none" stroke="black" d="M716.918,-53.1014C648.112,-56.9742 513.552,-69.0572 406,-106.309 397.571,-109.229 388.829,-113.136 380.739,-117.164"/> -<polygon fill="black" stroke="black" points="378.871,-114.19 371.599,-121.895 382.089,-120.406 378.871,-114.19"/> -<text text-anchor="middle" x="551" y="-81.7094" font-family="Times,serif" font-size="12.00">manage (via API)</text> +<g id="edge35" class="edge"><title>inspect failed->manageable</title> +<path fill="none" stroke="black" d="M716.878,-56.0731C633.005,-61.4772 456.973,-76.3869 406,-108.55 386.09,-121.113 370.544,-142.979 360.419,-160.525"/> +<polygon fill="black" stroke="black" points="357.104,-159.292 355.359,-169.743 363.24,-162.661 357.104,-159.292"/> +<text text-anchor="middle" x="551" y="-81.95" font-family="Times,serif" font-size="12.00">manage (via API)</text> </g> <!-- inspect failed->inspecting --> -<g id="edge35" class="edge"><title>inspect failed->inspecting</title> -<path fill="none" stroke="black" d="M734.727,-37.0181C723.645,-31.9591 710.52,-26.887 698,-24.3094 665.054,-17.5267 627.033,-17.5072 598.051,-19.0618"/> -<polygon fill="black" stroke="black" points="597.573,-15.5846 587.809,-19.6961 598.006,-22.5712 597.573,-15.5846"/> -<text text-anchor="middle" x="655" y="-27.7094" font-family="Times,serif" font-size="12.00">inspect (via API)</text> +<g id="edge36" class="edge"><title>inspect failed->inspecting</title> +<path fill="none" stroke="black" d="M734.727,-39.2587C723.645,-34.1997 710.52,-29.1276 698,-26.55 665.054,-19.7673 627.033,-19.7478 598.051,-21.3024"/> +<polygon fill="black" stroke="black" points="597.573,-17.8252 587.809,-21.9367 598.006,-24.8118 597.573,-17.8252"/> +<text text-anchor="middle" x="655" y="-29.95" font-family="Times,serif" font-size="12.00">inspect (via API)</text> +</g> +<!-- adopt failed->manageable --> +<g id="edge40" class="edge"><title>adopt failed->manageable</title> +<path fill="none" stroke="black" d="M732.005,-418.332C651.178,-385.257 433.099,-294.955 406,-272.55 386.665,-256.564 370.867,-232.724 360.501,-214.356"/> +<polygon fill="black" stroke="black" points="363.439,-212.432 355.586,-205.32 357.29,-215.777 363.439,-212.432"/> +<text text-anchor="middle" x="551" y="-361.95" font-family="Times,serif" font-size="12.00">manage (via API)</text> +</g> +<!-- adopt failed->adopting --> +<g id="edge39" class="edge"><title>adopt failed->adopting</title> +<path fill="none" stroke="black" d="M728.401,-420.602C718.68,-418.147 707.995,-415.85 698,-414.55 663.074,-410.007 623.054,-409.806 593.757,-410.573"/> +<polygon fill="black" stroke="black" points="593.364,-407.083 583.478,-410.892 593.582,-414.08 593.364,-407.083"/> +<text text-anchor="middle" x="655" y="-417.95" font-family="Times,serif" font-size="12.00">adopt (via API)</text> </g> </g> </svg> diff --git a/doc/source/index.rst b/doc/source/index.rst index 77d17ffed..9981cacd5 100644 --- a/doc/source/index.rst +++ b/doc/source/index.rst @@ -41,6 +41,7 @@ Administrator's Guide deploy/raid deploy/inspection deploy/security + deploy/adoption deploy/troubleshooting Release Notes <http://docs.openstack.org/releasenotes/ironic/> diff --git a/doc/source/webapi/v1.rst b/doc/source/webapi/v1.rst index c0e2d6ae5..4ed0e10a0 100644 --- a/doc/source/webapi/v1.rst +++ b/doc/source/webapi/v1.rst @@ -32,7 +32,15 @@ always requests the newest supported API version. API Versions History -------------------- +**1.17** + + Addition of provision_state verb ``adopt`` which allows an operator + to move a node from ``manageable`` state to ``active`` state without + performing a deployment operation on the node. This is intended for + nodes that have already been deployed by external means. + **1.16** + Add ability to filter nodes by driver. **1.15** diff --git a/ironic/api/controllers/v1/node.py b/ironic/api/controllers/v1/node.py index c34cfc1ec..18a8fd7c7 100644 --- a/ironic/api/controllers/v1/node.py +++ b/ironic/api/controllers/v1/node.py @@ -94,7 +94,8 @@ _DEFAULT_RETURN_FIELDS = ('instance_uuid', 'maintenance', 'power_state', # States where calling do_provisioning_action makes sense PROVISION_ACTION_STATES = (ir_states.VERBS['manage'], ir_states.VERBS['provide'], - ir_states.VERBS['abort']) + ir_states.VERBS['abort'], + ir_states.VERBS['adopt']) _NODES_CONTROLLER_RESERVED_WORDS = None diff --git a/ironic/api/controllers/v1/utils.py b/ironic/api/controllers/v1/utils.py index 85ffd3486..027fd69e4 100644 --- a/ironic/api/controllers/v1/utils.py +++ b/ironic/api/controllers/v1/utils.py @@ -51,6 +51,7 @@ MIN_VERB_VERSIONS = { states.VERBS['inspect']: versions.MINOR_6_INSPECT_STATE, states.VERBS['abort']: versions.MINOR_13_ABORT_VERB, states.VERBS['clean']: versions.MINOR_15_MANUAL_CLEAN, + states.VERBS['adopt']: versions.MINOR_17_ADOPT_VERB, } diff --git a/ironic/api/controllers/v1/versions.py b/ironic/api/controllers/v1/versions.py index 0acdf6701..16d985946 100644 --- a/ironic/api/controllers/v1/versions.py +++ b/ironic/api/controllers/v1/versions.py @@ -46,6 +46,7 @@ BASE_VERSION = 1 # 2. '/v1/drivers/<driver-name>/properties' # v1.15: Add ability to do manual cleaning of nodes # v1.16: Add ability to filter nodes by driver. +# v1.17: Add 'adopt' verb for ADOPTING active nodes. MINOR_0_JUNO = 0 MINOR_1_INITIAL_VERSION = 1 @@ -64,11 +65,12 @@ MINOR_13_ABORT_VERB = 13 MINOR_14_LINKS_NODESTATES_DRIVERPROPERTIES = 14 MINOR_15_MANUAL_CLEAN = 15 MINOR_16_DRIVER_FILTER = 16 +MINOR_17_ADOPT_VERB = 17 # When adding another version, update MINOR_MAX_VERSION and also update # doc/source/webapi/v1.rst with a detailed explanation of what the version has # changed. -MINOR_MAX_VERSION = MINOR_16_DRIVER_FILTER +MINOR_MAX_VERSION = MINOR_17_ADOPT_VERB # String representations of the minor and maximum versions MIN_VERSION_STRING = '{}.{}'.format(BASE_VERSION, MINOR_1_INITIAL_VERSION) diff --git a/ironic/common/states.py b/ironic/common/states.py index 09c4aff85..e5251036c 100644 --- a/ironic/common/states.py +++ b/ironic/common/states.py @@ -47,6 +47,7 @@ VERBS = { 'inspect': 'inspect', 'abort': 'abort', 'clean': 'clean', + 'adopt': 'adopt', } """ Mapping of state-changing events that are PUT to the REST API @@ -161,19 +162,35 @@ INSPECTING = 'inspecting' """ Node is under inspection. This is the provision state used when inspection is started. A successfully -inspected node shall transition to MANAGEABLE status. +inspected node shall transition to MANAGEABLE state. """ INSPECTFAIL = 'inspect failed' """ Node inspection failed. """ +ADOPTING = 'adopting' +""" Node is being adopted. + +This provision state is intended for use to move a node from MANAGEABLE to +ACTIVE state to permit designation of nodes as being "managed" by Ironic, +however "deployed" previously by external means. +""" + +ADOPTFAIL = 'adopt failed' +""" Node failed to complete the adoption process. + +This state is the resulting state of a node that failed to complete adoption, +potentially due to invalid or incompatible information being defined for the +node. +""" UPDATE_ALLOWED_STATES = (DEPLOYFAIL, INSPECTING, INSPECTFAIL, CLEANFAIL, ERROR, - VERIFYING) + VERIFYING, ADOPTFAIL) """Transitional states in which we allow updating a node.""" -DELETE_ALLOWED_STATES = (AVAILABLE, NOSTATE, MANAGEABLE, ENROLL) +DELETE_ALLOWED_STATES = (AVAILABLE, NOSTATE, MANAGEABLE, ENROLL, + ADOPTFAIL) """States in which node deletion is allowed.""" STABLE_STATES = (ENROLL, MANAGEABLE, AVAILABLE, ACTIVE, ERROR) @@ -243,6 +260,10 @@ machine.add_transition(AVAILABLE, DEPLOYING, 'deploy') machine.add_state(INSPECTING, target=MANAGEABLE, **watchers) machine.add_state(INSPECTFAIL, target=MANAGEABLE, **watchers) +# Add adopt* states +machine.add_state(ADOPTING, target=ACTIVE, **watchers) +machine.add_state(ADOPTFAIL, target=ACTIVE, **watchers) + # A deployment may fail machine.add_transition(DEPLOYING, DEPLOYFAIL, 'fail') @@ -346,3 +367,19 @@ machine.add_transition(VERIFYING, MANAGEABLE, 'done') # Verification can fail with setting last_error and rolling back to ENROLL machine.add_transition(VERIFYING, ENROLL, 'fail') + +# Node Adoption is being attempted +machine.add_transition(MANAGEABLE, ADOPTING, 'adopt') + +# Adoption can succeed and the node should be set to ACTIVE +machine.add_transition(ADOPTING, ACTIVE, 'done') + +# Node adoptions can fail and as such nodes shall be set +# into a dedicated state to hold the nodes. +machine.add_transition(ADOPTING, ADOPTFAIL, 'fail') + +# Node adoption can be retried when it previously failed. +machine.add_transition(ADOPTFAIL, ADOPTING, 'adopt') + +# A node that failed adoption can be moved back to manageable +machine.add_transition(ADOPTFAIL, MANAGEABLE, 'manage') diff --git a/ironic/conductor/manager.py b/ironic/conductor/manager.py index 8cb9ba008..69d3d73c3 100644 --- a/ironic/conductor/manager.py +++ b/ironic/conductor/manager.py @@ -1120,6 +1120,16 @@ class ConductorManager(base_manager.BaseConductorManager): err_handler=utils.provisioning_error_handler) return + if (action == states.VERBS['adopt'] and + node.provision_state in (states.MANAGEABLE, + states.ADOPTFAIL)): + task.process_event( + 'adopt', + callback=self._spawn_worker, + call_args=(self._do_adoption, task), + err_handler=utils.provisioning_error_handler) + return + if (action == states.VERBS['abort'] and node.provision_state == states.CLEANWAIT): @@ -1306,6 +1316,54 @@ class ConductorManager(base_manager.BaseConductorManager): callback_method=utils.cleanup_after_timeout, err_handler=utils.provisioning_error_handler) + @task_manager.require_exclusive_lock + def _do_adoption(self, task): + """Adopt the node. + + Similar to node takeover, adoption performs a driver boot + validation and then triggers node takeover in order to make the + conductor responsible for the node. Upon completion of takeover, + the node is moved to ACTIVE state. + + The goal of this method is to set the conditions for the node to + be managed by Ironic as an ACTIVE node without having performed + a deployment operation. + + :param task: a TaskManager instance + """ + + node = task.node + LOG.debug('Conductor %(cdr)s attempting to adopt node %(node)s', + {'cdr': self.host, 'node': node.uuid}) + + try: + # NOTE(TheJulia): A number of drivers expect to know if a + # whole disk image was used prior to their takeover logic + # being triggered, as such we need to populate the + # internal info based on the configuration the user has + # supplied. + iwdi = images.is_whole_disk_image(task.context, + task.node.instance_info) + node.driver_internal_info['is_whole_disk_image'] = iwdi + # Calling boot validate to ensure that sufficient information + # is supplied to allow the node to be able to boot if takeover + # writes items such as kernel/ramdisk data to disk. + task.driver.boot.validate(task) + # NOTE(TheJulia): While task.driver.boot.validate() is called + # above, and task.driver.power.validate() could be called, it + # is called as part of the transition from ENROLL to MANAGEABLE + # states. As such it is redundant to call here. + self._do_takeover(task) + LOG.info(_LI("Successfully adopted node %(node)s"), + {'node': node.uuid}) + task.process_event('done') + except Exception as err: + msg = (_('Error while attempting to adopt node %(node)s: ' + '%(err)s.') % {'node': node.uuid, 'err': err}) + LOG.error(msg) + node.last_error = msg + task.process_event('fail') + def _do_takeover(self, task): """Take over this node. diff --git a/ironic/conductor/utils.py b/ironic/conductor/utils.py index 0aac967a0..f25340c22 100644 --- a/ironic/conductor/utils.py +++ b/ironic/conductor/utils.py @@ -41,6 +41,14 @@ CLEANING_INTERFACE_PRIORITY = { def node_set_boot_device(task, device, persistent=False): """Set the boot device for a node. + Sets the boot device for a node if the node's driver interface + contains a 'management' interface. + + If the node that the boot device change is being requested for + is in ADOPTING state, the boot device will not be set as that + change could potentially result in the future running state of + an adopted node being modified erroneously. + :param task: a TaskManager instance. :param device: Boot device. Values are vendor-specific. :param persistent: Whether to set next-boot, or make the change @@ -51,9 +59,14 @@ def node_set_boot_device(task, device, persistent=False): """ if getattr(task.driver, 'management', None): task.driver.management.validate(task) - task.driver.management.set_boot_device(task, - device=device, - persistent=persistent) + # NOTE(TheJulia): When a node is in the ADOPTING state, we must + # not attempt to change the default boot device as a side effect + # of a driver's node takeover process as it is modifying + # a working machine. + if task.node.provision_state != states.ADOPTING: + task.driver.management.set_boot_device(task, + device=device, + persistent=persistent) @task_manager.require_exclusive_lock diff --git a/ironic/tests/unit/api/v1/test_nodes.py b/ironic/tests/unit/api/v1/test_nodes.py index 5e26b5d6e..5eabd0047 100644 --- a/ironic/tests/unit/api/v1/test_nodes.py +++ b/ironic/tests/unit/api/v1/test_nodes.py @@ -2184,6 +2184,93 @@ class TestPut(test_api_base.BaseApiTest): mock_rpcapi.assert_called_once_with(mock.ANY, self.node.uuid, clean_steps, 'test-topic') + def test_adopt_raises_error_before_1_17(self): + """Test that a lower API client cannot use the adopt verb""" + ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid, + {'target': states.VERBS['adopt']}, + headers={api_base.Version.string: "1.16"}, + expect_errors=True) + self.assertEqual(http_client.NOT_ACCEPTABLE, ret.status_code) + + @mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action') + def test_adopt_from_manage(self, mock_dpa): + """Test that a node can be adopted from the manageable state""" + self.node.provision_state = states.MANAGEABLE + self.node.save() + + ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid, + {'target': states.VERBS['adopt']}, + headers={api_base.Version.string: "1.17"}) + self.assertEqual(http_client.ACCEPTED, ret.status_code) + self.assertEqual(b'', ret.body) + mock_dpa.assert_called_once_with(mock.ANY, self.node.uuid, + states.VERBS['adopt'], + 'test-topic') + + @mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action') + def test_adopt_from_adoption_failed(self, mock_dpa): + self.node.provision_state = states.ADOPTFAIL + self.node.save() + + ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid, + {'target': states.VERBS['adopt']}, + headers={api_base.Version.string: "1.17"}) + self.assertEqual(http_client.ACCEPTED, ret.status_code) + self.assertEqual(b'', ret.body) + mock_dpa.assert_called_once_with(mock.ANY, self.node.uuid, + states.VERBS['adopt'], + 'test-topic') + + @mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action') + def test_adopt_from_active_fails(self, mock_dpa): + self.node.provision_state = states.ACTIVE + self.node.save() + + ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid, + {'target': states.VERBS['adopt']}, + headers={api_base.Version.string: "1.17"}, + expect_errors=True) + self.assertEqual(http_client.BAD_REQUEST, ret.status_code) + self.assertEqual(0, mock_dpa.call_count) + + @mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action') + def test_manage_from_adoption_failed(self, mock_dpa): + self.node.provision_state = states.ADOPTFAIL + self.node.save() + + ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid, + {'target': states.VERBS['manage']}, + headers={api_base.Version.string: "1.17"}) + self.assertEqual(http_client.ACCEPTED, ret.status_code) + self.assertEqual(b'', ret.body) + mock_dpa.assert_called_once_with(mock.ANY, self.node.uuid, + states.VERBS['manage'], + 'test-topic') + + @mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action') + def test_bad_requests_in_adopting_state(self, mock_dpa): + self.node.provision_state = states.ADOPTING + self.node.save() + + for state in [states.ACTIVE, states.REBUILD, states.DELETED]: + ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid, + {'target': state}, + expect_errors=True) + self.assertEqual(http_client.BAD_REQUEST, ret.status_code) + self.assertEqual(0, mock_dpa.call_count) + + @mock.patch.object(rpcapi.ConductorAPI, 'do_provisioning_action') + def test_bad_requests_in_adoption_failed_state(self, mock_dpa): + self.node.provision_state = states.ADOPTFAIL + self.node.save() + + for state in [states.ACTIVE, states.REBUILD, states.DELETED]: + ret = self.put_json('/nodes/%s/states/provision' % self.node.uuid, + {'target': state}, + expect_errors=True) + self.assertEqual(http_client.BAD_REQUEST, ret.status_code) + self.assertEqual(0, mock_dpa.call_count) + def test_set_console_mode_enabled(self): with mock.patch.object(rpcapi.ConductorAPI, 'set_console_mode') as mock_scm: diff --git a/ironic/tests/unit/api/v1/test_utils.py b/ironic/tests/unit/api/v1/test_utils.py index 3f9da1e42..7c01dba22 100644 --- a/ironic/tests/unit/api/v1/test_utils.py +++ b/ironic/tests/unit/api/v1/test_utils.py @@ -182,6 +182,17 @@ class TestApiUtils(base.TestCase): mock_request.version.minor = 10 self.assertFalse(utils.allow_links_node_states_and_driver_properties()) + @mock.patch.object(pecan, 'request', spec_set=['version']) + def test_check_allow_adopt_verbs_fail(self, mock_request): + mock_request.version.minor = 16 + self.assertRaises(exception.NotAcceptable, + utils.check_allow_management_verbs, 'adopt') + + @mock.patch.object(pecan, 'request', spec_set=['version']) + def test_check_allow_adopt_verbs(self, mock_request): + mock_request.version.minor = 17 + utils.check_allow_management_verbs('adopt') + class TestNodeIdent(base.TestCase): diff --git a/ironic/tests/unit/conductor/test_manager.py b/ironic/tests/unit/conductor/test_manager.py index 9f327fa8b..d53db2a75 100644 --- a/ironic/tests/unit/conductor/test_manager.py +++ b/ironic/tests/unit/conductor/test_manager.py @@ -2610,6 +2610,16 @@ class DestroyNodeTestCase(mgr_utils.ServiceSetUpMixin, self.dbapi.get_node_by_uuid, node.uuid) + def test_destroy_node_adopt_failed_no_power_change(self): + self._start_service() + node = obj_utils.create_test_node(self.context, + driver='fake', + provision_state=states.ADOPTFAIL) + with mock.patch.object(self.driver.power, + 'set_power_state') as mock_power: + self.service.destroy_node(self.context, node.uuid) + self.assertFalse(mock_power.called) + @mgr_utils.mock_record_keepalive class UpdatePortTestCase(mgr_utils.ServiceSetUpMixin, @@ -4760,3 +4770,143 @@ class DoNodeTakeOverTestCase(mgr_utils.ServiceSetUpMixin, mock_prepare.assert_called_once_with(mock.ANY) mock_take_over.assert_called_once_with(mock.ANY) mock_start_console.assert_called_once_with(mock.ANY) + + +@mgr_utils.mock_record_keepalive +class DoNodeAdoptionTestCase( + mgr_utils.ServiceSetUpMixin, + tests_db_base.DbTestCase): + + @mock.patch('ironic.drivers.modules.fake.FakePower.validate') + @mock.patch('ironic.drivers.modules.fake.FakeBoot.validate') + @mock.patch('ironic.drivers.modules.fake.FakeConsole.start_console') + @mock.patch('ironic.drivers.modules.fake.FakeDeploy.take_over') + @mock.patch('ironic.drivers.modules.fake.FakeDeploy.prepare') + def test__do_adoption_with_takeover(self, + mock_prepare, + mock_take_over, + mock_start_console, + mock_boot_validate, + mock_power_validate): + self._start_service() + node = obj_utils.create_test_node( + self.context, driver='fake', + provision_state=states.ADOPTING) + task = task_manager.TaskManager(self.context, node.uuid) + + self.service._do_adoption(task) + node.refresh() + + self.assertEqual(states.ACTIVE, node.provision_state) + self.assertIsNone(node.last_error) + self.assertFalse(node.console_enabled) + mock_prepare.assert_called_once_with(mock.ANY) + mock_take_over.assert_called_once_with(mock.ANY) + self.assertFalse(mock_start_console.called) + self.assertTrue(mock_boot_validate.called) + + @mock.patch('ironic.drivers.modules.fake.FakeBoot.validate') + @mock.patch('ironic.drivers.modules.fake.FakeConsole.start_console') + @mock.patch('ironic.drivers.modules.fake.FakeDeploy.take_over') + @mock.patch('ironic.drivers.modules.fake.FakeDeploy.prepare') + def test__do_adoption_take_over_failure(self, + mock_prepare, + mock_take_over, + mock_start_console, + mock_boot_validate): + # Note(TheJulia): Use of an actual possible exception that + # can be raised due to a misconfiguration. + mock_take_over.side_effect = exception.IPMIFailure( + "something went wrong") + + self._start_service() + node = obj_utils.create_test_node( + self.context, driver='fake', + provision_state=states.ADOPTING) + task = task_manager.TaskManager(self.context, node.uuid) + + self.service._do_adoption(task) + node.refresh() + + self.assertEqual(states.ADOPTFAIL, node.provision_state) + self.assertIsNotNone(node.last_error) + self.assertFalse(node.console_enabled) + mock_prepare.assert_called_once_with(mock.ANY) + mock_take_over.assert_called_once_with(mock.ANY) + self.assertFalse(mock_start_console.called) + self.assertTrue(mock_boot_validate.called) + + @mock.patch('ironic.drivers.modules.fake.FakeBoot.validate') + @mock.patch('ironic.drivers.modules.fake.FakeConsole.start_console') + @mock.patch('ironic.drivers.modules.fake.FakeDeploy.take_over') + @mock.patch('ironic.drivers.modules.fake.FakeDeploy.prepare') + def test__do_adoption_boot_validate_failure(self, + mock_prepare, + mock_take_over, + mock_start_console, + mock_boot_validate): + # Note(TheJulia): Use of an actual possible exception that + # can be raised due to a misconfiguration. + mock_boot_validate.side_effect = exception.MissingParameterValue( + "something is missing") + + self._start_service() + node = obj_utils.create_test_node( + self.context, driver='fake', + provision_state=states.ADOPTING) + task = task_manager.TaskManager(self.context, node.uuid) + + self.service._do_adoption(task) + node.refresh() + + self.assertEqual(states.ADOPTFAIL, node.provision_state) + self.assertIsNotNone(node.last_error) + self.assertFalse(node.console_enabled) + self.assertFalse(mock_prepare.called) + self.assertFalse(mock_take_over.called) + self.assertFalse(mock_start_console.called) + self.assertTrue(mock_boot_validate.called) + + @mock.patch('ironic.conductor.manager.ConductorManager._spawn_worker') + def test_do_provisioning_action_adopt_node(self, mock_spawn): + node = obj_utils.create_test_node( + self.context, driver='fake', + provision_state=states.MANAGEABLE, + target_provision_state=states.NOSTATE) + + self._start_service() + self.service.do_provisioning_action(self.context, node.uuid, 'adopt') + node.refresh() + self.assertEqual(states.ADOPTING, node.provision_state) + self.assertEqual(states.ACTIVE, node.target_provision_state) + self.assertIsNone(node.last_error) + mock_spawn.assert_called_with(self.service._do_adoption, mock.ANY) + + @mock.patch('ironic.conductor.manager.ConductorManager._spawn_worker') + def test_do_provisioning_action_adopt_node_retry(self, mock_spawn): + node = obj_utils.create_test_node( + self.context, driver='fake', + provision_state=states.ADOPTFAIL, + target_provision_state=states.ACTIVE) + + self._start_service() + self.service.do_provisioning_action(self.context, node.uuid, 'adopt') + node.refresh() + self.assertEqual(states.ADOPTING, node.provision_state) + self.assertEqual(states.ACTIVE, node.target_provision_state) + self.assertIsNone(node.last_error) + mock_spawn.assert_called_with(self.service._do_adoption, mock.ANY) + + def test_do_provisioning_action_manage_of_failed_adoption(self): + node = obj_utils.create_test_node( + self.context, driver='fake', + provision_state=states.ADOPTFAIL, + target_provision_state=states.ACTIVE) + + self._start_service() + self.service.do_provisioning_action(self.context, node.uuid, 'manage') + node.refresh() + + self.assertEqual(states.MANAGEABLE, node.provision_state) + self.assertEqual(states.NOSTATE, node.target_provision_state) + self.assertIsNone(node.last_error) diff --git a/ironic/tests/unit/conductor/test_utils.py b/ironic/tests/unit/conductor/test_utils.py index a04a1c8f4..15b0117b4 100644 --- a/ironic/tests/unit/conductor/test_utils.py +++ b/ironic/tests/unit/conductor/test_utils.py @@ -60,6 +60,23 @@ class NodeSetBootDeviceTestCase(base.DbTestCase): device='pxe', persistent=False) + def test_node_set_boot_device_adopting(self): + mgr_utils.mock_the_extension_manager(driver="fake_ipmitool") + self.driver = driver_factory.get_driver("fake_ipmitool") + ipmi_info = utils.get_test_ipmi_info() + node = obj_utils.create_test_node(self.context, + uuid=uuidutils.generate_uuid(), + driver='fake_ipmitool', + driver_info=ipmi_info, + provision_state=states.ADOPTING) + task = task_manager.TaskManager(self.context, node.uuid) + + with mock.patch.object(self.driver.management, + 'set_boot_device') as mock_sbd: + conductor_utils.node_set_boot_device(task, + device='pxe') + self.assertFalse(mock_sbd.called) + class NodePowerActionTestCase(base.DbTestCase): diff --git a/releasenotes/notes/active-node-creation-a41c9869c966c82b.yaml b/releasenotes/notes/active-node-creation-a41c9869c966c82b.yaml new file mode 100644 index 000000000..91abd8e91 --- /dev/null +++ b/releasenotes/notes/active-node-creation-a41c9869c966c82b.yaml @@ -0,0 +1,14 @@ +--- +features: + - Addition of the provision state target verb of ``adopt`` + which allows an operator to move a node into an ``active`` + state from ``manageable`` state, without performing a deployment + operation on the node. This can be used to represent nodes that have + been previously deployed by other means that will now be managed by + ironic and be later released to the available hardware pool. +other: + - When a node is enrolled into ironic, upon transition to the + ``manageable`` state, the current power state of the node is + recorded. Once the node is adopted and in an ``active`` state, + that recorded power state will be enfored by ironic unless an + operator changes the power state in ironic. |
