Improve trove guest agent image building

- Add a new element 'guest-agent' for image building. This element is
  used when dev_mode=false, so that the trove code is downloaded into
  the image during the building phase rather than during the guest
  agent initialization.
- Improve trovestack sub-command 'build-image'.

  ./trovestack build-image ${datastore_type} \
     ${guest_os} \
     ${guest_release} \
     ${dev_mode}

- Improve documentation.

Story: #2005387
Task: #30375

Change-Id: I9d7acbd6a97f8c01b48b0f2cf94398d549d89124

1 files changed, 26 insertions, 420 deletions

diff --git a/doc/source/admin/building_guest_images.rst b/doc/source/admin/building_guest_images.rst
index a9ea3991..389325cc 100644
--- a/doc/source/admin/building_guest_images.rst
+++ b/doc/source/admin/building_guest_images.rst
@@ -85,12 +85,12 @@ Injected Configuration for the Guest Agent
 
 When TaskManager launches the guest VM it injects the specific settings
 for the guest into the VM, into the file /etc/trove/conf.d/guest_info.conf.
-The file is injected one of three ways. If use_heat=True, it is injected
-during the heat launch process. If use_nova_server_config_drive=True
-it is injected via ConfigDrive. Otherwise it is passed to the nova
-create call as the 'files' parameter and will be injected based on
-the configuration of Nova; the Nova default is to discard the files.
-If the settings in guest_info.conf are not present on the guest
+The file is injected one of three ways.
+
+If use_nova_server_config_drive=True, it is injected via ConfigDrive. Otherwise
+it is passed to the nova create call as the 'files' parameter and will be
+injected based on the configuration of Nova; the Nova default is to discard the
+files. If the settings in guest_info.conf are not present on the guest
 Guest Agent will fail to start up.
 
 ------------------------------
@@ -108,135 +108,23 @@ Building Guest Images using DIB
 
 A Trove Guest Image can be built with any tool that produces an image
 accepted by Nova. In this document we describe how to build guest
-images using the 'Disk Image Builder' (DIB) tool, and we focus on
-building qemu images [1]_. DIB is an OpenStack tool and is available for
-download at
-https://git.openstack.org/cgit/openstack/diskimage-builder/tree/ or
-https://pypi.org/project/diskimage-builder/0.1.38.
+images using the
+`'Disk Image Builder' (DIB) <https://docs.openstack.org/diskimage-builder/latest/>`_
+tool, and we focus on building qemu images.
 
 DIB uses a chroot'ed environment to construct the image. The goal is
 to build a bare machine that has all the components required for
 launch by Nova.
 
-----------
-Invocation
-----------
-
-You can download the DIB tool from OpenStack's public git
-repository. Note that DIB works with Ubuntu and Fedora (RedHat). Other
-operating systems are not yet fully supported.
-
-.. code-block:: bash
-
-   user@machine:/opt/stack$ git clone https://git.openstack.org/openstack/diskimage-builder
-   Cloning into 'diskimage-builder'...
-   remote: Counting objects: 8881, done.
-   remote: Total 8881 (delta 0), reused 0 (delta 0)
-   Receiving objects: 100% (8881/8881), 1.92 MiB | 0 bytes/s, done.
-   Resolving deltas: 100% (4668/4668), done.
-   Checking connectivity... done.
-   user@machine:/opt/stack$ cd diskimage-builder
-   user@machine:/opt/stack/diskimage-builder$ sudo pip install -r requirements.txt
-   user@machine:/opt/stack/diskimage-builder$ sudo python setup.py install
-
-
-Ensure that you have qemu-img [2]_ and kpartx installed.
-
-The disk-image-create command is the main command in the DIB tool that
-is used to build guest images for Trove. The disk-image-create command
-takes the following options:
-
-.. code-block:: bash
-
-    user@machine:/opt/stack/diskimage-builder$ disk-image-create -h
-    Usage: disk-image-create [OPTION]... [ELEMENT]...
-
-    Options:
-        -a i386|amd64|armhf -- set the architecture of the image(default amd64)
-        -o imagename -- set the imagename of the output image file(default image)
-        -t qcow2,tar -- set the image types of the output image files (default qcow2)
-           File types should be comma separated
-        -x -- turn on tracing
-        -u -- uncompressed; do not compress the image - larger but faster
-        -c -- clear environment before starting work
-        --image-size size -- image size in GB for the created image
-        --image-cache directory -- location for cached images(default ~/.cache/image-create)
-        --max-online-resize size -- max number of filesystem blocks to support when resizing.
-           Useful if you want a really large root partition when the image is deployed.
-           Using a very large value may run into a known bug in resize2fs.
-           Setting the value to 274877906944 will get you a 1PB root file system.
-           Making this value unnecessarily large will consume extra disk space
-           on the root partition with extra file system inodes.
-        --min-tmpfs size -- minimum size in GB needed in tmpfs to build the image
-        --no-tmpfs -- do not use tmpfs to speed image build
-        --offline -- do not update cached resources
-        --qemu-img-options -- option flags to be passed directly to qemu-img.
-           Options need to be comma separated, and follow the key=value pattern.
-        --root-label label -- label for the root filesystem.  Defaults to 'cloudimg-rootfs'.
-        --ramdisk-element -- specify the main element to be used for building ramdisks.
-           Defaults to 'ramdisk'.  Should be set to 'dracut-ramdisk' for platforms such
-           as RHEL and CentOS that do not package busybox.
-        --install-type -- specify the default installation type. Defaults to 'source'. Set
-           to 'package' to use package based installations by default.
-        -n skip the default inclusion of the 'base' element
-        -p package[,package,package] -- list of packages to install in the image
-        -h|--help -- display this help and exit
-
-    ELEMENTS_PATH will allow you to specify multiple locations for the elements.
-
-    NOTE: At least one distribution root element must be specified.
-
-    Examples:
-        disk-image-create -a amd64 -o ubuntu-amd64 vm ubuntu
-        export ELEMENTS_PATH=~/source/tripleo-image-elements/elements
-        disk-image-create -a amd64 -o fedora-amd64-heat-cfntools vm fedora heat-cfntools
-    user@machine:/opt/stack/diskimage-builder$
-
-The example command provided above would build a perfectly functional
-Nova image with the 64 bit Fedora operating system.
-
-In addition to the -a argument which specifies to build an amd64 (64
-bit) image, and the -o which specifies the output file, the command
-line lists the various elements that should be used in building the
-image. The next section of this document talks about image elements.
-
-Building a Trove guest image is a little more involved and the standard
-elements (more about this later) are highly configurable through the use
-of environment variables.
-
-This command will create a guest image usable by Trove:
+----------------------------
+Build image using trovestack
+----------------------------
 
-.. code-block:: bash
-
-    # assign a suitable value for each of these environment
-    # variables that change the way the elements behave.
-    export HOST_USERNAME
-    export HOST_SCP_USERNAME
-    export GUEST_USERNAME
-    export CONTROLLER_IP
-    export TROVESTACK_SCRIPTS
-    export SERVICE_TYPE
-    export PATH_TROVE
-    export ESCAPED_PATH_TROVE
-    export SSH_DIR
-    export GUEST_LOGDIR
-    export ESCAPED_GUEST_LOGDIR
-    export DIB_CLOUD_INIT_DATASOURCES="ConfigDrive"
-    export DATASTORE_PKG_LOCATION
-    export BRANCH_OVERRIDE
-
-    # you typically do not have to change these variables
-    export ELEMENTS_PATH=$TROVESTACK_SCRIPTS/files/elements
-    export ELEMENTS_PATH+=:$PATH_DISKIMAGEBUILDER/elements
-    export ELEMENTS_PATH+=:$PATH_TRIPLEO_ELEMENTS/elements
-    export DIB_APT_CONF_DIR=/etc/apt/apt.conf.d
-    export DIB_CLOUD_INIT_ETC_HOSTS=true
-    local QEMU_IMG_OPTIONS="--qemu-img-options compat=1.1"
-
-    # run disk-image-create that actually causes the image to be built
-    $disk-image-create -a amd64 -o "${VM}" \
-        -x ${QEMU_IMG_OPTIONS} ${DISTRO} ${EXTRA_ELEMENTS} vm \
-        cloud-init-datasources ${DISTRO}-guest ${DISTRO}-${SERVICE_TYPE}
+Trove provides a script called ``trovestack`` that could do most of the
+management and test tasks. Refer to the "Build guest agent image" section
+in
+`trovestack document <https://github.com/openstack/trove/blob/master/integration/README.md>`_
+for how to build trove guest agent images.
 
 -----------------------------
 Disk Image Builder 'Elements'
@@ -248,106 +136,16 @@ scripts that are executed by DIB in a specific order to generate the
 image. You provide the names of the elements that you would like
 executed, in order, on the command line to disk-image-create.
 
-Elements are executed within the chroot'ed environment while DIB is
-run. Elements are executed in phases and the various phases are (in
-order) root.d, extra-data.d, pre-install.d, install.d, post-install.d,
-block-device.d, finalise.d [3]_, and cleanup.d [4]_. The latter
-reference provides a very good outline on writing elements and is a
-'must read'.
-
-Some elements use environment.d to setup environment
-variables. Element dependencies can be established using the
-element-deps and element-provides files which are plain text files.
-
------------------
-Existing Elements
------------------
-
-DIB comes with some tools that are located in the elements directory.
+DIB comes with some
+`built-in elements <https://docs.openstack.org/diskimage-builder/latest/elements.html>`_.
+In addition, projects like
+`TripleO <https://github.com/openstack/tripleo-image-elements>`_ provide
+elements as well.
 
-.. code-block:: bash
-
-    user@machine:/opt/stack/diskimage-builder/elements$ ls
-    apt-conf                         dpkg                      ramdisk
-    apt-preferences                  dracut-network            ramdisk-base
-    apt-sources                      dracut-ramdisk            rax-nova-agent
-    architecture-emulation-binaries  element-manifest          redhat-common
-    baremetal                        enable-serial-console     rhel
-    base                             epel                      rhel7
-    cache-url                        fedora                    rhel-common
-    centos7                          hwburnin                  rpm-distro
-    cleanup-kernel-initrd            hwdiscovery               select-boot-kernel-initrd
-    cloud-init-datasources           ilo                       selinux-permissive
-    cloud-init-nocloud               ironic-agent              serial-console
-    debian                           ironic-discoverd-ramdisk  source-repositories
-    debian-systemd                   iso                       stable-interface-names
-    debian-upstart                   local-config              svc-map
-    deploy                           manifests                 uboot
-    deploy-baremetal                 mellanox                  ubuntu
-    deploy-ironic                    modprobe-blacklist        ubuntu-core
-    deploy-kexec                     opensuse                  vm
-    dhcp-all-interfaces              package-installs          yum
-    dib-run-parts                    pip-cache                 zypper
-    disable-selinux                  pkg-map
-    dkms                             pypi
-
-In addition, projects like TripleO [5]_ provide elements as well.
-
-Trove provides a set of elements as part of the trove [6]_
-project which will be described in the next section.
-
-Trove Reference Elements
-========================
-
-Reference elements provided by Trove are part of the trove project.
-
-In keeping with the philosophy of making elements 'layered', Trove
-provides two sets of elements. The first implements the guest agent
-for various operating systems and the second implements the database
-for these operating systems.
-
----------------------------
-Provided Reference Elements
----------------------------
-
-The Trove reference elements are located in the
-trove/integration/scripts/files/elements directory. The elements
-[operating-system]-guest provide the Trove Guest capabilities and the
-[operating-system]-[database] elements provide support for each
-database on the specified database.
-
-.. code-block:: bash
-
-  user@machine:/opt/stack/trove/integration/scripts/files/elements$ ls -l
-  total 56
-  drwxrwxr-x 5 user group 4096 Jan  7 12:47 fedora-guest
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 fedora-mongodb
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 fedora-mysql
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 fedora-percona
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 fedora-postgresql
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 fedora-redis
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 ubuntu-cassandra
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 ubuntu-couchbase
-  drwxrwxr-x 6 user group 4096 Jan  7 12:47 ubuntu-guest
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 ubuntu-mongodb
-  drwxrwxr-x 4 user group 4096 Jan  7 12:47 ubuntu-mysql
-  drwxrwxr-x 4 user group 4096 Jan  7 12:47 ubuntu-percona
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 ubuntu-postgresql
-  drwxrwxr-x 3 user group 4096 Jan  7 12:47 ubuntu-redis
-  user@machine:/opt/stack/trove/integration/scripts/files/elements$
-
-With this infrastructure in place, and the elements from DIB and
-TripleO accessible to the DIB command, one can generate the (for
-example) Ubuntu guest image for Percona Server with the command line:
-
-.. code-block:: bash
-
-  ${DIB} -a amd64 -o ${output-file} Ubuntu vm \
-      cloud-init-datasources ubuntu-guest ubuntu-percona
-
-Where ${DIB} is the fully qualified path to the disk-image-create
-command and ${output-file} is the name of the output file to be
-created.
+Trove also provides a set of its own elements. In keeping with the philosophy
+of making elements 'layered', Trove provides two sets of elements. The first
+implements the guest agent for various operating systems and the second
+implements the database for these operating systems.
 
 -------------------------------------------------------------------
 Contributing Reference Elements When Implementing a New 'Datastore'
@@ -357,81 +155,6 @@ When contributing a new datastore, you should contribute elements
 that will allow any user of Trove to be able to build a guest image
 for that datastore.
 
-This is typically accomplished by submitting files into the
-trove project, as above.
-
-Getting the Guest Agent Code onto a Trove Guest Instance
-========================================================
-
-The guest agent code typically runs on the guest instance alongside
-the database. There are two ways in which the guest agent code can be
-placed on the guest instance and we describe both of these here.
-
-----------------------------------------
-Guest Agent Code Installed at Build Time
-----------------------------------------
-
-In this option, the guest agent code is built into the guest image,
-thereby ensuring that all database instances that are launched with
-the image will have the exact same version of the guest image.
-
-This can be accomplished by placing suitable code in the elements for
-the image and these elements will ensure that the guest agent code is
-installed on the image.
-
---------------------------------------
-Guest Agent Code Installed at Run Time
---------------------------------------
-
-In this option, the guest agent code is not part of the guest image
-and instead the guest agent code is obtained at runtime, potentially
-from some well known location.
-
-In devstack, this is implemented in trove-guest.upstart.conf and
-trove-guest.systemd.conf. Shown below is the code from
-trove-guest.upstart.conf (this code may change in the future and
-is shown here as an example only). See the code highlighted below:
-
-.. code-block:: bash
-
-    description "Trove Guest"
-    author "Auto-Gen"
-
-    start on (filesystem and net-device-up IFACE!=lo)
-    stop on runlevel [016]
-    chdir /var/run
-    pre-start script
-        mkdir -p /var/run/trove
-        chown GUEST_USERNAME:root /var/run/trove/
-
-        mkdir -p /var/lock/trove
-        chown GUEST_USERNAME:root /var/lock/trove/
-
-        mkdir -p GUEST_LOGDIR
-        chown GUEST_USERNAME:root GUEST_LOGDIR
-        chmod +r /etc/guest_info
-
-        # If /etc/trove does not exist, copy the trove source and the
-        # guest agent config from the user's development environment
-        if [ ! -d /etc/trove ]; then
-    ->      sudo -u GUEST_USERNAME rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --exclude='.*' HOST_SCP_USERNAME@NETWORK_GATEWAY:PATH_TROVE/ /home/GUEST_USERNAME/trove
-            mkdir -p /etc/trove
-    ->      sudo -u GUEST_USERNAME rsync -e 'ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no' -avz --exclude='.*' HOST_SCP_USERNAME@NETWORK_GATEWAY:/etc/trove/trove-guestagent.conf ~GUEST_USERNAME/
-            mv ~GUEST_USERNAME/trove-guestagent.conf /etc/trove/trove-guestagent.conf
-        fi
-
-    end script
-
-    exec su -c "/home/GUEST_USERNAME/trove/contrib/trove-guestagent -config-file=/etc/guest_info --config-file=/etc/trove/trove-guestagent.conf" GUEST_USERNAME
-
-In building an image for a production Trove deployment, it is a very
-bad idea to use this mechanism. It makes sense in a development
-environment where the thing that you are developing is in Trove and
-part of the Guest Agent! This is because you get to merely boot a new
-Trove instance and the freshly modified code gets run on the
-Guest. But, in any other circumstance, it is much better to have the
-guest image include the guest agent code.
-
 Considerations in Building a Guest Image
 ========================================
 
@@ -522,120 +245,3 @@ repository, this would equate to:
 .. code-block:: bash
 
   DATASTORE_PKG_LOCATION=/path/to/new_db.deb ./trovestack kick-start new_db
-
-Building Guest Images Using Standard Elements
-=============================================
-
-A very good reference for how one builds guest images can be found by
-reviewing the trovestack script (trove/integration/scripts). Lower level
-routines that actually invoke Disk Image Builder can be found in
-trove/integration/scripts/functions_qemu.
-
-The following block of code illustrates the most basic invocation of
-DIB to create a guest image. This code is in
-trove/integration/scripts/functions_qemu as part of the function
-build_vm().  We look at this section of code in detail below.
-
-.. code-block:: bash
-
-    # assign a suitable value for each of these environment
-    # variables that change the way the elements behave.
-    export HOST_USERNAME
-    export HOST_SCP_USERNAME
-    export GUEST_USERNAME
-    export CONTROLLER_IP
-    export TROVESTACK_SCRIPTS
-    export SERVICE_TYPE
-    export PATH_TROVE
-    export ESCAPED_PATH_TROVE
-    export SSH_DIR
-    export GUEST_LOGDIR
-    export ESCAPED_GUEST_LOGDIR
-    export DIB_CLOUD_INIT_DATASOURCES="ConfigDrive"
-    export DATASTORE_PKG_LOCATION
-    export BRANCH_OVERRIDE
-
-    # you typically do not have to change these variables
-    export ELEMENTS_PATH=$TROVESTACK_SCRIPTS/files/elements
-    export ELEMENTS_PATH+=:$PATH_DISKIMAGEBUILDER/elements
-    export ELEMENTS_PATH+=:$PATH_TRIPLEO_ELEMENTS/elements
-    export DIB_APT_CONF_DIR=/etc/apt/apt.conf.d
-    export DIB_CLOUD_INIT_ETC_HOSTS=true
-    local QEMU_IMG_OPTIONS="--qemu-img-options compat=1.1"
-
-    # run disk-image-create that actually causes the image to be built
-    $disk-image-create -a amd64 -o "${VM}" \
-        -x ${QEMU_IMG_OPTIONS} ${DISTRO} ${EXTRA_ELEMENTS} vm \
-        cloud-init-datasources ${DISTRO}-guest ${DISTRO}-${SERVICE_TYPE}
-
-Several of the environment variables referenced above are referenced
-in the course of the Disk Image Building process.
-
-For example, let's look at GUEST_LOGDIR. Looking at the element
-elements/fedora-guest/extra-data.d/20-guest-systemd, we find:
-
-.. code-block:: bash
-
-        #!/bin/bash
-
-        set -e
-        set -o xtrace
-
-        # CONTEXT: HOST prior to IMAGE BUILD as SCRIPT USER
-        # PURPOSE: stages the bootstrap file and upstart conf file while replacing variables so that guest image is properly
-        # configured
-
-        source $_LIB/die
-
-        [ -n "$TMP_HOOKS_PATH" ] || die "Temp hook path not set"
-
-        [ -n "${GUEST_USERNAME}" ] || die "GUEST_USERNAME needs to be set to the user for the guest image"
-        [ -n "${HOST_SCP_USERNAME}" ] || die "HOST_SCP_USERNAME needs to be set to the user for the host instance"
-        [ -n "${CONTROLLER_IP}" ] || die "CONTROLLER_IP needs to be set to the ip address that guests will use to contact the controller"
-        [ -n "${ESCAPED_PATH_TROVE}" ] || die "ESCAPED_PATH_TROVE needs to be set to the path to the trove directory on the trovestack host"
-        [ -n "${TROVESTACK_SCRIPTS}" ] || die "TROVESTACK_SCRIPTS needs to be set to the trove/integration/scripts dir"
-        [ -n "${ESCAPED_GUEST_LOGDIR}" ] || die "ESCAPED_GUEST_LOGDIR must be set to the escaped guest log dir"
-
-        sed "s/GUEST_USERNAME/${GUEST_USERNAME}/g;s/GUEST_LOGDIR/${ESCAPED_GUEST_LOGDIR}/g;s/HOST_SCP_USERNAME/${HOST_SCP_USERNAME}/g;s/CONTROLLER_IP/${CONTROLLER_IP}/g;s/PATH_TROVE/${ESCAPED_PATH_TROVE}/g" \
-        ${TROVESTACK_SCRIPTS}/files/trove-guest.systemd.conf >
-        ${TMP_HOOKS_PATH}/trove-guest.service
-
-As you can see, the value of GUEST_LOGDIR is used in the extra-data.d
-script to appropriately configure the trove-guest.systemd.conf file.
-
-This pattern is one that you can expect in your own building of guest
-images.  The invocation of disk-image-create provides a list of
-elements that are to be invoked 'in order'.
-
-That list of elements is:
-
-.. code-block:: bash
-
-         ${DISTRO}
-         ${EXTRA_ELEMENTS}
-         vm
-         cloud-init-datasources
-         ${DISTRO}-guest
-         ${DISTRO}-${SERVICE_TYPE}
-
-When invoked to (for example) create a MySQL guest image on Ubuntu, we
-can expect that DISTRO would be 'Ubuntu' and SERVICE_TYPE would be
-MySQL. And therefore these would end up being the elements:
-
-.. code-block:: bash
-
-  ubuntu                        From diskimage-builder/elements/ubuntu
-  vm                            From diskimage-builder/elements/vm
-  cloud-init-datasources        From diskimage-builder/elements/cloud-init-datasources
-  ubuntu-guest                  From trove/integration/scripts/files/elements/ubuntu-guest
-  ubuntu-mysql                  From trove/integration/scripts/files/elements/ubuntu-mysql
-
-References
-==========
-
-.. [1] For more information about QEMU, refer to http://wiki.qemu.org/Main_Page
-.. [2] On Ubuntu, qemu-img is part of the package qemu-utils, on Fedora and RedHat it is part of the qemu package.
-.. [3] User (especially in the USA) are cautioned about this spelling which once resulted in several sleepless nights.
-.. [4] https://git.openstack.org/cgit/openstack/diskimage-builder/tree/README.rst#writing-an-element
-.. [5] https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/elements
-.. [6] https://git.openstack.org/cgit/openstack/trove/tree/integration/scripts/files/elements