diff options
author | Zach Marano <zmarano@google.com> | 2017-08-21 14:36:46 -0700 |
---|---|---|
committer | GitHub <noreply@github.com> | 2017-08-21 14:36:46 -0700 |
commit | 6906ad03cae98f5f5f7ddc2d3324af4cbe357f14 (patch) | |
tree | d5c9c3240ba7e42231bbe8c3e08102defd715c76 | |
parent | 35be86d29b9c73d530cb623f286bcc8d4ed116f4 (diff) | |
download | google-compute-image-packages-6906ad03cae98f5f5f7ddc2d3324af4cbe357f14.tar.gz |
New packages, package builds, and documentation updates. v2.6.0 (#463)
* Debian and RPM spec's for new packages along with build workflows.
* Update README, bump new packages to 2.6.0, remove old fpm package
scripts.
* Update the "Versions" section of the readme.
* Python is capitalized.
* Improved wording and formatting.
* Improved style of the workflow README.
* Remove trailing new line.
* Remove trailing new line.
-rw-r--r-- | README.md | 97 | ||||
-rwxr-xr-x | build_packages.sh | 105 | ||||
-rw-r--r-- | debian/changelog | 5 | ||||
-rw-r--r-- | debian/compat | 1 | ||||
-rw-r--r-- | debian/control | 69 | ||||
-rw-r--r-- | debian/copyright | 27 | ||||
-rw-r--r-- | debian/google-compute-engine.install | 4 | ||||
-rw-r--r-- | debian/rules | 31 | ||||
-rw-r--r-- | debian/source/format | 1 | ||||
-rwxr-xr-x | google_compute_engine_init/build_packages.sh | 123 | ||||
-rwxr-xr-x | google_config/build_packages.sh | 112 | ||||
-rw-r--r-- | packaging/README.md | 37 | ||||
-rw-r--r-- | packaging/build_deb_packages.sh | 64 | ||||
-rw-r--r-- | packaging/build_debian.wf.json | 71 | ||||
-rw-r--r-- | packaging/build_el6.wf.json | 71 | ||||
-rw-r--r-- | packaging/build_el7.wf.json | 71 | ||||
-rw-r--r-- | packaging/build_el_packages.sh | 65 | ||||
-rwxr-xr-x | setup.py | 2 | ||||
-rw-r--r-- | specs/google-compute-engine.spec | 150 | ||||
-rw-r--r-- | specs/python-google-compute-engine.spec | 54 |
20 files changed, 786 insertions, 374 deletions
@@ -24,6 +24,7 @@ Compute Engine [images](https://cloud.google.com/compute/docs/images). * [Network Setup](#network-setup) * [Configuration](#configuration) * [Packaging](#packaging) +* [Version Updates](#version-updates) * [Package Distribution](#package-distribution) * [Troubleshooting](#troubleshooting) * [Contributing](#contributing) @@ -275,14 +276,12 @@ provided Python versions. Distro | Package Type | Python Version | Init System ------------ | ------------ | -------------- | ----------- -Debian 7 | deb | 2.7 | sysvinit Debian 8 | deb | 2.7 | systemd -Debian 9 | deb | 2.7 | systemd +Debian 9 | deb | 3.5 or 2.7 | systemd CentOS 6 | rpm | 2.6 | upstart CentOS 7 | rpm | 2.7 | systemd RHEL 6 | rpm | 2.6 | upstart RHEL 7 | rpm | 2.7 | systemd -Ubuntu 12.04 | deb | 2.7 | upstart Ubuntu 14.04 | deb | 2.7 | upstart Ubuntu 16.04 | deb | 3.5 or 2.7 | systemd SLES 11 | rpm | 2.6 | sysvinit @@ -290,31 +289,58 @@ SLES 12 | rpm | 2.7 | systemd We build the following packages for the Linux guest environment. -* `google-compute-engine` is a Python package for Linux daemons, scripts, and - libraries. - * The package is installed to its distro default Python package location - (e.g. `/usr/lib/python2.7/site-packages`). - * Entry point scripts, created by the Python package, are located in - `/usr/bin`. -* `google-compute-engine-init` is a package that contains init configuration - for the `google-compute-engine` Python package. Installing this package - will configure the `google-compute-engine` package to run on system - startup on sysvinit, upstart, or systemd init systems. -* `google-config` is a package containing non-Python scripts and guest - configuration. - * Sets up udev rules and sysctl rules. - * Configures the SysLog output that gets sent to serial port output. - * Includes bash scripts needed by `instance_setup`. - -The package build tools are published in this project. +* `google-compute-engine` + * System init scripts (systemd, upstart, or sysvinit). + * Includes udev rules, sysctl rules, rsyslog configs, dhcp configs for + hostname setting. + * Entry point scripts created by the Python package located in `/usr/bin`. + * Includes bash scripts used by `instance_setup`. +* `python-google-compute-engine` + * The Python 2 package for Linux daemons and libraries. +* `python3-google-compute-engine` + * The Python 3 package for Linux daemons and libraries. + +The package source for Debian and RPM specs for Enterprise Linux 6 and 7 are +included in this project. There are also +[Daisy](https://github.com/GoogleCloudPlatform/compute-image-tools/tree/master/daisy) +workflows for spinning up GCE VM's to automatically build the packages for +Debian, Red Hat, and CentOS. See the [README](packaging/README.md) in the +packaging directory for more details. + +## Version Updates + +There are several places where package versions have to be updated and must +match to successfully release an update. + +* `setup.py` Update the version string with the Python package version. Used + for entry points through the Python egg and PyPI. +* `specs/google-compute-engine.spec` Update the version of the + `google-compute-engine` package for EL6 and EL7. +* `specs/python-google-compute-engine.spec` Update the version string of the + `python-google-compute-engine` package for EL6 and EL7. +* `debian/changelog` Update `google-compute-image-packages (VERSION) stable`, + the version of the Debian packages. +* Update the variable `package_version` when invoking the package build workflows. ## Package Distribution The deb and rpm packages used in some GCE images are published to Google Cloud -repositories. Debian 8, Debian 9, CentOS 6 and 7, and RHEL 6 and 7 use these -repositories to install and update the `google-compute-engine`, -`google-compute-engine-init`, and `google-config` packages. If you are creating -a custom image, you can also use these repositories in your image. +repositories. Debian 8 and 9, CentOS 6 and 7, and RHEL 6 and 7 use these +repositories to install and update the `google-compute-engine`, and +`python-google-compute-engine` (and `python3-google-compute-engine` for Python 3) +packages. If you are creating a custom image, you can also use these repositories +in your image. + +The following older packages will be replaced by the new packages: + +* `google-compute-engine` (`-jessie`, or `-stretch` for Debian) are replaced + by `google-compute-engine` and `python-google-compute-engine`. +* `google-compute-engine-init` (`-jessie`, or `-stretch` for Debian) are + replaced by `google-compute-engine`. +* `google-config` (`-jessie`, or `-stretch` for Debian) are replaced by + `google-compute-engine`. +* `google-compute-daemon` is replaced by `python-google-compute-engine`. +* `google-startup-scripts` is replaced by `google-compute-engine` **For Debian 8, run the following commands as root:** @@ -336,10 +362,9 @@ Install the packages to maintain the public key over time: apt-get update; apt-get install google-cloud-packages-archive-keyring ``` -Install the `google-compute-engine-jessie`, -`google-compute-engine-init-jessie`, and `google-config-jessie` packages: +Install the `google-compute-engine` and `python-google-compute-engine` packages: ``` -apt-get update; apt-get install -y google-config-jessie google-compute-engine-jessie google-compute-engine-init-jessie +apt-get update; apt-get install -y google-compute-engine python-google-compute-engine ``` **For Debian 9, run the following commands as root:** @@ -362,10 +387,9 @@ Install the packages to maintain the public key over time: apt-get update; apt-get install google-cloud-packages-archive-keyring ``` -Install the `google-compute-engine-stretch`, -`google-compute-engine-init-stretch`, and `google-config-stretch` packages: +Install the `google-compute-engine` and `python-google-compute-engine` packages: ``` -apt-get update; apt-get install -y google-config-stretch google-compute-engine-stretch google-compute-engine-init-stretch +apt-get update; apt-get install -y google-compute-engine python-google-compute-engine ``` **For EL6 and EL7 based distributions, run the following commands as root:** @@ -386,10 +410,9 @@ gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg EOM ``` -Install the `google-compute-engine`, `google-compute-engine-init`, and -`google-config` packages: +Install the `google-compute-engine`, `python-google-compute-engine` packages: ``` -yum install -y google-compute-engine google-compute-engine-init google-config +yum install -y google-compute-engine python-google-compute-engine ``` ## Troubleshooting @@ -405,6 +428,14 @@ http://mirror.centos.org/centos/6/SCL/x86_64/repodata/repomd.xml: [Errno 14] PYC Remove the stale repository file: `sudo rm -f /etc/yum.repos.d/CentOS-SCL.repo` +**On some CentOS or RHEL 6 systems, extraneous python egg directories can cause +the python daemons to fail.** + +In `/usr/lib/python2.6/site-packages` look for +`google_compute_engine-2.4.1-py27.egg-info` directories and +`google_compute_engine-2.5.2.egg-info` directories and delete them if you run +into this problem. + **Using boto with virtualenv** Specific to running `boto` inside of a Python diff --git a/build_packages.sh b/build_packages.sh deleted file mode 100755 index a2c5cef..0000000 --- a/build_packages.sh +++ /dev/null @@ -1,105 +0,0 @@ -#!/bin/bash -# Copyright 2016 Google Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -#/ Usage: build_packages.sh [options] -#/ -#/ Build the Python package for Linux daemons, scripts, and libraries. -#/ -#/ OPTIONS: -#/ -h Show this message -#/ -o DISTRO,... Build only specified distros - -function usage() { - grep '^#/' < "$0" | cut -c 4- -} - -function build_distro() { - declare -r distro="$1" - declare -r pkg_type="$2" - declare -r py_path="$3" - declare depends='google-compute-engine-init' - declare name='google-compute-engine' - - export CONFIG="${init_config}" - - if [[ "${pkg_type}" == 'deb' ]]; then - depends="${depends}-${distro}" - name="${name}-${distro}" - fi - - fpm \ - -s python \ - -t "${pkg_type}" \ - --depends "${depends}" \ - --depends 'python-boto' \ - --depends 'python-setuptools' \ - --iteration "0.${TIMESTAMP}" \ - --maintainer 'gc-team@google.com' \ - --name "${name}" \ - --no-python-fix-name \ - --python-install-bin '/usr/bin' \ - --python-install-lib "${py_path}" \ - --python-install-data "/usr/share/doc/${name}" \ - --rpm-dist "${distro}" \ - setup.py -} - -TIMESTAMP="$(date +%s)" - -while getopts 'ho:' OPTION; do - case "$OPTION" in - h) - usage - exit 2 - ;; - o) - set -f - IFS=',' - BUILD=($OPTARG) - set +f - ;; - ?) - usage - exit - ;; - esac -done - -if [ -z "$BUILD" ]; then - BUILD=('el6' 'el7' 'jessie' 'stretch') -fi - -for build in "${BUILD[@]}"; do - case "$build" in - el6) # RHEL/CentOS 6 - build_distro 'el6' 'rpm' '/usr/lib/python2.6/site-packages' - ;; - el7) # RHEL/CentOS 7 - build_distro 'el7' 'rpm' '/usr/lib/python2.7/site-packages' - ;; - wheezy) # Debian 7 - build_distro 'wheezy' 'deb' '/usr/lib/python2.7/dist-packages' - ;; - jessie) # Debian 8 - build_distro 'jessie' 'deb' '/usr/lib/python2.7/dist-packages' - ;; - stretch) # Debian 9 - build_distro 'stretch' 'deb' '/usr/lib/python2.7/dist-packages' - ;; - *) - echo "Invalid build '${build}'. Use 'el6', 'el7', 'wheezy', 'jessie', or 'stretch'." - ;; - esac -done diff --git a/debian/changelog b/debian/changelog new file mode 100644 index 0000000..c89fa10 --- /dev/null +++ b/debian/changelog @@ -0,0 +1,5 @@ +google-compute-image-packages (2.6.0-1) stable; urgency=low + + * New packaging. + + -- Google Cloud Team <gc-team@google.com> Mon, 27 Jun 2017 12:00:00 -0700 diff --git a/debian/compat b/debian/compat new file mode 100644 index 0000000..ec63514 --- /dev/null +++ b/debian/compat @@ -0,0 +1 @@ +9 diff --git a/debian/control b/debian/control new file mode 100644 index 0000000..1a9daf5 --- /dev/null +++ b/debian/control @@ -0,0 +1,69 @@ +Source: google-compute-image-packages +Section: admin +Priority: extra +Maintainer: Google Cloud Team <gc-team@google.com> +Build-Depends: debhelper (>= 9), + dh-python, + dh-systemd (>= 1.5), + python-all, + python-setuptools, + python3-all, + python3-setuptools, + python-pytest, + python3-pytest, + python-mock, + python-boto, + python3-boto +Standards-Version: 3.9.8 +Homepage: https://github.com/GoogleCloudPlatform/compute-image-packages + +Package: google-compute-engine +Architecture: all +Depends: python:any, + python3:any, + ${python:Depends}, + ${python3:Depends}, + ${misc:Depends}, + python-google-compute-engine (= ${source:Version}), + python3-google-compute-engine (= ${source:Version}), + ntp, + systemd +Recommends: google-cloud-sdk +Conflicts: google-compute-engine-jessie, + google-compute-engine-init-jessie, + google-config-jessie, + google-compute-engine-stretch, + google-compute-engine-init-stretch, + google-config-stretch, + google-compute-daemon, + google-startup-scripts +Replaces: google-compute-engine-jessie, + google-compute-engine-init-jessie, + google-config-jessie, + google-compute-engine-stretch, + google-compute-engine-init-stretch, + google-config-stretch, + google-compute-daemon, + google-startup-scripts +Description: Google Compute Engine guest environment. + This package contains scripts, configuration, and systemd init files for features specific to the Google Compute Engine cloud environment. + +Package: python-google-compute-engine +Section: python +Architecture: all +Depends: ${python:Depends}, ${misc:Depends} +Conflicts: google-compute-engine-jessie, + google-compute-engine-stretch, + google-compute-daemon, + google-startup-scripts +Description: Google Compute Engine python library for Python 2.x. + +Package: python3-google-compute-engine +Section: python +Architecture: all +Depends: ${python3:Depends}, ${misc:Depends} +Conflicts: google-compute-engine-jessie, + google-compute-engine-stretch, + google-compute-daemon, + google-startup-scripts +Description: Google Compute Engine python library for Python 3.x. diff --git a/debian/copyright b/debian/copyright new file mode 100644 index 0000000..9f4356b --- /dev/null +++ b/debian/copyright @@ -0,0 +1,27 @@ +Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/ +Upstream-Name: google-compute-engine +Upstream-Contact: gc-team@google.com + +Files: * +Copyright: Copyright 2017 Google Inc. +License: Apache-2.0 + +Files: debian/* +Copyright: Copyright 2017 Google Inc. +License: Apache-2.0 + +License: Apache-2.0 + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + . + http://www.apache.org/licenses/LICENSE-2.0 + . + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + . + On Debian systems, the complete text of the Apache version 2.0 license + can be found in "/usr/share/common-licenses/Apache-2.0". diff --git a/debian/google-compute-engine.install b/debian/google-compute-engine.install new file mode 100644 index 0000000..2b4b2bc --- /dev/null +++ b/debian/google-compute-engine.install @@ -0,0 +1,4 @@ +google_compute_engine_init/systemd/*.service lib/systemd/system +google_config/rsyslog/* etc/rsyslog.d +google_config/sysctl/* etc/sysctl.d +google_config/udev/* lib/udev/rules.d diff --git a/debian/rules b/debian/rules new file mode 100644 index 0000000..46ac69b --- /dev/null +++ b/debian/rules @@ -0,0 +1,31 @@ +#!/usr/bin/make -f + +export PYBUILD_NAME=google-compute-engine +export PYBUILD_TEST_PYTEST=1 +export PYBUILD_TEST_ARGS={dir}/google_compute_engine/ +export PYBUILD_SYSTEM=distutils + +%: + dh $@ --with python2,python3,systemd --buildsystem=pybuild + +override_dh_systemd_start: + dh_systemd_start --no-restart-on-upgrade --no-start --name=google-startup-scripts + dh_systemd_start --no-restart-on-upgrade --no-start --name=google-shutdown-scripts + +override_dh_python3: + dh_python3 + + # Remove the Python 3 entry scripts. + rm -rf debian/python3-google-compute-engine/usr/bin + +override_dh_python2: + dh_python2 + + # Split the Python 2 entry script to the google-compute-engine package. + mkdir -p debian/google-compute-engine/usr/bin + mv debian/python-google-compute-engine/usr/bin/* debian/google-compute-engine/usr/bin/ + rm -rf debian/python-google-compute-engine/usr/bin + +override_dh_clean: + rm -rf google_compute_engine.egg-info + dh_clean diff --git a/debian/source/format b/debian/source/format new file mode 100644 index 0000000..163aaf8 --- /dev/null +++ b/debian/source/format @@ -0,0 +1 @@ +3.0 (quilt) diff --git a/google_compute_engine_init/build_packages.sh b/google_compute_engine_init/build_packages.sh deleted file mode 100755 index 1b46916..0000000 --- a/google_compute_engine_init/build_packages.sh +++ /dev/null @@ -1,123 +0,0 @@ -#!/bin/bash -# Copyright 2016 Google Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -#/ Usage: build_packages.sh [options] -#/ -#/ Build the package that contains init configuration for the -#/ google-compute-engine Python package. -#/ -#/ OPTIONS: -#/ -h Show this message -#/ -o DISTRO,... Build only specified distros - -function usage() { - grep '^#/' < "$0" | cut -c 4- -} - -function build_distro() { - declare -r distro="$1" - declare -r pkg_type="$2" - declare -r init_config="$3" - declare -r init_prefix="$4" - declare py_depends='google-compute-engine' - declare config_depends='google-config' - declare file_pattern='*[^.sh]' - declare init_files=(${init_config}/${file_pattern}) - declare name='google-compute-engine-init' - - for ((i=0; i<${#init_files[@]}; i++)); do - file_name="$(basename ${init_files[$i]})" - file_entry="${init_config}/${file_name}=${init_prefix}/${file_name}" - init_files[$i]="$file_entry" - done - - if [[ "${pkg_type}" == 'deb' ]]; then - py_depends="${py_depends}-${distro}" - config_depends="${config_depends}-${distro}" - name="${name}-${distro}" - fi - - fpm \ - -s dir \ - -t "${pkg_type}" \ - --after-install "${init_config}/postinst.sh" \ - --before-remove "${init_config}/prerm.sh" \ - --depends "${py_depends}" \ - --depends "${config_depends}" \ - --description 'Google Compute Engine Linux initialization scripts' \ - --iteration "0.${TIMESTAMP}" \ - --license 'Apache Software License' \ - --maintainer 'gc-team@google.com' \ - --name "${name}" \ - --replaces 'gce-daemon' \ - --replaces 'gce-startup-scripts' \ - --replaces 'google-compute-daemon' \ - --replaces 'google-startup-scripts' \ - --rpm-dist "${distro}" \ - --rpm-trigger-after-target-uninstall \ - "google-compute-daemon: ${init_config}/rpm_replace" \ - --url 'https://github.com/GoogleCloudPlatform/compute-image-packages' \ - --vendor 'Google Compute Engine Team' \ - --version '2.1.2' \ - "${init_files[@]}" -} - -TIMESTAMP="$(date +%s)" - -while getopts 'ho:' OPTION; do - case "$OPTION" in - h) - usage - exit 2 - ;; - o) - set -f - IFS=',' - BUILD=($OPTARG) - set +f - ;; - ?) - usage - exit - ;; - esac -done - -if [ -z "$BUILD" ]; then - BUILD=('el6' 'el7' 'jessie' 'stretch') -fi - -for build in "${BUILD[@]}"; do - case "$build" in - el6) # RHEL/CentOS 6 - build_distro 'el6' 'rpm' 'upstart' '/etc/init' - ;; - el7) # RHEL/CentOS 7 - build_distro 'el7' 'rpm' 'systemd' '/usr/lib/systemd/system' - ;; - wheezy) # Debian 7 - build_distro 'wheezy' 'deb' 'sysvinit' '/etc/init.d' - ;; - jessie) # Debian 8 - build_distro 'jessie' 'deb' 'systemd' '/usr/lib/systemd/system' - ;; - stretch) # Debian 9 - build_distro 'stretch' 'deb' 'systemd' '/usr/lib/systemd/system' - ;; - *) - echo "Invalid build '${build}'. Use 'el6', 'el7', 'wheezy', 'jessie', or 'stretch'." - ;; - esac -done diff --git a/google_config/build_packages.sh b/google_config/build_packages.sh deleted file mode 100755 index 1ebec14..0000000 --- a/google_config/build_packages.sh +++ /dev/null @@ -1,112 +0,0 @@ -#!/bin/bash -# Copyright 2016 Google Inc. All Rights Reserved. -# -# Licensed under the Apache License, Version 2.0 (the "License"); -# you may not use this file except in compliance with the License. -# You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -#/ Usage: build_packages.sh [options] -#/ -#/ Build the package containing non-Python scripts and guest configuration. -#/ -#/ OPTIONS: -#/ -h Show this message -#/ -o DISTRO,... Build only specified distros - -function usage() { - grep '^#/' < "$0" | cut -c 4- -} - -function build_distro() { - declare -r distro="$1" - declare -r pkg_type="$2" - declare files=("$@") - declare name='google-config' - - if [[ "${pkg_type}" == 'deb' ]]; then - name="${name}-${distro}" - fi - - fpm \ - -s dir \ - -t "${pkg_type}" \ - --description 'Google Compute Engine Linux guest configuration' \ - --iteration "0.${TIMESTAMP}" \ - --license 'Apache Software License' \ - --maintainer 'gc-team@google.com' \ - --name "${name}" \ - --rpm-dist "${distro}" \ - --url 'https://github.com/GoogleCloudPlatform/compute-image-packages' \ - --vendor 'Google Compute Engine Team' \ - --version '2.1.2' \ - "${COMMON_FILES[@]}" \ - "${files[@]:2}" -} - -COMMON_FILES=( - 'rsyslog/90-google.conf=/etc/rsyslog.d/90-google.conf' - 'sysctl/11-gce-network-security.conf=/etc/sysctl.d/11-gce-network-security.conf' - 'udev/64-gce-disk-removal.rules=/etc/udev/rules.d/64-gce-disk-removal.rules' - 'udev/65-gce-disk-naming.rules=/etc/udev/rules.d/65-gce-disk-naming.rules') -TIMESTAMP="$(date +%s)" - -while getopts 'ho:' OPTION; do - case "$OPTION" in - h) - usage - exit 2 - ;; - o) - set -f - IFS=',' - BUILD=($OPTARG) - set +f - ;; - ?) - usage - exit - ;; - esac -done - -if [ -z "$BUILD" ]; then - BUILD=('el6' 'el7' 'jessie' 'stretch') -fi - -for build in "${BUILD[@]}"; do - case "$build" in - el6) # RHEL/CentOS 6 - build_distro 'el6' 'rpm' \ - 'bin/set_hostname=/etc/dhcp/dhclient-exit-hooks' \ - 'sbin/google-dhclient-script=/sbin/google-dhclient-script' - ;; - el7) # RHEL/CentOS 7 - build_distro 'el7' 'rpm' \ - 'bin/set_hostname=/usr/bin/set_hostname' \ - 'dhcp/google_hostname.sh=/etc/dhcp/dhclient.d/google_hostname.sh' - ;; - wheezy) # Debian 7 - build_distro 'wheezy' 'deb' \ - 'bin/set_hostname=/etc/dhcp/dhclient-exit-hooks.d/set_hostname' - ;; - jessie) # Debian 8 - build_distro 'jessie' 'deb' \ - 'bin/set_hostname=/etc/dhcp/dhclient-exit-hooks.d/set_hostname' - ;; - stretch) # Debian 9 - build_distro 'stretch' 'deb' \ - 'bin/set_hostname=/etc/dhcp/dhclient-exit-hooks.d/set_hostname' - ;; - *) - echo "Invalid build '${build}'. Use 'el6', 'el7', 'wheezy', 'jessie', or 'stretch'." - ;; - esac -done diff --git a/packaging/README.md b/packaging/README.md new file mode 100644 index 0000000..c7c59a6 --- /dev/null +++ b/packaging/README.md @@ -0,0 +1,37 @@ +## Daisy Workflows for building packages. + +For more information on Daisy and how workflows work, refer to the +[Daisy documentation](https://github.com/GoogleCloudPlatform/compute-image-tools/tree/master/daisy). + +# Workflow invocation + +```shell +# Builds Debian packages. +./daisy -project YOUR_PROJECT \ + -zone ZONE \ + -gcs_path YOUR_GCS_PATCH \ + -variables package_version=2.6.0 \ + build_debian.wf.json + +# Builds EL6 packages. +./daisy -project YOUR_PROJECT \ + -zone ZONE \ + -gcs_path YOUR_GCS_PATCH \ + -variables package_version=2.6.0 \ + build_el6.wf.json + +# Builds EL7 packages. +./daisy -project YOUR_PROJECT \ + -zone ZONE \ + -gcs_path YOUR_GCS_PATCH \ + -variables package_version=2.6.0 \ + build_el7.wf.json +``` + +# Variables + +* `output_path` Specify a different GCS path to save resulting packages to. +* `github_repo` Specify a different github repo (for example a forked repo). +* `github_branch` Specify a different github branch. +* `package_version` The version of the package. This version has to match the + version of the python `setup.py` files, spec files, and Debian changelog. diff --git a/packaging/build_deb_packages.sh b/packaging/build_deb_packages.sh new file mode 100644 index 0000000..80381da --- /dev/null +++ b/packaging/build_deb_packages.sh @@ -0,0 +1,64 @@ +#!/bin/bash +# Copyright 2017 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +URL="http://metadata/computeMetadata/v1/instance/attributes" +BRANCH="$(curl -f -H Metadata-Flavor:Google ${URL}/github_branch)" +GIT_REPO="$(curl -f -H Metadata-Flavor:Google ${URL}/github_repo)" +VERSION="$(curl -f -H Metadata-Flavor:Google ${URL}/package_version)" +OUTPUT="$(curl -f -H Metadata-Flavor:Google ${URL}/output_path)" +DEB_VER="$(cut -d "." -f 1 /etc/debian_version)" + +if [ -z ${OUTPUT} ]; then + OUTPUT="$(curl -f -H Metadata-Flavor:Google ${URL}/daisy-outs-path)" +fi + +# Install build dependencies. +apt-get update +apt-get -y install git debhelper devscripts dh-python dh-systemd \ + python-all python-boto python-setuptools python-pytest python-mock \ + python3-all python3-boto python3-setuptools python3-pytest python3-mock \ + libcurl4-openssl-dev libjson-c-dev libpam-dev build-essential +if [ $? -ne 0 ]; then + echo "BuildFailed: Unable to install build dependencies." + exit 1 +fi + +# Clone the github repo. +git clone ${GIT_REPO} -b ${BRANCH} +if [ $? -ne 0 ]; then + echo "BuildFailed: Unable to clone github repo ${GIT_REPO} and branch ${BRANCH}" + exit 1 +fi + +# Create tar's for package builds. +tar -czvf google-compute-image-packages_${VERSION}.orig.tar.gz --exclude .git compute-image-packages + +# Build the deb's +pushd compute-image-packages +debuild -us -uc +if [ $? -ne 0 ]; then + echo "BuildFailed: debuild failed." + exit 1 +fi +popd + +# Copy the deb and dsc files to the output. +gsutil cp *.dsc *.deb ${OUTPUT}/ +if [ $? -ne 0 ]; then + echo "BuildFailed: copying to ${OUTPUT} failed." + exit 1 +fi + +echo "BuildSuccess: Packages are in ${OUTPUT}." diff --git a/packaging/build_debian.wf.json b/packaging/build_debian.wf.json new file mode 100644 index 0000000..f3fad62 --- /dev/null +++ b/packaging/build_debian.wf.json @@ -0,0 +1,71 @@ +{ + "Name": "build-debian-packages", + "Vars": { + "output_path": { + "Value": "${OUTSPATH}", + "Description": "GCS output path for packages." + }, + "github_repo": { + "Value": "https://github.com/GoogleCloudPlatform/compute-image-packages.git", + "Description": "Github repo to build packages from." + }, + "github_branch": { + "Value": "master", + "Description": "Github branch to build packages from." + }, + "package_version": { + "Required": true, + "Description": "The package version to build." + } + }, + "Sources": { + "build_deb_packages.sh": "./build_deb_packages.sh" + }, + "Steps": { + "setup-disk": { + "CreateDisks": [ + { + "Name": "disk-deb-build", + "SourceImage": "projects/debian-cloud/global/images/family/debian-9", + "SizeGb": "10", + "Type": "pd-ssd" + } + ] + }, + "package-build": { + "CreateInstances": [ + { + "Name": "inst-deb-build", + "Disks": [ + {"Source": "disk-deb-build"} + ], + "MachineType": "n1-standard-2", + "Metadata": { + "github_branch": "${github_branch}", + "github_repo": "${github_repo}", + "output_path": "${output_path}", + "package_version": "${package_version}" + }, + "Scopes": ["https://www.googleapis.com/auth/devstorage.read_write"], + "StartupScript": "build_deb_packages.sh" + } + ] + }, + "wait-for-build": { + "WaitForInstancesSignal": [ + { + "Name": "inst-deb-build", + "SerialOutput": { + "Port": 1, + "SuccessMatch": "BuildSuccess:", + "FailureMatch": "BuildFailed:" + } + } + ] + } + }, + "Dependencies": { + "package-build": ["setup-disk"], + "wait-for-build": ["package-build"] + } +} diff --git a/packaging/build_el6.wf.json b/packaging/build_el6.wf.json new file mode 100644 index 0000000..07e8471 --- /dev/null +++ b/packaging/build_el6.wf.json @@ -0,0 +1,71 @@ +{ + "Name": "build-el6-packages", + "Vars": { + "output_path": { + "Value": "${OUTSPATH}", + "Description": "GCS output path for packages." + }, + "github_repo": { + "Value": "https://github.com/GoogleCloudPlatform/compute-image-packages.git", + "Description": "Github repo to build packages from." + }, + "github_branch": { + "Value": "master", + "Description": "Github branch to build packages from." + }, + "package_version": { + "Required": true, + "Description": "The package version to build." + } + }, + "Sources": { + "build_el_packages.sh": "./build_el_packages.sh" + }, + "Steps": { + "setup-disk": { + "CreateDisks": [ + { + "Name": "disk-el6-build", + "SourceImage": "projects/centos-cloud/global/images/family/centos-6", + "SizeGb": "10", + "Type": "pd-ssd" + } + ] + }, + "package-build": { + "CreateInstances": [ + { + "Name": "inst-el6-build", + "Disks": [ + {"Source": "disk-el6-build"} + ], + "MachineType": "n1-standard-2", + "Metadata": { + "github_branch": "${github_branch}", + "github_repo": "${github_repo}", + "output_path": "${output_path}", + "package_version": "${package_version}" + }, + "Scopes": ["https://www.googleapis.com/auth/devstorage.read_write"], + "StartupScript": "build_el_packages.sh" + } + ] + }, + "wait-for-build": { + "WaitForInstancesSignal": [ + { + "Name": "inst-el6-build", + "SerialOutput": { + "Port": 1, + "SuccessMatch": "BuildSuccess:", + "FailureMatch": "BuildFailed:" + } + } + ] + } + }, + "Dependencies": { + "package-build": ["setup-disk"], + "wait-for-build": ["package-build"] + } +} diff --git a/packaging/build_el7.wf.json b/packaging/build_el7.wf.json new file mode 100644 index 0000000..10700b6 --- /dev/null +++ b/packaging/build_el7.wf.json @@ -0,0 +1,71 @@ +{ + "Name": "build-el7-packages", + "Vars": { + "output_path": { + "Value": "${OUTSPATH}", + "Description": "GCS output path for packages." + }, + "github_repo": { + "Value": "https://github.com/GoogleCloudPlatform/compute-image-packages.git", + "Description": "Github repo to build packages from." + }, + "github_branch": { + "Value": "master", + "Description": "Github branch to build packages from." + }, + "package_version": { + "Required": true, + "Description": "The package version to build." + } + }, + "Sources": { + "build_el_packages.sh": "./build_el_packages.sh" + }, + "Steps": { + "setup-disk": { + "CreateDisks": [ + { + "Name": "disk-el7-build", + "SourceImage": "projects/centos-cloud/global/images/family/centos-7", + "SizeGb": "10", + "Type": "pd-ssd" + } + ] + }, + "package-build": { + "CreateInstances": [ + { + "Name": "inst-el7-build", + "Disks": [ + {"Source": "disk-el7-build"} + ], + "MachineType": "n1-standard-2", + "Metadata": { + "github_branch": "${github_branch}", + "github_repo": "${github_repo}", + "output_path": "${output_path}", + "package_version": "${package_version}" + }, + "Scopes": ["https://www.googleapis.com/auth/devstorage.read_write"], + "StartupScript": "build_el_packages.sh" + } + ] + }, + "wait-for-build": { + "WaitForInstancesSignal": [ + { + "Name": "inst-el7-build", + "SerialOutput": { + "Port": 1, + "SuccessMatch": "BuildSuccess:", + "FailureMatch": "BuildFailed:" + } + } + ] + } + }, + "Dependencies": { + "package-build": ["setup-disk"], + "wait-for-build": ["package-build"] + } +} diff --git a/packaging/build_el_packages.sh b/packaging/build_el_packages.sh new file mode 100644 index 0000000..a91a468 --- /dev/null +++ b/packaging/build_el_packages.sh @@ -0,0 +1,65 @@ +#!/bin/bash +# Copyright 2017 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +URL="http://metadata/computeMetadata/v1/instance/attributes" +BRANCH="$(curl -f -H Metadata-Flavor:Google ${URL}/github_branch)" +GIT_REPO="$(curl -f -H Metadata-Flavor:Google ${URL}/github_repo)" +VERSION="$(curl -f -H Metadata-Flavor:Google ${URL}/package_version)" +OUTPUT="$(curl -f -H Metadata-Flavor:Google ${URL}/output_path)" + +if [ -z ${OUTPUT} ]; then + OUTPUT="$(curl -f -H Metadata-Flavor:Google ${URL}/daisy-outs-path)" +fi + +# Install build dependencies. +yum -y install git rpmdevtools \ + python2-devel python-setuptools python-boto \ + make gcc-c++ libcurl-devel json-c json-c-devel pam-devel \ + policycoreutils-python + +# Clone the github repo. +git clone ${GIT_REPO} -b ${BRANCH} +if [ $? -ne 0 ]; then + echo "BuildFailed: Unable to clone github repo ${GIT_REPO} and branch ${BRANCH}" + exit 1 +fi + +# Create tar's for package builds. +tar -czvf google-compute-engine_${VERSION}.orig.tar.gz --exclude .git compute-image-packages + +# Setup rpmbuild tree. +for d in BUILD BUILDROOT RPMS SOURCES SPECS SRPMS; do + mkdir -p /rpmbuild/$d +done +cp compute-image-packages/specs/*.spec /rpmbuild/SPECS/ +cp google-compute-engine_${VERSION}.orig.tar.gz /rpmbuild/SOURCES/ + +# Build the RPM's +for spec in $(ls /rpmbuild/SPECS/*.spec); do + rpmbuild --define "_topdir /rpmbuild" -ba $spec + if [ $? -ne 0 ]; then + echo "BuildFailed: rpmbuild for $spec failed." + exit 1 + fi +done + +# Copy the rpm and srpms to the output. +gsutil cp /rpmbuild/RPMS/*/*.rpm /rpmbuild/SRPMS/*.src.rpm ${OUTPUT}/ +if [ $? -ne 0 ]; then + echo "BuildFailed: copying to ${OUTPUT} failed." + exit 1 +fi + +echo "BuildSuccess: Packages are in ${OUTPUT}." @@ -32,7 +32,7 @@ setuptools.setup( packages=setuptools.find_packages(), scripts=glob.glob('scripts/*'), url='https://github.com/GoogleCloudPlatform/compute-image-packages', - version='2.5.2', + version='2.6.0', # Entry points create scripts in /usr/bin that call a function. entry_points={ 'console_scripts': [ diff --git a/specs/google-compute-engine.spec b/specs/google-compute-engine.spec new file mode 100644 index 0000000..fc67585 --- /dev/null +++ b/specs/google-compute-engine.spec @@ -0,0 +1,150 @@ +# Copyright 2017 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Force the dist to be el7 to avoid el7.centos. +%if 0%{?rhel} == 7 + %define dist .el7 +%endif + +Name: google-compute-engine +Version: 2.6.0 +Release: 1%{?dist} +Summary: Google Compute Engine guest environment. +License: ASL 2.0 +Url: https://github.com/GoogleCloudPlatform/compute-image-packages +Source0: %{name}_%{version}.orig.tar.gz + +BuildArch: noarch +BuildRequires: python2-devel python-setuptools python-boto +%if 0%{?el7} +BuildRequires: systemd +%endif + +Requires: python-setuptools +Requires: python-google-compute-engine +Requires: ntp +Requires: rsyslog +%if 0%{?el7} +Requires: systemd +%endif + +Obsoletes: google-compute-engine-init +Obsoletes: google-config +Obsoletes: google-startup-scripts +Conflicts: google-compute-engine-init +Conflicts: google-config +Conflicts: google-startup-scripts + +%description +This package contains scripts, configuration, and init files for features specific to the Google Compute Engine cloud environment. + +%prep +%autosetup -n compute-image-packages + +%install +mkdir -p %{buildroot}%{_bindir} +mkdir -p %{buildroot}/etc/dhcp +mkdir -p %{buildroot}/etc/rsyslog.d +mkdir -p %{buildroot}/etc/sysctl.d +mkdir -p %{buildroot}/etc/udev/rules.d + +cp google_config/rsyslog/90-google.conf %{buildroot}/etc/rsyslog.d/ +cp google_config/sysctl/11-gce-network-security.conf %{buildroot}/etc/sysctl.d/ +cp google_config/udev/*.rules %{buildroot}/etc/udev/rules.d/ + +# Install the python package to get the entry scripts. +python setup.py install --prefix=%{_prefix} --root %{buildroot} +rm -Rf %{buildroot}/usr/lib/python* + +%if 0%{?el6} +mkdir %{buildroot}/sbin +mkdir -p %{buildroot}/etc/init +cp google_compute_engine_init/upstart/*.conf %{buildroot}/etc/init/ +cp google_config/bin/set_hostname %{buildroot}/etc/dhcp/dhclient-exit-hooks +cp google_config/sbin/google-dhclient-script %{buildroot}/sbin/ +%endif + +%if 0%{?el7} +mkdir -p %{buildroot}/etc/dhcp/dhclient.d +mkdir -p %{buildroot}%{_unitdir} +cp google_compute_engine_init/systemd/*.service %{buildroot}%{_unitdir} +cp google_config/bin/set_hostname %{buildroot}%{_bindir} +cp google_config/dhcp/google_hostname.sh %{buildroot}/etc/dhcp/dhclient.d/google_hostname.sh +%endif + + +%files +%defattr(0644,root,root,0755) +%if 0%{?el6} +%attr(0755,root,root) /sbin/google-dhclient-script +%attr(0755,root,root) /etc/dhcp/dhclient-exit-hooks +/etc/init/*.conf +%endif +%if 0%{?el7} +%attr(0755,root,root) /etc/dhcp/dhclient.d/google_hostname.sh +%{_unitdir}/*.service +%endif +%config /etc/rsyslog.d/90-google.conf +%config /etc/sysctl.d/11-gce-network-security.conf +/etc/udev/rules.d/*.rules +%attr(0755,root,root) %{_bindir}/* + + +%post +%if 0%{?el6} +# On upgrade run instance setup again to handle any new configs and restart daemons. +if [ $1 -eq 2 ]; then + restart -q -n google-accounts-daemon + restart -q -n google-clock-skew-daemon + restart -q -n google-ip-forwarding-daemon + /usr/bin/google_instance_setup +fi +%endif + +%if 0%{?el7} +%systemd_post google-accounts-daemon.service +%systemd_post google-clock-skew-daemon.service +%systemd_post google-instance-setup.service +%systemd_post google-ip-forwarding-daemon.service +%systemd_post google-network-setup.service +%systemd_post google-shutdown-scripts.service +%systemd_post google-startup-scripts.service +# On upgrade run instance setup again to handle any new configs and restart daemons. +if [ $1 -eq 2 ]; then + systemctl reload-or-restart google-accounts-daemon.service + systemctl reload-or-restart google-clock-skew-daemon.service + systemctl reload-or-restart google-ip-forwarding-daemon.service + /usr/bin/google_instance_setup +fi +%endif + + +%preun +# On uninstall only. +if [ $1 -eq 0 ]; then +%if 0%{?el6} + stop -q -n google-accounts-daemon + stop -q -n google-clock-skew-daemon + stop -q -n google-ip-forwarding-daemon +%endif +%if 0%{?el7} + %systemd_preun google-accounts-daemon.service + %systemd_preun google-clock-skew-daemon.service + %systemd_preun google-instance-setup.service + %systemd_preun google-ip-forwarding-daemon.service + %systemd_preun google-network-setup.service + %systemd_preun google-shutdown-scripts.service + %systemd_preun google-startup-scripts.service +%endif +fi diff --git a/specs/python-google-compute-engine.spec b/specs/python-google-compute-engine.spec new file mode 100644 index 0000000..c4a915d --- /dev/null +++ b/specs/python-google-compute-engine.spec @@ -0,0 +1,54 @@ +# Copyright 2017 Google Inc. All Rights Reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Force the dist to be el7 to avoid el7.centos. +%if 0%{?rhel} == 7 + %define dist .el7 +%endif + +Name: python-google-compute-engine +Version: 2.6.0 +Release: 1%{?dist} +Summary: Google Compute Engine python library +License: ASL 2.0 +Url: https://github.com/GoogleCloudPlatform/compute-image-packages +Source0: google-compute-engine_%{version}.orig.tar.gz + +BuildArch: noarch +BuildRequires: python2-devel python-setuptools python-boto + +Requires: python-boto python-setuptools + +Provides: python2-google-compute-engine + +Obsoletes: google-compute-daemon +Obsoletes: google-startup-scripts +Conflicts: google-compute-daemon +Conflicts: google-startup-scripts + +%description +Google Compute Engine python library for Python 2.x. + +%prep +%autosetup -n compute-image-packages + +%build +python setup.py build + +%install +python setup.py install --prefix=%{_prefix} --root %{buildroot} +rm -Rf %{buildroot}/usr/bin + +%files +%{python_sitelib}/* |