diff options
| -rw-r--r-- | .gitreview | 5 | ||||
| -rw-r--r-- | Makefile | 7 | ||||
| -rw-r--r-- | README.mdwn | 12 | ||||
| -rw-r--r-- | changelog.mdwn | 7 | ||||
| -rw-r--r-- | data/favicon.ico | bin | 0 -> 709 bytes | |||
| -rw-r--r-- | deployment.mdwn | 120 | ||||
| -rw-r--r-- | index.mdwn | 42 | ||||
| -rwxr-xr-x | migrations/008-submodules-in-strata.py | 237 | ||||
| -rw-r--r-- | mkdocs.yml | 29 | ||||
| -rw-r--r-- | schemas/cluster.json-schema | 6 | ||||
| -rw-r--r-- | schemas/stratum.json-schema | 9 | ||||
| -rwxr-xr-x | scripts/yaml-jsonschema | 50 | ||||
| -rw-r--r-- | spec.mdwn | 465 |
13 files changed, 726 insertions, 263 deletions
diff --git a/.gitreview b/.gitreview new file mode 100644 index 0000000..99d9c4d --- /dev/null +++ b/.gitreview @@ -0,0 +1,5 @@ +[gerrit] +host=gerrit.baserock.org +port=29418 +project=baserock/baserock/spec +defaultbranch=master diff --git a/Makefile b/Makefile deleted file mode 100644 index 3cfeb6f..0000000 --- a/Makefile +++ /dev/null @@ -1,7 +0,0 @@ -# Makefile for Baserock definitions format specification - -# This is available in the 'python3-markdown2' package on Fedora. -MARKDOWN2 = markdown2-3.4 - -spec.html: spec.mdwn - $(MARKDOWN2) $< > $@ diff --git a/README.mdwn b/README.mdwn index c49202c..66702d5 100644 --- a/README.mdwn +++ b/README.mdwn @@ -13,6 +13,18 @@ The definitions format is documentated as a Markdown text file. See that text for more information about the Baserock definitions format. +The contents of the 'master' branch is rendered as HTML at +<http://docs.baserock.org/>. This is done using the 'mkdocs' static site +generator. + +If you want to try out changes to the HTML rendering locally, install +mkdocs and the required theme, then run `mkdocs serve` from the root +directory of the repository. For example: + + pip3.4 install --user mkdocs mkdocs-material + mkdocs serve + # Rendered content should now be visible at http://localhost:8000/ + JSON-Schemas ------------ diff --git a/changelog.mdwn b/changelog.mdwn index 82dbb52..b88a131 100644 --- a/changelog.mdwn +++ b/changelog.mdwn @@ -1,12 +1,7 @@ -[[!meta title="Baserock definitions format, version history"]] +Title: Baserock definitions format, version history This page describes the history of the Baserock definitions format. -FIXME: A changelog should no longer be needed now that the spec is -defined in a version control repository. - -See also: [[current]] version, and [[planned]] versions. - # Version 7 [Version 7](http://listmaster.pepperfish.net/pipermail/baserock-dev-baserock.org/2015-July/013220.html) adds a DEFAULTS file that specifies predefined build systems and predefined splitting rules. Previously it was up to the build tool to define these. diff --git a/data/favicon.ico b/data/favicon.ico Binary files differnew file mode 100644 index 0000000..bbc9bbd --- /dev/null +++ b/data/favicon.ico diff --git a/deployment.mdwn b/deployment.mdwn new file mode 100644 index 0000000..a7505fd --- /dev/null +++ b/deployment.mdwn @@ -0,0 +1,120 @@ +Deploying Baserock systems +========================== + +Building systems is quite a well-defined operation: source code goes in, a tree +of binary files comes out. + +Deployment is rather open-ended. Since a rootfs tarball isn't often useful +in itself, Baserock tools can follow the approach presented below to get a set +of built systems actually running somewhere. This might involve copying it to +a disk, creating an OpenStack instance or some other kind of virtual machine, +using the [Baserock upgrade +mechanism](http://wiki.baserock.org/guides/upgrades/) to upgrade an existing +deployment, or anything else. + +Within this document, consider "deployment" to be a process of first +post-processing a filesystem tree with one or more 'configure extensions', then +performing an operation to convert and/or transfer the filesystem tree using a +'write extension'. + +### Deployment sequence + +A Baserock deployment tool should, for each labelled deployment in the +'cluster', follow this sequence. + +1. Run the program '$type.check' (where $type is the value of the 'type' + field), if it exists. If it fails, exit now with an error. + + The .check extensions exist only to work around the fact that (2), (3) and + (4) may be slow, because programs that wait several minutes just to raise an + error that could have been detected right away are very annoying. + +2. Create a writeable temporary directory containing the contents of the system + artifact. + +3. For each 'subsystem' specified, recursively run steps 1, 3, 4 and 5. The + intention is that the result of the subsystem deployments end up *inside* + the temporary directory created in step 2 for the 'outer' deployment. + +4. For each path in the `configuration-extensions` field of the corresponding + system morphology, run `$path.configure`. The order of running these is, + sadly, unspecified. + +5. Run the .write extension for that deployment 'type'. + +The deployment of that system and its subsystems is now considered complete, +and temporary data can be removed. + +### Locating and running extensions + +The deployment extensions must live in the definitions repository. They can be +in a subdirectory, we recommend `extensions/`. + +> Previously extensions could and did live in the build tool's code. This was a +bad idea because we needed to define a new VERSION every time we changed +anything in the API of any deployment extension that the build tool provide, +and every new build tool that wanted to do deployment needed to reimplement or +duplicate every such extension. + +An extension must be executable using the [POSIX `exec()` system call]. We +encourage writing them as Python scripts and using a `#!/usr/bin/env python` +[hashbang], but any executable code is permitted. + +The 'execution environment' for an extension is sadly unspecified at the moment. +[Morph] runs extensions without any kid of chroot environment, but each one +is run in a separate [mount namespace]. Running extensions chrooted into the +system being deployed does not make sense, as it may not contain the right +dependencies for the extensions to work, and it may expect to run on a +different, incompatible architecture to the system running the deployment in +any case. + +A .check extension must be passed one commandline argument: the value of the +`location` or `upgrade-location` field. + +A .configure extension must be passed one commandline argument: the path to +the unpacked filesystem tree, which it can modify in some way. It is expected +that a .configure extension will do nothing unless it is enabled using a +configuration variable, but it is up to the code in the .configure extension +to do this, and this behaviour is not currently enforced. + +A .write extension must be passed two commandline arguments: the value of the +`location` or `upgrade-location` field, then the path to the unpacked +filesystem tree (which it could modify in some way). + +All key/value pairs from the 'cluster' definition for the given labelled +deployment of a system must be passed to all extensions, as environment +variables. The `type`, `location`, `upgrade-type` and `upgrade-location` fields +do not need to be passed in. + +Extensions are expected to set exit code 0 on success, and a non-zero value on +failure. If any extension returns non-zero, the deployment tool should abort +the deployment. + +Extensions are expected to write status information to 'stdout', and any error +messages to 'stderr'. This is for user feedback ONLY, deployment tools should +not do anything with this data other than log and/or display it. + +[Morph] sets an extra environment variable `MORPH_LOG_FD` which is a file +descriptor that the extension can write log messages to, in order for them to +appear in morph.log but not on stdout or stderr. + +[POSIX `exec()` system call]: http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html +[hashbang]: https://en.wikipedia.org/wiki/Shebang_%28Unix%29 +[mount namespace]: https://stackoverflow.com/questions/22889241/linux-understanding-the-mount-namespace-clone-clone-newns-flag#22889401 + +### Help for extensions + +The .configure and .write extensions may provide .help files as documentation. +The .help file should be valid [YAML], containing a dict with the key `help` +and a string value. + +For example, the `tar.write` extension could provide `tar.write.help` with the +following content: + + help: | + Deploy a system as a .tar file. + +### Common configuration parameters used by extensions + +Fill me in! + diff --git a/index.mdwn b/index.mdwn new file mode 100644 index 0000000..7f5dd63 --- /dev/null +++ b/index.mdwn @@ -0,0 +1,42 @@ +# Welcome to the Baserock Project's documentation! + +Baserock is about improving how we build, integrate and maintain collections of +software. + +For more information, see these pages on the wiki: + + - <http://wiki.baserock.org/overview/> + - <http://wiki.baserock.org/moving-parts/> + +## The Baserock definitions format + +The Baserock definitions format is a way of specifying components that +need to be built, deployed and distributed in some way. + +The current format is described in the [Spec](spec) section. + +Historical changes are listed in the [Changelog](changelog). + +To propose changes to the version, please submit patches against 'master' of +the [spec.git repo](http://git.baserock.org/cgit/baserock/baserock/spec.git). +See the [contributing](http://wiki.baserock.org/contributing/) page for +information. If you send a patch to Gerrit for a major change, please notify +the baserock-dev@baserock.org mailing list too. + +The wiki has more [information on the Definitions +format](http://wiki.baserock.org/definitions/), including a [comparison with +other +formats](http://wiki.baserock.org/definitions/comparison-with-other-formats), +and a [style guide](http://wiki.baserock.org/definitions/best-practices). + +## The Baserock reference definitions + +We maintain a collection of definitions describing how to build lots of popular +free software components. These are in a repository at +[git.baserock.org/baserock/baserock/definitions](http://git.baserock.org/cgit/baserock/baserock/definitions.git/tree/). + +When modifying or defining your own Baserock systems, normally you will 'fork' +this repository and use Git tooling to keep your changes in sync with ours. + +It's also possible to write definitions completely independently of ours, if +you want. diff --git a/migrations/008-submodules-in-strata.py b/migrations/008-submodules-in-strata.py new file mode 100755 index 0000000..a536852 --- /dev/null +++ b/migrations/008-submodules-in-strata.py @@ -0,0 +1,237 @@ +#!/usr/bin/env python +# Copyright (C) 2016 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see <http://www.gnu.org/licenses/>. + + +# THIS MIGRATION REQUIRES NETWORK ACCESS TO A BASEROCK GIT CACHE SERVER! If +# you do not have your own Trove, or don't know what a Trove is, it should +# work as-is, provided you have internet access that allows access to +# http://git.baserock.org:8080/. +# +# If you do have your own Trove, change the value of TROVE_HOST below to +# point to it. +# + +'''Migration to Baserock Definitions format version 8. + +In definitions version 8, submodules must be declared explicitly for all chunks +that contains a .gitmodules file in their root. This is so that mirrored source +repositories don't need to maintain branches that point to the mirrored +submodules, and can instead translate these at build time. + +''' + +import requests +import string +import logging +import re +import os +import sys +import warnings +import migrations +from subprocess import call, Popen +from ConfigParser import RawConfigParser +from StringIO import StringIO + + +TROVE_HOST = 'git.baserock.org' + +REPO_ALIASES = { + 'baserock:': 'git://%s/baserock/' % TROVE_HOST, + 'freedesktop:': 'git://anongit.freedesktop.org/', + 'github:': 'git://github.com/', + 'gnome:': 'git://git.gnome.org/', + 'upstream:': 'git://%s/delta/' % TROVE_HOST, +} + +GIT_CACHE_SERVER_URL = 'http://%s:8080/' % TROVE_HOST + +FAIL_ON_REMOTE_CACHE_ERRORS = False + + +TO_VERSION = 8 + + +# From ybd.git file repos.py at commit eb3bf397ba729387f0d4145a8df8d3c1f9eb707f + +def get_repo_url(repo): + for alias, url in REPO_ALIASES.items(): + repo = repo.replace(alias, url) + if repo.endswith('.git'): + repo = repo[:-4] + return repo + +def get_repo_name(repo): + ''' Convert URIs to strings that only contain digits, letters, _ and %. + NOTE: this naming scheme is based on what lorry uses + ''' + valid_chars = string.digits + string.ascii_letters + '%_' + transl = lambda x: x if x in valid_chars else '_' + return ''.join([transl(x) for x in get_repo_url(repo)]) + + +## End of code based on ybd repos.py + +def get_toplevel_file_list_from_repo(url, ref): + '''Try to list the set of files in the root directory of the repo at 'url'. + + ''' + try: + response = requests.get( + GIT_CACHE_SERVER_URL + '1.0/trees', + params={'repo': url, 'ref': ref}, + headers={'Accept': 'application/json'}, + timeout=9) + logging.debug("Got response: %s" % response) + try: + response.raise_for_status() + toplevel_tree = response.json()['tree'] + except Exception as e: + raise RuntimeError( + "Unexpected response from server %s for repo %s: %s" % + (GIT_CACHE_SERVER_URL, url, e.message)) + toplevel_filenames = toplevel_tree.keys() + except requests.exceptions.ConnectionError as e: + raise RuntimeError("Unable to connect to cache server %s while trying " + "to query file list of repo %s. Error was: %s" % + (GIT_CACHE_SERVER_URL, url, e.message)) + return toplevel_filenames + + +def validate_chunk_refs(contents, filename): + assert contents['kind'] == 'stratum' + + valid = True + for chunk_ref in contents.get('chunks', []): + if chunk_ref.get('morph') is None: + # No chunk .morph file -- this stratum was relying on build-system + # autodetection here. + + if 'repo' not in chunk_ref: + warnings.warn("%s: Chunk %s doesn't specify a source repo." % + (filename, chunk_ref.get('name'))) + valid = False + + if 'ref' not in chunk_ref: + warnings.warn("%s: Chunk %s doesn't specify a source ref." % + (filename, chunk_ref.get('name'))) + valid = False + return valid + + +def move_dict_entry_last(dict_object, key, error_if_missing=False): + '''Move an entry in a ordered dict to the end.''' + + # This is a hack, I couldn't find a method on the 'CommentedMap' type dict + # that we receive from ruamel.yaml that would allow doing this neatly. + if key in dict_object: + value = dict_object[key] + del dict_object[key] + dict_object[key] = value + else: + if error_if_missing: + raise KeyError(key) + +def submodules_to_dict(url, ref): + try: + response = requests.get( + GIT_CACHE_SERVER_URL + '1.0/files', + params={'repo': url, 'ref': ref, 'filename':'.gitmodules'}, + headers={'Accept': 'application/json'}, + timeout=9) + logging.debug("Got response: %s" % response) + try: + response.raise_for_status() + except Exception as e: + raise RuntimeError( + "Unexpected response from server %s for repo %s: %s" % + (GIT_CACHE_SERVER_URL, url, e.message)) + except requests.exceptions.ConnectionError as e: + raise RuntimeError("Unable to connect to cache server %s while trying " + "to read file '.gitmodules' of repo %s. Error " + "was: %s" % (GIT_CACHE_SERVER_URL, url, e.message)) + + gitmodules = '' + for line in response.text.splitlines(): + gitmodules += "%s\n" % (line.strip()) + io = StringIO(gitmodules) + parser = RawConfigParser() + parser.readfp(io) + stuff = {} + for section in parser.sections(): + submodule = re.sub(r'submodule "(.*)"', r'\1', section) + url = parser.get(section, 'url') + path = parser.get(section, 'path') + stuff[submodule] = {'url': url} + return stuff + +def add_submodules_to_strata(contents, filename): + assert contents['kind'] == 'stratum' + + changed = False + for chunk_ref in contents.get('chunks', []): + chunk_git_url = get_repo_url(chunk_ref['repo']) + chunk_git_ref = chunk_ref['ref'] + + if 'submodules' in chunk_ref: + continue + try: + toplevel_file_list = get_toplevel_file_list_from_repo( + chunk_git_url, chunk_git_ref) + except Exception as e: + message = ( + "Unable to look up repo %s on remote Git server %s. Check that " + "the repo URL is correct." % (chunk_git_url, TROVE_HOST)) + warning = ( + "If you are using a Trove that is not %s, please edit the " + "TROVE_HOST constant in this script and run it again." % + TROVE_HOST) + if FAIL_ON_REMOTE_CACHE_ERRORS: + raise RuntimeError(message + " " + warning) + else: + warnings.warn(message) + warnings.warn(warning) + continue + + logging.debug( + "%s: got file list %s", chunk_git_url, toplevel_file_list) + + path = get_repo_name(chunk_git_url) + if u'.gitmodules' in toplevel_file_list: + submodules = submodules_to_dict(chunk_git_url, chunk_git_ref) + if submodules: + chunk_ref['submodules'] = submodules + changed = True + + return changed +try: + if migrations.check_definitions_version(TO_VERSION - 1): + success = migrations.process_definitions( + kinds=['stratum'], + validate_cb=validate_chunk_refs, + modify_cb=add_submodules_to_strata) + if not success: + sys.stderr.write( + "Migration failed due to one or more warnings.\n") + sys.exit(1) + else: + migrations.set_definitions_version(TO_VERSION) + sys.stderr.write("Migration completed successfully.\n") + sys.exit(0) + else: + sys.stderr.write("Nothing to do.\n") + sys.exit(0) +except RuntimeError as e: + sys.stderr.write("Error: %s\n" % e.message) + sys.exit(1) diff --git a/mkdocs.yml b/mkdocs.yml new file mode 100644 index 0000000..2ecb636 --- /dev/null +++ b/mkdocs.yml @@ -0,0 +1,29 @@ +# This file controls the HTML rendering of the contents of this repo. +# +# Rendering is done using the 'mkdocs' tool, available from +# <http://www.mkdocs.org/>. + + +site_name: Baserock Project documentation +site_url: https://docs.baserock.org +site_favicon: data/favicon.ico + +copyright: | + Unless otherwise noted, text in this repository is licensed under the + Creative Commons Attribution-ShareAlike 4.0 International license (CC BY-SA + 4.0) + +pages: + - 'index.mdwn' + - 'Definitions format': 'spec.mdwn' + - 'Deployment': 'deployment.mdwn' + - 'changelog.mdwn' + +docs_dir: . + +site_dir: /srv/docs.baserock.org/ + +# I chose this theme because it mostly functions without Javascript. Many +# of the other mkdocs themes are pretty much unusable for users who have +# Javascript turned off! +theme: 'material' diff --git a/schemas/cluster.json-schema b/schemas/cluster.json-schema index 1a628c7..40cc439 100644 --- a/schemas/cluster.json-schema +++ b/schemas/cluster.json-schema @@ -40,12 +40,16 @@ definitions: system-deployment: type: object - required: [ 'type', 'location' ] + oneOf: [ {required: [ 'type', 'location' ]}, + {required: [ 'upgrade-type', 'upgrade-location' ]} ] + additionalProperties: true properties: type: { type: string } location: { type: string } + upgrade-type: { type: string } + upgrade-location: { type: string } # Corresponds to Cluster in Baserock data model. diff --git a/schemas/stratum.json-schema b/schemas/stratum.json-schema index 0330f37..1344edc 100644 --- a/schemas/stratum.json-schema +++ b/schemas/stratum.json-schema @@ -62,6 +62,15 @@ definitions: # artifact. patternProperties: ^.*$: { type: string } + submodules: + $ref: "#/definitions/submodule-reference" + + submodule-reference: + type: object + properties: + url: { type: string } + submodules: + $ref: "#/definitions/submodule-reference" # This doesn't need any special treatment in the Baserock data model because # it's a link to another stratum definition, without any extra info. diff --git a/scripts/yaml-jsonschema b/scripts/yaml-jsonschema new file mode 100755 index 0000000..64f52a7 --- /dev/null +++ b/scripts/yaml-jsonschema @@ -0,0 +1,50 @@ +#!/usr/bin/env python +# Copyright (C) 2015 Codethink Limited +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; version 2 of the License. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License along +# with this program. If not, see <http://www.gnu.org/licenses/>. + + +'''A tool to validate YAML files against the JSON-Schema schemas. + +This wraps Python `jsonschema` module so that YAML schemas can be understood +and YAML data can be validated. + +Usage: yaml-jsonschema SCHEMA INPUT1 [INPUT2, ...] + +''' + + +import jsonschema +import yaml + +import sys + + +schema_file = sys.argv[1] +input_files = sys.argv[2:] + + +with open(schema_file) as f: + schema = yaml.load(f) + + +for input_file in input_files: + with open(input_file) as f: + data = yaml.load(f) + + try: + jsonschema.validate(data, schema) + print("%s: valid" % input_file) + except jsonschema.ValidationError as e: + # Print 'e' instead of 'e.message' for more information! + print("%s: %s" % (input_file, e.message)) @@ -1,4 +1,4 @@ -[[!meta title="Baserock definitions format"]] +Title: Baserock definitions format The Baserock definitions format =============================== @@ -6,28 +6,18 @@ The Baserock definitions format This page describes the Baserock definitions format (morph files). It is intended to be useful as an *informal* specification. It is not guaranteed to be accurate or exhaustive. -If you are just getting started with Baserock, the [[quick-start]], [[devel-with]] and [[guides]] pages provide a more practical introduction. +If you are just getting started with Baserock, the wiki pages [quick-start](http://wiki.baserock.org/quick-start), [devel-with](http://wiki.baserock.org/devel-with) and [guides](http://wiki.baserock.org/guides) pages provide a more practical introduction. -The allowed YAML constructs are described in json-schema format here: <http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/schemas>. +The allowed YAML constructs are described in json-schema format here: <http://git.baserock.org/cgit/baserock/baserock/spec.git/tree/schemas>. -The data model is described using OWL here: <http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git/tree/schemas/baserock.owl>. +The data model is described using OWL here: <http://git.baserock.org/cgit/baserock/baserock/spec.git/tree/schemas/baserock.owl>. -The source code of [[Morph]] and [[YBD]] might be more useful if you need a completely accurate description of how the current Baserock definition format is used in practice. - -[[!toc startlevel=2 levels=2]] +The source code of [Morph] and [YBD] might be more useful if you need a completely accurate description of how the current Baserock definition format is used in practice. Versioning ---------- -The current version of the definitions format is version 7. - -See also: the [[planned]] versions, and the [[historical]] versions of the -format. - -Please propose changes to the format described here on the -[[baserock-dev@baserock.org mailing list|mailinglist]]. Ideally, provide a -patch for this file against the <git://baserock.branchable.com/> repo, but just -describing the change you want to make is also fine. +The current version of the definitions format is version 8. Definitions repository ---------------------- @@ -46,7 +36,7 @@ well, such as configuration data and documentation. The Baserock Project maintains a set of 'reference system definitions' at [git://git.baserock.org/baserock/baserock/definitions] (which can also be referred to as [baserock:baserock/definitions], when using the repo aliasing -feature of [[Morph]]). That repo contains systems that can be built and +feature of [Morph]). That repo contains systems that can be built and deployed as-is, but it is important that users can fork this repo as well, and work on systems in their version using `git merge` or `git rebase` to keep up to date with changes from upstream. @@ -55,8 +45,8 @@ Baserock tooling should not mandate anything about the definitions repo that the user wants to process, other than the rules defined below. -[git://git.baserock.org/baserock/baserock/definitions]: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git -[baserock:baserock/definitions]: http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/definitions.git +[git://git.baserock.org/baserock/baserock/definitions]: http://git.baserock.org/cgit/baserock/baserock/definitions.git +[baserock:baserock/definitions]: http://git.baserock.org/cgit/baserock/baserock/definitions.git ### Structure @@ -69,23 +59,15 @@ integer. The integer specifies the version of the definitions format that this repo uses. A tool should refuse to process a version that it doesn't support, to -avoid unpredictable errors. See the "Versioning" heading above for more detail -on versions. +avoid unpredictable errors. See also the [Versioning](#versioning) section. + +The top directory of the repo can also contain a file named `DEFAULTS`. This +holds repo-wide 'build-system' and 'split-rules' information. See the +[defaults](#defaults) section below. To find all the Baserock definition files in the repo, tooling can recursively scan the contents of the repo for files matching the glob pattern "\*.morph". -Deployment tooling should look in the toplevel directory /only/ for files -matching the following globs. (The purpose of these files is described in the -"deployment" section below). - - - `\*.check` - - `\*.check.help` - - `\*.configure` - - `\*.configure.help` - - `\*.write` - - `\*.write.help` - Definitions file syntax ----------------------- @@ -95,7 +77,7 @@ The toplevel entity in a definition is a dict, in all cases. Any syntax errors or type errors (such the toplevel entity being a number, or something) should be reported to the user. -The [[Morph]] tool raises an error if any unknown dictionary keys are found in +The [Morph] tool raises an error if any unknown dictionary keys are found in the definition, mainly so that it reports any spelling errors in key names. ### Common fields @@ -107,90 +89,103 @@ For all definitions, use the following fields: * `kind`: the kind of thing being built; **required** * `description`: a comment to describe what the definition is for; optional -### Build definitions: Chunks, Systems and Strata +Build definitions: Chunks, Systems and Strata +--------------------------------------------- Within this document, consider 'building' to be the act of running a series of commands in a given 'environment', where the commands and how to build the environment are completely specified by the definitions and the build tool. -For chunks, use the following fields: - -* `build-system`: if the program is built using a build system known to - `morph`, you can set this field and avoid having to set the various - `*-commands` fields; the commands that the build system specifies can - be overridden; the following build-systems are known: - - - `autotools` - - `python-distutils` - - `cpan` - - `cmake` - - `qmake` - - optional - -* `pre-configure-commands`: a list of shell commands to run at - the configuration phase of a build, before the list in `configure-commands`; - optional -* `configure-commands`: a list of shell commands to run at the configuraiton - phase of a build; optional -* `post-configure-commands`: a list of shell commands to run at - the configuration phase of a build, after the list in `configure-commands`; - optional - -* `pre-build-commands`: a list of shell commands to run at - the build phase of a build, before the list in `build-commands`; - optional -* `build-commands`: a list of shell commands to run to build (compile) the - project; optional -* `post-build-commands`: a list of shell commands to run at - the build phase of a build, after the list in `build-commands`; - optional - -* `pre-test-commands`: a list of shell commands to run at - the test phase of a build, before the list in `test-commands`; - optional -* `test-commands`: a list of shell commands to run unit tests and other - non-interactive tests on the built but un-installed project; optional -* `post-test-commands`: a list of shell commands to run at - the test phase of a build, after the list in `test-commands`; - optional - -* `pre-install-commands`: a list of shell commands to run at - the install phase of a build, before the list in `install-commands`; - optional -* `install-commands`: a list of shell commands to ; optional -* `post-install-commands`: a list of shell commands to run at - the install phase of a build, after the list in `strip-commands`; - optional - -* `pre-strip-commands`: a list of shell commands to run at - the strip phase of a build, before the list in `strip-commands`; - optional -* `strip-commands`: a list of shell commands to strip debug symbols from binaries; - this should strip binaries in the directory named in the `DESTDIR` environment - variable, not the actual system; optional -* `post-strip-commands`: a list of shell commands to run at - the strip phase of a build, after the list in `install-commands`; - optional - -* `max-jobs`: a string to be given to `make` as the argument to the `-j` - option to specify the maximum number of parallel jobs; the only sensible - value is `"1"` (including the quotes), to prevent parallel jobs to run - at all; parallel jobs are only used during the `build-commands` phase, - since the other phases are often not safe when run in parallel; `morph` - picks a default value based on the number of CPUs on the host system; - optional - -* `chunks`: a key/value map of lists of regular expressions; - the key is the name - of a binary chunk, the regexps match the pathnames that will be - included in that chunk; the patterns match the pathnames that get installed - by `install-commands` (the whole path below `DESTDIR`); every file must - be matched by at least one pattern; by default, a single chunk gets - created, named according to the definition, and containing all files; - optional - -For strata, use the following fields: +### Chunks + +A 'chunk' definition describes an individual component, which can be built from +a Git repository by executing the given sequence of commands. + +The structure of a 'chunk' definition is described using [JSON-Schema] in the +[spec.git] repo: +<http://git.baserock.org/cgit/baserock/baserock/spec.git/tree/schemas/chunk.json-schema> + +The build sequence consists of four phases: + +1. configure +2. build +3. install +4. strip + +You can define one or more commands for each phase, or none. Here is an example +chunk: + + name: glibc + kind: chunk + description: GNU C library (example) + + configure-commands: + - mkdir o + - cd o && ../libc/configure --prefix=/usr + + build-commands: + - cd o && make + + install-commands: + - cd o && make install_root="$DESTDIR" install + +A chunk can optionally make use of a repo-wide build system defined in +['DEFAULTS'](#defaults), by using the `build-system` field. In this case, the +command sequences defined in DEFAULTS are used. + +Any predefined command sequence can be overridden by specifying a new value. +You can also extend a predefined build system with the `pre-` or `post-` +fields. For example, `pre-configure-commands` are run directly before +`configure-commands`, and all `post-configure-commands` are run directly after. + +For example: + + name: git + kind: chunk + description: Git version control tool (example) + + build-system: autotools + + # This command will run before the normal 'configure' command sequence. + pre-configure-commands: + - make configure + + # This command overrides the normal 'build' command sequence. + build-commands: + - make all + +If a chunk doesn't need to override anything from ['DEFAULTS'](#defaults), you can avoid +having a chunk .morph file altogether, and just set 'build-system' when +referring to the chunk from the stratum. + +The `max-jobs` field can be used to pass a custom value for --jobs to Make, +via the `MAKEFLAGS` environment variable. This is useful for Makefiles which +don't work in parallel: you can set `max-jobs: 1` to work around the problem. +Parallel jobs are only used during the `build-commands` phase, since the other +phases are often not safe when run in parallel; `morph` picks a default value +based on the number of CPUs on the host system. + +Built chunks are split up into multiple artifacts. The default splitting rules +for a chunk are defined in ['DEFAULTS'](#defaults). You can use the `chunks` field of a +chunk to override these. The `chunks` field is a key/value map of lists of +regular expressions; the key is the name of a binary chunk, the regexps match +the pathnames that will be included in that chunk; the patterns match the +pathnames that get installed by `install-commands` (the whole path below +`DESTDIR`); every file must be matched by at least one pattern; by default, a +single chunk gets created, named according to the definition, and containing +all files. + +### Strata + +A 'stratum' is a group of related chunks. A stratum can contain only chunks. +Certain information about how to build a chunk is defined in the containing +stratum, rather than in the chunk definition. + +The structure of a 'stratum' definition is described using [JSON-Schema] in the +[spec.git] repo: +<http://git.baserock.org/cgit/baserock/baserock/spec.git/tree/schemas/stratum.json-schema> + +The fields mean the following: * `build-depends`: a list of strings, each of which refers to another stratum that the current stratum depends on. This list may be omitted @@ -207,6 +202,11 @@ For strata, use the following fields: to a chunk .morph file. - `build-system` specifies one of the predefined build systems. You must specify ONE of `morph` or `build-system` for each chunk. + - `submodules` is a list of key/value mappings that specifies url + overrides for .gitmodules. The key should be the name of the submodule + as listed in .gitmodules, not the path. The value is a dictionary + containing one key/value pair: + * `url`: The url override for the submodule, this can include aliasing. In addition to these keys, each of the sources can specify a list of build dependencies using the `build-depends` field. To specify one or more chunk dependencies, `build-depends` needs to be set to a list @@ -220,26 +220,23 @@ For strata, use the following fields: of a workaround for the lack of distinction between build and runtime dependencies. -For systems, use the following fields: +### Systems + +In the Baserock model, a 'system' is the top level entity that you actually +build and execute. Systems contain one or more strata. + +The structure of a 'system' definition is described using [JSON-Schema] in the +[spec.git] repo: +<http://git.baserock.org/cgit/baserock/baserock/spec.git/tree/schemas/system.json-schema> + +The fields mean the following: * `strata`: a list of key/value mappings, similar to the 'chunks' field of a stratum. Two fields are allowed (are both required?): - `name`: name of the artifact when the stratum is build - `morph`: path to a stratum .morph file relative to the top of the containing repo -Example chunk (simplified commands): - - name: eglibc - kind: chunk - configure-commands: - - mkdir o - - cd o && ../libc/configure --prefix=/usr - build-commands: - - cd o && make - install-commands: - - cd o && make install_root="$DESTDIR" install - -Example stratum: +### Example stratum: name: foundation kind: stratum @@ -264,8 +261,16 @@ Example stratum: build-depends: - fhs-dirs - linux-api-headers + - name: ansible + repo: upstream:ansible + ref: v2.0 + submodules: + lib/ansible/modules/core: + url: upstream:ansible-modules-core + lib/ansible/modules/extras: + url: upstream:ansible-modules-extras -Example system: +### Example system: name: base kind: system @@ -273,25 +278,26 @@ Example system: - morph: foundation - morph: linux-stratum -### Deployment definitions: Clusters - -**NOTE**: The deployment mechanism specified here is quite abstract. Most of -the code used to do real-world deployments is currently tied to [[Morph]] and -kept in [morph.git]. +Deployment definitions: Clusters +-------------------------------- -For 'deployment', [[Morph]] defines an API for running 'extensions'. The +For 'deployment', Baserock defines an API for running 'extensions'. The 'cluster' and 'system' definitions together describe what extensions should be run, and what should be set in their environment, in order to deploy the -system. See the "Deployment extension API" section below for how to find and -execute the extensions. +system. See the [Deployment](deployment) section for how to find and execute +the extensions. Within this document, consider "deployment" to be a process of first post-processing a filesystem tree with one or more 'configure extensions', then performing an operation to convert and/or transfer the filesystem tree using a 'write extension'. +The structure of the 'cluster' definitions is described using [JSON-Schema] in +the [spec.git] repo: +<http://git.baserock.org/cgit/baserock/baserock/spec.git/tree/schemas/cluster.json-schema> + A cluster morphology defines a list of systems to deploy, and for each system a -list of ways to deploy them. Use the following field: +list of ways to deploy them. The fields are used as follows: * **systems**: a list of systems to deploy; the value is a list of mappings, where each mapping has the @@ -303,8 +309,8 @@ list of ways to deploy them. Use the following field: * **deploy**: a mapping where each key identifies a system and each system has at least the following keys: - * **type**: identifies the type of deployment e.g. (kvm, - pxeboot), and thus what '.write' extension should be used. + * **type**: identifies the relative path, without extension, to the + '.write' program that should be used for this system. * **location**: where the deployed system should end up at. The syntax depends on the '.write' extension chosen in the 'type' field. @@ -353,7 +359,8 @@ Example: cluster-foo-armv7-2: HOSTNAME: cluster-foo-armv7-2 -### Repo URLs +Repo URLs +--------- Git repository locations can (and should) be abbreviated using the 'repo-alias' feature of Baserock definitions. This is a kind of [Compact URI](http://www.w3.org/TR/2009/CR-curie-20090116/). It currently only affects the 'repo' fields in a stratum .morph file. @@ -380,139 +387,99 @@ There are two repo aliases that *must* be defined: Baserock tools should allow changing these values. The main benefit of this compact URI scheme is that definitions are not tied to a specific Git server, or protocol. You can build against a mirror of the original Git server, or change the protocol that is used, just by altering the repo-alias configuration. -Build environment ------------------ - -### Sandboxing - -Builds should be done an isolated 'staging area', with only the specified dependencies available to the build process. The simplest approach is to install the dependencies in an empty directory, then [chroot](https://en.wikipedia.org/wiki/Chroot) into it. The more sandboxing the build tool can do, the better, because it lowers the chance of unexpected and unreproducible errors in the build process. The [Sandboxlib](https://github.com/CodethinkLabs/sandboxlib) Python library may be useful. - -The exception to the above is if the 'build-mode' field for a chunk is set to 'bootstrap'. Chunks in bootstrap mode are treated specially and do have access to tools from the host system. - -FIXME: more detail is needed here! - -### Environment variables +Defaults +-------- -The following environment variables can be used in chunk configure/build/install commands, and must be defined by the build tool. +A definitions repository can contain a file named DEFAULTS. This file sets up +repo-wide configuration. - - `MORPH_ARCH`: the Morph-specific architecture name; see <http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/tree/morphlib/util.py#n473> for a list of valid architectures - - `PREFIX`: the value of the 'prefix' field for this chunk (set in the stratum .morph file); default /usr - - `TARGET`: the [GNU architecture triplet](http://wiki.osdev.org/Target_Triplet) for the target architecture (for example, x86_64-baserock-linux) - - `TARGET_STAGE1`: the 'bootstrap' variant of the GNU architecture triplet. This must be different from $TARGET -- you can just change the vendor field to achieve that (e.g. x86_64-bootstrap-linux). +The structure of the DEFAULTS file is described using [JSON-Schema] in +the [spec.git] repo: +<http://git.baserock.org/cgit/baserock/baserock/spec.git/tree/schemas/defaults.json-schema> -FIXME: The `TARGET` and `TARGET_STAGE1` fields are specific to building GNU/Linux based systems, they shouldn't be mandated in the spec. +### Build systems -Deployment extension API ------------------------- +You can define common command sequences using the 'build-systems' key. These +can then be referenced in chunk (and stratum) definitions, to avoid repeating +common patterns. -**NOTE**: This section is more of an explaination of how the `morph deploy` -command works than a general-purpose spec, in places. We hope to fix that in -future. +Here is an example DEFAULTS file that defines the python-distutils build +system: -As noted in the previous section: within this document, consider "deployment" -to be a process of first post-processing a filesystem tree with one or more -'configure extensions', then performing an operation to convert and/or transfer -the filesystem tree using a 'write extension'. + build-systems: + python-distutils: + configure-commands: [] + build-commands: + - python setup.py build + install-commands: + - python setup.py install --prefix "$PREFIX" --root "$DESTDIR" -### Deployment sequence +### Splitting rules -A Baserock deployment tool should, for each labelled deployment in the -'cluster', follow this sequence. +One 'source' in Baserock can produce multiple binary 'artifacts'. This allows +you to separate programs, libraries, documentation, debugging information, and +whatever else into different artifacts. You can then choose to exclude some of +these artifacts from your final system. -1. Run the .check extension for that deployment 'type', if one exists. Abort - now, if it fails. +Split rules are defined separately for chunks, and for strata. - The .check extensions exist only to work around the fact that (2), (3) and - (4) may be slow, because programs that wait several minutes just to raise an - error that could have been detected right away are very annoying. +Chunks define a list of artifacts, and a series of regular expression patterns +that are matched against filenames to define what to include. The list of +artifacts is evaluated *in order*, so if two artifacts match the same pattern, +those files will go into whichever artifact is defined first. -2. Create a writeable temporary directory containing the contents of the system - artifact. +Stratum matches work similarly, except the stratum artifacts are a series of +patterns that match the *names of chunk artifacts*. -3. For each 'subsystem' specified, recursively run steps 1, 3, 4 and 5. The - intention is that the result of the subsystem deployments end up *inside* - the temporary directory created in step 2 for the 'outer' deployment. +Here is an example DEFAULTS file that sets up some simple splitting rules. -4. Run each .configure extension that was specified in the - `configuration-extensions` field of the system morphology. The order of - running these is, sadly, unspecified. + split-rules: + chunk: + - artifact: -devel + include: + - (usr/)?include/.* + - (usr/)?share/man/.* + - artifact: -runtime + include: + - .* -5. Run the .write extension for that deployment 'type'. + stratum: + - artifact: -devel + include: + - .*-devel + - artifact: -runtime + include: + - .*-runtime -The deployment of that system and its subsystems is now considered complete, -and temporary data can be removed. +You could use these rules to keep header files (/usr/include/\*) and manual +pages (/usr/share/man/\*) out of your systems, by only including the -runtime +stratum artifacts. -### Locating and running extensions - -Deployment extensions are executable files that live in the top level directory -of the definitions.git repo. Configure extensions have the extension -`.configure`, check extensions have the extension `.check`, and write -extensions have the extension `.write`. - -[[Morph]] will look for extensions inside the 'morphlib' Python package, in the -`exts/` dir, before looking in the definitions.git repo. - -An extension must be executable using the [POSIX `exec()` system call]. We -encourage writing them as Python scripts and using a `#!/usr/bin/env python` -[hashbang], but any executable code is permitted. - -The 'execution environment' for an extension is sadly unspecified at the moment. -[[Morph]] runs extensions without any kid of chroot environment, but each one -is run in a separate [mount namespace]. Running extensions chrooted into the -system being deployed does not make sense, as it may not contain the right -dependencies for the extensions to work, and it may expect to run on a -different, incompatible architecture to the system running the deployment in -any case. - -A .check extension must be passed one commandline argument: the value of the -`location` or `upgrade-location` field. - -A .configure extension must be passed one commandline argument: the path to -the unpacked filesystem tree, which it can modify in some way. It is expected -that a .configure extension will do nothing unless it is enabled using a -configuration variable, but it is up to the code in the .configure extension -to do this, and this behaviour is not currently enforced. - -A .write extension must be passed two commandline arguments: the value of the -`location` or `upgrade-location` field, then the path to the unpacked -filesystem tree (which it could modify in some way). - -All key/value pairs from the 'cluster' definition for the given labelled -deployment of a system must be passed to all extensions, as environment -variables. The `type`, `location`, `upgrade-type` and `upgrade-location` fields -do not need to be passed in. - -Extensions are expected to set exit code 0 on success, and a non-zero value on -failure. If any extension returns non-zero, the deployment tool should abort -the deployment. - -Extensions are expected to write status information to 'stdout', and any error -messages to 'stderr'. This is for user feedback ONLY, deployment tools should -not do anything with this data other than log and/or display it. +Build environment +----------------- -[[Morph]] sets an extra environment variable `MORPH_LOG_FD` which is a file -descriptor that the extension can write log messages to, in order for them to -appear in morph.log but not on stdout or stderr. +### Sandboxing -[POSIX `exec()` system call]: http://pubs.opengroup.org/onlinepubs/009695399/functions/exec.html -[hashbang]: https://en.wikipedia.org/wiki/Shebang_%28Unix%29 -[mount namespace]: https://stackoverflow.com/questions/22889241/linux-understanding-the-mount-namespace-clone-clone-newns-flag#22889401 +Builds should be done an isolated 'staging area', with only the specified dependencies available to the build process. The simplest approach is to install the dependencies in an empty directory, then [chroot](https://en.wikipedia.org/wiki/Chroot) into it. The more sandboxing the build tool can do, the better, because it lowers the chance of unexpected and unreproducible errors in the build process. The [Sandboxlib](https://github.com/CodethinkLabs/sandboxlib) Python library may be useful. -### Help for extensions +The exception to the above is if the 'build-mode' field for a chunk is set to 'bootstrap'. Chunks in bootstrap mode are treated specially and do have access to tools from the host system. -The .configure and .write extensions may provide .help files as documentation. -The .help file should be valid [YAML], containing a dict with the key `help` -and a string value. +FIXME: more detail is needed here! -For example, the `tar.write` extension could provide `tar.write.help` with the -following content: +### Environment variables - help: | - Deploy a system as a .tar file. +The following environment variables can be used in chunk configure/build/install commands, and must be defined by the build tool. -### Common configuration parameters used by extensions + - `MORPH_ARCH`: the Morph-specific architecture name; see <http://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph.git/tree/morphlib/util.py#n473> for a list of valid architectures + - `PREFIX`: the value of the 'prefix' field for this chunk (set in the stratum .morph file); default /usr + - `TARGET`: the [GNU architecture triplet](http://wiki.osdev.org/Target_Triplet) for the target architecture (for example, x86_64-baserock-linux) + - `TARGET_STAGE1`: the 'bootstrap' variant of the GNU architecture triplet. This must be different from $TARGET -- you can just change the vendor field to achieve that (e.g. x86_64-bootstrap-linux). -Fill me in! +FIXME: The `TARGET` and `TARGET_STAGE1` fields are specific to building GNU/Linux based systems, they shouldn't be mandated in the spec. -[morph.git]: git://git.baserock.org/cgi-bin/cgit.cgi/baserock/baserock/morph +[JSON-Schema]: https://www.json-schema.org/ +[Morph]: http://wiki.baserock.org/Morph/ +[YBD]: http://wiki.baserock.org/ybd/ +[morph.git]: git://git.baserock.org/cgit/baserock/baserock/morph.git/ +[spec.git]: git://git.baserock.org/cgit/baserock/baserock/spec.git/ [YAML]: http://yaml.org/ |