summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Fixupssam/distbuild-nfsboot.writeSam Thursfield2015-04-201-5/+7
| | | | Change-Id: I7c89d0ea096e967241db8b1aa01e6a5eeefa87f2
* Add distbuild-trove-nfsboot.writeSam Thursfield2015-04-203-0/+479
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The nfsboot.write deployment extension has been deprecated for a while because it's not generally useful. It's only used for deploying distbuild nodes to a Trove, as far as I know. We still need to support setting up a bunch of machines that boot over NFS from a Trove. But we can do this in a special-purpose .write extension. The new distbuild-trove-nfsboot.write is much more efficient than the more generic nfsboot.write: instead of treating each system individually (thus copying an almost identical ~2GB rootfs to the Trove once per node) it copies the system image to the Trove once, and /then/ sets up a rootfs per node. Upgrades are now supported, although the code assumes distbuild nodes are stateless (as they should be) so nothing special is done for upgrades, other than checking that there is already a version of the given system in existance. The new extension does not create an orig/ and run/ version of each system, because there is no need when the deployed system is stateless. There could be further gains in efficiency, but I don't have time to do them right now. This write extension is full of compromises, its goal is to better support the existing users who have a Trove and a distbuild network deployed via NFS. It is specifically not intended to be useful for other purposes. Change-Id: I9a50c58b714ed272212d1d6c55b289aaa96051b1
* OSTREE FIXUPSam Thursfield2015-04-201-3/+4
| | | | Change-Id: I834d21942427884420b153b49764d53a19e72988
* Download stratum metadata along with stratumSam Thursfield2015-04-201-1/+1
| | | | | | | | | | Otherwise, the checkout_stratum() function fails because the .meta file is missing. I would rather Morph never downloaded part of an artifact -- all the consituent files should be tracked and kept together. Change-Id: I939147e4035cf891b9a7809db36df1271f057b7b
* Consolidate parsing of stratum artifacts in one placeSam Thursfield2015-04-203-23/+27
| | | | | | | This allows handing parse errors that might occur if the files in the cache have become corrupted, instead of crashing with a traceback. Change-Id: Ibaa372ba6a14e7a04de907cb3a664b92cf61fbf3
* Flush temporary file after downloadSam Thursfield2015-04-201-0/+1
| | | | | | | Without this, stratum files can turn up in the cache as 0 bytes long, which breaks everything. Change-Id: I4c0712d97a372232360113e388a09c61f2e7b875
* Remove the chunk hardlink cacheAdam Coldrick2015-04-204-81/+30
| | | | | | | | The OSTree checkout uses hardlinks anyway, so this is thankfully now redundant. Most of this work was done by Sam Thursfield <sam.thursfield@codethink.co.uk>. Change-Id: I5be211ad3034331834f7b0542721f1bfc024b863
* Fix the cmdtests to understand the OSTree artifact cacheAdam Coldrick2015-04-2017-56/+111
| | | | | | | This commit also disables the cross-bootstrap test, as the cross-bootstrap plugin needs rewriting to use OSTree. Change-Id: I6183b1a396525f2e60dcbc2dbf5730acfb9df30b
* Add a plugin containing some commands to help with the OSTree artifact cacheAdam Coldrick2015-04-203-2/+173
| | | | Change-Id: I37855e48a8ff099d4ea32d6ec4d44048f9eae077
* yarns: Disable the cross-bootstrap yarnAdam Coldrick2015-04-201-7/+9
| | | | | | | | | The cross-bootstrap plugin is currently full of hacks, so it makes sense to rewrite it properly rather than extending these hacks to work with OSTree. I don't have time to do this right now, so disable the cross-bootstrap yarn. Change-Id: I343e238778f7f6ab290a3e22ff930edf740bf27c
* yarns: Make the distbuild yarn expose the worker's artifact cache over HTTPAdam Coldrick2015-04-203-5/+32
| | | | | | | | | | | OSTree can easily pull over HTTP. All that is necessary is to expose the repo directory. This commit adds a simple HTTP server to do this in the test suite. Actual implementations should use something better, like lighttpd. Also add some logging of the cache servers in this yarn to help debug. Change-Id: Ic65390f550c972c5f1072bbca8d80e8b56723158
* morph-cache-server: Add support for an OSTree artifact cacheAdam Coldrick2015-04-201-52/+72
| | | | Change-Id: I8409eb5816e5fbd6416437f51b97a703e3bddac4
* gc: Make `morph gc` use the OSTree artifact cacheAdam Coldrick2015-04-201-2/+5
| | | | Change-Id: Ib2c8b0af414252acb5964eacae68f4dc692f85e3
* deploy: Use OSTree to checkout the system for deploymentAdam Coldrick2015-04-201-23/+101
| | | | | | | | | | | Now that we have an OSTree artifact cache, the deploy plugin needs to use that to get the system to be deployed. Due to the changes in how we store systems, we need to get the contents of each stratum then put the system delta on top of that. This is still much quicker than unpacking stuff from tarballs. Change-Id: I209dc43a49bb00fa828907cb72715afc7061d43f
* builder: Use the OSTree artifact cache when buildingAdam Coldrick2015-04-203-43/+45
| | | | | | | | | The API of the OSTree artifact cache is slightly different to that of the old tarball cache, so adjust things accordingly. Also, only store the files changed at system-construction-time rather than everything in system artifacts. Change-Id: Iec74f37ed1ae769280c552a1d67f5f6c5fdae9e3
* Make morph use OSTreeArtifactCache instead of LocalArtifactCacheAdam Coldrick2015-04-202-4/+6
| | | | Change-Id: I957e0b085e9aed3426b164b18e8b50fe47aa918f
* RemoteArtifactCache: Support multiple cache methodsAdam Coldrick2015-04-201-0/+25
| | | | | | | This commit updates RemoteArtifactCache to enable it to interact with a remote OSTree artifact cache. Change-Id: Icdb2d3071adb6532e54ab7f203c391b3a30a1c63
* Add an artifact cache which uses OSTreeAdam Coldrick2015-04-204-5/+298
| | | | Change-Id: Ie591875d0507b4f43da3e91d4ca63e0a4dc7cbf2
* Add a class to wrap the OSTree APIAdam Coldrick2015-04-204-0/+189
| | | | Change-Id: I6c193597f136d1a0f4af14f3fd670c01f9ae2875
* bins: We no longer want chunks to be tarballsAdam Coldrick2015-04-202-120/+37
| | | | | | | | Change create_chunk to put the contents of an artifact into a directory rather than storing them in a tarball, as we want to store chunks as directory trees in OSTree rather than tarballs now. Change-Id: Ic8aab162fdbbc03ec92538759fef06f35ff0696e
* Move the chunk cache logic into buildcommandAdam Coldrick2015-04-202-23/+23
| | | | | | | | This avoids needing to pass the cache to the staging area since lac.get returns a path to a directory tree rather than a file handle now, so we also don't need to do any unpacking. Change-Id: I918a96cf210f9f8b7256493d9e544ffbca8c6d38
* Create device nodes in staging areaAdam Coldrick2015-04-203-35/+39
| | | | | | | | | | | We can't store devices nodes in OSTree, so we can't create them at artifact build time and store them in the chunk artifacts anymore. Instead, we should create them in the staging area when installing an artifact with a source which has a morphology which defines them into the staging area. Change-Id: I423c5e4b8d6595b95894935329c3bf3517cfa475
* Add support for unionfs-fuseAdam Coldrick2015-04-204-16/+39
| | | | | | | | Overlayfs is new in version 3.18 of the kernel, so add support for a different implementation of a union/overlay filesystem in order to allow morph to work on older kernels. Change-Id: I63f05265c645dfcc92f3987582bb3f06d853e740
* Use overlayfs when deployingAdam Coldrick2015-04-201-4/+34
| | | | | | | | | | | | | | When deploying, configuration extensions are run against the unpacked tarball of a system created by a build. If we are to use OSTree to store systems (rather than tarballs) this will not be possible as the result of `ostree checkout` would be read-only. To solve this, we use overlayfs to mount the unpacked tarball underneath a temporary directory somewhere, and run the configuration extensions on that mount point. This means that the changes are made in the temporary directory rather than directly on the tarball. Change-Id: Iafcb78feaa1d6d37be4f52420c03d5b96d5c81ee
* Use overlayfs when building systemsAdam Coldrick2015-04-203-5/+45
| | | | | | | This will allow us to cache systems as a list of chunks and a small filesystem delta, rather than a massive tarball. Change-Id: Ic5182137ddd88dc9679e0941550d08954cb4d306
* Allow the passing of options to fsutils.mountAdam Coldrick2015-04-201-2/+5
| | | | | | | In order to mount using overlayfs, fsutils.mount needs to take a string of options to pass to the mount command. Change-Id: Ic190784353ef8c841a1cd9ff632a7996351b436c
* deploy: Clean up `morph deploy` documentation a bitSam Thursfield2015-04-201-133/+30
| | | | | | | | | | | | Remove documentation specific to certain .configure and .write extensions, and point the user to `morph help-extensions` and `morph help xxx.write` instead. Don't mention the deprecated nfsboot.write extension. Other small cleanups. Change-Id: Ie7ed030c71ade4876aafb3d8a43e864a28ddc279
* deploy: Deploy and upgrade systems from the same 'cluster' definitionSam Thursfield2015-04-201-17/+83
| | | | | | | | | | | | | | | | | | | This patch adds two fields to deployment (cluster) .morph files: 'upgrade-type' and 'upgrade-location'. The `morph deploy` command ignores these. The `morph upgrade` command will honour them if present, instead of the existing 'type' and 'location' fields. If they are not present, `morph upgrade` will give a warning, and will use the 'type' and 'location' fields as before. This avoids the need to edit the deployment .morph file after deploying a system. Small detail: the 'type' and 'location' variables are no longer removed from the environment that is passed to the .configure and .write extensions. This shouldn't affect anything. Change-Id: Id2a4e4f229b8adebdb57eded2049ac113a82a4be
* deploy: Factor out a bit of code into its own functionSam Thursfield2015-04-201-10/+37
| | | | | | | This behaviour is complex, it shouldn't be hidden away in a larger function. Change-Id: I953c9477e0210d395b97f5f8219eaebe4dbcd272
* distbuild: Don't create a directory for build output until we get someSam Thursfield2015-04-151-9/+17
| | | | | | | | Currently, it leaves around empty directories called build-00, build-01, etc. when you run a distbuild that fails to get as far as building something, which is annoying. Change-Id: Id3466e248c327dedaf973bc2fe22d42e5c5570d4
* Add dns option to simple-network conf extRichard Ipsum2015-04-151-1/+8
| | | | Change-Id: I0f4490d76caca802536b21085ea0d770fb8c0798
* Give more helpful error when stratum artifacts in cache are corruptedSam Thursfield2015-04-141-10/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | I had a stratum artifact in my artifact cache which for some reason was 0 bytes long. When building a system that included this stratum, `morph build` gave me the following output 2015-04-13 13:48:57 ERROR Traceback (most recent call last): File "/src/morph/morphlib/builder.py", line 539, in build_and_cache self.unpack_strata(fs_root) File "/src/morph/morphlib/builder.py", line 600, in unpack_strata chunks = [ArtifactCacheReference(c) for c in json.load(f)] File "/usr/lib/python2.7/json/__init__.py", line 290, in load **kw) File "/usr/lib/python2.7/json/__init__.py", line 338, in loads return _default_decoder.decode(s) File "/usr/lib/python2.7/json/decoder.py", line 366, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode raise ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded With this patch, I get a better error: ERROR: Corruption detected: No JSON object could be decoded while loading /src/build/cache/artifacts/8b4422c58ecb2a085b142fbba74b760f501f65d4b2885bf707994973230e0c58.stratum.build-essential-minimal Change-Id: I0ad359901c5da75bd26d5a1a8108ef4e6f1d7708
* More robust parsing of plugin pathsBob Mottram2015-04-141-6/+8
| | | | | | | Previously it was possible to have some strange values for pluginmgr.locations if the same path was appeared twice Change-Id: I3ec257e48e85c3fd30759c3dcc2064f0b151ec45
* _parse_version_file tweakRichard Ipsum2015-04-131-8/+6
| | | | | | | | Swap nested ifs for and Also replace type() with isinstance() Also fix doc string Change-Id: I58177566f19b601bc734f7542c6a5dd52938063a
* Add template option to install-files conf extRichard Ipsum2015-04-111-10/+32
| | | | | | | | | This adds an optional 'template' option to the install-files manifest format. A file declared as a template will be rendered using jinja2 with variables substituted in from the environment. Change-Id: I2ed6fe58f5fff315b42b7e4ec478ada851e0a70d
* Fix: check that path to chunk morph existsbaserock-definitions-v2Richard Ipsum2015-04-111-5/+33
| | | | | | | | | | | | | | | | | | | | | This fixes a bug that allows stratum definitions to carry paths to non-existent chunk morphs. This fix changes the interpretation of definitions so necessarily introduces a new definitions version (version 2) Assuming a morph field defined in a stratum: "morph: strata/cats/xattr.morph" and assuming "strata/cats/xattr.morph" does not exist, this patch will not alter current behaviour for definitions versions 0 and 1, besides producing a warning message like this: "Warning! `strata/cats/xattr.morph' referenced in `strata/swift.morph' does not exist" for definitions version 2 and greater the following error will be raised: "ERROR: Couldn't find morphology: strata/cats/xattr.morph referenced in strata/swift.morph" Change-Id: I4def5e92741cce25168f1038136503022ab27ffd
* Make _check_version_file return versionRichard Ipsum2015-04-111-6/+16
| | | | | | Also add a distinct error message for an invalid version file Change-Id: I674e6af284b33705ae8235054906f56b8cd8a130
* Fix _env_for_arch()Paul Martin2015-04-101-1/+1
| | | | Change-Id: I1ffb63340d3facb608708d04a0a21c5a9e290c14
* Change build environment to support MIPS32/64 BE and LESimon Hoinkis2015-04-103-6/+36
| | | | Change-Id: I9344b9b80a6ec008715559390b63c9003f34bf90
* Remove description of definitions format from Morph's READMESam Thursfield2015-04-101-183/+6
| | | | | | | | | | I propose putting it here instead: <http://wiki.baserock.org/definitions/current> I've also updated a few anachronisms in the README, but it could do with further work. Change-Id: I803246c123d99990e941afa66f96ba9fd210c28e
* distbuild: Remove unneeded debugging statementSam Thursfield2015-04-091-6/+0
| | | | | | | A JsonMachine object can be set to log all messages that it sends, we don't need to handle it in the WorkerConnection class as well. Change-Id: Idfdc06953363a016708b5dda50c978eb93b1113c
* distbuild: Disable extra message debugging in worker log filesSam Thursfield2015-04-091-1/+0
| | | | | | | | | | | | | | | | | | | | | | | Worker log files are overly verbose with this enabled, each message is dumped 6 times: 2015-03-19 11:00:11 DEBUG JsonMachine: Received: '"{...}\\n"\n' 2015-03-19 11:00:11 DEBUG JsonMachine: line: '"{...}\\n"' 2015-03-19 11:00:11 DEBUG JsonRouter: got msg: {...} 2015-03-19 11:00:11 DEBUG JsonMachine: Sending message {...} 2015-03-19 11:00:11 DEBUG JsonMachine: As '"{...}\\n"' 2015-03-19 11:00:11 DEBUG JsonRouter: sent to client: {...} With this setting disabled, the message is only logged by the JsonRouter class, so appears only twice: 2015-03-19 11:00:11 DEBUG JsonRouter: got msg: {...} 2015-03-19 11:00:11 DEBUG JsonRouter: sent to client: {...} We've not seen any issues with message encoding/decoding recently so I think it's safe to disable this debugging output by default. Change-Id: I7d22ed29e81d6c594cb2c639abf3b40bfb27e3ad
* distbuild: Make 'Current jobs' log message more usefulSam Thursfield2015-04-091-2/+11
| | | | | | | | | | | | | | | | | | | It's good to know which jobs are in progress and which are queued, when reading morph-controller.log. Old output: 2015-04-09 10:40:58 DEBUG Current jobs: ['3f647933a1effbb128c857225ba77e9aa775d92314ef0acf3e58e084a7248c73.chunk.stage1-binutils-misc', 'd7279e4179a31d8a3a98c27d5b01ad1bb7387c7fab623fee1086ab68af2784bb.chunk.stage2-fhs-dirs-misc'] New output: 2015-04-09 10:40:58 DEBUG Current jobs: ['3f647933a1effbb128c857225ba77e9aa775d92314ef0acf3e58e084a7248c73.chunk.stage1-binutils-misc (given to worker1:3434)', 'd7279e4179a31d8a3a98c27d5b01ad1bb7387c7fab623fee1086ab68af2784bb.chunk.stage2-fhs-dirs-misc (given to worker2:3434)'] Change-Id: Ie89e6723b0da5f930813591a3166301fd3966804
* Make kvm deploy check that host has virt-installRichard Ipsum2015-04-071-0/+9
| | | | | | | This allows us to catch a case where virt-install hasn't been installed on the host we're deploying to much earlier in the deployment process. Change-Id: I413ad804a7e8bef4fc2d1231411e01d30d0cb9e8
* Make source resolver pass filenameRichard Ipsum2015-04-041-1/+1
| | | | | | | | | | | | | Passing the filename when loading a morphology lets us give a more useful error message: e.g. ERROR: Missing field name from morphology strata/build-essential.morph instead of ERROR: Missing field name from morphology string Change-Id: I2e5b33e8154567469aa56adbf942fb7b6e9dd1ac
* distbuild: Fix issues in build cancellationSam Thursfield2015-04-021-8/+13
| | | | | | | | | A cancel during the 'graphing' or 'annotating' stages would be ignored as the BuildController was listening for the InitiatorDisconnect message from the wrong event source. In 'building' state the actual build would be stopped, but the BuildController instance would stick around due to sending the message class instead of an instance of the message. Change-Id: I222a8aa39bf7fffab4d89e12997ffd18cd1b54fc
* Add a test for partial buildsAdam Coldrick2015-04-022-0/+17
| | | | Change-Id: Ibc95fe1d78c8d68dd20a4d2187f93f363369dfec
* Implement partial distbuildsAdam Coldrick2015-04-025-32/+120
| | | | | | | | | | | | | | In addition to partial builds we also want to be able to do partial distbuilds, and distbuild uses a different codepath. This commit updates the distbuild code to know what to do if a partial build is requested. It only builds up to the latest chunk/stratum that was requested, and displays where to find the artifacts for each of the chunks/strata requested upon completion of the build. The usage is the same as for local builds. Change-Id: I0537f74e2e65c7aefe5e71795f17999e2415fce5
* Add `morph certify` plugin to check for build reproducibility.Michael Drake2015-04-012-0/+141
| | | | | | | | | | | Currently does three checks: 1. Checks that all chunks in given system(s) have sha1 refs. 2. Checks that all sha1 refs are anchored. Unanchored refs can be removed on `git gc`. 3. Checks that all chunk repos are on the trove-host. Change-Id: Iaf105b1614a45616684e68a08f28b8529d4321fa
* Make warning messages more user friendly.Michael Drake2015-04-011-0/+7
| | | | | | | | Printing the file and line number of the warning's origin makes the warning itself harder to spot, and can make it appear as if something has actually gone wrong. Change-Id: Ie77357d9a061d31e234e6e434521d2d274409773
View Lorry Controller Status