| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
| |
This is mostly to check that the 'cancel entire subprocess tree' works
as expected. Revert that patch and the test fails.
There are also some tweaks included in this commit.
Change-Id: If297522e6589ebb3a07dac66a39eb243789e53aa
|
|
|
|
|
|
|
| |
This check is intended for local builds, not distbuilds. Much less disk
space is needed to distbuild.
Change-Id: Iaff77805119129cbf6584de418ef97c7f1b82ad8
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The nfsboot.write deployment extension has been deprecated for a while
because it's not generally useful. It's only used for deploying
distbuild nodes to a Trove, as far as I know.
We still need to support setting up a bunch of machines that boot over
NFS from a Trove. But we can do this in a special-purpose .write
extension.
The new distbuild-trove-nfsboot.write is much more efficient than
the more generic nfsboot.write: instead of treating each system
individually (thus copying an almost identical ~2GB rootfs to the Trove
once per node) it copies the system image to the Trove once, and /then/
sets up a rootfs per node.
Upgrades are now supported, although the code assumes distbuild nodes
are stateless (as they should be) so nothing special is done for
upgrades, other than checking that there is already a version of the
given system in existance. The new extension does not create an
orig/ and run/ version of each system, because there is no need when the
deployed system is stateless.
There could be further gains in efficiency, but I don't have time to do
them right now. This write extension is full of compromises, its goal is
to better support the existing users who have a Trove and a distbuild
network deployed via NFS. It is specifically not intended to be useful
for other purposes.
Change-Id: I9a50c58b714ed272212d1d6c55b289aaa96051b1
|
|
|
|
|
|
|
|
| |
Currently, it leaves around empty directories called build-00, build-01,
etc. when you run a distbuild that fails to get as far as building
something, which is annoying.
Change-Id: Id3466e248c327dedaf973bc2fe22d42e5c5570d4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We discovered a case where a user of distbuild began a build of
'qtbase', then cancelled it 2 minutes in. The `morph worker-build`
process didn't exit for over an hour -- it ran right through until the
chunk artifacts had been created. Then it exited with code -9 (SIGKILL).
This seems to be due to the fact that SIGKILL doesn't kill subprocesses,
and so any file descriptors the subprocesses have open will remain open.
If we set up the `morph worker-build` process as a process group
leader, using os.setpgid(), then we can use os.killpg() to kill the
entire process group. This should ensure that the `morph worker-build`
command exits straight away, as all of its subprocesses will be killed
at the same time it is.
Change-Id: I38707d18004d8c5bc994fd0cb99e90fd5def58e4
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove documentation specific to certain .configure and .write
extensions, and point the user to `morph help-extensions` and `morph
help xxx.write` instead.
Don't mention the deprecated nfsboot.write extension.
Other small cleanups.
Change-Id: Ie7ed030c71ade4876aafb3d8a43e864a28ddc279
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch adds two fields to deployment (cluster) .morph files:
'upgrade-type' and 'upgrade-location'.
The `morph deploy` command ignores these. The `morph upgrade` command
will honour them if present, instead of the existing 'type' and
'location' fields. If they are not present, `morph upgrade` will give
a warning, and will use the 'type' and 'location' fields as before.
This avoids the need to edit the deployment .morph file after deploying
a system.
Small detail: the 'type' and 'location' variables are no longer removed
from the environment that is passed to the .configure and .write
extensions. This shouldn't affect anything.
Change-Id: Id2a4e4f229b8adebdb57eded2049ac113a82a4be
|
|
|
|
|
|
|
| |
This behaviour is complex, it shouldn't be hidden away in a larger
function.
Change-Id: I953c9477e0210d395b97f5f8219eaebe4dbcd272
|
|
|
|
|
|
|
|
|
| |
Previously it was only available in the distbuild-helper program. Moving
it to its own module means we can test it and reuse it.
This commit also adds a docstring to the class.
Change-Id: Iaf7854048cf0ff463a87894f1f500cdcb6a34d8b
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A log message was printing the 'remote name' of a socket that was
listening for connections. There isn't one, so the message always
shows this:
2015-04-14 17:05:19 INFO Binding socket to sam-jetson-mason
2015-04-14 17:05:19 INFO Listening at None
Print the local name instead:
2015-04-14 17:05:19 INFO Binding socket to sam-jetson-mason
2015-04-14 17:05:19 INFO Listening at 10.24.2.125:7878
Change-Id: I22c1bbe8c9f78ef63e587b6ace516afc861fae0f
|
|
|
|
|
|
|
|
|
|
| |
Add InitiatorListJobs class and list-jobs message template, add
distbuild-list-jobs to morph commandlist, send running job
information back to initiator, split out handling of build request
and list-jobs messages to separate functions and change generating
a random integer to UUID for message identification
Change-Id: Id02604f2c1201dbc10f6bbd7f501b8ce1ce0deae
|
|
|
|
| |
Change-Id: I0f4490d76caca802536b21085ea0d770fb8c0798
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I had a stratum artifact in my artifact cache which for some reason was
0 bytes long. When building a system that included this stratum, `morph
build` gave me the following output
2015-04-13 13:48:57 ERROR Traceback (most recent call last):
File "/src/morph/morphlib/builder.py", line 539, in build_and_cache
self.unpack_strata(fs_root)
File "/src/morph/morphlib/builder.py", line 600, in unpack_strata
chunks = [ArtifactCacheReference(c) for c in json.load(f)]
File "/usr/lib/python2.7/json/__init__.py", line 290, in load
**kw)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
With this patch, I get a better error:
ERROR: Corruption detected: No JSON object could be decoded while
loading
/src/build/cache/artifacts/8b4422c58ecb2a085b142fbba74b760f501f65d4b2885bf707994973230e0c58.stratum.build-essential-minimal
Change-Id: I0ad359901c5da75bd26d5a1a8108ef4e6f1d7708
|
|
|
|
|
|
|
| |
Previously it was possible to have some strange values for
pluginmgr.locations if the same path was appeared twice
Change-Id: I3ec257e48e85c3fd30759c3dcc2064f0b151ec45
|
|
|
|
|
|
|
|
| |
Swap nested ifs for and
Also replace type() with isinstance()
Also fix doc string
Change-Id: I58177566f19b601bc734f7542c6a5dd52938063a
|
|
|
|
|
|
|
|
|
| |
This adds an optional 'template' option to the install-files manifest format.
A file declared as a template will be rendered using jinja2
with variables substituted in from the environment.
Change-Id: I2ed6fe58f5fff315b42b7e4ec478ada851e0a70d
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes a bug that allows stratum definitions
to carry paths to non-existent chunk morphs.
This fix changes the interpretation of definitions
so necessarily introduces a new definitions version (version 2)
Assuming a morph field defined in a stratum: "morph: strata/cats/xattr.morph"
and assuming "strata/cats/xattr.morph" does not exist, this patch will not
alter current behaviour for definitions versions 0 and 1,
besides producing a warning message like this:
"Warning! `strata/cats/xattr.morph' referenced in `strata/swift.morph' does not exist"
for definitions version 2 and greater the following error will be raised:
"ERROR: Couldn't find morphology: strata/cats/xattr.morph referenced in strata/swift.morph"
Change-Id: I4def5e92741cce25168f1038136503022ab27ffd
|
|
|
|
|
|
| |
Also add a distinct error message for an invalid version file
Change-Id: I674e6af284b33705ae8235054906f56b8cd8a130
|
|
|
|
| |
Change-Id: I1ffb63340d3facb608708d04a0a21c5a9e290c14
|
|
|
|
| |
Change-Id: I9344b9b80a6ec008715559390b63c9003f34bf90
|
|
|
|
|
|
|
|
|
|
| |
I propose putting it here instead:
<http://wiki.baserock.org/definitions/current>
I've also updated a few anachronisms in the README, but it could do with
further work.
Change-Id: I803246c123d99990e941afa66f96ba9fd210c28e
|
|
|
|
|
|
|
| |
A JsonMachine object can be set to log all messages that it sends, we
don't need to handle it in the WorkerConnection class as well.
Change-Id: Idfdc06953363a016708b5dda50c978eb93b1113c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Worker log files are overly verbose with this enabled, each message is
dumped 6 times:
2015-03-19 11:00:11 DEBUG JsonMachine: Received: '"{...}\\n"\n'
2015-03-19 11:00:11 DEBUG JsonMachine: line: '"{...}\\n"'
2015-03-19 11:00:11 DEBUG JsonRouter: got msg: {...}
2015-03-19 11:00:11 DEBUG JsonMachine: Sending message {...}
2015-03-19 11:00:11 DEBUG JsonMachine: As '"{...}\\n"'
2015-03-19 11:00:11 DEBUG JsonRouter: sent to client: {...}
With this setting disabled, the message is only logged by the JsonRouter
class, so appears only twice:
2015-03-19 11:00:11 DEBUG JsonRouter: got msg: {...}
2015-03-19 11:00:11 DEBUG JsonRouter: sent to client: {...}
We've not seen any issues with message encoding/decoding recently so I
think it's safe to disable this debugging output by default.
Change-Id: I7d22ed29e81d6c594cb2c639abf3b40bfb27e3ad
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It's good to know which jobs are in progress and which are queued, when
reading morph-controller.log.
Old output:
2015-04-09 10:40:58 DEBUG Current jobs:
['3f647933a1effbb128c857225ba77e9aa775d92314ef0acf3e58e084a7248c73.chunk.stage1-binutils-misc',
'd7279e4179a31d8a3a98c27d5b01ad1bb7387c7fab623fee1086ab68af2784bb.chunk.stage2-fhs-dirs-misc']
New output:
2015-04-09 10:40:58 DEBUG Current jobs:
['3f647933a1effbb128c857225ba77e9aa775d92314ef0acf3e58e084a7248c73.chunk.stage1-binutils-misc (given to worker1:3434)',
'd7279e4179a31d8a3a98c27d5b01ad1bb7387c7fab623fee1086ab68af2784bb.chunk.stage2-fhs-dirs-misc (given to worker2:3434)']
Change-Id: Ie89e6723b0da5f930813591a3166301fd3966804
|
|
|
|
|
|
|
| |
This allows us to catch a case where virt-install hasn't been installed
on the host we're deploying to much earlier in the deployment process.
Change-Id: I413ad804a7e8bef4fc2d1231411e01d30d0cb9e8
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Passing the filename when loading a morphology
lets us give a more useful error message:
e.g. ERROR: Missing field name from morphology strata/build-essential.morph
instead of
ERROR: Missing field name from morphology string
Change-Id: I2e5b33e8154567469aa56adbf942fb7b6e9dd1ac
|
|
|
|
|
|
|
|
|
| |
A cancel during the 'graphing' or 'annotating' stages would be ignored
as the BuildController was listening for the InitiatorDisconnect message
from the wrong event source. In 'building' state the actual build would
be stopped, but the BuildController instance would stick around due to
sending the message class instead of an instance of the message.
Change-Id: I222a8aa39bf7fffab4d89e12997ffd18cd1b54fc
|
|
|
|
| |
Change-Id: Ibc95fe1d78c8d68dd20a4d2187f93f363369dfec
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In addition to partial builds we also want to be able to do partial
distbuilds, and distbuild uses a different codepath.
This commit updates the distbuild code to know what to do if a partial
build is requested. It only builds up to the latest chunk/stratum that
was requested, and displays where to find the artifacts for each of
the chunks/strata requested upon completion of the build.
The usage is the same as for local builds.
Change-Id: I0537f74e2e65c7aefe5e71795f17999e2415fce5
|
|
|
|
|
|
|
|
|
|
|
| |
Currently does three checks:
1. Checks that all chunks in given system(s) have sha1 refs.
2. Checks that all sha1 refs are anchored.
Unanchored refs can be removed on `git gc`.
3. Checks that all chunk repos are on the trove-host.
Change-Id: Iaf105b1614a45616684e68a08f28b8529d4321fa
|
|
|
|
|
|
|
|
| |
Printing the file and line number of the warning's origin
makes the warning itself harder to spot, and can make it
appear as if something has actually gone wrong.
Change-Id: Ie77357d9a061d31e234e6e434521d2d274409773
|
|
|
|
| |
Change-Id: I63f0d9da8b12930bd9539710429756a0821ce884
|
|
|
|
| |
Change-Id: Ic6e613c21ed26c528ad7c75f41af01d7552729fd
|
|
|
|
| |
Change-Id: Ica2a270b254b01b9fb15956556efb74ee641e1b4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit allows you to build only a single chunk or stratum and its
dependencies, instead of a whole system. This will be useful for
developers who want to quickly check the build commands of a chunk for
example. You can give more than one chunk or stratum to the command,
and the build will run up to the latest one, building the dependencies
of both.
An example of the usage:
morph build systems/devel-system-x86_64-generic.morph \
strata/build-essential.morph strata/core/curl.morph
This will only build up to curl in the core stratum.
Change-Id: Id55b8335fc61e8fdf847f5e383cfedd13d07e6d2
|
|
|
|
|
|
| |
Also ensure NIC_CONFIG begins with 'network=', 'bridge=' or is 'user'
Change-Id: I3bcbd25eb2c9a05b7fa276697f97a1080cb0316e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The temporary build branch (--local-changes=include) feature can be
convenient, but it is also slow, and causes the resulting system to be
'unreproducible' (built from temporary commits).
Before:
$ morph build systems/devel-system-x86_64-generic.morph
2015-03-31 10:47:18 Starting build 53546647f92a4283bf96e33ea93e767e
2015-03-31 10:47:18 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master
2015-03-31 10:47:25 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph
2015-03-31 10:47:25 Deciding on task order
...
After:
$ morph build systems/devel-system-x86_64-generic.morph
2015-03-31 10:46:21 Looking for uncommitted changes (pass --local-changes=ignore to skip)
2015-03-31 10:46:21 Creating temporary branch in /src/ws/master/baserock/baserock/definitions named refs/heads/baserock/builds/f0b21fe240b244edb7e4142b6e201658/8df11f234ab24d22a9616ce911542332
2015-03-31 10:46:28 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph
2015-03-31 10:46:28 Deciding on task order
...
Change-Id: I21a3e65c29adf0b07f743289c6b3f0f4dddc34be
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Before:
$ morph build systems/devel-system-x86_64-generic.morph --verbose
2015-03-31 09:35:36 new environment variable GIT_NO_REPLACE_OBJECTS = "1"
2015-03-31 09:35:36 Starting build 3fba9406f75242afbfdb413bf9658516
2015-03-31 09:35:36 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master
2015-03-31 09:25:08 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f6137060cd0>
After:
$ morph build systems/devel-system-x86_64-generic.morph --verbose
2015-03-31 09:35:36 new environment variable GIT_NO_REPLACE_OBJECTS = "1"
2015-03-31 09:35:36 Starting build 3fba9406f75242afbfdb413bf9658516
2015-03-31 09:35:36 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master
2015-03-31 09:35:37 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to refs/heads/baserock/builds/f0b21fe240b244edb7e4142b6e201658/8df11f234ab24d22a9616ce911542332
Change-Id: I5b61ae6f10b26458178ead551e7ec12c1926014e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reduces noise on the console when using `morph build --verbose`.
Before (example from my current chroot):
$ morph build systems/devel-system-x86_64-generic.morph --verbose
2015-03-31 09:25:08 new environment variable SCHROOT_UID = "0"
2015-03-31 09:25:08 new environment variable SCHROOT_CHROOT_NAME = "baserock-2015-03-24-ostree"
2015-03-31 09:25:08 new environment variable LOGNAME = "root"
2015-03-31 09:25:08 new environment variable XDG_SEAT = "seat0"
2015-03-31 09:25:08 new environment variable PATH = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
2015-03-31 09:25:08 new environment variable XDG_VTNR = "1"
2015-03-31 09:25:08 new environment variable HOME = "/root"
2015-03-31 09:25:08 new environment variable SCHROOT_ALIAS_NAME = "baserock-2015-03-24-ostree"
2015-03-31 09:25:08 new environment variable TERM = "xterm"
2015-03-31 09:25:08 new environment variable SHELL = "/bin/bash"
2015-03-31 09:25:08 new environment variable SHLVL = "1"
2015-03-31 09:25:08 new environment variable SCHROOT_USER = "root"
2015-03-31 09:25:08 new environment variable EDITOR = "vim"
2015-03-31 09:25:08 new environment variable SCHROOT_GID = "0"
2015-03-31 09:25:08 new environment variable SCHROOT_GROUP = "root"
2015-03-31 09:25:08 new environment variable GIT_NO_REPLACE_OBJECTS = "1"
2015-03-31 09:25:08 new environment variable SCHROOT_COMMAND = "/usr/bin/env SSH_AUTH_SOCK=/tmp/tmp.eVw95CcpjS/auth-sock /bin/bash -l"
2015-03-31 09:25:08 new environment variable PYTHONPATH = "/src/morph/"
2015-03-31 09:25:08 new environment variable SSH_AUTH_SOCK = "/tmp/tmp.eVw95CcpjS/auth-sock"
2015-03-31 09:25:08 new environment variable SCHROOT_SESSION_ID = "baserock-2015-03-24-ostree-f3c36e20-c3db-4404-8808-e805ca0c1ac4"
2015-03-31 09:25:08 new environment variable MANPAGER = "less -R"
2015-03-31 09:25:08 new environment variable XDG_SESSION_ID = "2"
2015-03-31 09:25:08 new environment variable _ = "/src/morph/morph"
2015-03-31 09:25:08 new environment variable OLDPWD = "/src/morph"
2015-03-31 09:25:08 new environment variable PWD = "/src/ws/master/baserock/baserock/definitions"
2015-03-31 09:25:08 new environment variable USER = "root"
2015-03-31 09:25:08 Starting build 8f80a816dfe44f6c8352a4242b84bf48
2015-03-31 09:25:08 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master
2015-03-31 09:25:08 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f6137060cd0>
2015-03-31 09:25:15 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph
...
After:
$ morph build systems/devel-system-x86_64-generic.morph --verbose
2015-03-31 09:23:38 new environment variable GIT_NO_REPLACE_OBJECTS = "1"
2015-03-31 09:23:38 Starting build acfdf9a405384e1ea4e1b62cd1bce27f
2015-03-31 09:23:38 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master
2015-03-31 09:23:38 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f8fcbe86bd0>
2015-03-31 09:23:45 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph
...
Change-Id: I20bbdf35b280ccd305bc7e4a70b5f3de7679a368
|
|
|
|
|
|
|
| |
In the native build phase, building of cmake will fail if the order of
pathes in $PATH is twisted. So we set it to fixed to avoid problem.
Change-Id: I6b4d3ed3d5331c2db4f5f3324207fabb4310985c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Usefully, this provides the resolved repo URL, which may be
aliased in the actual defintions. This makes it easier to
use a workflow that avoids `morph edit`. Resolved repo URLs
can be git cloned.
It also provides other info about the chunk. In the case
of multiple strata containing the same chunk name, the tool
lists them all, saying which stratum they come from.
Example output:
# morph get-chunk-details libpng
In stratum graphics-common:
Chunk: libpng
Repo: git://git.baserock.org/delta/libpng
Ref: 259fb7761d747655c607efcec7a12ff1f3c24561
Change-Id: I6b540e38d9521f3f473b51c9031e508ddcb458ee
|
| |
|
|\ |
|
| |
| |
| |
| |
| |
| |
| | |
Let morph recognise the architecture string reported by
a big-endian ARMv8 kernel.
Change-Id: I64323e6c2a4675a8cc34f185b10fa04754ebe8c8
|
|\ \
| | |
| | |
| | | |
exist"
|
| | |
| | |
| | |
| | | |
Change-Id: I1521c0bdec4d7a6812f8988a2349e66b08161de8
|
|\ \ \
| |/ / |
|
| | |
| | |
| | |
| | | |
Change-Id: I44693d15aa5e92d5f09720065788adff34f8685c
|
|\ \ \
| |/ / |
|
| | |
| | |
| | |
| | | |
Change-Id: If011a5518fd30914c89b00099b9d05cff5cd3959
|