| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some failing commands in a yaml block were being ignored. For
example the following commands...
pre-configure-commands:
- |
cp nonexistent foo
echo foo
... where giving the following result without making the build
fail:
### PRE-CONFIGURE-COMMANDS ###
+ cp nonexistent foo
cp: can't stat 'nonexistent': No such file or directory
+ echo foo
foo
This commit also adjusts the test-shell.c to allow the use of '-e'
flag. No other changes because the shell was already failing after
the first failure found.
Change-Id: Ie78b25a99f1b5d6d0d26be74ddc377025dff69fd
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is done by adding an status update when a command fails. An example
of the new output is:
2016-06-03 10:35:04 [Build 5/133] [stage2-libstdc++] Creating staging area
2016-06-03 10:35:04 [Build 5/133] [stage2-libstdc++] Starting actual build: stage2-libstdc++ fc4868e
2016-06-03 10:35:04 [Build 5/133] [stage2-libstdc++] Extracting /src/gcc-tarball into /src/tmp/staging/tmpE5b84y/stage2-libstdc++.build
2016-06-03 10:35:38 [Build 5/133] [stage2-libstdc++] Running configure-commands
2016-06-03 10:35:55 [Build 5/133] [stage2-libstdc++] Running build-commands
2016-06-03 10:36:14 [Build 5/133] [stage2-libstdc++] Error when running build-commands
Change-Id: Iac94794311e649d13c534f453d3473ac8003e8f4
|
|
|
|
|
|
|
|
|
| |
This code is a rework from changes done by:
- Tiago Gomes <tiago.gomes@codethink.co.uk>
https://storyboard.baserock.org/#!/story/86
Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's not really any reason you'd want to use the RemoteRepoCache
class except as a workaround for the slow speed of some LocalRepoCache
operations, so I can't see this ruining anyone's day.
The main reason for doing this is so we can simply the sourceresolver
code. One reason that the sourceresolver class is so hopelessly
complicated is that it right now has to use two incompatible interfaces
for Git repo caches.
I've taken the opportunity to detangle the RepoCache class from the
App class. Now all of the configuration for the RepoCache class is
passed into the constructor explicitly. This makes the class usable from
outside Morph:
resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[])
repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver)
Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For a long time the CachedRepo class has basically been a wrapper around
the GitDir class, but with a few extra methods that don't really even
belong there.
It is now a tiny class in the localrepocache module which just keeps
track of a few extra attributes. All other functionality is provided
by the gitdir module.
This commit also removes the `git clone` approach for copying repos out
of the cache. The alternative approach implemented by
git.copy_repository() was slightly faster when I tested, so for now we
should use that everywhere. Longer term we should find out why this is
quicker than `git clone`, and fix Git itself to be fast.
Change-Id: I1686ab43253d44c3903d9a0bad8bb75528e9cf75
|
|
|
|
|
|
|
|
|
|
| |
This allows us to simplify a couple of places.
I cannot think of a single situation where'd you want to get the cached
copy of a repo, but not update it. Think about it -- the repo might be
*years* behind the upstream remote.
Change-Id: I60340c7fb33e7bfe871ad30c0a9322a7202548e2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's no longer any need to look up the buildsystem in builder,
since the morphloader now sets commands directly in the morphology.
Removing this unused variable also fixes a bug where morph can
crash if the chunk being built is being built with a buildsystem
that is not hardcoded in morph, error below:
2016-02-15 15:29:46 Starting build of devel-system-x86_64-generic
2016-02-15 15:29:46 [Build 70/407] [cython.morph] Building chunk cython.morph
2016-02-15 15:29:46 [Build 70/407] [cython.morph] Creating staging area
2016-02-15 15:29:47 [Build 70/407] [cython.morph] Starting actual build: cython.morph 4dd8e76
2016-02-15 15:29:47 [Build 70/407] [cython.morph] Extracting upstream:cython into /src/tmp/staging/tmpLI9TUj/cython.morph.build
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 299, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/build_plugin.py", line 208, in build
self._build(source_pool, filename, component_names=component_names)
File "/src/morph/morphlib/plugins/build_plugin.py", line 237, in _build
bc.build_in_order(component)
File "/src/morph/morphlib/buildcommand.py", line 297, in build_in_order
self.cache_or_build_source(s, build_env)
File "/src/morph/morphlib/buildcommand.py", line 317, in cache_or_build_source
self.build_source(source, build_env)
File "/src/morph/morphlib/buildcommand.py", line 374, in build_source
self.build_and_cache(staging_area, source, setup_mounts)
File "/src/morph/morphlib/buildcommand.py", line 545, in build_and_cache
return builder.build_and_cache(source)
File "/src/morph/morphlib/builder.py", line 748, in build_and_cache
built_artifacts = o.build_and_cache()
File "/src/morph/morphlib/builder.py", line 288, in build_and_cache
self.run_commands(temppath, stdout)
File "/src/morph/morphlib/builder.py", line 314, in run_commands
bs = morphlib.buildsystem.lookup_build_system(m['build-system'])
File "/src/morph/morphlib/buildsystem.py", line 300, in lookup_build_system
raise KeyError('Unknown build system: %s' % name)
KeyError: 'Unknown build system: python3-distutils'
Change-Id: Iddf651b408c612c7c80296012cf1c97ab99e8ccc
|
|
|
|
|
|
|
| |
Using app.status() produces nicer logs as the messages will have the
status prefix prepended.
Change-Id: I0573d7066784afdbfb878bed85af704e82bb9797
|
|
|
|
|
|
|
|
|
| |
Replace cliapp.shell_quote() with pipes.quote() to produce the chroot
scripts. Despite pipes.quote() being deprecated, it produces much less
escaping than cliapp.shell_quote(). This method should be replaced with
shlex.quote() when migrating to Python 3.
Change-Id: I4b8e8eefe0cd321458bb3fae72f6d0552680f84f
|
|
|
|
| |
Change-Id: Iedefd1745e85e1f30d45ee8d26b315760a542a0b
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pass the Source to the staging area constructor so that we don't need to
pass it as a parameter when we call some StagingArea methods. Also move
the creation of the build and destdir directories to the constructor so
that we can get rid of the chroot_open() and chroot_close() methods.
Also, provide API to retrieve the relative locations for buildir and
destdir.
Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Improve error messages and remove redundant information
* Stop displaying the containerised settings, as they are not very
helpful to debug a build failure
* Display the error code
* Change system integration commands logging to consistent with build
commands logging.
* Provide a log file in the staging directory for each chunk build
failure or system integration failure.
* Provide a chroot script in the staging directory for each chunk build
failure or system integration build failure. This script allows the
user to enter an environment similiar to the one where the build
failed.
* Fix a bug where the system integration commands stdout was being
omitted when these commands failed.
* Fix a bug where the error_message_for_containerised_commandline() was
being called with an empty error message, as the stderr was being
directed to stdout.
Chunk build failure output before the this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands
build failed
# configure
# # echo "configure-commands stdout"
configure-commands stdout
# # echo "configure-commands stderr" >&2
configure-commands stderr
# build
# # echo "build commands stdout"
build commands stdout
# # echo "build commands stderr" >&2
build commands stderr
# # echo "some more sdout"
echo "some more stderr" >&2
echo $PATH
foo
some more sdout
some more stderr
/tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
sh: foo: not found
ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c
'echo "some more sdout"
echo "some more stderr" >&2
echo $PATH
foo
':
Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball',
'/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True,
'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths':
['/src/tmp/staging/tmpglc5Sz/test.build',
'/src/tmp/staging/tmpglc5Sz/test.inst',
'/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc',
'/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz',
'cwd': '/test.build'}
Error output:
$>
Chunk build failure output after this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands
### CONFIGURE-COMMANDS ###
+ echo configure-commands stdout
configure-commands stdout
+ echo configure-commands stderr
configure-commands stderr
### BUILD-COMMANDS ###
+ echo build commands stdout
build commands stdout
+ echo build commands stderr
build commands stderr
+ echo some more sdout
some more sdout
+ echo some more stderr
some more stderr
+ echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
/tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
+ foo
sh: foo: not found
ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi
t_code=127)
$>
System integration build failure output before this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the
system integration commands
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while
running system integration commands
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while
building system
ERROR: Command failed:
baserock/system-integration/00-test-test-misc-0002:
Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'),
('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root':
'/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'}
Error output:
+ foo
baserock/system-integration/00-test-test-misc-0002: line 1: foo: not
found
$>
System integration build failure output after this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the
system integration commands
+ echo stdout
stdout
+ echo stderr
stderr
+ foo
baserock/system-integration/00-test-test-misc-0002: line 1: foo: not
found
ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration
commands failed (exit_code=127)
$>
Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stop moving staging areas of failed builds from the 'staging' directory
to the 'failed' directory. Moving staging areas make it very difficult
to debug build failures on the build essential chunks, as the paths set
on the configure scripts and some environment variables (e.g.
STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area.
This change will also make it easier to create scripts that chroot n
environment similiar to the one where the build failure occurred.
To make it still possible to safely do a build an run `morph gc` in
parallel, we use flock(2) to control access to the staging area
directory.
Also, move the `test_supports_non_isolated_mode` test into a different
class, as it requires a different SetUp() routine (the staging area is
contructed with different parameters).
Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of logging every command for ourselves, use the shell X-ray
mode. As side effect, each command argument will now be printed out in
its evaluated form, which aids debugging.
Also, improve the visual separation between the different build steps,
and display those steps with finer granularity.
Change-Id: I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8
|
|
|
|
|
|
|
| |
This is probably a sign that something is broken, but it's better to
cope with it rather than crashing.
Change-Id: I4e9c40d0b7db031ea271143938db61c4ad70a888
|
|
|
|
|
|
|
|
|
|
|
|
| |
The system-integration field in a chunk .morph file defines which of the
chunk artifacts the integration script should live. If you list an
artifact that doesn't exist, the integration script will not go anywhere
and the commands won't run.
Ideally system-integration commands will be become a bit more simple to
use, but hopefully this warning helps in the meantime.
Change-Id: I10a068e79eb46f98f0bd308a5caf3c445dda22ed
|
|
|
|
|
|
|
| |
Chunk commands will currently block if they read from stdin.
To fix this we set stdin to /dev/null so that the first read returns EOF.
Change-Id: I143396f4c443c098b2880c5925e6f0907cb4fb17
|
|
|
|
|
|
|
|
|
|
|
| |
It's quite easy to forget that "true" and "false" are special words in YAML,
and write "false" to halt the build process while debugging. This was treated
as a Boolean instead of a string, so causes an unhelpful error in morph.
Since there is no use for booleans in morphologies, they should be treated as
strings.
Change-Id: I7c872f9696611920febec5f375b599eee89d040e
|
|
|
|
| |
Change-Id: I19a6c31979aa36ff3c03f41e16e2d25ef407533b
|
|
|
|
|
|
|
|
| |
Do not remove this code for now as its possible that new versions
of morph build old sytems without the install-essential-files
extension
Change-Id: I5e49d0bc6baee73bedf16469d922822e76286bc1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I had a stratum artifact in my artifact cache which for some reason was
0 bytes long. When building a system that included this stratum, `morph
build` gave me the following output
2015-04-13 13:48:57 ERROR Traceback (most recent call last):
File "/src/morph/morphlib/builder.py", line 539, in build_and_cache
self.unpack_strata(fs_root)
File "/src/morph/morphlib/builder.py", line 600, in unpack_strata
chunks = [ArtifactCacheReference(c) for c in json.load(f)]
File "/usr/lib/python2.7/json/__init__.py", line 290, in load
**kw)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
With this patch, I get a better error:
ERROR: Corruption detected: No JSON object could be decoded while
loading
/src/build/cache/artifacts/8b4422c58ecb2a085b142fbba74b760f501f65d4b2885bf707994973230e0c58.stratum.build-essential-minimal
Change-Id: I0ad359901c5da75bd26d5a1a8108ef4e6f1d7708
|
|
|
|
| |
Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
|
|\ |
|
| |
| |
| |
| | |
Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
|
|/
|
|
| |
Change-Id: I771c3de9cecda7a503f4d36ae5d9fabc040892e4
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
| |
This helps refactoring morph to get rid of treeishes.
|
|
|
|
|
|
|
|
| |
When making a copy of the repository to build from, builder copies the
.git directory then checks out the ref it wants.
However sometimes this doesn't add the files to the working directory,
possibly because the files it is missing weren't changed in the difference
between what the .git thought HEAD was and what is now checked out.
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
| |
I found the write_cache_metadata and prepare_binary_metadata methods
be confusingly similar, when skimming code. Since write_cache_metadata
was very short and only used from one place, inlined it, which removes
the confusion without, I think, adding complexity.
|
|
|
|
|
|
|
|
| |
The --target-cflags option is really just a temporary solution. At some
point we will want to have a proper target architecture/device
description that morph uses to define compiler flags etc.
For now, I hope this will do.
|
| |
|
| |
|
|
|
|
|
| |
The system images will later need to be read, so useful commands
want to be shared
|
|
|
|
| |
It only needs a morph, not a blob
|
|
|
|
|
| |
This is achieved by copying vmlinuz, System.map and extlinux.conf
to the default subvolume.
|
| |
|
| |
|
|\ |
|
| |
| |
| |
| |
| |
| | |
This should stop tar from replacing symbolic links with directories
and instead follow the symbolic links, extracting the files into
the linked directories
|
| |
| |
| |
| | |
f.write does not add them
|