| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's no longer any need to look up the buildsystem in builder,
since the morphloader now sets commands directly in the morphology.
Removing this unused variable also fixes a bug where morph can
crash if the chunk being built is being built with a buildsystem
that is not hardcoded in morph, error below:
2016-02-15 15:29:46 Starting build of devel-system-x86_64-generic
2016-02-15 15:29:46 [Build 70/407] [cython.morph] Building chunk cython.morph
2016-02-15 15:29:46 [Build 70/407] [cython.morph] Creating staging area
2016-02-15 15:29:47 [Build 70/407] [cython.morph] Starting actual build: cython.morph 4dd8e76
2016-02-15 15:29:47 [Build 70/407] [cython.morph] Extracting upstream:cython into /src/tmp/staging/tmpLI9TUj/cython.morph.build
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 299, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/build_plugin.py", line 208, in build
self._build(source_pool, filename, component_names=component_names)
File "/src/morph/morphlib/plugins/build_plugin.py", line 237, in _build
bc.build_in_order(component)
File "/src/morph/morphlib/buildcommand.py", line 297, in build_in_order
self.cache_or_build_source(s, build_env)
File "/src/morph/morphlib/buildcommand.py", line 317, in cache_or_build_source
self.build_source(source, build_env)
File "/src/morph/morphlib/buildcommand.py", line 374, in build_source
self.build_and_cache(staging_area, source, setup_mounts)
File "/src/morph/morphlib/buildcommand.py", line 545, in build_and_cache
return builder.build_and_cache(source)
File "/src/morph/morphlib/builder.py", line 748, in build_and_cache
built_artifacts = o.build_and_cache()
File "/src/morph/morphlib/builder.py", line 288, in build_and_cache
self.run_commands(temppath, stdout)
File "/src/morph/morphlib/builder.py", line 314, in run_commands
bs = morphlib.buildsystem.lookup_build_system(m['build-system'])
File "/src/morph/morphlib/buildsystem.py", line 300, in lookup_build_system
raise KeyError('Unknown build system: %s' % name)
KeyError: 'Unknown build system: python3-distutils'
Change-Id: Iddf651b408c612c7c80296012cf1c97ab99e8ccc
|
|
|
|
|
|
|
| |
Using app.status() produces nicer logs as the messages will have the
status prefix prepended.
Change-Id: I0573d7066784afdbfb878bed85af704e82bb9797
|
|
|
|
|
|
|
|
|
| |
Replace cliapp.shell_quote() with pipes.quote() to produce the chroot
scripts. Despite pipes.quote() being deprecated, it produces much less
escaping than cliapp.shell_quote(). This method should be replaced with
shlex.quote() when migrating to Python 3.
Change-Id: I4b8e8eefe0cd321458bb3fae72f6d0552680f84f
|
|
|
|
| |
Change-Id: Iedefd1745e85e1f30d45ee8d26b315760a542a0b
|
|
|
|
|
|
|
|
|
|
|
|
| |
Pass the Source to the staging area constructor so that we don't need to
pass it as a parameter when we call some StagingArea methods. Also move
the creation of the build and destdir directories to the constructor so
that we can get rid of the chroot_open() and chroot_close() methods.
Also, provide API to retrieve the relative locations for buildir and
destdir.
Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Improve error messages and remove redundant information
* Stop displaying the containerised settings, as they are not very
helpful to debug a build failure
* Display the error code
* Change system integration commands logging to consistent with build
commands logging.
* Provide a log file in the staging directory for each chunk build
failure or system integration failure.
* Provide a chroot script in the staging directory for each chunk build
failure or system integration build failure. This script allows the
user to enter an environment similiar to the one where the build
failed.
* Fix a bug where the system integration commands stdout was being
omitted when these commands failed.
* Fix a bug where the error_message_for_containerised_commandline() was
being called with an empty error message, as the stderr was being
directed to stdout.
Chunk build failure output before the this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands
build failed
# configure
# # echo "configure-commands stdout"
configure-commands stdout
# # echo "configure-commands stderr" >&2
configure-commands stderr
# build
# # echo "build commands stdout"
build commands stdout
# # echo "build commands stderr" >&2
build commands stderr
# # echo "some more sdout"
echo "some more stderr" >&2
echo $PATH
foo
some more sdout
some more stderr
/tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
sh: foo: not found
ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c
'echo "some more sdout"
echo "some more stderr" >&2
echo $PATH
foo
':
Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball',
'/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True,
'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths':
['/src/tmp/staging/tmpglc5Sz/test.build',
'/src/tmp/staging/tmpglc5Sz/test.inst',
'/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc',
'/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz',
'cwd': '/test.build'}
Error output:
$>
Chunk build failure output after this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands
### CONFIGURE-COMMANDS ###
+ echo configure-commands stdout
configure-commands stdout
+ echo configure-commands stderr
configure-commands stderr
### BUILD-COMMANDS ###
+ echo build commands stdout
build commands stdout
+ echo build commands stderr
build commands stderr
+ echo some more sdout
some more sdout
+ echo some more stderr
some more stderr
+ echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
/tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
+ foo
sh: foo: not found
ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi
t_code=127)
$>
System integration build failure output before this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the
system integration commands
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while
running system integration commands
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while
building system
ERROR: Command failed:
baserock/system-integration/00-test-test-misc-0002:
Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'),
('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root':
'/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'}
Error output:
+ foo
baserock/system-integration/00-test-test-misc-0002: line 1: foo: not
found
$>
System integration build failure output after this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the
system integration commands
+ echo stdout
stdout
+ echo stderr
stderr
+ foo
baserock/system-integration/00-test-test-misc-0002: line 1: foo: not
found
ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration
commands failed (exit_code=127)
$>
Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stop moving staging areas of failed builds from the 'staging' directory
to the 'failed' directory. Moving staging areas make it very difficult
to debug build failures on the build essential chunks, as the paths set
on the configure scripts and some environment variables (e.g.
STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area.
This change will also make it easier to create scripts that chroot n
environment similiar to the one where the build failure occurred.
To make it still possible to safely do a build an run `morph gc` in
parallel, we use flock(2) to control access to the staging area
directory.
Also, move the `test_supports_non_isolated_mode` test into a different
class, as it requires a different SetUp() routine (the staging area is
contructed with different parameters).
Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of logging every command for ourselves, use the shell X-ray
mode. As side effect, each command argument will now be printed out in
its evaluated form, which aids debugging.
Also, improve the visual separation between the different build steps,
and display those steps with finer granularity.
Change-Id: I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8
|
|
|
|
|
|
|
| |
This is probably a sign that something is broken, but it's better to
cope with it rather than crashing.
Change-Id: I4e9c40d0b7db031ea271143938db61c4ad70a888
|
|
|
|
|
|
|
|
|
|
|
|
| |
The system-integration field in a chunk .morph file defines which of the
chunk artifacts the integration script should live. If you list an
artifact that doesn't exist, the integration script will not go anywhere
and the commands won't run.
Ideally system-integration commands will be become a bit more simple to
use, but hopefully this warning helps in the meantime.
Change-Id: I10a068e79eb46f98f0bd308a5caf3c445dda22ed
|
|
|
|
|
|
|
| |
Chunk commands will currently block if they read from stdin.
To fix this we set stdin to /dev/null so that the first read returns EOF.
Change-Id: I143396f4c443c098b2880c5925e6f0907cb4fb17
|
|
|
|
|
|
|
|
|
|
|
| |
It's quite easy to forget that "true" and "false" are special words in YAML,
and write "false" to halt the build process while debugging. This was treated
as a Boolean instead of a string, so causes an unhelpful error in morph.
Since there is no use for booleans in morphologies, they should be treated as
strings.
Change-Id: I7c872f9696611920febec5f375b599eee89d040e
|
|
|
|
| |
Change-Id: I19a6c31979aa36ff3c03f41e16e2d25ef407533b
|
|
|
|
|
|
|
|
| |
Do not remove this code for now as its possible that new versions
of morph build old sytems without the install-essential-files
extension
Change-Id: I5e49d0bc6baee73bedf16469d922822e76286bc1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I had a stratum artifact in my artifact cache which for some reason was
0 bytes long. When building a system that included this stratum, `morph
build` gave me the following output
2015-04-13 13:48:57 ERROR Traceback (most recent call last):
File "/src/morph/morphlib/builder.py", line 539, in build_and_cache
self.unpack_strata(fs_root)
File "/src/morph/morphlib/builder.py", line 600, in unpack_strata
chunks = [ArtifactCacheReference(c) for c in json.load(f)]
File "/usr/lib/python2.7/json/__init__.py", line 290, in load
**kw)
File "/usr/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/usr/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/usr/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
With this patch, I get a better error:
ERROR: Corruption detected: No JSON object could be decoded while
loading
/src/build/cache/artifacts/8b4422c58ecb2a085b142fbba74b760f501f65d4b2885bf707994973230e0c58.stratum.build-essential-minimal
Change-Id: I0ad359901c5da75bd26d5a1a8108ef4e6f1d7708
|
|
|
|
| |
Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
|
|\ |
|
| |
| |
| |
| | |
Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
|
|/
|
|
| |
Change-Id: I771c3de9cecda7a503f4d36ae5d9fabc040892e4
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
| |
|
|
|
|
| |
This helps refactoring morph to get rid of treeishes.
|
|
|
|
|
|
|
|
| |
When making a copy of the repository to build from, builder copies the
.git directory then checks out the ref it wants.
However sometimes this doesn't add the files to the working directory,
possibly because the files it is missing weren't changed in the difference
between what the .git thought HEAD was and what is now checked out.
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
| |
I found the write_cache_metadata and prepare_binary_metadata methods
be confusingly similar, when skimming code. Since write_cache_metadata
was very short and only used from one place, inlined it, which removes
the confusion without, I think, adding complexity.
|
|
|
|
|
|
|
|
| |
The --target-cflags option is really just a temporary solution. At some
point we will want to have a proper target architecture/device
description that morph uses to define compiler flags etc.
For now, I hope this will do.
|
| |
|
| |
|
|
|
|
|
| |
The system images will later need to be read, so useful commands
want to be shared
|
|
|
|
| |
It only needs a morph, not a blob
|
|
|
|
|
| |
This is achieved by copying vmlinuz, System.map and extlinux.conf
to the default subvolume.
|
| |
|
| |
|
|\ |
|
| |
| |
| |
| |
| |
| | |
This should stop tar from replacing symbolic links with directories
and instead follow the symbolic links, extracting the files into
the linked directories
|
| |
| |
| |
| | |
f.write does not add them
|
|\ \
| |/ |
|
| |
| |
| |
| |
| |
| |
| | |
Previously, we could get a situation where chunks named the same way
in two morphologies could interfere with each other: they could generate
the same cache key, resulting in their chunks overwriting each other.
Now this won't happen anymore.
|
| |
| |
| |
| | |
The mkimage.sh script should now be unnecessary again.
|
| | |
|
| |
| |
| |
| |
| | |
Add PREFIX to the whitelist of environment variables that
can affect the cache keys.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previous change would share the same environment between all Builders
this may not have broken anything yet, but this change in behaviour
was unintentional.
The relevant cache keys are not altered by Builders and they are set
at the time the Builder is created, so the build string can be cached
rather than generated every time a cache key is needed
|