| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
| |
Also, add integration tests to prevent regressions.
Change-Id: I80421112382168a71ca69e4bafe3c3a174e7f9ef
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Stop moving staging areas of failed builds from the 'staging' directory
to the 'failed' directory. Moving staging areas make it very difficult
to debug build failures on the build essential chunks, as the paths set
on the configure scripts and some environment variables (e.g.
STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area.
This change will also make it easier to create scripts that chroot n
environment similiar to the one where the build failure occurred.
To make it still possible to safely do a build an run `morph gc` in
parallel, we use flock(2) to control access to the staging area
directory.
Also, move the `test_supports_non_isolated_mode` test into a different
class, as it requires a different SetUp() routine (the staging area is
contructed with different parameters).
Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
|
|
|
|
|
|
| |
They were all run onto the same line before with no linebreaks.
Change-Id: Ibdff91a23221034a3d345542f1268f8863cf4515
|
|
|
|
| |
Change-Id: Ife16546fc90e26919672f6658b30084bafa42f94
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit allows the specification of one or more strata/chunks
in a deployment entry in a cluster morphology to deploy instead of
the full system if --partial is set. These are listed in a
'partial-deploy-components' field in each deployment definition.
The components must be in the system, and this only works for
tarball or sysroot deployments. It SHOULD NOT be used when
deploying production systems, as it has a number of limitations.
Change-Id: I04ac58af57216335d9257f6620d09f18f61ea714
|
|
|
|
| |
Change-Id: I048687305054f08b0d8e01daa1366170b3eec323
|
|
|
|
|
|
|
|
|
|
|
| |
Its going to be more likely that a baserock user is more interesed
in the build log than actually the debug output from morph.
And also the intuitive option would be to use -v for this.
--verbose/-v: show build output
--debug/-d: show morph debug output
Change-Id: I1fb99034dc8680a5f168f6306724663aea33ebc5
|
|
|
|
|
|
|
| |
Previously it was possible to have some strange values for
pluginmgr.locations if the same path was appeared twice
Change-Id: I3ec257e48e85c3fd30759c3dcc2064f0b151ec45
|
|
|
|
|
|
|
|
| |
Printing the file and line number of the warning's origin
makes the warning itself harder to spot, and can make it
appear as if something has actually gone wrong.
Change-Id: Ie77357d9a061d31e234e6e434521d2d274409773
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reduces noise on the console when using `morph build --verbose`.
Before (example from my current chroot):
$ morph build systems/devel-system-x86_64-generic.morph --verbose
2015-03-31 09:25:08 new environment variable SCHROOT_UID = "0"
2015-03-31 09:25:08 new environment variable SCHROOT_CHROOT_NAME = "baserock-2015-03-24-ostree"
2015-03-31 09:25:08 new environment variable LOGNAME = "root"
2015-03-31 09:25:08 new environment variable XDG_SEAT = "seat0"
2015-03-31 09:25:08 new environment variable PATH = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
2015-03-31 09:25:08 new environment variable XDG_VTNR = "1"
2015-03-31 09:25:08 new environment variable HOME = "/root"
2015-03-31 09:25:08 new environment variable SCHROOT_ALIAS_NAME = "baserock-2015-03-24-ostree"
2015-03-31 09:25:08 new environment variable TERM = "xterm"
2015-03-31 09:25:08 new environment variable SHELL = "/bin/bash"
2015-03-31 09:25:08 new environment variable SHLVL = "1"
2015-03-31 09:25:08 new environment variable SCHROOT_USER = "root"
2015-03-31 09:25:08 new environment variable EDITOR = "vim"
2015-03-31 09:25:08 new environment variable SCHROOT_GID = "0"
2015-03-31 09:25:08 new environment variable SCHROOT_GROUP = "root"
2015-03-31 09:25:08 new environment variable GIT_NO_REPLACE_OBJECTS = "1"
2015-03-31 09:25:08 new environment variable SCHROOT_COMMAND = "/usr/bin/env SSH_AUTH_SOCK=/tmp/tmp.eVw95CcpjS/auth-sock /bin/bash -l"
2015-03-31 09:25:08 new environment variable PYTHONPATH = "/src/morph/"
2015-03-31 09:25:08 new environment variable SSH_AUTH_SOCK = "/tmp/tmp.eVw95CcpjS/auth-sock"
2015-03-31 09:25:08 new environment variable SCHROOT_SESSION_ID = "baserock-2015-03-24-ostree-f3c36e20-c3db-4404-8808-e805ca0c1ac4"
2015-03-31 09:25:08 new environment variable MANPAGER = "less -R"
2015-03-31 09:25:08 new environment variable XDG_SESSION_ID = "2"
2015-03-31 09:25:08 new environment variable _ = "/src/morph/morph"
2015-03-31 09:25:08 new environment variable OLDPWD = "/src/morph"
2015-03-31 09:25:08 new environment variable PWD = "/src/ws/master/baserock/baserock/definitions"
2015-03-31 09:25:08 new environment variable USER = "root"
2015-03-31 09:25:08 Starting build 8f80a816dfe44f6c8352a4242b84bf48
2015-03-31 09:25:08 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master
2015-03-31 09:25:08 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f6137060cd0>
2015-03-31 09:25:15 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph
...
After:
$ morph build systems/devel-system-x86_64-generic.morph --verbose
2015-03-31 09:23:38 new environment variable GIT_NO_REPLACE_OBJECTS = "1"
2015-03-31 09:23:38 Starting build acfdf9a405384e1ea4e1b62cd1bce27f
2015-03-31 09:23:38 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master
2015-03-31 09:23:38 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f8fcbe86bd0>
2015-03-31 09:23:45 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph
...
Change-Id: I20bbdf35b280ccd305bc7e4a70b5f3de7679a368
|
|
|
|
| |
Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This saves a duplicate `git remote update origin` that was being run as
part of each chunk build. For any repos that have submodules, it also
avoids updating repos if the SHA1 we need to build is already present
locally.
As well as speeding up builds slightly, this means Morph can now build
without being connected to a network, as long as the local Git cache
all of the necessary repos and commits in the build, without needing the
'--no-git-update' option.
The code is also now in a more logical place than before.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Not everyone is a fan of the `morph build` magic that collects up your
changes and puts them in a temporary branch. Now you can disable it by
setting 'local-changes=ignore' in your morph.conf file.
This speeds up `morph build` and `morph deploy` by 5-10 seconds on my
machine.
I looked an option to make `morph build` warn if there are uncommitted
changes. I found that with a cold cache, it takes about 5 seconds on my
machine to verify that there are no uncommitted changes to a checkout of
definitions.git. That defeats the main purpose of this patch for me, so
I didn't include the option.
|
|
|
|
|
|
|
| |
This avoids writing each command to the log file twice, as we did
previously. Also, the user-visible message is now only constructed if
we are definitely going to write it to the screen (a tiny optimisation).
Hopefully the logic is clearer now too.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We were attempting to display commandlines with correct shell-escaping,
but the logic was super broken so users would end up seeing this sort of
thing instead:
2015-01-02 10:25:42 # g | i | t | | r | e | v | - | p | a | r | s | e
| | - | - | v | e | r | i | f | y | | ' | 9 | 8 | f | e | a | 8 | 7
| b | 7 | 2 | 7 | 2 | 5 | 3 | e | 7 | f | f | 8 | 1 | 0 | 5 | 4 | 3 | 4
| c | 9 | e | a | 9 | 0 | 2 | b | b | 6 | a | 6 | f | 7 | e | ^ | { | c
| o | m | m | i | t | } | '
Commandlines should now display as intended, more like this:
2015-01-02 10:57:17 # git rev-parse --verify
'9df9643842e4b4d8ece710fe6105f32fa38a0d22^{commit}'
This broken logic was introduced as a post-review fixup in merge commit
c57952ef44a0f1f161441970fcf2f27a39b0de7c.
|
|\
| |
| |
| |
| |
| |
| | |
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
Reviewed-By: Mike Smith <mike.smith@codethink.co.uk>
|
| | |
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This affects errors encountered at build time and at system-integration time.
New errors look like this:
ERROR: Command failed: baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000:
Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/var/tmp/staging/tmp1YQ2yN/minimal-system-x86_64-generic.inst'}
Error output:
+ install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644
install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
Previously the error message would have been this:
Command failed: unshare --mount -- sh -ec.
mount --make-rprivate /
root="$1"
shift
while true; do
case "$1" in
--)
shift
break
;;
*)
mount_point="$1"
mount_type="$2"
mount_source="$3"
shift 3
path="$root/$mount_point"
mount -t "$mount_type" "$mount_source" "$path"
;;
esac
done
exec "$@"
- /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ dev/shm tmpfs none tmp tmpfs none -- linux-user-chroot --chdir . --mount-proc proc
/var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000
+ install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644
install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
|
|
|
|
|
| |
We were passing a nonsensical value for the 'kind' parameter so it
would always return an empty list.
|
|
|
|
|
|
|
|
| |
This code is an essential part of 'morph build'. It's quite complex and
really shouldn't be mixed in with the base Application class.
Given a dedicated class we can store some state in the object and avoid
functions with seven parameters, too.
|
|
|
|
|
|
| |
The MorphologyFactory class only uses the status() function of the
morphlib.Application instance that it gets passed. So make it require
only a status callback.
|
|
|
|
|
| |
This means that we can force the building of a specific commit without
losing the original branch name in the metadata of the resulting system.
|
|\
| |
| |
| |
| |
| | |
Reviewed-by: Lars Wirzenius (+2 to misc fixups)
Reviewed-by: Sam Thursfield (+1 to per-source building)
Reviewed-by: Paul Sherwood (+1 to per-source building)
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Building per-artifact results in undesirable behaviour,
as multiple artifacts are produced for every chunk build.
It therefore makes more sense to build per-source.
This implies that actually, the model of one source per
morphology is wrong and we should move the dependencies
into the source.
Unlike chunks however, where every chunk artifact has the
same dependencies, stratum artifacts can have different
dependencies.
So before we can move the dependencies into the Source,
we need to have as many Sources as Stratum Artifacts.
|
| |
| |
| |
| |
| | |
The upstream cliapp project is not interested in this functionality
right now.
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This involved rewriting the util.log_dict_diff() function. It has been
renamed to log_environment_changes() to better reflect its purpose.
It no longer logs both the old and new values in the event of an
environment variable changing. It now just logs the new value. This makes
the code simpler and seems like it should not be a big problem.
Some projects recommend passing credentials through the environment.
OpenStack does this, for example, see:
<http://docs.openstack.org/user-guide/content/cli_openrc.html>
It's unlikely that users would be happy about applications saving
these passwords in log files all over their system.
I do not recommend ever storing valuable passwords in the environment.
|
|
|
|
|
|
|
|
|
|
|
| |
This was previously used just so it could get the right repo and ref to
read the file out of.
However, there was a subtle bug in this behaviour, as if we had not
previously used morph build in that branch, it would attempt to read the
extensions from a branch which didn't exist.
So now it reads it from the working tree, which always exists.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It will now load the morphology from the definitions repository if the
"morph" field is present in the chunk spec.
Rather than adapting the loop to fit yet-more changing circumstances, it
has been partially rewritten, so there is one loop for loading
systems and strata from definitions.git, another for chunks from
definitions.git, and a third for chunks in the source repository.
This is tidier than attempting to fit the logic in the main loop, as it
makes it easier to remove afterwards when we no longer need to load
chunk morphologies from the source repository.
|
|\
| |
| |
| |
| |
| |
| | |
'origin/baserock/richardmaw/S11284/morphologies-by-path-v4'
Reviewed-by: Sam Thursfield
Reviewed-by: Lars Wirzenius
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I was getting the following error when running the 'do-release.py'
script:
ERROR:root:Command failed: morph --quiet --trove-host=git.baserock.org list-artifacts baserock:baserock/definitions sam/auto-release minimal-system-x86_32-generic
ERROR: Ref d67a0e110187abd560a1de63fa172894a52839d5 is an invalid reference for repo git://git.baserock.org/delta/linux
The commit that it wants did actually exist in git.baserock.org and the
logs show that Morph hadn't done a `git remote update` to get the commit
locally.
This turned out to be because it'd had already looked up a different ref
in linux.git. It hadn't needed to run 'git remote update', but it *had*
added linux.git to a "don't need to update this repo again" list. Oops!
I've moved the code in question to the cachedrepo module so that the repo
object keeps that of whether it should be updated. The bug is now fixed.
As a side effect this patch fixes the spurious 'Updating repo
file:///...' messages, which were printed even though repos with
file:/// URLs are never updated.
|
|
|
|
|
|
|
|
|
|
| |
When the given ref points to a specific commit, and it's already available in the
locally cached copy of repo, there's no need to update the repo.
If the ref points to a branch or tag, and the user didn't pass --no-git-update, the
locally cached copy of the repo will still be updated.
This should speed up many Morph commands.
|
| |
|
|
|
|
|
|
| |
Add a command 'help-extensions' to list all extensions.
Add the ability to find help on an extension by calling
'morph help [extension name]'.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously this resulted in a confusing traceback.
This is a quick fix. I'd rather insert the error in the buildcommand
module instead, but that code assumes the source being built is a
system in several places before it actually checks the kind. Those
would all need to be changed, or the code would need to reworked to
call _validate_root_kind() much earlier.
The Application.traverse_morphs() method is rather ugly anyway, so I'm
happy to add further ugliness to it for the time being.
|
|
|
|
|
| |
'--help' when used with a subcommand will show the subcommand help.
Do not reflow the help text by using a custom formatter.
|
|
|
|
| |
This reverts commit a72c8dca6965d1ac239e4f0102f08fbf7fe59ac7.
|
|
|
|
| |
This reverts commit 329b81419be20e7b1f2651a47030186216044eec.
|
|
|
|
|
|
|
| |
Add a command 'help-extensions' to list all extensions.
Add the ability to find help on an extension by calling
'morph help [extension name]'. This will then call
the extension with the '--help' option to obtain help text.
|
|
|
|
|
| |
'--help' when used with a subcommand will show the subcommand help.
Do not reflow the help text by using a custom formatter.
|
| |
|
| |
|
|\
| |
| |
| |
| | |
Reviewed by: Lars Wirzenius
Reviewed by: Pedro Alvarezwq
|
| | |
|
|/ |
|
| |
|
|
|
|
|
|
|
|
| |
The cliapp formatting doesn't understand Markdown. Adding
Markdown support in Morph is more work than there is time
for right now, but we'll do it later. This quick hack just
means we output the raw Markdown rather than something that
is incomprehensible due to ruined formatting.
|
|
|
|
|
| |
This also removes the long-obsolete code to install staging fillers
in the staging area. We've not allowed users to do that for ages now.
|
| |
|
| |
|