| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
| |
A cancel during the 'graphing' or 'annotating' stages would be ignored
as the BuildController was listening for the InitiatorDisconnect message
from the wrong event source. In 'building' state the actual build would
be stopped, but the BuildController instance would stick around due to
sending the message class instead of an instance of the message.
Change-Id: I222a8aa39bf7fffab4d89e12997ffd18cd1b54fc
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In addition to partial builds we also want to be able to do partial
distbuilds, and distbuild uses a different codepath.
This commit updates the distbuild code to know what to do if a partial
build is requested. It only builds up to the latest chunk/stratum that
was requested, and displays where to find the artifacts for each of
the chunks/strata requested upon completion of the build.
The usage is the same as for local builds.
Change-Id: I0537f74e2e65c7aefe5e71795f17999e2415fce5
|
|
|
|
| |
Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
|
|
|
|
| |
Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This message was hundreds of kilobytes in size, as it contained a
recursive list of dependencies for each artifact in the build graph. It
was used in the initiator only to print this message:
Build steps in total: 592
This message is now gone. The 'Need to build %d artifacts'
build-progress message now indicates the total build steps instead:
Need to build 300 artifacts, of 592 total
This is a compatible change to the distbuild protocol: old initiators
will continue to work as normal with new controllers that don't send
the build-steps message.
|
|
|
|
|
|
|
|
| |
Users build sources, not artifacts. So the log files should be called
build-step-systemd.log and not build-step-systemd-misc.log.
Note strata are a kind of special case so you will still see
build-step-foundation-runtime.log, build-step-foundation-devel.log etc.
|
| |
|
|
|
|
|
|
| |
Let the end-user see the URL that distbuild was attempting to talk to,
so they can more easily spot configuration errors. It's kind of silly
to say 'HTTP request failed' without saying where the request was going.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The previous error looked like this by the time it had reached the
initiator's console:
ERROR: Failed to build baserock:baserock/definitions
c7292b7c81cdd7e5b9e85722406371748453c44f
systems/base-system-x86_64-generic.morph.frodsham: Failed to compute
build graph. Problem with serialise-artifact: ERROR: Couldn't find
morphology: systems/base-system-x86_64-generic.morph.frodsham
New message is at least a bit simpler:
ERROR: Failed to build baserock:baserock/definitions
c7292b7c81cdd7e5b9e85722406371748453c44f
systems/base-system-x86_64-generic.morph.frodsham: ERROR: Couldn't
find morphology: systems/base-system-x86_64-generic.morph.frodsham
|
|
|
|
|
|
| |
If there's no distbuild-helper process running on the controller then
the controller would hang forever. This situation is unlikely, but it's
important to give the user feedback instead of silently hanging forever.
|
|
|
|
|
|
|
| |
There's no need to handle failure differently at each stage of the
build. Simpler to use the BuildFailed message for all errors. This
then allows us to have a single self.fail() function that can be used
everywhere.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
The recent changes to the BuildCommand.build() function caused distbuild
to break, because I didn't make the same change to the
InitiatorBuildCommand.build() function but did change how it was called.
This commit adds the ability to have optional fields in distbuild
messages. This is used to add an optional 'original_ref' field, which
will get passed to `morph serialise-artifact` by new distbuild
controllers, and will be ignored by older ones.
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The "unicode fix" worked for the subset of cases relevant, and only
broke distbuild because its tests have not been integrated with ./check,
so the fact that it broke for any string ending with a \ escaped notice,
if you will excuse the pun.
During json.load, the encode option is for specifying the character
encoding of the file or string that is being loaded.
During json.dump, the encode option is for the encoding of `str` keys
and values.
The fact that it worked for the set of cases we cared about is a small
mystery, probably caused by the strings we happened to give it being
valid unicode-escape encoded `str`ings.
A full fix would require either converting all these cases to a
different format, such as YAML, which will handle input data not being
valid Unicode, or pre-processing the data that is passed to `json.dump`
to convert all `str` instances to an appropriately escaped `unicode`,
and converting back on `json.load`, but this is a quick fix to get the
distbuild code working again.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
| |
|
| |
|
|\
| |
| |
| |
| |
| |
| | |
Reviewed by:
Sam Thursfield
Richard Maw
Lars Wirzenius
|
| |
| |
| |
| |
| |
| | |
The contents of the message has changed for several events,
event messages that need to be sent to several initiators have a list
of ids instead of a single id.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There are two new messages:
WorkerBuildStepAlreadyStarted tells the initiator that the artifact
they want to build is already being built, e.g.
'eglibc-misc is already building on 172.17.1.37:3434'
WorkerBuildWaiting tells the initiator that the artifact they want
to build can't be built yet because there aren't any workers free, e.g.
'Ready to build eglibc-misc: waiting for a worker to become available'
|
|/ |
|
|
|
|
|
|
| |
Most of the time knowing the state of the build
graph isn't that useful for debugging distbuild,
but it may be useful in some situations.
|
|
|
|
|
|
| |
The controller should check the response event is
actually the response it was waiting for before
logging that there was a cache response
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
distbuild/build_controller.py
Reviewed by:
Lars Wirzenius
Daniel Silverstone
Sam Thursfield
|
| |
| |
| |
| | |
We now get the state of all artifacts with a single request.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There are cases where a state machine handles an event but stays in the
same state. A callback is registered which filters messages further
before possibly taking action. There have been bugs caused by this
pattern being incorrectly implemented (where the callback is expected to
filter the message, but a transition takes place anyway). Hopefully a
consistent naming convention will make the pattern clearer.
|
| |
| |
| |
| |
| |
| | |
There is always one BuildController object per InitiatorConnection.
By coupling the objects slightly closer we can simplify some transitions
in BuildController.
|
| |
| |
| |
| |
| | |
This is similar to the issue fixed by commit
c38b77bed86acc8b90f253ce354f3ecf98e475e7.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Each BuildController instance sets itself up to receive all messages from
all workers (via the WorkerConnection instances). In the case of a build
failure, all BuildController objects would transition to 'None' state
(causing them all to be destroyed) on any WorkerBuildFailed message.
This meant that if any one build failed on a distbuild network:
- the user whose build actually failed would receive the error
messages correctly
- any concurrent users would see no further build messages from the
controller, making it look like their builds had hung.
Ctrl+C from the 'hung' users would still be correctly handled by the
controller, as their InitiatorConnection instance would still be alive.
|
| |
| |
| |
| | |
Makes it easier to see what they mean at a glance.
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
New DistbuildSocket class that wraps socket.socket(), providing a
descriptive repr() handler showing where the socket is connected, and
providing a couple of helper methods for fetching local and remote
endpoint names.
This commit also adds a descriptive repr() handler to a few other
objects (mostly giving socket connection details).
|
|/ |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Whenever the controller finds a source artifact
it wants to build, it changes its state to BUILDING.
We build all a chunk's source artifacts in one go.
So for any chunk artifact, we change the state of
all chunk artifacts that are built from the
same source to BUILDING
|
|
|