summaryrefslogtreecommitdiff
path: root/zuul/merger
Commit message (Collapse)AuthorAgeFilesLines
* Git driverFabien Boucher2017-12-153-0/+39
| | | | | | | | | | | | | | | | | This patch improves the existing git driver by adding a refs watcher thread. This refs watcher looks at refs added, deleted, updated and trigger a ref-updated event. When a refs is updated and that the related commits from oldrev to newrev include a change on .zuul.yaml/zuul.yaml or zuul.d/*.yaml then tenants including that ref is reconfigured. Furthermore the patch includes a triggering model. Events are sent to the scheduler so jobs can be attached to a pipeline for running jobs. Change-Id: I529660cb20d011f36814abe64f837945dd3f1f33
* Add command socket support to zuul-mergerPaul Belanger2017-12-061-0/+29
| | | | | | | | Like we have in zuul-executor, add command socket support for zuul-merger. Change-Id: I66a2cb2ba3f55bdd03e884f47648278e30d2f6ab Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Add support for override-checkout, deprecate override-branchJames E. Blair2017-10-301-1/+3
| | | | | | | | | | | | | | | | We want to support jobs specifying that they check out a tag rather than merely a branch. This accidentally worked in Zuul v2 with zuul-cloner, and some jobs have come to rely on the behavior. There's no reason not to support it, so let's do so. However, for clarity, change the name of the option to "override-checkout". This is intended to be intuitive (anything you can tell git to 'checkout' you can put here), while avoiding the suggestion that only branches or tags may be checked out. The old form, "override-branch" is deprecated and will be removed. Change-Id: Icc2907e72596626e96d2dc9f6ab1c3026f4085ab
* Add git timeoutJames E. Blair2017-10-101-21/+44
| | | | | | | | | | | | | | | | | | | | | Timeout remote git operations after 300 seconds. Because it could be in an invalid state, delete the local repo if a timeout occurs (subsequent operations will recreate it). This replaces our use of the clone_from() and fetch() methods from GitPython with lower-level equivalents. The high-level methods do not currently permit the hard timeout. The GitPython requirement is changed to a temporary fork until both https://github.com/gitpython-developers/GitPython/pull/682 and https://github.com/gitpython-developers/GitPython/pull/686 end up in a release. Change-Id: I7f680472a8d67ff2dbe7956a8585fb3714119e65
* Create git_http_low_speed_limit / git_http_low_speed_timePaul Belanger2017-10-052-11/+17
| | | | | | | | | Expose 2 new settings to zuul.conf allowing an operator better control over them. By default we set the speed limit to 1000 bytes and speed time to 30 seconds. Change-Id: I9da80fcfc312cbc12ea11ee7284eaec23adb97c9 Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Add git timeout for HTTP(S) operationsPaul Belanger2017-10-051-3/+7
| | | | | | | | | | Using the GIT_HTTP_LOW_SPEED_LIMIT and GIT_HTTP_LOW_SPEED_TIME environmental variables, we have the ability to support timeouts via HTTP(S) for git. Currently, it is possible for a merger to block for ever if something happens to the network. Change-Id: I8ab245b221eae3a9faacd0b9ba6f7b14b27e6b0e Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Fix typo with merger.getRepoStatePaul Belanger2017-09-121-1/+1
| | | | | | | getItemRepoState() doesn't exist, rename to getRepoState(). Change-Id: I95a42293397ac75fa7e0039a353db37e933d2915 Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Abort reconfiguration when cat jobs failJames E. Blair2017-09-121-3/+3
| | | | | | | | | Currently, if a cat job fails during reconfiguration, we simply proceed without that section of the config, which usually doesn't work out well. Instead, raise an exception which will abort the reconfiguration. Change-Id: I87f2d870f007e3df5f47c04ef49add27c8a0b554
* Merge "Clean up obsolete TODO for _setGitSsh()" into feature/zuulv3Zuul2017-09-091-4/+0
|\
| * Clean up obsolete TODO for _setGitSsh()Jeremy Stanley2017-09-051-4/+0
| | | | | | | | | | | | | | | | | | Change Idcc9808948b018a271b32492766a96876979d1fa addressed the need for connection context in zuul.merger.merger.Merger.updateRepo() but failed to remove the TODO comment associated with it. Clean up the unnecessary comment now. Change-Id: I57a48a27a09b39efdc4d15f1a3a674b83a1d1199
* | Use 'git checkout' when checking out a tagJames E. Blair2017-09-081-1/+4
|/ | | | | | | | | | | | | | | The way we were checking out tags did not match the results that would happen from a simple 'git checkout tagname', which made it difficult for us to push repos from the executor to the workers. It may also have eventually caused inconsistent results with git tooling in jobs since it didn't match the way most folks would check out a tag. (In particular, this method did not appear to update .git/logs/HEAD, which is used by git status.) Change to just use the git checkout command rather than the lower-level manipulation we were doing. Change-Id: I77e9900c73caf16032dbadb9991358caed0e07ed
* Ensure ref-updated jobs run with their refJames E. Blair2017-08-023-8/+81
| | | | | | | | We were incorrectly preparing the current state of the repo for ref updated (eg, post) jobs. This ensures that we run with the actual supplied ref, even if the remote has moved on since then. Change-Id: I52f05406246e6e39805fd8365412f3cb77fe3a0a
* Remove zuul_url from merger configTobias Henkel2017-07-312-7/+3
| | | | | | | Currently the zuul_url is not used anywhere but still a required merger setting. This removes it. Change-Id: I627c8a18015f4c148c28d2f7e735b30cc1ef3862
* Remove refspecJames E. Blair2017-07-211-9/+10
| | | | | | | | This internal-only attribute is basically the same as "ref" but spelled differently only in the case of a change. Just use the "ref" name in all cases for improved developer sanity. Change-Id: I476f8d32dae37309ab0c9e11c8a5337b213f985e
* Add support for zuul.d configuration splitTristan Cacqueray2017-07-103-13/+25
| | | | | | | This change implements the zuul_split spec to support configuration split in a zuul.d directory. Change-Id: I6bc7250b2045b73dfba109aa0b2f1ba5d66752b2
* config: refactor config get defaultTristan Cacqueray2017-06-172-46/+14
| | | | | | | This change adds a new get_default library procedure to simplify getting default value of config object. Change-Id: I0546b1175b259472a10690273af611ef4bad5a99
* Add ssl support to gearman / gearman_serverPaul Belanger2017-06-142-2/+26
| | | | | | | | | Enable SSL support for gearman. We also created an new SSLZuulBaseTest class to provide a simple way to use SSL end to end where possible. A future patch will enable support in zookeeper. Change-Id: Ia8b89bab475d758cc6a021988f8d79ead8836a9d Signed-off-by: Paul Belanger <pabelanger@redhat.com>
* Fix git ssh env in mergerJames E. Blair2017-06-091-44/+36
| | | | | | | | | | | | A recent change switched from using .ssh_wrapper files to using a custom environment to set the ssh command in the merger. However, it was incomplete. Ensure that all instances of git.Repo objects have the necessary environment variable set. Change-Id: Ie69c736ded1628e5ccf7afd4535d31c6c9b20215 NB: this feature (enabled in the earlier change) requires Git 2.3+.
* Actually check out the branch in the jobdirJames E. Blair2017-05-311-2/+4
| | | | | | | | This *actually* checks out the branch in the staged git repos on the executor, rather than checking out the branch tip commit as a detached head. Change-Id: I2edb79d14f51ef61f74745d2409df8f89ba70589
* Update the merger recent dict when saving the repo stateJames E. Blair2017-05-311-7/+9
| | | | | | | | | | | | | | | | | | | | | This will have the effect of only saving or restoring the repo state once in the merger. The "recent" dictionary is consulted by the merger before it merges each change to find upon which change the merge should be based. If it contains an entry for a project-branch, that entry is used. Otherwise, it resets the repo and finds the branch tip. By populating it when we save the repo state, we can ensure that a repo is reset only once during a merge, and since the repo_state dict is only updated when there is no recent entry for a project-branch, the repo state is only updated once per project as well. The same applies to restoring the repo state -- the restore is performed right before the save, so by fully populating recent after the restore, we will only restore a repo state once. Change-Id: I6fbdf4ec9be303bb6b90caa724ee2c813673a431
* Add LoggerAdapter for executor jobsJames E. Blair2017-05-261-7/+14
| | | | | | | | | | | | Create a logger adapter for executor jobs that ensures that every log entry has the job uuid associated with it. Pass this into the associated mergers as well, since they are also performing interesting tasks in parallel on the executor. The uuid is also added to the extras dict so that it's available to logging systems which can handle extra data. Change-Id: I28e0bcd0f030361659e5a72c162d549c1f0d6acb
* Check out the appropriate branch in executorJames E. Blair2017-05-251-0/+4
| | | | | | | | | | | This mimics the behavior of Zuul cloner and allows the user to specify what branches of each repo should be checked out in the jobdir at the start of the job. (Of course, the job is free to check out other branches as needed; all of them will have the appropriate future state.) Change-Id: I93af5c49cb0404944636c7e63d203cdb564b267c
* Use the executor cached repos more oftenJames E. Blair2017-05-251-12/+39
| | | | | | | | | | | | Currently, we clone the job's working repos from the repo cache that the executor maintains. However, some playbook and role repos do not use that cache. To facilitate more operations using the merger class, add the concept of a repo cache to it, and use that in all job related git repo operations on the executor. Change-Id: I747b6602540458506e1f6d480a95b80c6543c5a8
* Return resulting commits from mergerJames E. Blair2017-05-252-6/+15
| | | | | | | | | | | | | | | | | | | | | Currently the results (i.e., the exact commits on each project-branch) of a speculative merge operation are only available by examining the Zuul refs created by the merger. The executor appears to use the results of the speculative merge in its tests, but that's only surface deep -- it simply leaves the repo in the last merge state, which is sufficient for simple tests, but not for multi-branch cross-repo dependencies. This change returns the branch names and commits for each project-branch involved in a speculative merge. The executor then updates the branch head references in the job's working source directory to reflect the merger results. In other words, even if the job is running for a change on master, if that change depends on a change in stable then the stable branch of the repo will now include the stable change. Note: this change does not yet alter the branch checked out -- we're still on a detached head. That will be addressed in a future change. Change-Id: Id842d64312d87709f0ed93121735fe97faccc189
* Use previously stored repo state on executorJames E. Blair2017-05-242-1/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When the initial speculative merge for a change is performed at the request of the pipeline manager, the repo state used to construct that merge is saved in a data structure. Pass that structure to the executor when running jobs so that, after cloning each repo into the jobdir, the repos are made to appear the same as those an the merger before it started its merge. The subsequent merge operatons on the executor will repeat the same operations producing the same content (though the actual commits will be different due to timestamps). It would be more efficient to have the executors pull changes from the mergers, however, that would require the mergers to run an accessible git service, which is one of the things that adds significant complexity to a zuul deployment. This method only requires that the mergers be able to initiate outgoing connections to gearman and sources. Because the initial merge may happen well before jobs are executed, save the dependency chain for a given BuildSet when it's configuration is being finalized. This will cause us to save not only the repository configuration that the merger uses, but also the exact sequence of changes applied on top of that state. (Currently, we build the series of changes we apply before running each job, however, the queue state can change (especially if items are merged) in the period between the inital merge and job launch). The initial merge is performed before we have a shadow layout for the item, yet, we must specify a merge mode for each project for which we merge a change. Currently, we are defaulting to the 'merge-resolve' merge mode for every project during the initial speculative merge, but then the secondary merge on the executor will use the correct merge mode since we have a layout at that point. With this change, where we are trying to replicate the initial merge exactly, we can't rely on that behavior any more. Instead, when attempting to find the merge mode to use for a project, we use the shadow layout of the nearest item ahead, or else the current live layout, to find the merge mode, and only if those fail, do we use the default. This means that a change to a project's merge-mode will not use that merge mode. However, subsequent changes will. This seems to be the best we can do, short of detecting this case and merging such changes twice. This seems rare enough that we don't need to do that. The test_delayed_merge_conflict method is updated to essentially invert the meaning of the test. Since the old behavior was for the initial merge check to be completely independent of the executor merge, this test examined the case where the initial merge worked but between that time and when the executor performed its merge, a conflicting change landed. That should no longer be possible since the executor merge now uses the results of the initial merge. We keep the test, but invert its final assertion -- instead of checking for a merge conflict being reported, we check that no merge conflict is reported. Change-Id: I34cd58ec9775c1d151db02034c342bd971af036f
* Store initial repo state in the mergerJames E. Blair2017-05-243-30/+38
| | | | | | | | | | | | | | | | | | | | | | | | When we ask a merger to speculatively merge changes, record the complete starting state of each repo (defined as all of the refs other than Zuul refs) and return that at the completion of all of the merges. This will later be used so that when a pipeline manager asks a merger to speculatively merge a change, the process can later be repeated by the (potentially multiple) executors which will end up running jobs for the change. Between the time that the merger runs and the jobs run, the underlying repos may have changed. This ensures a consistent state throughout. The facility which used saved zuul refs within the merger repo to short-cut the merge sequence for an additional change added to a previously completed merge sequence is removed, because in that case, we would not be able to know the original repo state for the earlier merge sequence. This is slightly less efficient, however, we are proposing removing zuul refs anyway due to the maintenance burden they cause. Change-Id: If0215d53c3b08877ded7276955a55fc5e617b244
* Merge "Remove unused merger:update task" into feature/zuulv3Jenkins2017-05-202-17/+0
|\
| * Remove unused merger:update taskJesse Keating2017-05-172-17/+0
| | | | | | | | | | | | | | | | | | | | This task is no longer used and was the last thing that merger claimed to do that executor did not. Now what merger does is a subset of executor, thus merger can scale out to handle things that leave the executor(s) free to focus on running jobs. Change-Id: Ibc8638cf7c2109d9b32c27fb98fb84605f5d5ac0 Signed-off-by: Jesse Keating <omgjlk@us.ibm.com>
* | Encoding changes in tests for py3Clint Byrum2017-05-191-1/+1
| | | | | | | | | | | | | | In many cases we do need to be explicit about bytes vs. strings for python 3 compatibility. Change-Id: I9cbc5c73004d03f711f8a6e5752a0865ae55fb9f
* | py3 Changes in __del__ for gitpythonClint Byrum2017-05-191-7/+6
| | | | | | | | | | | | | | | | | | | | gitpython in py2 relied upon the behavior of __del__ to release locks on config_writer objects. In py3, because __del__ may happen much later than when an object drops out of scope, we must explicitly release locks either with the .release() method or by using the config writer as a context manager. Change-Id: Ic09eb7af6757eefbb0f8c86cb1bf3e8ca5f007b9
* | Use gear Text interfaceClint Byrum2017-05-192-2/+2
|/ | | | | | | | This makes the transition to python3 much smoother. Change-Id: I9d8638dd98502bdd91cbe6caf3d94ce197f06c6f Depends-On: If6bfc35d916cfb84d630af59f4fde4ccae5187d4 Depends-On: I93bfe33f898294f30a82c0a24a18a081f9752354
* Use connection to qualify projects in mergerJames E. Blair2017-04-273-45/+57
| | | | | | | | | | | | | | | | | | | | | | | Fully qualify projects in the merger with connection names. This lets us drop the URL parameter (which always seemed unecessary, as the merger can figure that out on its own given a uniquely identified project). On disk, use the canonical hostname, so that the checked out versions of repositories include the canonical hostname, and so that repos on mergers survive changes in connection names. This simplifies both the API and the JSON data structure passed to the merger. The addProject method of the merger is flagged as an internal method now, as all "public" API methods indirectly call it. In the executor, after cloning and merging are completed, the 'origin' remote is removed from the resulting repositories since it may not be valid for use within a running job. Change-Id: Idcc9808948b018a271b32492766a96876979d1fa
* Perform pre-launch merge checksK Jonathan Harker2017-04-041-1/+4
| | | | | | | | | | | | | | | | | | Move merge scheduling to its own function in the pipeline manager, and call it in the main _processOneItem loop once the item has entered the active window, in addition to the previous location in the executor when getting the layout information. Since we are now scheduling merges for all items in any pipeline, make sure we properly handle both Ref and Change objects. Also, if the executor encounters a merger failure, immediately report that result. Change-Id: I1c9db6993994bf8e841ecd8554c37a3ec0afc798 Co-Authored-By: Adam Gandelman <adamg@ubuntu.com> Story: 2000773 Task: 3468
* Merge "Merge branch 'master' into feature/zuulv3" into feature/zuulv3Jenkins2017-03-062-1/+14
|\
| * Merge branch 'master' into feature/zuulv3Joshua Hesketh2017-03-062-1/+14
| |\ | | | | | | | | | Change-Id: I37a3c5d4f12917b111b7eb624f8b68689687ebc4
| | * Fix setting of GIT_SSH for timer merge jobsEvgeny Antyshev2017-02-083-11/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Problem is that merge jobs triggered by timer don't supply "refspec" event attribute, and, naturally, nothing is merged but the repo is updated. But, in such case, no connection info is passed to merger, and GIT_SSH is not set. So, the merge job will fail (perhaps, until some other change events with "refspec" defined will make this merger to set GIT_SSH properly) Change-Id: I6dd96ae0bdcdf4c3db4c931a8c6ef7d5edfc1257
| | * Merger: fully update repo on repo updateJames E. Blair2016-10-221-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a merger gets a request to update a repo for anything other than a pre-merge job (eg, ref-updated, periodic, etc), it simply updates the repository from upstream. While it did update its tags and remote branch references, it did not actually update its local branch heads. Switch to using the 'reset' method so that happens (along with the update which was already happening). This corrects a situation where a periodic or post job might pull data from a repo on a zuul merger and end up with out of date information. Change-Id: Iccc691974ae5efedec9bfa5b553b451a1b5ab686
* | | Catch gear.InterruptedErrorJames E. Blair2017-03-031-0/+2
|/ / | | | | | | | | | | | | | | | | This is how gear lets us know that it has been stopped by another thread. Since we weren't catching these exceptions explicitly, they were polluting the logs on shutdown (especially in tests). Catch them and return immediately from the daemon run methods. Change-Id: I6f845566c759651e630d7f93c94dce975ad53d30
* | Fix race in sshwrapperJoshua Hesketh2017-02-211-48/+39
| | | | | | | | | | | | | | | | With multiple mergers modifying the os-env GIT_SSH this caused some races. Instead use gitpython's context manager for setting the ssh command and therefore private key. Change-Id: I55e628ffd429f6028ae2f9931d86731fe9807365
* | Merge "Split merger and launcher git roots" into feature/zuulv3Jenkins2017-02-151-1/+1
|\ \
| * | Split merger and launcher git rootsJames E. Blair2017-02-091-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In case a user runs a merger and a launcher on the same host, make sure that they don't share a git directory (used by the launcher's internal merger). They could end up colliding. Incidentally, that's basically the configuraton used in tests, so update the test configuration likewise. Change-Id: I64a690c706d00583973bd2d542a5f42ae6e9ef36
* | | Reset merger repo when performing an updateJames E. Blair2017-02-141-1/+1
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When a merger "updates", it fetches remote data but does not alter any of the local branches. However, the new "getFiles" merger function expects to be able to fetch data from any given branch. If a merger is "updated" and then calls "getFiles", it will return data from whatever commit the local branch points to, not what the newly-updated remote branch points to. This change alters the "update" command to "reset" the repo. A merger reset is an update as above, followed by a forced re-creation of all the local branches to mirror the remotes. This will cause getFiles (as long as it is preceded by an update) to return the correct data. Change-Id: Ic475c4437dcd045d8d07a391159d555a379e97b8
* | Add some gearman related debuggingJames E. Blair2017-02-061-1/+1
| | | | | | | | | | | | | | | | Make sure all clients are identified. Log the port on which the gearman server is listening in tests. Log the arguments for the launch job. Change-Id: Ia99ea5272241799aa8dd089bdb99f6058838ddff
* | Use playbooks defined in reposJames E. Blair2017-02-031-0/+10
| | | | | | | | | | | | | | | | | | This replaces the stubbed-out 'hello world' Ansible playbook with an implementation which actually runs the corresponding playbook defined in the repo where the job is defined. Change-Id: I73a6b3b067c7d61bb2a2b2140ab98c4944a6adfe Story: 2000772
* | Merge branch 'master' into feature/zuulv3Joshua Hesketh2016-09-201-3/+14
|\ \ | |/ | | | | Change-Id: Id2789441d6b651c736c3d046a373eaea1921cf2d
| * Handle non-valid HEADArie2016-08-311-2/+6
| | | | | | | | | | | | | | | | | | | | Repo might have HEAD that points on master branch while in reality it doesn't exists. This fix will choose first reference in case origin/HEAD doesn't exist. Change-Id: I22fec070af4445c1cb3dd34e9c3b7f10e349bd0f
| * Cloner: correct fetch on pre-1.9 git2.5.0James E. Blair2016-07-291-0/+7
| | | | | | | | | | | | | | | | | | | | Commit 6c54b531a36644d53ae054d3fdf846fb2e538677 changed the cloner to use 'fetch --tags' instead of update, however, 'fetch --tags' does not actually update heads before git version 1.9. So if we are using old versions of git, run both commands so that we get persistent behavior. Change-Id: I8ea8ac2c6a8384845a3310da63974afc3279564f
| * Use fetch instead of update in mergerJames E. Blair2016-07-141-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | According to a well written and therefore almost certainly correct answer on SO[1], fetch and update are nearly the same, especially when used in the context of a single remote (as we are here). Since we sometimes want correct tag information as well as correct branch information for our repos, switch from update to fetch, with the '--tags' argument. [1] http://stackoverflow.com/questions/2688251/what-is-the-difference-between-git-fetch-origin-and-git-remote-update-origin Change-Id: I3a31b84e7d65efa5430fb29a77047ae4b16ad18c
* | Add dynamic reconfigurationJames E. Blair2016-07-183-11/+28
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | If a change alters .zuul.yaml in a repo that is permitted to use in-repo configuration, create a shadow configuration layout specifically for that and any following changes with the new configuration in place. Such configuration changes extend only to altering jobs and job trees. More substantial changes such as altering pipelines will be ignored. This only applies to "project" repos (ie, the repositories under test which may incidentally have .zuul.yaml files) rather than "config" repos (repositories specifically designed to hold Zuul configuration in zuul.yaml files). This is to avoid the situation where a user might propose a change to a config repository (and Zuul would therefore run) that would perform actions that the gatekeepers of that repository would not normally permit. This change also corrects an issue with job inheritance in that the Job instances attached to the project pipeline job trees (ie, those that represent the job as invoked in the specific pipeline configuration for a project) were inheriting attributes at configuration time rather than when job trees are frozen when a change is enqueued. This could mean that they would inherit attributes from the wrong variant of a job. Change-Id: If3cd47094e6c6914abf0ffaeca45997c132b8e32
* | Merge branch 'master' into v3_mergeJoshua Hesketh2016-07-142-2/+2
|\ \ | |/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Includes minor py3 fixes (for pep8 on py3). Conflicts: tests/base.py tests/test_model.py tests/test_scheduler.py tox.ini zuul/model.py zuul/reporter/__init__.py zuul/scheduler.py zuul/source/gerrit.py Change-Id: I99daf9acd746767967b42396881a2dff82134a07
View Lorry Controller Status