summaryrefslogtreecommitdiff
path: root/lorrycontroller
Commit message (Collapse)AuthorAgeFilesLines
...
* | STATEDB: Generalise 'troves' table to allow more Upstream Host typesBen Hutchings2020-06-014-41/+90
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We need to store different parameters for different upstream host types. Currently the troves table has a 'gitlab_token' column for GitLab upstreams, but in general we might need to have arbitrary parameters. That column's value is also how we determine the host type(!). * Add an explicit 'type' column * Replace the 'gitlab_token' column with a 'type_params' column that is a JSON dictionary of type-specific parameters While we're at it, and since it makes the migration simpler: * Replace 'troves' table with 'hosts' table Closes #10.
* | STATEDB: Comment why the original schema is here, not the current schemaBen Hutchings2020-06-011-0/+5
| | | | | | | | | | | | We are using yoyo to do migrations, but did not use it to set the original schema. We can't move the original schema to a migration since it wouldn't be recorded as already done in existing databases.
* | lorrycontroller.readconf: Fix up 'troves' section type earlyBen Hutchings2020-06-011-2/+5
| | | | | | | | | | | | The 'troves' section type is an alias for 'trove'. Replace it with the latter in LorryControllerConfValidator() before anything else sees it.
* | Update terminology for hosts in internal APIs and web templatesBen Hutchings2020-06-016-134/+132
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is intended to replace all uses of "Trove" that should really be "Downstream Host" or "Upstream Host", except in the database schema and the REST API (which will probably change later). * ARCH: Update example code to reflect API name change * yarns.webapp: Update test descriptions and uses of internal APIs * units: Update and rename the ls-troves units (although they still use the ls-troves endpoint for now) Some references that really are specific to Trove integration are retained. Related to #3.
* | lorrycontroller.statedb: Allow internal keys and column names to divergeBen Hutchings2020-06-011-13/+16
|/ | | | | | | I want to remove "trove" from internal APIs without having to immediately change the database schema. So we'll still have a column "from_trovehost" in the database schema but get_lorry_info should map that to "from_host".
* lorrycontroller.gitano: Fix repeated import of urllib.parseBen Hutchings2020-05-061-1/+0
| | | | This was introduced by "Update to python3". Found by pyflakes.
* Convert output from runcmd{,_unchecked} to textBen Hutchings2020-05-062-1/+7
| | | | | | The cliapp.Application.runcmd{,_unchecked} methods return bytes objects under Python 3. Decode these as UTF-8 in non-strict (replace) mode.
* readconf: Stop comparing dicts directly (python3)Pedro Alvarez2020-01-271-1/+1
|
* Update to python3Pedro Alvarez2019-11-278-51/+51
|
* Make status call copy the static filesPedro Alvarez2019-07-121-1/+17
|
* Support 'local' git-server-type for local mirrorsPedro Alvarez2019-07-091-0/+2
| | | | | In this scenario, where the mirror would be in the local filesystem, lorry will be the one initialising the repository.
* Fix incorrect debug message on statedb removalBen Brown2018-05-201-2/+2
|
* Make migrations run only once per executionpedro/publish-failuresPedro Alvarez Piedehierro2017-07-131-15/+25
| | | | | | | | Yoyo migration libraries were failing in some cases due to "database is locked" errors. It was difficult to track down what parallel operations were causing the problems, so I dediced it was better to run the migrations once per execution instead of executing them everytime we opened a connection with the database (for every request).
* status: Encode generated html as UTF-8Pedro Alvarez Piedehierro2017-07-131-1/+1
| | | | | | Some job outputs include Unicode characters. This made the status.html generation fail given that some job logs are included in the status page.
* status: send 'publish-failures' option to templatePedro Alvarez Piedehierro2017-07-131-4/+9
|
* jobupdate: Store job information in lorry if job failedPedro Alvarez Piedehierro2017-07-132-1/+16
|
* statedb: add columns for last failure informationPedro Alvarez Piedehierro2017-07-132-3/+28
|
* Add support for YAML lorriesAdam Coldrick2016-10-061-2/+6
| | | | | | | | | | | This commit makes lorry-controller understand YAML lorry files. If the file cannot be parsed as YAML then we fall back to attempting to load it as JSON, before giving up on loading it completely. The test for broken JSON is modified to use a string which is invalid for both YAML and JSON. Change-Id: If83e2e44b38e6fb63dbf0b857e143fdcabab78ac
* Make use of the exceptions response_code member variableBen Brown2016-06-171-1/+1
| | | | Change-Id: Iad9139bb6bbce07926ca800c0525601a43341c69
* Add docstring for suitable_path()Ben Brown2016-06-171-0/+12
| | | | Change-Id: I3a95880618218da00991f8ac58dee54d06a0da41
* Remove unused version tableBen Brown2016-06-171-4/+0
| | | | Change-Id: Id79d4d05a5cabd7f7a634dd0decdee6bd4005f29
* Migrations by yoyoBen Brown2016-06-172-9/+30
| | | | Change-Id: I6d06353462badac114c685a2f655bc41c6820aff
* Allow host to be specified instead of trovehostBen Brown2016-06-171-3/+5
| | | | Change-Id: I23c23886b5389d75032c64bf4e78db7edca1af63
* Workaround for GitLab's lack of multi-level namespacingBen Brown2016-06-171-11/+31
| | | | Change-Id: Ic1ef8b983cc84b0ce75dc57f9e05b5cdda343dc6
* Add support for lorrying from a Gitlab serverBen Brown2016-06-174-21/+80
| | | | Change-Id: I2bb0aaf428e331a0bcd5a1e3111d4c7bca4afede
* Add support for mirroring to a GitLab serverBen Brown2016-06-173-2/+101
| | | | Change-Id: I74dc0265fb3c92259101317d655eb55ccb62c119
* Reduce nesting a littleBen Brown2016-04-182-4/+3
| | | | | | Should have done it this way in 2e3195f. Change-Id: I015a3c1241a59277801cf39865f87e8c3ac96747
* Remove trailing whitespaceBen Brown2015-12-156-16/+16
| | | | | | While we're here, seeing as Adam mentioned it. Change-Id: I5ddb86c70d76a84cf12fbd4eb91f3802e490d745
* Perform some sanity checking for 'ignore' and 'globs' fieldsBen Brown2015-12-151-0/+11
| | | | | | | The 'ignore' and 'globs' fields expect values to be string lists, add checks for this in config validation. Change-Id: I96fc8fa217a6f62b6b67f604bd50b2fbea723db5
* 'ignore' field is no longer treated as if it were mandatoryBen Brown2015-12-152-3/+4
| | | | | | | | The README lists 'ignore' in the group of optional keys for trove specifications, yet any attempt to read the configuration without the key being present would result in a KeyError. Change-Id: I05121535b970c6d7382def46ffa720209f794633
* Allow glob patterns in the 'ignore' field for upstream Trovessam/ignore-globsSam Thursfield2015-03-111-5/+9
| | | | | | | The documentation on wiki.baserock.org described the 'ignore' field as "glob patterns on repository pathnames which to not mirror (trove only)" but it was actually being treated as a simple list of repos to ignore. Globs are much more useful.
* Add initial support for mirroring to a Gerrit instancesam/support-gerritSam Thursfield2015-03-113-7/+84
|
* Use 'requests' instead of the awful 'urllib2'Sam Thursfield2014-12-192-21/+7
| | | | | | | | | urllib2 doesn't honour proxy settings despite our best efforts. It turns out that lorry-controller currently ignores all proxy settings. Requests, by comparison, does whatever the 'http_proxy' and 'https_proxy' environment variables tell it to do. This makes lorry-controller actually work behind a proxy.
* Support HTTP(S) proxies which don't require username and passwordSam Thursfield2014-11-241-10/+25
| | | | | Previously the proxy.conf file had to contain 'username' and 'password' fields, which meant that it didn't work for some HTTP proxy setups.
* Merge remote-tracking branch 'origin/baserock/liw/lc-list-failed-jobs'Sam Thursfield2014-11-172-12/+36
|\ | | | | | | | | Reviewed-By: Francisco Redondo Marchena <francisco.marchena@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * Add lists of failed, all jobs to lorry spec infobaserock/liw/lc-list-failed-jobsLars Wirzenius2014-09-092-12/+36
| | | | | | | | This adds them to both the JSON and HTML outputs.
* | Handle case when there is no disk usage infoLars Wirzenius2014-10-221-1/+1
|/ | | | | | | Use 0 instead. This was exposed by a test suite change: test suite creates a dummy job, but doesn't fill in all the fields. With this change, WEBAPP doesn't crash when it tries to report a job's information when not all fields are filled in.
* Implement /1.0/remove-ghost-jobsLars Wirzenius2014-09-044-11/+100
|
* Merge branch 'liw/move-error-msgs'Lars Wirzenius2014-07-071-22/+34
|\ | | | | | | | | Reviewed-by: Sam Thursfield Reviewed-by: Richard Maw
| * Improve move-to-top/bottom API call error messagesliw/move-error-msgsLars Wirzenius2014-06-231-0/+12
| |
| * Move shared code to a base class, less code duplicationLars Wirzenius2014-06-231-22/+22
| |
* | Also catch IOErrorRichard Ipsum2014-06-231-1/+1
| | | | | | | | | | Failure to remove the temp file due to full disk or other IO error is acceptable.
* | Write static status HTML page via temporary fileLars Wirzenius2014-06-231-1/+26
| | | | | | | | This will avoid the file being truncated if the disk fills up.
* | Forget "kill_job" flag for future runs of a lorryliw/anti-exterminateLars Wirzenius2014-06-233-17/+36
|/ | | | | | | | | We do this by moving the "kill_job" column from the lorries table to the jobs table, renaming it to "kill" in the process. It makes no sense to have the flag to kill a specific job in the lorries table. This avoids the need to reset the flag, since it affects only a specific job, instead of all jobs of a lorry.
* Fix job showerbaserock/richardipsum/fix_job_showerRichard Ipsum2014-06-191-1/+1
| | | | | | | | | | If our job hasn't ended then our query for (started, ended) returns the start time and NULL for the end time. With sqlite3 NULL maps to None. time.gmtime(None) returns the current time, so instead of displaying nothing a job that hasn't ended will display the current time as its ended time.
* Merge branch 'baserock/liw/lc-list-all-jobs-optimisation'Lars Wirzenius2014-05-222-3/+22
|\ | | | | | | | | Reviewed-by: Sam Thursfield Reviewed-by: Daniel Silverstone
| * Make listjobs faster by getting all info in one querybaserock/liw/lc-list-all-jobs-optimisationLars Wirzenius2014-05-222-3/+22
| |
* | Merge branch 'master' of ↵Lars Wirzenius2014-05-221-3/+3
|\ \ | | | | | | | | | git://git.baserock.org/baserock/baserock/lorry-controller
| * | Fix over-indentation in readconf.pybaserock/pedroalvarez/fix-readconfPedro Alvarez2014-05-211-3/+3
| |/ | | | | | | | | | | The bottle.redirect method raises an HTTP Response. With the over-indentation it was being called inside a 'with' block, making the entire block fail.
* | Render static HTML page without linksbaserock/liw/lc-static-html-without-linksLars Wirzenius2014-05-091-1/+4
|/ | | | The links won't work, so there's no point in having them.
View Lorry Controller Status