summaryrefslogtreecommitdiff
path: root/lorrycontroller
Commit message (Collapse)AuthorAgeFilesLines
* Fix job queue getting stuck on repo preparationKyle Mckay2022-01-121-3/+21
| | | | | | | | | | | | | | | | | | | | | - When a downstream type failed to prepare a repository the loop over lorries would abort unexpectly meaning any following lorries were never added into the minion's queue. Such cases are now simply skipped over so that lorry controller does not get stuck. - Catching Exception doesn't feel nice or correct, but we don't want any unexpected exceptions to cause the same problem of getting stuck so I don't see any obvious alternative in the short term. Perhaps the overall code can be reworked to avoid needing to do this. Log exceptions when skipping lorries When a repository fails to prepare, ensure there is a log of the exception that occured and the associated lorry. Update skipped lorry last run and exit data - Update last run to prevent frequent repeated failure - Update exit for user visiblity on status page
* Enable LFS in gitlab for raw filesEllis Barnwell2021-11-251-1/+1
|
* Update admin interface to support raw-fileBen Brown2021-11-181-1/+4
|
* Bump copyright yearBen Brown2021-10-131-1/+1
|
* Remove unclear commentBen Brown2021-10-131-3/+0
|
* Fetch without the need to touch remotesBen Brown2021-10-131-13/+8
|
* Drop duplicate hard resetBen Brown2021-10-131-5/+0
|
* readconf: unify clone and update into single fetch methodBen Brown2021-10-131-12/+6
|
* Ensure that origin url is set to what's in settingsEllis Barnwell2021-10-131-0/+5
|
* Reset to the branch supplied in the configEllis Barnwell2021-10-131-1/+2
|
* Add a page listing failing lorriesBen Hutchings2020-10-062-6/+25
| | | | | | | | | | | | | Provide a status page that lists only the lorries that failed to update on the last attempt. This will make it easier to spot problems that require an update to the lorry configuration. Reuse some of the existing status page logic and render the run queue with a different template. Add a link to this page from the status page. Closes #20.
* lorrycontroller: Explicitly list all names to be re-exportedBen Hutchings2020-08-101-1/+32
| | | | | | | | | | | | pyflakes isn't able to recognise that: __all__ = locals() effectively uses all the imported names. Replace this with an explicit list of names. Running pyflakes in CI should ensure that the list doesn't get out of sync.
* Delete unused module importsBen Hutchings2020-08-106-14/+0
| | | | | | Various modules are imported and not used. Caught by pyflakes.
* Delete assignments to local variables that are not used laterBen Hutchings2020-08-102-7/+2
| | | | | | | pyflakes found various assignments to local variables which are not used again. In some cases we still need to evaluate the expression that's assigned. In most places we can delete the assignment entirely.
* showjob: Use local variable instead of re-evaluating the same expressionBen Hutchings2020-08-101-1/+1
| | | | | | | | JobShower.get_job_as_json assigns to a local 'path' variable, but doesn't use it. Use it instead of re-evaluating the same expression assigned to it. Caught by pyflakes.
* readconf: Fix handling of invalid time values in configurationBen Hutchings2020-08-101-1/+1
| | | | | | | | | ReadConfiguration.fix_up_interval was returning default_value when the string value did not match the expected format. This variable is not defined, but default_interval is. This bug has been present since the beginning of LC development, and was caught by pyflakes.
* givemejob: Fix upstream host metadata lookupBen Hutchings2020-08-101-2/+2
| | | | | | | | Commit bc7f80d "givemejob: Move upstream host metadata lookup out of get_repo_metadata" failed to pass through the statedb variable into the new function, so it will never work. Caught by pyflakes.
* Rewind lorry FDs before parsing as JSON.Robert J. Heywood2020-08-031-0/+1
| | | | | | | | | First, the lorry files are parsed as YAML. If the parser fails, then the lorry files are parsed as JSON. If the YAML parser has advanced the Lorry file's file-descriptor, this can lead to JSON parser errors. This change resets the file descriptor to the start of the file before calling the JSON parser.
* Merge branch 'bwh/single-lorry-metadata' into 'master'Ben Brown2020-08-032-24/+121
|\ | | | | | | | | | | | | Add metadata for single repositories Closes #15 See merge request CodethinkLabs/lorry/lorry-controller!17
| * gitlab: Ignore failure to set the default branchBen Hutchings2020-07-301-3/+12
| | | | | | | | | | | | | | | | | | | | In testing the change to define the default branch for single repositories, I found that GitLab will reject a change of default branch if the branch does not yet exist. (It doesn't seem to do this when creating a repository.) Ignore failure to change the default branch, as we will fix it on the next run after the branch has been created.
| * givemejob: Add metadata for single repositoriesBen Hutchings2020-07-301-2/+89
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When mirroring a repository found through a 'lorries' configuration section (instead of 'trove' or 'gitlab'), we currently don't set a description or default branch. * Set the description to the upstream repository path, but allow this to be overridden by a description field in the .lorry file. Prepend the host-name, just as we do when mirroring an Upstream Host. * Set the default branch to: - Bazaar: 'trunk' - Git: upstream default branch, found using 'git ls-remote' - others: 'master' Closes #15.
| * givemejob: Move upstream host metadata lookup out of get_repo_metadataBen Hutchings2020-07-151-10/+18
| | | | | | | | | | | | In preparation for adding metadata for single repositories, move the code for looking up metadata through the Upstream Host connector into a separate function.
| * givemejob: Stop synthesising host information when it's missingBen Hutchings2020-07-151-10/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | Any lorry in the database should have been either: 1. Generated based on a 'trove' or 'gitlab' section, with 'from_host' set to the Upstream Host name. 2. Read from a lorry referred to in a 'lorries' section, with 'from_host' set to an empty string. So this error case indicates a dangling reference in the database. The current schema does not use foreign keys so this could happen, but it's reasonable to just give up and return empty metadata.
* | local: Create repos with empty description and enabled for exportBen Hutchings2020-07-291-1/+6
| | | | | | | | | | | | | | | | | | | | | | | | 'git init' copies files into the repository from a template directory, which contains a 'replace me' description that we don't want. Provide and use our own template directory instead. While we're at it, also add the git-daemon-export-ok flag file that git-daemon and git-http-backend may check for (depending on their configuration). Closes #16.
* | status: Use integer division where appropriateBen Hutchings2020-07-161-4/+4
|/ | | | | | | | | | | | | | | In Python 2, the / operator between two integer-typed values produces another integer. In Python 3, it produces a float. We need to use the // operator to get the old behaviour. The change to floats in get_free_disk_space resulted in unnecessarily precise values for free disk space. The change in formats_secs_nicely doesn't seem to make a difference since we format the quotients with '%d', but I'm making it use // so it's clearer what we want. Closes #14.
* Merge branch 'bwh/statedb-no-json1' into 'master'Ben Brown2020-06-141-1/+1
|\ | | | | | | | | STATEDB: Avoid depending on SQLite json1 extension See merge request CodethinkLabs/lorry-controller!13
| * STATEDB: Avoid depending on SQLite json1 extensionBen Hutchings2020-06-081-1/+1
| | | | | | | | | | | | | | | | | | | | The migration to a generalised hosts table uses json_object(), which is part of the optional json1 extension to SQLite. This is not enabled by default, and isn't enabled in the Freedesktop-SDK build of SQLite. Since GitLab tokens shouldn't include double-quotes, we can avoid this by using standard SQL concatenation instead.
* | Update copyright dates for files I recently changedBen Hutchings2020-06-089-9/+9
|/
* Merge branch 'master' into 'bwh/gitea-support'Ben Hutchings2020-06-081-12/+28
|\ | | | | | | # Conflicts: # lorry-controller-webapp
| * Merge branch 'master' into 'bwh/gitlab-config'Ben Hutchings2020-06-083-34/+84
| |\ | | | | | | | | | Resolved conflicts (neighbouring insertions) in lorry-controller-webapp and lorrycontroller/gerrit.py.
| | * GerritDownstream: Pass default branch and description to create-projectBen Hutchings2020-06-011-14/+18
| | | | | | | | | | | | | | | | | | | | | Lorry's account on Gerrit might not be permitted to change these attributes later, so include them in the create-project command. Closes #7.
| | * GerritDownstream: Add application setting for parent projectBen Hutchings2020-06-011-1/+13
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | GerritDownstream does not specify a parent project for create-project, so it implicitly creates projects directly under 'All-Projects'. This requires that Lorry's user account is given broad permissions on All-Projects, which seems undesirable. Allow specifying an alternate parent project, so that Lorry can be given permissions on only that project.
* | | Add gitea Downstream Host connectorBen Hutchings2020-06-012-0/+140
| | | | | | | | | | | | | | | | | | | | | Implement organisation and repository creation using the Gitea REST API v1. Add a 'gitea-access-token' application setting for this. Closes #9.
* | | lsupstreams: Prefer more specific remote prefixes in prefixmapBen Hutchings2020-06-011-3/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | In order to support Downstream Hosts with restricted repository hierarchies (like Gitea's 2 levels), it is useful to be able to map both a directory and a subdirectory of it from an Upstream Host that has a deeper hierarchy. Convert the prefixmap into a list of (remote, local) tuples and sort in reverse order, so that a subdirectory prefix will appear before its parent directory prefix if both are present.
* | | Merge branch 'bwh/remote-downstreams' into bwh/gitea-supportBen Hutchings2020-06-013-28/+62
|\ \ \ | |/ / |/| / | |/
| * GitlabUpstream: Use HTTPS unless explicitly configured not toBen Hutchings2020-06-011-1/+4
| | | | | | | | | | | | Currently we always use the 'http' scheme to talk to GitLab's REST API. Use the 'https' scheme instead, if the configured protocol is 'https' or 'ssh' (as the SSH server doesn't expose this API).
| * Introduce and use downstream-{http,ssh}-url settingsBen Hutchings2020-06-012-11/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Replace the current hardcoded Downstream Host parameters with application settings. In general, Downstream Host connectors will use some combination of HTTP(S) and/or SSH, so these are added in lorry-controller-webapp. Currently only the gerrit connector will use downstream-ssh-url and only the gitlab connector will use downstream-http-url. Turn *on* StrictHostKeyChecking for the gerrit connector when not talking to localhost. Closes #4.
| * Introduce and use SshCommand class for running commands via sshBen Hutchings2020-06-012-21/+47
| | | | | | | | | | | | | | | | | | | | | | | | Most of the ssh parameters are given by a URL, which should be configurable. Additional options can be set by keyword arguments. runcmd already has a function like this, but it doesn't take a URL and runcmd.shell_quote doesn't handle empty strings correctly. Use this in the Gerrit host connector. It could be used in the Gitano host connector as well, but I want to avoid making further changes there that I can't test.
| * Merge branch 'bwh/gerrit-ssh-quoting' into bwh/remote-downstreamsBen Hutchings2020-06-011-1/+2
| |\
| | * gerrit: Shell-quote ssh command argumentsBen Hutchings2020-06-011-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The Secure Shell protocol does not allow passing an argument vector, only a command line that will be interpreted by a specific program (usually a shell, but in this case Gerrit) on the remote host. OpenSSH's ssh command simply concatenates the remote command arguments with spaces between them, so we need to shell-quote them first. Closes #8.
* | | GitlabDownstream: Create repositories with most optional features disabledBen Hutchings2020-06-011-8/+20
| | | | | | | | | | | | | | | | | | | | | | | | It generally doesn't make sense for a pure mirror to have issues, merge requests, CI, a wiki, snippets, GitLab Pages, container images, or LFS. Disable all of these. Closes #6.
* | | Allow creating repos/groups as internal or public on GitLabBen Hutchings2020-06-013-3/+20
|/ / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | GitLab groups can have a visibility of 'private' (only members), 'internal'/'limited' (only authenticated users), or 'public' (anyone). GitLab subgroups and projects can have the same visibilities as groups, but cannot be more visible than the containing group. When mirroring public source code, it may be desirable to make the entities we create more visible than the current default of 'private'. Add an option to set any of the three levels of visibility supported by GitLab, with the default unchanged. (The same levels are supported by Gitea, though with one named differently.) Never change the visibility of a group or repository that already exists. Note that the GitLab connector previously tried to create all projects as public, but this stopped working after the change to API v4. In Gerrit and Gitano the access control model is lot more complex, and generally only administrators can set visibility. Make their connectors reject visibility other than 'private'. Relates to #6, #9.
* | Implement getting and setting repo metadata for all host typesBen Hutchings2020-06-013-17/+45
| | | | | | | | | | * GitlabUpstream: Implement get_repo_metadata * {Gerrit,Gitlab,Local}Downstream: Use given metadata in prepare_repo
* | local: Create the repository in prepare_repoBen Hutchings2020-06-012-7/+29
| | | | | | | | | | | | | | | | | | We won't be able to set metadata on a repository before we've created it. So do that in prepare_repo instead of deferring it to Lorry. This requires adding an application setting for the base directory. Also update the method doc-string so the old lazy behaviour is not allowed.
* | Move Upstream Host type-specific code into modulesBen Hutchings2020-06-017-102/+184
| | | | | | | | | | | | | | | | | | * Introduce UpnstreamHost abstract base class * Define a subclass of this for each Upstream Host type * Define a name-type map for these in the lorrycontroller package, * Use these classes to replace type-specific code elsewhere Related to #5.
* | Move Downstream Host type-specific code into modulesBen Hutchings2020-06-017-103/+230
| | | | | | | | | | | | | | | | | | * Introduce hosts module and DownstreamHost abstract base class * Define a subclass of this for each Downstream Host type * Define a name-type map for these in the lorrycontroller package * Use these classes to replace type-specific code elsewhere Related to #5.
* | lorrycontroller.readconf: Make general validator methods publicBen Hutchings2020-06-011-11/+11
| | | | | | | | | | | | | | | | | | | | I will move validation of the GitLab-specific field out to the gitlab module. In future we may also add further fields for specific Upstream Host types, that will also need to be validated in the associated modules. In preparation for that, drop the leading '_' from the more general methods in LorryControllerConfValidator, to mark them as public.
* | Merge branch 'bwh/update-gitlab' into bwh/cleanup-host-typesBen Hutchings2020-06-011-56/+29
|\ \
| * | lorrycontroller.gitlab: Use and create subgroups as neededBen Hutchings2020-06-011-56/+29
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The current code only uses top-level groups since GitLab did not originally support subgroups. For repositories with more than two path components, it joins multiple components together as the project name. However, subgroups were implemented in version 9.0, so we can use them and remove this mangling.
| * | lorrycontroller.gitlab: Update for GitLab v4 APIBen Hutchings2020-06-011-2/+2
| |/ | | | | | | | | python-gitlab dropped support for projects.search(...) along with the GitLab v3 API. Instead we have to use projects.list(search=...).
View Lorry Controller Status