summaryrefslogtreecommitdiff
path: root/lorry
Commit message (Collapse)AuthorAgeFilesLines
* Set version to 1.0.0Ben Brown2023-04-071-1/+1
|
* Remove persistent configuration downstreamChris Shepherd2022-02-181-21/+96
| | | | | | | | | | | Checks whether downstream files are now redundant according to current .lorry file, using `git worktree`. If so, their paths are deleted downstream. Also checks whether redundant files are not needed locally, ie the files haven't just been moved to a new path. If so, they're deleted. Resolves #30
* Make flake8 happyBen Brown2022-02-051-5/+5
|
* Commit changes from from black and isortBen Brown2022-02-051-424/+549
|
* Set 'destination' field as optionalBen Brown2022-02-051-1/+1
|
* (v2) Make raw importer honour multiple filesshprdchris2022-02-021-5/+5
| | | | | | | | | | | | | | | | | | | | | | | Resolves issue where raw importer was only honouring last entry for several files with same upstream path. These files were being passed as comma-separated URLs under the 'urls' tag in the .lorry file. The yaml importer could only store one entry per 'urls', updating a dictionary entry after each comma. The changes in this commit allow for multiple files to be added, but the URLs must now be added in the form: ``` urls: - destination: libosinfo url: https://releases.pagure.org/libosinfo/osinfo-db-20211216.tar.xz - destination: libosinfo url: https://releases.pagure.org/libosinfo/libosinfo-1.2.0.tar.gz ``` A previous branch was created to resolve this issue, but the solution required URLs to be separted by whitespace. It was decided that the format implemented here is preferable. Resolves: #29
* Create the gitdir regardless of no new archive/lfs contentBen Brown2022-01-151-2/+2
|
* Create tarball dest only when neededKyle Mckay2021-12-291-1/+2
|
* Use a default tarball output locationKyle Mckay2021-12-291-2/+2
| | | | Prevent them being created in root of current directory
* Create tarball destination when non-existentKyle Mckay2021-12-151-4/+9
| | | | | | Just a quality of life improvement Closes #23
* Set url-agent to Lorry if we get 403 HTTP statusEllis Barnwell2021-12-151-4/+19
| | | | | | If we get an HTTPError 403 then set the user-agent so it is clear that the request is coming from Lorry. This is used to get around situations where the url has blocked python user-agents.
* Use context manager for raw file requestsKyle Mckay2021-11-161-3/+2
| | | | | | | - Automaitcally clean up when done - Improve code readability - Use of parentheses around a with statement is a Python 3.10 change (though supported in 3.9)
* Use local install for git lfsKyle Mckay2021-11-151-1/+1
| | | | | Don't want to polute the environment and install globally
* Add support for multiple raw files per lorryKyle Mckay2021-11-041-29/+35
| | | | | | | | | | - This allows consumer to group raw files so that they end up in the same mirrored lfs repository - Useful for raw file mirroring scenarios where a common shared prefix is desired (e.g. for a BuildStream project) - Use "urls" key for consistency with other lorry types (all use "url")
* Remove use of raw-file monorepoKyle Mckay2021-11-041-16/+4
|
* Gitify raw files, storing them in an LFS repoJonathan Maw2021-11-041-4/+64
| | | | | Now, if a lorry has type 'raw-file', it'll be stored as a large file in a 'raw-file-mirrors' repo (or whatever else 'raw-file-repo' is set to)
* lorry: Install and use fudge_user_ids plugin for hg-fast-exportBen Hutchings2020-10-011-1/+23
| | | | | | | | | | | | | | | | | | | | | Mercurial allows arbitrary strings as user (committer) ids, while Git requires a name and email address, and specific punctuation around the address. hg-fast-export has some provision for automatically fixing-up invalid committer and author ids, but it doesn't catch everything. Its maintainer does not want to extend this, so we use a plugin instead. * Add a plugin (fudge_user_ids) that should fix up all invalid ids. * In setup.py: - Compile it at build time - Install it under a private data directory (/usr/share/lorry) - Clean up the bytecode * In gitify_hg, check whether hg-fast-export supports plugins, and where our plugins are. If this succeeds, add --plugin-path and --plugin options to enable fudge_user_ids. Closes #11.
* lorry: Use a working tree in gitify_cvsBen Hutchings2020-09-171-8/+55
| | | | | | | | | | | Sadly, git cvsimport requires a working tree for most operations, and only an initial import can be done successfully with the '-i' option. Rather than adding back general support for non-bare repositories: * When creating a mirror, let git cvsimport create it non-bare and then convert it to be bare * When updating a mirror, add a temporary working tree with "git worktree add"
* lorry: Make cvsps store cache inside working git repositoryBen Hutchings2020-09-171-0/+1
| | | | | | | | | git cvsimport uses cvsps, which creates state files under ~/.cvsps. This means the cache may become inconsistent with the git repository. I observed this to happen when adding a second test case for cvs conversion. Set $HOME to the git directory so that this doesn't happen.
* lorry: Avoid creating non-bare repository in gitify_svnBen Hutchings2020-09-151-18/+16
| | | | | | | | | | | | | | | | Our working repositories should always be initialised as bare, and this should be done in one place. gitify_svn currently uses "git svn init", which always creates a non-bare repository, and then modifies the repository to be bare (though still with core.logallrefupdates enabled). Thankfully "git svn init" does very little beyond "git init", so we can avoid this: 1. Call ensure_gitdir, like other gitify methods do 2. Create the directory svn/refs/remotes/git-svn 3. Set config parameters svn-remote.svn.url (which we already did) and svn-remote.svn.fetch
* lorry: Create bare repository in gitify_cvsBen Hutchings2020-09-151-2/+9
| | | | | | | | | | | | | | Since commit 50383f3bcce7 "lorry: Crash-proof updates to working git repos with A/B switching", prepare_working_repos assumes that the existing working repositories are bare. Most gitify functions ensure that, but gitify_cvs does not. * Specify the git directory to "git cvsimport" using $GIT_DIR, not the -C option (which specifies a working tree) * Add the -i option to inhibit it from checking out a working tree * Delete any index file that it creates Closes #12.
* lorry: Set migrated working repositories to be bareBen Hutchings2020-09-151-0/+7
| | | | | | | | | | Since commit 50383f3bcce7 "lorry: Crash-proof updates to working git repos with A/B switching", prepare_working_repos assumes that the existing working repositories are bare. migrate_oldstyle_repos did not ensure that, although it did discard any working tree. Run "git config" to set the core.bare and core.logallrefupdates flags appropriately.
* lorry: Use common method to create working git repositoriesBen Hutchings2020-09-151-14/+17
| | | | | | | | | | | | Add an ensure_gitdir method, that creates a bare git repository if the given directory doesn't yet exist. It also logs what happened and returns a flag for whether it created the directory. Use this to replace the open-coded versions of this in (most) gitify functions. Add a comment in prepare_working_repos explaining why we can't just call it there.
* Merge branch 'bwh/validate-server-certs' into 'master'Ben Brown2020-08-121-8/+25
|\ | | | | | | | | | | | | lorry: Enable TLS server certificate validation by default Closes #9 See merge request CodethinkLabs/lorry/lorry!15
| * lorry: Enable TLS server certificate validation by defaultBen Hutchings2020-08-121-8/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Lorry is not only used in Baserock, and it's reasonable to assume that there is normally a useful CA certificate store available. It's also no longer common for open source projects to avoid the "CA cartel" by using self-signed or CAcert certificates. * Enable validation by default for Bazaar, Git, and Mercurial * Add a configuration option to disable it * Add and document a .lorry keyword to disable it We already validate server certificates for file downloads since the Python standard library enabled it by default. We also never disabled validation for Subversion. Since this seems to have worked OK, don't add the option to disable it for these upstream types. Closes #9.
* | lorry: Crash-proof updates to working git repos with A/B switchingBen Hutchings2020-08-101-47/+98
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | lorry currently updates a git repository in its working area like this: 1. Delete old backup and debug repositories 2. Copy active repository (git) to backup (git-pre-update) 3. Perform conversion in active repository 4. If conversion failed, restore the backup: 1. Copy active repository to debug (git-post-fail) 2. Delete active repository 3. Copy backup repository to active 5. If conversion succeeded: delete git-pre-update If step 3 or 4 is interrupted, for example because of a system reboot, the active repository may be left missing or broken. On the next update, the backup will be deleted. Replace this with the following process: 1. Look for repositories 'git-a' and 'git-b'. The one with a higher update count (or the only one that exists and has an update count file) is the active repository. The other is the the temporary repository. In case neither exists, 'git-a' is the temporary repository. 2. Delete old temporary repository, if it exists 3. Copy active repository, if it exists, to temporary repository (excluding the update count file) 4. Perform conversion in temporary repository 5. If conversion succeeded, make the temporary repository active by writing the new update count file 6. If conversion failed, move the temporary repository to debug as before We need to migrate the working state, so if there is a 'git' and no 'git-a', add an update count of 1 to 'git' and move it to 'git-a'. Also delete the old-style backup repository. Update test cases to expect the new filenames. lorry should now be crash-proof, in the sense that killing it at any point will leave the last successful update in place. However, an unclean system shutdown (e.g. kernel crash) that does not flush all writes to persistent storage may still result in the active repository being left broken. Closes: #8.
* Merge branch 'bwh/mercurial-work-around-unnamed-heads' into 'master'Ben Hutchings2020-08-061-1/+56
|\ | | | | | | | | | | | | lorry: Prune unreachable commits from hg-fast-export marks file Closes #7 See merge request CodethinkLabs/lorry/lorry!12
| * lorry: Only log number of commits pruned from marks filesBen Hutchings2020-08-051-2/+6
| | | | | | | | | | | | | | | | | | | | Logging each pruned commit can result in very long logs for some repositories. This is a problem for Lorry Controller because job status updates result in copying the entire log and not just the new log lines. There's not much value in listing all the commit hashes, so log the number of commits instead. Update the test case accordingly.
| * lorry: Prune unreachable commits from hg-fast-export marks fileBen Hutchings2020-08-051-1/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | By default, hg-fast-export refuses to export a Mercurial repository if it has unnamed heads. lorry passes the --force option that overrides this since we still want to convert the named heads, but doing this sets us up for failure later. If the unnamed heads also have no tags pointing to them, the corresponding git commits may be deleted by 'git gc'. However they will still be listed in the 'marks' file used to record state for incremental conversions, so a later run of hg-fast-export may assume they exist and refer to them by hash. Before running hg-fast-export, delete any lines in its marks file that refer to unreachable or non-existent commits. Closes #7.
* | Add warnings about using BreezyBen Hutchings2020-08-051-0/+4
|/ | | | | | | | | | We should not recommend using Breezy until we can point to a version where fast-export is reliable. * README: describe support for Breezy as experiemntal * lorry: warn if attempting a Bazaar conversion using 'brz' Relates to lorry-buildstream#14.
* Restore copyright dates before 2013Ben Hutchings2020-07-151-1/+1
| | | | | | This mostly reverts commit 5f5435647ada "Update copyright notices", which changed all copyright dates to 2013. Most files were added in 2011 and 2012.
* lorry: Copy modification time from server to downloaded filesBen Hutchings2020-07-151-0/+10
| | | | | | | | | | | | lorry.gzip-importer uses the given file's modification time (mtime) as the commit time, but this is currently unrelated to the mtime on the server and so is unreproducible. Copy the Last-Modified field of the urllib response header, if present, to the mtime (and atime) of a file after downloading it. For FTP downloads, this response field is synthesised by our FTP handler. Closes #4.
* lorry: Get modification time for FTP downloadsBen Hutchings2020-07-151-1/+110
| | | | | | | | | | | | | lorry.gzip-importer uses the given file's modification time (mtime) as the commit time, but this is currently unrelated to the mtime on the server and so is unreproducible. For FTP downloads, urllib's default handler does not provide an mtime and there is no reasonable way to add that in a subclass. Define and use a new handler class that attempts to get the mtime from the server using the MDTM extension. Related to #4.
* Add support for Breezy (brz) as alternative Bazaar implementationBen Hutchings2020-07-091-4/+27
| | | | | | | | | | | | | | | | | | | | The original Bazaar implementation (bzr) only runs on Python 2, but there is a fork known as Breezy (brz) that has been ported to Python 3. To support environments without Python 2, allow using Breezy: * Add a setting 'bazaar-command' that specifies which command to run * Set the default based on a path search, preferring 'bzr' but using 'brz' if only 'brz' is found * Implement a similar fallback in the test setup for Bazaar import * Update README to mention Breezy as an alternative. The 'fastimport' extension is included in it rather than being a separate package. Closes #3.
* Merge branch 'bwh/push-options' into 'master'Ben Hutchings2020-06-011-2/+9
|\ | | | | | | | | | | | | lorry: Add support for push options Closes #2 See merge request CodethinkLabs/lorry!4
| * lorry: Add support for git push optionsBen Hutchings2020-05-291-2/+9
| | | | | | | | | | | | | | | | | | | | Gerrit requires passing the skip-validation option when pushing a branch with a long history. Add a 'push-option' application setting that allows adding any such option to the git push command. Closes #2.
* | lorry: Delete unused local variablesBen Hutchings2020-05-291-2/+2
|/ | | | Found by pyflakes.
* lorry: Decode sub-command stdout and stderr to textBen Hutchings2020-05-291-0/+4
| | | | | cliapp.Application.runcmd_unchecked returns bytes objects under Python 3. Decode these as UTF-8 in non-strict (replace) mode.
* lorry: Fix deprecated syntax in sed commandBen Hutchings2020-05-291-1/+1
| | | | | | | | | | | The sed command includes the sequences '\{' and '\}' which are not valid escape sequences and so expand to themselves (which is what is needed). Unrecognised escape sequences are deprecated since Python 3.6 and will become a syntax error in future. Use raw-string syntax to avoid the need for double-backslashes. Found with pycodestyle.
* Migrate to Python 3Pedro Alvarez2019-11-271-12/+12
|
* Create bare repo before pushing if local remotePedro Alvarez2019-07-091-0/+7
|
* Add support for YAML lorriesAdam Coldrick2016-10-061-2/+8
| | | | | | | If parsing the lorry as YAML fails, we attempt to parse it as JSON before giving up completely. Change-Id: I7e3502f5ae9c32e10d63047e5f280bfe1d7e52dc
* Set 'gc.autodetach=false' for `git fetch`Pedro Alvarez2016-02-231-2/+3
| | | | | | | | This prevent git from running processes in the background. We were having issues when `git gc` was running automatically, making `git gc` call fail because it was already running. Change-Id: Iee3e8e29eb0697f5241c7de4744c8e080bfd8e28
* lorry: Mirror without touching configRichard Maw2016-01-151-20/+3
| | | | | | | | We can do everything we need to on the command-line, which makes more sense than syncing lorry file config every time, and it makes it easier to add extra options later. Change-Id: Ieaabc9fe7a83b5c29371ddb92b991d128c90e6c6
* Pass in -r before --force in hg-fast-exportTristan Van Berkom2015-12-181-1/+1
| | | | | | | | | | | | | | | HG lorries are currently broken with current versions of hg-fast-export Filed upstream bug here because it technically should not be broken: https://github.com/frej/fast-export/issues/57 Note the issue in the upstream git repository here: http://repo.or.cz/fast-export.git/blob/refs/heads/master:/hg-fast-export.sh#l54 This patch works around the issue by simply reordering the arguments of hg-fast-export in the lorry program. Change-Id: I2c8b303e2d90edb301186c0b3a6c57ba2e0b8179
* Added gzip file support to lorryTristan Van Berkom2015-11-101-2/+3
| | | | | | | | The new 'gzip' lorry file is practical for any gzipped single file repositories, this can be useful for some static blob resources, such as fonts. Change-Id: I931e15e4c1291d820db82d0fc764aa767d3df9ab
* Make branches and tags optionalRichard Ipsum2015-09-241-7/+27
| | | | | | | | | | | | | | | | | | Some svn repos don't have a branches dir or a tags dir in their layout. This will for example allow us to lorry xmlto with the following lorry, { "xmlto": { "type": "svn", "url": "http://svn.fedorahosted.org/svn/xmlto", "layout": { "trunk": "." } } } Change-Id: I74c032ab7642f4b22ed0569b2a163c0b78cba658
* Add note about license of fast-import frontendsRichard Ipsum2015-07-091-1/+1
| | | | | | Also update copyright Change-Id: Iff42a32048482520cee695bc1262990f5989ce63
* Add zip support to lorryRichard Ipsum2015-07-071-10/+14
| | | | Change-Id: I29368ddcc64c82bafcbb834c81db30018a144f64
* Merge branch 'sam/no-default-mirror-url'Sam Thursfield2015-02-041-11/+13
|\ | | | | | | | | Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk> Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
View Lorry Controller Status