summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
...
| * Use branch ref when getting last commitKyle Mckay2021-11-151-3/+3
| | | | | | | | | | | | | | - Ensures raw file importer doesn't fail when `init.defaultBranch` is not set to `master` on the machine where lorry is running. - Avoids use of `--initial-branch` on repository init which is only supported in git versions >=2.28.0
| * Improve retrieval of most recent commitKyle Mckay2021-11-151-2/+5
| | | | | | | | | | - Use of rev-parse more correct since `git log` is porcelain
| * Add support for multiple raw files per lorryKyle Mckay2021-11-041-29/+35
| | | | | | | | | | | | | | | | | | | | - This allows consumer to group raw files so that they end up in the same mirrored lfs repository - Useful for raw file mirroring scenarios where a common shared prefix is desired (e.g. for a BuildStream project) - Use "urls" key for consistency with other lorry types (all use "url")
| * Remove use of raw-file monorepoKyle Mckay2021-11-041-16/+4
| |
| * Gitify raw files, storing them in an LFS repoJonathan Maw2021-11-043-4/+195
|/ | | | | Now, if a lorry has type 'raw-file', it'll be stored as a large file in a 'raw-file-mirrors' repo (or whatever else 'raw-file-repo' is set to)
* Merge branch 'bwh/fix-author-ids' into 'master'Ben Hutchings2020-10-033-3/+91
|\ | | | | | | | | | | | | lorry: Install and use fudge_user_ids plugin for hg-fast-export Closes #11 See merge request CodethinkLabs/lorry/lorry!20
| * lorry: Install and use fudge_user_ids plugin for hg-fast-exportBen Hutchings2020-10-013-3/+91
|/ | | | | | | | | | | | | | | | | | | | | Mercurial allows arbitrary strings as user (committer) ids, while Git requires a name and email address, and specific punctuation around the address. hg-fast-export has some provision for automatically fixing-up invalid committer and author ids, but it doesn't catch everything. Its maintainer does not want to extend this, so we use a plugin instead. * Add a plugin (fudge_user_ids) that should fix up all invalid ids. * In setup.py: - Compile it at build time - Install it under a private data directory (/usr/share/lorry) - Clean up the bytecode * In gitify_hg, check whether hg-fast-export supports plugins, and where our plugins are. If this succeeds, add --plugin-path and --plugin options to enable fudge_user_ids. Closes #11.
* Merge branch 'bwh/cvs-use-worktree' into 'master'Ben Brown2020-09-1724-187/+271
|\ | | | | | | | | | | | | Use a working tree in gitify_cvs Closes #12 See merge request CodethinkLabs/lorry/lorry!19
| * lorry: Use a working tree in gitify_cvsBen Hutchings2020-09-171-8/+55
| | | | | | | | | | | | | | | | | | | | | | Sadly, git cvsimport requires a working tree for most operations, and only an initial import can be done successfully with the '-i' option. Rather than adding back general support for non-bare repositories: * When creating a mirror, let git cvsimport create it non-bare and then convert it to be bare * When updating a mirror, add a temporary working tree with "git worktree add"
| * tests: Add test of incremental cvs lorryingBen Hutchings2020-09-173-0/+195
| | | | | | | | | | This currently fails because git cvsimport doesn't really work with bare repositories.
| * lorry: Make cvsps store cache inside working git repositoryBen Hutchings2020-09-173-0/+3
| | | | | | | | | | | | | | | | | | git cvsimport uses cvsps, which creates state files under ~/.cvsps. This means the cache may become inconsistent with the git repository. I observed this to happen when adding a second test case for cvs conversion. Set $HOME to the git directory so that this doesn't happen.
| * tests: Do not hide lorry errorsBen Hutchings2020-09-1710-12/+12
| | | | | | | | | | Redirecting lorry's stderr to /dev/null just makes test failures harder to debug. Only do that when we expect it to fail.
| * tests: Clear out $DATADIR after every testBen Hutchings2020-09-169-168/+7
|/ | | | | | | | | | Some tests have a teardown script that removes their temporary files, but most do not. To avoid interference or dependencies between tests, use a global teardown script that removes almost everything in $DATADIR. It excludes git-upstream, which is intentionally set up once and read by multiple tests.
* Merge branch 'bwh/bare-working-repos' into 'master'Ben Brown2020-09-165-35/+123
|\ | | | | | | | | | | | | Ensure all working repositories are bare Closes #12 See merge request CodethinkLabs/lorry/lorry!18
| * lorry: Avoid creating non-bare repository in gitify_svnBen Hutchings2020-09-151-18/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Our working repositories should always be initialised as bare, and this should be done in one place. gitify_svn currently uses "git svn init", which always creates a non-bare repository, and then modifies the repository to be bare (though still with core.logallrefupdates enabled). Thankfully "git svn init" does very little beyond "git init", so we can avoid this: 1. Call ensure_gitdir, like other gitify methods do 2. Create the directory svn/refs/remotes/git-svn 3. Set config parameters svn-remote.svn.url (which we already did) and svn-remote.svn.fetch
| * lorry: Create bare repository in gitify_cvsBen Hutchings2020-09-151-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Since commit 50383f3bcce7 "lorry: Crash-proof updates to working git repos with A/B switching", prepare_working_repos assumes that the existing working repositories are bare. Most gitify functions ensure that, but gitify_cvs does not. * Specify the git directory to "git cvsimport" using $GIT_DIR, not the -C option (which specifies a working tree) * Add the -i option to inhibit it from checking out a working tree * Delete any index file that it creates Closes #12.
| * lorry: Set migrated working repositories to be bareBen Hutchings2020-09-151-0/+7
| | | | | | | | | | | | | | | | | | | | Since commit 50383f3bcce7 "lorry: Crash-proof updates to working git repos with A/B switching", prepare_working_repos assumes that the existing working repositories are bare. migrate_oldstyle_repos did not ensure that, although it did discard any working tree. Run "git config" to set the core.bare and core.logallrefupdates flags appropriately.
| * lorry: Use common method to create working git repositoriesBen Hutchings2020-09-151-14/+17
| | | | | | | | | | | | | | | | | | | | | | | | Add an ensure_gitdir method, that creates a bare git repository if the given directory doesn't yet exist. It also logs what happened and returns a flag for whether it created the directory. Use this to replace the open-coded versions of this in (most) gitify functions. Add a comment in prepare_working_repos explaining why we can't just call it there.
| * tests: Verify that working repos for cvs and svn mirroring are bareBen Hutchings2020-09-154-2/+75
|/ | | | | | | | | | These will currently fail: * gitify_cvs creates non-bare repositories * gitify_svn creates non-bare repositories and changes them to bare, but leaves ref-logs enabled Related to #12.
* Merge branch 'bwh/include-gpl' into 'master'Ben Brown2020-08-131-3/+341
|\ | | | | | | | | LICENSE: Include full text of GPLv2 See merge request CodethinkLabs/lorry/lorry!17
| * LICENSE: Include full text of GPLv2Ben Hutchings2020-08-131-3/+341
|/ | | | | Since Lorry is now separate from Baserock we shouldn't assume that it's accompanied by any license texts.
* Merge branch 'bwh/validate-server-certs' into 'master'Ben Brown2020-08-122-8/+31
|\ | | | | | | | | | | | | lorry: Enable TLS server certificate validation by default Closes #9 See merge request CodethinkLabs/lorry/lorry!15
| * lorry: Enable TLS server certificate validation by defaultBen Hutchings2020-08-122-8/+31
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Lorry is not only used in Baserock, and it's reasonable to assume that there is normally a useful CA certificate store available. It's also no longer common for open source projects to avoid the "CA cartel" by using self-signed or CAcert certificates. * Enable validation by default for Bazaar, Git, and Mercurial * Add a configuration option to disable it * Add and document a .lorry keyword to disable it We already validate server certificates for file downloads since the Python standard library enabled it by default. We also never disabled validation for Subversion. Since this seems to have worked OK, don't add the option to disable it for these upstream types. Closes #9.
* | Merge branch 'bwh/pyflakes' into 'master'Ben Brown2020-08-122-1/+17
|\ \ | | | | | | | | | | | | Fix pyflakes warning; add it to CI configuration See merge request CodethinkLabs/lorry/lorry!16
| * | gitlab-ci: Run pyflakes3 over all Python sourcesBen Hutchings2020-08-101-0/+17
| | |
| * | lorry.zip-importer: Delete unused import of time moduleBen Hutchings2020-08-101-1/+0
| |/ | | | | | | | | The time module is unused since "lorry.zip-importer: Interpret zip file timestamps as UTC".
* | Merge branch 'bwh/crash-proofing' into 'master'Ben Brown2020-08-1223-139/+478
|\ \ | |/ |/| | | | | | | | | lorry: Crash-proof updates to working git repos with A/B switching Closes #8 See merge request CodethinkLabs/lorry/lorry!14
| * tests: Check update counts in git-backup-on-error test casesBen Hutchings2020-08-104-0/+4
| |
| * tests: Add test case for migration of old-style working repositoriesBen Hutchings2020-08-104-0/+164
| | | | | | | | | | Make sure migration of existing working directories to the A/B scheme works properly.
| * tests: Add test case for update counts in the A/B schemeBen Hutchings2020-08-104-0/+158
| |
| * lorry: Crash-proof updates to working git repos with A/B switchingBen Hutchings2020-08-1013-139/+152
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | lorry currently updates a git repository in its working area like this: 1. Delete old backup and debug repositories 2. Copy active repository (git) to backup (git-pre-update) 3. Perform conversion in active repository 4. If conversion failed, restore the backup: 1. Copy active repository to debug (git-post-fail) 2. Delete active repository 3. Copy backup repository to active 5. If conversion succeeded: delete git-pre-update If step 3 or 4 is interrupted, for example because of a system reboot, the active repository may be left missing or broken. On the next update, the backup will be deleted. Replace this with the following process: 1. Look for repositories 'git-a' and 'git-b'. The one with a higher update count (or the only one that exists and has an update count file) is the active repository. The other is the the temporary repository. In case neither exists, 'git-a' is the temporary repository. 2. Delete old temporary repository, if it exists 3. Copy active repository, if it exists, to temporary repository (excluding the update count file) 4. Perform conversion in temporary repository 5. If conversion succeeded, make the temporary repository active by writing the new update count file 6. If conversion failed, move the temporary repository to debug as before We need to migrate the working state, so if there is a 'git' and no 'git-a', add an update count of 1 to 'git' and move it to 'git-a'. Also delete the old-style backup repository. Update test cases to expect the new filenames. lorry should now be crash-proof, in the sense that killing it at any point will leave the last successful update in place. However, an unclean system shutdown (e.g. kernel crash) that does not flush all writes to persistent storage may still result in the active repository being left broken. Closes: #8.
* Merge branch 'bwh/mercurial-work-around-unnamed-heads' into 'master'Ben Hutchings2020-08-0610-1/+341
|\ | | | | | | | | | | | | lorry: Prune unreachable commits from hg-fast-export marks file Closes #7 See merge request CodethinkLabs/lorry/lorry!12
| * lorry: Only log number of commits pruned from marks filesBen Hutchings2020-08-053-4/+9
| | | | | | | | | | | | | | | | | | | | Logging each pruned commit can result in very long logs for some repositories. This is a problem for Lorry Controller because job status updates result in copying the entire log and not just the new log lines. There's not much value in listing all the commit hashes, so log the number of commits instead. Update the test case accordingly.
| * tests: Add a test case for unnamed heads in MercurialBen Hutchings2020-08-054-0/+137
| | | | | | | | | | | | Test that we can cope with conversion of unnamed heads now. Related to #7.
| * tests: Add a test case for incremental conversion from MercurialBen Hutchings2020-08-055-0/+147
| | | | | | | | | | Also add a teardown script for hg-single-commit so these two don't get in each other's way.
| * lorry: Prune unreachable commits from hg-fast-export marks fileBen Hutchings2020-08-051-1/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | By default, hg-fast-export refuses to export a Mercurial repository if it has unnamed heads. lorry passes the --force option that overrides this since we still want to convert the named heads, but doing this sets us up for failure later. If the unnamed heads also have no tags pointing to them, the corresponding git commits may be deleted by 'git gc'. However they will still be listed in the 'marks' file used to record state for incremental conversions, so a later run of hg-fast-export may assume they exist and refer to them by hash. Before running hg-fast-export, delete any lines in its marks file that refer to unreachable or non-existent commits. Closes #7.
* | Merge branch 'bwh/breezy-warning' into 'master'Ben Brown2020-08-062-3/+9
|\ \ | |/ |/| | | | | Add warnings about using Breezy See merge request CodethinkLabs/lorry/lorry!13
| * Add warnings about using BreezyBen Hutchings2020-08-052-3/+9
|/ | | | | | | | | | We should not recommend using Breezy until we can point to a version where fast-export is reliable. * README: describe support for Breezy as experiemntal * lorry: warn if attempting a Bazaar conversion using 'brz' Relates to lorry-buildstream#14.
* Merge branch 'bwh/fix-cvs-test-failure' into 'master'Ben Hutchings2020-07-221-0/+4
|\ | | | | | | | | | | | | tests: Fix flaky cvs-single-commit test Closes #6 See merge request CodethinkLabs/lorry/lorry!11
| * tests: Fix flaky cvs-single-commit testBen Hutchings2020-07-221-0/+4
|/ | | | | | | | | | | | | | | | | The cvs-single-commit test actually involves two commits: one made by 'cvs init', and another by 'cvs commit'. Occasionally these commit appear in the wrong order in the git repository, apparently because they ended up with the same timestamp. This seems to be a similar issue to <https://bugs.launchpad.net/ubuntu/+source/cvs/+bug/12230>, which has a workaround (a 20 ms sleep after checkout). However, this issue is not readily reproducible - I had to use libfaketime to do it - and I can't tell whether that will work for us. Instead, add a 1 second sleep after checkout. Closes #6.
* Merge branch 'bwh/fix-copyright-notices' into 'master'Ben Brown2020-07-1536-47/+44
|\ | | | | | | | | Fix copyright notices See merge request CodethinkLabs/lorry/lorry!10
| * Add copyright notices and years to several scriptsBen Hutchings2020-07-156-5/+7
| | | | | | | | | | * lorry.gzip-importer was missing a copyright notice * Various changes were not included in the copyright dates
| * Restore copyright dates before 2013Ben Hutchings2020-07-1533-33/+33
| | | | | | | | | | | | This mostly reverts commit 5f5435647ada "Update copyright notices", which changed all copyright dates to 2013. Most files were added in 2011 and 2012.
| * Move references to original git scripts from LICENSE to derived scriptsBen Hutchings2020-07-153-14/+2
| | | | | | | | | | This seems like a more obvious to look for such information, and it's also where the GPL says we should mention changes.
| * Add proper copyright notices for lorry.{tar,zip}-importerBen Hutchings2020-07-153-0/+7
| | | | | | | | | | | | These are based on scripts in the git project, which we noted but didn't include a copyright statement for. Put copyright notices (including for our changes) in those files and in LICENSE.
* | Merge branch 'bwh/doc-names' into 'master'Ben Brown2020-07-152-25/+26
|\ \ | |/ | | | | | | Use document names expected by GitLab See merge request CodethinkLabs/lorry/lorry!9
| * README: Move copyright and license statements to a LICENSE fileBen Hutchings2020-07-152-25/+26
| | | | | | | | This is what git hosting services like GitLab expect.
| * README: Rename to README.mdBen Hutchings2020-07-151-0/+0
|/ | | | | This will cause it to be rendered on GitLab and other git hosts' web interfaces.
* Merge branch 'bwh/reproducibility' into 'master'Ben Hutchings2020-07-154-10/+165
|\ | | | | | | | | | | | | Fix reproducibility of file imports Closes #1 and #4 See merge request CodethinkLabs/lorry/lorry!8
| * lorry.tar-importer: Delete unused variableBen Hutchings2020-07-151-1/+0
| | | | | | | | | | | | The $commit_time variable is unused since commit cf4a0dff36819 "Make tarball lorries reproducible", but it confused me into thinking the issue was still unfixed. Delete it to avoid confusing anyone else.
View Lorry Controller Status