| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Checks whether downstream files are now redundant according to current
.lorry file, using `git worktree`. If so, their paths are deleted
downstream.
Also checks whether redundant files are not needed locally, ie the files
haven't just been moved to a new path. If so, they're deleted.
Resolves #30
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Resolves issue where raw importer was only honouring last entry for
several files with same upstream path.
These files were being passed as comma-separated URLs under the 'urls'
tag in the .lorry file. The yaml importer could only store one entry per
'urls', updating a dictionary entry after each comma.
The changes in this commit allow for multiple files to be added, but the
URLs must now be added in the form:
```
urls:
- destination: libosinfo
url: https://releases.pagure.org/libosinfo/osinfo-db-20211216.tar.xz
- destination: libosinfo
url: https://releases.pagure.org/libosinfo/libosinfo-1.2.0.tar.gz
```
A previous branch was created to resolve this issue, but the solution required URLs to be separted by whitespace. It was decided that the format implemented here is preferable.
Resolves: #29
|
| |
|
| |
|
|
|
|
| |
Prevent them being created in root of current directory
|
|
|
|
|
|
| |
Just a quality of life improvement
Closes #23
|
|
|
|
|
|
| |
If we get an HTTPError 403 then set the user-agent so it is clear that
the request is coming from Lorry. This is used to get around situations where
the url has blocked python user-agents.
|
|
|
|
|
|
|
| |
- Automaitcally clean up when done
- Improve code readability
- Use of parentheses around a with statement is a Python 3.10 change
(though supported in 3.9)
|
|
|
|
|
| |
Don't want to polute the environment and install
globally
|
|
|
|
|
|
|
|
|
|
| |
- This allows consumer to group raw files so that
they end up in the same mirrored lfs repository
- Useful for raw file mirroring scenarios where a
common shared prefix is desired (e.g. for a
BuildStream project)
- Use "urls" key for consistency with other lorry
types (all use "url")
|
| |
|
|
|
|
|
| |
Now, if a lorry has type 'raw-file', it'll be stored as a large file in
a 'raw-file-mirrors' repo (or whatever else 'raw-file-repo' is set to)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Mercurial allows arbitrary strings as user (committer) ids, while
Git requires a name and email address, and specific punctuation
around the address.
hg-fast-export has some provision for automatically fixing-up invalid
committer and author ids, but it doesn't catch everything. Its
maintainer does not want to extend this, so we use a plugin instead.
* Add a plugin (fudge_user_ids) that should fix up all invalid ids.
* In setup.py:
- Compile it at build time
- Install it under a private data directory (/usr/share/lorry)
- Clean up the bytecode
* In gitify_hg, check whether hg-fast-export supports plugins, and
where our plugins are. If this succeeds, add --plugin-path and
--plugin options to enable fudge_user_ids.
Closes #11.
|
|
|
|
|
|
|
|
|
|
|
| |
Sadly, git cvsimport requires a working tree for most operations, and
only an initial import can be done successfully with the '-i' option.
Rather than adding back general support for non-bare repositories:
* When creating a mirror, let git cvsimport create it non-bare
and then convert it to be bare
* When updating a mirror, add a temporary working tree with
"git worktree add"
|
|
|
|
|
|
|
|
|
| |
git cvsimport uses cvsps, which creates state files under ~/.cvsps.
This means the cache may become inconsistent with the git repository.
I observed this to happen when adding a second test case for cvs
conversion.
Set $HOME to the git directory so that this doesn't happen.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Our working repositories should always be initialised as bare, and
this should be done in one place. gitify_svn currently uses "git svn
init", which always creates a non-bare repository, and then modifies
the repository to be bare (though still with core.logallrefupdates
enabled).
Thankfully "git svn init" does very little beyond "git init", so
we can avoid this:
1. Call ensure_gitdir, like other gitify methods do
2. Create the directory svn/refs/remotes/git-svn
3. Set config parameters svn-remote.svn.url (which we already did) and
svn-remote.svn.fetch
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since commit 50383f3bcce7 "lorry: Crash-proof updates to working git
repos with A/B switching", prepare_working_repos assumes that the
existing working repositories are bare. Most gitify functions ensure
that, but gitify_cvs does not.
* Specify the git directory to "git cvsimport" using $GIT_DIR, not
the -C option (which specifies a working tree)
* Add the -i option to inhibit it from checking out a working tree
* Delete any index file that it creates
Closes #12.
|
|
|
|
|
|
|
|
|
|
| |
Since commit 50383f3bcce7 "lorry: Crash-proof updates to working git
repos with A/B switching", prepare_working_repos assumes that the
existing working repositories are bare. migrate_oldstyle_repos did
not ensure that, although it did discard any working tree.
Run "git config" to set the core.bare and core.logallrefupdates flags
appropriately.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add an ensure_gitdir method, that creates a bare git repository if the
given directory doesn't yet exist. It also logs what happened and
returns a flag for whether it created the directory.
Use this to replace the open-coded versions of this in (most) gitify
functions.
Add a comment in prepare_working_repos explaining why we can't just
call it there.
|
|\
| |
| |
| |
| |
| |
| | |
lorry: Enable TLS server certificate validation by default
Closes #9
See merge request CodethinkLabs/lorry/lorry!15
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Lorry is not only used in Baserock, and it's reasonable to assume that
there is normally a useful CA certificate store available. It's also
no longer common for open source projects to avoid the "CA cartel" by
using self-signed or CAcert certificates.
* Enable validation by default for Bazaar, Git, and Mercurial
* Add a configuration option to disable it
* Add and document a .lorry keyword to disable it
We already validate server certificates for file downloads since the
Python standard library enabled it by default. We also never disabled
validation for Subversion. Since this seems to have worked OK, don't
add the option to disable it for these upstream types.
Closes #9.
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
lorry currently updates a git repository in its working area like this:
1. Delete old backup and debug repositories
2. Copy active repository (git) to backup (git-pre-update)
3. Perform conversion in active repository
4. If conversion failed, restore the backup:
1. Copy active repository to debug (git-post-fail)
2. Delete active repository
3. Copy backup repository to active
5. If conversion succeeded: delete git-pre-update
If step 3 or 4 is interrupted, for example because of a system reboot,
the active repository may be left missing or broken. On the next
update, the backup will be deleted.
Replace this with the following process:
1. Look for repositories 'git-a' and 'git-b'. The one with a higher
update count (or the only one that exists and has an update count
file) is the active repository. The other is the the temporary
repository. In case neither exists, 'git-a' is the temporary
repository.
2. Delete old temporary repository, if it exists
3. Copy active repository, if it exists, to temporary repository
(excluding the update count file)
4. Perform conversion in temporary repository
5. If conversion succeeded, make the temporary repository active by
writing the new update count file
6. If conversion failed, move the temporary repository to debug as
before
We need to migrate the working state, so if there is a 'git' and no
'git-a', add an update count of 1 to 'git' and move it to 'git-a'.
Also delete the old-style backup repository.
Update test cases to expect the new filenames.
lorry should now be crash-proof, in the sense that killing it at any
point will leave the last successful update in place. However, an
unclean system shutdown (e.g. kernel crash) that does not flush all
writes to persistent storage may still result in the active repository
being left broken.
Closes: #8.
|
|\
| |
| |
| |
| |
| |
| | |
lorry: Prune unreachable commits from hg-fast-export marks file
Closes #7
See merge request CodethinkLabs/lorry/lorry!12
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Logging each pruned commit can result in very long logs for some
repositories. This is a problem for Lorry Controller because job
status updates result in copying the entire log and not just the new
log lines.
There's not much value in listing all the commit hashes, so log the
number of commits instead. Update the test case accordingly.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
By default, hg-fast-export refuses to export a Mercurial repository if
it has unnamed heads. lorry passes the --force option that overrides
this since we still want to convert the named heads, but doing this
sets us up for failure later.
If the unnamed heads also have no tags pointing to them, the
corresponding git commits may be deleted by 'git gc'. However they
will still be listed in the 'marks' file used to record state for
incremental conversions, so a later run of hg-fast-export may assume
they exist and refer to them by hash.
Before running hg-fast-export, delete any lines in its marks file that
refer to unreachable or non-existent commits.
Closes #7.
|
|/
|
|
|
|
|
|
|
|
| |
We should not recommend using Breezy until we can point to a version
where fast-export is reliable.
* README: describe support for Breezy as experiemntal
* lorry: warn if attempting a Bazaar conversion using 'brz'
Relates to lorry-buildstream#14.
|
|
|
|
|
|
| |
This mostly reverts commit 5f5435647ada "Update copyright notices",
which changed all copyright dates to 2013. Most files were added in
2011 and 2012.
|
|
|
|
|
|
|
|
|
|
|
|
| |
lorry.gzip-importer uses the given file's modification time (mtime) as
the commit time, but this is currently unrelated to the mtime on the
server and so is unreproducible.
Copy the Last-Modified field of the urllib response header, if
present, to the mtime (and atime) of a file after downloading it. For
FTP downloads, this response field is synthesised by our FTP handler.
Closes #4.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
lorry.gzip-importer uses the given file's modification time (mtime) as
the commit time, but this is currently unrelated to the mtime on the
server and so is unreproducible.
For FTP downloads, urllib's default handler does not provide an mtime
and there is no reasonable way to add that in a subclass. Define and
use a new handler class that attempts to get the mtime from the server
using the MDTM extension.
Related to #4.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The original Bazaar implementation (bzr) only runs on Python 2, but
there is a fork known as Breezy (brz) that has been ported to Python 3.
To support environments without Python 2, allow using Breezy:
* Add a setting 'bazaar-command' that specifies which command to run
* Set the default based on a path search, preferring 'bzr' but using
'brz' if only 'brz' is found
* Implement a similar fallback in the test setup for Bazaar import
* Update README to mention Breezy as an alternative. The
'fastimport' extension is included in it rather than being a
separate package.
Closes #3.
|
|\
| |
| |
| |
| |
| |
| | |
lorry: Add support for push options
Closes #2
See merge request CodethinkLabs/lorry!4
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Gerrit requires passing the skip-validation option when pushing a
branch with a long history.
Add a 'push-option' application setting that allows adding any
such option to the git push command.
Closes #2.
|
|/
|
|
| |
Found by pyflakes.
|
|
|
|
|
| |
cliapp.Application.runcmd_unchecked returns bytes objects under Python
3. Decode these as UTF-8 in non-strict (replace) mode.
|
|
|
|
|
|
|
|
|
|
|
| |
The sed command includes the sequences '\{' and '\}' which are not
valid escape sequences and so expand to themselves (which is what is
needed). Unrecognised escape sequences are deprecated since Python
3.6 and will become a syntax error in future.
Use raw-string syntax to avoid the need for double-backslashes.
Found with pycodestyle.
|
| |
|
| |
|
|
|
|
|
|
|
| |
If parsing the lorry as YAML fails, we attempt to parse it as JSON
before giving up completely.
Change-Id: I7e3502f5ae9c32e10d63047e5f280bfe1d7e52dc
|
|
|
|
|
|
|
|
| |
This prevent git from running processes in the background. We were
having issues when `git gc` was running automatically, making `git gc`
call fail because it was already running.
Change-Id: Iee3e8e29eb0697f5241c7de4744c8e080bfd8e28
|
|
|
|
|
|
|
|
| |
We can do everything we need to on the command-line,
which makes more sense than syncing lorry file config every time,
and it makes it easier to add extra options later.
Change-Id: Ieaabc9fe7a83b5c29371ddb92b991d128c90e6c6
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
HG lorries are currently broken with current versions of hg-fast-export
Filed upstream bug here because it technically should not be broken:
https://github.com/frej/fast-export/issues/57
Note the issue in the upstream git repository here:
http://repo.or.cz/fast-export.git/blob/refs/heads/master:/hg-fast-export.sh#l54
This patch works around the issue by simply reordering the arguments
of hg-fast-export in the lorry program.
Change-Id: I2c8b303e2d90edb301186c0b3a6c57ba2e0b8179
|
|
|
|
|
|
|
|
| |
The new 'gzip' lorry file is practical for any gzipped single file
repositories, this can be useful for some static blob resources,
such as fonts.
Change-Id: I931e15e4c1291d820db82d0fc764aa767d3df9ab
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some svn repos don't have a branches dir or a tags dir in their layout.
This will for example allow us to lorry xmlto with the following lorry,
{
"xmlto": {
"type": "svn",
"url": "http://svn.fedorahosted.org/svn/xmlto",
"layout": {
"trunk": "."
}
}
}
Change-Id: I74c032ab7642f4b22ed0569b2a163c0b78cba658
|
|
|
|
|
|
| |
Also update copyright
Change-Id: Iff42a32048482520cee695bc1262990f5989ce63
|
|
|
|
| |
Change-Id: I29368ddcc64c82bafcbb834c81db30018a144f64
|
|\
| |
| |
| |
| | |
Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
|