diff options
author | Ben Hutchings <ben.hutchings@codethink.co.uk> | 2020-07-15 13:55:00 +0100 |
---|---|---|
committer | Ben Hutchings <ben.hutchings@codethink.co.uk> | 2020-07-15 13:55:00 +0100 |
commit | ad2ca653536adfccbb90beec6bd717a0caafaac8 (patch) | |
tree | 079b0da2b7a849d0e068cb7b0d52f2c6ccdb4ec2 /README.md | |
parent | 4600f8f0beba655f72b4bb0098f94d47a0152a29 (diff) | |
download | lorry-ad2ca653536adfccbb90beec6bd717a0caafaac8.tar.gz |
README: Rename to README.md
This will cause it to be rendered on GitLab and other git hosts'
web interfaces.
Diffstat (limited to 'README.md')
-rw-r--r-- | README.md | 320 |
1 files changed, 320 insertions, 0 deletions
diff --git a/README.md b/README.md new file mode 100644 index 0000000..e46408a --- /dev/null +++ b/README.md @@ -0,0 +1,320 @@ +README for Lorry +================ + +Lorry is a tool to take upstream source code (in various formats, +though preferably in version control) and converts it into a git +repository. + +If you want to try this, use `--pull-only` and/or `--mirror-base-url-push` +so that you do not accidentally overwrite important stuff for Baserock. +(If you don't have direct commit access to Baserock on git.baserock.org +then you're not dangerous.) + +See the manual page for instructions on using. + +The usual use is that a server hosts lorry and checks for updates, +which it then lorries. Lorry should not generally be run from the +developer's machine apart from testing. This is because Lorry has to +keep the git trees to avoid it having to pull everything. + +You can find a lot of lorries to crib ideas from at +<https://git.baserock.org/cgit/baserock/local-config/lorries.git/>. + + +Dependencies +------------ + +Required: + +* **Python 3** + +* **Git** + +* **cliapp**: Can be installed as the `python3-cliapp` package in + Debian, or with: + + pip3 install https://gitlab.com/trovekube/cliapp/-/archive/cliapp-1.20180812.1/cliapp-cliapp-1.20180812.1.tar.gz + + or from the source at <https://liw.fi/cliapp/>. + +* **PyYAML**: Can be installed from PyPI. or as the `python3-yaml` + package in Debian. + +Optional: + +* **bzr-fastimport** or **Breezy**: Needed if you want to import + Bazaar (bzr) repositories. Can be installed as the `bzr-fastimport` + or `brz` package in Debian. + +* **cmdtest**: Needed if you want to run the test suite. Can be + installed as the `cmdtest` package in Debian, or from the source at + <https://liw.fi/cmdtest/>. + +* **git-cvsimport**: Needed if you want to import CVS repositories. + Can be installed as the `git-cvs` package in Debian. + +* **git-svn**: Needed if you want to import Subversion (svn) + repositories. Can be installed as the `git-svn` package in Debian. + +* **hg-fast-export**: Needed if you want to import Mercurial (hg) + repositories. Can be installed from the source at + <https://repo.or.cz/w/fast-export.git>. + +* **OpenSSH**: The OpenSSH client is needed if you want to import or + push to repositories using the SSH protocol. + +* **Perl**: Needed if you want to import tarballs. + +Lorry file specification +------------------------ + +Lorry files are json dicts where the repository names are the keys and the +values are dicts with the data required to mirror it. + +So a simple lorry that mirrors a git project looks like + + { + "git": { + "type": "git", + "url": "git://github.com/gitster/git.git" + } + } + +Multiple repositories can be specified in the same .lorry file, in which case +all of them will be processed by lorry. The following shows two repositories. + + { + "git": { + "type": "git", + "url": "git://github.com/gitster/git.git" + }, + "curl": { + "type": "git", + "url": "git://github.com/bagder/curl.git" + } + } + +Lorry can import other version control systems into git. + +### Mercurial +Mercurial is very similar to git, just change the type field to "hg" + + { + "sudo": { + "type": "hg", + "url": "http://www.sudo.ws/repos/sudo" + } + } + +### Bazaar +Repositories and branches in Bazaar mean different things to Git. +The practical difference for Lorry is that it is not possible to have +a url for a repository, urls map directly to branches. + + { + "libpipeline": { + "type": "bzr", + "branches": { + "trunk": "http://bzr.savannah.gnu.org/r/libpipeline/trunk" + } + } + } + +For convenience if the project only needs one branch mirrored, the url +is assumed to be the master branch. + + { + "libpipeline": { + "type": "bzr", + "url": "http://bzr.savannah.gnu.org/r/libpipeline/trunk" + } + } + +### Subversion +To support all the branches and tags a layout needs to be specified as svn is +very flexible with the possible layouts, however the most common is to have the +working branch in a directory called trunk, and the branches and tags in +respectively named subdirectories. +Because this is so common "standard" can be used as the layout + + { + "mpc": { + "type": "svn", + "url": "svn://scm.gforge.inria.fr/svn/mpc", + "layout": "standard" + } + } + +This is equivalent to + + { + "mpc": { + "type": "svn", + "url": "svn://scm.gforge.inria.fr/svn/mpc", + "layout": { + "trunk": "trunk", + "branches": "branches/*", + "tags": "tags/*" + } + } + } + +Trunk is the path to the directory where the main branch is located. +Branches and Tags are glob expressions to allow finer control over which paths +are used. +Trunk is mandatory, but Branches and Tags are optional. +Texlive keeps a lot of resources in their svn repository, we are only concerned +with the source code, so this layout should select the correct subdirectory for +each branch. + + { + "texlive": { + "type": "svn", + "url": "svn://tug.org/texlive/", + "layout": { + "trunk": "trunk/Build/source", + "branches": "branches/*/Build/source", + "tags": "tags/*/Build/source" + } + } + } + +Brace expansions can be used to specify subsets of paths. +Netpbm for example, keeps all its branches in the root directory + + { + "netpbm": { + "type": "svn", + "url": "https://netpbm.svn.sourceforge.net/svnroot/netpbm", + "layout": { + "trunk": "trunk", + "branches": "{advanced,stable,super_stable}", + "tags": "release_number/*" + } + } + } + +Note that git-svn can provide better history tracking if the url is as close to +the root of the repository as possible, so it may be more effective if the lorry +was specified similar to this, assuming svnroot is the real root of the repo + + { + "netpbm": { + "type": "svn", + "url": "https://netpbm.svn.sourceforge.net/svnroot/", + "layout": { + "trunk": "netpbm/trunk", + "branches": "netpbm/{advanced,stable,super_stable}", + "tags": "netpbm/release_number/*" + } + } + } + +### CVS +The url for CVS repositories is the CVSROOT string. The module is required as +cvs repositories usually have multiple modules, the module is usually the same +as the project name. + + { + "openssl": { + "type": "cvs", + "url": "anonymous@cvs.openssl.org:/openssl-cvs", + "module": "openssl" + } + } + +### Tarball + +Lorry can import a tarball fetched from a URL. The contents will be +committed on a branch named after the basename of the tar file (e.g. +bc-1.06.tar.gz will be imported into a branch named 'bc-1.06'. + +The import is done by the `lorry.tar-importer` subprocess. It can detect and +handle common compression formats including gzip, bz2, xz and lzma. It will +also detect if there is a 'top directory' that contains the tarball contents +and strip this out of the imported filenames. + +Tarball imports once required 'compression' and 'strip' to be specified. +These are obsolete now and are ignored by Lorry. + + { + "bc": { + "type": "tarball", + "url": "http://ftp.gnu.org/gnu/bc/bc-1.06.tar.gz" + } + } + +### Zip + +Lorry can import a zip file fetched from a URL. The contents will be +extracted in place and committed directly to master. The new commit will +be tagged with the basename of the imported zip file (e.g. docbook-xml-4.5.zip +will be tagged as 'docbook-xml-4.5') + + { + "docbook-xml": { + "type": "zip", + "url": "http://www.docbook.org/xml/4.5/docbook-xml-4.5.zip" + } + } + + +### Gzip + +Lorry can import a gzip file fetched from a URL. The file will be extracted in place +and committed directly to master after removing the .gz extension from the filename. + +The new commit will be tagged with the basename of the imported zip file (e.g. bkai00mp.ttf.gz +will be tagged as 'bkai00mp.ttf') + + { + "ttf-bkai00mp": { + "type": "gzip", + "url": "ftp://ftp.gnu.org/non-gnu/chinese-fonts-truetype/bkai00mp.ttf.gz" + } + } + + +Tips +---- + +1. Use upstream's git repository whenever possible + + Importing from foreign version control systems is always slower than a + git mirror (with the exception of tarballs because they have less history). + +2. GNU Projects often have a git repository + + Most GNU projects are old compared to git, so were mainly developed in CVS. + Many official websites only mention the CVS or SVN repositories. + They will tend to have a git repository as well though, especially if they + are hosted on savannah. + +Legal stuff +----------- + +Copyright (C) 2013-2020 Codethink Limited + +This program is free software; you can redistribute it and/or modify +it under the terms of the GNU General Public License as published by +the Free Software Foundation; version 2 of the License. + +This program is distributed in the hope that it will be useful, +but WITHOUT ANY WARRANTY; without even the implied warranty of +MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +GNU General Public License for more details. + +You should have received a copy of the GNU General Public License along +with this program; if not, write to the Free Software Foundation, Inc., +51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA. + +We carry 2 slightly modified git-fast-import frontends: + +lorry.tar-importer which is based on +https://github.com/git/git/blob/master/contrib/fast-import/import-tars.perl + +and lorry.zip-importer, which is based on +https://github.com/git/git/blob/master/contrib/fast-import/import-zips.py + +all parts of git are under a GPLv2 compatible license, +see the git source code for more details. |