diff options
Diffstat (limited to 'README')
| -rw-r--r-- | README | 208 |
1 files changed, 137 insertions, 71 deletions
@@ -1,103 +1,169 @@ README for lorry-controller =========================== -Lorry Controller mirrors Troves and mirrors or converts upstream -projects into git repositories on the local Trove. Lorry Controller -reads a configuration file (see below) and runs the Lorry program -against the intended targets at suitable intervals. +Overview +-------- -Lorry Controller configuration +Lorry Controller, or LC for short, manages the importing of source +code from external sources into git repositories on a Trove. A Trove +is a component in the Baserock system for hosting source code, and LC +runs inside a Trove. + +LC uses the Lorry tool to do the actual import. Lorry can read code +from several different version control systems, and convert them to +git. External repositories can be specfied individually, as Lorry +`.lorry` specification files. In addition, LC can be told to mirror +all the git repositories on another Trove. + +LC runs Lorry for the right external repositories, and takes care of +running a suitable number of Lorry instances concurrently, and +recovering from any problems. LC has a web based administration +interface, and an HTTP API for reporting and controlling its state. + +This README file documents the LC configuration file and general use. +For the architecture of LC and the HTTP API, see the `ARCH` file. + +Lorry Controller configuration: overview ------------------------------ -Lorry Controller reads a configuration file of the following format. - -* The file uses JSON syntax. -* The file is a list. -* Each item in the list is a mapping specifying a Trove or a set of - `.lorry` files. -* A Trove spec specifies another Trove to mirror completely. -* A Lorry spec specifies a set of Lorry specification (`.lorry` files) - for individual project repositories to convert or mirror to git on - the local Trove. - -Each spec (mapping) has a number of key/value pairs. The following are -shared between Trove and Lorry specs: - -* `type` is the type of the spec; value MUST be either `trove` or - `lorries`. -* `interval` specifies how often Lorry Controller should mirror the - repositories in the spec. See below for INTERVAL. -* `protocol`: specifies how Lorry Controller (and Lorry) should talk - to remove Troves. Allowed values are `ssh`, `https`, `http`. This - field is mandatory. -* `auth`: Specifies how to authenticate to the remote Trove over - https. The is an optional field. If present, it should be a - dictionary with the fields `username` and `password`. -* Additionally, the following seem to be supported by an old version - of Lorry Controller, but are ignored by the new Lorry Controller: - `uuid`, `serial`, `create`, `destroy`, `stagger`, `tarball`. - -Trove specs have the following keys: - -* `trovehost` is the other Trove to mirror; a domain name or IP - address. It is mandatory. -* `ls-interval` determines how often should Lorry Controller query the - other Trove for a list of repositories it may mirror. See below for - INTERVAL. `ls-interval` is mandatory. -* `prefixmap` maps repository path prefixes from the other Trove to - the local Trove. It is mandatory in a Trove spec. If the remote - prefix is `foo`, and the local prefix is `bar`, then remote - repository `foo/baserock/yeehaa` gets mirrored to local repository - `bar/baserock/yeehaa`. If the remote Trove has a repository that - does not match a prefix, that repository gets ignored. -* `ignore` is a list of git repositories from the other Trove that - should NOT be mirrored. Each list element is a path to the git - repository (not including leading slash). `ignore` is optional. +Lorry Controller has two levels of configuration. The first level is +command line options and configuration files. This level specifies +things such as log levels, network addresses to listen on, and such. +Most importantly, this level specifies the location of the second +level. For information about these options, run +`lorry-controller-webapp --help` to get a list of them. -An INTERVAL value (for `interval` or `ls-interval`) is number and a -unit to indicate a time interval. Allowed units are minutes (`m`), -hours (`h`), and days (`d`), expressed as single-letter codes in upper -or lower case. +The second level is a git repository that specifies which external +repositories and other Troves to import into the Trove LC runs on. +This git repository is referred to as CONFGIT in documentation, and is +specified with the the `--confgit-url` command line option, or the +`confgit-url` key in the configuration file. The configuration file +could contain this, for example: -Lorry specs have the following keys: + [config] + confgit-url = ssh://git@localhost/baserock/local-config/lorries -* `prefix` is a path prefix to be prepended to all repositories - created from the `.lorry` files from this spec. It is mandatory. -* `globs` is a list of globs (as strings) for `.lorry` files to use. - The glob is matched in the directory containing the configuration - file in which this spec is. It is OK for the globs to not match - anything. A `globs` entry is mandatory, however. +The system integration of a Trove automatically includes a +configuration file that contains a configuration such as the above. +The URL contains the name of the Trove, so it needs to be customised +for each Trove, but as long as you're only using LC as part of a +Baserock Trove, it's all taken care of for you automatically. -A fairly minimal example for mirroring `git.baserock.org` and using -local `.lorry` files. + +The CONFGIT repository +---------------------- + +The CONFGIT repository must contain at least the file +`lorry-controller.conf`. It may also contain other files, including +`.lorry` files for Lorry, but all other files are ignored unless +referenced by `lorry-controller.conf`. + + + +The `lorry-controller.conf` file +-------------------------------- + +`lorry-controller.conf` is a JSON file containing a list of maps. Each +map specifies another Trove, or one set of `.lorry` files. Here's an +example that tells LC to mirror the `git.baserock.org` Trove and +anything in the `open-source-lorries/*.lorry` files (if any exist). [ { - "type": "trove", - "trovehost": "git.baserock.org", - "ls-interval": "4d", - "interval": "2h", + "ignore": [ + "baserock/lorries" + ], + "interval": "2H", + "ls-interval": "4H", "prefixmap": { - "baserock": "baserock", + "baserock": "baserock", "delta": "delta" - } + }, + "protocol": "http", + "tarball": "always", + "trovehost": "git.baserock.org", + "type": "trove" }, { "type": "lorries", - "interval": "1h", + "interval": "6H", + "create": "always", + "destroy": "never", "prefix": "delta", + "tarball": "always", "globs": [ "open-source-lorries/*.lorry" ] } ] +A Trove specification (map) uses the following mandatory keys: + +* `type: trove` -- specify it's a Trove specifiation. + +* `trovehost` -- the other Trove to mirror; a domain name or IP + address. + +* `protocol` -- specify how Lorry Controller (and Lorry) should talk + to other Troves. Allowed values are `ssh`, `https`, `http`. + +* `prefixmap` -- map repository path prefixes from the other Trove to + the local Trove. If the remote prefix is `foo`, and the local prefix + is `bar`, then remote repository `foo/baserock/yeehaa` gets mirrored + to local repository `bar/baserock/yeehaa`. If the remote Trove has a + repository that does not match a prefix, that repository gets + ignored. + +* `ls-interval` -- determine how often should Lorry Controller query + the other Trove for a list of repositories it may mirror. See below + for how the value is interpreted. The default is 24 hours. + +* `interval` -- specify how often Lorry Controller should mirror the + repositories in the spec. See below for INTERVAL. The default + interval is 24 hours. + +Additionally, the following optional keys are allowed in Trove +specifications: + +* `ignore` -- a list of git repositories from the other Trove that + should NOT be mirrored. Each list element is a path to the git + repository (not including leading slash). + +* `auth` -- specify how to authenticate to the remote Trove over https + (only). It should be a dictionary with the fields `username` and + `password`. + +A Lorry specification (map) uses the following keys, all of them +mandatory: + +* `type: lorries` -- specify it's a Trove specifiation. + +* `interval` -- identical in meaning to the `interval` in a + Trove specification. + +* `prefix` -- a path prefix to be prepended to all repositories + created from the `.lorry` files from this spec. + +* `globs` -- a list of globs (as strings) for `.lorry` files to use. + The glob is matched in the directory containing the configuration + file in which this spec is. It is OK for the globs to not match + anything. + +For backwards compatibility with another implementation of Lorry +Controller, other fields in either type of specification are allowed +and silently ignored. + +An INTERVAL value (for `interval` or `ls-interval`) is a number and a +unit to indicate a time interval. Allowed units are minutes (`m`), +hours (`h`), and days (`d`), expressed as single-letter codes in upper +or lower case. + The syntax of `.lorry` files is specified by the Lorry program; see its documentation for details. -HTTP proxy configuration ------------------------- +HTTP proxy configuration: `proxy.conf` +-------------------------------------- Lorry Controller will look for a file called `proxy.conf` in the same directory as the `lorry-controller.conf` configuration file. |