Merge branch 'bwh/update-docs' into 'master'

Update documentation of dependencies and installation

Closes #2 and #1

See merge request CodethinkLabs/lorry-controller!15

2 files changed, 278 insertions, 0 deletions

diff --git a/INSTALL b/INSTALL
new file mode 100644
index 0000000..5344093
--- /dev/null
+++ b/INSTALL
@@ -0,0 +1,273 @@
+# Installing Lorry Controller
+
+## Dependencies
+
+Required:
+
+* **Python 3**: Tested with versions 3.6 and 3.8.
+
+* **Git**: Tested with versions 2.11 and 2.27.
+
+* **Lorry**: Also note the dependencies in its README.
+
+* **Bottle**: Can be installed from PyPI, or as the `python3-bottle`
+  package in Debian.  Tested with version 0.12.18.
+
+* **cliapp**: Can be installed as the `python3-cliapp` package in
+  Debian, or with:
+
+        pip3 install https://gitlab.com/trovekube/cliapp/-/archive/cliapp-1.20180812.1/cliapp-cliapp-1.20180812.1.tar.gz
+
+  or from the source at <https://liw.fi/cliapp/>.  Tested with version
+  1.20180812.1.
+
+* **PyYAML**: Can be installed from PyPI, or as the `python3-yaml`
+  package in Debian.  Tested with version 5.3.1.
+
+* **yoyo-migrations**: Can be installed from PyPI.  Tested with
+  version 7.0.2.
+
+Optional:
+
+* **curl**: Needed if you want to run the test suite.  Can be
+  installed as a distribution package.  Tested with versions 7.52.1
+  and 7.70.0.
+
+* **flup**: Needed if you want to run the web application as a FastCGI
+  server.  Can be installed from PyPI.  Tested with version 1.0.3, and
+  earlier versions won't run on Python 3.
+
+* **OpenSSH**: The OpenSSH client is needed if you want to use any
+  Downstream Host other than the local filesystem.  Tested with
+  versions 7.4p1 and 8.2p1.
+
+* **python-gitlab**: Needed if you want to use GitLab as a Downstream
+  or Upstream Host.  Can be installed from PyPI or as the
+  `python3-gitlab` package in Debian 10 onward.  Tested with version
+  1.15.0.
+
+* **Requests**: Needed if you want to use Gitano or GitLab as the
+  Downstream Host.  Can be installed from PyPI or as the
+  `python3-requests` package in Debian.  Tested with version 2.23.0.
+
+* **yarn**: Needed if you want to run the test suite.  Can be
+  installed as the `cmdtest` package in Debian, or from the source at
+  <https://liw.fi/cmdtest/>.  Tested with version 0.27-1 and with
+  commit `cdfe14e45134` on the master branch.
+
+## User account
+
+Create a single user account and home directory for Lorry and Lorry
+Controller on the host where they will run.
+
+Create an SSH key pair for Lorry, and install the *private* key in
+`.ssh` in Lorry's home directory.
+
+## Configuring the Downstream Host
+
+### Gerrit
+
+These instructions were written for Gerrit 3.1.
+
+1. Create a user for Lorry in Gerrit's authentication provider.
+   Add Lorry's SSH *public* key to the Gerrit user account.
+
+2. Create a group in Gerrit, or add the user to a group, that will be
+   permitted to create repositories and push changes to them.  The
+   Lorry user should be a member but not an owner of this group.
+
+3. (Optional but strongly recommended) Create a parent project for
+   the mirror repositories and make this group the owner.
+
+   Use the `gerrit create-project` command with the
+   `--permissions-only` option.  Alternately, in the web UI, create
+   a new project and fill out the form as follows:
+
+   * Set 'Repository name' as you wish.  This is independent of the
+     names of repositories that Lorry will create.
+   * Leave 'Rights inherit' blank
+   * Set Owner to the group
+   * Set 'Create initial empty commit' to 'False'
+   * Set 'Only server as parent for other repositories' to 'True'
+
+4. Give the group permission to create repositories,
+   [bypass review](https://gerrit-review.googlesource.com/Documentation/user-upload.html#bypass_review),
+   [skip validation](https://gerrit-review.googlesource.com/Documentation/user-upload.html#skip_validation),
+   and push tags that aren't on a branch:
+
+   * In 'All-Projects', give the group 'Create Project' permission.
+     In the web UI this is in the Global Capabilities section.
+   * In the parent project (or 'All-Projects'), give the group 'Forge
+     Author Identity', 'Forge Committer Identity', 'Forge Server
+     Identity', 'Push', and 'Push Merge Commit' permissions over
+     `refs/*`
+   * If you *did not* create a parent project, then in 'All-Projects'
+     also give the group 'Create Reference', 'Create Signed Tag', and
+     'Create Annotated Tag' permissions over `refs/*`
+
+5. In `lorry.conf`:
+
+   * Set `mirror-base-url-{fetch,push}` to
+   `git+ssh://`*username*`@`*hostname*`:29418`
+   * Set `push-option = skip-validation`
+
+6. In `webapp.conf`:
+
+   * Set `downstream-host-type = gerrit`
+   * Set `downstream-ssh-url = ssh://`*username*`@`*hostname*`:29418`
+   * Set `gerrit-parent-project =` *parent-project*
+
+7. Add Gerrit's SSH host public key to `.ssh/known_hosts` in Lorry's
+   home directory.
+
+### Gitano
+
+Gitano and Lorry Controller would normally be deployed together as
+part of a Trove: <http://wiki.baserock.org/Trove/reference/>.
+
+### Gitea
+
+These instructions were written for Gitea 1.11.
+
+1. Create a user for Lorry in Gitea (or its authentication provider).
+   Log in as the user and add Lorry's SSH *public* key to the user
+   account.  Generate an access token for the user.
+
+2. Set `mirror-base-url-{fetch,push}` in `lorry.conf` to
+   `git+ssh://git@`*hostname*
+
+3. In `webapp.conf`:
+
+   * Set `downstream-host-type = gitea`
+   * Set `downstream-visibility` to the desired visibility of
+     repositories: `private`, `internal`, or `public`
+   * Set `downstream-http-url` to the HTTPS or HTTP (not recommended)
+     URL of the Gitea server.
+   * Set `gitea-access-token =` *access-token*
+
+4. Add Gitea's SSH host public key to `.ssh/known_hosts` in Lorry's
+   home directory.
+
+Gitea requires all repositories to be organised under a user or
+organisation, and organisations cannot contain other organisations.
+You must therefore ensure that the CONFGIT specifies repository paths
+with exactly two path components.
+
+Lorry Controller will attempt to create organisations as needed to
+contain repositories.  If your Gitea configuration does not allow
+users to do this, you will need to create organisations in advance and
+give the Lorry user permission to create repositories under them.
+
+### GitLab
+
+These instructions were written for GitLab CE 12.10.
+
+1. Create a user for Lorry in GitLab (or its authentication provider).
+   Add Lorry's SSH *public* key to the user account.  Generate an
+   impersonation token for the user.
+
+2. Set `mirror-base-url-{fetch,push}` in `lorry.conf` to
+   `git+ssh://git@`*hostname*
+
+3. In `webapp.conf`:
+
+   * Set `downstream-host-type = gitlab`
+   * Set `downstream-visibility` to the desired visibility of
+     repositories: `private`, `internal`, or `public`
+   * Set `downstream-http-url` to the HTTPS or HTTP (not recommended)
+     URL of the GitLab server.
+   * Set `gitlab-private-token =` *impersonation-token*
+
+4. Add GitLab's SSH host public key to `.ssh/known_hosts` in Lorry's
+   home directory.
+
+GitLab requires all projects to be organised under a user or group.
+You must therefore ensure that the CONFGIT specifies repository paths
+with at least two path components.
+
+Lorry Controller will attempt to create groups as needed to contain
+projects.  If your GitLab configuration does not allow users to do
+this, you will need to create top-level groups in advance and give the
+Lorry user permission to create subgroups and projects under them.
+
+### Local filesystem
+
+1. Create a directory to contain the repositories, writable by
+   the Lorry user.
+
+2. Set `mirror-base-url-{fetch,push}` in `lorry.conf` to the directory
+   name.
+
+3. In `webapp.conf`:
+
+   * Set `downstream-host-type = local`
+   * Set `local-base-directory =` *directory*
+
+## Configuring a front-end web server
+
+WEBAPP can run behind a front-end web server connected through FastCGI.
+To enable FastCGI, set `wsgi = yes` in `webapp.conf`.
+
+The front-end web server must be configured so that:
+
+* It does any necessary access control
+
+* It passes the request path as `PATH_INFO`, not split into
+  `SCRIPT_NAME` and `PATH_INFO`
+
+* It creates the FastCGI socket and starts WEBAPP as the Lorry user.
+  WEBAPP should normally be started with the command:
+
+        /usr/bin/lorry-controller-webapp --config=/etc/lorry-controller/webapp.conf
+
+An example configuration for lighttpd, and a corresponding systemd
+unit file, are included in the source as
+`etc/lighttpd/lorry-controller-webapp-httpd.conf` and
+`units/lighttpd-lorry-controller-webapp.service`.
+
+## Creating the CONFGIT repository
+
+The CONFGIT repository can be hosted anywhere, but will normally be a
+private repository on the Downstream Host.  The Lorry user account on
+the Downstream Host must be given permission to read, but not write,
+to it.  Only administrators of Lorry Controller should be permitted
+to push to it.
+
+The configuration files stored in CONFGIT are documented in README.
+
+## Installing Lorry and Lorry Controller
+
+1. In a copy of the Lorry source tree, run:
+
+        python3 setup.py install --install-layout=deb
+
+2. Install `lorry.conf` in `/etc/`
+
+3. Create the directories named in `lorry.conf`
+   (`bundle-dest`, `tarball-dest`, `working-area` settings),
+   owned by the Lorry user.
+
+4. In a copy of the Lorry Controller source tree, run:
+
+        python3 setup.py install --install-layout=deb
+
+5. Install `webapp.conf` in `/etc/lorry-controller/`
+
+6. Create the directories named in `webapp.conf`
+   (`configuration-directory` setting and parent directories for the
+   `statedb` and `status-html` settings), owned by the Lorry user.
+
+7. Install the unit files from Lorry Controller's source tree
+   (`units/lorry-controller-*`) in `/lib/systemd/system/`.  These may
+   need to be modified to specify the correct URL for the front-end
+   web server.
+
+8. Run `systemctl daemon-reload` to make systemd load the unit files.
+
+9. Enable and start as many MINION services as you want to run in
+   parallel.  For example, to create 5 services:
+
+        for i in $(seq 5); do
+		    systemctl enable lorry-controller-minion@$i.service
+			systemctl start lorry-controller-minion@$i.service
+		done
diff --git a/README b/README
index 34a1d1f..f2afb80 100644
--- a/README
+++ b/README
@@ -22,6 +22,11 @@ interface, and an HTTP API for reporting and controlling its state.
 This README file documents the LC configuration file and general use.
 For the architecture of LC and the HTTP API, see the `ARCH` file.
 
+Installation
+------------
+
+See the INSTALL file.
+
 Lorry Controller configuration: overview
 ------------------------------