path: root/INSTALL.md

# Installing Lorry Controller

## Dependencies

Required:

* **Python 3**: Tested with versions 3.6 and 3.8.

* **Git**: Tested with versions 2.11 and 2.27.

* **Lorry**: Also note the dependencies in its README.

* **Bottle**: Can be installed from PyPI, or as the `python3-bottle`
  package in Debian.  Tested with version 0.12.18.

* **cliapp**: Can be installed as the `python3-cliapp` package in
  Debian, or with:

        pip3 install https://gitlab.com/trovekube/cliapp/-/archive/cliapp-1.20180812.1/cliapp-cliapp-1.20180812.1.tar.gz

  or from the source at <https://liw.fi/cliapp/>.  Tested with version
  1.20180812.1.

* **PyYAML**: Can be installed from PyPI, or as the `python3-yaml`
  package in Debian.  Tested with version 5.3.1.

* **yoyo-migrations**: Can be installed from PyPI.  Tested with
  version 7.0.2.

Optional:

* **curl**: Needed if you want to run the test suite.  Can be
  installed as a distribution package.  Tested with versions 7.52.1
  and 7.70.0.

* **flup**: Needed if you want to run the web application as a FastCGI
  server.  Can be installed from PyPI.  Tested with version 1.0.3, and
  earlier versions won't run on Python 3.

* **OpenSSH**: The OpenSSH client is needed if you want to use any
  Downstream Host other than the local filesystem.  Tested with
  versions 7.4p1 and 8.2p1.

* **python-gitlab**: Needed if you want to use GitLab as a Downstream
  or Upstream Host.  Can be installed from PyPI or as the
  `python3-gitlab` package in Debian 10 onward.  Tested with version
  1.15.0.

* **Requests**: Needed if you want to use Gitano or GitLab as the
  Downstream Host.  Can be installed from PyPI or as the
  `python3-requests` package in Debian.  Tested with version 2.23.0.

* **yarn**: Needed if you want to run the test suite.  Can be
  installed as the `cmdtest` package in Debian, or from the source at
  <https://liw.fi/cmdtest/>.  Tested with version 0.27-1 and with
  commit `cdfe14e45134` on the master branch.

## User account

Create a single user account and home directory for Lorry and Lorry
Controller on the host where they will run.

Create an SSH key pair for Lorry, and install the *private* key in
`.ssh` in Lorry's home directory.

## Configuring the Downstream Host

### Gerrit

These instructions were written for Gerrit 3.1.

1. Create a user for Lorry in Gerrit's authentication provider.
   Add Lorry's SSH *public* key to the Gerrit user account.

2. Create a group in Gerrit, or add the user to a group, that will be
   permitted to create repositories and push changes to them.  The
   Lorry user should be a member but not an owner of this group.

3. (Optional but strongly recommended) Create a parent project for
   the mirror repositories and make this group the owner.

   Use the `gerrit create-project` command with the
   `--permissions-only` option.  Alternately, in the web UI, create
   a new project and fill out the form as follows:

   * Set 'Repository name' as you wish.  This is independent of the
     names of repositories that Lorry will create.
   * Leave 'Rights inherit' blank
   * Set Owner to the group
   * Set 'Create initial empty commit' to 'False'
   * Set 'Only server as parent for other repositories' to 'True'

4. Give the group permission to create repositories,
   [bypass review](https://gerrit-review.googlesource.com/Documentation/user-upload.html#bypass_review),
   [skip validation](https://gerrit-review.googlesource.com/Documentation/user-upload.html#skip_validation),
   and push tags that aren't on a branch:

   * In 'All-Projects', give the group 'Create Project' permission.
     In the web UI this is in the Global Capabilities section.
   * In the parent project (or 'All-Projects'), give the group 'Forge
     Author Identity', 'Forge Committer Identity', 'Forge Server
     Identity', 'Push', and 'Push Merge Commit' permissions over
     `refs/*`
   * If you *did not* create a parent project, then in 'All-Projects'
     also give the group 'Create Reference', 'Create Signed Tag', and
     'Create Annotated Tag' permissions over `refs/*`

5. In `lorry.conf`:

   * Set `mirror-base-url-{fetch,push}` to
   `git+ssh://`*username*`@`*hostname*`:29418`
   * Set `push-option = skip-validation`

6. In `webapp.conf`:

   * Set `downstream-host-type = gerrit`
   * Set `downstream-ssh-url = ssh://`*username*`@`*hostname*`:29418`
   * Set `gerrit-parent-project =` *parent-project*

7. Add Gerrit's SSH host public key to `.ssh/known_hosts` in Lorry's
   home directory.

### Gitano

Gitano and Lorry Controller would normally be deployed together as
part of a Trove: <http://wiki.baserock.org/Trove/reference/>.

### Gitea

These instructions were written for Gitea 1.11.

1. Create a user for Lorry in Gitea (or its authentication provider).
   Log in as the user and add Lorry's SSH *public* key to the user
   account.  Generate an access token for the user.

2. Set `mirror-base-url-{fetch,push}` in `lorry.conf` to
   `git+ssh://git@`*hostname*

3. In `webapp.conf`:

   * Set `downstream-host-type = gitea`
   * Set `downstream-visibility` to the desired visibility of
     repositories: `private`, `internal`, or `public`
   * Set `downstream-http-url` to the HTTPS or HTTP (not recommended)
     URL of the Gitea server.
   * Set `gitea-access-token =` *access-token*

4. Add Gitea's SSH host public key to `.ssh/known_hosts` in Lorry's
   home directory.

Gitea requires all repositories to be organised under a user or
organisation, and organisations cannot contain other organisations.
You must therefore ensure that the CONFGIT specifies repository paths
with exactly two path components.

Lorry Controller will attempt to create organisations as needed to
contain repositories.  If your Gitea configuration does not allow
users to do this, you will need to create organisations in advance and
give the Lorry user permission to create repositories under them.

### GitLab

These instructions were written for GitLab CE 12.10.

1. Create a user for Lorry in GitLab (or its authentication provider).
   Add Lorry's SSH *public* key to the user account.  Generate an
   impersonation token for the user.

2. Set `mirror-base-url-{fetch,push}` in `lorry.conf` to
   `git+ssh://git@`*hostname*

3. In `webapp.conf`:

   * Set `downstream-host-type = gitlab`
   * Set `downstream-visibility` to the desired visibility of
     repositories: `private`, `internal`, or `public`
   * Set `downstream-http-url` to the HTTPS or HTTP (not recommended)
     URL of the GitLab server.
   * Set `gitlab-private-token =` *impersonation-token*

4. Add GitLab's SSH host public key to `.ssh/known_hosts` in Lorry's
   home directory.

GitLab requires all projects to be organised under a user or group.
You must therefore ensure that the CONFGIT specifies repository paths
with at least two path components.

Lorry Controller will attempt to create groups as needed to contain
projects.  If your GitLab configuration does not allow users to do
this, you will need to create top-level groups in advance and give the
Lorry user permission to create subgroups and projects under them.

### Local filesystem

1. Create a directory to contain the repositories, writable by
   the Lorry user.

2. Set `mirror-base-url-{fetch,push}` in `lorry.conf` to the directory
   name.

3. In `webapp.conf`:

   * Set `downstream-host-type = local`
   * Set `local-base-directory =` *directory*

## Configuring a front-end web server

WEBAPP can run behind a front-end web server connected through FastCGI.
To enable FastCGI, set `wsgi = yes` in `webapp.conf`.

The front-end web server must be configured so that:

* It does any necessary access control

* It passes the request path as `PATH_INFO`, not split into
  `SCRIPT_NAME` and `PATH_INFO`

* It creates the FastCGI socket and starts WEBAPP as the Lorry user.
  WEBAPP should normally be started with the command:

        /usr/bin/lorry-controller-webapp --config=/etc/lorry-controller/webapp.conf

An example configuration for lighttpd, and a corresponding systemd
unit file, are included in the source as
`etc/lighttpd/lorry-controller-webapp-httpd.conf` and
`units/lighttpd-lorry-controller-webapp.service`.

## Creating the CONFGIT repository

The CONFGIT repository can be hosted anywhere, but will normally be a
private repository on the Downstream Host.  The Lorry user account on
the Downstream Host must be given permission to read, but not write,
to it.  Only administrators of Lorry Controller should be permitted
to push to it.

The configuration files stored in CONFGIT are documented in `README.md`.

## Installing Lorry and Lorry Controller

1. In a copy of the Lorry source tree, run:

        python3 setup.py install --install-layout=deb

2. Install `lorry.conf` in `/etc/`

3. Create the directories named in `lorry.conf`
   (`bundle-dest`, `tarball-dest`, `working-area` settings),
   owned by the Lorry user.

4. In a copy of the Lorry Controller source tree, run:

        python3 setup.py install --install-layout=deb

5. Install `webapp.conf` in `/etc/lorry-controller/`

6. Create the directories named in `webapp.conf`
   (`configuration-directory` setting and parent directories for the
   `statedb` and `status-html` settings), owned by the Lorry user.

7. Install the unit files from Lorry Controller's source tree
   (`units/lorry-controller-*`) in `/lib/systemd/system/`.  These may
   need to be modified to specify the correct URL for the front-end
   web server.

8. Run `systemctl daemon-reload` to make systemd load the unit files.

9. Enable and start as many MINION services as you want to run in
   parallel.  For example, to create 5 services:

        for i in $(seq 5); do
		    systemctl enable lorry-controller-minion@$i.service
			systemctl start lorry-controller-minion@$i.service
		done