diff options
Diffstat (limited to 'INSTALL')
| -rw-r--r-- | INSTALL | 273 |
1 files changed, 273 insertions, 0 deletions
@@ -0,0 +1,273 @@ +# Installing Lorry Controller + +## Dependencies + +Required: + +* **Python 3**: Tested with versions 3.6 and 3.8. + +* **Git**: Tested with versions 2.11 and 2.27. + +* **Lorry**: Also note the dependencies in its README. + +* **Bottle**: Can be installed from PyPI, or as the `python3-bottle` + package in Debian. Tested with version 0.12.18. + +* **cliapp**: Can be installed as the `python3-cliapp` package in + Debian, or with: + + pip3 install https://gitlab.com/trovekube/cliapp/-/archive/cliapp-1.20180812.1/cliapp-cliapp-1.20180812.1.tar.gz + + or from the source at <https://liw.fi/cliapp/>. Tested with version + 1.20180812.1. + +* **PyYAML**: Can be installed from PyPI, or as the `python3-yaml` + package in Debian. Tested with version 5.3.1. + +* **yoyo-migrations**: Can be installed from PyPI. Tested with + version 7.0.2. + +Optional: + +* **curl**: Needed if you want to run the test suite. Can be + installed as a distribution package. Tested with versions 7.52.1 + and 7.70.0. + +* **flup**: Needed if you want to run the web application as a FastCGI + server. Can be installed from PyPI. Tested with version 1.0.3, and + earlier versions won't run on Python 3. + +* **OpenSSH**: The OpenSSH client is needed if you want to use any + Downstream Host other than the local filesystem. Tested with + versions 7.4p1 and 8.2p1. + +* **python-gitlab**: Needed if you want to use GitLab as a Downstream + or Upstream Host. Can be installed from PyPI or as the + `python3-gitlab` package in Debian 10 onward. Tested with version + 1.15.0. + +* **Requests**: Needed if you want to use Gitano or GitLab as the + Downstream Host. Can be installed from PyPI or as the + `python3-requests` package in Debian. Tested with version 2.23.0. + +* **yarn**: Needed if you want to run the test suite. Can be + installed as the `cmdtest` package in Debian, or from the source at + <https://liw.fi/cmdtest/>. Tested with version 0.27-1 and with + commit `cdfe14e45134` on the master branch. + +## User account + +Create a single user account and home directory for Lorry and Lorry +Controller on the host where they will run. + +Create an SSH key pair for Lorry, and install the *private* key in +`.ssh` in Lorry's home directory. + +## Configuring the Downstream Host + +### Gerrit + +These instructions were written for Gerrit 3.1. + +1. Create a user for Lorry in Gerrit's authentication provider. + Add Lorry's SSH *public* key to the Gerrit user account. + +2. Create a group in Gerrit, or add the user to a group, that will be + permitted to create repositories and push changes to them. The + Lorry user should be a member but not an owner of this group. + +3. (Optional but strongly recommended) Create a parent project for + the mirror repositories and make this group the owner. + + Use the `gerrit create-project` command with the + `--permissions-only` option. Alternately, in the web UI, create + a new project and fill out the form as follows: + + * Set 'Repository name' as you wish. This is independent of the + names of repositories that Lorry will create. + * Leave 'Rights inherit' blank + * Set Owner to the group + * Set 'Create initial empty commit' to 'False' + * Set 'Only server as parent for other repositories' to 'True' + +4. Give the group permission to create repositories, + [bypass review](https://gerrit-review.googlesource.com/Documentation/user-upload.html#bypass_review), + [skip validation](https://gerrit-review.googlesource.com/Documentation/user-upload.html#skip_validation), + and push tags that aren't on a branch: + + * In 'All-Projects', give the group 'Create Project' permission. + In the web UI this is in the Global Capabilities section. + * In the parent project (or 'All-Projects'), give the group 'Forge + Author Identity', 'Forge Committer Identity', 'Forge Server + Identity', 'Push', and 'Push Merge Commit' permissions over + `refs/*` + * If you *did not* create a parent project, then in 'All-Projects' + also give the group 'Create Reference', 'Create Signed Tag', and + 'Create Annotated Tag' permissions over `refs/*` + +5. In `lorry.conf`: + + * Set `mirror-base-url-{fetch,push}` to + `git+ssh://`*username*`@`*hostname*`:29418` + * Set `push-option = skip-validation` + +6. In `webapp.conf`: + + * Set `downstream-host-type = gerrit` + * Set `downstream-ssh-url = ssh://`*username*`@`*hostname*`:29418` + * Set `gerrit-parent-project =` *parent-project* + +7. Add Gerrit's SSH host public key to `.ssh/known_hosts` in Lorry's + home directory. + +### Gitano + +Gitano and Lorry Controller would normally be deployed together as +part of a Trove: <http://wiki.baserock.org/Trove/reference/>. + +### Gitea + +These instructions were written for Gitea 1.11. + +1. Create a user for Lorry in Gitea (or its authentication provider). + Log in as the user and add Lorry's SSH *public* key to the user + account. Generate an access token for the user. + +2. Set `mirror-base-url-{fetch,push}` in `lorry.conf` to + `git+ssh://git@`*hostname* + +3. In `webapp.conf`: + + * Set `downstream-host-type = gitea` + * Set `downstream-visibility` to the desired visibility of + repositories: `private`, `internal`, or `public` + * Set `downstream-http-url` to the HTTPS or HTTP (not recommended) + URL of the Gitea server. + * Set `gitea-access-token =` *access-token* + +4. Add Gitea's SSH host public key to `.ssh/known_hosts` in Lorry's + home directory. + +Gitea requires all repositories to be organised under a user or +organisation, and organisations cannot contain other organisations. +You must therefore ensure that the CONFGIT specifies repository paths +with exactly two path components. + +Lorry Controller will attempt to create organisations as needed to +contain repositories. If your Gitea configuration does not allow +users to do this, you will need to create organisations in advance and +give the Lorry user permission to create repositories under them. + +### GitLab + +These instructions were written for GitLab CE 12.10. + +1. Create a user for Lorry in GitLab (or its authentication provider). + Add Lorry's SSH *public* key to the user account. Generate an + impersonation token for the user. + +2. Set `mirror-base-url-{fetch,push}` in `lorry.conf` to + `git+ssh://git@`*hostname* + +3. In `webapp.conf`: + + * Set `downstream-host-type = gitlab` + * Set `downstream-visibility` to the desired visibility of + repositories: `private`, `internal`, or `public` + * Set `downstream-http-url` to the HTTPS or HTTP (not recommended) + URL of the GitLab server. + * Set `gitlab-private-token =` *impersonation-token* + +4. Add GitLab's SSH host public key to `.ssh/known_hosts` in Lorry's + home directory. + +GitLab requires all projects to be organised under a user or group. +You must therefore ensure that the CONFGIT specifies repository paths +with at least two path components. + +Lorry Controller will attempt to create groups as needed to contain +projects. If your GitLab configuration does not allow users to do +this, you will need to create top-level groups in advance and give the +Lorry user permission to create subgroups and projects under them. + +### Local filesystem + +1. Create a directory to contain the repositories, writable by + the Lorry user. + +2. Set `mirror-base-url-{fetch,push}` in `lorry.conf` to the directory + name. + +3. In `webapp.conf`: + + * Set `downstream-host-type = local` + * Set `local-base-directory =` *directory* + +## Configuring a front-end web server + +WEBAPP can run behind a front-end web server connected through FastCGI. +To enable FastCGI, set `wsgi = yes` in `webapp.conf`. + +The front-end web server must be configured so that: + +* It does any necessary access control + +* It passes the request path as `PATH_INFO`, not split into + `SCRIPT_NAME` and `PATH_INFO` + +* It creates the FastCGI socket and starts WEBAPP as the Lorry user. + WEBAPP should normally be started with the command: + + /usr/bin/lorry-controller-webapp --config=/etc/lorry-controller/webapp.conf + +An example configuration for lighttpd, and a corresponding systemd +unit file, are included in the source as +`etc/lighttpd/lorry-controller-webapp-httpd.conf` and +`units/lighttpd-lorry-controller-webapp.service`. + +## Creating the CONFGIT repository + +The CONFGIT repository can be hosted anywhere, but will normally be a +private repository on the Downstream Host. The Lorry user account on +the Downstream Host must be given permission to read, but not write, +to it. Only administrators of Lorry Controller should be permitted +to push to it. + +The configuration files stored in CONFGIT are documented in README. + +## Installing Lorry and Lorry Controller + +1. In a copy of the Lorry source tree, run: + + python3 setup.py install --install-layout=deb + +2. Install `lorry.conf` in `/etc/` + +3. Create the directories named in `lorry.conf` + (`bundle-dest`, `tarball-dest`, `working-area` settings), + owned by the Lorry user. + +4. In a copy of the Lorry Controller source tree, run: + + python3 setup.py install --install-layout=deb + +5. Install `webapp.conf` in `/etc/lorry-controller/` + +6. Create the directories named in `webapp.conf` + (`configuration-directory` setting and parent directories for the + `statedb` and `status-html` settings), owned by the Lorry user. + +7. Install the unit files from Lorry Controller's source tree + (`units/lorry-controller-*`) in `/lib/systemd/system/`. These may + need to be modified to specify the correct URL for the front-end + web server. + +8. Run `systemctl daemon-reload` to make systemd load the unit files. + +9. Enable and start as many MINION services as you want to run in + parallel. For example, to create 5 services: + + for i in $(seq 5); do + systemctl enable lorry-controller-minion@$i.service + systemctl start lorry-controller-minion@$i.service + done |