Merge branch 'jej-pages-to-ce' into 'master'

Adding GitLab Pages to CE

Closes #14605, gitlab-com/infrastructure#1058, gitlab-ee#1333, and #323

See merge request !8463

3 files changed, 573 insertions, 1 deletions

diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md
index 1824829903c..dad8e956c0e 100644
--- a/doc/administration/high_availability/load_balancer.md
+++ b/doc/administration/high_availability/load_balancer.md
@@ -66,4 +66,4 @@ Read more on high-availability configuration:
       configure custom domains with custom SSL, which would not be possible
       if SSL was terminated at the load balancer.
 
-[gitlab-pages]: http://docs.gitlab.com/ee/pages/administration.html
+[gitlab-pages]: ../pages/index.md
diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md
new file mode 100644
index 00000000000..c352caf1115
--- /dev/null
+++ b/doc/administration/pages/index.md
@@ -0,0 +1,249 @@
+# GitLab Pages Administration
+
+> **Notes:**
+- [Introduced][ee-80] in GitLab EE 8.3.
+- Custom CNAMEs with TLS support were [introduced][ee-173] in GitLab EE 8.5.
+- GitLab Pages [were ported][ce-14605] to Community Edition in GitLab 8.17.
+- This guide is for Omnibus GitLab installations. If you have installed
+  GitLab from source, follow the [Pages source installation document](source.md).
+
+---
+
+This document describes how to set up the _latest_ GitLab Pages feature. Make
+sure to read the [changelog](#changelog) if you are upgrading to a new GitLab
+version as it may include new features and changes needed to be made in your
+configuration.
+
+If you are looking for ways to upload your static content in GitLab Pages, you
+probably want to read the [user documentation][pages-userguide].
+
+## Overview
+
+GitLab Pages makes use of the [GitLab Pages daemon], a simple HTTP server
+written in Go that can listen on an external IP address and provide support for
+custom domains and custom certificates. It supports dynamic certificates through
+SNI and exposes pages using HTTP2 by default.
+You are encouraged to read its [README][pages-readme] to fully understand how
+it works.
+
+---
+
+In the case of custom domains, the Pages daemon needs to listen on ports `80`
+and/or `443`. For that reason, there is some flexibility in the way which you
+can set it up:
+
+1. Run the pages daemon in the same server as GitLab, listening on a secondary IP
+1. Run the pages daemon in a separate server. In that case, the
+   [Pages path](#change-storage-path) must also be present in the server that
+   the pages daemon is installed, so you will have to share it via network.
+1. Run the pages daemon in the same server as GitLab, listening on the same IP
+   but on different ports. In that case, you will have to proxy the traffic with
+   a loadbalancer. If you choose that route note that you should use TCP load
+   balancing for HTTPS. If you use TLS-termination (HTTPS-load balancing) the
+   pages will not be able to be served with user provided certificates. For
+   HTTP it's OK to use HTTP or TCP load balancing.
+
+In this document, we will proceed assuming the first option.
+
+## Prerequisites
+
+Before proceeding with the Pages configuration, you will need to:
+
+1. Have a separate domain under which the GitLab Pages will be served. In this
+   document we assume that to be `example.io`.
+1. Configure a **wildcard DNS record**.
+1. (Optional) Have a **wildcard certificate** for that domain if you decide to
+   serve Pages under HTTPS.
+1. (Optional but recommended) Enable [Shared runners](../ci/runners/README.md)
+   so that your users don't have to bring their own.
+
+### DNS configuration
+
+GitLab Pages expect to run on their own virtual host. In your DNS server/provider
+you need to add a [wildcard DNS A record][wiki-wildcard-dns] pointing to the
+host that GitLab runs. For example, an entry would look like this:
+
+```
+*.example.io. 1800 IN A 1.2.3.4
+```
+
+where `example.io` is the domain under which GitLab Pages will be served
+and `1.2.3.4` is the IP address of your GitLab instance.
+
+> **Note:**
+You should not use the GitLab domain to serve user pages. For more information
+see the [security section](#security).
+
+[wiki-wildcard-dns]: https://en.wikipedia.org/wiki/Wildcard_DNS_record
+
+## Configuration
+
+Depending on your needs, you can install GitLab Pages in four different ways.
+
+### Option 1. Custom domains with HTTPS support
+
+| URL scheme | Wildcard certificate | Custom domain with HTTP support | Custom domain with HTTPS support | Secondary IP |
+| --- |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `https://page.example.io` and `https://page.com` | yes |  redirects to HTTPS | yes | yes |
+
+Pages enabled, daemon is enabled AND pages has external IP support enabled.
+In that case, the pages daemon is running, NGINX still proxies requests to
+the daemon but the daemon is also able to receive requests from the outside
+world. Custom domains and TLS are supported.
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    pages_external_url "https://example.io"
+    nginx['listen_addresses'] = ['1.1.1.1']
+    pages_nginx['enable'] = false
+    gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt"
+    gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key"
+    gitlab_pages['external_http'] = '1.1.1.2:80'
+    gitlab_pages['external_https'] = '1.1.1.2:443'
+    ```
+
+    where `1.1.1.1` is the primary IP address that GitLab is listening to and
+    `1.1.1.2` the secondary IP where the GitLab Pages daemon listens to.
+
+1. [Reconfigure GitLab][reconfigure]
+
+### Option 2. Custom domains without HTTPS support
+
+| URL scheme |  Wildcard certificate | Custom domain with HTTP support | Custom domain with HTTPS support | Secondary IP |
+| --- |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `http://page.example.io` and `http://page.com` | no  |  yes    | no  | yes |
+
+Pages enabled, daemon is enabled AND pages has external IP support enabled.
+In that case, the pages daemon is running, NGINX still proxies requests to
+the daemon but the daemon is also able to receive requests from the outside
+world. Custom domains and TLS are supported.
+
+1. Edit `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    pages_external_url "http://example.io"
+    nginx['listen_addresses'] = ['1.1.1.1']
+    pages_nginx['enable'] = false
+    gitlab_pages['external_http'] = '1.1.1.2:80'
+    ```
+
+    where `1.1.1.1` is the primary IP address that GitLab is listening to and
+    `1.1.1.2` the secondary IP where the GitLab Pages daemon listens to.
+
+1. [Reconfigure GitLab][reconfigure]
+
+### Option 3. Wildcard HTTPS domain without custom domains
+
+| URL scheme | Wildcard certificate | Custom domain with HTTP support | Custom domain with HTTPS support | Secondary IP |
+| --- |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `https://page.example.io` | yes |  no | no | no |
+
+Pages enabled, daemon is enabled and NGINX will proxy all requests to the
+daemon. Pages daemon doesn't listen to the outside world.
+
+1. Place the certificate and key inside `/etc/gitlab/ssl`
+1. In `/etc/gitlab/gitlab.rb` specify the following configuration:
+
+    ```ruby
+    pages_external_url 'https://example.io'
+
+    pages_nginx['redirect_http_to_https'] = true
+    pages_nginx['ssl_certificate'] = "/etc/gitlab/ssl/pages-nginx.crt"
+    pages_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/pages-nginx.key"
+    ```
+
+    where `pages-nginx.crt` and `pages-nginx.key` are the SSL cert and key,
+    respectively.
+
+1. [Reconfigure GitLab][reconfigure]
+
+### Option 4. Wildcard HTTP domain without custom domains
+
+| URL scheme | Wildcard certificate | Custom domain with HTTP support | Custom domain with HTTPS support | Secondary IP |
+| --- |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `http://page.example.io`  | no  |  no | no | no |
+
+Pages enabled, daemon is enabled and NGINX will proxy all requests to the
+daemon. Pages daemon doesn't listen to the outside world.
+
+1. Set the external URL for GitLab Pages in `/etc/gitlab/gitlab.rb`:
+
+    ```ruby
+    pages_external_url 'http://example.io'
+    ```
+
+1. [Reconfigure GitLab][reconfigure]
+
+## Change storage path
+
+Follow the steps below to change the default path where GitLab Pages' contents
+are stored.
+
+1. Pages are stored by default in `/var/opt/gitlab/gitlab-rails/shared/pages`.
+   If you wish to store them in another location you must set it up in
+   `/etc/gitlab/gitlab.rb`:
+
+     ```ruby
+     gitlab_rails['pages_path'] = "/mnt/storage/pages"
+     ```
+
+1. [Reconfigure GitLab][reconfigure]
+
+## Set maximum pages size
+
+The maximum size of the unpacked archive per project can be configured in the
+Admin area under the Application settings in the **Maximum size of pages (MB)**.
+The default is 100MB.
+
+## Backup
+
+Pages are part of the [regular backup][backup] so there is nothing to configure.
+
+## Security
+
+You should strongly consider running GitLab pages under a different hostname
+than GitLab to prevent XSS attacks.
+
+## Changelog
+
+GitLab Pages were first introduced in GitLab EE 8.3. Since then, many features
+where added, like custom CNAME and TLS support, and many more are likely to
+come. Below is a brief changelog. If no changes were introduced or a version is
+missing from the changelog, assume that the documentation is the same as the
+latest previous version.
+
+---
+
+**GitLab 8.17 ([documentation][8-17-docs])**
+
+- GitLab Pages were ported to Community Edition in GitLab 8.17.
+- Documentation was refactored to be more modular and easy to follow.
+
+**GitLab 8.5 ([documentation][8-5-docs])**
+
+- In GitLab 8.5 we introduced the [gitlab-pages][] daemon which is now the
+  recommended way to set up GitLab Pages.
+- The [NGINX configs][] have changed to reflect this change. So make sure to
+  update them.
+- Custom CNAME and TLS certificates support.
+- Documentation was moved to one place.
+
+**GitLab 8.3 ([documentation][8-3-docs])**
+
+- GitLab Pages feature was introduced.
+
+[8-3-docs]: https://gitlab.com/gitlab-org/gitlab-ee/blob/8-3-stable-ee/doc/pages/administration.md
+[8-5-docs]: https://gitlab.com/gitlab-org/gitlab-ee/blob/8-5-stable-ee/doc/pages/administration.md
+[8-17-docs]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-17-stable-ce/doc/administration/pages/index.md
+[backup]: ../raketasks/backup_restore.md
+[ce-14605]: https://gitlab.com/gitlab-org/gitlab-ce/issues/14605
+[ee-80]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/80
+[ee-173]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/173
+[gitlab pages daemon]: https://gitlab.com/gitlab-org/gitlab-pages
+[NGINX configs]: https://gitlab.com/gitlab-org/gitlab-ee/tree/8-5-stable-ee/lib/support/nginx
+[pages-readme]: https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md
+[pages-userguide]: ../../user/project/pages/index.md
+[reconfigure]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
+[restart]: ../administration/restart_gitlab.md#installations-from-source
+[gitlab-pages]: https://gitlab.com/gitlab-org/gitlab-pages/tree/v0.2.4
diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md
new file mode 100644
index 00000000000..d4468b99992
--- /dev/null
+++ b/doc/administration/pages/source.md
@@ -0,0 +1,323 @@
+# GitLab Pages administration for source installations
+
+This is the documentation for configuring a GitLab Pages when you have installed
+GitLab from source and not using the Omnibus packages.
+
+You are encouraged to read the [Omnibus documentation](index.md) as it provides
+some invaluable information to the configuration of GitLab Pages. Please proceed
+to read it before going forward with this guide.
+
+We also highly recommend that you use the Omnibus GitLab packages, as we
+optimize them specifically for GitLab, and we will take care of upgrading GitLab
+Pages to the latest supported version.
+
+## Overview
+
+[Read the Omnibus overview section.](index.md#overview)
+
+## Prerequisites
+
+[Read the Omnibus prerequisites section.](index.md#prerequisites)
+
+## Configuration
+
+Depending on your needs, you can install GitLab Pages in four different ways.
+
+### Option 1. Custom domains with HTTPS support
+
+| URL scheme | Wildcard certificate | Custom domain with HTTP support | Custom domain with HTTPS support | Secondary IP |
+| --- |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `https://page.example.io` and `https://page.com` | yes |  redirects to HTTPS | yes | yes |
+
+Pages enabled, daemon is enabled AND pages has external IP support enabled.
+In that case, the pages daemon is running, NGINX still proxies requests to
+the daemon but the daemon is also able to receive requests from the outside
+world. Custom domains and TLS are supported.
+
+1. Install the Pages daemon:
+
+    ```
+    cd /home/git
+    sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git
+    cd gitlab-pages
+    sudo -u git -H git checkout v0.2.4
+    sudo -u git -H make
+    ```
+
+1. Edit `gitlab.yml` to look like the example below. You need to change the
+   `host` to the FQDN under which GitLab Pages will be served. Set
+   `external_http` and `external_https` to the secondary IP on which the pages
+   daemon will listen for connections:
+
+     ```yaml
+     ## GitLab Pages
+     pages:
+       enabled: true
+       # The location where pages are stored (default: shared/pages).
+       # path: shared/pages
+
+       host: example.io
+       port: 443
+       https: true
+
+       external_http: 1.1.1.2:80
+       external_https: 1.1.1.2:443
+     ```
+
+1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in
+   order to enable the pages daemon. In `gitlab_pages_options` the
+   `-pages-domain`, `-listen-http` and `-listen-https` must match the `host`,
+   `external_http` and `external_https` settings that you set above respectively.
+   The `-root-cert` and `-root-key` settings are the wildcard TLS certificates
+   of the `example.io` domain:
+
+    ```
+    gitlab_pages_enabled=true
+    gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 1.1.1.2:80 -listen-https 1.1.1.2:443 -root-cert /path/to/example.io.crt -root-key /path/to/example.io.key
+    ```
+
+1. Copy the `gitlab-pages-ssl` Nginx configuration file:
+
+    ```bash
+    sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf
+    sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf
+    ```
+
+      Replace `gitlab-pages-ssl` with `gitlab-pages` if you are not using SSL.
+
+1. Edit all GitLab related configs in `/etc/nginx/site-available/` and replace
+   `0.0.0.0` with `1.1.1.1`, where `1.1.1.1` the primary IP where GitLab
+   listens to.
+1. Restart NGINX
+1. [Restart GitLab][restart]
+
+### Option 2. Custom domains without HTTPS support
+
+| URL scheme |  Wildcard certificate | Custom domain with HTTP support | Custom domain with HTTPS support | Secondary IP |
+| --- |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `http://page.example.io` and `http://page.com` | no  |  yes    | no  | yes |
+
+Pages enabled, daemon is enabled AND pages has external IP support enabled.
+In that case, the pages daemon is running, NGINX still proxies requests to
+the daemon but the daemon is also able to receive requests from the outside
+world. Custom domains and TLS are supported.
+
+1. Install the Pages daemon:
+
+    ```
+    cd /home/git
+    sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git
+    cd gitlab-pages
+    sudo -u git -H git checkout v0.2.4
+    sudo -u git -H make
+    ```
+
+1. Edit `gitlab.yml` to look like the example below. You need to change the
+   `host` to the FQDN under which GitLab Pages will be served. Set
+   `external_http` to the secondary IP on which the pages daemon will listen
+   for connections:
+
+     ```yaml
+     pages:
+       enabled: true
+       # The location where pages are stored (default: shared/pages).
+       # path: shared/pages
+
+       host: example.io
+       port: 80
+       https: false
+
+       external_http: 1.1.1.2:80
+     ```
+
+1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in
+   order to enable the pages daemon. In `gitlab_pages_options` the
+   `-pages-domain` and `-listen-http` must match the `host` and `external_http`
+   settings that you set above respectively:
+
+    ```
+    gitlab_pages_enabled=true
+    gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 -listen-http 1.1.1.2:80"
+    ```
+
+1. Copy the `gitlab-pages-ssl` Nginx configuration file:
+
+    ```bash
+    sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf
+    sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf
+    ```
+
+      Replace `gitlab-pages-ssl` with `gitlab-pages` if you are not using SSL.
+
+1. Edit all GitLab related configs in `/etc/nginx/site-available/` and replace
+   `0.0.0.0` with `1.1.1.1`, where `1.1.1.1` the primary IP where GitLab
+   listens to.
+1. Restart NGINX
+1. [Restart GitLab][restart]
+
+### Option 3. Wildcard HTTPS domain without custom domains
+
+| URL scheme | Wildcard certificate | Custom domain with HTTP support | Custom domain with HTTPS support | Secondary IP |
+| --- |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `https://page.example.io` | yes |  no | no | no |
+
+Pages enabled, daemon is enabled and NGINX will proxy all requests to the
+daemon. Pages daemon doesn't listen to the outside world.
+
+1. Install the Pages daemon:
+
+    ```
+    cd /home/git
+    sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git
+    cd gitlab-pages
+    sudo -u git -H git checkout v0.2.4
+    sudo -u git -H make
+    ```
+1. In `gitlab.yml`, set the port to `443` and https to `true`:
+
+     ```bash
+     ## GitLab Pages
+     pages:
+       enabled: true
+       # The location where pages are stored (default: shared/pages).
+       # path: shared/pages
+
+       host: example.io
+       port: 443
+       https: true
+     ```
+
+1. Copy the `gitlab-pages-ssl` Nginx configuration file:
+
+    ```bash
+    sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf
+    sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf
+    ```
+
+      Replace `gitlab-pages-ssl` with `gitlab-pages` if you are not using SSL.
+
+1. Restart NGINX
+1. [Restart GitLab][restart]
+
+### Option 4. Wildcard HTTP domain without custom domains
+
+| URL scheme | Wildcard certificate | Custom domain with HTTP support | Custom domain with HTTPS support | Secondary IP |
+| --- |:---:|:---:|:---:|:---:|:---:|:---:|:---:|
+| `http://page.example.io`  | no  |  no | no | no |
+
+Pages enabled, daemon is enabled and NGINX will proxy all requests to the
+daemon. Pages daemon doesn't listen to the outside world.
+
+1. Install the Pages daemon:
+
+    ```
+    cd /home/git
+    sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-pages.git
+    cd gitlab-pages
+    sudo -u git -H git checkout v0.2.4
+    sudo -u git -H make
+    ```
+
+1. Go to the GitLab installation directory:
+
+     ```bash
+     cd /home/git/gitlab
+     ```
+
+1. Edit `gitlab.yml` and under the `pages` setting, set `enabled` to `true` and
+   the `host` to the FQDN under which GitLab Pages will be served:
+
+     ```yaml
+     ## GitLab Pages
+     pages:
+       enabled: true
+       # The location where pages are stored (default: shared/pages).
+       # path: shared/pages
+
+       host: example.io
+       port: 80
+       https: false
+     ```
+
+1. Copy the `gitlab-pages-ssl` Nginx configuration file:
+
+    ```bash
+    sudo cp lib/support/nginx/gitlab-pages-ssl /etc/nginx/sites-available/gitlab-pages-ssl.conf
+    sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf
+    ```
+
+      Replace `gitlab-pages-ssl` with `gitlab-pages` if you are not using SSL.
+
+1. Restart NGINX
+1. [Restart GitLab][restart]
+
+## NGINX caveats
+
+>**Note:**
+The following information applies only for installations from source.
+
+Be extra careful when setting up the domain name in the NGINX config. You must
+not remove the backslashes.
+
+If your GitLab pages domain is `example.io`, replace:
+
+```bash
+server_name ~^.*\.YOUR_GITLAB_PAGES\.DOMAIN$;
+```
+
+with:
+
+```
+server_name ~^.*\.example\.io$;
+```
+
+If you are using a subdomain, make sure to escape all dots (`.`) except from
+the first one with a backslash (\). For example `pages.example.io` would be:
+
+```
+server_name ~^.*\.pages\.example\.io$;
+```
+
+## Change storage path
+
+Follow the steps below to change the default path where GitLab Pages' contents
+are stored.
+
+1. Pages are stored by default in `/home/git/gitlab/shared/pages`.
+   If you wish to store them in another location you must set it up in
+   `gitlab.yml` under the `pages` section:
+
+     ```yaml
+     pages:
+       enabled: true
+       # The location where pages are stored (default: shared/pages).
+       path: /mnt/storage/pages
+     ```
+
+1. [Restart GitLab][restart]
+
+## Set maximum Pages size
+
+The maximum size of the unpacked archive per project can be configured in the
+Admin area under the Application settings in the **Maximum size of pages (MB)**.
+The default is 100MB.
+
+## Backup
+
+Pages are part of the [regular backup][backup] so there is nothing to configure.
+
+## Security
+
+You should strongly consider running GitLab pages under a different hostname
+than GitLab to prevent XSS attacks.
+
+[backup]: ../raketasks/backup_restore.md
+[ee-80]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/80
+[ee-173]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/173
+[gitlab pages daemon]: https://gitlab.com/gitlab-org/gitlab-pages
+[NGINX configs]: https://gitlab.com/gitlab-org/gitlab-ee/tree/8-5-stable-ee/lib/support/nginx
+[pages-readme]: https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md
+[pages-userguide]: ../../user/project/pages/index.md
+[reconfigure]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure
+[restart]: ../administration/restart_gitlab.md#installations-from-source
+[gitlab-pages]: https://gitlab.com/gitlab-org/gitlab-pages/tree/v0.2.4