Add latest changes from gitlab-org/gitlab@13-7-stable-eev13.7.0-rc42

3 files changed, 140 insertions, 24 deletions

diff --git a/doc/administration/pages/img/zip_cache_configuration.png b/doc/administration/pages/img/zip_cache_configuration.png
new file mode 100644
index 00000000000..a332d8d0eb5
--- /dev/null
+++ b/doc/administration/pages/img/zip_cache_configuration.png
diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md
index 0ebeb96d247..af0476b72e9 100644
--- a/doc/administration/pages/index.md
+++ b/doc/administration/pages/index.md
@@ -1,7 +1,7 @@
 ---
 stage: Release
-group: Release Management
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+group: Release
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
 description: 'Learn how to administer GitLab Pages.'
 ---
 
@@ -16,7 +16,7 @@ description: 'Learn how to administer GitLab Pages.'
 GitLab Pages allows for hosting of static sites. It must be configured by an
 administrator. Separate [user documentation](../../user/project/pages/index.md) is available.
 
-NOTE: **Note:**
+NOTE:
 This guide is for Omnibus GitLab installations. If you have installed
 GitLab from source, see
 [GitLab Pages administration for source installations](source.md).
@@ -53,7 +53,7 @@ supporting custom domains a secondary IP is not needed.
 
 Before proceeding with the Pages configuration, you will need to:
 
-1. Have a domain for Pages that is not a subdomain of your GitLab's instance domain.
+1. Have a domain for Pages that is not a subdomain of your GitLab instance domain.
 
    | GitLab domain | Pages domain | Does it work? |
    | :---: | :---: | :---: |
@@ -68,7 +68,7 @@ Before proceeding with the Pages configuration, you will need to:
    so that your users don't have to bring their own.
 1. (Only for custom domains) Have a **secondary IP**.
 
-NOTE: **Note:**
+NOTE:
 If your GitLab instance and the Pages daemon are deployed in a private network or behind a firewall, your GitLab Pages websites will only be accessible to devices/users that have access to the private network.
 
 ### Add the domain to the Public Suffix List
@@ -101,7 +101,7 @@ where `example.io` is the domain under which GitLab Pages will be served
 and `192.0.2.1` is the IPv4 address of your GitLab instance and `2001::1` is the
 IPv6 address. If you don't have IPv6, you can omit the AAAA record.
 
-NOTE: **Note:**
+NOTE:
 You should not use the GitLab domain to serve user pages. For more information see the [security section](#security).
 
 ## Configuration
@@ -181,7 +181,7 @@ behavior:
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 
-NOTE: **Note:**
+NOTE:
 `inplace_chroot` option might not work with the other features, such as [Pages Access Control](#access-control).
 The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) has more information about caveats and workarounds.
 
@@ -204,13 +204,13 @@ control over how the Pages daemon runs and serves content in your environment.
 | `artifacts_server_url` |  API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`.
 | `auth_redirect_uri` |  Callback URL for authenticating with GitLab. Defaults to project's subdomain of `pages_external_url` + `/auth`.
 | `auth_secret` |  Secret key for signing authentication requests. Leave blank to pull automatically from GitLab during OAuth registration.
-| `dir` |  Working directory for config and secrets files.
+| `dir` |  Working directory for configuration and secrets files.
 | `enable` |  Enable or disable GitLab Pages on the current system.
 | `external_http` |  Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`.
 | `external_https` |  Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`.
 | `gitlab_client_http_timeout`  | GitLab API HTTP client connection timeout in seconds (default: 10s).
 | `gitlab_client_jwt_expiry`  | JWT Token expiry time in seconds (default: 30s).
-| `domain_config_source` | Domain configuration source (default: `disk`)
+| `domain_config_source` | Domain configuration source (default: `auto`)
 | `gitlab_id` |  The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab.
 | `gitlab_secret` |  The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab.
 | `gitlab_server` |  Server to use for authentication when access control is enabled; defaults to GitLab `external_url`.
@@ -241,7 +241,7 @@ control over how the Pages daemon runs and serves content in your environment.
 | `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`.
 | `pages_nginx[]` | |
 | `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains).
-| `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more info. |
+| `FF_ENABLE_REDIRECTS` | Feature flag to disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#disable-redirects) for more information. |
 
 ---
 
@@ -375,7 +375,7 @@ Pages access control is disabled by default. To enable it:
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure).
 1. Users can now configure it in their [projects' settings](../../user/project/pages/pages_access_control.md).
 
-NOTE: **Important:**
+NOTE:
 For this setting to be effective with multi-node setups, it has to be applied to
 all the App nodes and Sidekiq nodes.
 
@@ -395,7 +395,7 @@ To do that:
 1. Check the **Disable public access to Pages sites** checkbox.
 1. Click **Save changes**.
 
-CAUTION: **Warning:**
+WARNING:
 For self-managed installations, all public websites remain private until they are
 redeployed. This issue will be resolved by
 [sourcing domain configuration from the GitLab API](https://gitlab.com/gitlab-org/gitlab/-/issues/218357).
@@ -427,6 +427,42 @@ Authority (CA) in the system certificate store.
 
 For Omnibus, this is fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates).
 
+### Zip serving and cache configuration
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/392) in GitLab 13.7.
+
+WARNING:
+These are advanced settings. The recommended default values are set inside GitLab Pages. You should
+change these settings only if absolutely necessary. Use extreme caution.
+
+GitLab Pages can serve content from zip archives through object storage (an
+[issue](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/485) exists for supporting disk storage
+as well). It uses an in-memory cache to increase the performance when serving content from a zip
+archive. You can modify the cache behavior by changing the following configuration flags.
+
+| Setting | Description |
+| ------- | ----------- |
+| `zip_cache_expiration` | The cache expiration interval of zip archives. Must be greater than zero to avoid serving stale content. Default is 60s. |
+| `zip_cache_cleanup` | The interval at which archives are cleaned from memory if they have already expired. Default is 30s. |
+| `zip_cache_refresh` | The time interval in which an archive is extended in memory if accessed before `zip_cache_expiration`. This works together with `zip_cache_expiration` to determine if an archive is extended in memory. See the [example below](#zip-cache-refresh-example) for important details. Default is 30s. |
+| `zip_open_timeout` | The maximum time allowed to open a zip archive. Increase this time for big archives or slow network connections, as doing so may affect the latency of serving Pages. Default is 30s. |
+
+#### Zip cache refresh example
+
+Archives are refreshed in the cache (extending the time they are held in memory) if they're accessed
+before `zip_cache_expiration`, and the time left before expiring is less than or equal to
+`zip_cache_refresh`. For example, if `archive.zip` is accessed at time 0s, it expires in 60s (the
+default for `zip_cache_expiration`). In the example below, if the archive is opened again after 15s
+it is **not** refreshed because the time left for expiry (45s) is greater than `zip_cache_refresh`
+(default 30s). However, if the archive is accessed again after 45s (from the first time it was
+opened) it's refreshed. This extends the time the archive remains in memory from
+`45s + zip_cache_expiration (60s)`, for a total of 105s.
+
+After an archive reaches `zip_cache_expiration`, it's marked as expired and removed on the next
+`zip_cache_cleanup` interval.
+
+![Zip cache configuration](img/zip_cache_configuration.png)
+
 ## Activate verbose logging for daemon
 
 Verbose logging was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in
@@ -539,7 +575,7 @@ your main application server.
 
 To configure GitLab Pages on a separate server:
 
-DANGER: **Warning:**
+WARNING:
 The following procedure includes steps to back up and edit the
 `gitlab-secrets.json` file. This file contains secrets that control
 database encryption. Proceed with caution.
@@ -632,13 +668,14 @@ Pages server.
 > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217912) in GitLab 13.3.
 
 GitLab Pages can use different sources to get domain configuration.
-The default value is `nil`; however, GitLab Pages will default to `disk`.
+The default value is `nil`; however, GitLab Pages will default to `auto`.
 
    ```ruby
    gitlab_pages['domain_config_source'] = nil
    ```
 
-You can specify `gitlab` to enable [API-based configuration](#gitlab-api-based-configuration).
+If left unchanged, GitLab Pages tries to use any available source (either `gitlab` or `disk`). The
+preferred source is `gitlab`, which uses [API-based configuration](#gitlab-api-based-configuration).
 
 For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/).
 
@@ -655,15 +692,86 @@ was used prior to GitLab 13.0. Follow these steps to enable it:
 
 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect.
 
-If you encounter an issue, you can disable it by choosing `disk` or `nil`:
+If you encounter an issue, you can disable it by choosing `disk`:
 
 ```ruby
-gitlab_pages['domain_config_source'] = nil
+gitlab_pages['domain_config_source'] = "disk"
 ```
 
 For other common issues, see the [troubleshooting section](#failed-to-connect-to-the-internal-gitlab-api)
 or report an issue.
 
+## Using object storage
+
+> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5577) in GitLab 13.6.
+
+[Read more about using object storage with GitLab](../object_storage.md).
+
+### Object storage settings
+
+The following settings are:
+
+- Nested under `pages:` and then `object_store:` on source installations.
+- Prefixed by `pages_object_store_` on Omnibus GitLab installations.
+
+| Setting | Description | Default |
+|---------|-------------|---------|
+| `enabled` | Whether object storage is enabled. | `false` |
+| `remote_directory` | The name of the bucket where Pages site content is stored. | |
+| `connection` | Various connection options described below. | |
+
+#### S3-compatible connection settings
+
+See [the available connection settings for different providers](../object_storage.md#connection-settings).
+
+In Omnibus installations:
+
+1. Add the following lines to `/etc/gitlab/gitlab.rb` and replace the values with the ones you want:
+
+   ```ruby
+   gitlab_rails['pages_object_store_enabled'] = true
+   gitlab_rails['pages_object_store_remote_directory'] = "pages"
+   gitlab_rails['pages_object_store_connection'] = {
+     'provider' => 'AWS',
+     'region' => 'eu-central-1',
+     'aws_access_key_id' => 'AWS_ACCESS_KEY_ID',
+     'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY'
+   }
+   ```
+
+   If you use AWS IAM profiles, be sure to omit the AWS access key and secret access key/value
+   pairs:
+
+   ```ruby
+   gitlab_rails['pages_object_store_connection'] = {
+     'provider' => 'AWS',
+     'region' => 'eu-central-1',
+     'use_iam_profile' => true
+   }
+   ```
+
+1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure)
+   for the changes to take effect.
+
+In installations from source:
+
+1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following lines:
+
+   ```yaml
+   pages:
+     object_store:
+       enabled: true
+       remote_directory: "pages" # The bucket name
+       connection:
+         provider: AWS # Only AWS supported at the moment
+         aws_access_key_id: AWS_ACESS_KEY_ID
+         aws_secret_access_key: AWS_SECRET_ACCESS_KEY
+         region: eu-central-1
+   ```
+
+1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source)
+   for the changes to take effect.
+
 ## Backup
 
 GitLab Pages are part of the [regular backup](../../raketasks/backup_restore.md), so there is no separate backup to configure.
@@ -734,7 +842,7 @@ x509: certificate signed by unknown authority
 ```
 
 The reason for those errors is that the files `resolv.conf` and `ca-bundle.pem` are missing inside the chroot.
-The fix is to copy the host's `/etc/resolv.conf` and GitLab's certificate bundle inside the chroot:
+The fix is to copy the host's `/etc/resolv.conf` and the GitLab certificate bundle inside the chroot:
 
 ```shell
 sudo mkdir -p /var/opt/gitlab/gitlab-rails/shared/pages/etc/ssl
@@ -811,3 +919,11 @@ Upgrading to an [officially supported operating system](https://about.gitlab.com
 This problem comes from the permissions of the GitLab Pages OAuth application. To fix it, go to
 **Admin > Applications > GitLab Pages** and edit the application. Under **Scopes**, ensure that the
 `api` scope is selected and save your changes.
+
+### Workaround in case no wildcard DNS entry can be set
+
+If the wildcard DNS [prerequisite](#prerequisites) can't be met, you can still use GitLab Pages in a limited fashion:
+
+1. [Move](../../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace)
+   all projects you need to use Pages with into a single group namespace, for example `pages`.
+1. Configure a [DNS entry](#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`.
diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md
index 87217b141a4..c7c25f0f3a7 100644
--- a/doc/administration/pages/source.md
+++ b/doc/administration/pages/source.md
@@ -1,12 +1,12 @@
 ---
 stage: Release
-group: Release Management
-info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+group: Release
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments
 ---
 
 # GitLab Pages administration for source installations
 
-NOTE: **Note:**
+NOTE:
 Before attempting to enable GitLab Pages, first make sure you have
 [installed GitLab](../../install/installation.md) successfully.
 
@@ -77,7 +77,7 @@ host that GitLab runs. For example, an entry would look like this:
 where `example.io` is the domain under which GitLab Pages will be served
 and `192.0.2.1` is the IP address of your GitLab instance.
 
-NOTE: **Note:**
+NOTE:
 You should not use the GitLab domain to serve user pages. For more information
 see the [security section](#security).
 
@@ -349,7 +349,7 @@ world. Custom domains and TLS are supported.
 
 ## NGINX caveats
 
-NOTE: **Note:**
+NOTE:
 The following information applies only for installations from source.
 
 Be extra careful when setting up the domain name in the NGINX configuration. You must
@@ -395,7 +395,7 @@ API to check that the user is authorized to read that site.
 From [GitLab 12.8](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3689) onward,
 Access Control parameters for Pages are set in a configuration file, which
 by convention is named `gitlab-pages-config`. The configuration file is passed to
-pages using the `-config flag` or CONFIG environment variable.
+pages using the `-config flag` or `CONFIG` environment variable.
 
 Pages access control is disabled by default. To enable it: