Document docs.gitlab.com deployment processdocs/release-process

Port content from the gitlab-docs project.

3 files changed, 317 insertions, 22 deletions

diff --git a/doc/development/documentation/site_architecture/deployment.md b/doc/development/documentation/site_architecture/deployment.md
new file mode 100644
index 00000000000..6b334dccd19
--- /dev/null
+++ b/doc/development/documentation/site_architecture/deployment.md
@@ -0,0 +1,16 @@
+# GitLab Docs deployment process
+
+The docs site is deployed to production with [GitLab Pages](../../../user/project/pages/index.md)
+and previewed in merge requests with Review Apps. By default, we pull from the
+master branch of [all the projects](index.md#repositories).
+
+The [scheduled pipelines](../../../user/project/pipelines/schedules.md)
+are used to build the site once an hour.
+
+For more information, see [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/.gitlab-ci.yml).
+
+<-- TODO
+
+Document Rakefile and .gitlab-ci.yml.
+
+-->
diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md
index 1aef0ed855c..ded5cd0b40e 100644
--- a/doc/development/documentation/site_architecture/index.md
+++ b/doc/development/documentation/site_architecture/index.md
@@ -4,14 +4,16 @@ description: "Learn how GitLab's documentation website is architectured."
 
 # Documentation site architecture
 
-Learn how we build and architecture [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs)
-and deploy it to <https://docs.gitlab.com>.
+The [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) project hosts
+the repository which is used to generate the GitLab documentation website and
+is deployed to <https://docs.gitlab.com>. It uses the [Nanoc](http://nanoc.ws)
+static site generator.
 
-## Repository
+## Architecture
 
 While the source of the documentation content is stored in GitLab's respective product
-repositories, the source that is used to build the documentation site _from that content_
-is located at <https://gitlab.com/gitlab-org/gitlab-docs>.
+[repositories](#repositories), the source that is used to build the documentation
+site _from that content_ is located at <https://gitlab.com/gitlab-org/gitlab-docs>.
 
 The following diagram illustrates the relationship between the repositories
 from where content is sourced, the `gitlab-docs` project, and the published output.
@@ -43,9 +45,6 @@ from where content is sourced, the `gitlab-docs` project, and the published outp
     G --> L
 ```
 
-See the [README there](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/README.md)
-for detailed information.
-
 ## Assets
 
 To provide an optimized site structure, design, and a search-engine friendly
@@ -71,28 +70,133 @@ the GitLab Documentation website.
 To understand how the global nav (left sidebar) is built, please
 read through the [global navigation](global_nav.md) doc.
 
-## Deployment
-
-The docs site is deployed to production with GitLab Pages, and previewed in
-merge requests with Review Apps.
+<!--
+## Helpers
 
-The deployment aspects will be soon transferred from the [original document](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/README.md)
-to this page.
+TBA
+-->
 
-<!--
 ## Repositories
 
-TBA
+You will not find any GitLab docs content in the `gitlab-docs` repository.
+All documentation files are hosted in the respective repository of each
+product, and all together are pulled to generate the docs website:
 
-## Search engine
+- [GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab-ee)
+- [GitLab Community Edition](https://gitlab.com/gitlab-org/gitlab-ce)
+- [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab)
+- [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner)
+- [GitLab Chart](https://gitlab.com/charts/gitlab)
 
-TBA
+NOTE: **Note:**
+Although GitLab Community Edition is generated, it is hidden from the website
+as it's a subset of the Enterprise Edition. We generate it for consistency,
+until the [GitLab codebase is unified](https://gitlab.com/gitlab-org/gitlab-ee/issues/2952).
+
+## Using YAML data files
+
+The easiest way to achieve something similar to
+[Jekyll's data files](https://jekyllrb.com/docs/datafiles/) in Nanoc is by
+using the [`@items`](https://nanoc.ws/doc/reference/variables/#items-and-layouts)
+variable.
+
+The data file must be placed inside the `content/` directory and then it can
+be referenced in an ERB template.
+
+Suppose we have the `content/_data/versions.yaml` file with the content:
+
+```yaml
+versions:
+- 10.6
+- 10.5
+- 10.4
+```
+
+We can then loop over the `versions` array with something like:
+
+```erb
+<% @items['/_data/versions.yaml'][:versions].each do | version | %>
+
+<h3><%= version %></h3>
+
+<% end &>
+```
+
+Note that the data file must have the `yaml` extension (not `yml`) and that
+we reference the array with a symbol (`:versions`).
+
+## Bumping versions of CSS and Javascript
+
+Whenever the custom CSS and Javascript files under `content/assets/` change,
+make sure to bump their version in the frontmatter. This method guarantees that
+your changes will take effect by clearing the cache of previous files.
+
+Always use Nanoc's way of including those files, do not hardcode them in the
+layouts. For example use:
+
+```erb
+<script async type="application/javascript" src="<%= @items['/assets/javascripts/badges.*'].path %>"></script>
+
+<link rel="stylesheet" href="<%= @items['/assets/stylesheets/toc.*'].path %>">
+```
+
+The links pointing to the files should be similar to:
+
+```erb
+<%= @items['/path/to/assets/file.*'].path %>
+```
+
+Nanoc will then build and render those links correctly according with what's
+defined in [`Rules`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/Rules).
+
+## Linking to source files
+
+A helper called [`edit_on_gitlab`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/lib/helpers/edit_on_gitlab.rb) can be used
+to link to a page's source file. We can link to both the simple editor and the
+web IDE. Here's how you can use it in a Nanoc layout:
+
+- Default editor: `<a href="<%= edit_on_gitlab(@item, editor: :simple) %>">Simple editor</a>`
+- Web IDE: `<a href="<%= edit_on_gitlab(@item, editor: :webide) %>">Web IDE</a>`
+
+If you don't specify `editor:`, the simple one is used by default.
+
+## Algolia search engine
+
+The docs site uses [Algolia docsearch](https://community.algolia.com/docsearch/)
+for its search function. This is how it works:
+
+1. GitLab is a member of the [docsearch program](https://community.algolia.com/docsearch/#join-docsearch-program),
+   which is the free tier of [Algolia](https://www.algolia.com/).
+1. Algolia hosts a [doscsearch config](https://github.com/algolia/docsearch-configs/blob/master/configs/gitlab.json)
+   for the GitLab docs site, and we've worked together to refine it.
+1. That [config](https://community.algolia.com/docsearch/config-file.html) is
+   parsed by their [crawler](https://community.algolia.com/docsearch/crawler-overview.html)
+   every 24h and [stores](https://community.algolia.com/docsearch/inside-the-engine.html)
+   the [docsearch index](https://community.algolia.com/docsearch/how-do-we-build-an-index.html)
+   on [Algolia's servers](https://community.algolia.com/docsearch/faq.html#where-is-my-data-hosted%3F).
+1. On the docs side, we use a [docsearch layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/docsearch.html) which
+   is present on pretty much every page except <https://docs.gitlab.com/search/>,
+   which uses its [own layout](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/layouts/instantsearch.html). In those layouts,
+   there's a javascript snippet which initiates docsearch by using an API key
+   and an index name (`gitlab`) that are needed for Algolia to show the results.
+
+NOTE: **For GitLab employees:**
+The credentials to access the Algolia dashboard are stored in 1Password. If you
+want to receive weekly reports of the search usage, search the Google doc with
+title "Email, Slack, and GitLab Groups and Aliases", search for `docsearch`,
+and add a comment with your email to be added to the alias that gets the weekly
+reports.
+
+## Deployment
+
+Read about the [deployment process](deployment.md).
 
 ## Versions
 
-TBA
+The docs website supports versions. For more information, read about
+[versions](versions.md).
 
-## Helpers
+## Review Apps for documentation merge requests
 
-TBA
--->
+If you are contributing to GitLab docs read how to [create a Review App with each
+merge request](../index.md#previewing-the-changes-live).
diff --git a/doc/development/documentation/site_architecture/versions.md b/doc/development/documentation/site_architecture/versions.md
new file mode 100644
index 00000000000..b5e9764f99f
--- /dev/null
+++ b/doc/development/documentation/site_architecture/versions.md
@@ -0,0 +1,175 @@
+# How the website versions are built
+
+This directory contains all needed Dockerfiles to build and deploy the
+versioned website. It is heavily inspired by Docker's
+[publish tools](https://github.com/docker/docker.github.io/tree/publish-tools).
+
+The following Dockerfiles are used.
+
+| Dockerfile | Docker image | Description |
+| ---------- | ------------ | ----------- |
+| [`Dockerfile.bootstrap`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.bootstrap) | `gitlab-docs:bootstrap` | Contains all the dependencies that are needed to build the website. If the gems are updated and `Gemfile{,.lock}` changes, the image must be rebuilt. |
+| [`Dockerfile.builder.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.builder.onbuild) | `gitlab-docs:builder-onbuild` | Base image to build the docs website. It uses `ONBUILD` to perform all steps and depends on `gitlab-docs:bootstrap`. |
+| [`Dockerfile.nginx.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.nginx.onbuild) | `gitlab-docs:nginx-onbuild` | Base image to use for building documentation archives. It uses `ONBUILD` to perform all required steps to copy the archive, and relies upon its parent `Dockerfile.builder.onbuild` that is invoked when building single documentation achives (see the `Dockerfile` of each branch. |
+| [`Dockerfile.archives`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/dockerfiles/Dockerfile.archives) | `gitlab-docs:archives` | Contains all the versions of the website in one archive. It copies all generated HTML files from every version in one location. |
+
+## How to build the images
+
+You can build and tag all tooling images locally:
+
+1. Make sure you're on the `dockerfiles/` directory of the `gitlab-docs` repo.
+1. Build the images:
+
+   ```sh
+   docker build -t registry.gitlab.com/gitlab-org/gitlab-docs:bootstrap -f Dockerfile.bootstrap ../
+   docker build -t registry.gitlab.com/gitlab-org/gitlab-docs:builder-onbuild -f Dockerfile.builder.onbuild ../
+   docker build -t registry.gitlab.com/gitlab-org/gitlab-docs:nginx-onbuild -f Dockerfile.nginx.onbuild ../
+   ```
+
+For each image, there's a manual job under the `images` stage in
+[`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/.gitlab-ci.yml) which can be invoked at will.
+
+## Release process
+
+When a new GitLab version is released on the 22nd, we need to create the respective
+single Docker image, and update some files.
+
+### 1. Add the chart version
+
+Since the charts use a different version number than all the other GitLab
+products, we need to add the new
+[version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html):
+
+1. Make sure you're on the root path of the `gitlab-docs` repo.
+1. Open `content/_data/chart_versions.yaml` and add the new version using the
+   [version mapping](https://docs.gitlab.com/charts/installation/version_mappings.html).
+   Note that only the `major.minor` version is needed.
+1. Create a new merge request and merge it.
+
+TIP: **Tip:**
+It can be handy to create the future mappings since they are pretty much known.
+In that case, when a new GitLab version is released, you don't have to repeat
+this first step.
+
+### 2. Create an image for a single version
+
+1. Make sure you're on the root path of the `gitlab-docs` repo.
+1. Run the raketask to create the single version:
+
+    ```
+    bundle exec rake "release:single[12.0]"
+    ```
+
+    A new `Dockerfile.12.0` should have been created.
+
+1. Test locally by building the image and running it:
+
+    ```
+    docker build -t docs:12.0 -f Dockerfile.12.0 .
+    docker run -it --rm -p 4000:4000 docs:12.0
+    ```
+
+1. Visit `http://localhost:4000/12.0/` and make sure everything works correctly.
+1. Commit your changes and push the newly created branch, but **don't create a merge request**.
+
+Once you push, the `image:docker-singe` job will create a new Docker image
+tagged with the branch name you created in the first step.
+
+### 3. Create the release merge request
+
+1. Make sure you're on the root path of the `gitlab-docs` repo.
+1. Create a branch `release-X-Y`.
+1. **Rotate the online and offline versions:**
+   At any given time, there are 4 browsable online versions: one pulled from
+   the upstream master branches and the three latest stable versions.
+
+   Edit [`content/_data/versions.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/master/content/_data/versions.yaml) and rotate
+   the versions to reflect the new changes:
+
+   - `online`: The 3 latest stable versions.
+   - `offline`: All the previous versions not available online.
+
+1. **Add the new offline version in the 404 page redirect script:**
+
+   Since we're deprecating the oldest version each month, we need to redirect
+   those URLs in order not to create [404 entries](https://gitlab.com/gitlab-org/gitlab-docs/issues/221).
+   There's a temporary hack for now:
+
+   1. Edit `content/404.html`, making sure all offline versions under
+      `content/_data/versions.yaml` are in the javascript snippet at the end of
+      the document.
+
+1. **Update the `latest` and `archives` images:**
+
+   The following two Dockerfiles need to be updated:
+
+   1. `Dockerfile.archives` - Add the latest version to the
+      list.
+   1. `Dockerfile.master` - Rotate the versions (oldest
+      gets removed and latest is added at the end of the list).
+
+1. Commit and push to create the merge request using the "Release" template.
+
+### 4. Update the dropdown for all online versions
+
+The versions dropdown is in a way "hardcoded". When the site is built, it looks
+at the contents of `content/_data/versions.yaml` and based on that, the dropdown
+is populated. So, older branches will have different content, which means the
+dropdown will be one release behind. Remember that the new changes of the dropdown
+are included in the unmerged `release-X-Y` branch.
+
+The content of `content/_data/versions.yaml` needs to change for all online
+versions. Here's a script to do this in one go:
+
+```sh
+# Change this to reflect the latest stable version
+CURRENT_VERSION="12.2"
+CURRENT_BRANCH="12-2"
+
+# Change this to the latest 3 stable versions
+for branch in 12.2 12.1 12.0
+do
+  git checkout ${branch}
+  git checkout release-${CURRENT_BRANCH} -- content/_data/versions.yaml
+  git commit -m "Update dropdown for ${branch} to ${CURRENT_VESION}"
+  git push origin ${branch}
+done
+```
+
+Replace `CURRENT_VERSION` and `CURRENT_BRANCH` with the current values.
+
+### 5. Merge the release merge request
+
+After all the pipelines of the online versions succeed, it's time to merge the
+release merge request.
+
+Once you merge, you may need to [run the scheduled pipeline](https://gitlab.com/gitlab-org/gitlab-docs/pipeline_schedules)
+(press the play button), since this is built on a schedule, once an hour.
+
+Once the scheduled pipeline succeeds, the docs site will be deployed with all
+new versions online.
+
+## Update an old image with new upstream content
+
+If there are upstream changes not included in the single Docker image, just
+[rerun the pipeline](https://gitlab.com/gitlab-org/gitlab-docs/pipelines/new)
+for the branch in question.
+
+## Porting new website changes to old versions
+
+CAUTION: **Warning:**
+Porting changes to older branches can have unintended effects as we're constantly
+changing the backend of the website. Use only when you know what you're doing
+and make sure to test locally.
+
+The website will keep changing and being improved. In order to consolidate
+those changes to the stable branches, we'd need to pick certain changes
+from time to time.
+
+If this is not possible or there are many changes, merge master into them:
+
+```sh
+git branch 12.0
+git fetch origin master
+git merge origin/master
+```