Finish rest of the sections

1 files changed, 147 insertions, 56 deletions

diff --git a/doc/articles/runner_autoscale_aws/index.md b/doc/articles/runner_autoscale_aws/index.md
index 14fcb5b7072..d7953e6be69 100644
--- a/doc/articles/runner_autoscale_aws/index.md
+++ b/doc/articles/runner_autoscale_aws/index.md
@@ -22,8 +22,37 @@ quite powerful autoscaling machines.
 
 ## Prerequisites
 
+Your GitLab instance is going to need to talk to the Runners over the network,
+so consider the sensitivity of your projects and communication between nodes
+when moving forward. This has implications on the AWS security groups you'll
+need, your DNS configuration, and other networking factors.
+
+For example, you can keep the EC2 resources segmented away from public traffic
+in a different VPC. Your environment is likely different, so consider what works
+best for your situation.
+
+### AWS security groups
+
+Docker Machine will attempt to use a
+[default security group](https://docs.docker.com/machine/drivers/aws/#security-group)
+with rules for port 2376, which is required for communication with the Docker
+daemon. Instead of relying on Docker, you can create a security group with the
+rules you need and provide that in the Runner options as we will see below.
+This way, you can customize it to your liking ahead of time based on your
+networking environment.
+
+### AWS credentials
+
+You'll need an AWS Access Key tied to a user with permission to scale (EC2) and
+update the cache (via S3). Create a new user with policies for EC2
+(AmazonEC2FullAccess) and S3 (AmazonS3FullAccess). To be more secure, you can
+disable console login for that user. Grab the security credentials as we'll use
+them later during the Runner configuration.
+
+## Prepare the bastion instance
+
 The first step is to install GitLab Runner in an EC2 instance that will serve
-as the bastion to spawning new machines. This doesn't have to be a powerful
+as the bastion that spawns new machines. This doesn't have to be a powerful
 machine since it will not run any jobs itself, a `t2.micro` instance will do.
 This machine will be a dedicated host since we need it always up and running,
 thus it will be the only standard cost.
@@ -34,11 +63,19 @@ Runner support, for example either Ubuntu, Debian, CentOS or RHEL will work fine
 
 Install the prerequisites:
 
-1. Log in your server
+1. Log in to your server
 1. [Install GitLab Runner from the official GitLab repository](https://docs.gitlab.com/runner/install/linux-repository.html)
 1. [Install Docker](https://docs.docker.com/engine/installation/#server)
 1. [Install Docker Machine](https://docs.docker.com/machine/install-machine/)
 
+Now that the Runner is installed, it's time to register it.
+
+## Registering GitLab Runner
+
+TIP: **Tip:**
+If you want every user in your instance to be able to use the autoscaled Runners,
+register the Runner as a shared one.
+
 Before configuring the GitLab Runner, you need to first register it, so that
 it connects with your GitLab instance:
 
@@ -46,10 +83,6 @@ it connects with your GitLab instance:
 1. [Register the Runner](https://docs.gitlab.com/runner/register/index.html#gnu-linux)
 1. When asked the executor type, enter `docker+machine`
 
-TIP: **Tip:**
-If you want every user in your instance to be able to use the autoscaled Runners,
-register the Runner as a shared one.
-
 You can now move on to the most important part, configuring GitLab Runner.
 
 ## Configuring GitLab Runner to use the AWS machine driver
@@ -69,6 +102,7 @@ check_interval = 0
   url = "<URL of your GitLab instance>"
   token = "<Runner's token>"
   executor = "docker+machine"
+  limit = 20
   [runners.docker]
     image = "alpine"
     privileged = true
@@ -85,11 +119,14 @@ check_interval = 0
     IdleCount = 1
     IdleTime = 1800
     MaxBuilds = 100
-    MachineDriver = "amazonec2"
-    MachineName = "gitlab-docker-machine-%s"
-    OffPeakPeriods = ["* * 0-7,19-23 * * mon-fri *", "* * * * * sat,sun *"]
+    OffPeakPeriods = [
+      "* * 0-9,18-23 * * mon-fri *",
+      "* * * * * sat,sun *"
+    ]
     OffPeakIdleCount = 0
     OffPeakIdleTime = 1200
+    MachineDriver = "amazonec2"
+    MachineName = "gitlab-docker-machine-%s"
     MachineOptions = [
       "amazonec2-access-key=XXXX",
       "amazonec2-secret-key=XXXX",
@@ -100,10 +137,6 @@ check_interval = 0
       "amazonec2-tags=Name,gitlab-runner-autoscale",
       "amazonec2-security-group=docker-machine-scaler",
       "amazonec2-instance-type=m4.2xlarge",
-      "amazonec2-ssh-user=ubuntu",
-      "amazonec2-ssh-keypath=/etc/gitlab-runner/certs/gitlab-aws-autoscaler",
-      "amazonec2-zone=a",
-      "amazonec2-root-size=32",
     ]
 ```
 
@@ -111,11 +144,6 @@ Let's break it down to pieces.
 
 ### Global section
 
-```toml
-concurrent = 10
-check_interval = 0
-```
-
 In the global section, you can define the limit of the jobs that can be run
 concurrently across all Runners (`concurrent`). This heavily depends on your
 needs, like how many users your Runners will accommodate, how much time your
@@ -125,10 +153,25 @@ forward.
 The `check_interval` setting defines in seconds how often the Runner should
 check GitLab for new jobs.
 
+Example:
+
+```toml
+concurrent = 10
+check_interval = 0
+```
+
 [Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section)
 about all the options you can use.
 
-### `[[runners]]`
+### `[[runners]]` section
+
+From the `[[runners]]` section, the most important part is the `executor` which
+must be set to `docker+machine`. Most of those settings are taken care of when
+you register the Runner for the first time.
+
+`limit` defines how many jobs can be handled concurrently by this token.
+
+Example:
 
 ```toml
 [[runners]]
@@ -136,23 +179,13 @@ about all the options you can use.
   url = "<URL of your GitLab instance>"
   token = "<Runner's token>"
   executor = "docker+machine"
+  limit = 20
 ```
 
-From the `[[runners]]` section, the most important part is the `executor` which
-must be set to `docker+machine`. All those settings are taken care of when you
-register the Runner for the first time.
-
 [Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section)
 about all the options you can use under `[[runners]]`.
 
-### `[runners.docker]`
-
-```toml
-  [runners.docker]
-    image = "alpine"
-    privileged = true
-    disable_cache = true
-```
+### `[runners.docker]` section
 
 In the `[runners.docker]` section you can define the default Docker image to
 be used by the child Runners if it's not defined in [`.gitlab-ci.yml`](../../ci/yaml/README.md).
@@ -163,10 +196,19 @@ Next, we use `disable_cache = true` to disable the Docker executor's inner
 cache mechanism since we will use the distributed cache mode as described
 below.
 
+Example:
+
+```toml
+  [runners.docker]
+    image = "alpine"
+    privileged = true
+    disable_cache = true
+```
+
 [Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-docker-section)
 about all the options you can use under `[runners.docker]`.
 
-### `[runners.cache]`
+### `[runners.cache]` section
 
 To speed up your jobs, GitLab Runner provides a cache mechanism where selected
 directories and/or files are saved and shared between subsequent jobs.
@@ -193,21 +235,33 @@ Here's some more info to get you started:
 - [Deploying and using a cache server for GitLab Runner](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching)
 - [How cache works](../../ci/yaml/README.md#cache)
 
-### `[runners.machine]`
+### `[runners.machine]` section
 
 This is the most important part of the configuration and it's the one that
-tells GitLab Runner how and when to spawn new Docker Machine instances.
+tells GitLab Runner how and when to spawn new or remove old Docker Machine
+instances.
+
+We will focus on the AWS machine options, for the rest of the settings read
+about the:
+
+- [autoscaling algorithm and the parameters it's based on](https://docs.gitlab.com/runner/configuration/autoscale.html#autoscaling-algorithm-and-parameters) - depends on the needs of your organization
+- [off peak time configuration](https://docs.gitlab.com/runner/configuration/autoscale.html#off-peak-time-mode-configuration) - useful when there are regular time periods in your organization when no work is done, for example weekends
+
+Example:
 
 ```toml
   [runners.machine]
     IdleCount = 1
     IdleTime = 1800
-    MaxBuilds = 100
-    MachineDriver = "amazonec2"
-    MachineName = "gitlab-docker-machine-%s"
-    OffPeakPeriods = ["* * 0-7,19-23 * * mon-fri *", "* * * * * sat,sun *"]
+    MaxBuilds = 10
+    OffPeakPeriods = [
+      "* * 0-9,18-23 * * mon-fri *",
+      "* * * * * sat,sun *"
+    ]
     OffPeakIdleCount = 0
     OffPeakIdleTime = 1200
+    MachineDriver = "amazonec2"
+    MachineName = "gitlab-docker-machine-%s"
     MachineOptions = [
       "amazonec2-access-key=XXXX",
       "amazonec2-secret-key=XXXX",
@@ -218,18 +272,44 @@ tells GitLab Runner how and when to spawn new Docker Machine instances.
       "amazonec2-tags=Name,gitlab-runner-autoscale",
       "amazonec2-security-group=docker-machine-scaler",
       "amazonec2-instance-type=m4.2xlarge",
-      "amazonec2-ssh-user=ubuntu",
-      "amazonec2-ssh-keypath=/etc/gitlab-runner/certs/gitlab-aws-autoscaler",
-      "amazonec2-zone=a",
-      "amazonec2-root-size=32",
     ]
 ```
 
+The Docker Machine driver is set to `amazonec2` and the machine name has a
+standard prefix followed by `%s` (required) that is replaced by the ID of the
+child Runner: `gitlab-docker-machine-%s`.
+
+Now, depending on your AWS infrastructure, there are many options you can set up
+under `MachineOptions`. Let's see the most common ones:
+
+- `amazonec2-access-key=XXXX` - The AWS access key of the user that has permissions to create EC2 instances, see [AWS credentials](#aws-credentials).
+- `amazonec2-secret-key=XXXX` - The AWS secret key of the user that has permissions to create EC2 instances, see [AWS credentials](#aws-credentials).
+- `amazonec2-region=eu-central-1` - The region to use when launching the instance. You can omit this entirely and the default `us-east-1` will be used.
+- `amazonec2-vpc-id=vpc-xxxxx` - Your VPC ID to launch the instance in, read more in [Docker docs about the VPC ID](https://docs.docker.com/machine/drivers/aws/#vpc-id).
+- `amazonec2-subnet-id=subnet-xxxx` - AWS VPC subnet ID.
+- `amazonec2-use-private-address=true` - Use the private IP address for docker-machine, but still create a public IP address. Useful to keep the traffic internal and avoid extra costs.
+- `amazonec2-tags=Name,gitlab-runner-autoscale` - AWS extra tag key-value pairs, useful to identify the instances on the AWS console.
+- `amazonec2-security-group=docker-machine-scaler` - AWS VPC security group name, see [AWS security groups](#aws-security-groups).
+- `amazonec2-instance-type=m4.2xlarge` - The instance type that the child Runners will run on.
+
 TIP: **Tip:**
 Under `MachineOptions` you can add anything that the [AWS Docker Machine driver
-supports](https://docs.docker.com/machine/drivers/aws/#options).
+supports](https://docs.docker.com/machine/drivers/aws/#options). You are highly
+encouraged to read Docker's docs as your infrastructure setup may warrant
+different options to be applied.
+
+NOTE: **Note:**
+The child instances will use by default Ubuntu 16.04 unless you choose a
+different AMI ID by setting `amazonec2-ami`.
+
+NOTE: **Note:**
+If you specify `amazonec2-private-address-only=true` as one of the machine
+options, your EC2 instance won't get assigned a public IP. This is fine if your
+VPC is configured correctly with an Internet Gateway (IGW) and routing is fine,
+but it’s something to consider if you've got a more exotic configuration. Read
+more in [Docker docs about VPC connectivity](https://docs.docker.com/machine/drivers/aws/#vpc-connectivity).
 
-[Read more](/runner/configuration/advanced-configuration.html#the-runners-machine-section)
+[Read more](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-machine-section)
 about all the options you can use under `[runners.machine]`.
 
 ## Cutting down costs with Amazon EC2 Spot instances
@@ -243,7 +323,9 @@ pricing, you can significantly reduce the cost of running your applications,
 grow your application’s compute capacity and throughput for the same budget,
 and enable new types of cloud computing applications.
 
-In `/etc/gitlab-runner/config.toml` under the `MachineOptions` section:
+In addition to the [`[runners.machine]`](#runners-machine-section) options
+you picked above, in `/etc/gitlab-runner/config.toml` under the `MachineOptions`
+section, add the following:
 
 ```toml
     MachineOptions = [
@@ -255,7 +337,8 @@ In `/etc/gitlab-runner/config.toml` under the `MachineOptions` section:
 
 With this configuration, Docker Machines are created on Spot instances with a
 maximum bid price of $0.03 per hour and the duration of the Spot instance is
-capped at 60 minutes.
+capped at 60 minutes. Be sure to check on the current pricing based on the
+region you picked.
 
 To learn more about Amazon EC2 Spot instances, visit the following links:
 
@@ -265,29 +348,37 @@ To learn more about Amazon EC2 Spot instances, visit the following links:
 
 ### Caveats of Spot instances
 
-If the Spot price raises, the auto-scale Runner would fail to create new machines.
+While Spot instances is a great way to use unused resources and minimize the
+costs of your infrastructure, you must be aware of the implications.
+
+Running CI jobs on Spot instances may increase the failure rates because of the
+Spot instances pricing model. If the price exceeds your bid, the existing Spot
+instances will be immediately terminated and all your jobs on that host will fail.
 
-This eventually eats 60 requests and then AWS won't accept any more. Then once
-the spot price is acceptable, you are locked out for a bit because the call amount
-limit is exceeded.
+As a consequence, the auto-scale Runner would fail to create new machines while
+it will continue to request new instances. This eventually will make 60 requests
+and then AWS won't accept any more. Then once the Spot price is acceptable, you
+are locked out for a bit because the call amount limit is exceeded.
 
-You can use the following command in the bastion machine to see the Docker Machines
-state:
+If you encounter that case, you can use the following command in the bastion
+machine to see the Docker Machines state:
 
 ```sh
 docker-machine ls -q --filter state=Error --format "{{.NAME}}"
 ```
 
 NOTE: **Note:**
-Follow [issue 2771](https://gitlab.com/gitlab-org/gitlab-runner/issues/2771)
-for more information.
+In [issue 2771](https://gitlab.com/gitlab-org/gitlab-runner/issues/2771)
+there is a discussion to make GitLab Runner gracefully handle Spot price changes.
 
 ## Conclusion
 
 Using the autoscale feature of GitLab Runner can save you both time and money.
-Using the spot instances that AWS provides can save you even more.
+Using the Spot instances that AWS provides can save you even more, but you must
+be aware of the implications. As long as your bid is high enough, there won't
+be an issue.
 
-You can read the following user cases from which this tutorial was influenced:
+You can read the following use cases from which this tutorial was influenced:
 
 - [HumanGeo - Scaling GitLab CI](http://blog.thehumangeo.com/gitlab-autoscale-runners.html)
 - [subtrakt Health - Autoscale GitLab CI Runners and save 90% on EC2 costs](https://substrakthealth.com/news/gitlab-ci-cost-savings/)