diff options
Diffstat (limited to 'doc')
| -rw-r--r-- | doc/README.md | 2 | ||||
| -rw-r--r-- | doc/topics/autodevops/index.md | 2 | ||||
| -rw-r--r-- | doc/user/project/clusters/add_remove_clusters.md | 202 | ||||
| -rw-r--r-- | doc/user/project/clusters/runbooks/index.md | 4 | ||||
| -rw-r--r-- | doc/user/project/clusters/serverless/index.md | 93 |
5 files changed, 267 insertions, 36 deletions
diff --git a/doc/README.md b/doc/README.md index ac64d8df83d..af573a3eb34 100644 --- a/doc/README.md +++ b/doc/README.md @@ -305,7 +305,7 @@ The following documentation relates to the DevOps **Configure** stage: | Configure Topics | Description | |:-----------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------| | [Auto DevOps](topics/autodevops/index.md) | Automatically employ a complete DevOps lifecycle. | -| [Create Kubernetes clusters on GKE](user/project/clusters/add_remove_clusters.md#add-new-gke-cluster) | Use Google Kubernetes Engine and GitLab. | +| [Create Kubernetes clusters](user/project/clusters/add_remove_clusters.md#add-new-cluster) | Use Kubernetes and GitLab. | | [Executable Runbooks](user/project/clusters/runbooks/index.md) | Documented procedures that explain how to carry out particular processes. | | [GitLab ChatOps](ci/chatops/README.md) | Interact with CI/CD jobs through chat services. | | [Installing Applications](user/project/clusters/index.md#installing-applications) | Deploy Helm, Ingress, and Prometheus on Kubernetes. | diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 0aa0d18bf77..93549ac4de5 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -100,7 +100,7 @@ To make full use of Auto DevOps, you will need: To enable deployments, you will need: 1. A [Kubernetes 1.12+ cluster](../../user/project/clusters/index.md) for the project. The easiest - way is to add a [new GKE cluster using the GitLab UI](../../user/project/clusters/add_remove_clusters.md#add-new-gke-cluster). + way is to add a [new cluster using the GitLab UI](../../user/project/clusters/add_remove_clusters.md#add-new-cluster). 1. NGINX Ingress. You can deploy it to your Kubernetes cluster by installing the [GitLab-managed app for Ingress](../../user/clusters/applications.md#ingress), once you have configured GitLab's Kubernetes integration in the previous step. diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index 549dd4dd916..150a451dfe5 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -5,9 +5,6 @@ GitLab can integrate with the following Kubernetes providers: - Google Kubernetes Engine (GKE). - Amazon Elastic Kubernetes Service (EKS). -GitLab is more deeply integrated with GKE, but deeper integration with EKS -[is planned](https://gitlab.com/gitlab-org/gitlab/issues/22392). - TIP: **Tip:** Every new Google Cloud Platform (GCP) account receives [$300 in credit upon sign up](https://console.cloud.google.com/freetrial), and in partnership with Google, GitLab is able to offer an additional $200 for new GCP accounts to get started with GitLab's @@ -62,17 +59,17 @@ Note the following about access controls: GitLab creates the following resources for RBAC clusters. -| Name | Type | Details | Created when | -|:----------------------|:---------------------|:-----------------------------------------------------------------------------------------------------------|:---------------------------| -| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | -| `gitlab-admin` | `ClusterRoleBinding` | [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Creating a new GKE Cluster | -| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | -| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | -| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | -| Environment namespace | `Namespace` | Contains all environment-specific resources | Deploying to a cluster | -| Environment namespace | `ServiceAccount` | Uses namespace of environment | Deploying to a cluster | -| Environment namespace | `Secret` | Token for environment ServiceAccount | Deploying to a cluster | -| Environment namespace | `RoleBinding` | [`edit`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Deploying to a cluster | +| Name | Type | Details | Created when | +|:----------------------|:---------------------|:-----------------------------------------------------------------------------------------------------------|:-----------------------| +| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new cluster | +| `gitlab-admin` | `ClusterRoleBinding` | [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Creating a new cluster | +| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new cluster | +| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | +| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | +| Environment namespace | `Namespace` | Contains all environment-specific resources | Deploying to a cluster | +| Environment namespace | `ServiceAccount` | Uses namespace of environment | Deploying to a cluster | +| Environment namespace | `Secret` | Token for environment ServiceAccount | Deploying to a cluster | +| Environment namespace | `RoleBinding` | [`edit`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Deploying to a cluster | ### ABAC cluster resources @@ -80,13 +77,13 @@ GitLab creates the following resources for ABAC clusters. | Name | Type | Details | Created when | |:----------------------|:---------------------|:-------------------------------------|:---------------------------| -| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | -| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | -| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | -| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | -| Environment namespace | `Namespace` | Contains all environment-specific resources | Deploying to a cluster | -| Environment namespace | `ServiceAccount` | Uses namespace of environment | Deploying to a cluster | -| Environment namespace | `Secret` | Token for environment ServiceAccount | Deploying to a cluster | +| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new cluster | +| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new cluster | +| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | +| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | +| Environment namespace | `Namespace` | Contains all environment-specific resources | Deploying to a cluster | +| Environment namespace | `ServiceAccount` | Uses namespace of environment | Deploying to a cluster | +| Environment namespace | `Secret` | Token for environment ServiceAccount | Deploying to a cluster | ### Security of GitLab Runners @@ -112,7 +109,9 @@ If you don't want to use GitLab Runner in privileged mode, either: 1. Installing a Runner [using `docker+machine`](https://docs.gitlab.com/runner/executors/docker_machine.html). -## Add new GKE cluster +## Add new cluster + +### GKE cluster GitLab supports: @@ -126,7 +125,7 @@ The [Google authentication integration](../../../integration/google.md) must be enabled in GitLab at the instance level. If that's not the case, ask your GitLab administrator to enable it. On GitLab.com, this is enabled. -### Requirements +#### GKE Requirements Before creating your first cluster on Google Kubernetes Engine with GitLab's integration, make sure the following requirements are met: @@ -151,7 +150,7 @@ order to setup an [initial service account](#access-controls). Starting from [Gi 11.10](https://gitlab.com/gitlab-org/gitlab-foss/issues/58208), the cluster creation process will explicitly request that basic authentication and client certificate is enabled. -### Creating the cluster +#### Creating the cluster on GKE If all of the above requirements are met, you can proceed to create and add a new Kubernetes cluster to your project: @@ -185,7 +184,7 @@ new Kubernetes cluster to your project: After a couple of minutes, your cluster will be ready to go. You can now proceed to install some [pre-defined applications](index.md#installing-applications). -### Cloud Run on GKE +#### Cloud Run on GKE > [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/16566) in GitLab 12.4. @@ -194,6 +193,159 @@ separately after the cluster has been created. This means that Cloud Run (Knative), Istio, and HTTP Load Balancing will be enabled on the cluster at create time and cannot be [installed or uninstalled](../../clusters/applications.md) separately. +### EKS Cluster + +GitLab supports: + +- Creating a new EKS cluster using the GitLab UI + ([Introduced](https://gitlab.com/gitlab-org/gitlab/issues/22392) in GitLab 12.5). +- Providing credentials to add an [existing Kubernetes cluster](#add-existing-cluster). + +#### EKS Requirements + +Before creating your first cluster on Amazon EKS with GitLab's integration, +make sure the following requirements are met: + +- An [Amazon Web Services](https://aws.amazon.com/) account is set up and you are able to log in. +- You have permissions to manage IAM resources. + +##### Additional requirements for self-managed instances + +If you are using a self-managed GitLab instance, GitLab must first +be configured with a set of Amazon credentials. These credentials +will be used to assume an Amazon IAM role provided by the user +creating the cluster. Create an IAM user and ensure it has permissions +to assume the role(s) that your users will use to create EKS clusters. + +For example, the following policy document allows assuming a role whose name starts with +`gitlab-eks-` in account `123456789012`: + +```json +{ + "Version": "2012-10-17", + "Statement": { + "Effect": "Allow", + "Action": "sts:AssumeRole", + "Resource": "arn:aws:iam::123456789012:role/gitlab-eks-*" + } +} +``` + +Generate an access key for the IAM user, and configure GitLab with the credentials: + +1. Navigate to **Admin Area > Settings > Integrations** and expand the **Amazon EKS** section. +1. Check **Enable Amazon EKS integration**. +1. Enter the account ID and access key credentials into the respective + `Account ID`, `Access key ID` and `Secret access key` fields. +1. Click **Save changes**. + +#### Creating the cluster on EKS + +If all of the above requirements are met, you can proceed to create and add a +new Kubernetes cluster to your project: + +1. Navigate to your project's **Operations > Kubernetes** page. + + NOTE: **Note:** + You need Maintainer [permissions](../../permissions.md) and above to access the Kubernetes page. + +1. Click **Add Kubernetes cluster**. +1. Click **Amazon EKS**. You will be provided with an `Account ID` and `External ID` to use in the next step. +1. In the [IAM Management Console](https://console.aws.amazon.com/iam/home), create an IAM role: + 1. From the left panel, select **Roles**. + 1. Click **Create role**. + 1. Under `Select type of trusted entity`, select **Another AWS account**. + 1. Enter the Account ID from GitLab into the `Account ID` field. + 1. Check **Require external ID**. + 1. Enter the External ID from GitLab into the `External ID` field. + 1. Click **Next: Permissions**. + 1. Click **Create Policy**, which will open a new window. + 1. Select the **JSON** tab, and paste in the following snippet in place of the existing content: + + ```json + { + "Version": "2012-10-17", + "Statement": [ + { + "Effect": "Allow", + "Action": [ + "autoscaling:CreateAutoScalingGroup", + "autoscaling:DescribeAutoScalingGroups", + "autoscaling:DescribeScalingActivities", + "autoscaling:UpdateAutoScalingGroup", + "autoscaling:CreateLaunchConfiguration", + "autoscaling:DescribeLaunchConfigurations", + "cloudformation:CreateStack", + "cloudformation:DescribeStacks", + "ec2:AuthorizeSecurityGroupEgress", + "ec2:AuthorizeSecurityGroupIngress", + "ec2:RevokeSecurityGroupEgress", + "ec2:RevokeSecurityGroupIngress", + "ec2:CreateSecurityGroup", + "ec2:createTags", + "ec2:DescribeImages", + "ec2:DescribeKeyPairs", + "ec2:DescribeRegions", + "ec2:DescribeSecurityGroups", + "ec2:DescribeSubnets", + "ec2:DescribeVpcs", + "eks:CreateCluster", + "eks:DescribeCluster", + "iam:AddRoleToInstanceProfile", + "iam:AttachRolePolicy", + "iam:CreateRole", + "iam:CreateInstanceProfile", + "iam:GetRole", + "iam:ListRoles", + "iam:PassRole", + "ssm:GetParameters" + ], + "Resource": "*" + } + ] + } + ``` + + NOTE: **Note:** + These permissions give GitLab the ability to create resources, but not delete them. + This means that if an error is encountered during the creation process, changes will + not be rolled back and you must remove resources manually. You can do this by deleting + the relevant [CloudFormation stack](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-console-delete-stack.html) + + 1. Click **Review policy**. + 1. Enter a suitable name for this policy, and click **Create Policy**. You can now close this window. + 1. Switch back to the "Create role" window, and select the policy you just created. + 1. Click **Next: Tags**, and optionally enter any tags you wish to associate with this role. + 1. Click **Next: Review**. + 1. Enter a role name and optional description into the fields provided. + 1. Click **Create role**, the new role name will appear at the top. Click on its name and copy the `Role ARN` from the newly created role. +1. In GitLab, enter the copied role ARN into the `Role ARN` field. +1. Click **Authenticate with AWS**. +1. Choose your cluster's settings: + - **Kubernetes cluster name** - The name you wish to give the cluster. + - **Environment scope** - The [associated environment](index.md#setting-the-environment-scope-premium) to this cluster. + - **Kubernetes version** - The Kubernetes version to use. Currently the only version supported is 1.14. + - **Role name** - Select the [IAM role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) + to allow Amazon EKS and the Kubernetes control plane to manage AWS resources on your behalf. + - **Region** - The [region](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html) + in which the cluster will be created. + - **Key pair name** - Select the [key pair](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) + that you can use to connect to your worker nodes if required. + - **VPC** - Select a [VPC](https://docs.aws.amazon.com/vpc/latest/userguide/what-is-amazon-vpc.html) + to use for your EKS Cluster resources. + - **Subnets** - Choose the [subnets](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_Subnets.html) + in your VPC where your worker nodes will run. + - **Security group** - Choose the [security group](https://docs.aws.amazon.com/vpc/latest/userguide/VPC_SecurityGroups.html) + to apply to the EKS-managed Elastic Network Interfaces that are created in your worker node subnets. + - **Instance type** - The [instance type](https://aws.amazon.com/ec2/instance-types/) of your worker nodes. + - **Node count** - The number of worker nodes. + - **GitLab-managed cluster** - Leave this checked if you want GitLab to manage namespaces and service accounts for this cluster. + See the [Managed clusters section](index.md#gitlab-managed-clusters) for more information. +1. Finally, click the **Create Kubernetes cluster** button. + +After about 10 minutes, your cluster will be ready to go. You can now proceed +to install some [pre-defined applications](index.md#installing-applications). + ## Add existing cluster If you have either of the following types of clusters already, you can add them to a project: diff --git a/doc/user/project/clusters/runbooks/index.md b/doc/user/project/clusters/runbooks/index.md index 5e8ae5b0595..bffae4c5069 100644 --- a/doc/user/project/clusters/runbooks/index.md +++ b/doc/user/project/clusters/runbooks/index.md @@ -35,7 +35,7 @@ for an overview of how this is accomplished in GitLab!** To create an executable runbook, you will need: 1. **Kubernetes** - A Kubernetes cluster is required to deploy the rest of the applications. - The simplest way to get started is to add a cluster using [GitLab's GKE integration](../add_remove_clusters.md#add-new-gke-cluster). + The simplest way to get started is to add a cluster using one of [GitLab's integrations](../add_remove_clusters.md#add-new-cluster). 1. **Helm Tiller** - Helm is a package manager for Kubernetes and is required to install all the other applications. It is installed in its own pod inside the cluster which can run the Helm CLI in a safe environment. @@ -60,7 +60,7 @@ the components outlined above and the preloaded demo runbook. ### 1. Add a Kubernetes cluster -Follow the steps outlined in [Add new GKE cluster](../add_remove_clusters.md#add-new-gke-cluster) +Follow the steps outlined in [Add new cluster](../add_remove_clusters.md#add-new-cluster) to add a Kubernetes cluster to your project. ### 2. Install Helm Tiller, Ingress, and JupyterHub diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index 26db2133d09..ffd7b0c0f2a 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -41,7 +41,7 @@ To run Knative on GitLab, you will need: - If you are planning on deploying a serverless application, clone the sample [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) to get started. 1. **Kubernetes Cluster:** An RBAC-enabled Kubernetes cluster is required to deploy Knative. - The simplest way to get started is to add a cluster using [GitLab's GKE integration](../add_remove_clusters.md#add-new-gke-cluster). + The simplest way to get started is to add a cluster using [GitLab's GKE integration](../add_remove_clusters.md#gke-cluster). The set of minimum recommended cluster specifications to run Knative is 3 nodes, 6 vCPUs, and 22.50 GB memory. 1. **Helm Tiller:** Helm is a package manager for Kubernetes and is required to install Knative. @@ -64,6 +64,8 @@ To run Knative on GitLab, you will need: using our [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes). 1. **Prometheus** (optional): Installing Prometheus allows you to monitor the scale and traffic of your serverless function/application. See [Installing Applications](../index.md#installing-applications) for more information. +1. **Logging** (optional): Configuring logging allows you to view and search request logs for your serverless function/application. + See [Configuring logging](#configuring-logging) for more information. ## Installing Knative via GitLab's Kubernetes integration @@ -166,13 +168,61 @@ You must do the following: or [serverless applications](#deploying-serverless-applications) onto your cluster. -## Deploying functions +## Configuring logging -> Introduced in GitLab 11.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/33330) in GitLab 12.5. + +### Prerequisites + +- A GitLab-managed cluster. +- `kubectl` installed and working. + +Running `kubectl` commands on your cluster requires setting up access to the +cluster first. For clusters created on: + +- GKE, see [GKE Cluster Access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl) +- Other platforms, see [Install and Set Up kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/). + +### Enable request log template + +Run the following command to enable request logs: + +```shell +kubectl edit cm -n knative-serving config-observability +``` + +Copy the `logging.request-log-template` from the `data._example` field to the data field one level up in the hierarchy. + +### Enable request logs + +Run the following commands to install Elasticsearch, Kibana, and Filebeat into a `kube-logging` namespace and configure all nodes to forward logs using Filebeat: + +```shell +kubectl apply -f https://gitlab.com/gitlab-org/serverless/configurations/knative/raw/v0.7.0/kube-logging-filebeat.yaml +kubectl label nodes --all beta.kubernetes.io/filebeat-ready="true" +``` + +### Viewing request logs + +To view request logs: + +1. Run `kubectl proxy`. +1. Navigate to Kibana UI. + +Or: -Using functions is useful for dealing with independent events without needing -to maintain a complex unified infrastructure. This allows you to focus on a -single task that can be executed/scaled automatically and independently. +1. Open the Kibana UI. +1. Click on **Discover**, then select `filebeat-*` from the dropdown on the left. +1. Enter `kubernetes.container.name:"queue-proxy" AND message:/httpRequest/` into the search box. + +## Supported runtimes + +Serverless functions for GitLab can be written in 6 supported languages: + +- NodeJS and Ruby, with GitLab-managed and OpenFaas runtimes. +- C#, Go, PHP, and Python with OpenFaaS runtimes only. + +### GitLab managed runtimes Currently the following [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes) are offered: @@ -182,6 +232,31 @@ Currently the following [runtimes](https://gitlab.com/gitlab-org/serverless/runt `Dockerfile` presence is assumed when a runtime is not specified. +### OpenFaaS runtimes + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/29253) in GitLab 12.5. + +[OpenFaaS classic runtimes](https://github.com/openfaas/templates#templates-in-store) can be used with GitLab serverless. +Runtimes are specified using the pattern: `openfaas/classic/<template_name>`. The following +example shows how to define a function in `serverless.yml` using an OpenFaaS runtime: + +```yaml +hello: + source: ./hello + runtime: openfaas/classic/ruby + description: "Ruby function using OpenFaaS classic runtime" +``` + +`handler` is not needed for OpenFaaS functions. The location of the handler is defined +by the conventions of the runtime. + +See the [`ruby-openfaas-function`](https://gitlab.com/knative-examples/ruby-openfaas-function) +project for an example of a function using an OpenFaaS runtime. + +## Deploying functions + +> Introduced in GitLab 11.6. + You can find and import all the files referenced in this doc in the **[functions example project](https://gitlab.com/knative-examples/functions)**. @@ -322,7 +397,7 @@ Running functions locally requires: - Go 1.12 or newer installed. - Docker Engine installed and running. - `gitlabktl` installed using the Go package manager: - + ```shell GO111MODULE=on go get gitlab.com/gitlab-org/gitlabktl ``` @@ -352,6 +427,10 @@ To run a function locally: > Introduced in GitLab 11.5. +Serverless applications are the building block of serverless functions. They are useful in scenarios where an existing +runtime does not meet the needs of an application, such as one written in a language that has no runtime available. Note +though that serverless applications should be stateless! + NOTE: **Note:** You can reference and import the sample [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) to get started. |