diff options
Diffstat (limited to 'doc/user/project/clusters')
17 files changed, 201 insertions, 573 deletions
diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index c0fb8f5848f..58bdb3d698f 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -74,10 +74,10 @@ Instance profiles dynamically retrieve temporary credentials from AWS when neede To create and add a new Kubernetes cluster to your project, group, or instance: 1. Navigate to your: - - Project's **Operations > Kubernetes** page, for a project-level cluster. + - Project's **Infrastructure > Kubernetes clusters** page, for a project-level cluster. - Group's **Kubernetes** page, for a group-level cluster. - **Admin Area > Kubernetes**, for an instance-level cluster. -1. Click **Add Kubernetes cluster**. +1. Click **Integrate with a cluster certificate**. 1. Under the **Create new cluster** tab, click **Amazon EKS** to display an `Account ID` and `External ID` needed for later steps. 1. In the [IAM Management Console](https://console.aws.amazon.com/iam/home), create an IAM policy: @@ -184,13 +184,10 @@ To create and add a new Kubernetes cluster to your project, group, or instance: See the [Managed clusters section](index.md#gitlab-managed-clusters) for more information. 1. Finally, click the **Create Kubernetes cluster** button. -After about 10 minutes, your cluster is ready to go. You can now proceed -to install some [pre-defined applications](index.md#installing-applications). +After about 10 minutes, your cluster is ready to go. NOTE: -You must add your AWS external ID to the -[IAM Role in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-xaccount) -to manage your cluster using `kubectl`. +If you have [installed and configured](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#get-started-kubectl) `kubectl` and you would like to manage your cluster with it, you must add your AWS external ID in the AWS configuration. For more information on how to configure AWS CLI, see [using an IAM role in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-role.html#cli-configure-role-xaccount). ### Cluster creation flow @@ -292,7 +289,7 @@ you've assigned the role the correct permissions. ### Key Pairs are not loaded -GitLab loads the key pairs from the **Cluster Region** specified. Ensure that key pair exists in that region. +GitLab loads the key pairs from the **Cluster Region** specified. Ensure that key pair exists in that region. #### `ROLLBACK_FAILED` during cluster creation diff --git a/doc/user/project/clusters/add_gke_clusters.md b/doc/user/project/clusters/add_gke_clusters.md index af3a17dc60c..9f0e5603785 100644 --- a/doc/user/project/clusters/add_gke_clusters.md +++ b/doc/user/project/clusters/add_gke_clusters.md @@ -46,10 +46,11 @@ Note the following: To create and add a new Kubernetes cluster to your project, group, or instance: 1. Navigate to your: - - Project's **{cloud-gear}** **Operations > Kubernetes** page, for a project-level cluster. + - Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level + cluster. - Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. - **Admin Area >** **{cloud-gear}** **Kubernetes** page, for an instance-level cluster. -1. Click **Add Kubernetes cluster**. +1. Click **Integrate with a cluster certificate**. 1. Under the **Create new cluster** tab, click **Google GKE**. 1. Connect your Google account if you haven't done already by clicking the **Sign in with Google** button. @@ -70,8 +71,7 @@ To create and add a new Kubernetes cluster to your project, group, or instance: See the [Managed clusters section](index.md#gitlab-managed-clusters) for more information. 1. Finally, click the **Create Kubernetes cluster** button. -After a couple of minutes, your cluster is ready. You can now proceed -to install some [pre-defined applications](index.md#installing-applications). +After a couple of minutes, your cluster is ready. ### Cloud Run for Anthos @@ -79,8 +79,8 @@ to install some [pre-defined applications](index.md#installing-applications). You can choose to use Cloud Run for Anthos in place of installing Knative and Istio separately after the cluster has been created. This means that Cloud Run -(Knative), Istio, and HTTP Load Balancing are enabled on the cluster at -create time and cannot be [installed or uninstalled](../../clusters/applications.md) separately. +(Knative), Istio, and HTTP Load Balancing are enabled on the cluster +from the start, and cannot be installed or uninstalled. ## Existing GKE cluster diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index 1b4b4f38f4b..2ecbc4a2ff5 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -4,7 +4,16 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Adding and removing Kubernetes clusters **(FREE)** +# Add a cluster using cluster certificates **(FREE)** + +> [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/6049) in GitLab 14.0. + +WARNING: +Creating a new cluster or adding an existing cluster to GitLab through the certificate-based method +is deprecated and no longer recommended. Kubernetes cluster, similar to any other +infrastructure, should be created, updated, and maintained using [Infrastructure as Code](../../infrastructure/index.md). +GitLab is developing a built-in capability to create clusters with Terraform. +You can follow along in this [epic](https://gitlab.com/groups/gitlab-org/-/epics/6049). GitLab offers integrated cluster creation for the following Kubernetes providers: @@ -35,9 +44,9 @@ Before [adding a Kubernetes cluster](#create-new-cluster) using GitLab, you need - A [self-managed installation](https://about.gitlab.com/pricing/#self-managed) with GitLab version 12.5 or later. This ensures the GitLab UI can be used for cluster creation. - The following GitLab access: - - [Maintainer access to a project](../../permissions.md#project-members-permissions) for a + - [Maintainer role for a project](../../permissions.md#project-members-permissions) for a project-level cluster. - - [Maintainer access to a group](../../permissions.md#group-members-permissions) for a + - [Maintainer role for a group](../../permissions.md#group-members-permissions) for a group-level cluster. - [Admin Area access](../../admin_area/index.md) for a self-managed instance-level cluster. **(FREE SELF)** @@ -52,16 +61,10 @@ When creating a cluster in GitLab, you are asked if you would like to create eit cluster, which is the GitLab default and recommended option. - An [Attribute-based access control (ABAC)](https://kubernetes.io/docs/reference/access-authn-authz/abac/) cluster. -GitLab creates the necessary service accounts and privileges to install and run -[GitLab managed applications](index.md#installing-applications). When GitLab creates the cluster, +When GitLab creates the cluster, a `gitlab` service account with `cluster-admin` privileges is created in the `default` namespace to manage the newly created cluster. -The first time you install an application into your cluster, the `tiller` service -account is created with `cluster-admin` privileges in the -`gitlab-managed-apps` namespace. This service account is used by Helm to -install and run [GitLab managed applications](index.md#installing-applications). - Helm also creates additional service accounts and other resources for each installed application. Consult the documentation of the Helm charts for each application for details. @@ -132,11 +135,8 @@ If you don't want to use a runner in privileged mode, either: - Use shared runners on GitLab.com. They don't have this security issue. - Set up your own runners using the configuration described at - [shared runners](../../gitlab_com/index.md#shared-runners). This involves: - 1. Making sure that you don't have it installed via - [the applications](index.md#installing-applications). - 1. Installing a runner - [using `docker+machine`](https://docs.gitlab.com/runner/executors/docker_machine.html). + [shared runners](../../gitlab_com/index.md#shared-runners) using + [`docker+machine`](https://docs.gitlab.com/runner/executors/docker_machine.html). ## Create new cluster @@ -144,36 +144,38 @@ New clusters can be created using GitLab on Google Kubernetes Engine (GKE) or Amazon Elastic Kubernetes Service (EKS) at the project, group, or instance level: 1. Navigate to your: - - Project's **{cloud-gear}** **Operations > Kubernetes** page, for a project-level cluster. + - Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level + cluster. - Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. - **Admin Area >** **{cloud-gear}** **Kubernetes** page, for an instance-level cluster. -1. Click **Add Kubernetes cluster**. +1. Click **Integrate with a cluster certificate**. 1. Click the **Create new cluster** tab. 1. Click either **Amazon EKS** or **Google GKE**, and follow the instructions for your desired service: - [Amazon EKS](add_eks_clusters.md#new-eks-cluster). - [Google GKE](add_gke_clusters.md#creating-the-cluster-on-gke). -After creating a cluster, you can install runners for it as described in -[GitLab Managed Apps](../../clusters/applications.md). +After creating a cluster, you can [install runners](https://docs.gitlab.com/runner/install/kubernetes.html), +add a [cluster management project](../../clusters/management_project.md), +configure [Auto DevOps](../../../topics/autodevops/index.md), +or start [deploying right away](index.md#deploying-to-a-kubernetes-cluster). ## Add existing cluster If you have an existing Kubernetes cluster, you can add it to a project, group, -or instance. - -Kubernetes integration isn't supported for arm64 clusters. See the issue -[Helm Tiller fails to install on arm64 cluster](https://gitlab.com/gitlab-org/gitlab/-/issues/29838) -for details. +or instance, and [install runners](https://docs.gitlab.com/runner/install/kubernetes.html) +on it (the cluster does not need to be added to GitLab first). -After adding an existing cluster, you can install runners for it as described in -[GitLab Managed Apps](../../clusters/applications.md). +After adding a cluster, you can add a [cluster management project](../../clusters/management_project.md), +configure [Auto DevOps](../../../topics/autodevops/index.md), +or start [deploying right away](index.md#deploying-to-a-kubernetes-cluster). ### Existing Kubernetes cluster To add a Kubernetes cluster to your project, group, or instance: 1. Navigate to your: - 1. Project's **{cloud-gear}** **Operations > Kubernetes** page, for a project-level cluster. + 1. Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level + cluster. 1. Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. 1. **Admin Area >** **{cloud-gear}** **Kubernetes** page, for an instance-level cluster. 1. Click **Add Kubernetes cluster**. @@ -316,8 +318,7 @@ To add a Kubernetes cluster to your project, group, or instance: 1. Finally, click the **Create Kubernetes cluster** button. -After a couple of minutes, your cluster is ready. You can now proceed -to install some [pre-defined applications](index.md#installing-applications). +After a couple of minutes, your cluster is ready. #### Disable Role-Based Access Control (RBAC) (optional) @@ -351,7 +352,8 @@ The Kubernetes cluster integration enables after you have successfully either cr a new cluster or added an existing one. To disable Kubernetes cluster integration: 1. Navigate to your: - - Project's **{cloud-gear}** **Operations > Kubernetes** page, for a project-level cluster. + - Project's **{cloud-gear}** **Infrastructure > Kubernetes clusters** page, for a project-level + cluster. - Group's **{cloud-gear}** **Kubernetes** page, for a group-level cluster. - **Admin Area >** **{cloud-gear}** **Kubernetes** page, for an instance-level cluster. 1. Click on the name of the cluster. diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index c2d06e0a22c..97296d22dd9 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -31,7 +31,7 @@ Besides integration at the project level, Kubernetes clusters can also be integrated at the [group level](../../group/clusters/index.md) or [GitLab instance level](../../instance/clusters/index.md). -To view your project level Kubernetes clusters, navigate to **Operations > Kubernetes** +To view your project level Kubernetes clusters, navigate to **Infrastructure > Kubernetes** from your project. On this page, you can [add a new cluster](#adding-and-removing-clusters) and view information about your existing clusters, such as: @@ -61,6 +61,9 @@ Kubernetes version to any supported version at any time: Some GitLab features may support versions outside the range provided here. +NOTE: +[GKE Cluster creation](add_remove_clusters.md#create-new-cluster) by GitLab is currently not supported for Kubernetes 1.19+. For these versions you can create the cluster through GCP, then [Add existing cluster](add_remove_clusters.md#add-existing-cluster). See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331922) for more information. + ### Adding and removing clusters See [Adding and removing Kubernetes clusters](add_remove_clusters.md) for details on how @@ -169,14 +172,9 @@ for your deployment jobs to use. Otherwise, a namespace is created for you. #### Important notes -Note the following with GitLab and clusters: - -- If you [install applications](#installing-applications) on your cluster, GitLab will - create the resources required to run these even if you have chosen to manage your own - cluster. -- Be aware that manually managing resources that have been created by GitLab, like - namespaces and service accounts, can cause unexpected errors. If this occurs, try - [clearing the cluster cache](#clearing-the-cluster-cache). +Be aware that manually managing resources that have been created by GitLab, like +namespaces and service accounts, can cause unexpected errors. If this occurs, try +[clearing the cluster cache](#clearing-the-cluster-cache). #### Clearing the cluster cache @@ -189,7 +187,7 @@ your cluster. This can cause deployment jobs to fail. To clear the cache: -1. Navigate to your project's **Operations > Kubernetes** page, and select your cluster. +1. Navigate to your project's **Infrastructure > Kubernetes** page, and select your cluster. 1. Expand the **Advanced settings** section. 1. Click **Clear cluster cache**. @@ -197,19 +195,15 @@ To clear the cache: > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24580) in GitLab 11.8. -You do not need to specify a base domain on cluster settings when using GitLab Serverless. The domain in that case -is specified as part of the Knative installation. See [Installing Applications](#installing-applications). - Specifying a base domain automatically sets `KUBE_INGRESS_BASE_DOMAIN` as an deployment variable. If you are using [Auto DevOps](../../../topics/autodevops/index.md), this domain is used for the different stages. For example, Auto Review Apps and Auto Deploy. The domain should have a wildcard DNS configured to the Ingress IP address. -After Ingress has been installed (see [Installing Applications](#installing-applications)), -you can either: +You can either: - Create an `A` record that points to the Ingress IP address with your domain provider. -- Enter a wildcard DNS address using a service such as nip.io or xip.io. For example, `192.168.1.1.xip.io`. +- Enter a wildcard DNS address using a service such as `nip.io` or `xip.io`. For example, `192.168.1.1.xip.io`. To determine the external Ingress IP address, or external Ingress hostname: @@ -259,13 +253,11 @@ This list provides a generic solution, and some GitLab-specific approaches: If you see a trailing `%` on some Kubernetes versions, do not include it. -## Installing applications +## Cluster management project -GitLab can install and manage some applications like Helm, GitLab Runner, Ingress, -Prometheus, and so on, in your project-level cluster. For more information on -installing, upgrading, uninstalling, and troubleshooting applications for -your project cluster, see -[GitLab Managed Apps](../../clusters/applications.md). +Attach a [Cluster management project](../../clusters/management_project.md) +to your cluster to manage shared resources requiring `cluster-admin` privileges for +installation, such as an Ingress controller. ## Auto DevOps @@ -351,16 +343,17 @@ You can customize the deployment namespace in a few ways: When you customize the namespace, existing environments remain linked to their current namespaces until you [clear the cluster cache](#clearing-the-cluster-cache). -WARNING: +#### Protecting credentials + By default, anyone who can create a deployment job can access any CI/CD variable in an environment's deployment job. This includes `KUBECONFIG`, which gives access to any secret available to the associated service account in your cluster. To keep your production credentials safe, consider using [protected environments](../../../ci/environments/protected_environments.md), -combined with either +combined with *one* of the following: -- a GitLab-managed cluster and namespace per environment, -- *or*, an environment-scoped cluster per protected environment. The same cluster +- A GitLab-managed cluster and namespace per environment. +- An environment-scoped cluster per protected environment. The same cluster can be added multiple times with multiple restricted service accounts. ### Integrations @@ -453,6 +446,6 @@ Automatically detect and monitor Kubernetes metrics. Automatic monitoring of > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4701) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/208224) to GitLab Free in 13.2. -When [Prometheus is deployed](#installing-applications), GitLab monitors the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start. +When [the Prometheus cluster integration is enabled](../../clusters/integrations.md#prometheus-cluster-integration), GitLab monitors the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start.  diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index bafb7d472c6..7a9c7eb423d 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4752) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26383) to [GitLab Free](https://about.gitlab.com/pricing/) 12.9. -GitLab makes it easy to view the logs of running pods or managed applications in +GitLab makes it easy to view the logs of running pods in [connected Kubernetes clusters](index.md). By displaying the logs directly in GitLab in the **Log Explorer**, developers can avoid managing console tools or jumping to a different interface. The **Log Explorer** interface provides a set of filters @@ -18,10 +18,11 @@ above the log file data, depending on your configuration:  - **Namespace** - Select the environment to display. Users with Maintainer or - greater [permissions](../../permissions.md) can also select Managed Apps. -- **Search** - Only available if the Elastic Stack managed application is installed. -- **Select time range** - Select the range of time to display. Only available if the - Elastic Stack managed application is installed. + greater [permissions](../../permissions.md) can also see pods in the + `gitlab-managed-apps` namespace. +- **Search** - Only available if the [Elastic Stack integration](../../clusters/integrations.md#elastic-stack-cluster-integration) is enabled. +- **Select time range** - Select the range of time to display. + Only available if the [Elastic Stack integration](../../clusters/integrations.md#elastic-stack-cluster-integration) is enabled. - **Scroll to bottom** **{scroll_down}** - Scroll to the end of the displayed logs. - **Refresh** **{retry}** - Reload the displayed logs. @@ -43,12 +44,11 @@ a [metrics dashboard](../../../operations/metrics/index.md) and select **View lo 1. Sign in as a user with the _View pod logs_ [permissions](../../permissions.md#project-members-permissions) in the project. -1. *To navigate to the **Log Explorer** from the sidebar menu,* go to - **{cloud-gear}** **Operations > Pod logs**. - ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22011) in GitLab 12.5.) -1. *To navigate to the **Log Explorer** from a specific pod on a [Deploy Board](../deploy_boards.md):* +1. To navigate to the **Log Explorer** from the sidebar menu, go to **Monitor > Logs** + ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22011) in GitLab 12.5.). +1. To navigate to the **Log Explorer** from a specific pod on a [Deploy Board](../deploy_boards.md): - 1. Go to **{cloud-gear}** **Operations > Environments** and find the environment + 1. Go to **Deployments > Environments** and find the environment which contains the desired pod, like `production`. 1. On the **Environments** page, you should see the status of the environment's pods with [Deploy Boards](../deploy_boards.md). @@ -81,7 +81,7 @@ Support for historical data is coming > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197879) in GitLab 12.8. -When you enable [Elastic Stack](../../clusters/applications.md#elastic-stack) +When you enable [Elastic Stack](../../clusters/integrations.md#elastic-stack-cluster-integration) on your cluster, you can filter logs displayed in the **Log Explorer** by date. Click **Show last** in the **Log Explorer** to see the available options. @@ -90,7 +90,7 @@ Click **Show last** in the **Log Explorer** to see the available options. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21656) in GitLab 12.7. -When you enable [Elastic Stack](../../clusters/applications.md#elastic-stack) on your cluster, +When you enable [Elastic Stack](../../clusters/integrations.md#elastic-stack-cluster-integration) on your cluster, you can search the content of your logs through a search bar. The search is passed to Elasticsearch using the [simple_query_string](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html) diff --git a/doc/user/project/clusters/protect/container_host_security/index.md b/doc/user/project/clusters/protect/container_host_security/index.md index 102001d4f87..5e4df6009f0 100644 --- a/doc/user/project/clusters/protect/container_host_security/index.md +++ b/doc/user/project/clusters/protect/container_host_security/index.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Container Host Security +# Container Host Security **(FREE)** Container Host Security in GitLab provides Intrusion Detection and Prevention capabilities that can monitor and (optionally) block activity inside the containers themselves. This is done by leveraging @@ -28,8 +28,8 @@ users define profiles for these technologies. See the [installation guide](quick_start_guide.md) for the recommended steps to install the Container Host Security capabilities. This guide shows the recommended way of installing Container -Host Security through GMAv2. However, it's also possible to do a manual installation through our -Helm chart. +Host Security through the Cluster Management Project. However, it's also possible to do a manual +installation through our Helm chart. ## Features diff --git a/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md index fa4a5fb61d0..ebcd56078ae 100644 --- a/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_host_security/quick_start_guide.md @@ -4,11 +4,9 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Getting started with Container Host Security +# Getting started with Container Host Security **(FREE)** -The following steps are recommended for installing Container Host Security. Although you can install -some capabilities through GMAv1, we [recommend](#using-gmav1-with-gmav2) that you install -applications through GMAv2 exclusively when using Container Network Security. +The following steps are recommended for installing Container Host Security. ## Installation steps @@ -21,8 +19,7 @@ The following steps are recommended to install and use Container Host Security t 1. Install and configure an Ingress node: - - [Install the Ingress node via CI/CD (GMAv2)](../../../../clusters/applications.md#install-ingress-using-gitlab-cicd). - - [Determine the external endpoint via the manual method](../../../../clusters/applications.md#determining-the-external-endpoint-manually). + - [Install the Ingress node via CI/CD (Cluster Management Project)](../../../../clusters/applications.md#install-ingress-using-gitlab-cicd). - Navigate to the Kubernetes page and enter the [DNS address for the external endpoint](../../index.md#base-domain) into the **Base domain** field on the **Details** tab. Save the changes to the Kubernetes cluster. @@ -63,19 +60,6 @@ initial troubleshooting steps that resolve the most common problems: `kubectl delete namespaces <insert-some-namespace-name>` in your Kubernetes cluster. - Rerun the application project pipeline to redeploy the application. -### Using GMAv1 with GMAv2 - -When GMAv1 and GMAv2 are used together on the same cluster, users may experience problems with -applications being uninstalled or removed from the cluster. This is because GMAv2 actively -uninstalls applications that are installed with GMAv1 and not configured to be installed with GMAv2. -It's possible to use a mixture of applications installed with GMAv1 and GMAv2 by ensuring that the -GMAv1 applications are installed **after** the GMAv2 cluster management project pipeline runs. GMAv1 -applications must be reinstalled after each run of that pipeline. This approach isn't recommended as -it's error-prone and can lead to downtime as applications are uninstalled and later reinstalled. -When using Container Network Security, the preferred and recommended path is to install all -necessary components with GMAv2 and the cluster management project. - **Related documentation links:** -- [GitLab Managed Apps v1 (GMAv1)](../../../../clusters/applications.md#install-with-one-click-deprecated) -- [GitLab Managed Apps v2 (GMAv2)](../../../../clusters/management_project.md) +- [Cluster Management Project](../../../../clusters/management_project.md) diff --git a/doc/user/project/clusters/protect/container_network_security/index.md b/doc/user/project/clusters/protect/container_network_security/index.md index a7cdd73acd7..3daa48e1811 100644 --- a/doc/user/project/clusters/protect/container_network_security/index.md +++ b/doc/user/project/clusters/protect/container_network_security/index.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Container Network Security +# Container Network Security **(FREE)** Container Network Security in GitLab provides basic firewall functionality by leveraging Cilium NetworkPolicies to filter traffic going in and out of the cluster as well as traffic between pods @@ -20,8 +20,8 @@ disabled by default, as they must usually be customized to match application-spe See the [installation guide](quick_start_guide.md) for the recommended steps to install GitLab Container Network Security. This guide shows the recommended way of installing Container Network -Security through GMAv2. However, it's also possible to install Cilium manually through our Helm -chart. +Security through the Cluster Management Project. However, it's also possible to install Cilium +manually through our Helm chart. ## Features diff --git a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md index bf419c69885..33aefec224a 100644 --- a/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md +++ b/doc/user/project/clusters/protect/container_network_security/quick_start_guide.md @@ -4,11 +4,9 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Getting started with Container Network Security +# Getting started with Container Network Security **(FREE)** -The following steps are recommended for installing Container Network Security. Although you can -install some capabilities through GMAv1, we [recommend](#using-gmav1-with-gmav2) that you install -applications through GMAv2 exclusively when using Container Network Security. +The following steps are recommended for installing Container Network Security. ## Installation steps @@ -21,8 +19,7 @@ The following steps are recommended to install and use Container Network Securit 1. Install and configure an Ingress node: - - [Install the Ingress node via CI/CD (GMAv2)](../../../../clusters/applications.md#install-ingress-using-gitlab-cicd). - - [Determine the external endpoint via the manual method](../../../../clusters/applications.md#determining-the-external-endpoint-manually). + - [Install the Ingress node via CI/CD (Cluster Management Project)](../../../../clusters/applications.md#install-ingress-using-gitlab-cicd). - Navigate to the Kubernetes page and enter the [DNS address for the external endpoint](../../index.md#base-domain) into the **Base domain** field on the **Details** tab. Save the changes to the Kubernetes cluster. @@ -60,7 +57,7 @@ use both methods simultaneously, when the application project pipeline runs the NetworkPolicy in the `auto-deploy-values.yaml` file may override policies configured in the UI editor. -## Monitoring throughput `**(ULTIMATE)**` +## Monitoring throughput **(ULTIMATE)** To view statistics for Container Network Security, you must follow the installation steps above and configure GitLab integration with Prometheus. Also, if you use custom Helm values for Cilium, you @@ -83,12 +80,8 @@ Additional information about the statistics page is available in the ## Forwarding logs to a SIEM Cilium logs can be forwarded to a SIEM or an external logging system through syslog protocol by -installing and configuring Fluentd. Fluentd can be installed through GitLab in two ways: - -- The [GMAv1 method](../../../../clusters/applications.md#fluentd) -- The [GMAv2 method](../../../../clusters/applications.md#install-fluentd-using-gitlab-cicd) - -GitLab strongly encourages using only the GMAv2 method to install Fluentd. +installing and configuring Fluentd. Fluentd can be installed through the GitLab +[Cluster Management Project](../../../../clusters/applications.md#install-fluentd-using-gitlab-cicd). ## Viewing the logs @@ -135,19 +128,6 @@ initial troubleshooting steps that resolve the most common problems: - Delete the relevant namespace in Kubernetes by running `kubectl delete namespaces <insert-some-namespace-name>` in your Kubernetes cluster. - Rerun the application project pipeline to redeploy the application. -### Using GMAv1 with GMAv2 - -When GMAv1 and GMAv2 are used together on the same cluster, users may experience problems with -applications being uninstalled or removed from the cluster. This is because GMAv2 actively -uninstalls applications that are installed with GMAv1 and not configured to be installed with GMAv2. -It's possible to use a mixture of applications installed with GMAv1 and GMAv2 by ensuring that the -GMAv1 applications are installed **after** the GMAv2 cluster management project pipeline runs. GMAv1 -applications must be reinstalled after each run of that pipeline. This approach isn't recommended as -it's error-prone and can lead to downtime as applications are uninstalled and later reinstalled. -When using Container Network Security, the preferred and recommended path is to install all -necessary components with GMAv2 and the cluster management project. - **Related documentation links:** -- [GitLab Managed Apps v1 (GMAv1)](../../../../clusters/applications.md#install-with-one-click-deprecated) -- [GitLab Managed Apps v2 (GMAv2)](../../../../clusters/management_project.md) +- [Cluster Management Project](../../../../clusters/management_project.md) diff --git a/doc/user/project/clusters/protect/index.md b/doc/user/project/clusters/protect/index.md index c489a0ddd30..1314a1948d5 100644 --- a/doc/user/project/clusters/protect/index.md +++ b/doc/user/project/clusters/protect/index.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Protecting your deployed applications +# Protecting your deployed applications **(FREE)** GitLab makes it straightforward to protect applications deployed in [connected Kubernetes clusters](index.md). These protections are available in the Kubernetes network layer and in the container itself. At @@ -18,9 +18,6 @@ containers themselves. The following capabilities are available to protect deployed applications in Kubernetes: -- Web Application Firewall - - [Overview](web_application_firewall/index.md) - - [Installation guide](web_application_firewall/quick_start_guide.md) - Container Network Security - [Overview](container_network_security/index.md) - [Installation guide](container_network_security/quick_start_guide.md) diff --git a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_disabled_settings_v12_10.png b/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_disabled_settings_v12_10.png Binary files differdeleted file mode 100644 index 2dd6df3d37b..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_disabled_settings_v12_10.png +++ /dev/null diff --git a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_installation_v12_10.png b/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_installation_v12_10.png Binary files differdeleted file mode 100644 index e88f62a2eba..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_installation_v12_10.png +++ /dev/null diff --git a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_save_changes_v12_10.png b/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_save_changes_v12_10.png Binary files differdeleted file mode 100644 index 1c99d4f7f96..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/img/guide_waf_ingress_save_changes_v12_10.png +++ /dev/null diff --git a/doc/user/project/clusters/protect/web_application_firewall/index.md b/doc/user/project/clusters/protect/web_application_firewall/index.md deleted file mode 100644 index 6e2e71c6ced..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/index.md +++ /dev/null @@ -1,103 +0,0 @@ ---- -stage: Protect -group: Container Security -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# Web Application Firewall - -WARNING: -The Web Application Firewall is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/271276) -in GitLab 13.6, and planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/271349) -in GitLab 14.0. - -A web application firewall (or WAF) filters, monitors, and blocks HTTP traffic to -and from a web application. By inspecting HTTP traffic, it can prevent attacks -stemming from web application security flaws. It can be used to detect SQL injection, -Cross-Site Scripting (XSS), Remote File Inclusion, Security Misconfigurations, and -much more. - -## Overview - -GitLab provides a WAF out of the box after Ingress is deployed. All you need to do is deploy your -application along with a service and Ingress resource. In the GitLab [Ingress](../../../../clusters/applications.md#ingress) -deployment, the [ModSecurity](https://modsecurity.org/) -module is loaded into Ingress-NGINX by default and monitors the traffic to the applications -which have an Ingress. The ModSecurity module runs with the [OWASP Core Rule Set (CRS)](https://coreruleset.org/) -by default. The OWASP CRS detects and logs a wide range of common attacks. - -By default, the WAF is deployed in Detection-only mode and only logs attack attempts. - -## Requirements - -The Web Application Firewall requires: - -- **Kubernetes** - - To enable the WAF, you need: - - - Kubernetes 1.12+. - - A load balancer. You can use NGINX-Ingress by deploying it to your - Kubernetes cluster by either: - - Using the [`nginx-ingress` Helm chart](https://github.com/helm/charts/tree/master/stable/nginx-ingress). - - Installing the [Ingress GitLab Managed App](../../../../clusters/applications.md#ingress) with WAF enabled. - -- **Configured Kubernetes objects** - - To use the WAF on an application, you need to deploy the following Kubernetes resources: - - - [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) - - [Service](https://kubernetes.io/docs/concepts/services-networking/service/) - - [Ingress Resource](https://kubernetes.io/docs/concepts/services-networking/ingress/) - -## Quick start - -If you are using GitLab.com, see the [quick start guide](quick_start_guide.md) for -how to use the WAF with GitLab.com and a Kubernetes cluster on Google Kubernetes Engine (GKE). - -If you are using a self-managed instance of GitLab, you must configure the -[Google OAuth2 OmniAuth Provider](../../../../../integration/google.md) before -you can configure a cluster on GKE. Once this is set up, you can follow the steps on the -[quick start guide](quick_start_guide.md) -to get started. - -NOTE: -This guide shows how the WAF can be deployed using Auto DevOps. The WAF -is available by default to all applications no matter how they are deployed, -as long as they are using Ingress. - -## Network firewall vs. Web Application Firewall - -A network firewall or packet filter looks at traffic at the Network (L3) and Transport (L4) layers -of the [OSI Model](https://en.wikipedia.org/wiki/OSI_model), and denies packets from entry based on -a set of rules regarding the network in general. - -A Web Application Firewall operates at the Application (L7) layer of the OSI Model and can -examine all the packets traveling to and from a specific application. A WAF can set -more advanced rules around threat detection. - -## Features - -ModSecurity is enabled with the [OWASP Core Rule Set (CRS)](https://github.com/coreruleset/coreruleset/) by -default. The OWASP CRS logs attempts to the following attacks: - -- [SQL Injection](https://wiki.owasp.org/index.php/OWASP_Periodic_Table_of_Vulnerabilities_-_SQL_Injection) -- [Cross-Site Scripting](https://wiki.owasp.org/index.php/OWASP_Periodic_Table_of_Vulnerabilities_-_Cross-Site_Scripting_(XSS)) -- [Local File Inclusion](https://wiki.owasp.org/index.php/Testing_for_Local_File_Inclusion) -- [Remote File Inclusion](https://wiki.owasp.org/index.php/OWASP_Periodic_Table_of_Vulnerabilities_-_Remote_File_Inclusion) -- [Code Injection](https://wiki.owasp.org/index.php/Code_Injection) -- [Session Fixation](https://wiki.owasp.org/index.php/Session_fixation) -- [Scanner Detection](https://wiki.owasp.org/index.php/Category:Vulnerability_Scanning_Tools) -- [Metadata/Error Leakages](https://wiki.owasp.org/index.php/Improper_Error_Handling) - -It is good to have a basic knowledge of the following: - -- [Kubernetes](https://kubernetes.io/docs/home/) -- [Ingress](https://kubernetes.github.io/ingress-nginx/) -- [ModSecurity](https://www.modsecurity.org/) -- [OWASP Core Rule Set](https://github.com/coreruleset/coreruleset/) - -## Roadmap - -You can find more information on the product direction of the WAF in -[Category Direction - Web Application Firewall](https://about.gitlab.com/direction/protect/web_application_firewall/). diff --git a/doc/user/project/clusters/protect/web_application_firewall/quick_start_guide.md b/doc/user/project/clusters/protect/web_application_firewall/quick_start_guide.md deleted file mode 100644 index e7d8d591510..00000000000 --- a/doc/user/project/clusters/protect/web_application_firewall/quick_start_guide.md +++ /dev/null @@ -1,265 +0,0 @@ ---- -stage: Protect -group: Container Security -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments ---- - -# Getting started with the Web Application Firewall - -WARNING: -The Web Application Firewall is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/271276) -in GitLab 13.6, and planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/271349) -in GitLab 14.0. - -This is a step-by-step guide to help you use the GitLab [Web Application Firewall](index.md) after -deploying a project hosted on GitLab.com to Google Kubernetes Engine using [Auto DevOps](../../../../../topics/autodevops/index.md). - -The GitLab native Kubernetes integration is used, so you do not need -to create a Kubernetes cluster manually using the Google Cloud Platform console. -A simple application is created and deployed based on a GitLab template. - -These instructions also work for a self-managed GitLab instance. However, you -need to ensure your own [runners are configured](../../../../../ci/runners/README.md) and -[Google OAuth is enabled](../../../../../integration/google.md). - -The GitLab Web Application Firewall is deployed with [Ingress](../../../../clusters/applications.md#ingress), -so it is available to your applications no matter how you deploy them to Kubernetes. - -## Configuring your Google account - -Before creating and connecting your Kubernetes cluster to your GitLab project, -you need a Google Cloud Platform account. If you do not already have one, -sign up at <https://console.cloud.google.com>. You need to either sign in with an existing -Google account (for example, one that you use to access Gmail, Drive, etc.) or create a new one. - -1. To enable the required APIs and related services, follow the steps in the ["Before you begin" section of the Kubernetes Engine docs](https://cloud.google.com/kubernetes-engine/docs/quickstart#before-you-begin). -1. Make sure you have created a [billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account). - -NOTE: -Every new Google Cloud Platform (GCP) account receives [$300 in credit](https://console.cloud.google.com/freetrial), -and in partnership with Google, GitLab is able to offer an additional $200 for new GCP accounts to get started with the GitLab -Google Kubernetes Engine integration. All you have to do is [follow this link](https://cloud.google.com/partners/partnercredit/?PCN=a0n60000006Vpz4AAC) and apply for credit. - -## Creating a new project from a template - -We use a GitLab project templates to get started. As the name suggests, -those projects provide a bare-bones application built on some well-known frameworks. - -1. In GitLab, click the plus icon (**+**) at the top of the navigation bar and select - **New project**. -1. Go to the **Create from template** tab where you can choose for example a Ruby on - Rails, Spring, or NodeJS Express project. - Use the Ruby on Rails template. - -  - -1. Give your project a name, optionally a description, and make it public so that - you can take advantage of the features available in the - [GitLab Ultimate plan](https://about.gitlab.com/pricing/). - -  - -1. Click **Create project**. - -Now that the project is created, the next step is to create the Kubernetes cluster -to deploy this application under. - -## Creating a Kubernetes cluster from within GitLab - -1. On the project's landing page, click **Add Kubernetes cluster** - (note that this option is also available when you navigate to **Operations > Kubernetes**). - -  - -1. On the **Create new cluster on GKE** tab, click **Sign in with Google**. - -  - -1. Connect with your Google account and click **Allow** when asked (this - appears only the first time you connect GitLab with your Google account). - -  - -1. The last step is to provide the cluster details. - 1. Give it a name, leave the environment scope as is, and choose the GCP project under which to create the cluster. - (Per the instructions to [configure your Google account](#configuring-your-google-account), a project should have already been created for you.) - 1. Choose the [region/zone](https://cloud.google.com/compute/docs/regions-zones/) to create the cluster in. - 1. Enter the number of nodes you want it to have. - 1. Choose the [machine type](https://cloud.google.com/compute/docs/machine-types). - -  - -1. Click **Create Kubernetes cluster**. - -After a couple of minutes, the cluster is created. You can also see its -status on your [GCP dashboard](https://console.cloud.google.com/kubernetes). - -The next step is to install some applications on your cluster that are needed -to take full advantage of Auto DevOps. - -## Install Ingress - -The GitLab Kubernetes integration comes with some -[pre-defined applications](../../index.md#installing-applications) -for you to install. - - - -For this guide, we need to install Ingress. Ingress provides load balancing, -SSL termination, and name-based virtual hosting, using NGINX behind -the scenes. Make sure to switch the toggle to the enabled position before installing. - -Both logging and blocking modes are available for WAF. While logging mode is useful for -auditing anomalous traffic, blocking mode ensures the traffic doesn't reach past Ingress. - - - -After Ingress is installed, wait a few seconds and copy the IP address that -is displayed in order to add in your base **Domain** at the top of the page. For -the purpose of this guide, we use the one suggested by GitLab. Once you have -filled in the domain, click **Save changes**. - - - -Prometheus should also be installed. It is an open-source monitoring and -alerting system that is used to supervise the deployed application. -Installing GitLab Runner is not required as we use the shared runners that -GitLab.com provides. - -## Enabling Auto DevOps (optional) - -Starting with GitLab 11.3, Auto DevOps is enabled by default. However, it is possible to disable -Auto DevOps at both the instance-level (for self-managed instances) and the group-level. -Follow these steps if Auto DevOps has been manually disabled: - -1. Navigate to **Settings > CI/CD > Auto DevOps**. -1. Select **Default to Auto DevOps pipeline**. -1. Select the [continuous deployment strategy](../../../../../topics/autodevops/index.md#deployment-strategy) - which automatically deploys the application to production once the pipeline - successfully runs on the `master` branch. -1. Click **Save changes**. - -  - -Once you complete all the above and save your changes, a new pipeline is -automatically created. To view the pipeline, go to **CI/CD > Pipelines**. - - - -The next section explains what each pipeline job does. - -## Deploying the application - -By now you should see the pipeline running, but what is it running exactly? - -To navigate inside the pipeline, click its status badge (its status should be "Running"). -The pipeline is split into a few stages, each running a couple of jobs. - - - -In the **build** stage, the application is built into a Docker image and then -uploaded to your project's [Container Registry](../../../../packages/container_registry/index.md) -([Auto Build](../../../../../topics/autodevops/stages.md#auto-build)). - -In the **test** stage, GitLab runs various checks on the application. - -The **production** stage is run after the tests and checks finish, and it automatically -deploys the application in Kubernetes ([Auto Deploy](../../../../../topics/autodevops/stages.md#auto-deploy)). - -The **production** stage creates Kubernetes objects -like a Deployment, Service, and Ingress resource. The -application is monitored by the WAF automatically. - -## Validating Ingress is running ModSecurity - -Now we can make sure that Ingress is running properly with ModSecurity and send -a request to ensure our application is responding correctly. You must connect to -your cluster either using [Cloud Shell](https://cloud.google.com/shell/) or the [Google Cloud SDK](https://cloud.google.com/sdk/docs/install). - -1. After connecting to your cluster, check if the Ingress-NGINX controller is running and ModSecurity is enabled. - - This is done by running the following commands: - - ```shell - $ kubectl get pods -n gitlab-managed-apps | grep 'ingress-controller' - ingress-nginx-ingress-controller-55f9cf6584-dxljn 2/2 Running - - $ kubectl -n gitlab-managed-apps exec -it $(kubectl get pods -n gitlab-managed-apps | grep 'ingress-controller' | awk '{print $1}') -- cat /etc/nginx/nginx.conf | grep 'modsecurity on;' - modsecurity on; - ``` - -1. Verify the Rails application has been installed properly. - - ```shell - $ kubectl get ns - auto-devv-2-16730183-production Active - - $ kubectl get pods -n auto-devv-2-16730183-production - NAME READY STATUS RESTARTS - production-5778cfcfcd-nqjcm 1/1 Running 0 - production-postgres-6449f8cc98-r7xgg 1/1 Running 0 - ``` - -1. To make sure the Rails application is responding, send a request to it by running: - - ```shell - $ kubectl get ing -n auto-devv-2-16730183-production - NAME HOSTS PORTS - production-auto-deploy fjdiaz-auto-devv-2.34.68.60.207.nip.io,le-16730183.34.68.60.207.nip.io 80, 443 - - $ curl --location --insecure "fjdiaz-auto-devv-2.34.68.60.207.nip.io" | grep 'Rails!' --after 2 --before 2 - <body> - <p>You're on Rails!</p> - </body> - ``` - -Now that we have confirmed our system is properly setup, we can go ahead and test -the WAF with OWASP CRS! - -## Testing out the OWASP Core Rule Set - -Now let's send a potentially malicious request, as if we were a scanner, -checking for vulnerabilities within our application and examine the ModSecurity logs: - -```shell -$ curl --location --insecure "fjdiaz-auto-devv-2.34.68.60.207.nip.io" --header "User-Agent: absinthe" | grep 'Rails!' --after 2 --before 2 -<body> - <p>You're on Rails!</p> -</body> - -$ kubectl -n gitlab-managed-apps exec -it $(kubectl get pods -n gitlab-managed-apps | grep 'ingress-controller' | awk '{print $1}') -- cat /var/log/modsec/audit.log | grep 'absinthe' -{ - "message": "Found User-Agent associated with security scanner", - "details": { - "match": "Matched \"Operator `PmFromFile' with parameter `scanners-user-agents.data' against variable `REQUEST_HEADERS:user-agent' (Value: `absinthe' )", - "reference": "o0,8v84,8t:lowercase", - "ruleId": "913100", - "file": "/etc/nginx/owasp-modsecurity-crs/rules/REQUEST-913-SCANNER-DETECTION.conf", - "lineNumber": "33", - "data": "Matched Data: absinthe found within REQUEST_HEADERS:user-agent: absinthe", - "severity": "2", - "ver": "OWASP_CRS/3.2.0", - "rev": "", - "tags": ["application-multi", "language-multi", "platform-multi", "attack-reputation-scanner", "OWASP_CRS", "OWASP_CRS/AUTOMATION/SECURITY_SCANNER", "WASCTC/WASC-21", "OWASP_TOP_10/A7", "PCI/6.5.10"], - "maturity": "0", - "accuracy": "0" - } -} -``` - -You can see that ModSecurity logs the suspicious behavior. By sending a request -with the `User Agent: absinthe` header, which [absinthe](https://github.com/cameronhotchkies/Absinthe), -a tool for testing for SQL injections uses, we can detect that someone was -searching for vulnerabilities on our system. Detecting scanners is useful, because we -can learn if someone is trying to exploit our system. - -## Conclusion - -You can now see the benefits of a using a Web Application Firewall. -ModSecurity and the OWASP Core Rule Set, offer many more benefits. -You can explore them in more detail: - -- [Category Direction - Web Application Firewall](https://about.gitlab.com/direction/protect/web_application_firewall/) -- [ModSecurity](https://www.modsecurity.org/) -- [OWASP Core Rule Set](https://github.com/coreruleset/coreruleset/) -- [AutoDevOps](../../../../../topics/autodevops/index.md) diff --git a/doc/user/project/clusters/runbooks/index.md b/doc/user/project/clusters/runbooks/index.md index e0b8c074fcf..b2054c4befb 100644 --- a/doc/user/project/clusters/runbooks/index.md +++ b/doc/user/project/clusters/runbooks/index.md @@ -63,18 +63,93 @@ information. Follow this step-by-step guide to configure an executable runbook in GitLab using the components outlined above and the pre-loaded demo runbook. -1. Add a Kubernetes cluster to your project by following the steps outlined in - [Create new cluster](../add_remove_clusters.md#create-new-cluster). - -1. Click the **Install** button next to the **Ingress** application to install Ingress. - -  - -1. After Ingress has been installed successfully, click the **Install** button next - to the **JupyterHub** application. You need the **Jupyter Hostname** provided - here in the next step. - -  +1. Create an [OAuth Application for JupyterHub](../../../../integration/oauth_provider.md#gitlab-as-oauth2-authentication-service-provider). +1. When [installing JupyterHub with Helm](https://zero-to-jupyterhub.readthedocs.io/en/latest/jupyterhub/installation.html), use the following values + + ```yaml + #----------------------------------------------------------------------------- + # The gitlab and ingress sections must be customized! + #----------------------------------------------------------------------------- + + gitlab: + clientId: <Your OAuth Application ID> + clientSecret: <Your OAuth Application Secret> + callbackUrl: http://<Jupyter Hostname>/hub/oauth_callback, + # Limit access to members of specific projects or groups: + # allowedGitlabGroups: [ "my-group-1", "my-group-2" ] + # allowedProjectIds: [ 12345, 6789 ] + + # ingress is required for OAuth to work + ingress: + enabled: true + host: <JupyterHostname> + # tls: + # - hosts: + # - <JupyterHostanme> + # secretName: jupyter-cert + # annotations: + # kubernetes.io/ingress.class: "nginx" + # kubernetes.io/tls-acme: "true" + + #----------------------------------------------------------------------------- + # NO MODIFICATIONS REQUIRED BEYOND THIS POINT + #----------------------------------------------------------------------------- + + hub: + extraEnv: + JUPYTER_ENABLE_LAB: 1 + extraConfig: | + c.KubeSpawner.cmd = ['jupyter-labhub'] + c.GitLabOAuthenticator.scope = ['api read_repository write_repository'] + + async def add_auth_env(spawner): + ''' + We set user's id, login and access token on single user image to + enable repository integration for JupyterHub. + See: https://gitlab.com/gitlab-org/gitlab-foss/issues/47138#note_154294790 + ''' + auth_state = await spawner.user.get_auth_state() + + if not auth_state: + spawner.log.warning("No auth state for %s", spawner.user) + return + + spawner.environment['GITLAB_ACCESS_TOKEN'] = auth_state['access_token'] + spawner.environment['GITLAB_USER_LOGIN'] = auth_state['gitlab_user']['username'] + spawner.environment['GITLAB_USER_ID'] = str(auth_state['gitlab_user']['id']) + spawner.environment['GITLAB_USER_EMAIL'] = auth_state['gitlab_user']['email'] + spawner.environment['GITLAB_USER_NAME'] = auth_state['gitlab_user']['name'] + + c.KubeSpawner.pre_spawn_hook = add_auth_env + + auth: + type: gitlab + state: + enabled: true + + singleuser: + defaultUrl: "/lab" + image: + name: registry.gitlab.com/gitlab-org/jupyterhub-user-image + tag: latest + lifecycleHooks: + postStart: + exec: + command: + - "sh" + - "-c" + - > + git clone https://gitlab.com/gitlab-org/nurtch-demo.git DevOps-Runbook-Demo || true; + echo "https://oauth2:${GITLAB_ACCESS_TOKEN}@${GITLAB_HOST}" > ~/.git-credentials; + git config --global credential.helper store; + git config --global user.email "${GITLAB_USER_EMAIL}"; + git config --global user.name "${GITLAB_USER_NAME}"; + jupyter serverextension enable --py jupyterlab_git + + proxy: + service: + type: ClusterIP + ``` 1. After JupyterHub has been installed successfully, open the **Jupyter Hostname** in your browser. Click the **Sign in with GitLab** button to log in to diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index e355b562c36..e4ac1eabffe 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -15,7 +15,7 @@ Serverless is currently in [alpha](https://about.gitlab.com/handbook/product/git Serverless architectures offer Operators and Developers the ability write highly scalable applications without provisioning a single server. -GitLab supports several ways deploy Serverless applications in both Kubernetes Environments and also major cloud FAAS environments. +GitLab supports several ways deploy Serverless applications in both Kubernetes Environments and also major cloud Function as a Service (FaaS) environments. Currently we support: @@ -35,7 +35,7 @@ of the box through its main components: For more information on Knative, visit the [Knative docs repository](https://github.com/knative/docs). -With GitLab Serverless, you can deploy both functions-as-a-service (FaaS) and serverless applications. +With GitLab Serverless, you can deploy both FaaS and serverless applications. ## Prerequisites @@ -53,7 +53,7 @@ To run Knative on GitLab, you need: The set of minimum recommended cluster specifications to run Knative is 3 nodes, 6 vCPUs, and 22.50 GB memory. 1. **GitLab Runner:** A runner is required to run the CI jobs that deploy serverless applications or functions onto your cluster. You can install GitLab Runner - onto the existing Kubernetes cluster. See [Installing Applications](../index.md#installing-applications) for more information. + onto the [existing Kubernetes cluster](https://docs.gitlab.com/runner/install/kubernetes.html). 1. **Domain Name:** Knative provides its own load balancer using Istio, and an external IP address or hostname for all the applications served by Knative. Enter a wildcard domain to serve your applications. Configure your DNS server to use the @@ -68,54 +68,18 @@ To run Knative on GitLab, you need: `Dockerfile` in order to build your applications. It should be included at the root of your project's repository and expose port `8080`. `Dockerfile` is not require if you plan to build serverless functions using our [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes). -1. **Prometheus** (optional): Installing Prometheus allows you to monitor the scale and traffic of your serverless function/application. - See [Installing Applications](../index.md#installing-applications) for more information. +1. **Prometheus** (optional): The [Prometheus cluster integration](../../../clusters/integrations.md#prometheus-cluster-integration) + allows you to monitor the scale and traffic of your serverless function/application. 1. **Logging** (optional): Configuring logging allows you to view and search request logs for your serverless function/application. See [Configuring logging](#configuring-logging) for more information. -## Installing Knative via the GitLab Kubernetes integration - -The minimum recommended cluster size to run Knative is 3-nodes, 6 vCPUs, and 22.50 GB -memory. **RBAC must be enabled.** - -1. [Add a Kubernetes cluster](../add_remove_clusters.md). -1. Select the **Applications** tab and scroll down to the Knative app section. Enter the domain to be used with - your application/functions (e.g. `example.com`) and click **Install**. - -  - -1. After the Knative installation has finished, you can wait for the IP address or hostname to be displayed in the - **Knative Endpoint** field or [retrieve the Istio Ingress Endpoint manually](../../../clusters/applications.md#determining-the-external-endpoint-manually). - - NOTE: - Running `kubectl` commands on your cluster requires setting up access to the cluster first. - For clusters created on GKE, see [GKE Cluster Access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl), - for other platforms [Install kubectl](https://kubernetes.io/docs/tasks/tools/). - -1. The Ingress is now available at this address and routes incoming requests to the proper service based on the DNS - name in the request. To support this, a wildcard DNS record should be created for the desired domain name. For example, - if your Knative base domain is `knative.info` then you need to create an A record or CNAME record with domain `*.knative.info` - pointing the IP address or hostname of the Ingress. - -  - -You can deploy either [functions](#deploying-functions) or [serverless applications](#deploying-serverless-applications) -on a given project, but not both. The current implementation makes use of a -`serverless.yml` file to signal a FaaS project. - -## Using an existing installation of Knative +## Configuring Knative > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/58941) in GitLab 12.0. -The _invocations_ monitoring feature of GitLab serverless is unavailable when -adding an existing installation of Knative. - -It's also possible to use GitLab Serverless with an existing Kubernetes cluster -which already has Knative installed. You must do the following: - 1. Follow the steps to - [add an existing Kubernetes - cluster](../add_remove_clusters.md#add-existing-cluster). + [add a Kubernetes + cluster](../add_remove_clusters.md). 1. Ensure GitLab can manage Knative: - For a non-GitLab managed cluster, ensure that the service account for the token @@ -164,13 +128,17 @@ which already has Knative installed. You must do the following: kubectl apply -f knative-serving-only-role.yaml ``` - If you would rather grant permissions on a per service account basis, you can do this - using a `Role` and `RoleBinding` specific to the service account and namespace. + Alternatively, permissions can be granted on a per-service account basis + using `Role`s and `RoleBinding`s (see the [Kubernetes RBAC + documentation](https://kubernetes.io/docs/reference/access-authn-authz/rbac/) + for more information). 1. Follow the steps to deploy [functions](#deploying-functions) or [serverless applications](#deploying-serverless-applications) onto your cluster. +1. **Optional:** For invocation metrics to show in GitLab, additional Istio metrics need to be configured in your cluster. For example, with Knative v0.9.0, you can use [this manifest](https://gitlab.com/gitlab-org/charts/knative/-/raw/v0.10.0/vendor/istio-metrics.yml). + ## Supported runtimes Serverless functions for GitLab can be run using: @@ -183,7 +151,7 @@ If a runtime is not available for the required programming language, consider de ### GitLab-managed runtimes -Currently the following GitLab-managed [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes) +The following GitLab-managed [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes) are available: - `go` (proof of concept) @@ -352,7 +320,7 @@ After the `gitlab-ci.yml` template has been added and the `serverless.yml` file has been created, pushing a commit to your project results in a CI pipeline being executed which deploys each function as a Knative service. After the deploy stage has finished, additional details for the function display -under **Operations > Serverless**. +under **Infrastructure > Serverless platform**.  @@ -486,7 +454,7 @@ With all the pieces in place, the next time a CI pipeline runs the Knative appli ### Function details -Go to the **Operations > Serverless** page to see the final URL of your functions. +Go to the **Infrastructure > Serverless platform** page to see the final URL of your functions.  @@ -499,10 +467,10 @@ rows to bring up the function details page. The pod count gives you the number of pods running the serverless function instances on a given cluster. -For the Knative function invocations to appear, -[Prometheus must be installed](../index.md#installing-applications). +For the Knative function invocations to appear, the +[Prometheus cluster integration must be enabled](../../../clusters/integrations.md#prometheus-cluster-integration). -Once Prometheus is installed, a message may appear indicating that the metrics data _is +Once Prometheus is enabled, a message may appear indicating that the metrics data _is loading or is not available at this time._ It appears upon the first access of the page, but should go away after a few seconds. If the message does not disappear, then it is possible that GitLab is unable to connect to the Prometheus instance running on the @@ -611,7 +579,7 @@ or with other versions of Python. Where `<namespace>` is the namespace created by GitLab for your serverless project (composed of `<project_name>-<project_id>-<environment>`) and `example.com` is the domain being used for your project. If you are unsure what the namespace of your project is, navigate - to the **Operations > Serverless** page of your project and inspect + to the **Infrastructure > Serverless platform** page of your project and inspect the endpoint provided for your function/app.  |
