diff options
Diffstat (limited to 'doc/topics/autodevops')
-rw-r--r-- | doc/topics/autodevops/customize.md | 65 | ||||
-rw-r--r-- | doc/topics/autodevops/index.md | 10 | ||||
-rw-r--r-- | doc/topics/autodevops/quick_start_guide.md | 30 | ||||
-rw-r--r-- | doc/topics/autodevops/requirements.md | 77 | ||||
-rw-r--r-- | doc/topics/autodevops/stages.md | 130 | ||||
-rw-r--r-- | doc/topics/autodevops/upgrading_auto_deploy_dependencies.md | 20 |
6 files changed, 178 insertions, 154 deletions
diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index a1a12ccd451..42c54961c1d 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -16,15 +16,25 @@ staging and canary deployments, ## Custom buildpacks -If the automatic buildpack detection fails for your project, or if you want to -use a custom buildpack, you can override the buildpack using a project CI/CD variable -or a `.buildpacks` file in your project: +If the automatic buildpack detection fails for your project, or if you +need more control over your build, you can customize the buildpacks +used for the build. -- **Project variable** - Create a project variable `BUILDPACK_URL` with the URL - of the buildpack to use. -- **`.buildpacks` file** - Add a file in your project's repository called `.buildpacks`, - and add the URL of the buildpack to use on a line in the file. If you want to - use multiple buildpacks, enter one buildpack per line. +### Custom buildpacks with Cloud Native Buildpacks + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28165) in GitLab 12.10. + +Specify either: + +- The CI/CD variable `BUILDPACK_URL` according to [`pack`'s specifications](https://buildpacks.io/docs/app-developer-guide/specific-buildpacks/). +- A [`project.toml` project descriptor](https://buildpacks.io/docs/app-developer-guide/using-project-descriptor/) with the buildpacks you would like to include. + +### Custom buildpacks with Herokuish + +Specify either: + +- The CI/CD variable `BUILDPACK_URL`. +- A `.buildpacks` file at the root of your project, containing one buildpack URL per line. The buildpack URL can point to either a Git repository URL or a tarball URL. For Git repositories, you can point to a specific Git reference (such as @@ -176,7 +186,7 @@ to the desired environment. See [Limit environment scope of CI/CD variables](../ ## Customizing `.gitlab-ci.yml` Auto DevOps is completely customizable because the -[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) +[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) is just an implementation of a [`.gitlab-ci.yml`](../../ci/yaml/README.md) file, and uses only features available to any implementation of `.gitlab-ci.yml`. @@ -191,11 +201,11 @@ include: ``` Add your changes, and your additions are merged with the -[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) +[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) using the behavior described for [`include`](../../ci/yaml/README.md#include). If you need to specifically remove a part of the file, you can also copy and paste the contents of the -[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) +[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) into your project and edit it as needed. ## Customizing the Kubernetes namespace @@ -241,7 +251,7 @@ include: - template: Jobs/Build.gitlab-ci.yml ``` -See the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) for information on available jobs. +See the [Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) for information on available jobs. WARNING: Auto DevOps templates using the [`only`](../../ci/yaml/README.md#only--except) or @@ -255,6 +265,27 @@ base template is migrated to use the `rules` syntax. For users who cannot migrate just yet, you can alternatively pin your templates to the [GitLab 12.10 based templates](https://gitlab.com/gitlab-org/auto-devops-v12-10). +## Use images hosted in a local Docker registry + +You can configure many Auto DevOps jobs to run in an [offline environment](../../user/application_security/offline_deployments/index.md): + +1. Copy the required Auto DevOps Docker images from Docker Hub and `registry.gitlab.com` to their local GitLab container registry. +1. After the images are hosted and available in a local registry, edit `.gitlab-ci.yml` to point to the locally-hosted images. For example: + + ```yaml + include: + - template: Auto-DevOps.gitlab-ci.yml + + variables: + REGISTRY_URL: "registry.gitlab.example" + + build: + image: "$REGISTRY_URL/docker/auto-build-image:v0.6.0" + services: + - name: "$REGISTRY_URL/greg/docker/docker:20.10.6-dind" + command: ['--tls=false', '--host=tcp://0.0.0.0:2375'] + ``` + ## PostgreSQL database support To support applications requiring a database, @@ -326,8 +357,8 @@ applications. | `ADDITIONAL_HOSTS` | Fully qualified domain names specified as a comma-separated list that are added to the Ingress hosts. | | `<ENVIRONMENT>_ADDITIONAL_HOSTS` | For a specific environment, the fully qualified domain names specified as a comma-separated list that are added to the Ingress hosts. This takes precedence over `ADDITIONAL_HOSTS`. | | `AUTO_DEVOPS_ATOMIC_RELEASE` | As of GitLab 13.0, Auto DevOps uses [`--atomic`](https://v2.helm.sh/docs/helm/#options-43) for Helm deployments by default. Set this variable to `false` to disable the use of `--atomic` | -| `AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED` | When set to a non-empty value and no `Dockerfile` is present, Auto Build builds your application using Cloud Native Buildpacks instead of Herokuish. [More details](stages.md#auto-build-using-cloud-native-buildpacks-beta). | -| `AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER` | The builder used when building with Cloud Native Buildpacks. The default builder is `heroku/buildpacks:18`. [More details](stages.md#auto-build-using-cloud-native-buildpacks-beta). | +| `AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED` | Set to `false` to use Herokuish instead of Cloud Native Buildpacks with Auto Build. [More details](stages.md#auto-build-using-cloud-native-buildpacks). | +| `AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER` | The builder used when building with Cloud Native Buildpacks. The default builder is `heroku/buildpacks:18`. [More details](stages.md#auto-build-using-cloud-native-buildpacks). | | `AUTO_DEVOPS_BUILD_IMAGE_EXTRA_ARGS` | Extra arguments to be passed to the `docker build` command. Note that using quotes doesn't prevent word splitting. [More details](#passing-arguments-to-docker-build). | | `AUTO_DEVOPS_BUILD_IMAGE_FORWARDED_CI_VARIABLES` | A [comma-separated list of CI/CD variable names](#forward-cicd-variables-to-the-build-environment) to be forwarded to the build environment (the buildpack builder or `docker build`). | | `AUTO_DEVOPS_CHART` | Helm Chart used to deploy your apps. Defaults to the one [provided by GitLab](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/tree/master/assets/auto-deploy-app). | @@ -337,8 +368,7 @@ applications. | `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From GitLab 11.11, used to set a password to connect to the Helm repository. Defaults to no credentials. Also set `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME`. | | `AUTO_DEVOPS_DEPLOY_DEBUG` | From GitLab 13.1, if this variable is present, Helm outputs debug logs. | | `AUTO_DEVOPS_ALLOW_TO_FORCE_DEPLOY_V<N>` | From [auto-deploy-image](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image) v1.0.0, if this variable is present, a new major version of chart is forcibly deployed. For more information, see [Ignore warnings and continue deploying](upgrading_auto_deploy_dependencies.md#ignore-warnings-and-continue-deploying). | -| `AUTO_DEVOPS_MODSECURITY_SEC_RULE_ENGINE` | From GitLab 12.5, used in combination with [ModSecurity feature flag](../../user/clusters/applications.md#web-application-firewall-modsecurity) to toggle [ModSecurity's `SecRuleEngine`](https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecRuleEngine) behavior. Defaults to `DetectionOnly`. | -| `BUILDPACK_URL` | Buildpack's full URL. Can point to either [a Git repository URL or a tarball URL](#custom-buildpacks). | +| `BUILDPACK_URL` | Buildpack's full URL. [Must point to a URL supported by Pack or Herokuish](#custom-buildpacks). | | `CANARY_ENABLED` | From GitLab 11.0, used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments). | | `CANARY_PRODUCTION_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. Takes precedence over `CANARY_REPLICAS`. Defaults to 1. | | `CANARY_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md). Defaults to 1. | @@ -411,7 +441,8 @@ The following table lists variables used to disable jobs. | `license_scanning` | `LICENSE_MANAGEMENT_DISABLED` | [From GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22773) | If the variable is present, the job isn't created. | | `load_performance` | `LOAD_PERFORMANCE_DISABLED` | From GitLab 13.2 | If the variable is present, the job isn't created. | | `nodejs-scan-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | -| `performance` | `PERFORMANCE_DISABLED` | From GitLab 11.0 | Browser performance. If the variable is present, the job isn't created. | +| `performance` | `PERFORMANCE_DISABLED` | GitLab 11.0 to GitLab 13.12 | Browser performance. If the variable is present, the job isn't created. Replaced by `browser_peformance`. | +| `browser_performance` | `BROWSER_PERFORMANCE_DISABLED` | From GitLab 14.0 | Browser performance. If the variable is present, the job isn't created. Replaces `performance`. | | `phpcs-security-audit-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | | `pmd-apex-sast` | `SAST_DISABLED` | | If the variable is present, the job isn't created. | | `retire-js-dependency_scanning` | `DEPENDENCY_SCANNING_DISABLED` | | If the variable is present, the job isn't created. | diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 03454649c7e..f2ce61044ef 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -264,7 +264,7 @@ When using Auto DevOps, you can deploy different environments to different Kubernetes clusters, due to the 1:1 connection [existing between them](../../user/project/clusters/index.md#multiple-kubernetes-clusters). -The [Deploy Job template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) +The [Deploy Job template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml) used by Auto DevOps currently defines 3 environment names: - `review/` (every environment starting with `review/`) @@ -286,14 +286,14 @@ The following table is an example of how to configure the three different cluste To add a different cluster for each environment: -1. Navigate to your project's **Operations > Kubernetes**. +1. Navigate to your project's **Infrastructure > Kubernetes clusters**. 1. Create the Kubernetes clusters with their respective environment scope, as described from the table above. -1. After creating the clusters, navigate to each cluster and install - Ingress. Wait for the Ingress IP address to be assigned. +1. After creating the clusters, navigate to each cluster and [install + Ingress](quick_start_guide.md#install-ingress). Wait for the Ingress IP address to be assigned. 1. Make sure you've [configured your DNS](#auto-devops-base-domain) with the specified Auto DevOps domains. -1. Navigate to each cluster's page, through **Operations > Kubernetes**, +1. Navigate to each cluster's page, through **Infrastructure > Kubernetes clusters**, and add the domain based on its Ingress IP address. After completing configuration, you can test your setup by creating a merge request diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index 631180f696c..448e9526b88 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -63,7 +63,8 @@ to deploy this project to. ## Create a Kubernetes cluster from within GitLab 1. On your project's landing page, click **Add Kubernetes cluster** - (note that this option is also available when you navigate to **Operations > Kubernetes**). + (note that this option is also available when you navigate to + **Infrastructure > Kubernetes clusters**). ![Project landing page](img/guide_project_landing_page_v12_10.png) @@ -106,7 +107,8 @@ status on your [GCP dashboard](https://console.cloud.google.com/kubernetes). After your cluster is running, you must install NGINX Ingress Controller as a load balancer, to route traffic from the internet to your application. Because you've created a Google GKE cluster in this guide, you can install NGINX Ingress Controller -with Google Cloud Shell: +through the GitLab [Cluster management project template](../../user/clusters/management_project_template.md), +or manually with Google Cloud Shell: 1. Go to your cluster's details page, and click the **Advanced Settings** tab. 1. Click the link to Google Kubernetes Engine to visit the cluster on Google Cloud Console. @@ -114,21 +116,28 @@ with Google Cloud Shell: 1. After the Cloud Shell starts, run these commands to install NGINX Ingress Controller: ```shell - helm repo add nginx-stable https://helm.nginx.com/stable + kubectl create ns gitlab-managed-apps + helm repo add stable https://charts.helm.sh/stable helm repo update - helm install nginx-ingress nginx-stable/nginx-ingress + helm install ingress stable/nginx-ingress -n gitlab-managed-apps # Check that the ingress controller is installed successfully - kubectl get service nginx-ingress-nginx-ingress + kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps ``` -1. A few minutes after you install NGINX, the load balancer obtains an IP address, and you can - get the external IP address with this command: +## Configure your base domain + +Follow these steps to configure the Base Domain where your apps will be accessible. +1. A few minutes after you install NGINX, the load balancer obtains an IP address, and you can + get the external IP address with the following command: + ```shell - kubectl get service nginx-ingress-nginx-ingress -ojson | jq -r '.status.loadBalancer.ingress[].ip' + kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps -ojson | jq -r '.status.loadBalancer.ingress[].ip' ``` + Replace `gitlab-managed-apps` if you have overwritten your namespace. + Copy this IP address, as you need it in the next step. 1. Go back to the cluster page on GitLab, and go to the **Details** tab. @@ -186,7 +195,7 @@ The jobs are separated into stages: - Jobs suffixed with `-sast` run static analysis on the current code to check for potential security issues, and are allowed to fail ([Auto SAST](stages.md#auto-sast)) **(ULTIMATE)** - The `secret-detection` job checks for leaked secrets and is allowed to fail ([Auto Secret Detection](stages.md#auto-secret-detection)) **(ULTIMATE)** - - The `license_management` job searches the application's dependencies to determine each of their + - The `license_scanning` job searches the application's dependencies to determine each of their licenses and is allowed to fail ([Auto License Compliance](stages.md#auto-license-compliance)) **(ULTIMATE)** @@ -208,7 +217,7 @@ to monitor it. After successfully deploying your application, you can view its website and check on its health on the **Environments** page by navigating to -**Operations > Environments**. This page displays details about +**Deployments > Environments**. This page displays details about the deployed applications, and the right-hand column displays icons that link you to common environment tasks: @@ -308,6 +317,5 @@ and customized to fit your workflow. Here are some helpful resources for further 1. [Multiple Kubernetes clusters](index.md#using-multiple-kubernetes-clusters) 1. [Incremental rollout to production](customize.md#incremental-rollout-to-production) **(PREMIUM)** 1. [Disable jobs you don't need with CI/CD variables](customize.md#cicd-variables) -1. [Use a static IP for your cluster](../../user/clusters/applications.md#using-a-static-ip) 1. [Use your own buildpacks to build your application](customize.md#custom-buildpacks) 1. [Prometheus monitoring](../../user/project/integrations/prometheus.md) diff --git a/doc/topics/autodevops/requirements.md b/doc/topics/autodevops/requirements.md index 8fb16511e34..7e59ecb4916 100644 --- a/doc/topics/autodevops/requirements.md +++ b/doc/topics/autodevops/requirements.md @@ -26,23 +26,26 @@ To make full use of Auto DevOps with Kubernetes, you need: [new cluster using the GitLab UI](../../user/project/clusters/add_remove_clusters.md#create-new-cluster). For Kubernetes 1.16+ clusters, you must perform additional configuration for [Auto Deploy for Kubernetes 1.16+](stages.md#kubernetes-116). - 1. NGINX Ingress. You can deploy it to your Kubernetes cluster by installing - the [GitLab-managed app for Ingress](../../user/clusters/applications.md#ingress), - after configuring the GitLab integration with Kubernetes in the previous step. - - Alternatively, you can use the - [`nginx-ingress`](https://github.com/helm/charts/tree/master/stable/nginx-ingress) - Helm chart to install Ingress manually. + 1. For external HTTP traffic, an Ingress controller is required. For regular + deployments, any Ingress controller should work, but as of GitLab 14.0, + [canary deployments](../../user/project/canary_deployments.md) require + NGINX Ingress. You can deploy the NGINX Ingress controller to your + Kubernetes cluster either through the GitLab [Cluster management project template](../../user/clusters/management_project_template.md) + or manually by using the [`ingress-nginx`](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx) + Helm chart. NOTE: - If you use your own Ingress instead of the one provided by GitLab Managed - Apps, ensure you're running at least version 0.9.0 of NGINX Ingress and - [enable Prometheus metrics](https://github.com/helm/charts/tree/master/stable/nginx-ingress#prometheus-metrics) - for the response metrics to appear. You must also + For metrics to appear when using the [Prometheus cluster integration](../../user/clusters/integrations.md#prometheus-cluster-integration), you must [enable Prometheus metrics](https://github.com/kubernetes/ingress-nginx/tree/master/charts/ingress-nginx#prometheus-metrics). + + When deploying [using custom charts](customize.md#custom-helm-chart), you must also [annotate](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/) - the NGINX Ingress deployment to be scraped by Prometheus using + the Ingress manifest to be scraped by Prometheus using `prometheus.io/scrape: "true"` and `prometheus.io/port: "10254"`. + NOTE: + If your cluster is installed on bare metal, see + [Auto DevOps Requirements for bare metal](#auto-devops-requirements-for-bare-metal). + - **Base domain** (for [Auto Review Apps](stages.md#auto-review-apps), [Auto Deploy](stages.md#auto-deploy), and [Auto Monitoring](stages.md#auto-monitoring)) @@ -61,42 +64,34 @@ To make full use of Auto DevOps with Kubernetes, you need: You can configure Docker-based runners to autoscale as well, using [Docker Machine](https://docs.gitlab.com/runner/install/autoscaling.html). - If you've configured the GitLab integration with Kubernetes in the first step, you - can deploy it to your cluster by installing the - [GitLab-managed app for GitLab Runner](../../user/clusters/applications.md#gitlab-runner). - - Runners should be registered as [shared runners](../../ci/runners/README.md#shared-runners) - for the entire GitLab instance, or [specific runners](../../ci/runners/README.md#specific-runners) - that are assigned to specific projects (the default if you've installed the - GitLab Runner managed application). + Runners should be registered as [shared runners](../../ci/runners/runners_scope.md#shared-runners) + for the entire GitLab instance, or [specific runners](../../ci/runners/runners_scope.md#specific-runners) + that are assigned to specific projects. - **Prometheus** (for [Auto Monitoring](stages.md#auto-monitoring)) To enable Auto Monitoring, you need Prometheus installed either inside or outside your cluster, and configured to scrape your Kubernetes cluster. - If you've configured the GitLab integration with Kubernetes, you can deploy it to - your cluster by installing the - [GitLab-managed app for Prometheus](../../user/clusters/applications.md#prometheus). + If you've configured the GitLab integration with Kubernetes, you can + instruct GitLab to query an in-cluster Prometheus by enabling + the [Prometheus cluster integration](../../user/clusters/integrations.md#prometheus-cluster-integration). - The [Prometheus service](../../user/project/integrations/prometheus.md) - integration must be enabled for the project, or enabled as a - [default service template](../../user/project/integrations/services_templates.md) - for the entire GitLab installation. + The [Prometheus integration](../../user/project/integrations/prometheus.md) + integration must be activated for the project, or activated at the group or instance level. + Learn more about [Project integration management](../../user/admin_area/settings/project_integration_management.md). To get response metrics (in addition to system metrics), you must [configure Prometheus to monitor NGINX](../../user/project/integrations/prometheus_library/nginx_ingress.md#configuring-nginx-ingress-monitoring). - **cert-manager** (optional, for TLS/HTTPS) - To enable HTTPS endpoints for your application, you must install cert-manager, + To enable HTTPS endpoints for your application, you can [install cert-manager](https://cert-manager.io/docs/installation/kubernetes/), a native Kubernetes certificate management controller that helps with issuing certificates. Installing cert-manager on your cluster issues a [Let's Encrypt](https://letsencrypt.org/) certificate and ensures the - certificates are valid and up-to-date. If you've configured the GitLab integration - with Kubernetes, you can deploy it to your cluster by installing the - [GitLab-managed app for cert-manager](../../user/clusters/applications.md#cert-manager). + certificates are valid and up-to-date. -If you don't have Kubernetes or Prometheus installed, then +If you don't have Kubernetes or Prometheus configured, then [Auto Review Apps](stages.md#auto-review-apps), [Auto Deploy](stages.md#auto-deploy), and [Auto Monitoring](stages.md#auto-monitoring) are skipped. @@ -124,9 +119,6 @@ When you trigger a pipeline, if you have Auto DevOps enabled and if you have cor [entered AWS credentials as variables](../../ci/cloud_deployment/index.md#deploy-your-application-to-the-aws-elastic-container-service-ecs), your application is deployed to AWS ECS. -[GitLab Managed Apps](../../user/clusters/applications.md) are not available when deploying to AWS ECS. -You must manually configure your application (such as Ingress or Help) on AWS ECS. - If you have both a valid `AUTO_DEVOPS_PLATFORM_TARGET` variable and a Kubernetes cluster tied to your project, only the deployment to Kubernetes runs. @@ -149,3 +141,18 @@ specific CI/CD variable. For more details, see [Custom build job for Auto DevOps](../../ci/cloud_deployment/index.md#custom-build-job-for-auto-devops) for deployments to AWS EC2. + +## Auto DevOps requirements for bare metal + +According to the [Kubernetes Ingress-NGINX docs](https://kubernetes.github.io/ingress-nginx/deploy/baremetal/): + +> In traditional cloud environments, where network load balancers are available on-demand, +a single Kubernetes manifest suffices to provide a single point of contact to the NGINX Ingress +controller to external clients and, indirectly, to any application running inside the cluster. +Bare-metal environments lack this commodity, requiring a slightly different setup to offer the +same kind of access to external consumers. + +The docs linked above explain the issue and present possible solutions, for example: + +- Through [MetalLB](https://github.com/metallb/metallb). +- Through [PorterLB](https://github.com/kubesphere/porterlb). diff --git a/doc/topics/autodevops/stages.md b/doc/topics/autodevops/stages.md index 66b37f30bbc..d3f217d3749 100644 --- a/doc/topics/autodevops/stages.md +++ b/doc/topics/autodevops/stages.md @@ -33,15 +33,24 @@ your own `Dockerfile`, you must either: - Override the default values by [customizing the Auto Deploy Helm chart](customize.md#custom-helm-chart). -### Auto Build using Heroku buildpacks +### Auto Build using Cloud Native Buildpacks + +> - Introduced in [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28165). +> - Auto Build using Cloud Native Buildpacks by default was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63351) in GitLab 14.0. Auto Build builds an application using a project's `Dockerfile` if present. If no -`Dockerfile` is present, it uses [Herokuish](https://github.com/gliderlabs/herokuish) -and [Heroku buildpacks](https://devcenter.heroku.com/articles/buildpacks) -to detect and build the application into a Docker image. +`Dockerfile` is present, Auto Build builds your application using +[Cloud Native Buildpacks](https://buildpacks.io) to detect and build the +application into a Docker image. The feature uses the +[`pack` command](https://github.com/buildpacks/pack). +The default [builder](https://buildpacks.io/docs/concepts/components/builder/) +is `heroku/buildpacks:18` but a different builder can be selected using +the CI/CD variable `AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER`. Each buildpack requires your project's repository to contain certain files for -Auto Build to build your application successfully. For example, your application's +Auto Build to build your application successfully. The structure is +specific to the builder and buildpacks you have selected. +For example, when using the Heroku's builder (the default), your application's root directory must contain the appropriate file for your application's language: @@ -52,39 +61,38 @@ For the requirements of other languages and frameworks, read the [Heroku buildpacks documentation](https://devcenter.heroku.com/articles/buildpacks#officially-supported-buildpacks). NOTE: -If Auto Build fails despite the project meeting the buildpack requirements, set -a project CI/CD variable `TRACE=true` to enable verbose logging, which may help you -troubleshoot. +Auto Test still uses Herokuish, as test suite detection is not +yet part of the Cloud Native Buildpack specification. For more information, see +[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/212689). -### Auto Build using Cloud Native Buildpacks (beta) +### Auto Build using Herokuish -> Introduced in [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28165). +> [Replaced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63351) with Cloud Native Buildpacks in GitLab 14.0. -Auto Build supports building your application using [Cloud Native Buildpacks](https://buildpacks.io) -through the [`pack` command](https://github.com/buildpacks/pack). To use Cloud Native Buildpacks, -set the CI/CD variable `AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED` to a non-empty -value. The default builder is `heroku/buildpacks:18` but a different builder -can be selected using the CI/CD variable `AUTO_DEVOPS_BUILD_IMAGE_CNB_BUILDER`. +Prior to GitLab 14.0, [Herokuish](https://github.com/gliderlabs/herokuish) was +the default build method for projects without a `Dockerfile`. Herokuish can +still be used by setting the CI/CD variable `AUTO_DEVOPS_BUILD_IMAGE_CNB_ENABLED` +to `false`. -Cloud Native Buildpacks (CNBs) are an evolution of Heroku buildpacks, and -GitLab expects them to eventually supersede Herokuish-based builds within Auto DevOps. For more -information, see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/212692). +NOTE: +If Auto Build fails despite the project meeting the buildpack requirements, set +a project CI/CD variable `TRACE=true` to enable verbose logging, which may help you +troubleshoot. + +### Moving from Herokuish to Cloud Native Buildpacks Builds using Cloud Native Buildpacks support the same options as builds using -Heroku buildpacks, with the following caveats: +Herokuish, with the following caveats: - The buildpack must be a Cloud Native Buildpack. A Heroku buildpack can be converted to a Cloud Native Buildpack using Heroku's [`cnb-shim`](https://github.com/heroku/cnb-shim). -- `BUILDPACK_URL` must be in a form +- `BUILDPACK_URL` must be in a format [supported by `pack`](https://buildpacks.io/docs/app-developer-guide/specific-buildpacks/). -- The `/bin/herokuish` command is not present in the resulting image, and prefixing +- The `/bin/herokuish` command is not present in the built image, and prefixing commands with `/bin/herokuish procfile exec` is no longer required (nor possible). - -NOTE: -Auto Test still uses Herokuish, as test suite detection is not -yet part of the Cloud Native Buildpack specification. For more information, see -[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/212689). + Instead, custom commands should be prefixed with `/cnb/lifecycle/launcher` + to receive the correct execution environment. ## Auto Test @@ -208,9 +216,9 @@ documentation. ## Auto Container Scanning **(ULTIMATE)** -Vulnerability Static Analysis for containers uses either [Clair](https://github.com/quay/clair) -or [Trivy](https://aquasecurity.github.io/trivy/latest/) to check for potential security issues in -Docker images. The Auto Container Scanning stage is skipped on licenses other than [Ultimate](https://about.gitlab.com/pricing/). +Vulnerability static analysis for containers uses [Trivy](https://aquasecurity.github.io/trivy/latest/) +to check for potential security issues in Docker images. The Auto Container Scanning stage is +skipped on licenses other than [Ultimate](https://about.gitlab.com/pricing/). After creating the report, it's uploaded as an artifact which you can later download and check out. The merge request displays any detected security issues. @@ -349,7 +357,7 @@ project ID, such as `project-4321`. Auto Deploy does not include deployments to staging or canary environments by default, but the -[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) +[Auto DevOps template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml) contains job definitions for these tasks if you want to enable them. You can use [CI/CD variables](customize.md#cicd-variables) to automatically @@ -461,15 +469,16 @@ If present, `DB_MIGRATE` is run as a shell command within an application pod as a Helm pre-upgrade hook. For example, in a Rails application in an image built with -[Herokuish](https://github.com/gliderlabs/herokuish): +[Cloud Native Buildpacks](#auto-build-using-cloud-native-buildpacks): -- `DB_INITIALIZE` can be set to `RAILS_ENV=production /bin/herokuish procfile exec bin/rails db:setup` -- `DB_MIGRATE` can be set to `RAILS_ENV=production /bin/herokuish procfile exec bin/rails db:migrate` +- `DB_INITIALIZE` can be set to `RAILS_ENV=production /cnb/lifecycle/launcher bin/rails db:setup` +- `DB_MIGRATE` can be set to `RAILS_ENV=production /cnb/lifecycle/launcher bin/rails db:migrate` Unless your repository contains a `Dockerfile`, your image is built with -Herokuish, and you must prefix commands run in these images with -`/bin/herokuish procfile exec` (for Herokuish) or `/cnb/lifecycle/launcher` -(for Cloud Native Buildpacks) to replicate the environment where your +Cloud Native Buildpacks, and you must prefix commands run in these images with +`/cnb/lifecycle/launcher`, (or `/bin/herokuish procfile exec` when +using [Herokuish](#auto-build-using-herokuish)) +to replicate the environment where your application runs. ### Upgrade auto-deploy-app Chart @@ -508,14 +517,10 @@ workers: sidekiq: replicaCount: 1 command: - - /bin/herokuish - - procfile - - exec + - /cnb/lifecycle/launcher - sidekiq preStopCommand: - - /bin/herokuish - - procfile - - exec + - /cnb/lifecycle/launcher - sidekiqctl - quiet terminationGracePeriodSeconds: 60 @@ -645,42 +650,6 @@ ciliumNetworkPolicy: For more information on installing Network Policies, see [Install Cilium using GitLab CI/CD](../../user/clusters/applications.md#install-cilium-using-gitlab-cicd). -### Web Application Firewall (ModSecurity) customization - -> [Introduced](https://gitlab.com/gitlab-org/charts/auto-deploy-app/-/merge_requests/44) in GitLab 12.8. - -Customization on an [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) -or on a deployment base is available for clusters with -[ModSecurity installed](../../user/clusters/applications.md#web-application-firewall-modsecurity). - -To enable ModSecurity with Auto Deploy, you must create a `.gitlab/auto-deploy-values.yaml` -file in your project with the following attributes. - -|Attribute | Description | Default | ------------|-------------|---------| -|`enabled` | Enables custom configuration for ModSecurity, defaulting to the [Core Rule Set](https://coreruleset.org/) | `false` | -|`secRuleEngine` | Configures the [rules engine](https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#secruleengine) | `DetectionOnly` | -|`secRules` | Creates one or more additional [rule](https://github.com/SpiderLabs/ModSecurity/wiki/Reference-Manual-(v2.x)#SecRule) | `nil` | - -In the following `auto-deploy-values.yaml` example, some custom settings -are enabled for ModSecurity. Those include setting its engine to -process rules instead of only logging them, while adding two specific -header-based rules: - -```yaml -ingress: - modSecurity: - enabled: true - secRuleEngine: "On" - secRules: - - variable: "REQUEST_HEADERS:User-Agent" - operator: "printer" - action: "log,deny,id:'2010',status:403,msg:'printer is an invalid agent'" - - variable: "REQUEST_HEADERS:Content-Type" - operator: "text/plain" - action: "log,deny,id:'2011',status:403,msg:'Text is not supported as content type'" -``` - ### Running commands in the container Applications built with [Auto Build](#auto-build) using Herokuish, the default @@ -723,11 +692,6 @@ The metrics include: - **Response Metrics:** latency, throughput, error rate - **System Metrics:** CPU utilization, memory utilization -GitLab provides some initial alerts for you after you install Prometheus: - -- Ingress status code `500` > 0.1% -- NGINX status code `500` > 0.1% - To use Auto Monitoring: 1. [Install and configure the Auto DevOps requirements](requirements.md). diff --git a/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md index 0dabb80204a..62dc061aba6 100644 --- a/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md +++ b/doc/topics/autodevops/upgrading_auto_deploy_dependencies.md @@ -100,9 +100,15 @@ If your Auto DevOps project has an active environment that was deployed with the MIGRATE_HELM_2TO3: "true" .auto-deploy: - image: registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v2.0.0-beta.1 + # Optional: If you are on GitLab 13.12 or older, pin the auto-deploy-image + # image: registry.gitlab.com/gitlab-org/cluster-integration/auto-deploy-image:v2.6.0 variables: AUTO_DEVOPS_FORCE_DEPLOY_V2: 1 + # If you have non-public pipelines, you can back up the entire namespace in a job artifact + # prior to the migration by setting the CI variable BACKUP_NAMESPACE to a non-empty value. + # WARNING: If you have public pipelines, this artifact will be public and can + # expose your secrets. + # BACKUP_HELM2_RELEASES: 1 ``` 1. Run the `<environment-name>:helm-2to3:migrate` job. @@ -110,10 +116,16 @@ If your Auto DevOps project has an active environment that was deployed with the 1. If the deployment succeeds, you can safely run `environment:helm-2to3:cleanup`. This deletes all Helm 2 release data from the namespace. - If you accidentally delete the Helm 2 releases before you are ready, the `<environment-name>:helm2to3:migrate` + If you set `BACKUP_HELM2_RELEASES` to a non-empty value, the `<environment-name>:helm2to3:migrate` job saves a backup for 1 week in a job artifact called `helm-2-release-backups`. - The backup is in a Kubernetes manifest file that can be restored using + If you accidentally delete the Helm 2 releases before you are ready, then + this backup is in a Kubernetes manifest file that can be restored using `kubectl apply -f $backup`. + + **WARNING:** + This artifact can contain secrets and is visible to any + user who can see your job. + 1. Remove the `MIGRATE_HELM_2TO3` CI/CD variable. #### In-Cluster PostgreSQL Channel 2 @@ -164,6 +176,8 @@ include: - remote: https://gitlab.com/gitlab-org/gitlab/-/raw/v13.3.0-ee/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml ``` +Alternatively, you can use the [v13.12 Auto DevOps templates archive](https://gitlab.com/hfyngvason/auto-devops-v13-12). + ### Ignore warnings and continue deploying If you are certain that the new chart version is safe to be deployed, you can add |