diff options
Diffstat (limited to 'doc/development/fips_compliance.md')
| -rw-r--r-- | doc/development/fips_compliance.md | 411 |
1 files changed, 411 insertions, 0 deletions
diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md index 8fe5af56f9d..d4274c6275b 100644 --- a/doc/development/fips_compliance.md +++ b/doc/development/fips_compliance.md @@ -97,3 +97,414 @@ virtual machine: ```shell fips-mode-setup --disable ``` + +#### Detect FIPS enablement in code + +You can query `GitLab::FIPS` in Ruby code to determine if the instance is FIPS-enabled: + +```ruby +def default_min_key_size(name) + if Gitlab::FIPS.enabled? + Gitlab::SSHPublicKey.supported_sizes(name).select(&:positive?).min || -1 + else + 0 + end +end +``` + +## Nightly Omnibus FIPS builds + +The Distribution team has created [nightly FIPS Omnibus builds](https://packages.gitlab.com/gitlab/nightly-fips-builds). These +GitLab builds are compiled to use the system OpenSSL instead of the Omnibus-embedded version of OpenSSL. + +See [the section on how FIPS builds are created](#how-fips-builds-are-created). + +## Runner + +See the [documentation on installing a FIPS-compliant GitLab Runner](https://docs.gitlab.com/runner/install/#fips-compliant-gitlab-runner). + +## Set up a FIPS-enabled cluster + +You can use the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) to spin +up a FIPS-enabled cluster for development and testing. These instructions use Amazon Web Services (AWS) +because that is the first target environment, but you can adapt them for other providers. + +### Set up your environment + +To get started, your AWS account must subscribe to a FIPS-enabled Amazon +Machine Image (AMI) in the [AWS Marketplace console](https://aws.amazon.com/premiumsupport/knowledge-center/launch-ec2-marketplace-subscription/). + +This example assumes that the `Ubuntu Pro 20.04 FIPS LTS` AMI by +`Canonical Group Limited` has been added your account. This operating +system is used for virtual machines running in Amazon EC2. + +### Omnibus + +The simplest way to get a FIPS-enabled GitLab cluster is to use an Omnibus reference architecture. +See the [GET Quick Start Guide](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/docs/environment_quick_start_guide.md) +for more details. The following instructions build on the Quick Start and are also necessary for [Cloud Native Hybrid](#cloud-native-hybrid) installations. + +#### Terraform: Use a FIPS AMI + +1. Follow the guide to set up Terraform and Ansible. +1. After [step 2b](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/docs/environment_quick_start_guide.md#2b-setup-config), + create a `data.tf` in your environment (for example, `gitlab-environment-toolkit/terraform/environments/gitlab-10k/inventory/data.tf`): + + ```tf + data "aws_ami" "ubuntu_20_04_fips" { + count = 1 + + most_recent = true + + filter { + name = "name" + values = ["ubuntu-pro-fips-server/images/hvm-ssd/ubuntu-focal-20.04-amd64-pro-fips-server-*"] + } + + filter { + name = "virtualization-type" + values = ["hvm"] + } + + owners = ["aws-marketplace"] + } + ``` + +1. Add the custom `ami_id` to use this AMI in `environment.tf`. For + example, in `gitlab-environment-toolkit/terraform/environments/gitlab-10k/inventory/environment.tf`: + + ```tf + module "gitlab_ref_arch_aws" { + source = "../../modules/gitlab_ref_arch_aws" + + prefix = var.prefix + ami_id = data.aws_ami.ubuntu_20_04_fips[0].id + ... + ``` + +NOTE: +GET does not allow the AMI to change on EC2 instances after it has +been deployed via `terraform apply`. Since an AMI change would tear down +an instance, this would result in data loss: not only would disks be +destroyed, but also GitLab secrets would be lost. There is a [Terraform lifecycle rule](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/blob/2aaeaff8ac8067f23cd7b6bb5bf131061649089d/terraform/modules/gitlab_aws_instance/main.tf#L40) +to ignore AMI changes. + +#### Ansible: Specify the FIPS Omnibus builds + +The standard Omnibus GitLab releases build their own OpenSSL library, +which is not FIPS-validated. However, we have nightly builds that create +Omnibus packages that link against the operating system's OpenSSL library. To +use this package, update the `gitlab_repo_script_url` field in the +Ansible `vars.yml`. For example, you might modify +`gitlab-environment-toolkit/ansible/environments/gitlab-10k/inventory/vars.yml` +in this way: + +```yaml +all: + vars: + ... + gitlab_repo_script_url: "https://packages.gitlab.com/install/repositories/gitlab/nightly-fips-builds/script.deb.sh" +``` + +### Cloud Native Hybrid + +A Cloud Native Hybrid install uses both Omnibus and Cloud Native GitLab +(CNG) images. The previous instructions cover the Omnibus part, but two +additional steps are needed to enable FIPS in CNG: + +1. Use a custom Amazon Elastic Kubernetes Service (EKS) AMI. +1. Use GitLab containers built with RedHat's Universal Base Image (UBI). + +#### Build a custom EKS AMI + +Because Amazon does not yet publish a FIPS-enabled AMI, you have to +build one yourself with Packer. + +Amazon publishes the following Git repositories with information about custom EKS AMIs: + +- [Amazon EKS AMI Build Specification](https://github.com/awslabs/amazon-eks-ami) +- [Sample EKS custom AMIs](https://github.com/aws-samples/amazon-eks-custom-amis/) + +This [GitHub pull request](https://github.com/awslabs/amazon-eks-ami/pull/898) makes +it possible to create an Amazon Linux 2 EKS AMI with FIPS enabled for Kubernetes v1.21. +To build an image: + +1. [Install Packer](https://learn.hashicorp.com/tutorials/packer/get-started-install-cli). +1. Run the following: + + ```shell + git clone https://github.com/awslabs/amazon-eks-ami + cd amazon-eks-ami + git fetch origin pull/898/head:fips-ami + git checkout fips-ami + AWS_DEFAULT_REGION=us-east-1 make 1.21-fips # Be sure to set the region accordingly + ``` + +If you are using a different version of Kubernetes, adjust the `make` +command and `Makefile` accordingly. + +When the AMI build is done, a new AMI should be created with a message +such as the following: + +```plaintext +==> Builds finished. The artifacts of successful builds are: +--> amazon-ebs: AMIs were created: +us-west-2: ami-0a25e760cd00b027e +``` + +In this example, the AMI ID is `ami-0a25e760cd00b027e`, but your value may +be different. + +Building a RHEL-based system with FIPS enabled should be possible, but +there is [an outstanding issue preventing the Packer build from completing](https://github.com/aws-samples/amazon-eks-custom-amis/issues/51). + +#### Terraform: Use a custom EKS AMI + +Now you can set the custom EKS AMI. + +1. In `environment.tf`, add `eks_ami_id = var.eks_ami_id` so you can pass this variable to the + AWS reference architecture module. For example, in + `gitlab-environment-toolkit/terraform/environments/gitlab-10k/inventory/environment.tf`: + + ```tf + module "gitlab_ref_arch_aws" { + source = "../../modules/gitlab_ref_arch_aws" + + prefix = var.prefix + ami_id = data.aws_ami.ubuntu_20_04_fips[0].id + eks_ami_id = var.eks_ami_id + .... + ``` + +1. In `variables.tf`, define a `eks_ami_id` with the AMI ID in the + previous step: + + ```tf + variable "eks_ami_id" { + default = "ami-0a25e760cd00b027e" + } + ``` + +#### Ansible: Use UBI images + +CNG uses a Helm Chart to manage which container images to deploy. By default, GET +deploys the latest released versions that use Debian-based containers. + +To switch to UBI-based containers, edit the Ansible `vars.yml` to use custom +Charts variables: + +```yaml +all: + vars: + ... + gitlab_charts_custom_config_file: '/path/to/gitlab-environment-toolkit/ansible/environments/gitlab-10k/inventory/charts.yml' +``` + +Now create `charts.yml` in the location specified above and specify tags with a `-ubi8` suffix. For example: + +```yaml +global: + image: + pullPolicy: Always + certificates: + image: + tag: master-ubi8 + +gitlab: + gitaly: + image: + tag: master-ubi8 + gitlab-exporter: + image: + tag: master-ubi8 + gitlab-shell: + image: + tag: main-ubi8 # The default branch is main, not master + gitlab-mailroom: + image: + tag: master-ubi8 + migrations: + image: + tag: master-ubi8 + sidekiq: + image: + tag: master-ubi8 + toolbox: + image: + tag: master-ubi8 + webservice: + image: + tag: master-ubi8 + workhorse: + tag: master-ubi8 + +nginx-ingress: + controller: + image: + repository: registry.gitlab.com/stanhu/gitlab-test-images/k8s-staging-ingress-nginx/controller + tag: v1.2.0-beta.1 + pullPolicy: Always + digest: sha256:ace38833689ad34db4a46bc1e099242696eb800def88f02200a8615530734116 +``` + +The above example shows a FIPS-enabled [`nginx-ingress`](https://github.com/kubernetes/ingress-nginx) image. +See [this issue](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/3153#note_917782207) for more details on +how to build NGINX and the Ingress Controller. + +You can also use release tags, but the versioning is tricky because each +component may use its own versioning scheme. For example, for GitLab v14.10: + +```yaml +global: + certificates: + image: + tag: 20191127-r2-ubi8 + +gitlab: + gitaly: + image: + tag: v14.10.0-ubi8 + gitlab-exporter: + image: + tag: 11.14.0-ubi8 + gitlab-shell: + image: + tag: v13.25.1-ubi8 + gitlab-mailroom: + image: + tag: v14.10.0-ubi8 + migrations: + image: + tag: v14.10.0-ubi8 + sidekiq: + image: + tag: v14.10.0-ubi8 + toolbox: + image: + tag: v14.10.0-ubi8 + webservice: + image: + tag: v14.10.0-ubi8 + workhorse: + tag: v14.10.0-ubi8 +``` + +## Verify FIPS + +The following sections describe ways you can verify if FIPS is enabled. + +### Kernel + +```shell +$ cat /proc/sys/crypto/fips_enabled +1 +``` + +### Ruby (Omnibus images) + +```ruby +$ /opt/gitlab/embedded/bin/irb +irb(main):001:0> require 'openssl'; OpenSSL.fips_mode +=> true +``` + +### Ruby (CNG images) + +```ruby +$ irb +irb(main):001:0> require 'openssl'; OpenSSL.fips_mode +=> true +``` + +### Go + +Google maintains a [`dev.boringcrypto` branch](https://github.com/golang/go/tree/dev.boringcrypto) in the Golang compiler +that makes it possible to statically link BoringSSL, a FIPS-validated module forked from OpenSSL. +However, BoringSSL is not intended for public use. + +We use [`golang-fips`](https://github.com/golang-fips/go), [a fork of the `dev.boringcrypto` branch](https://github.com/golang/go/blob/2fb6bf8a4a51f92f98c2ae127eff2b7ac392c08f/README.boringcrypto.md) to build Go programs that +[dynamically link OpenSSL via `dlopen`](https://github.com/golang-fips/go/blob/go1.18.1-1-openssl-fips/src/crypto/internal/boring/boring.go#L47-L65). This has several advantages: + +- Using a FIPS-validated, system OpenSSL is straightforward. +- This is the source code used by [Red Hat's go-toolset package](https://gitlab.com/redhat/centos-stream/rpms/golang#sources). +- Unlike [go-toolset](https://developers.redhat.com/blog/2019/06/24/go-and-fips-140-2-on-red-hat-enterprise-linux#), this fork appears to keep up with the latest Go releases. + +However, [cgo](https://pkg.go.dev/cmd/cgo) must be enabled via `CGO_ENABLED=1` for this to work. There +is a performance hit when calling into C code. + +Projects that are compiled with `golang-fips` on Linux x86 automatically +get built the crypto routines that use OpenSSL. While the `boringcrypto` +build tag is automatically present, no extra build tags are actually +needed. There are [specific build tags](https://github.com/golang-fips/go/blob/go1.18.1-1-openssl-fips/src/crypto/internal/boring/boring.go#L6) +that disable these crypto hooks. + +We can [check whether a given binary is using OpenSSL](https://go.googlesource.com/go/+/dev.boringcrypto/misc/boring/#caveat) via `go tool nm` +and look for symbols named `Cfunc__goboringcrypto`. For example: + +```plaintext +$ go tool nm nginx-ingress-controller | grep Cfunc__goboringcrypto | tail + 2a0b650 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA384_Final + 2a0b658 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA384_Init + 2a0b660 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA384_Update + 2a0b668 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA512_Final + 2a0b670 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA512_Init + 2a0b678 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_SHA512_Update + 2a0b680 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ECDSA_sign + 2a0b688 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ECDSA_verify + 2a0b690 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ERR_error_string_n + 2a0b698 D crypto/internal/boring._cgo_71ae3cd1ca33_Cfunc__goboringcrypto_internal_ERR_get_error +``` + +In addition, LabKit contains routines to [check whether FIPS is enabled](https://gitlab.com/gitlab-org/labkit/-/tree/master/fips). + +## How FIPS builds are created + +Many GitLab projects (for example: Gitaly, GitLab Pages) have +standardized on using `FIPS_MODE=1 make` to build FIPS binaries locally. + +### Omnibus + +The Omnibus FIPS builds are triggered with the `USE_SYSTEM_SSL` +environment variable set to `true`. When this environment variable is +set, the Omnibus recipes dependencies such as `curl`, NGINX, and libgit2 +will link against the system OpenSSL. OpenSSL will NOT be included in +the Omnibus build. + +The Omnibus builds are created using container images [that use the `golang-fips` compiler](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/blob/master/docker/snippets/go_fips). For +example, [this job](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/jobs/2363742108) created +the `registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/centos_8_fips:3.3.1` image used to +build packages for RHEL 8. + +#### Add a new FIPS build for another Linux distribution + +First, you need to make sure there is an Omnibus builder image for the +desired Linux distribution. The images used to build Omnibus packages are +created with [Omnibus Builder images](https://gitlab.com/gitlab-org/gitlab-omnibus-builder). + +Review [this merge request](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/merge_requests/218). A +new image can be added by: + +1. Adding CI jobs with the `_fips` suffix (for example: `ubuntu_18.04_fips`). +1. Making sure the `Dockerfile` uses `Snippets.new(fips: fips).populate` instead of `Snippets.new.populate`. + +After this image has been tagged, add a new [CI job to Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/911fbaccc08398dfc4779be003ea18014b3e30e9/gitlab-ci-config/dev-gitlab-org.yml#L594-602). + +### Cloud Native GitLab (CNG) + +The Cloud Native GitLab CI pipeline generates images using several base images: + +- Debian +- [Red Hat's Universal Base Image (UBI)](https://developers.redhat.com/products/rhel/ubi) + +UBI images ship with the same OpenSSL package as those used by +RHEL. This makes it possible to build FIPS-compliant binaries without +needing RHEL. Note that RHEL 8.2 ships a [FIPS-validated OpenSSL](https://access.redhat.com/articles/2918071), but 8.5 is in +review for FIPS validation. + +[This merge request](https://gitlab.com/gitlab-org/build/CNG/-/merge_requests/981) +introduces a FIPS pipeline for CNG images. Images tagged for FIPS have the `-fips` suffix. For example, +the `webservice` container has the following tags: + +- `master` +- `master-ubi8` +- `master-fips` |