diff options
Diffstat (limited to 'doc/development/gitlab_shell')
| -rw-r--r-- | doc/development/gitlab_shell/features.md | 89 | ||||
| -rw-r--r-- | doc/development/gitlab_shell/gitlab_sshd.md | 36 | ||||
| -rw-r--r-- | doc/development/gitlab_shell/index.md | 222 | ||||
| -rw-r--r-- | doc/development/gitlab_shell/process.md | 71 |
4 files changed, 418 insertions, 0 deletions
diff --git a/doc/development/gitlab_shell/features.md b/doc/development/gitlab_shell/features.md new file mode 100644 index 00000000000..f7931c4b94d --- /dev/null +++ b/doc/development/gitlab_shell/features.md @@ -0,0 +1,89 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# GitLab Shell feature list + +## Discover + +Allows users to identify themselves on an instance via SSH. The command helps to +confirm quickly whether a user has SSH access to the instance: + +```shell +ssh git@<hostname> + +PTY allocation request failed on channel 0 +Welcome to GitLab, @username! +Connection to staging.gitlab.com closed. +``` + +When permission is denied, it returns: + +```shell +ssh git@<hostname> +git@<hostname>: Permission denied (publickey). +``` + +## Git operations + +GitLab Shell provides support for Git operations over SSH by processing +`git-upload-pack`, `git-receive-pack` and `git-upload-archive` SSH commands. +It limits the set of commands to predefined Git commands: + +- `git archive` +- `git clone` +- `git pull` +- `git push` + +## Generate new 2FA recovery codes + +Enables users to +[generate new 2FA recovery codes](../../user/profile/account/two_factor_authentication.md#generate-new-recovery-codes-using-ssh): + +```shell +$ ssh git@<hostname> 2fa_recovery_codes + +Are you sure you want to generate new two-factor recovery codes? +Any existing recovery codes you saved will be invalidated. (yes/no) +yes + +Your two-factor authentication recovery codes are: +... +``` + +## Verify 2FA OTP + +Allows users to verify their +[2FA one-time password (OTP)](../../security/two_factor_authentication.md#2fa-for-git-over-ssh-operations): + +```shell +$ ssh git@<hostname> 2fa_verify + +OTP: 347419 + +OTP validation failed. +``` + +## LFS authentication + +Enables users to generate credentials for LFS authentication: + +```shell +$ ssh git@<hostname> git-lfs-authenticate <project-path> <upload/download> + +{"header":{"Authorization":"Basic ..."},"href":"https://gitlab.com/user/project.git/info/lfs","expires_in":7200} +``` + +## Personal access token + +Enables users to use personal access tokens via SSH: + +```shell +$ ssh git@<hostname> personal_access_token <name> <scope1[,scope2,...]> [ttl_days] + +Token: glpat-... +Scopes: api +Expires: 2022-02-05 +``` diff --git a/doc/development/gitlab_shell/gitlab_sshd.md b/doc/development/gitlab_shell/gitlab_sshd.md new file mode 100644 index 00000000000..4c2cd6396c1 --- /dev/null +++ b/doc/development/gitlab_shell/gitlab_sshd.md @@ -0,0 +1,36 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# `gitlab-sshd` in GitLab Shell + +`gitlab-sshd` is a binary in [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) +which runs as a persistent SSH daemon. It is intended to replace `OpenSSH` on GitLab SaaS, +and eventually other cloud-native environments. Instead of running an `sshd` process, +we run a `gitlab-sshd` process that does the same job, in a more focused manner: + +```mermaid +sequenceDiagram + participant Git on client + participant GitLab SSHD + participant Rails + participant Gitaly + participant Git on server + + Note left of Git on client: git fetch + Git on client->>+GitLab SSHD: ssh git fetch-pack request + GitLab SSHD->>+Rails: GET /internal/api/authorized_keys?key=AAAA... + Note right of Rails: Lookup key ID + Rails-->>-GitLab SSHD: 200 OK, command="gitlab-shell upload-pack key_id=1" + GitLab SSHD->>+Rails: GET /internal/api/allowed?action=upload_pack&key_id=1 + Note right of Rails: Auth check + Rails-->>-GitLab SSHD: 200 OK, { gitaly: ... } + GitLab SSHD->>+Gitaly: SSHService.SSHUploadPack request + Gitaly->>+Git on server: git upload-pack request + Note over Git on client,Git on server: Bidirectional communication between Git client and server + Git on server-->>-Gitaly: git upload-pack response + Gitaly -->>-GitLab SSHD: SSHService.SSHUploadPack response + GitLab SSHD-->>-Git on client: ssh git fetch-pack response +``` diff --git a/doc/development/gitlab_shell/index.md b/doc/development/gitlab_shell/index.md new file mode 100644 index 00000000000..7f2c113fa0c --- /dev/null +++ b/doc/development/gitlab_shell/index.md @@ -0,0 +1,222 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab Shell + +[](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main) [](https://gitlab.com/gitlab-org/gitlab-shell/-/pipelines?ref=main) [](https://codeclimate.com/github/gitlabhq/gitlab-shell) + +GitLab Shell handles Git SSH sessions for GitLab and modifies the list of authorized keys. +GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh. + +GitLab supports Git LFS authentication through SSH. + +## Requirements + +GitLab Shell is written in Go, and needs a Go compiler to build. It still requires +Ruby to build and test, but not to run. + +GitLab Shell runs on `port 22` on an Omnibus installation. To use a regular SSH +service, configure it on an alternative port. + +Download and install the current version of Go from [golang.org](https://golang.org/dl/). +We follow the [Golang Release Policy](https://golang.org/doc/devel/release.html#policy) +and support: + +- The current stable version. +- The previous two major versions. + +## How GitLab Shell works + +When you access the GitLab server over SSH, GitLab Shell then: + +1. Limits you to predefined Git commands (`git push`, `git pull`, `git fetch`). +1. Calls the GitLab Rails API to check if you are authorized, and what Gitaly server your repository is on. +1. Copies data back and forth between the SSH client and the Gitaly server. + +If you access a GitLab server over HTTP(S) you end up in [`gitlab-workhorse`](../workhorse/index.md). + +### `git pull` over SSH + +```mermaid +graph LR + A[Git pull] --> |via SSH| B[gitlab-shell] + B -->|API call| C[gitlab-rails<br>authorization] + C -->|accept or decline| D[Gitaly session] +``` + +### `git push` over SSH + +The `git push` command is not performed until after `gitlab-rails` accepts the push: + +```mermaid +graph LR +subgraph User initiates + A[Git push] -->|via SSH| B[gitlab-shell] +end +subgraph Gitaly + B -->|establish Gitaly session| C[gitlab-shell pre-receive hook] + C -->|API auth call| D[Gitlab-rails] + D --> E[accept or decline push] +end +``` + +[Full feature list](features.md) + +### Modifies `authorized_keys` + +GitLab Shell modifies the `authorized_keys` file on the client machine. + +## Contribute to GitLab Shell + +To contribute to GitLab Shell: + +1. Check if GitLab API access, and Redis via the internal API, can be reached: `make check` +1. Compile the `gitlab-shell` binaries, placing them into `bin/`: `make compile` +1. Run `make install` to build the `gitlab-shell` binaries and install. them onto the file system. + The default location is `/usr/local`. To change it, set the `PREFIX` and `DESTDIR` environment variables. +1. To install GitLab from source on a single machine, run `make setup`. + It compiles the GitLab Shell binaries, and ensures that various paths on the file system + exist with the correct permissions. Do not run this command unless your installation method + documentation instructs you to. + +For more information, see +[CONTRIBUTING.md](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/CONTRIBUTING.md). + +### Run tests + +When contributing, run tests: + +1. Run tests with `bundle install` and `make test`. +1. Run Gofmt: `make verify` +1. Run both test and verify (the default Makefile target): + + ```shell + bundle install + make validate + ``` + +1. If needed, configure Gitaly. + +### Configure Gitaly for local testing + +Some tests need a Gitaly server. The +[`docker-compose.yml`](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/docker-compose.yml) file runs Gitaly on port 8075. +To tell the tests where Gitaly is, set `GITALY_CONNECTION_INFO`: + +```plaintext +export GITALY_CONNECTION_INFO='{"address": "tcp://localhost:8075", "storage": "default"}' +make test +``` + +If no `GITALY_CONNECTION_INFO` is set, the test suite still runs, but any +tests requiring Gitaly are skipped. The tests always run in the CI environment. + +## Rate limiting + +GitLab Shell performs rate-limiting by user account and project for Git operations. +GitLab Shell accepts Git operation requests and then makes a call to the Rails +rate-limiter, backed by Redis. If the `user + project` exceeds the rate limit, +then GitLab Shell then drop further connection requests for that `user + project`. + +The rate-limiter is applied at the Git command (plumbing) level. Each command has +a rate limit of 600 per minute. For example, `git push` has 600 per minute, and +`git pull` has another 600 per minute. + +Because they are using the same plumbing command, `git-upload-pack`, `git pull`, +and `git clone` are in effect the same command for the purposes of rate-limiting. + +Gitaly also has a rate-limiter in place, but calls are never made to Gitaly if +the rate limit is exceeded in GitLab Shell (Rails). + +## Logs in GitLab Shell + +In general, you can determine the structure, but not content, of a GitLab Shell +or `gitlab-sshd` session by inspecting the logs. Some guidelines: + +- We use [`gitlab.com/gitlab-org/labkit/log`](https://pkg.go.dev/gitlab.com/gitlab-org/labkit/log) + for logging. +- Always include a correlation ID. +- Log messages should be invariant and unique. Include accessory information in + fields, using `log.WithField`, `log.WithFields`, or `log.WithError`. +- Log both success cases and error cases. +- Logging too much is better than not logging enough. If a message seems too + verbose, consider reducing the log level before removing the message. + +## GitLab SaaS + +A diagram of the flow of `gitlab-shell` on GitLab.com: + +```mermaid +graph LR + a2 --> b2 + a2 --> b3 + a2 --> b4 + b2 --> c1 + b3 --> c1 + b4 --> c1 + c2 --> d1 + c2 --> d2 + c2 --> d3 + d1 --> e1 + d2 --> e1 + d3 --> e1 + a1[Cloudflare] --> a2[TCP<br/> load balancer] + e1[Git] + + subgraph HAProxy Fleet + b2[HAProxy] + b3[HAProxy] + b4[HAProxy] + end + + subgraph GKE + c1[Internal TCP<br/> load balancer<br/>port 2222] --> c2[GitLab-shell<br/> pods] + end + + subgraph Gitaly + d1[Gitaly] + d2[Gitaly] + d3[Gitaly] + end +``` + +## GitLab Shell architecture + +```mermaid +sequenceDiagram + participant Git on client + participant SSH server + participant AuthorizedKeysCommand + participant GitLab Shell + participant Rails + participant Gitaly + participant Git on server + + Note left of Git on client: git fetch + Git on client->>+SSH server: ssh git fetch-pack request + SSH server->>+AuthorizedKeysCommand: gitlab-shell-authorized-keys-check git AAAA... + AuthorizedKeysCommand->>+Rails: GET /internal/api/authorized_keys?key=AAAA... + Note right of Rails: Lookup key ID + Rails-->>-AuthorizedKeysCommand: 200 OK, command="gitlab-shell upload-pack key_id=1" + AuthorizedKeysCommand-->>-SSH server: command="gitlab-shell upload-pack key_id=1" + SSH server->>+GitLab Shell: gitlab-shell upload-pack key_id=1 + GitLab Shell->>+Rails: GET /internal/api/allowed?action=upload_pack&key_id=1 + Note right of Rails: Auth check + Rails-->>-GitLab Shell: 200 OK, { gitaly: ... } + GitLab Shell->>+Gitaly: SSHService.SSHUploadPack request + Gitaly->>+Git on server: git upload-pack request + Note over Git on client,Git on server: Bidirectional communication between Git client and server + Git on server-->>-Gitaly: git upload-pack response + Gitaly -->>-GitLab Shell: SSHService.SSHUploadPack response + GitLab Shell-->>-SSH server: gitlab-shell upload-pack response + SSH server-->>-Git on client: ssh git fetch-pack response +``` + +## Related topics + +- [LICENSE](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/LICENSE). +- [PROCESS.md](https://gitlab.com/gitlab-org/gitlab-shell/-/blob/main/PROCESS.md) +- [Using the GitLab Shell chart](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/) diff --git a/doc/development/gitlab_shell/process.md b/doc/development/gitlab_shell/process.md new file mode 100644 index 00000000000..cc6f44b865c --- /dev/null +++ b/doc/development/gitlab_shell/process.md @@ -0,0 +1,71 @@ +--- +stage: Create +group: Source Code +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Processes for GitLab Shell + +## Releasing a new version + +GitLab Shell is versioned by Git tags, and the version used by the Rails +application is stored in +[`GITLAB_SHELL_VERSION`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_SHELL_VERSION). + +For each version, there is a raw version and a tag version: + +- The **raw version** is the version number. For instance, `15.2.8`. +- The **tag version** is the raw version prefixed with `v`. For instance, `v15.2.8`. + +To release a new version of GitLab Shell and have that version available to the +Rails application: + +1. Create a merge request to update the [`CHANGELOG`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/CHANGELOG.md) with the + **tag version** and the [`VERSION`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/VERSION) file with the **raw version**. +1. Ask a maintainer to review and merge the merge request. If you're already a + maintainer, second maintainer review is not required. +1. Add a new Git tag with the **tag version**. +1. Update `GITLAB_SHELL_VERSION` in the Rails application to the **raw + version**. + + NOTE: + This can be done as a separate merge request, or in a merge request + that uses the latest GitLab Shell changes. + +## Security releases + +GitLab Shell is included in the packages we create for GitLab. Each version of +GitLab specifies the version of GitLab Shell it uses in the `GITLAB_SHELL_VERSION` +file. Because of this specification, security fixes in GitLab Shell are tightly coupled to the +[GitLab security release](https://about.gitlab.com/handbook/engineering/workflow/#security-issues) workflow. + +For a security fix in GitLab Shell, two sets of merge requests are required: + +1. The fix itself, in the `gitlab-org/security/gitlab-shell` repository and its + backports to the previous versions of GitLab Shell. +1. Merge requests to change the versions of GitLab Shell included in the GitLab + security release, in the `gitlab-org/security/gitlab` repository. + +The first step could be to create a merge request with a fix targeting `main` +in `gitlab-org/security/gitlab-shell`. When the merge request is approved by maintainers, +backports targeting previous 3 versions of GitLab Shell must be created. The stable +branches for those versions may not exist, so feel free to ask a maintainer to create +them. The stable branches must be created out of the GitLab Shell tags or versions +used by the 3 previous GitLab releases. + +To find out the GitLab Shell version used on a particular GitLab stable release, +run this command, replacing `13-9-stable-ee` with the version you're interested in. +These commands show the version used by the `13.9` version of GitLab: + +```shell +git fetch security 13-9-stable-ee +git show refs/remotes/security/13-9-stable-ee:GITLAB_SHELL_VERSION +``` + +Close to the GitLab security release, a maintainer should merge the fix and backports, +and cut all the necessary GitLab Shell versions. This allows bumping the +`GITLAB_SHELL_VERSION` for `gitlab-org/security/gitlab`. The GitLab merge request +is handled by the general GitLab security release process. + +After the security release is done, a GitLab Shell maintainer is responsible for +syncing tags and `main` to the `gitlab-org/gitlab-shell` repository. |