diff options
Diffstat (limited to 'doc/administration')
| -rw-r--r-- | doc/administration/auth/jwt.md | 2 | ||||
| -rw-r--r-- | doc/administration/high_availability/nfs.md | 94 | ||||
| -rw-r--r-- | doc/administration/high_availability/redis.md | 2 | ||||
| -rw-r--r-- | doc/administration/index.md | 1 | ||||
| -rw-r--r-- | doc/administration/job_artifacts.md | 4 | ||||
| -rw-r--r-- | doc/administration/job_traces.md | 95 | ||||
| -rw-r--r-- | doc/administration/monitoring/prometheus/gitlab_metrics.md | 2 | ||||
| -rw-r--r-- | doc/administration/operations/fast_ssh_key_lookup.md | 2 | ||||
| -rw-r--r-- | doc/administration/repository_checks.md | 26 | ||||
| -rw-r--r-- | doc/administration/uploads.md | 2 |
10 files changed, 174 insertions, 56 deletions
diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index b51e705ab52..8b00f52ffc1 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -50,7 +50,7 @@ JWT will provide you with a secret key for you to use. required_claims: ["name", "email"], info_map: { name: "name", email: "email" }, auth_url: 'https://example.com/', - valid_within: nil, + valid_within: null, } } ``` diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index d8928a7fe4c..ad8ffc46559 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -75,43 +75,33 @@ Notice several options that you should consider using: | `nobootwait` | Don't halt boot process waiting for this mount to become available | `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. -## Mount locations +## A single NFS mount -When using default Omnibus configuration you will need to share 5 data locations -between all GitLab cluster nodes. No other locations should be shared. The -following are the 5 locations you need to mount: - -| Location | Description | Default configuration | -| -------- | ----------- | --------------------- | -| `/var/opt/gitlab/git-data` | Git repository data. This will account for a large portion of your data | `git_data_dirs({"default" => "/var/opt/gitlab/git-data"})` -| `/var/opt/gitlab/.ssh` | SSH `authorized_keys` file and keys used to import repositories from some other Git services | `user['home'] = '/var/opt/gitlab/'` -| `/var/opt/gitlab/gitlab-rails/uploads` | User uploaded attachments | `gitlab_rails['uploads_directory'] = '/var/opt/gitlab/gitlab-rails/uploads'` -| `/var/opt/gitlab/gitlab-rails/shared` | Build artifacts, GitLab Pages, LFS objects, temp files, etc. If you're using LFS this may also account for a large portion of your data | `gitlab_rails['shared_path'] = '/var/opt/gitlab/gitlab-rails/shared'` -| `/var/opt/gitlab/gitlab-ci/builds` | GitLab CI build traces | `gitlab_ci['builds_directory'] = '/var/opt/gitlab/gitlab-ci/builds'` +It's recommended to nest all gitlab data dirs within a mount, that allows automatic +restore of backups without manually moving existing data. -Other GitLab directories should not be shared between nodes. They contain -node-specific files and GitLab code that does not need to be shared. To ship -logs to a central location consider using remote syslog. GitLab Omnibus packages -provide configuration for [UDP log shipping][udp-log-shipping]. - -### Consolidating mount points - -If you don't want to configure 5-6 different NFS mount points, you have a few -alternative options. +``` +mountpoint +└── gitlab-data + ├── builds + ├── git-data + ├── home-git + ├── shared + └── uploads +``` -#### Change default file locations +To do so, we'll need to configure Omnibus with the paths to each directory nested +in the mount point as follows: -Omnibus allows you to configure the file locations. With custom configuration -you can specify just one main mountpoint and have all of these locations -as subdirectories. Mount `/gitlab-data` then use the following Omnibus +Mount `/gitlab-nfs` then use the following Omnibus configuration to move each data location to a subdirectory: ```ruby -git_data_dirs({"default" => "/gitlab-data/git-data"}) -user['home'] = '/gitlab-data/home' -gitlab_rails['uploads_directory'] = '/gitlab-data/uploads' -gitlab_rails['shared_path'] = '/gitlab-data/shared' -gitlab_ci['builds_directory'] = '/gitlab-data/builds' +git_data_dirs({"default" => "/gitlab-nfs/gitlab-data/git-data"}) +user['home'] = '/gitlab-nfs/gitlab-data/home' +gitlab_rails['uploads_directory'] = '/gitlab-nfs/gitlab-data/uploads' +gitlab_rails['shared_path'] = '/gitlab-nfs/gitlab-data/shared' +gitlab_ci['builds_directory'] = '/gitlab-nfs/gitlab-data/builds' ``` To move the `git` home directory, all GitLab services must be stopped. Run @@ -122,22 +112,52 @@ Run `sudo gitlab-ctl reconfigure` to start using the central location. Please be aware that if you had existing data you will need to manually copy/rsync it to these new locations and then restart GitLab. -#### Bind mounts +## Bind mounts + +Alternatively to changing the configuration in Omnibus, bind mounts can be used +to store the data on an NFS mount. Bind mounts provide a way to specify just one NFS mount and then bind the default GitLab data locations to the NFS mount. Start by defining your single NFS mount point as you normally would in `/etc/fstab`. Let's assume your -NFS mount point is `/gitlab-data`. Then, add the following bind mounts in +NFS mount point is `/gitlab-nfs`. Then, add the following bind mounts in `/etc/fstab`: ```bash -/gitlab-data/git-data /var/opt/gitlab/git-data none bind 0 0 -/gitlab-data/.ssh /var/opt/gitlab/.ssh none bind 0 0 -/gitlab-data/uploads /var/opt/gitlab/gitlab-rails/uploads none bind 0 0 -/gitlab-data/shared /var/opt/gitlab/gitlab-rails/shared none bind 0 0 -/gitlab-data/builds /var/opt/gitlab/gitlab-ci/builds none bind 0 0 +/gitlab-nfs/gitlab-data/git-data /var/opt/gitlab/git-data none bind 0 0 +/gitlab-nfs/gitlab-data/.ssh /var/opt/gitlab/.ssh none bind 0 0 +/gitlab-nfs/gitlab-data/uploads /var/opt/gitlab/gitlab-rails/uploads none bind 0 0 +/gitlab-nfs/gitlab-data/shared /var/opt/gitlab/gitlab-rails/shared none bind 0 0 +/gitlab-nfs/gitlab-data/builds /var/opt/gitlab/gitlab-ci/builds none bind 0 0 ``` +Using bind mounts will require manually making sure the data directories +are empty before attempting a restore. Read more about the +[restore prerequisites](../../raketasks/backup_restore.md). + +## Multiple NFS mounts + +When using default Omnibus configuration you will need to share 5 data locations +between all GitLab cluster nodes. No other locations should be shared. The +following are the 5 locations need to be shared: + +| Location | Description | Default configuration | +| -------- | ----------- | --------------------- | +| `/var/opt/gitlab/git-data` | Git repository data. This will account for a large portion of your data | `git_data_dirs({"default" => "/var/opt/gitlab/git-data"})` +| `/var/opt/gitlab/.ssh` | SSH `authorized_keys` file and keys used to import repositories from some other Git services | `user['home'] = '/var/opt/gitlab/'` +| `/var/opt/gitlab/gitlab-rails/uploads` | User uploaded attachments | `gitlab_rails['uploads_directory'] = '/var/opt/gitlab/gitlab-rails/uploads'` +| `/var/opt/gitlab/gitlab-rails/shared` | Build artifacts, GitLab Pages, LFS objects, temp files, etc. If you're using LFS this may also account for a large portion of your data | `gitlab_rails['shared_path'] = '/var/opt/gitlab/gitlab-rails/shared'` +| `/var/opt/gitlab/gitlab-ci/builds` | GitLab CI build traces | `gitlab_ci['builds_directory'] = '/var/opt/gitlab/gitlab-ci/builds'` + +Other GitLab directories should not be shared between nodes. They contain +node-specific files and GitLab code that does not need to be shared. To ship +logs to a central location consider using remote syslog. GitLab Omnibus packages +provide configuration for [UDP log shipping][udp-log-shipping]. + +Having multiple NFS mounts will require manually making sure the data directories +are empty before attempting a restore. Read more about the +[restore prerequisites](../../raketasks/backup_restore.md). + --- Read more on high-availability configuration: diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 430f865f1e7..031fb31ca4f 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -323,7 +323,7 @@ The prerequisites for a HA Redis setup are the following: # machines to connect to it. redis['port'] = 6379 - # The same password for Redeis authentication you set up for the master node. + # The same password for Redis authentication you set up for the master node. redis['password'] = 'redis-password-goes-here' # The IP of the master Redis node. diff --git a/doc/administration/index.md b/doc/administration/index.md index b472ca5b4d8..5551a04959c 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -40,6 +40,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. [source installations](../install/installation.md#installation-from-source). - [Environment variables](environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. - [Plugins](plugins.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. +- [Enforcing Terms of Service](../user/admin_area/settings/terms.md) #### Customizing GitLab's appearance diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 896cb93e5ed..77fe4d561a1 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -107,7 +107,7 @@ For source installations the following settings are nested under `artifacts:` an | Setting | Description | Default | |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | -| `remote_directory` | The bucket name where Artfacts will be stored| | +| `remote_directory` | The bucket name where Artifacts will be stored| | | `direct_upload` | Set to true to enable direct upload of Artifacts without the need of local shared storage. Option may be removed once we decide to support only single storage for all files. Currently only `Google` provider is supported | `false` | | `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 | `true` | | `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | @@ -148,7 +148,7 @@ _The artifacts are stored by default in ``` NOTE: For GitLab 9.4+, if you are using AWS IAM profiles, be sure to omit the - AWS access key and secret acces key/value pairs. For example: + AWS access key and secret access key/value pairs. For example: ```ruby gitlab_rails['artifacts_object_store_connection'] = { diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md index 84a1ffeec98..f0b2054a7f3 100644 --- a/doc/administration/job_traces.md +++ b/doc/administration/job_traces.md @@ -40,3 +40,98 @@ To change the location where the job logs will be stored, follow the steps below [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" [restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" + +## New live trace architecture + +> [Introduced][ce-18169] in GitLab 10.4. + +> **Notes**: +- This feature is still Beta, which could impact GitLab.com/on-premises instances, and in the worst case scenario, traces will be lost. +- This feature is still being discussed in [an issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/46097) for the performance improvements. +- This feature is off by default. Please check below how to enable/disable this featrue. + +**What is "live trace"?** + +Job trace that is sent by runner while jobs are running. You can see live trace in job pages UI. +The live traces are archived once job finishes. + +**What is new architecture?** + +So far, when GitLab Runner sends a job trace to GitLab-Rails, traces have been saved to file storage as text files. +This was a problem for [Cloud Native-compatible GitLab application](https://gitlab.com/gitlab-com/migration/issues/23) where GitLab had to rely on File Storage. + +This new live trace architecture stores chunks of traces in Redis and database instead of file storage. +Redis is used as first-class storage, and it stores up-to 128kB. Once the full chunk is sent it will be flushed to database. Afterwhile, the data in Redis and database will be archived to ObjectStorage. + +Here is the detailed data flow. + +1. GitLab Runner picks a job from GitLab-Rails +1. GitLab Runner sends a piece of trace to GitLab-Rails +1. GitLab-Rails appends the data to Redis +1. If the data in Redis is fulfilled 128kB, the data is flushed to Database. +1. 2.~4. is continued until the job is finished +1. Once the job is finished, GitLab-Rails schedules a sidekiq worker to archive the trace +1. The sidekiq worker archives the trace to Object Storage, and cleanup the trace in Redis and Database + +**How to check if it's on or off?** + +```ruby +Feature.enabled?('ci_enable_live_trace') +``` + +**How to enable?** + +```ruby +Feature.enable('ci_enable_live_trace') +``` + +>**Note:** +The transition period will be handled gracefully. Upcoming traces will be generated with the new architecture, and on-going live traces will stay with the legacy architecture (i.e. on-going live traces won't be re-generated forcibly with the new architecture). + +**How to disable?** + +```ruby +Feature.disable('ci_enable_live_trace') +``` + +>**Note:** +The transition period will be handled gracefully. Upcoming traces will be generated with the legacy architecture, and on-going live traces will stay with the new architecture (i.e. on-going live traces won't be re-generated forcibly with the legacy architecture). + +**Redis namespace:** + +`Gitlab::Redis::SharedState` + +**Potential impact:** + +- This feature could incur data loss: + - Case 1: When all data in Redis are accidentally flushed. + - On-going live traces could be recovered by re-sending traces (This is supported by all versions of GitLab Runner) + - Finished jobs which has not archived live traces will lose the last part (~128kB) of trace data. + - Case 2: When sidekiq workers failed to archive (e.g. There was a bug that prevents archiving process, Sidekiq inconsistancy, etc): + - Currently all trace data in Redis will be deleted after one week. If the sidekiq workers can't finish by the expiry date, the part of trace data will be lost. +- This feature could consume all memory on Redis instance. If the number of jobs is 1000, 128MB (128kB * 1000) is consumed. +- This feature could pressure Database replication lag. `INSERT` are generated to indicate that we have trace chunk. `UPDATE` with 128kB of data is issued once we receive multiple chunks. +- and so on + +**How to test?** + +We're currently evaluating this feature on dev.gitalb.org or staging.gitlab.com to verify this features. Here is the list of tests/measurements. + +- Features: + - Live traces should be visible on job pages + - Archived traces should be visible on job pages + - Live traces should be archived to Object storage + - Live traces should be cleaned up after archived + - etc +- Performance: + - Schedule 1000~10000 jobs and let GitLab-runners process concurrently. Measure memoery presssure, IO load, etc. + - etc +- Failover: + - Simulate Redis outage + - etc + +**How to verify the correctnesss?** + +- TBD + +[ce-44935]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18169 diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index f495990d9a4..69600cad25c 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -46,7 +46,7 @@ In this experimental phase, only a few metrics are available: | redis_ping_latency_seconds | Gauge | 9.4 | Round trip time of the redis ping | | user_session_logins_total | Counter | 9.4 | Counter of how many users have logged in | | filesystem_circuitbreaker_latency_seconds | Gauge | 9.5 | Time spent validating if a storage is accessible | -| filesystem_circuitbreaker | Gauge | 9.5 | Wether or not the circuit for a certain shard is broken or not | +| filesystem_circuitbreaker | Gauge | 9.5 | Whether or not the circuit for a certain shard is broken or not | | circuitbreaker_storage_check_duration_seconds | Histogram | 10.3 | Time a single storage probe took | ## Metrics shared directory diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index bd6c7bb07b5..89331238ce4 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -31,7 +31,7 @@ GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup to the GitLab database. GitLab Shell uses the fingerprint of the SSH key to check whether the user is authorized to access GitLab. -Add the following to your `sshd_config` file. This is usuaully located at +Add the following to your `sshd_config` file. This is usually located at `/etc/ssh/sshd_config`, but it will be `/assets/sshd_config` if you're using Omnibus Docker: diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index ee37ea49874..efeec9db517 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -13,12 +13,12 @@ checks failed you can see their output on the admin log page under ## Periodic checks -When enabled, GitLab periodically runs a repository check on all project -repositories and wiki repositories in order to detect data corruption problems. +When enabled, GitLab periodically runs a repository check on all project +repositories and wiki repositories in order to detect data corruption. A project will be checked no more than once per month. If any projects fail their repository checks all GitLab administrators will receive an email -notification of the situation. This notification is sent out once a week on -Sunday, by default. +notification of the situation. This notification is sent out once a week, +by default, midnight at the start of Sunday. ## Disabling periodic checks @@ -28,16 +28,18 @@ panel. ## What to do if a check failed If the repository check fails for some repository you should look up the error -in repocheck.log (in the admin panel or on disk; see -`/var/log/gitlab/gitlab-rails` for Omnibus installations or -`/home/git/gitlab/log` for installations from source). Once you have -resolved the issue use the admin panel to trigger a new repository check on -the project. This will clear the 'check failed' state. +in `repocheck.log`: + +- in the [admin panel](logs.md#repocheck.log) +- or on disk, see: + - `/var/log/gitlab/gitlab-rails` for Omnibus installations + - `/home/git/gitlab/log` for installations from source If for some reason the periodic repository check caused a lot of false -alarms you can choose to clear ALL repository check states from the -'Settings' page of the admin panel. +alarms you can choose to clear *all* repository check states by +clicking "Clear all repository checks" on the **Settings** page of the +admin panel (`/admin/application_settings`). --- [ce-3232]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3232 "Auto git fsck" -[git-fsck]: https://www.kernel.org/pub/software/scm/git/docs/git-fsck.html "git fsck documentation" +[git-fsck]: https://git-scm.com/docs/git-fsck "git fsck documentation" diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 2fa3284b6be..7f0bd8f04e3 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -104,7 +104,7 @@ _The uploads are stored by default in ``` >**Note:** -If you are using AWS IAM profiles, be sure to omit the AWS access key and secret acces key/value pairs. +If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. ```ruby gitlab_rails['uploads_object_store_connection'] = { |