summaryrefslogtreecommitdiff
path: root/doc/administration
diff options
context:
space:
mode:
authorSimon Knox <psimyn@gmail.com>2019-08-06 15:07:18 +1000
committerSimon Knox <psimyn@gmail.com>2019-08-06 15:07:18 +1000
commitfc77b9df8b6a49c86e9c1eb949f1b1162470d2ee (patch)
tree96aad0a31543fa520626dc1c5efabff1367a0bab /doc/administration
parent9c71bf3e6df2dcb20ea19df21a127823bbe5e615 (diff)
parentfa216b0e86433192ba4e39a05f42217fb4685173 (diff)
downloadgitlab-ce-fc77b9df8b6a49c86e9c1eb949f1b1162470d2ee.tar.gz
Merge branch 'master' of gitlab.com:gitlab-org/gitlab-ce into alerts-dropdown-to-modal-part-2-cealerts-dropdown-to-modal-part-2-ce
Diffstat (limited to 'doc/administration')
-rw-r--r--doc/administration/audit_events.md2
-rw-r--r--doc/administration/auth/ldap-ee.md126
-rw-r--r--doc/administration/auth/ldap.md67
-rw-r--r--doc/administration/custom_hooks.md2
-rw-r--r--doc/administration/geo/replication/index.md77
-rw-r--r--doc/administration/geo/replication/updating_the_geo_nodes.md34
-rw-r--r--doc/administration/gitaly/index.md102
-rw-r--r--doc/administration/high_availability/README.md35
-rw-r--r--doc/administration/index.md2
-rw-r--r--doc/administration/monitoring/performance/img/performance_bar.pngbin127198 -> 113617 bytes
-rw-r--r--doc/administration/monitoring/performance/img/performance_bar_line_profiling.pngbin93063 -> 0 bytes
-rw-r--r--doc/administration/monitoring/performance/performance_bar.md2
-rw-r--r--doc/administration/operations/fast_ssh_key_lookup.md4
-rw-r--r--doc/administration/plugins.md32
-rw-r--r--doc/administration/raketasks/ldap.md25
-rw-r--r--doc/administration/repository_storage_paths.md16
-rw-r--r--doc/administration/troubleshooting/diagnostics_tools.md27
17 files changed, 393 insertions, 160 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md
index aaa43f67760..02de2caf558 100644
--- a/doc/administration/audit_events.md
+++ b/doc/administration/audit_events.md
@@ -73,6 +73,8 @@ From there, you can see the following actions:
- User was added to project and with which [permissions]
- Permission changes of a user assigned to a project
- User was removed from project
+- Project export was downloaded
+- Project repository was downloaded
### Instance events **(PREMIUM ONLY)**
diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md
index 1b7af2d945a..34f3cfa353f 100644
--- a/doc/administration/auth/ldap-ee.md
+++ b/doc/administration/auth/ldap-ee.md
@@ -4,37 +4,27 @@ type: reference
# LDAP Additions in GitLab EE **(STARTER ONLY)**
-This is a continuation of the main [LDAP documentation](ldap.md), detailing LDAP
-features specific to GitLab Enterprise Edition Starter, Premium and Ultimate.
+This section documents LDAP features specific to to GitLab Enterprise Edition
+[Starter](https://about.gitlab.com/pricing/#self-managed) and above.
-## Overview
-
-[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol)
-stands for **Lightweight Directory Access Protocol**, which
-is a standard application protocol for
-accessing and maintaining distributed directory information services
-over an Internet Protocol (IP) network.
-
-GitLab integrates with LDAP to support **user authentication**. This integration
-works with most LDAP-compliant directory servers, including Microsoft Active
-Directory, Apple Open Directory, Open LDAP, and 389 Server.
-**GitLab Enterprise Edition** includes enhanced integration, including group
-membership syncing.
+For documentation relevant to both Community Edition and Enterprise Edition,
+see the main [LDAP documentation](ldap.md).
## Use cases
-- User sync: Once a day, GitLab will update users against LDAP
+- User sync: Once a day, GitLab will update users against LDAP.
- Group sync: Once an hour, GitLab will update group membership
- based on LDAP group members
+ based on LDAP group members.
-## Multiple LDAP servers
+## Multiple LDAP servers **(STARTER ONLY)**
With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers
that your GitLab instance will connect to.
-To add another LDAP server, you can start by duplicating the settings under
-[the main configuration](ldap.md#configuration) and edit them to match the
-additional LDAP server.
+To add another LDAP server:
+
+1. Duplicating the settings under [the main configuration](ldap.md#configuration).
+1. Edit them to match the additional LDAP server.
Be sure to choose a different provider ID made of letters a-z and numbers 0-9.
This ID will be stored in the database so that GitLab can remember which LDAP
@@ -47,19 +37,19 @@ users against LDAP.
The process will execute the following access checks:
-1. Ensure the user is still present in LDAP
-1. If the LDAP server is Active Directory, ensure the user is active (not
- blocked/disabled state). This will only be checked if
- `active_directory: true` is set in the LDAP configuration [^1]
+- Ensure the user is still present in LDAP.
+- If the LDAP server is Active Directory, ensure the user is active (not
+ blocked/disabled state). This will only be checked if
+ `active_directory: true` is set in the LDAP configuration. [^1]
The user will be set to `ldap_blocked` state in GitLab if the above conditions
fail. This means the user will not be able to login or push/pull code.
The process will also update the following user information:
-1. Email address
-1. If `sync_ssh_keys` is set, SSH public keys
-1. If Kerberos is enabled, Kerberos identity
+- Email address.
+- If `sync_ssh_keys` is set, SSH public keys.
+- If Kerberos is enabled, Kerberos identity.
NOTE: **Note:**
The LDAP sync process updates existing users while new users will
@@ -72,9 +62,6 @@ first time GitLab will trigger a sync for groups the user should be a member of.
That way they don't need to wait for the hourly sync to be granted
access to their groups and projects.
-In GitLab Premium, we can also add a GitLab group to sync with one or multiple LDAP groups or we can
-also add a filter. The filter must comply with the syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254).
-
A group sync process will run every hour on the hour, and `group_base` must be set
in LDAP configuration for LDAP synchronizations based on group CN to work. This allows
GitLab group membership to be automatically updated based on LDAP group members.
@@ -120,13 +107,26 @@ following.
1. [Restart GitLab][restart] for the changes to take effect.
----
-
To take advantage of group sync, group owners or maintainers will need to create an
-LDAP group link in their group **Settings > LDAP Groups** page. Multiple LDAP
-groups and/or filters can be linked with a single GitLab group. When the link is
-created, an access level/role is specified (Guest, Reporter, Developer, Maintainer,
-or Owner).
+LDAP group link in their group **Settings > LDAP Groups** page.
+
+Multiple LDAP groups and [filters](#filters-premium-only) can be linked with
+a single GitLab group. When the link is created, an access level/role is
+specified (Guest, Reporter, Developer, Maintainer, or Owner).
+
+### Filters **(PREMIUM ONLY)**
+
+In GitLab Premium, you can add an LDAP user filter for group synchronization.
+Filters allow for complex logic without creating a special LDAP group.
+
+To sync GitLab group membership based on an LDAP filter:
+
+1. Open the **LDAP Synchronization** page for the GitLab group.
+1. Select **LDAP user filter** as the **Sync method**.
+1. Enter an LDAP user filter in the **LDAP user filter** field.
+
+The filter must comply with the
+syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254).
## Administrator sync
@@ -190,10 +190,13 @@ group, as opposed to the full DN.
## Global group memberships lock
"Lock memberships to LDAP synchronization" setting allows instance administrators
-to lock down user abilities to invite new members to a group. When enabled following happens:
+to lock down user abilities to invite new members to a group.
+
+When enabled, the following applies:
-1. Only administrator can manage memberships of any group including access levels.
-1. Users are not allowed to share project with other groups or invite members to a project created in a group.
+- Only administrator can manage memberships of any group including access levels.
+- Users are not allowed to share project with other groups or invite members to
+ a project created in a group.
## Adjusting LDAP user sync schedule
@@ -333,10 +336,18 @@ administrative duties.
### Supported LDAP group types/attributes
-GitLab supports LDAP groups that use member attributes `member`, `submember`,
-`uniquemember`, `memberof` and `memberuid`. This means group sync supports, at
-least, LDAP groups with object class `groupOfNames`, `posixGroup`, and
-`groupOfUniqueName`. Other object classes should work fine as long as members
+GitLab supports LDAP groups that use member attributes:
+
+- `member`
+- `submember`
+- `uniquemember`
+- `memberof`
+- `memberuid`.
+
+This means group sync supports, at least, LDAP groups with object class:
+`groupOfNames`, `posixGroup`, and `groupOfUniqueName`.
+
+Other object classes should work fine as long as members
are defined as one of the mentioned attributes. This also means GitLab supports
Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server.
Other LDAP servers should work, too.
@@ -344,11 +355,12 @@ Other LDAP servers should work, too.
Active Directory also supports nested groups. Group sync will recursively
resolve membership if `active_directory: true` is set in the configuration file.
-> **Note:** Nested group membership will only be resolved if the nested group
- also falls within the configured `group_base`. For example, if GitLab sees a
- nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but
- the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group`
- will be ignored.
+NOTE: **Note:**
+Nested group membership will only be resolved if the nested group
+also falls within the configured `group_base`. For example, if GitLab sees a
+nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but
+the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group`
+will be ignored.
### Queries
@@ -403,7 +415,7 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server
[^1]: In Active Directory, a user is marked as disabled/blocked if the user
account control attribute (`userAccountControl:1.2.840.113556.1.4.803`)
- has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/
+ has bit 2 set. See <https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/>
for more information.
### User DN has changed
@@ -423,10 +435,10 @@ things to check to debug the situation.
- Ensure LDAP configuration has a `group_base` specified. This configuration is
required for group sync to work properly.
- Ensure the correct LDAP group link is added to the GitLab group. Check group
- links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**.
-- Check that the user has an LDAP identity
+ links by visiting the GitLab group, then **Settings dropdown > LDAP groups**.
+- Check that the user has an LDAP identity:
1. Sign in to GitLab as an administrator user.
- 1. Navigate to **Admin area -> Users**.
+ 1. Navigate to **Admin area > Users**.
1. Search for the user
1. Open the user, by clicking on their name. Do not click 'Edit'.
1. Navigate to the **Identities** tab. There should be an LDAP identity with
@@ -437,7 +449,7 @@ Often, the best way to learn more about why group sync is behaving a certain
way is to enable debug logging. There is verbose output that details every
step of the sync.
-1. Start a Rails console
+1. Start a Rails console:
```bash
# For Omnibus installations
@@ -446,6 +458,7 @@ step of the sync.
# For installations from source
sudo -u git -H bundle exec rails console production
```
+
1. Set the log level to debug (only for this session):
```ruby
@@ -540,8 +553,9 @@ and more DNs may be added, or existing entries modified, based on additional
LDAP group lookups. The very last occurrence of this entry should indicate
exactly which users GitLab believes should be added to the group.
-> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer'
- and 50 is 'Owner'
+NOTE: **Note:**
+10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer'
+and 50 is 'Owner'.
```bash
Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30,
diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md
index be05a4d63a7..186bf4c4825 100644
--- a/doc/administration/auth/ldap.md
+++ b/doc/administration/auth/ldap.md
@@ -7,28 +7,44 @@ type: reference
# LDAP
GitLab integrates with LDAP to support user authentication.
-This integration works with most LDAP-compliant directory
-servers, including Microsoft Active Directory, Apple Open Directory, Open LDAP,
-and 389 Server. GitLab Enterprise Editions include enhanced integration,
+
+This integration works with most LDAP-compliant directory servers, including:
+
+- Microsoft Active Directory
+- Apple Open Directory
+- Open LDAP
+- 389 Server.
+
+GitLab Enterprise Editions (EE) include enhanced integration,
including group membership syncing as well as multiple LDAP servers support.
-## GitLab EE
+For more details about EE-specific LDAP features, see the
+[LDAP Enterprise Edition documentation](ldap-ee.md).
-The information on this page is relevant for both GitLab CE and EE. For more
-details about EE-specific LDAP features, see the
-[LDAP Enterprise Edition documentation](ldap-ee.md). **(STARTER ONLY)**
+NOTE: **Note:**
+The information on this page is relevant for both GitLab CE and EE.
+
+## Overview
+
+[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol)
+stands for **Lightweight Directory Access Protocol**, which is a standard
+application protocol for accessing and maintaining distributed directory
+information services over an Internet Protocol (IP) network.
## Security
-GitLab assumes that LDAP users are not able to change their LDAP 'mail', 'email'
-or 'userPrincipalName' attribute. An LDAP user who is allowed to change their
-email on the LDAP server can potentially
-[take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users)
-on your GitLab server.
+GitLab assumes that LDAP users:
+
+- Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attribute.
+ An LDAP user who is allowed to change their email on the LDAP server can potentially
+ [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users)
+ on your GitLab server.
+- Have unique email addresses, otherwise it is possible for LDAP users with the same
+ email address to share the same GitLab account.
We recommend against using LDAP integration if your LDAP users are
-allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on
-the LDAP server.
+allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on
+the LDAP server or share email addresses.
### User deletion
@@ -64,9 +80,12 @@ NOTE: **Note**:
In GitLab Enterprise Edition Starter, you can configure multiple LDAP servers
to connect to one GitLab server.
-For a complete guide on configuring LDAP with GitLab Community Edition, please check
-the admin guide [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md).
-For GitLab Enterprise Editions, see also [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)**
+For a complete guide on configuring LDAP with:
+
+- GitLab Community Edition, see
+ [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md).
+- Enterprise Editions, see
+ [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)**
To enable LDAP integration you need to add your LDAP server settings in
`/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus
@@ -384,7 +403,7 @@ production:
Tip: If you want to limit access to the nested members of an Active Directory
group, you can use the following syntax:
-```
+```text
(memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com)
```
@@ -402,13 +421,13 @@ The `user_filter` DN can contain special characters. For example:
- A comma:
- ```
+ ```text
OU=GitLab, Inc,DC=gitlab,DC=com
```
- Open and close brackets:
- ```
+ ```text
OU=Gitlab (Inc),DC=gitlab,DC=com
```
@@ -417,13 +436,13 @@ The `user_filter` DN can contain special characters. For example:
- Escape commas with `\2C`. For example:
- ```
+ ```text
OU=GitLab\2C Inc,DC=gitlab,DC=com
```
- Escape open and close brackets with `\28` and `\29`, respectively. For example:
- ```
+ ```text
OU=Gitlab \28Inc\29,DC=gitlab,DC=com
```
@@ -507,11 +526,11 @@ timeout), the login is rejected and a message will be logged to
### Debug LDAP user filter with ldapsearch
-This example uses ldapsearch and assumes you are using ActiveDirectory. The
+This example uses `ldapsearch` and assumes you are using ActiveDirectory. The
following query returns the login names of the users that will be allowed to
log in to GitLab if you configure your own user_filter.
-```
+```sh
ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName
```
diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md
index 113514e1ee8..32462a95a1a 100644
--- a/doc/administration/custom_hooks.md
+++ b/doc/administration/custom_hooks.md
@@ -21,7 +21,7 @@ administrators can add custom git hooks to any GitLab project.
## Create a custom Git hook for a repository
Server-side Git hooks are typically placed in the repository's `hooks`
-subdirectory. In GitLab, hook directories are are symlinked to the gitlab-shell
+subdirectory. In GitLab, hook directories are symlinked to the gitlab-shell
`hooks` directory for ease of maintenance between gitlab-shell upgrades.
Custom hooks are implemented differently, but the behavior is exactly the same
once the hook is created. Follow the steps below to set up a custom hook for a
diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md
index f0d329d5296..dc49a95e008 100644
--- a/doc/administration/geo/replication/index.md
+++ b/doc/administration/geo/replication/index.md
@@ -1,5 +1,11 @@
# Geo Replication **(PREMIUM ONLY)**
+> - Introduced in GitLab Enterprise Edition 8.9.
+> - Using Geo in combination with
+> [High Availability](../../high_availability/README.md)
+> is considered **Generally Available** (GA) in
+> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4.
+
Geo is the solution for widely distributed development teams.
## Overview
@@ -8,14 +14,14 @@ Fetching large repositories can take a long time for teams located far from a si
Geo provides local, read-only instances of your GitLab instances, reducing the time it takes to clone and fetch large repositories and speeding up development.
+> **Notes:**
+>
> - Geo is part of [GitLab Premium](https://about.gitlab.com/pricing/#self-managed).
-> - Introduced in GitLab Enterprise Edition 8.9.
> - We recommend you use:
> - At least GitLab Enterprise Edition 10.0 for basic Geo features.
> - The latest version for a better experience.
> - Make sure that all nodes run the same GitLab version.
> - Geo requires PostgreSQL 9.6 and Git 2.9, in addition to GitLab's usual [minimum requirements](../../../install/requirements.md).
-> - Using Geo in combination with [High Availability](../../high_availability/README.md) is considered **Generally Available** (GA) in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 10.4.
For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w).
@@ -239,35 +245,48 @@ This list of limitations only reflects the latest version of GitLab. If you are
- Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node.
- [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work.
-### Limitations on replication
-
-Only the following items are replicated to the **secondary** node:
-
-- All database content. For example, snippets, epics, issues, merge requests, groups, and project metadata.
-- Project repositories.
-- Project wiki repositories.
-- User uploads. For example, attachments to issues, merge requests, epics, and avatars.
-- CI job artifacts and traces.
+### Limitations on replication/verification
+
+The following table lists the GitLab features along with their replication
+and verification status on a **secondary** node.
+
+You can keep track of the progress to include the missing items in:
+
+- [ee-893](https://gitlab.com/groups/gitlab-org/-/epics/893).
+- [ee-1430](https://gitlab.com/groups/gitlab-org/-/epics/1430).
+
+| Feature | Replicated | Verified |
+|-----------|------------|----------|
+| All database content (e.g. snippets, epics, issues, merge requests, groups, and project metadata) | Yes | Yes |
+| Project repository | Yes | Yes |
+| Project wiki repository | Yes | Yes |
+| Project designs repository | No | No |
+| Uploads (e.g. attachments to issues, merge requests, epics, and avatars) | Yes | Yes, only on transfer, or manually (1) |
+| LFS Objects | Yes | Yes, only on transfer, or manually (1) |
+| CI job artifacts (other than traces) | Yes | No, only manually (1) |
+| Archived traces | Yes | Yes, only on transfer, or manually (1) |
+| Personal snippets | Yes | Yes |
+| Version-controlled personal snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No |
+| Project snippets | Yes | Yes |
+| Version-controlled project snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No |
+| Object pools for forked project deduplication | No | No |
+| [Server-side Git Hooks](../../custom_hooks.md) | No | No |
+| [Elasticsearch integration](../../../integration/elasticsearch.md) | No | No |
+| [GitLab Pages](../../pages/index.md) | No | No |
+| [Container Registry](../../container_registry.md) ([track progress](https://gitlab.com/gitlab-org/gitlab-ee/issues/2870)) | No | No |
+| [NPM Registry](../../npm_registry.md) | No | No |
+| [Maven Packages](../../maven_packages.md) | No | No |
+| [External merge request diffs](../../merge_request_diffs.md) | No, if they are on-disk | No |
+| Content in object storage ([track progress](https://gitlab.com/groups/gitlab-org/-/epics/1526)) | No | No |
+
+1. The integrity can be verified manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them.
DANGER: **DANGER**
-Data not on this list is unavailable on the **secondary** node. Failing over without manually replicating data not on this list will cause the data to be **lost**.
-
-### Examples of data not replicated
-
-Take special note that these examples of GitLab features are both:
-
-- Commonly used.
-- **Not** replicated by Geo at present.
-
-Examples include:
-
-- [Elasticsearch integration](../../../integration/elasticsearch.md).
-- [Container Registry](../../container_registry.md). [Object Storage](object_storage.md) can mitigate this.
-- [GitLab Pages](../../pages/index.md).
-- [Mattermost integration](https://docs.gitlab.com/omnibus/gitlab-mattermost/).
-
-CAUTION: **Caution:**
-If you wish to use them on a **secondary** node, or to execute a failover successfully, you will need to replicate their data using some other means.
+Features not on this list, or with **No** in the **Replicated** column,
+are not replicated on the **secondary** node. Failing over without manually
+replicating data from those features will cause the data to be **lost**.
+If you wish to use those features on a **secondary** node, or to execute a failover
+successfully, you must replicate their data using some other means.
## Frequently Asked Questions
diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md
index 550b3b07a95..39174780e24 100644
--- a/doc/administration/geo/replication/updating_the_geo_nodes.md
+++ b/doc/administration/geo/replication/updating_the_geo_nodes.md
@@ -10,10 +10,23 @@ all you need to do is update GitLab itself:
1. Log into each node (**primary** and **secondary** nodes).
1. [Update GitLab][update].
-1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when
- the tracking database is enabled.
1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each.
+### Check status after updating
+
+Now that the update process is complete, you may want to check whether
+everything is working correctly:
+
+1. Run the Geo raketask on all nodes, everything should be green:
+
+ ```sh
+ sudo gitlab-rake gitlab:geo:check
+ ```
+
+1. Check the **primary** node's Geo dashboard for any errors.
+1. Test the data replication by pushing code to the **primary** node and see if it
+ is received by **secondary** nodes.
+
## Upgrading to GitLab 12.1
By default, GitLab 12.1 will attempt to automatically upgrade the embedded PostgreSQL server to 10.7 from 9.6. Please see [the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) for the recommended procedure.
@@ -419,22 +432,7 @@ is prepended with the relevant node for better clarity:
sudo gitlab-ctl start
```
-## Check status after updating
-
-Now that the update process is complete, you may want to check whether
-everything is working correctly:
-
-1. Run the Geo raketask on all nodes, everything should be green:
-
- ```sh
- sudo gitlab-rake gitlab:geo:check
- ```
-
-1. Check the **primary** node's Geo dashboard for any errors.
-1. Test the data replication by pushing code to the **primary** node and see if it
- is received by **secondary** nodes.
-
-## Update tracking database on **secondary** node
+### Update tracking database on **secondary** node
After updating a **secondary** node, you might need to run migrations on
the tracking database. The tracking database was added in GitLab 9.1,
diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md
index 0f547ef03bf..432056d48c7 100644
--- a/doc/administration/gitaly/index.md
+++ b/doc/administration/gitaly/index.md
@@ -469,7 +469,16 @@ One current feature of GitLab that still requires a shared directory (NFS) is
There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196)
to eliminate the need for NFS to support GitLab Pages.
-## Troubleshooting
+## Troubleshooting Gitaly
+
+### Commits, pushes, and clones return a 401
+
+```
+remote: GitLab: 401 Unauthorized
+```
+
+You will need to sync your `gitlab-secrets.json` file with your GitLab
+app nodes.
### `gitaly-debug`
@@ -510,3 +519,94 @@ implemented as calls to `gitaly-ruby`:
```
sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0
```
+
+### Repository changes fail with a `401 Unauthorized` error
+
+If you're running Gitaly on its own server and notice that users can
+successfully clone and fetch repositories (via both SSH and HTTPS), but can't
+push to them or make changes to the repository in the web UI without getting a
+`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate
+with the other nodes due to having the [wrong secrets file](#3-gitaly-server-configuration).
+
+Confirm the following are all true:
+
+- When any user performs a `git push` to any repository on this Gitaly node, it
+ fails with the following error (note the `401 Unauthorized`):
+
+ ```sh
+ remote: GitLab: 401 Unauthorized
+ To <REMOTE_URL>
+ ! [remote rejected] branch-name -> branch-name (pre-receive hook declined)
+ error: failed to push some refs to '<REMOTE_URL>'
+ ```
+
+- When any user adds or modifies a file from the repository using the GitLab
+ UI, it immediatley fails with a red `401 Unauthorized` banner.
+- Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects)
+ successfully creates the project but doesn't create the README.
+- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.md#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors
+ when reaching the `/api/v4/internal/allowed` endpoint:
+
+ ```sh
+ # api_json.log
+ {
+ "time": "2019-07-18T00:30:14.967Z",
+ "severity": "INFO",
+ "duration": 0.57,
+ "db": 0,
+ "view": 0.57,
+ "status": 401,
+ "method": "POST",
+ "path": "\/api\/v4\/internal\/allowed",
+ "params": [
+ {
+ "key": "action",
+ "value": "git-receive-pack"
+ },
+ {
+ "key": "changes",
+ "value": "REDACTED"
+ },
+ {
+ "key": "gl_repository",
+ "value": "REDACTED"
+ },
+ {
+ "key": "project",
+ "value": "\/path\/to\/project.git"
+ },
+ {
+ "key": "protocol",
+ "value": "web"
+ },
+ {
+ "key": "env",
+ "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}"
+ },
+ {
+ "key": "user_id",
+ "value": "2"
+ },
+ {
+ "key": "secret_token",
+ "value": "[FILTERED]"
+ }
+ ],
+ "host": "gitlab.example.com",
+ "ip": "REDACTED",
+ "ua": "Ruby",
+ "route": "\/api\/:version\/internal\/allowed",
+ "queue_duration": 4.24,
+ "gitaly_calls": 0,
+ "gitaly_duration": 0,
+ "correlation_id": "XPUZqTukaP3"
+ }
+
+ # nginx_access.log
+ [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby"
+ ```
+
+To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-server-configuration)
+on the Gitaly node matches the one on all other nodes. If it doesn't match,
+update the secrets file on the Gitaly node to match the others, then
+[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure).
diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md
index 41ef68f5b57..56665ba8b9a 100644
--- a/doc/administration/high_availability/README.md
+++ b/doc/administration/high_availability/README.md
@@ -164,25 +164,24 @@ contention due to certain workloads.
#### Reference Architecture
-- **Status:** Work-in-progress
- **Supported Users (approximate):** 10,000
-- **Related Issues:** [gitlab-com/support/support-team-meta#1513](https://gitlab.com/gitlab-com/support/support-team-meta/issues/1513),
- [gitlab-org/quality/team-tasks#110](https://gitlab.com/gitlab-org/quality/team-tasks/issues/110)
-
-The Support and Quality teams are in the process of building and performance testing
-an environment that will support about 10,000 users. The specifications below
-are a work-in-progress representation of the work so far. Quality will be
-certifying this environment in FY20-Q2. The specifications may be adjusted
-prior to certification based on performance testing.
-
-- 3 PostgreSQL - 4 CPU, 8GB RAM per node
-- 1 PgBouncer - 2 CPU, 4GB RAM
-- 2 Redis - 2 CPU, 8GB RAM per node
-- 3 Consul/Sentinel - 2 CPU, 2GB RAM per node
-- 4 Sidekiq - 4 CPU, 8GB RAM per node
-- 5 GitLab application nodes - 20 CPU, 64GB RAM per node
-- 1 Gitaly - 20 CPU, 64GB RAM
-- 1 Monitoring node - 4 CPU, 8GB RAM
+- **Known Issues:** While validating the reference architecture, slow endpoints were discovered and are being investigated. [gitlab-org/gitlab-ce/issues/64335](https://gitlab.com/gitlab-org/gitlab-ce/issues/64335)
+
+The Support and Quality teams built, performance tested, and validated an
+environment that supports about 10,000 users. The specifications below are a
+representation of the work so far. The specifications may be adjusted in the
+future based on additional testing and iteration.
+
+NOTE: **Note:** The specifications here were performance tested against a specific coded workload. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size.
+
+- 3 PostgreSQL - 4 CPU, 16GiB memory per node
+- 1 PgBouncer - 2 CPU, 4GiB memory
+- 2 Redis - 2 CPU, 8GiB memory per node
+- 3 Consul/Sentinel - 2 CPU, 2GiB memory per node
+- 4 Sidekiq - 4 CPU, 16GiB memory per node
+- 5 GitLab application nodes - 16 CPU, 64GiB memory per node
+- 1 Gitaly - 16 CPU, 64GiB memory
+- 1 Monitoring node - 2 CPU, 8GiB memory, 100GiB local storage
### Fully Distributed
diff --git a/doc/administration/index.md b/doc/administration/index.md
index 00c8863f200..febee3e5af8 100644
--- a/doc/administration/index.md
+++ b/doc/administration/index.md
@@ -185,3 +185,5 @@ Learn how to install, configure, update, and maintain your GitLab instance.
- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong
- [Log system](logs.md): Where to look for logs.
- [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs.
+- Useful [diagnostics tools](troubleshooting/diagnostics_tools.md) that are sometimes used by the GitLab
+ Support team.
diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png
index 8a6f8b3b273..89b09054d46 100644
--- a/doc/administration/monitoring/performance/img/performance_bar.png
+++ b/doc/administration/monitoring/performance/img/performance_bar.png
Binary files differ
diff --git a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png b/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png
deleted file mode 100644
index a55ce753101..00000000000
--- a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png
+++ /dev/null
Binary files differ
diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md
index 15eab7dcab0..2cc78ccc03c 100644
--- a/doc/administration/monitoring/performance/performance_bar.md
+++ b/doc/administration/monitoring/performance/performance_bar.md
@@ -14,8 +14,6 @@ It allows you to see (from left to right):
![Gitaly profiling using the Performance Bar](img/performance_bar_gitaly_calls.png)
- time taken and number of [Rugged] calls, click through for details of these calls
![Rugged profiling using the Performance Bar](img/performance_bar_rugged_calls.png)
-- profile of the code used to generate the page, line by line. In the profile view, the numbers in the left panel represent wall time, cpu time, and number of calls (based on [rblineprof](https://github.com/tmm1/rblineprof)).
- ![Line profiling using the Performance Bar](img/performance_bar_line_profiling.png)
- time taken and number of Redis calls, click through for details of these calls
![Redis profiling using the Performance Bar](img/performance_bar_redis_calls.png)
- time taken and number of Ruby GC calls
diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md
index ea69378b249..e787af798bc 100644
--- a/doc/administration/operations/fast_ssh_key_lookup.md
+++ b/doc/administration/operations/fast_ssh_key_lookup.md
@@ -71,10 +71,10 @@ sudo service sshd reload
Confirm that SSH is working by removing your user's SSH key in the UI, adding a
new one, and attempting to pull a repo.
-> **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in
+NOTE: **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in
GitLab 11.11 and later.
-> **Warning:** Do not disable writes until SSH is confirmed to be working
+CAUTION: **Caution:** Do not disable writes until SSH is confirmed to be working
perfectly, because the file will quickly become out-of-date.
In the case of lookup failures (which are common), the `authorized_keys`
diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md
index 4302667caf5..4cf3c607dae 100644
--- a/doc/administration/plugins.md
+++ b/doc/administration/plugins.md
@@ -52,7 +52,37 @@ as appropriate. The plugins file list is updated for each event, there is no
need to restart GitLab to apply a new plugin.
If a plugin executes with non-zero exit code or GitLab fails to execute it, a
-message will be logged to `plugin.log`.
+message will be logged to:
+
+- `gitlab-rails/plugin.log` in an Omnibus installation.
+- `log/plugin.log` in a source installation.
+
+## Creating plugins
+
+Below is an example that will only response on the event `project_create` and
+will inform the admins from the GitLab instance that a new project has been created.
+
+```ruby
+# By using the embedded ruby version we eliminate the possibility that our chosen language
+# would be unavailable from
+#!/opt/gitlab/embedded/bin/ruby
+require 'json'
+require 'mail'
+
+# The incoming variables are in JSON format so we need to parse it first.
+ARGS = JSON.parse(STDIN.read)
+
+# We only want to trigger this plugin on the event project_create
+return unless ARGS['event_name'] == 'project_create'
+
+# We will inform our admins of our gitlab instance that a new project is created
+Mail.deliver do
+ from 'info@gitlab_instance.com'
+ to 'admin@gitlab_instance.com'
+ subject "new project " + ARGS['name']
+ body ARGS['owner_name'] + 'created project ' + ARGS['name']
+end
+```
## Validation
diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md
index 91fc0133d56..e880d76e756 100644
--- a/doc/administration/raketasks/ldap.md
+++ b/doc/administration/raketasks/ldap.md
@@ -28,6 +28,31 @@ limit by passing a number to the check task:
rake gitlab:ldap:check[50]
```
+## Run a Group Sync
+
+> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.3.
+
+The following task will run a [group sync](../auth/ldap-ee.md#group-sync) immediately. This is valuable
+when you'd like to update all configured group memberships against LDAP without
+waiting for the next scheduled group sync to be run.
+
+NOTE: **NOTE:**
+If you'd like to change the frequency at which a group sync is performed,
+[adjust the cron schedule](../auth/ldap-ee.md#adjusting-ldap-group-sync-schedule)
+instead.
+
+**Omnibus Installation**
+
+```
+sudo gitlab-rake gitlab:ldap:group_sync
+```
+
+**Source Installation**
+
+```bash
+bundle exec rake gitlab:ldap:group_sync
+```
+
## Rename a provider
If you change the LDAP server ID in `gitlab.yml` or `gitlab.rb` you will need
diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md
index ad3a9b19c3c..b1a870210a8 100644
--- a/doc/administration/repository_storage_paths.md
+++ b/doc/administration/repository_storage_paths.md
@@ -57,10 +57,10 @@ storage2:
Now that you've read that big fat warning above, let's edit the configuration
files and add the full paths of the alternative repository storage paths. In
-the example below, we add two more mountpoints that are named `nfs` and `cephfs`
+the example below, we add two more mountpoints that are named `nfs_1` and `nfs_2`
respectively.
-NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details.
+NOTE: **Note:** This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details.
**For installations from source**
@@ -73,10 +73,10 @@ NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS
storages: # You must have at least a 'default' storage path.
default:
path: /home/git/repositories
- nfs:
- path: /mnt/nfs/repositories
- cephfs:
- path: /mnt/cephfs/repositories
+ nfs_1:
+ path: /mnt/nfs1/repositories
+ nfs_2:
+ path: /mnt/nfs2/repositories
```
1. [Restart GitLab][restart-gitlab] for the changes to take effect.
@@ -96,8 +96,8 @@ working, you can remove the `repos_path` line.
```ruby
git_data_dirs({
"default" => { "path" => "/var/opt/gitlab/git-data" },
- "nfs" => { "path" => "/mnt/nfs/git-data" },
- "cephfs" => { "path" => "/mnt/cephfs/git-data" }
+ "nfs_1" => { "path" => "/mnt/nfs1/git-data" },
+ "nfs_2" => { "path" => "/mnt/nfs2/git-data" }
})
```
diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md
new file mode 100644
index 00000000000..ab3b25f0e97
--- /dev/null
+++ b/doc/administration/troubleshooting/diagnostics_tools.md
@@ -0,0 +1,27 @@
+---
+type: reference
+---
+
+# Diagnostics tools
+
+These are some of the diagnostics tools the GitLab Support team uses during troubleshooting.
+They are listed here for transparency, and they may be useful for users with experience
+with troubleshooting GitLab. If you are currently having an issue with GitLab, you
+may want to check your [support options](https://about.gitlab.com/support/) first,
+before attempting to use these tools.
+
+## gitlabsos
+
+The [gitlabsos](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) utility
+provides a unified method of gathering info and logs from GitLab and the system it's
+running on.
+
+## strace-parser
+
+[strace-parser](https://gitlab.com/wchandler/strace-parser) is a small tool to analyze
+and summarize raw strace data.
+
+## Pritaly
+
+[Pritaly](https://gitlab.com/wchandler/pritaly) takes Gitaly logs and colorizes output
+or converts the logs to JSON.