From 6cfd13726fbce725633d95140dd0e4bf1779c5db Mon Sep 17 00:00:00 2001 From: manojmj Date: Wed, 24 Jul 2019 19:45:50 +0530 Subject: CE: Add project download & project export audit event This change adds audit events for download of repository and export of project. --- doc/administration/audit_events.md | 2 ++ 1 file changed, 2 insertions(+) (limited to 'doc/administration') diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index aaa43f67760..02de2caf558 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -73,6 +73,8 @@ From there, you can see the following actions: - User was added to project and with which [permissions] - Permission changes of a user assigned to a project - User was removed from project +- Project export was downloaded +- Project repository was downloaded ### Instance events **(PREMIUM ONLY)** -- cgit v1.2.1 From 65d8e39dc49a25bc25fae71e19b51edd53fb091e Mon Sep 17 00:00:00 2001 From: Harish Ramachandran Date: Fri, 26 Jul 2019 11:36:48 -0400 Subject: Document how to use the GroupSync rake task --- doc/administration/raketasks/ldap.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) (limited to 'doc/administration') diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 91fc0133d56..e880d76e756 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -28,6 +28,31 @@ limit by passing a number to the check task: rake gitlab:ldap:check[50] ``` +## Run a Group Sync + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.3. + +The following task will run a [group sync](../auth/ldap-ee.md#group-sync) immediately. This is valuable +when you'd like to update all configured group memberships against LDAP without +waiting for the next scheduled group sync to be run. + +NOTE: **NOTE:** +If you'd like to change the frequency at which a group sync is performed, +[adjust the cron schedule](../auth/ldap-ee.md#adjusting-ldap-group-sync-schedule) +instead. + +**Omnibus Installation** + +``` +sudo gitlab-rake gitlab:ldap:group_sync +``` + +**Source Installation** + +```bash +bundle exec rake gitlab:ldap:group_sync +``` + ## Rename a provider If you change the LDAP server ID in `gitlab.yml` or `gitlab.rb` you will need -- cgit v1.2.1 From 173eaab379a380187838d6132e78ebf983ee30b5 Mon Sep 17 00:00:00 2001 From: Tanya Pazitny Date: Mon, 29 Jul 2019 14:34:27 +0000 Subject: Update reference architecture docs to remove status --- doc/administration/high_availability/README.md | 15 ++++++--------- 1 file changed, 6 insertions(+), 9 deletions(-) (limited to 'doc/administration') diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 41ef68f5b57..42516d811a0 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -164,16 +164,13 @@ contention due to certain workloads. #### Reference Architecture -- **Status:** Work-in-progress - **Supported Users (approximate):** 10,000 -- **Related Issues:** [gitlab-com/support/support-team-meta#1513](https://gitlab.com/gitlab-com/support/support-team-meta/issues/1513), - [gitlab-org/quality/team-tasks#110](https://gitlab.com/gitlab-org/quality/team-tasks/issues/110) - -The Support and Quality teams are in the process of building and performance testing -an environment that will support about 10,000 users. The specifications below -are a work-in-progress representation of the work so far. Quality will be -certifying this environment in FY20-Q2. The specifications may be adjusted -prior to certification based on performance testing. +- **Known Issues:** While validating the reference architecture, slow endpoints were discovered and are being investigated. [gitlab-org/gitlab-ce/issues/64335](https://gitlab.com/gitlab-org/gitlab-ce/issues/64335) + +The Support and Quality teams built, performance tested, and validated an +environment that supports about 10,000 users. The specifications below are a +representation of the work so far. The specifications may be adjusted in the +future based on additional testing and iteration. - 3 PostgreSQL - 4 CPU, 8GB RAM per node - 1 PgBouncer - 2 CPU, 4GB RAM -- cgit v1.2.1 From 18cdc5ba6ce1810c19982475eca89fd385fe31e2 Mon Sep 17 00:00:00 2001 From: Sean McGivern Date: Wed, 24 Jul 2019 12:37:05 +0100 Subject: Remove line profiler from performance bar 1. The output isn't great. It can be hard to find hotspots and, even when you do find them, to find why those are hotspots. 2. It uses some jQuery-specific frontend code which we can remove now that we don't have this any more. 3. It's only possible to profile the initial request, not any subsequent AJAX requests. --- .../monitoring/performance/img/performance_bar.png | Bin 127198 -> 113617 bytes .../img/performance_bar_line_profiling.png | Bin 93063 -> 0 bytes .../monitoring/performance/performance_bar.md | 2 -- 3 files changed, 2 deletions(-) delete mode 100644 doc/administration/monitoring/performance/img/performance_bar_line_profiling.png (limited to 'doc/administration') diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png index 8a6f8b3b273..89b09054d46 100644 Binary files a/doc/administration/monitoring/performance/img/performance_bar.png and b/doc/administration/monitoring/performance/img/performance_bar.png differ diff --git a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png b/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png deleted file mode 100644 index a55ce753101..00000000000 Binary files a/doc/administration/monitoring/performance/img/performance_bar_line_profiling.png and /dev/null differ diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 15eab7dcab0..2cc78ccc03c 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -14,8 +14,6 @@ It allows you to see (from left to right): ![Gitaly profiling using the Performance Bar](img/performance_bar_gitaly_calls.png) - time taken and number of [Rugged] calls, click through for details of these calls ![Rugged profiling using the Performance Bar](img/performance_bar_rugged_calls.png) -- profile of the code used to generate the page, line by line. In the profile view, the numbers in the left panel represent wall time, cpu time, and number of calls (based on [rblineprof](https://github.com/tmm1/rblineprof)). - ![Line profiling using the Performance Bar](img/performance_bar_line_profiling.png) - time taken and number of Redis calls, click through for details of these calls ![Redis profiling using the Performance Bar](img/performance_bar_redis_calls.png) - time taken and number of Ruby GC calls -- cgit v1.2.1 From 21977d9a16abe4ee87307aed33777a2a18436842 Mon Sep 17 00:00:00 2001 From: Douglas Barbosa Alexandre Date: Tue, 30 Jul 2019 17:10:23 -0300 Subject: Remove duplicated words --- doc/administration/custom_hooks.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) (limited to 'doc/administration') diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 113514e1ee8..32462a95a1a 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -21,7 +21,7 @@ administrators can add custom git hooks to any GitLab project. ## Create a custom Git hook for a repository Server-side Git hooks are typically placed in the repository's `hooks` -subdirectory. In GitLab, hook directories are are symlinked to the gitlab-shell +subdirectory. In GitLab, hook directories are symlinked to the gitlab-shell `hooks` directory for ease of maintenance between gitlab-shell upgrades. Custom hooks are implemented differently, but the behavior is exactly the same once the hook is created. Follow the steps below to set up a custom hook for a -- cgit v1.2.1 From 47def8eb2f1cd2ff4b3ec45992f5fbb4497258d3 Mon Sep 17 00:00:00 2001 From: Evan Read Date: Wed, 31 Jul 2019 08:36:37 +0000 Subject: Move version info for Geo --- doc/administration/geo/replication/index.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) (limited to 'doc/administration') diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index f0d329d5296..b5789c87e47 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -1,5 +1,11 @@ # Geo Replication **(PREMIUM ONLY)** +> - Introduced in GitLab Enterprise Edition 8.9. +> - Using Geo in combination with +> [High Availability](../../high_availability/README.md) +> is considered **Generally Available** (GA) in +> [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. + Geo is the solution for widely distributed development teams. ## Overview @@ -8,14 +14,14 @@ Fetching large repositories can take a long time for teams located far from a si Geo provides local, read-only instances of your GitLab instances, reducing the time it takes to clone and fetch large repositories and speeding up development. +> **Notes:** +> > - Geo is part of [GitLab Premium](https://about.gitlab.com/pricing/#self-managed). -> - Introduced in GitLab Enterprise Edition 8.9. > - We recommend you use: > - At least GitLab Enterprise Edition 10.0 for basic Geo features. > - The latest version for a better experience. > - Make sure that all nodes run the same GitLab version. > - Geo requires PostgreSQL 9.6 and Git 2.9, in addition to GitLab's usual [minimum requirements](../../../install/requirements.md). -> - Using Geo in combination with [High Availability](../../high_availability/README.md) is considered **Generally Available** (GA) in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). -- cgit v1.2.1 From b0d9b9fc7b7f40deb878ce286dde0c2bde35254b Mon Sep 17 00:00:00 2001 From: Ben Kochie Date: Tue, 23 Jul 2019 18:14:35 +0200 Subject: Update HA resource descriptions * Use `GiB memory` to describe memory needs. * Add local storage recommendation for monitoring node. * Better align CPU/Memory to match common cloud provider instnace types. * Update requirements based on the new 10k user standard. --- doc/administration/high_availability/README.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) (limited to 'doc/administration') diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 42516d811a0..bb824c9259b 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -172,14 +172,14 @@ environment that supports about 10,000 users. The specifications below are a representation of the work so far. The specifications may be adjusted in the future based on additional testing and iteration. -- 3 PostgreSQL - 4 CPU, 8GB RAM per node -- 1 PgBouncer - 2 CPU, 4GB RAM -- 2 Redis - 2 CPU, 8GB RAM per node -- 3 Consul/Sentinel - 2 CPU, 2GB RAM per node -- 4 Sidekiq - 4 CPU, 8GB RAM per node -- 5 GitLab application nodes - 20 CPU, 64GB RAM per node -- 1 Gitaly - 20 CPU, 64GB RAM -- 1 Monitoring node - 4 CPU, 8GB RAM +- 3 PostgreSQL - 4 CPU, 16GiB memory per node +- 1 PgBouncer - 2 CPU, 4GiB memory +- 2 Redis - 2 CPU, 8GiB memory per node +- 3 Consul/Sentinel - 2 CPU, 2GiB memory per node +- 4 Sidekiq - 4 CPU, 16GiB memory per node +- 5 GitLab application nodes - 16 CPU, 64GiB memory per node +- 1 Gitaly - 16 CPU, 64GiB memory +- 1 Monitoring node - 2 CPU, 8GiB memory, 100GiB local storage ### Fully Distributed -- cgit v1.2.1 From 9241ba276d1d4396b5d3eb3e847a914eeeba55b2 Mon Sep 17 00:00:00 2001 From: Douglas Barbosa Alexandre Date: Mon, 29 Jul 2019 21:05:24 -0300 Subject: Update list of unreplicated/unverified data types --- doc/administration/geo/replication/index.md | 67 +++++++++++++++++------------ 1 file changed, 40 insertions(+), 27 deletions(-) (limited to 'doc/administration') diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index b5789c87e47..dc49a95e008 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -245,35 +245,48 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. -### Limitations on replication - -Only the following items are replicated to the **secondary** node: - -- All database content. For example, snippets, epics, issues, merge requests, groups, and project metadata. -- Project repositories. -- Project wiki repositories. -- User uploads. For example, attachments to issues, merge requests, epics, and avatars. -- CI job artifacts and traces. +### Limitations on replication/verification + +The following table lists the GitLab features along with their replication +and verification status on a **secondary** node. + +You can keep track of the progress to include the missing items in: + +- [ee-893](https://gitlab.com/groups/gitlab-org/-/epics/893). +- [ee-1430](https://gitlab.com/groups/gitlab-org/-/epics/1430). + +| Feature | Replicated | Verified | +|-----------|------------|----------| +| All database content (e.g. snippets, epics, issues, merge requests, groups, and project metadata) | Yes | Yes | +| Project repository | Yes | Yes | +| Project wiki repository | Yes | Yes | +| Project designs repository | No | No | +| Uploads (e.g. attachments to issues, merge requests, epics, and avatars) | Yes | Yes, only on transfer, or manually (1) | +| LFS Objects | Yes | Yes, only on transfer, or manually (1) | +| CI job artifacts (other than traces) | Yes | No, only manually (1) | +| Archived traces | Yes | Yes, only on transfer, or manually (1) | +| Personal snippets | Yes | Yes | +| Version-controlled personal snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No | +| Project snippets | Yes | Yes | +| Version-controlled project snippets ([unsupported](https://gitlab.com/gitlab-org/gitlab-ce/issues/13426)) | No | No | +| Object pools for forked project deduplication | No | No | +| [Server-side Git Hooks](../../custom_hooks.md) | No | No | +| [Elasticsearch integration](../../../integration/elasticsearch.md) | No | No | +| [GitLab Pages](../../pages/index.md) | No | No | +| [Container Registry](../../container_registry.md) ([track progress](https://gitlab.com/gitlab-org/gitlab-ee/issues/2870)) | No | No | +| [NPM Registry](../../npm_registry.md) | No | No | +| [Maven Packages](../../maven_packages.md) | No | No | +| [External merge request diffs](../../merge_request_diffs.md) | No, if they are on-disk | No | +| Content in object storage ([track progress](https://gitlab.com/groups/gitlab-org/-/epics/1526)) | No | No | + +1. The integrity can be verified manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. DANGER: **DANGER** -Data not on this list is unavailable on the **secondary** node. Failing over without manually replicating data not on this list will cause the data to be **lost**. - -### Examples of data not replicated - -Take special note that these examples of GitLab features are both: - -- Commonly used. -- **Not** replicated by Geo at present. - -Examples include: - -- [Elasticsearch integration](../../../integration/elasticsearch.md). -- [Container Registry](../../container_registry.md). [Object Storage](object_storage.md) can mitigate this. -- [GitLab Pages](../../pages/index.md). -- [Mattermost integration](https://docs.gitlab.com/omnibus/gitlab-mattermost/). - -CAUTION: **Caution:** -If you wish to use them on a **secondary** node, or to execute a failover successfully, you will need to replicate their data using some other means. +Features not on this list, or with **No** in the **Replicated** column, +are not replicated on the **secondary** node. Failing over without manually +replicating data from those features will cause the data to be **lost**. +If you wish to use those features on a **secondary** node, or to execute a failover +successfully, you must replicate their data using some other means. ## Frequently Asked Questions -- cgit v1.2.1 From ad228df77421c1b312cdfc6288d1ec57c6a421df Mon Sep 17 00:00:00 2001 From: Evan Read Date: Thu, 1 Aug 2019 08:37:18 +0000 Subject: General improvements to LDAP docs --- doc/administration/auth/ldap-ee.md | 126 ++++++++++++++++++++----------------- doc/administration/auth/ldap.md | 50 ++++++++++----- 2 files changed, 103 insertions(+), 73 deletions(-) (limited to 'doc/administration') diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 1b7af2d945a..34f3cfa353f 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -4,37 +4,27 @@ type: reference # LDAP Additions in GitLab EE **(STARTER ONLY)** -This is a continuation of the main [LDAP documentation](ldap.md), detailing LDAP -features specific to GitLab Enterprise Edition Starter, Premium and Ultimate. +This section documents LDAP features specific to to GitLab Enterprise Edition +[Starter](https://about.gitlab.com/pricing/#self-managed) and above. -## Overview - -[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) -stands for **Lightweight Directory Access Protocol**, which -is a standard application protocol for -accessing and maintaining distributed directory information services -over an Internet Protocol (IP) network. - -GitLab integrates with LDAP to support **user authentication**. This integration -works with most LDAP-compliant directory servers, including Microsoft Active -Directory, Apple Open Directory, Open LDAP, and 389 Server. -**GitLab Enterprise Edition** includes enhanced integration, including group -membership syncing. +For documentation relevant to both Community Edition and Enterprise Edition, +see the main [LDAP documentation](ldap.md). ## Use cases -- User sync: Once a day, GitLab will update users against LDAP +- User sync: Once a day, GitLab will update users against LDAP. - Group sync: Once an hour, GitLab will update group membership - based on LDAP group members + based on LDAP group members. -## Multiple LDAP servers +## Multiple LDAP servers **(STARTER ONLY)** With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers that your GitLab instance will connect to. -To add another LDAP server, you can start by duplicating the settings under -[the main configuration](ldap.md#configuration) and edit them to match the -additional LDAP server. +To add another LDAP server: + +1. Duplicating the settings under [the main configuration](ldap.md#configuration). +1. Edit them to match the additional LDAP server. Be sure to choose a different provider ID made of letters a-z and numbers 0-9. This ID will be stored in the database so that GitLab can remember which LDAP @@ -47,19 +37,19 @@ users against LDAP. The process will execute the following access checks: -1. Ensure the user is still present in LDAP -1. If the LDAP server is Active Directory, ensure the user is active (not - blocked/disabled state). This will only be checked if - `active_directory: true` is set in the LDAP configuration [^1] +- Ensure the user is still present in LDAP. +- If the LDAP server is Active Directory, ensure the user is active (not + blocked/disabled state). This will only be checked if + `active_directory: true` is set in the LDAP configuration. [^1] The user will be set to `ldap_blocked` state in GitLab if the above conditions fail. This means the user will not be able to login or push/pull code. The process will also update the following user information: -1. Email address -1. If `sync_ssh_keys` is set, SSH public keys -1. If Kerberos is enabled, Kerberos identity +- Email address. +- If `sync_ssh_keys` is set, SSH public keys. +- If Kerberos is enabled, Kerberos identity. NOTE: **Note:** The LDAP sync process updates existing users while new users will @@ -72,9 +62,6 @@ first time GitLab will trigger a sync for groups the user should be a member of. That way they don't need to wait for the hourly sync to be granted access to their groups and projects. -In GitLab Premium, we can also add a GitLab group to sync with one or multiple LDAP groups or we can -also add a filter. The filter must comply with the syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). - A group sync process will run every hour on the hour, and `group_base` must be set in LDAP configuration for LDAP synchronizations based on group CN to work. This allows GitLab group membership to be automatically updated based on LDAP group members. @@ -120,13 +107,26 @@ following. 1. [Restart GitLab][restart] for the changes to take effect. ---- - To take advantage of group sync, group owners or maintainers will need to create an -LDAP group link in their group **Settings > LDAP Groups** page. Multiple LDAP -groups and/or filters can be linked with a single GitLab group. When the link is -created, an access level/role is specified (Guest, Reporter, Developer, Maintainer, -or Owner). +LDAP group link in their group **Settings > LDAP Groups** page. + +Multiple LDAP groups and [filters](#filters-premium-only) can be linked with +a single GitLab group. When the link is created, an access level/role is +specified (Guest, Reporter, Developer, Maintainer, or Owner). + +### Filters **(PREMIUM ONLY)** + +In GitLab Premium, you can add an LDAP user filter for group synchronization. +Filters allow for complex logic without creating a special LDAP group. + +To sync GitLab group membership based on an LDAP filter: + +1. Open the **LDAP Synchronization** page for the GitLab group. +1. Select **LDAP user filter** as the **Sync method**. +1. Enter an LDAP user filter in the **LDAP user filter** field. + +The filter must comply with the +syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). ## Administrator sync @@ -190,10 +190,13 @@ group, as opposed to the full DN. ## Global group memberships lock "Lock memberships to LDAP synchronization" setting allows instance administrators -to lock down user abilities to invite new members to a group. When enabled following happens: +to lock down user abilities to invite new members to a group. + +When enabled, the following applies: -1. Only administrator can manage memberships of any group including access levels. -1. Users are not allowed to share project with other groups or invite members to a project created in a group. +- Only administrator can manage memberships of any group including access levels. +- Users are not allowed to share project with other groups or invite members to + a project created in a group. ## Adjusting LDAP user sync schedule @@ -333,10 +336,18 @@ administrative duties. ### Supported LDAP group types/attributes -GitLab supports LDAP groups that use member attributes `member`, `submember`, -`uniquemember`, `memberof` and `memberuid`. This means group sync supports, at -least, LDAP groups with object class `groupOfNames`, `posixGroup`, and -`groupOfUniqueName`. Other object classes should work fine as long as members +GitLab supports LDAP groups that use member attributes: + +- `member` +- `submember` +- `uniquemember` +- `memberof` +- `memberuid`. + +This means group sync supports, at least, LDAP groups with object class: +`groupOfNames`, `posixGroup`, and `groupOfUniqueName`. + +Other object classes should work fine as long as members are defined as one of the mentioned attributes. This also means GitLab supports Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. Other LDAP servers should work, too. @@ -344,11 +355,12 @@ Other LDAP servers should work, too. Active Directory also supports nested groups. Group sync will recursively resolve membership if `active_directory: true` is set in the configuration file. -> **Note:** Nested group membership will only be resolved if the nested group - also falls within the configured `group_base`. For example, if GitLab sees a - nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but - the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` - will be ignored. +NOTE: **Note:** +Nested group membership will only be resolved if the nested group +also falls within the configured `group_base`. For example, if GitLab sees a +nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but +the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` +will be ignored. ### Queries @@ -403,7 +415,7 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server [^1]: In Active Directory, a user is marked as disabled/blocked if the user account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) - has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/ + has bit 2 set. See for more information. ### User DN has changed @@ -423,10 +435,10 @@ things to check to debug the situation. - Ensure LDAP configuration has a `group_base` specified. This configuration is required for group sync to work properly. - Ensure the correct LDAP group link is added to the GitLab group. Check group - links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**. -- Check that the user has an LDAP identity + links by visiting the GitLab group, then **Settings dropdown > LDAP groups**. +- Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. Navigate to **Admin area -> Users**. + 1. Navigate to **Admin area > Users**. 1. Search for the user 1. Open the user, by clicking on their name. Do not click 'Edit'. 1. Navigate to the **Identities** tab. There should be an LDAP identity with @@ -437,7 +449,7 @@ Often, the best way to learn more about why group sync is behaving a certain way is to enable debug logging. There is verbose output that details every step of the sync. -1. Start a Rails console +1. Start a Rails console: ```bash # For Omnibus installations @@ -446,6 +458,7 @@ step of the sync. # For installations from source sudo -u git -H bundle exec rails console production ``` + 1. Set the log level to debug (only for this session): ```ruby @@ -540,8 +553,9 @@ and more DNs may be added, or existing entries modified, based on additional LDAP group lookups. The very last occurrence of this entry should indicate exactly which users GitLab believes should be added to the group. -> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' - and 50 is 'Owner' +NOTE: **Note:** +10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' +and 50 is 'Owner'. ```bash Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index be05a4d63a7..beacaa99d60 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -7,16 +7,29 @@ type: reference # LDAP GitLab integrates with LDAP to support user authentication. -This integration works with most LDAP-compliant directory -servers, including Microsoft Active Directory, Apple Open Directory, Open LDAP, -and 389 Server. GitLab Enterprise Editions include enhanced integration, + +This integration works with most LDAP-compliant directory servers, including: + +- Microsoft Active Directory +- Apple Open Directory +- Open LDAP +- 389 Server. + +GitLab Enterprise Editions (EE) include enhanced integration, including group membership syncing as well as multiple LDAP servers support. -## GitLab EE +For more details about EE-specific LDAP features, see the +[LDAP Enterprise Edition documentation](ldap-ee.md). -The information on this page is relevant for both GitLab CE and EE. For more -details about EE-specific LDAP features, see the -[LDAP Enterprise Edition documentation](ldap-ee.md). **(STARTER ONLY)** +NOTE: **Note:** +The information on this page is relevant for both GitLab CE and EE. + +## Overview + +[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +stands for **Lightweight Directory Access Protocol**, which is a standard +application protocol for accessing and maintaining distributed directory +information services over an Internet Protocol (IP) network. ## Security @@ -64,9 +77,12 @@ NOTE: **Note**: In GitLab Enterprise Edition Starter, you can configure multiple LDAP servers to connect to one GitLab server. -For a complete guide on configuring LDAP with GitLab Community Edition, please check -the admin guide [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). -For GitLab Enterprise Editions, see also [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)** +For a complete guide on configuring LDAP with: + +- GitLab Community Edition, see + [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). +- Enterprise Editions, see + [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)** To enable LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus @@ -384,7 +400,7 @@ production: Tip: If you want to limit access to the nested members of an Active Directory group, you can use the following syntax: -``` +```text (memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) ``` @@ -402,13 +418,13 @@ The `user_filter` DN can contain special characters. For example: - A comma: - ``` + ```text OU=GitLab, Inc,DC=gitlab,DC=com ``` - Open and close brackets: - ``` + ```text OU=Gitlab (Inc),DC=gitlab,DC=com ``` @@ -417,13 +433,13 @@ The `user_filter` DN can contain special characters. For example: - Escape commas with `\2C`. For example: - ``` + ```text OU=GitLab\2C Inc,DC=gitlab,DC=com ``` - Escape open and close brackets with `\28` and `\29`, respectively. For example: - ``` + ```text OU=Gitlab \28Inc\29,DC=gitlab,DC=com ``` @@ -507,11 +523,11 @@ timeout), the login is rejected and a message will be logged to ### Debug LDAP user filter with ldapsearch -This example uses ldapsearch and assumes you are using ActiveDirectory. The +This example uses `ldapsearch` and assumes you are using ActiveDirectory. The following query returns the login names of the users that will be allowed to log in to GitLab if you configure your own user_filter. -``` +```sh ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName ``` -- cgit v1.2.1 From a0ccab454667157a6f7717dd8ed7dc1641618fc0 Mon Sep 17 00:00:00 2001 From: Harish Ramachandran Date: Thu, 1 Aug 2019 09:27:04 +0000 Subject: Troubleshooting steps when there are different Gitaly node secrets --- doc/administration/gitaly/index.md | 91 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) (limited to 'doc/administration') diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 0f547ef03bf..f6f02221fe3 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -510,3 +510,94 @@ implemented as calls to `gitaly-ruby`: ``` sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 ``` + +### Repository changes fail with a `401 Unauthorized` error + +If you're running Gitaly on its own server and notice that users can +successfully clone and fetch repositories (via both SSH and HTTPS), but can't +push to them or make changes to the repository in the web UI without getting a +`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate +with the other nodes due to having the [wrong secrets file](#3-gitaly-server-configuration). + +Confirm the following are all true: + +- When any user performs a `git push` to any repository on this Gitaly node, it + fails with the following error (note the `401 Unauthorized`): + + ```sh + remote: GitLab: 401 Unauthorized + To + ! [remote rejected] branch-name -> branch-name (pre-receive hook declined) + error: failed to push some refs to '' + ``` + +- When any user adds or modifies a file from the repository using the GitLab + UI, it immediatley fails with a red `401 Unauthorized` banner. +- Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) + successfully creates the project but doesn't create the README. +- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.md#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors + when reaching the `/api/v4/internal/allowed` endpoint: + + ```sh + # api_json.log + { + "time": "2019-07-18T00:30:14.967Z", + "severity": "INFO", + "duration": 0.57, + "db": 0, + "view": 0.57, + "status": 401, + "method": "POST", + "path": "\/api\/v4\/internal\/allowed", + "params": [ + { + "key": "action", + "value": "git-receive-pack" + }, + { + "key": "changes", + "value": "REDACTED" + }, + { + "key": "gl_repository", + "value": "REDACTED" + }, + { + "key": "project", + "value": "\/path\/to\/project.git" + }, + { + "key": "protocol", + "value": "web" + }, + { + "key": "env", + "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" + }, + { + "key": "user_id", + "value": "2" + }, + { + "key": "secret_token", + "value": "[FILTERED]" + } + ], + "host": "gitlab.example.com", + "ip": "REDACTED", + "ua": "Ruby", + "route": "\/api\/:version\/internal\/allowed", + "queue_duration": 4.24, + "gitaly_calls": 0, + "gitaly_duration": 0, + "correlation_id": "XPUZqTukaP3" + } + + # nginx_access.log + [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" + ``` + +To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-server-configuration) +on the Gitaly node matches the one on all other nodes. If it doesn't match, +update the secrets file on the Gitaly node to match the others, then +[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). -- cgit v1.2.1 From 3519111296c9f05ec7202b1a9c75cb09c65948d3 Mon Sep 17 00:00:00 2001 From: George Koltsov Date: Fri, 2 Aug 2019 06:32:03 +0000 Subject: Update ldap#security section --- doc/administration/auth/ldap.md | 17 ++++++++++------- 1 file changed, 10 insertions(+), 7 deletions(-) (limited to 'doc/administration') diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index beacaa99d60..186bf4c4825 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -33,15 +33,18 @@ information services over an Internet Protocol (IP) network. ## Security -GitLab assumes that LDAP users are not able to change their LDAP 'mail', 'email' -or 'userPrincipalName' attribute. An LDAP user who is allowed to change their -email on the LDAP server can potentially -[take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) -on your GitLab server. +GitLab assumes that LDAP users: + +- Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attribute. + An LDAP user who is allowed to change their email on the LDAP server can potentially + [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) + on your GitLab server. +- Have unique email addresses, otherwise it is possible for LDAP users with the same + email address to share the same GitLab account. We recommend against using LDAP integration if your LDAP users are -allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on -the LDAP server. +allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on +the LDAP server or share email addresses. ### User deletion -- cgit v1.2.1 From 4e23f45e0d6326940787e368030adeb4a9a417ee Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cindy=20Pallares=20=F0=9F=A6=89?= Date: Fri, 2 Aug 2019 16:56:58 +0000 Subject: Remove CephFS from example We recommend avoiding CephFS https://docs.gitlab.com/ee/administration/high_availability/nfs.html#avoid-using-cephfs-and-glusterfs --- doc/administration/repository_storage_paths.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) (limited to 'doc/administration') diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index ad3a9b19c3c..b1a870210a8 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -57,10 +57,10 @@ storage2: Now that you've read that big fat warning above, let's edit the configuration files and add the full paths of the alternative repository storage paths. In -the example below, we add two more mountpoints that are named `nfs` and `cephfs` +the example below, we add two more mountpoints that are named `nfs_1` and `nfs_2` respectively. -NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. +NOTE: **Note:** This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. **For installations from source** @@ -73,10 +73,10 @@ NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS storages: # You must have at least a 'default' storage path. default: path: /home/git/repositories - nfs: - path: /mnt/nfs/repositories - cephfs: - path: /mnt/cephfs/repositories + nfs_1: + path: /mnt/nfs1/repositories + nfs_2: + path: /mnt/nfs2/repositories ``` 1. [Restart GitLab][restart-gitlab] for the changes to take effect. @@ -96,8 +96,8 @@ working, you can remove the `repos_path` line. ```ruby git_data_dirs({ "default" => { "path" => "/var/opt/gitlab/git-data" }, - "nfs" => { "path" => "/mnt/nfs/git-data" }, - "cephfs" => { "path" => "/mnt/cephfs/git-data" } + "nfs_1" => { "path" => "/mnt/nfs1/git-data" }, + "nfs_2" => { "path" => "/mnt/nfs2/git-data" } }) ``` -- cgit v1.2.1 From 615a94435318717e1d18e79d2acfd21529c3fe1b Mon Sep 17 00:00:00 2001 From: Cody West Date: Fri, 2 Aug 2019 12:42:47 -0500 Subject: Update requirements wording I'm adding some wording to our hardware requirements to explain that needs vary by workload. I also included some examples of factors that may influence the workload. --- doc/administration/high_availability/README.md | 2 ++ 1 file changed, 2 insertions(+) (limited to 'doc/administration') diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index bb824c9259b..56665ba8b9a 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -172,6 +172,8 @@ environment that supports about 10,000 users. The specifications below are a representation of the work so far. The specifications may be adjusted in the future based on additional testing and iteration. +NOTE: **Note:** The specifications here were performance tested against a specific coded workload. Your exact needs may be more, depending on your workload. Your workload is influenced by factors such as - but not limited to - how active your users are, how much automation you use, mirroring, and repo/change size. + - 3 PostgreSQL - 4 CPU, 16GiB memory per node - 1 PgBouncer - 2 CPU, 4GiB memory - 2 Redis - 2 CPU, 8GiB memory per node -- cgit v1.2.1 From 61e1a1492507ad2b77a48ef0a890de62caa590b3 Mon Sep 17 00:00:00 2001 From: Marcel Amirault Date: Mon, 5 Aug 2019 02:13:46 +0000 Subject: Expand markdown linting rules for docs MD002 - First header should be level 1 MD006 - Start bullets at beginning of line MD019 - No multiple spaces after header style MD022 - Headers surrounded by blank lines MD025 - Only 1 level 1 header MD028 - No blank lines within blockquote MD038 - Spaces inside code span elements --- doc/administration/operations/fast_ssh_key_lookup.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) (limited to 'doc/administration') diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index ea69378b249..e787af798bc 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -71,10 +71,10 @@ sudo service sshd reload Confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo. -> **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in +NOTE: **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in GitLab 11.11 and later. -> **Warning:** Do not disable writes until SSH is confirmed to be working +CAUTION: **Caution:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. In the case of lookup failures (which are common), the `authorized_keys` -- cgit v1.2.1 From bcfef87afc08a64d460315d2f80c7cd3be0f0b8c Mon Sep 17 00:00:00 2001 From: Gabriel Mazetto Date: Mon, 5 Aug 2019 06:45:43 +0000 Subject: Geo: Some update instructions should be for 9.0.x only --- .../geo/replication/updating_the_geo_nodes.md | 34 ++++++++++------------ 1 file changed, 16 insertions(+), 18 deletions(-) (limited to 'doc/administration') diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 550b3b07a95..39174780e24 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -10,10 +10,23 @@ all you need to do is update GitLab itself: 1. Log into each node (**primary** and **secondary** nodes). 1. [Update GitLab][update]. -1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when - the tracking database is enabled. 1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. +### Check status after updating + +Now that the update process is complete, you may want to check whether +everything is working correctly: + +1. Run the Geo raketask on all nodes, everything should be green: + + ```sh + sudo gitlab-rake gitlab:geo:check + ``` + +1. Check the **primary** node's Geo dashboard for any errors. +1. Test the data replication by pushing code to the **primary** node and see if it + is received by **secondary** nodes. + ## Upgrading to GitLab 12.1 By default, GitLab 12.1 will attempt to automatically upgrade the embedded PostgreSQL server to 10.7 from 9.6. Please see [the omnibus documentation](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-geo-instance) for the recommended procedure. @@ -419,22 +432,7 @@ is prepended with the relevant node for better clarity: sudo gitlab-ctl start ``` -## Check status after updating - -Now that the update process is complete, you may want to check whether -everything is working correctly: - -1. Run the Geo raketask on all nodes, everything should be green: - - ```sh - sudo gitlab-rake gitlab:geo:check - ``` - -1. Check the **primary** node's Geo dashboard for any errors. -1. Test the data replication by pushing code to the **primary** node and see if it - is received by **secondary** nodes. - -## Update tracking database on **secondary** node +### Update tracking database on **secondary** node After updating a **secondary** node, you might need to run migrations on the tracking database. The tracking database was added in GitLab 9.1, -- cgit v1.2.1 From 969e6b3318c3f663ae378c2b470c5fa022f2652a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Cindy=20Pallares=20=F0=9F=A6=89?= Date: Mon, 5 Aug 2019 13:30:07 +0000 Subject: Update Gitaly troubleshooting --- doc/administration/gitaly/index.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) (limited to 'doc/administration') diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index f6f02221fe3..432056d48c7 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -469,7 +469,16 @@ One current feature of GitLab that still requires a shared directory (NFS) is There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) to eliminate the need for NFS to support GitLab Pages. -## Troubleshooting +## Troubleshooting Gitaly + +### Commits, pushes, and clones return a 401 + +``` +remote: GitLab: 401 Unauthorized +``` + +You will need to sync your `gitlab-secrets.json` file with your GitLab +app nodes. ### `gitaly-debug` -- cgit v1.2.1 From dd2b87241d3caccb240e901cbb2e6892d758f588 Mon Sep 17 00:00:00 2001 From: Marcel Amirault Date: Mon, 5 Aug 2019 18:02:23 +0000 Subject: Bring diagnostics_tools.md from debug project to docs --- doc/administration/index.md | 2 ++ .../troubleshooting/diagnostics_tools.md | 27 ++++++++++++++++++++++ 2 files changed, 29 insertions(+) create mode 100644 doc/administration/troubleshooting/diagnostics_tools.md (limited to 'doc/administration') diff --git a/doc/administration/index.md b/doc/administration/index.md index 00c8863f200..febee3e5af8 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -185,3 +185,5 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong - [Log system](logs.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. +- Useful [diagnostics tools](troubleshooting/diagnostics_tools.md) that are sometimes used by the GitLab + Support team. diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md new file mode 100644 index 00000000000..ab3b25f0e97 --- /dev/null +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -0,0 +1,27 @@ +--- +type: reference +--- + +# Diagnostics tools + +These are some of the diagnostics tools the GitLab Support team uses during troubleshooting. +They are listed here for transparency, and they may be useful for users with experience +with troubleshooting GitLab. If you are currently having an issue with GitLab, you +may want to check your [support options](https://about.gitlab.com/support/) first, +before attempting to use these tools. + +## gitlabsos + +The [gitlabsos](https://gitlab.com/gitlab-com/support/toolbox/gitlabsos/) utility +provides a unified method of gathering info and logs from GitLab and the system it's +running on. + +## strace-parser + +[strace-parser](https://gitlab.com/wchandler/strace-parser) is a small tool to analyze +and summarize raw strace data. + +## Pritaly + +[Pritaly](https://gitlab.com/wchandler/pritaly) takes Gitaly logs and colorizes output +or converts the logs to JSON. -- cgit v1.2.1 From 79a39a5c5543331cdd54f95fdfd323006fa9fcd2 Mon Sep 17 00:00:00 2001 From: Ronald van Zon Date: Mon, 5 Aug 2019 21:24:32 +0000 Subject: Add example to plugins file --- doc/administration/plugins.md | 32 +++++++++++++++++++++++++++++++- 1 file changed, 31 insertions(+), 1 deletion(-) (limited to 'doc/administration') diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md index 4302667caf5..4cf3c607dae 100644 --- a/doc/administration/plugins.md +++ b/doc/administration/plugins.md @@ -52,7 +52,37 @@ as appropriate. The plugins file list is updated for each event, there is no need to restart GitLab to apply a new plugin. If a plugin executes with non-zero exit code or GitLab fails to execute it, a -message will be logged to `plugin.log`. +message will be logged to: + +- `gitlab-rails/plugin.log` in an Omnibus installation. +- `log/plugin.log` in a source installation. + +## Creating plugins + +Below is an example that will only response on the event `project_create` and +will inform the admins from the GitLab instance that a new project has been created. + +```ruby +# By using the embedded ruby version we eliminate the possibility that our chosen language +# would be unavailable from +#!/opt/gitlab/embedded/bin/ruby +require 'json' +require 'mail' + +# The incoming variables are in JSON format so we need to parse it first. +ARGS = JSON.parse(STDIN.read) + +# We only want to trigger this plugin on the event project_create +return unless ARGS['event_name'] == 'project_create' + +# We will inform our admins of our gitlab instance that a new project is created +Mail.deliver do + from 'info@gitlab_instance.com' + to 'admin@gitlab_instance.com' + subject "new project " + ARGS['name'] + body ARGS['owner_name'] + 'created project ' + ARGS['name'] +end +``` ## Validation -- cgit v1.2.1