diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-09-16 18:09:47 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-09-16 18:09:47 +0000 |
| commit | bf1600d157465f9408aace91073954fd5790c054 (patch) | |
| tree | f317bb99330769c4eb37621c860af014810e554b /doc | |
| parent | 6de7d2c195a8a7fa5702cafa4365f7a9fcac37cd (diff) | |
| download | gitlab-ce-bf1600d157465f9408aace91073954fd5790c054.tar.gz | |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc')
19 files changed, 698 insertions, 79 deletions
diff --git a/doc/administration/geo/disaster_recovery/promotion_runbook.md b/doc/administration/geo/disaster_recovery/promotion_runbook.md new file mode 100644 index 00000000000..fb2353513df --- /dev/null +++ b/doc/administration/geo/disaster_recovery/promotion_runbook.md @@ -0,0 +1,269 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + +CAUTION: **Caution:** +This runbook is in **alpha**. For complete, production-ready documentation, see the +[disaster recovery documentation](index.md). + +# Disaster Recovery (Geo) promotion runbooks **(PREMIUM ONLY)** + +## Geo planned failover runbook 1 + +| Component | Configuration | +| ----------- | --------------- | +| PostgreSQL | Omnibus-managed | +| Geo site | Single-node | +| Secondaries | One | + +This runbook will guide you through a planned failover of a single-node Geo site +with one secondary. The following general architecture is assumed: + +```mermaid +graph TD + subgraph main[Geo deployment] + subgraph Primary[Primary site] + Node_1[(GitLab node)] + end + subgraph Secondary1[Secondary site] + Node_2[(GitLab node)] + end + end +``` + +This guide will result in the following: + +1. An offline primary. +1. A promoted secondary that is now the new primary. + +What is not covered: + +1. Re-adding the old **primary** as a secondary. +1. Adding a new secondary. + +### Preparation + +NOTE: **Note:** +Before following any of those steps, make sure you have `root` access to the +**secondary** to promote it, since there isn't provided an automated way to +promote a Geo replica and perform a failover. + +On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to +review its status. Replicated objects (shown in green) should be close to 100%, +and there should be no failures (shown in red). If a large proportion of +objects aren't yet replicated (shown in gray), consider giving the node more +time to complete. + + + +If any objects are failing to replicate, this should be investigated before +scheduling the maintenance window. After a planned failover, anything that +failed to replicate will be **lost**. + +You can use the +[Geo status API](../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) +to review failed objects and the reasons for failure. +A common cause of replication failures is the data being missing on the +**primary** node - you can resolve these failures by restoring the data from backup, +or removing references to the missing data. + +The maintenance window won't end until Geo replication and verification is +completely finished. To keep the window as short as possible, you should +ensure these processes are close to 100% as possible during active use. + +If the **secondary** node is still replicating data from the **primary** node, +follow these steps to avoid unnecessary data loss: + +1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) + is implemented, updates must be prevented from happening manually to the + **primary**. Note that your **secondary** node still needs read-only + access to the **primary** node during the maintenance window: + + 1. At the scheduled time, using your cloud provider or your node's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and + the **secondary** node's IP. + + For instance, you can run the following commands on the **primary** node: + + ```shell + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` + + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. + + 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + another IP. The server should refuse connection. + + 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + existing Git repository with an SSH remote URL. The server should refuse + connection. + + 1. On the **primary** node, disable non-Geo periodic background jobs by navigating + to **Admin Area > Monitoring > Background Jobs > Cron**, clicking `Disable All`, + and then clicking `Enable` for the `geo_sidekiq_cron_config_worker` cron job. + This job will re-enable several other cron jobs that are essential for planned + failover to complete successfully. + +1. Finish replicating and verifying all data: + + CAUTION: **Caution:** + Not all data is automatically replicated. Read more about + [what is excluded](planned_failover.md#not-all-data-is-automatically-replicated). + + 1. If you are manually replicating any + [data not managed by Geo](../replication/datatypes.md#limitations-on-replicationverification), + trigger the final replication process now. + 1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all queues except those with `geo` in the name to drop to 0. + These queues contain work that has been submitted by your users; failing over + before it is completed will cause the work to be lost. + 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the + following conditions to be true of the **secondary** node you are failing over to: + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). + + 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. + 1. On the **secondary** node, use [these instructions](../../raketasks/check.md) + to verify the integrity of CI artifacts, LFS objects, and uploads in file + storage. + + At this point, your **secondary** node will contain an up-to-date copy of everything the + **primary** node has, meaning nothing will be lost when you fail over. + +1. In this final step, you need to permanently disable the **primary** node. + + CAUTION: **Caution:** + When the **primary** node goes offline, there may be data saved on the **primary** node + that has not been replicated to the **secondary** node. This data should be treated + as lost if you proceed. + + TIP: **Tip:** + If you plan to [update the **primary** domain DNS record](index.md#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. + + When performing a failover, we want to avoid a split-brain situation where + writes can occur in two different GitLab instances. So to prepare for the + failover, you must disable the **primary** node: + + - If you have SSH access to the **primary** node, stop and disable GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + + Prevent GitLab from starting up again if the server unexpectedly reboots: + + ```shell + sudo systemctl disable gitlab-runsvdir + ``` + + NOTE: **Note:** + (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). + It may be safest to uninstall the GitLab package completely with `sudo yum remove gitlab-ee`. + + NOTE: **Note:** + (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu + or any other distribution based on the Upstart init system, you can prevent GitLab + from starting if the machine reboots as `root` with + `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. + + - If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting. Since there are many ways you may prefer to accomplish + this, we will avoid a single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the **primary** DNS record to the **secondary** + node in order to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +### Promoting the **secondary** node + +Note the following when promoting a secondary: + +- A new **secondary** should not be added at this time. If you want to add a new + **secondary**, do this after you have completed the entire process of promoting + the **secondary** to the **primary**. +- If you encounter an `ActiveRecord::RecordInvalid: Validation failed: Name has already been taken` + error during this process, read + [the troubleshooting advice](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). + +To promote the secondary node: + +1. SSH in to your **secondary** node and login as root: + + ```shell + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by + removing any lines that enabled the `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + +1. Run the following command to list out all preflight checks and automatically + check if replication and verification are complete before scheduling a planned + failover to ensure the process will go smoothly: + + ```shell + gitlab-ctl promotion-preflight-checks + ``` + +1. Promote the **secondary**: + + ```shell + gitlab-ctl promote-to-primary-node + ``` + + If you have already run the [preflight checks](planned_failover.md#preflight-checks) + or don't want to run them, you can skip them: + + ```shell + gitlab-ctl promote-to-primary-node --skip-preflight-check + ``` + + You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + + ```shell + sudo gitlab-ctl promote-to-primary-node --force + ``` + +1. Verify you can connect to the newly promoted **primary** node using the URL used + previously for the **secondary** node. + + If successful, the **secondary** node has now been promoted to the **primary** node. + +### Next steps + +To regain geographic redundancy as quickly as possible, you should +[add a new **secondary** node](../setup/index.md). To +do that, you can re-add the old **primary** as a new secondary and bring it back +online. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index c2a60b958d4..876904a2093 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1196,9 +1196,16 @@ CAUTION: **Caution:** ## Data recovery -If a Gitaly node fails replication jobs for any reason, it ends up hosting outdated versions of -the affected repositories. Praefect provides tools for automatically or manually reconciling -the outdated repositories in order to bring them fully up to date again. +If a Gitaly node fails replication jobs for any reason, it ends up hosting outdated versions of the +affected repositories. Praefect provides tools for: + +- [Automatic](#automatic-reconciliation) reconciliation, for GitLab 13.4 and later. +- [Manual](#manual-reconciliation) reconciliation, for: + - GitLab 13.3 and earlier. + - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. + A migration tool [is planned](https://gitlab.com/gitlab-org/gitaly/-/issues/3033). + +These tools reconcile the outdated repositories to bring them fully up to date again. ### Automatic reconciliation diff --git a/doc/api/users.md b/doc/api/users.md index 26d9a79a165..634e0bd0842 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -2,8 +2,6 @@ ## List users -Active users = Total accounts - Blocked users - Get a list of users. This function takes pagination parameters `page` and `per_page` to restrict the list of users. @@ -49,9 +47,9 @@ For example: GET /users?username=jack_smith ``` -In addition, you can filter users based on states eg. `blocked`, `active` -This works only to filter users who are `blocked` or `active`. -It does not support `active=false` or `blocked=false`. +In addition, you can filter users based on the states `blocked` and `active`. +It does not support `active=false` or `blocked=false`. The list of active users +is the total number of users minus the blocked users. ```plaintext GET /users?active=true diff --git a/doc/development/README.md b/doc/development/README.md index 398aea55e72..abdd5c662f3 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -57,6 +57,7 @@ Complementary reads: - [Generate a changelog entry with `bin/changelog`](changelog.md) - [Requesting access to Chatops on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLab team members) - [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers) +- [Adding a new service component to GitLab](adding_service_component.md) ## UX and Frontend guides diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md new file mode 100644 index 00000000000..2801e27145d --- /dev/null +++ b/doc/development/adding_service_component.md @@ -0,0 +1,89 @@ +# Adding a new Service Component to GitLab + +The GitLab product is made up of several service components that run as independent system processes in communication with each other. These services can be run on the same instance, or spread across different instances. A list of the existing components can be found in the [GitLab architecture overview](architecture.md). + +## Integration phases + +The following outline re-uses the [maturity metric](https://about.gitlab.com/direction/maturity) naming as an example of the various phases of integrating a component. These are only loosely coupled to a components actual maturity, and are intended as a guide for implementation order (for example, a component does not need to be enabled by default to be Lovable, and being enabled by default does not on its own cause a component to be Lovable). + +- Proposed + - [Proposing a new component](#proposing-a-new-component) +- Minimal + - [Integrating a new service with GitLab](#integrating-a-new-service-with-gitlab) + - [Handling service dependencies](#handling-service-dependencies) +- Viable + - [Bundled with GitLab installations](#bundling-a-service-with-gitlab) + - [End-to-end testing in GitLab QA](testing_guide/end_to_end/beginners_guide.md) + - [Release management](#release-management) + - [Enabled on GitLab.com](feature_flags/controls.md#enabling-a-feature-for-gitlabcom) +- Complete + - [Configurable by the GitLab orchestrator](https://gitlab.com/gitlab-org/gitlab-orchestrator) +- Lovable + - Enabled by default for the majority of users + +## Proposing a new component + +The initial step for integrating a new component with GitLab starts with creating a [Feature proposal in the issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Feature%20proposal). + +Identify the [product category](https://about.gitlab.com/handbook/product/categories/) the component falls under and assign the Engineering Manager and Product Manager responsible for that category. + +The general steps for getting any GitLab feature from proposal to release can be found in the [Product development flow](https://about.gitlab.com/handbook/product-development-flow/). + +## Integrating a new service with GitLab + +Adding a new service follows the same [merge request workflow](contributing/merge_request_workflow.md) as other contributions, and must meet the same [completion criteria](contributing/merge_request_workflow.md#definition-of-done) and in addition needs to cover the following: + +- The [architecture component list](architecture.md#component-list) has been updated to include the service. +- Features provided by the component have been accepted into the [GitLab Product Direction](https://about.gitlab.com/direction/). +- Documentation is available and the support team has been made aware of the new component. + +**For services that can operate completely separate from GitLab:** + +The first iteration should be to add the ability to connect and use the service as an externally installed component. Often this involves providing settings in GitLab to connect to the service, or allow connections from it. And then shipping documentation on how to install and configure the service with GitLab. + +TIP: **Tip:** +[Elasticsearch](../integration/elasticsearch.md#installing-elasticsearch) is an example of a service that has been integrated this way. And many of the other services, including internal projects like Gitaly, started off as separately installed alternatives. + +**For services that depend on the existing GitLab codebase:** + +The first iteration should be opt-in, either through the `gitlab.yml` configuration or through [feature flags](feature_flags.md). For these types of services it is often necessary to [bundle the service and its dependencies with GitLab](#bundling-a-service-with-gitlab) as part of the initial integration. + +TIP: **Tip:** +[ActionCable](https://docs.gitlab.com/omnibus/settings/actioncable.html) is an example of a service that has been added this way. + +## Bundling a service with GitLab + +NOTE: **Note:** +Code shipped with GitLab needs to use a license approved by the Legal team. See the list of [existing approved licenses](https://about.gitlab.com/handbook/engineering/open-source/#using-open-source-libraries). + +NOTE: **Note:** +Notify the [Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/) when adding a new dependency that must be compiled. We must be able to compile the dependency on all supported platforms. + +New services to be bundled with GitLab need to be available in the following environments. + +**Dev environment** + +The first step of bundling a new service is to provide it in the development environment to engage in collaboration and feedback. + +- [Include in the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) +- [Include in the source install instructions](../install/installation.md) + +**Standard install methods** + +In order for a service to be bundled for end-users or GitLab.com, it needs to be included in the standard install methods: + +- [Included in the Omnibus package](https://gitlab.com/gitlab-org/omnibus-gitlab) +- [Included in the GitLab Helm charts](https://gitlab.com/gitlab-org/charts/gitlab) + +## Handling service dependencies + +Dependencies should be kept up to date and be tracked for security updates. For the Rails codebase, the JavaScript and Ruby dependencies are +scanned for vulnerabilities using GitLab [dependency scanning](../user/application_security/dependency_scanning/index.md). + +In addition, any system dependencies used in Omnibus packages or the Cloud Native images should be added to the [dependency update automation](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/maintenance/dependencies.io.html#adding-new-dependencies). + +## Release management + +If the service component needs to be updated or released with the monthly GitLab release, then the component should be added to the [release tools automation](https://gitlab.com/gitlab-org/release-tools). This project is maintained by the [Delivery team](https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/). A list of the projects managed this way can be found in the [release tools project directory](https://about.gitlab.com/handbook/engineering/infrastructure/team/delivery/). + +For example, during the monthly GitLab release, the desired version of Gitaly, GitLab Workhorse, GitLab Shell, etc., need to synchronized through the various release pipelines. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 14f987ca87d..d88b159b666 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -242,6 +242,7 @@ request: 1. The [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit). 1. The [CI environment preparation](https://gitlab.com/gitlab-org/gitlab/blob/master/scripts/prepare_build.sh). 1. The [Omnibus package creator](https://gitlab.com/gitlab-org/omnibus-gitlab). +1. The [Cloud Native GitLab Dockerfiles](https://gitlab.com/gitlab-org/build/CNG) ## Incremental improvements diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index 09bb6b9da6a..984c64b9e9e 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -1120,16 +1120,14 @@ they need to interact with the application. When you take screenshots: - *Capture the most relevant area of the page.* Don't include unnecessary white - space or areas of the page that don't help illustrate your point. Also, don't - include the entire page if you don't have to, but also ensure the image - contains enough information to allow the user to determine where things are. -- *Be consistent.* Find a browser window size that works for you that also - displays all areas of the product, including the left navigation (usually > - 1200px wide). For consistency, use this browser window size for your - screenshots by installing a browser extension for setting a window to a - specific size (for example, - [Window Resizer](https://chrome.google.com/webstore/detail/window-resizer/kkelicaakdanhinjdeammmilcgefonfh/related?hl=en) - for Google Chrome). + space or areas of the page that don't help illustrate the point. The left + sidebar of the GitLab user interface can change, so don't include the sidebar + if it's not necessary. +- *Keep it small.* If you don't need to show the full width of the screen, don't. + A value of 1000 pixels is a good maximum width for your screenshot image. +- *Be consistent.* Coordinate screenshots with the other screenshots already on + a documentation page. For example, if other screenshots include the left + sidebar, include the sidebar in all screenshots. ### Save the image diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md index b5a2da1b3c5..29bd0ca0a7e 100644 --- a/doc/development/feature_flags/development.md +++ b/doc/development/feature_flags/development.md @@ -110,7 +110,7 @@ Each feature flag is defined in a separate YAML file consisting of a number of f |---------------------|----------|----------------------------------------------------------------| | `name` | yes | Name of the feature flag. | | `type` | yes | Type of feature flag. | -| `default_enabled` | yes | The default state of the feature flag that is strongly validated, with `default_enabled:` passed as an argument. | +| `default_enabled` | yes | The default state of the feature flag that is strictly validated, with `default_enabled:` passed as an argument. | | `introduced_by_url` | no | The URL to the Merge Request that introduced the feature flag. | | `rollout_issue_url` | no | The URL to the Issue covering the feature flag rollout. | | `group` | no | The [group](https://about.gitlab.com/handbook/product/product-categories/#devops-stages) that owns the feature flag. | @@ -129,16 +129,16 @@ Only feature flags that have a YAML definition file can be used when running the ```shell $ bin/feature-flag my-feature-flag ->> Please specify the group introducing feature flag, like `group::apm`: +>> Specify the group introducing the feature flag, like `group::apm`: ?> group::memory ->> If you have MR open, can you paste the URL here? (or enter to skip) +>> URL of the MR introducing the feature flag (enter to skip): ?> https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38602 ->> Open this URL and fill the rest of details: +>> Open this URL and fill in the rest of the details: https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue%5Btitle%5D=%5BFeature+flag%5D+Rollout+of+%60test-flag%60&issuable_template=Feature+Flag+Roll+Out ->> Paste URL of `rollout issue` here, or enter to skip: +>> URL of the rollout issue (enter to skip): ?> https://gitlab.com/gitlab-org/gitlab/-/issues/232533 create config/feature_flags/development/test-flag.yml --- @@ -305,7 +305,7 @@ used as an actor for `Feature.enabled?`. ### Feature flags for licensed features If a feature is license-gated, there's no need to add an additional -explicit feature flag check since the flag will be checked as part of the +explicit feature flag check since the flag is checked as part of the `License.feature_available?` call. Similarly, there's no need to "clean up" a feature flag once the feature has reached general availability. @@ -316,7 +316,7 @@ a by default enabled feature flag with the same name as the provided argument. **An important side-effect of the implicit feature flags mentioned above is that unless the feature is explicitly disabled or limited to a percentage of users, -the feature flag check will default to `true`.** +the feature flag check defaults to `true`.** NOTE: **Note:** Due to limitations with `feature_available?`, the YAML definition for `licensed` feature @@ -361,9 +361,9 @@ default_enabled: [false, true] Feature groups must be defined statically in `lib/feature.rb` (in the `.register_feature_groups` method), but their implementation can obviously be -dynamic (querying the DB etc.). +dynamic (querying the DB, for example). -Once defined in `lib/feature.rb`, you will be able to activate a +Once defined in `lib/feature.rb`, you can to activate a feature for a given feature group via the [`feature_group` parameter of the features API](../../api/features.md#set-or-create-a-feature) ### Enabling a feature flag locally (in development) @@ -374,7 +374,7 @@ In the rails console (`rails c`), enter the following command to enable a featur Feature.enable(:feature_flag_name) ``` -Similarly, the following command will disable a feature flag: +Similarly, the following command disables a feature flag: ```ruby Feature.disable(:feature_flag_name) @@ -388,7 +388,7 @@ Feature.enable(:feature_flag_name, Project.find_by_full_path("root/my-project")) ## Feature flags in tests -Introducing a feature flag into the codebase creates an additional codepath that should be tested. +Introducing a feature flag into the codebase creates an additional code path that should be tested. It is strongly advised to test all code affected by a feature flag, both when **enabled** and **disabled** to ensure the feature works properly. @@ -423,10 +423,10 @@ Feature.enabled?(:ci_live_trace, project2) # => false The behavior of FlipperGate is as follows: -1. You can enable an override for a specified actor to be enabled +1. You can enable an override for a specified actor to be enabled. 1. You can disable (remove) an override for a specified actor, - falling back to default state -1. There's no way to model that you explicitly disable a specified actor + falling back to the default state. +1. There's no way to model that you explicitly disabled a specified actor. ```ruby Feature.enable(:my_feature) @@ -467,7 +467,7 @@ Feature.enable_percentage_of_time(:my_feature_3, 50) Feature.enable_percentage_of_actors(:my_feature_4, 50) ``` -Each feature flag that has a defined state will be persisted +Each feature flag that has a defined state is persisted during test execution time: ```ruby diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index c999b19ab7d..b720a6ca47e 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -235,11 +235,10 @@ For example, to add support for files referenced by a `Widget` model with a `ee/lib/gitlab/geo.rb`: ```ruby - def self.replicator_classes - classes = [::Geo::PackageFileReplicator, - ::Geo::WidgetReplicator] - - classes.select(&:enabled?) + REPLICATOR_CLASSES = [ + ::Geo::PackageFileReplicator, + ::Geo::WidgetReplicator + ] end ``` @@ -315,10 +314,6 @@ For example, to add support for files referenced by a `Widget` model with a end ``` - The method `has_create_events?` should return `true` in most of the cases. - However, if the entity you add doesn't have the create event, don't add the - method at all. - 1. Update `REGISTRY_CLASSES` in `ee/app/workers/geo/secondary/registry_consistency_worker.rb`. 1. Add `widget_registry` to `ActiveSupport::Inflector.inflections` in `config/initializers_before_autoloader/000_inflections.rb`. @@ -435,7 +430,7 @@ for verification state to the widgets table: ``` 1. Add a partial index on `verification_failure` and `verification_checksum` to ensure - re-verification can be performed efficiently. Add a migration in `ee/db/geo/migrate/`: + re-verification can be performed efficiently: ```ruby # frozen_string_literal: true @@ -461,9 +456,9 @@ for verification state to the widgets table: ##### Option 2: Create a separate `widget_states` table with verification state fields -1. Add a migration in `ee/db/geo/migrate/` to create a `widget_states` table and add a - partial index on `verification_failure` and `verification_checksum` to ensure - re-verification can be performed efficiently. Order the columns according to [our guidelines](../ordering_table_columns.md): +1. Create a `widget_states` table and add a partial index on `verification_failure` and + `verification_checksum` to ensure re-verification can be performed efficiently. Order + the columns according to [our guidelines](../ordering_table_columns.md): ```ruby # frozen_string_literal: true diff --git a/doc/development/redis.md b/doc/development/redis.md index d5d42a3869e..d205082b9c6 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -1,10 +1,19 @@ # Redis guidelines -GitLab uses [Redis](https://redis.io) for three distinct purposes: +GitLab uses [Redis](https://redis.io) for the following distinct purposes: -- Caching via `Rails.cache`. +- Caching (mostly via `Rails.cache`). - As a job processing queue with [Sidekiq](sidekiq_style_guide.md). - To manage the shared application state. +- As a Pub/Sub queue backend for ActionCable. + +In most environments (including the GDK), all of these point to the same +Redis instance. + +On GitLab.com, we use [separate Redis +instances](../administration/redis/replication_and_failover.md#running-multiple-redis-clusters). +(We do not currently use [ActionCable on +GitLab.com](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/228)). Every application process is configured to use the same Redis servers, so they can be used for inter-process communication in cases where [PostgreSQL](sql.md) @@ -21,11 +30,11 @@ to key names to avoid collisions. Typically we use colon-separated elements to provide a semblance of structure at application level. An example might be `projects:1:somekey`. -Although we split our Redis usage into three separate purposes, and those may -map to separate Redis servers in a [Highly Available](../administration/high_availability/redis.md) -configuration, the default Omnibus and GDK setups share a single Redis server. -This means that keys should **always** be globally unique across the three -purposes. +Although we split our Redis usage by purpose into distinct categories, and +those may map to separate Redis servers in a Highly Available +configuration like GitLab.com, the default Omnibus and GDK setups share +a single Redis server. This means that keys should **always** be +globally unique across all categories. It is usually better to use immutable identifiers - project ID rather than full path, for instance - in Redis key names. If full path is used, the key will @@ -56,3 +65,127 @@ Currently, we validate this in the development and test environments with the [`RedisClusterValidator`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/instrumentation/redis_cluster_validator.rb), which is enabled for the `cache` and `shared_state` [Redis instances](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances).. + +## Redis in structured logging + +Our [structured logging](logging.md#use-structured-json-logging) for web +requests and Sidekiq jobs contains fields for the duration, call count, +bytes written, and bytes read per Redis instance, along with a total for +all Redis instances. For a particular request, this might look like: + +| Field | Value | +| --- | --- | +| `json.queue_duration_s` | 0.01 | +| `json.redis_cache_calls` | 1 | +| `json.redis_cache_duration_s` | 0 | +| `json.redis_cache_read_bytes` | 109 | +| `json.redis_cache_write_bytes` | 49 | +| `json.redis_calls` | 2 | +| `json.redis_duration_s` | 0.001 | +| `json.redis_read_bytes` | 111 | +| `json.redis_shared_state_calls` | 1 | +| `json.redis_shared_state_duration_s` | 0 | +| `json.redis_shared_state_read_bytes` | 2 | +| `json.redis_shared_state_write_bytes` | 206 | +| `json.redis_write_bytes` | 255 | + +As all of these fields are indexed, it is then straightforward to +investigate Redis usage in production. For instance, to find the +requests that read the most data from the cache, we can just sort by +`redis_cache_read_bytes` in descending order. + +### The slow log + +On GitLab.com, entries from the [Redis +slow log](https://redis.io/commands/slowlog) are available in the +`pubsub-redis-inf-gprd*` index with the [`redis.slowlog` +tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). +This shows commands that have taken a long time and may be a performance +concern. + +The +[fluent-plugin-redis-slowlog](https://gitlab.com/gitlab-org/fluent-plugin-redis-slowlog) +project is responsible for taking the slowlog entries from Redis and +passing to fluentd (and ultimately Elasticsearch). + +## Analyzing the entire keyspace + +The [Redis Keyspace +Analyzer](https://gitlab.com/gitlab-com/gl-infra/redis-keyspace-analyzer) +project contains tools for dumping the full key list and memory usage of a Redis +instance, and then analyzing those lists while elimating potentially sensitive +data from the results. It can be used to find the most frequent key patterns, or +those that use the most memory. + +Currently this is not run automatically for the GitLab.com Redis instances, but +is run manually on an as-needed basis. + +## Utility classes + +We have some extra classes to help with specific use cases. These are +mostly for fine-grained control of Redis usage, so they wouldn't be used +in combination with the `Rails.cache` wrapper: we'd either use +`Rails.cache` or these classes and literal Redis commands. + +`Rails.cache` or these classes and literal Redis commands. We prefer +using `Rails.cache` so we can reap the benefits of future optimizations +done to Rails. It is worth noting that Ruby objects are +[marshalled](https://github.com/rails/rails/blob/v6.0.3.1/activesupport/lib/active_support/cache/redis_cache_store.rb#L447) +when written to Redis, so we need to pay attention to not to store huge +objects, or untrusted user input. + +Typically we would only use these classes when at least one of the +following is true: + +1. We want to manipulate data on a non-cache Redis instance. +1. `Rails.cache` does not support the operations we want to perform. + +### `Gitlab::Redis::{Cache,SharedState,Queues}` + +These classes wrap the Redis instances (using +[`Gitlab::Redis::Wrapper`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/wrapper.rb)) +to make it convenient to work with them directly. The typical use is to +call `.with` on the class, which takes a block that yields the Redis +connection. For example: + +```ruby +# Get the value of `key` from the shared state (persistent) Redis +Gitlab::Redis::SharedState.with { |redis| redis.get(key) } + +# Check if `value` is a member of the set `key` +Gitlab::Redis::Cache.with { |redis| redis.sismember(key, value) } +``` + +### `Gitlab::Redis::Boolean` + +In Redis, every value is a string. +[`Gitlab::Redis::Boolean`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/boolean.rb) +makes sure that booleans are encoded and decoded consistently. + +### `Gitlab::Redis::HLL` + +The Redis [`PFCOUNT`](https://redis.io/commands/pfcount), +[`PFADD`](https://redis.io/commands/pfadd), and +[`PFMERGE`](https://redis.io/commands/pfmergge) commands operate on +HyperLogLogs, a data structure that allows estimating the number of unique +elements with low memory usage. (In addition to the `PFCOUNT` documentation, +Thoughtbot's article on [HyperLogLogs in +Redis](https://thoughtbot.com/blog/hyperloglogs-in-redis) provides a good +background here.) + +[`Gitlab::Redis::HLL`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/redis/hll.rb) +provides a convenient interface for adding and counting values in HyperLogLogs. + +### `Gitlab::SetCache` + +For cases where we need to efficiently check the whether an item is in a group +of items, we can use a Redis set. +[`Gitlab::SetCache`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/set_cache.rb) +provides an `#include?` method that will use the +[`SISMEMBER`](https://redis.io/commands/sismember) command, as well as `#read` +to fetch all entries in the set. + +This is used by the +[`RepositorySetCache`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/repository_set_cache.rb) +to provide a convenient way to use sets to cache repository data like branch +names. diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md new file mode 100644 index 00000000000..f26483e3b5e --- /dev/null +++ b/doc/integration/gitpod.md @@ -0,0 +1,74 @@ +--- +type: reference, how-to +stage: Create +group: Editor +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers" +--- + +# Gitpod Integration + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228893) in GitLab 13.4. +> - It's [deployed behind a feature flag](#enable-or-disable-the-gitpod-integration), disabled by default. +> - It's enabled on GitLab.com. +> - It's recommended for production use. +> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#configure-your-gitlab-instance-with-gitpod). **(CORE ONLY)** + +CAUTION: **Warning:** +This feature might not be available to you. Check the **version history** note above for details. + +With [Gitpod](https://gitpod.io/) you can describe your dev environment as code to get fully set +up, compiled, and tested dev environments for any GitLab project. The dev environments are not only +automated but also prebuilt which means that Gitpod continuously builds your Git branches like a CI +server. By that you don’t have to wait for dependencies to be downloaded and builds to finish, but +you can start coding immediately. + +In short: With Gitpod you can start coding instantly on any project, branch, and merge request from +any device, at any time. + + + +You can launch Gitpod directly from GitLab by clicking the **Gitpod** button from the **Web IDE** +dropdown on the project page: + + + +To learn more about Gitpod, see their [features](https://www.gitpod.io/features/) and +[documentation](https://www.gitpod.io/docs/). + +To use the GitLab-Gitpod integration, you need to enable it from your user preferences: + +1. From the GitLab UI, click your avatar in the top-right corner, then click **Settings**. +1. On the left-hand nav, click **Preferences**. +1. Under **Integrations**, find the **Gitpod** section. +1. Check **Enable Gitpod**. + +Users of GitLab.com can enable it and start using straightaway. Users of GitLab self-managed instances +can follow the same steps once the integration has been enabled and configured by a GitLab administrator. + +## Configure your GitLab instance with Gitpod **(CORE ONLY)** + +If you are new to Gitpod, head over to the [Gitpod documentation](https://www.gitpod.io/docs/self-hosted/latest/self-hosted/) +and get your instance up and running. + +1. In GitLab, go to **Admin Area > Settings > Integrations**. +1. Expand the **Gitpod** configuration section. +1. Check **Enable Gitpod**. +1. Add your Gitpod instance URL (for example, `https://gitpod.example.com`). + +## Enable or disable the Gitpod integration **(CORE ONLY)** + +The Gitpod integration is under development and not ready for production use. It is deployed behind a +feature flag that is **disabled by default**. +[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md) +can enable it. + +To enable it: + +```ruby +Feature.enable(:gitpod) +``` + +To disable it: + +```ruby +Feature.disable(:gitpod) diff --git a/doc/integration/img/gitpod_button_project_page_v13_4.png b/doc/integration/img/gitpod_button_project_page_v13_4.png Binary files differnew file mode 100644 index 00000000000..55a70d89169 --- /dev/null +++ b/doc/integration/img/gitpod_button_project_page_v13_4.png diff --git a/doc/integration/img/gitpod_web_interface_v13_4.png b/doc/integration/img/gitpod_web_interface_v13_4.png Binary files differnew file mode 100644 index 00000000000..5cd9a6aad0f --- /dev/null +++ b/doc/integration/img/gitpod_web_interface_v13_4.png diff --git a/doc/user/application_security/offline_deployments/index.md b/doc/user/application_security/offline_deployments/index.md index ea7c5e23d2a..3a7c0148388 100644 --- a/doc/user/application_security/offline_deployments/index.md +++ b/doc/user/application_security/offline_deployments/index.md @@ -64,10 +64,10 @@ Once a vulnerability is found, you can interact with it. Read more on how to Please note that in some cases the reported vulnerabilities provide metadata that can contain external links exposed in the UI. These links might not be accessible within an offline environment. -### Suggested Solutions for vulnerabilities +### Automatic remediation for vulnerabilities -The [suggested solutions](../index.md#solutions-for-vulnerabilities-auto-remediation) feature -(auto-remediation) is available for Dependency Scanning and Container Scanning, but may not work +The [automatic remediation for vulnerabilities](../index.md#solutions-for-vulnerabilities-auto-remediation) feature +(auto-remediation) is available for offline Dependency Scanning and Container Scanning, but may not work depending on your instance's configuration. We can only suggest solutions, which are generally more current versions that have been patched, when we are able to access up-to-date registry services hosting the latest versions of that dependency or image. diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index 03bb1bf677e..f84fc1ae898 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -182,6 +182,12 @@ Manage the availability of integrated code intelligence features powered by Sourcegraph. View [the Sourcegraph feature documentation](../../integration/sourcegraph.md#enable-sourcegraph-in-user-preferences) for more information. +### Gitpod + +Enable and disable the [GitLab-Gitpod integration](../../integration/gitpod.md). This is only +visible after the integration is configured by a GitLab administrator. View +[the Gitpod feature documentation](../../integration/gitpod.md) for more information. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index 8ecc0e1f9c4..d0499730bfe 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -37,7 +37,12 @@ Import your projects from Bitbucket Server to GitLab with minimal effort. empty changes. 1. Attachments in Markdown are currently not imported. 1. Task lists are not imported. -1. Emoji reactions are not imported +1. Emoji reactions are not imported. +1. [LFS objects](../../../topics/git/lfs/index.md) are not imported. + + NOTE: **Note:** + To import a repository including LFS objects from a Bitbucket server repository, use the [Repo by URL](../import/repo_by_url.md) importer. + 1. Project filtering does not support fuzzy search (only `starts with` or `full match strings` are currently supported) diff --git a/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md b/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md index 403067ba0cd..1d0299637bd 100644 --- a/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md +++ b/doc/user/project/merge_requests/reviewing_and_managing_merge_requests.md @@ -289,6 +289,17 @@ the command line. NOTE: **Note:** This section might move in its own document in the future. +### Copy the branch name for local checkout + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23767) in GitLab 13.4. + +The merge request sidebar contains the branch reference for the source branch +used to contribute changes for this merge request. + +To copy the branch reference into your clipboard, click the **Copy branch name** button +(**{copy-to-clipboard}**) in the right sidebar. You can then use it to checkout the branch locally +via command line by running `git checkout <branch-name>`. + ### Checkout merge requests locally through the `head` ref A merge request contains all the history from a repository, plus the additional diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index 597d8dfe3f1..ef32b0dbb18 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -15,10 +15,7 @@ type: reference, howto > - It's recommended for production use. > - For GitLab self-managed instances, GitLab administrators can [disable it](#enable-or-disable-project-access-tokens). -Project access tokens are scoped to a project and can be used to authenticate with the [GitLab API](../../../api/README.md#personalproject-access-tokens). - -<!-- Commented out until https://gitlab.com/gitlab-org/gitlab/-/issues/219551 is fixed --> -<!-- You can also use project access tokens with Git to authenticate over HTTP or SSH. --> +Project access tokens are scoped to a project and can be used to authenticate with the [GitLab API](../../../api/README.md#personalproject-access-tokens). You can also use project access tokens with Git to authenticate over HTTP or SSH. Project access tokens expire on the date you define, at midnight UTC. diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index d31e42e270e..821b42af049 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -53,21 +53,42 @@ If you are missing Syntax Highlighting support for any language, we prepared a s NOTE: **Note:** Single file editing is based on the [Ace Editor](https://ace.c9.io). -### Schema based validation +### Themes + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2389) in GitLab in 13.0. +> - Full Solarized Dark Theme [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219228) in GitLab 13.1. + +All the themes GitLab supports for syntax highlighting are added to the Web IDE's code editor. +You can pick a theme from your [profile preferences](../../profile/preferences.md). + +The themes are available only in the Web IDE file editor, except for the [dark theme](https://gitlab.com/gitlab-org/gitlab/-/issues/209808) and +the [solarized dark theme](https://gitlab.com/gitlab-org/gitlab/-/issues/219228), +which apply to the entire Web IDE screen. + +| Solarized Light Theme | Solarized Dark Theme | Dark Theme | +|---------------------------------------------------------------|-------------------------------------------------------------|-----------------------------------------| +|  |  |  | + +## Schema based validation -> - Support for `.gitlab-ci.yml` validation [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218472) in GitLab 13.2. +> - Support for validation based on predefined schemas [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218472) in GitLab 13.2. > - It was deployed behind a feature flag, disabled by default. > - It's enabled on GitLab.com. > - It cannot be enabled or disabled per-project. -> - For GitLab self-managed instances, GitLab administrators can opt to [enable it](#enable-or-disable-schema-based-validation). +> - For GitLab self-managed instances, GitLab administrators can opt to [enable it](#enable-or-disable-validation-based-on-predefined-schemas). +> - Support for validation based on custom schemas [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/226982) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. The Web IDE provides validation support for certain JSON and YAML files using schemas -based on the [JSON Schema Store](https://www.schemastore.org/json/). This feature is -only supported for the `.gitlab-ci.yml` file. +based on the [JSON Schema Store](https://www.schemastore.org/json/). -#### Enable or disable Schema based validation **(CORE ONLY)** +### Predefined schemas -Schema based validation is under development and not ready for production use. It is +The Web IDE has validation for certain files built in. This feature is only supported for +the `*.gitlab-ci.yml` files. + +#### Enable or disable validation based on predefined schemas **(CORE ONLY)** + +Validation based on predefined schemas is under development and not ready for production use. It is deployed behind a feature flag that is **disabled by default** for self-managed instances, [GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) can enable it for your instance. @@ -84,21 +105,35 @@ To disable it: Feature.disable(:schema_linting) ``` -### Themes +### Custom schemas **(PREMIUM)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2389) in GitLab in 13.0. -> - Full Solarized Dark Theme [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/219228) in GitLab 13.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/226982) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. -All the themes GitLab supports for syntax highlighting are added to the Web IDE's code editor. -You can pick a theme from your [profile preferences](../../profile/preferences.md). +The Web IDE also allows you to define custom schemas for certain JSON/YAML files in your project. +You can do so by defining a `schemas` entry in the `.gitlab/.gitlab-webide.yml` file inside the +repository's root. Here is an example configuration: -The themes are available only in the Web IDE file editor, except for the [dark theme](https://gitlab.com/gitlab-org/gitlab/-/issues/209808) and -the [solarized dark theme](https://gitlab.com/gitlab-org/gitlab/-/issues/219228), -which apply to the entire Web IDE screen. +```yaml +schemas: + - uri: https://json.schemastore.org/package + match: + - package.json + - uri: https://somewebsite.com/first/raw/url + match: + - data/release_posts/unreleased/*.{yml,yaml} + - uri: https://somewebsite.com/second/raw/url + match: + - "*.meta.json" +``` -| Solarized Light Theme | Solarized Dark Theme | Dark Theme | -|---------------------------------------------------------------|-------------------------------------------------------------|-----------------------------------------| -|  |  |  | +Each schema entry supports two properties: + +- `uri`: please provide an absolute URL for the schema definition file here. The schema from this URL +is loaded when a matching file is open. +- `match`: a list of matching paths or glob expressions. If a schema matches a particular path pattern, +it will be applied to that file. Please enclose the pattern in quotes if it begins with an asterisk (`*`), +it's be applied to that file. If a pattern begins with an asterisk (`*`), enclose it in quotation +marks. Otherwise, the configuration file is not valid YAML. ## Configure the Web IDE |
