diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-10-21 07:08:36 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-10-21 07:08:36 +0000 |
| commit | 48aff82709769b098321c738f3444b9bdaa694c6 (patch) | |
| tree | e00c7c43e2d9b603a5a6af576b1685e400410dee /doc/development | |
| parent | 879f5329ee916a948223f8f43d77fba4da6cd028 (diff) | |
| download | gitlab-ce-48aff82709769b098321c738f3444b9bdaa694c6.tar.gz | |
Add latest changes from gitlab-org/gitlab@13-5-stable-eev13.5.0-rc42
Diffstat (limited to 'doc/development')
109 files changed, 3786 insertions, 2182 deletions
diff --git a/doc/development/README.md b/doc/development/README.md index abdd5c662f3..7b8a5cd5f75 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -43,6 +43,7 @@ For information on how to install, configure, update, and upgrade your own GitLa **Must-reads:** +- [Guide on adapting existing and introducing new components](architecture.md#adapting-existing-and-introducing-new-components) - [Code review guidelines](code_review.md) for reviewing code and having code reviewed - [Database review guidelines](database_review.md) for reviewing database-related changes and complex SQL queries, and having them reviewed - [Secure coding guidelines](secure_coding_guidelines.md) @@ -115,6 +116,7 @@ Complementary reads: - [Code Intelligence](code_intelligence/index.md) - [Approval Rules](approval_rules.md) - [Feature categorization](feature_categorization/index.md) +- [Wikis development guide](wikis.md) ## Performance guides @@ -163,11 +165,11 @@ See [database guidelines](database/index.md). - [Externalization](i18n/externalization.md) - [Translation](i18n/translation.md) -## Telemetry guides +## Product Analytics guides -- [Telemetry guide](telemetry/index.md) -- [Usage Ping guide](telemetry/usage_ping.md) -- [Snowplow guide](telemetry/snowplow.md) +- [Product Analytics guide](product_analytics/index.md) +- [Usage Ping guide](product_analytics/usage_ping.md) +- [Snowplow guide](product_analytics/snowplow.md) ## Experiment guide diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 03557491e68..d2526d6c4f2 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Adding Database Indexes Indexes can be used to speed up database queries, but when should you add a new diff --git a/doc/development/adding_service_component.md b/doc/development/adding_service_component.md index 2801e27145d..bb5af286c1d 100644 --- a/doc/development/adding_service_component.md +++ b/doc/development/adding_service_component.md @@ -46,7 +46,7 @@ TIP: **Tip:** **For services that depend on the existing GitLab codebase:** -The first iteration should be opt-in, either through the `gitlab.yml` configuration or through [feature flags](feature_flags.md). For these types of services it is often necessary to [bundle the service and its dependencies with GitLab](#bundling-a-service-with-gitlab) as part of the initial integration. +The first iteration should be opt-in, either through the `gitlab.yml` configuration or through [feature flags](feature_flags/index.md). For these types of services it is often necessary to [bundle the service and its dependencies with GitLab](#bundling-a-service-with-gitlab) as part of the initial integration. TIP: **Tip:** [ActionCable](https://docs.gitlab.com/omnibus/settings/actioncable.html) is an example of a service that has been added this way. diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 18fc0fb7d33..3d4c033e676 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -49,6 +49,20 @@ See also: - [Exposing Global IDs](#exposing-global-ids). - [Mutation arguments](#object-identifier-arguments). +We have a custom scalar type (`Types::GlobalIDType`) which should be used as the +type of input and output arguments when the value is a `GlobalID`. The benefits +of using this type instead of `ID` are: + +- it validates that the value is a `GlobalID` +- it parses it into a `GlobalID` before passing it to user code +- it can be parameterized on the type of the object (e.g. + `GlobalIDType[Project]`) which offers even better validation and security. + +Consider using this type for all new arguments and result types. Remember that +it is perfectly possible to parameterize this type with a concern or a +supertype, if you want to accept a wider range of objects (e.g. +`GlobalIDType[Issuable]` vs `GlobalIDType[Issue]`). + ## Types We use a code-first schema, and we declare what type everything is in Ruby. @@ -371,8 +385,8 @@ end GitLab's GraphQL API is versionless, which means we maintain backwards compatibility with older versions of the API with every change. Rather than removing a field or [enum value](#enums), we need to _deprecate_ it instead. -In future, GitLab -[may remove deprecated parts of the schema](https://gitlab.com/gitlab-org/gitlab/-/issues/32292). +The deprecated parts of the schema can then be removed in a future release +in accordance with [GitLab's deprecation process](../api/graphql/index.md#deprecation-process). Fields and enum values are deprecated using the `deprecated` property. The value of the property is a `Hash` of: @@ -472,6 +486,28 @@ end Enum values can be deprecated using the [`deprecated` keyword](#deprecating-fields-and-enum-values). +### Defining GraphQL enums dynamically from Rails enums + +If your GraphQL enum is backed by a [Rails enum](creating_enums.md), then consider +using the Rails enum to dynamically define the GraphQL enum values. Doing so +binds the GraphQL enum values to the Rails enum definition, so if values are +ever added to the Rails enum then the GraphQL enum automatically reflects the change. + +Example: + +```ruby +module Types + class IssuableSeverityEnum < BaseEnum + graphql_name 'IssuableSeverity' + description 'Incident severity' + + ::IssuableSeverity.severities.keys.each do |severity| + value severity.upcase, value: severity, description: "#{severity.titleize} severity" + end + end +end +``` + ## JSON When data to be returned by GraphQL is stored as @@ -780,6 +816,25 @@ to advertise the need for lookahead: For an example of real world use, please see [`ResolvesMergeRequests`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/resolvers/concerns/resolves_merge_requests.rb). +### Negated arguments + +Negated filters can filter some resources (for example, find all issues that +have the `bug` label, but don't have the `bug2` label assigned). The `not` +argument is the preferred syntax to pass negated arguments: + +```graphql +issues(labelName: "bug", not: {labelName: "bug2"}) { + nodes { + id + title + } +} +``` + +To avoid duplicated argument definitions, you can place these arguments in a reusable module (or +class, if the arguments are nested). Alternatively, you can consider to add a +[helper resolver method](https://gitlab.com/gitlab-org/gitlab/-/issues/258969). + ## Pass a parent object into a child Presenter Sometimes you need to access the resolved query parent in a child context to compute fields. Usually the parent is only @@ -827,10 +882,39 @@ mutation. ### Building Mutations -Mutations live in `app/graphql/mutations` ideally grouped per +Mutations are stored in `app/graphql/mutations`, ideally grouped per resources they are mutating, similar to our services. They should inherit `Mutations::BaseMutation`. The fields defined on the mutation -will be returned as the result of the mutation. +are returned as the result of the mutation. + +#### Update mutation granularity + +GitLab's service-oriented architecture means that most mutations call a Create, Delete, or Update +service, for example `UpdateMergeRequestService`. +For Update mutations, a you might want to only update one aspect of an object, and thus only need a +_fine-grained_ mutation, for example `MergeRequest::SetWip`. + +It's acceptable to have both fine-grained mutations and coarse-grained mutations, but be aware +that too many fine-grained mutations can lead to organizational challenges in maintainability, code +comprehensibility, and testing. +Each mutation requires a new class, which can lead to technical debt. +It also means the schema becomes very big, and we want users to easily navigate our schema. +As each new mutation also needs tests (including slower request integration tests), adding mutations +slows down the test suite. + +To minimize changes: + +- Use existing mutations, such as `MergeRequest::Update`, when available. +- Expose existing services as a coarse-grained mutation. + +When a fine-grained mutation might be more appropriate: + +- Modifying a property that requires specific permissions or other specialized logic. +- Exposing a state-machine-like transition (locking issues, merging MRs, closing epics, etc). +- Accepting nested properties (where we accept properties for a child object). +- The semantics of the mutation can be expressed clearly and concisely. + +See [issue #233063](https://gitlab.com/gitlab-org/gitlab/-/issues/233063) for further context. ### Naming conventions @@ -1361,5 +1445,4 @@ For information on generating GraphQL documentation and schema files, see [updating the schema documentation](rake_tasks.md#update-graphql-documentation-and-schema-definitions). To help our readers, you should also add a new page to our [GraphQL API](../api/graphql/index.md) documentation. -For guidance, see the [GraphQL API](documentation/styleguide.md#graphql-api) section -of our documentation style guide. +For guidance, see the [GraphQL API](documentation/graphql_styleguide.md) page. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 400752c69e9..47c06377d88 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -17,7 +17,7 @@ Each new or updated API endpoint must come with documentation, unless it is inte The docs should be in the same merge request, or, if strictly necessary, in a follow-up with the same milestone as the original merge request. -See the [Documentation Style Guide RESTful API section](documentation/styleguide.md#restful-api) for details on documenting API resources in Markdown as well as in OpenAPI definition files. +See the [Documentation Style Guide RESTful API page](documentation/restful_api_styleguide.md) for details on documenting API resources in Markdown as well as in OpenAPI definition files. ## Methods and parameters description diff --git a/doc/development/architecture.md b/doc/development/architecture.md index f6b1c8cd914..513af491576 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -1,32 +1,91 @@ -# GitLab Architecture Overview +# GitLab architecture overview ## Software delivery -There are two software distributions of GitLab: the open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/) (CE), and the open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/) (EE). GitLab is available under [different subscriptions](https://about.gitlab.com/pricing/). +There are two software distributions of GitLab: -New versions of GitLab are released in stable branches and the master branch is for bleeding edge development. +- The open source [Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/) (CE). +- The open core [Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/) (EE). -For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/release/docs/-/tree/master#gitlab-release-process). +GitLab is available under [different subscriptions](https://about.gitlab.com/pricing/). -Both EE and CE require some add-on components called GitLab Shell and Gitaly. These components are available from the [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/master) and [Gitaly](https://gitlab.com/gitlab-org/gitaly/-/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with the exception of informal security updates deemed critical. +New versions of GitLab are released from stable branches, and the `master` branch is used for +bleeding-edge development. + +For more information, visit the [GitLab Release Process](https://about.gitlab.com/handbook/engineering/releases/). + +Both distributions require additional components. These components are described in the +[Component details](#components) section, and all have their own repositories. +New versions of each dependent component are usually tags, but staying on the `master` branch of the +GitLab codebase gives you the latest stable version of those components. New versions are +generally released around the same time as GitLab releases, with the exception of informal security +updates deemed critical. ## Components -A typical install of GitLab will be on GNU/Linux. It uses NGINX or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and pre-compiled assets. GitLab serves web pages and the [GitLab API](../api/README.md) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses Redis as a non-persistent database backend for job information, meta data, and incoming jobs. +A typical install of GitLab is on GNU/Linux, but growing number of deployments also use the +Kubernetes platform. The largest known GitLab instance is on GitLab.com, which is deployed using our +[official GitLab Helm chart](https://docs.gitlab.com/charts/) and the [official Linux package](https://about.gitlab.com/install/). -We also support deploying GitLab on Kubernetes using our [GitLab Helm chart](https://docs.gitlab.com/charts/). +A typical installation uses NGINX or Apache as a web server to proxy through +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) and into the [Puma](https://puma.io) +application server. GitLab serves web pages and the [GitLab API](../api/README.md) using the Puma +application server. It uses Sidekiq as a job queue which, in turn, uses Redis as a non-persistent +database backend for job information, metadata, and incoming jobs. -The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare Git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. +By default, communication between Puma and Workhorse is via a Unix domain socket, but forwarding +requests via TCP is also supported. Workhorse accesses the `gitlab/public` directory, bypassing the +Puma application server to serve static pages, uploads (for example, avatar images or attachments), +and pre-compiled assets. -When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving Git objects. +The GitLab application uses PostgreSQL for persistent database information (for example, users, +permissions, issues, or other metadata). GitLab stores the bare Git repositories in the location +defined in [the configuration file, `repositories:` section](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example). +It also keeps default branch and hook information with the bare repository. -The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. GitLab Shell accesses the bare repositories through Gitaly to serve Git objects and communicates with Redis to submit jobs to Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access. +When serving repositories over HTTP/HTTPS GitLab uses the GitLab API to resolve authorization and +access and to serve Git objects. -Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from Git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). +The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within the +location defined in [the configuration file, `GitLab Shell` section](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example). +The file in that location should never be manually edited. GitLab Shell accesses the bare +repositories through Gitaly to serve Git objects, and communicates with Redis to submit jobs to +Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access. + +Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to the +GitLab web app to get attributes from Git (for example, title, branches, tags, or other metadata), +and to get blobs (for example, diffs, commits, or files). You may also be interested in the [production architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/). -### Simplified Component Overview +## Adapting existing and introducing new components + +There are fundamental differences in how the application behaves when it is installed on a +traditional Linux machine compared to a containerized platform, such as Kubernetes. + +Compared to [our official installation methods](https://about.gitlab.com/install/), some of the +notable differences are: + +- Official Linux packages can access files on the same file system with different services. + [Shared files](shared_files.md) are not an option for the application running on the Kubernetes + platform. +- Official Linux packages by default have services that have access to the shared configuration and + network. This is not the case for services running in Kubernetes, where services might be running + in complete isolation, or only accessible through specific ports. + +In other words, the shared state between services needs to be carefully considered when +architecting new features and adding new components. Services that need to have access to the same +files, need to be able to exchange information through the appropriate APIs. Whenever possible, +this should not be done with files. + +Since components written with the API-first philosophy in mind are compatible with both methods, all +new features and services must be written to consider Kubernetes compatibility **first**. + +The simplest way to ensure this, is to add support for your feature or service to +[the official GitLab Helm chart](https://docs.gitlab.com/charts/) or reach out to +[the Distribution team](https://about.gitlab.com/handbook/engineering/development/enablement/distribution/#how-to-work-with-distribution). + +### Simplified component overview This is a simplified architecture diagram that can be used to understand GitLab's architecture. @@ -51,23 +110,25 @@ SSH -- TCP 22 --> GitLabShell[GitLab Shell] SMTP[SMTP Gateway] Geo[GitLab Geo Node] -- TCP 22, 80, 443 --> NGINX -GitLabShell --TCP 8080 -->Unicorn["Unicorn (GitLab Rails)"] +GitLabShell --TCP 8080 -->Puma["Puma (GitLab Rails)"] GitLabShell --> Praefect -Unicorn --> PgBouncer[PgBouncer] -Unicorn --> Redis -Unicorn --> Praefect +Puma --> PgBouncer[PgBouncer] +Puma --> Redis +Puma --> Praefect Sidekiq --> Redis Sidekiq --> PgBouncer Sidekiq --> Praefect -GitLabWorkhorse[GitLab Workhorse] --> Unicorn +GitLabWorkhorse[GitLab Workhorse] --> Puma GitLabWorkhorse --> Redis GitLabWorkhorse --> Praefect Praefect --> Gitaly NGINX --> GitLabWorkhorse NGINX -- TCP 8090 --> GitLabPages[GitLab Pages] NGINX --> Grafana[Grafana] +NGINX -- TCP 8150 --> GitLabKas[GitLab Kubernetes Agent Server] +GitLabKas --> Praefect Grafana -- TCP 9090 --> Prometheus[Prometheus] -Prometheus -- TCP 80, 443 --> Unicorn +Prometheus -- TCP 80, 443 --> Puma RedisExporter[Redis Exporter] --> Redis Prometheus -- TCP 9121 --> RedisExporter PostgreSQLExporter[PostgreSQL Exporter] --> PostgreSQL @@ -83,27 +144,27 @@ PgBouncer --> Consul PostgreSQL --> Consul PgBouncer --> PostgreSQL NGINX --> Registry -Unicorn --> Registry +Puma --> Registry NGINX --> Mattermost -Mattermost --- Unicorn +Mattermost --- Puma Prometheus --> Alertmanager Migrations --> PostgreSQL Runner -- TCP 443 --> NGINX -Unicorn -- TCP 9200 --> Elasticsearch +Puma -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 9200 --> Elasticsearch Sidekiq -- TCP 80, 443 --> Sentry -Unicorn -- TCP 80, 443 --> Sentry +Puma -- TCP 80, 443 --> Sentry Sidekiq -- UDP 6831 --> Jaeger -Unicorn -- UDP 6831 --> Jaeger +Puma -- UDP 6831 --> Jaeger Gitaly -- UDP 6831 --> Jaeger GitLabShell -- UDP 6831 --> Jaeger GitLabWorkhorse -- UDP 6831 --> Jaeger Alertmanager -- TCP 25 --> SMTP Sidekiq -- TCP 25 --> SMTP -Unicorn -- TCP 25 --> SMTP -Unicorn -- TCP 369 --> LDAP +Puma -- TCP 25 --> SMTP +Puma -- TCP 369 --> LDAP Sidekiq -- TCP 369 --> LDAP -Unicorn -- TCP 443 --> ObjectStorage["Object Storage"] +Puma -- TCP 443 --> ObjectStorage["Object Storage"] Sidekiq -- TCP 443 --> ObjectStorage GitLabWorkhorse -- TCP 443 --> ObjectStorage Registry -- TCP 443 --> ObjectStorage @@ -121,7 +182,7 @@ click Gitaly "./architecture.html#gitaly" click Jaeger "./architecture.html#jaeger" click GitLabWorkhorse "./architecture.html#gitlab-workhorse" click LDAP "./architecture.html#ldap-authentication" -click Unicorn "./architecture.html#unicorn" +click Puma "./architecture.html#puma" click GitLabShell "./architecture.html#gitlab-shell" click SSH "./architecture.html#ssh-request-22" click Sidekiq "./architecture.html#sidekiq" @@ -175,6 +236,7 @@ Table description links: | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ✅ | ❌ | ⚙ | EE Only | | [GitLab Managed Apps](#gitlab-managed-apps) | Deploy Helm, Ingress, Cert-Manager, Prometheus, GitLab Runner, JupyterHub, or Knative to a cluster | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE | | [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE | +| [GitLab Kubernetes Agent](#gitlab-kubernetes-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | ✅ | ⚙ | ⤓ | ✅ | ❌ | ❌ | CE & EE | | [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | ❌ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | CE & EE | @@ -201,7 +263,7 @@ Table description links: | [Runner](#gitlab-runner) | Executes GitLab CI/CD jobs | ⤓ | ✅ | ⚙ | ✅ | ⚙ | ⚙ | CE & EE | | [Sentry integration](#sentry) | Error tracking for deployed apps | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | ⤓ | CE & EE | | [Sidekiq](#sidekiq) | Background jobs processor | ✅ | ✅ | ✅ | ✅ | ✅ | ✅ | CE & EE | -| [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | +| [Puma (GitLab Rails)](#puma) | Handles requests for the web interface and API | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | ### Component details @@ -271,7 +333,7 @@ Consul is a tool for service discovery and configuration. Consul is distributed, - [Source](../integration/elasticsearch.md) - [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/elasticsearch.md) - Layer: Core Service (Data) -- GitLab.com: [Get Advanced Global Search working on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/153) epic. +- GitLab.com: [Get Advanced Search working on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/153) epic. Elasticsearch is a distributed RESTful search engine built for the cloud. @@ -321,6 +383,19 @@ repository updates to secondary nodes. GitLab Exporter is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's README](https://gitlab.com/gitlab-org/gitlab-exporter). +#### GitLab Kubernetes Agent + +- [Project page](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) +- Configuration: + - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) + - [Charts](https://docs.gitlab.com/charts/charts/gitlab/kas/index.html) + +[GitLab Kubernetes Agent](../user/clusters/agent/index.md) is an active in-cluster +component for solving GitLab and Kubernetes integration tasks in a secure and +cloud-native way. + +You can use it to sync deployments onto your Kubernetes cluster. + #### GitLab Pages - Configuration: @@ -368,13 +443,13 @@ GitLab CI/CD is the open-source continuous integration service included with Git - [Project page](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md) - Configuration: - [Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template) - - [Charts](https://docs.gitlab.com/charts/charts/gitlab/unicorn/) + - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) - [Source](../install/installation.md#install-gitlab-workhorse) - Layer: Core Service (Processor) - Process: `gitlab-workhorse` - GitLab.com: [Service Architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/#service-architecture) -[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/blog/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Puma. You can read more about the [historical reasons for developing](https://about.gitlab.com/blog/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. #### Grafana @@ -411,7 +486,8 @@ For monitoring deployed apps, see [Jaeger tracing documentation](../operations/t - Layer: Core Service - Process: `logrotate` -GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. +GitLab is comprised of a large number of services that all log. We started bundling our own Logrotate +as of GitLab 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. #### Mattermost @@ -560,7 +636,7 @@ Redis is packaged to provide a place to store: - [Source](../administration/packages/container_registry.md#enable-the-container-registry) - [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/registry.md) - Layer: Core Service (Processor) -- GitLab.com: [GitLab Container Registry](../user/packages/container_registry/index.md#build-and-push-images-using-gitlab-cicd) +- GitLab.com: [GitLab Container Registry](../user/packages/container_registry/index.md#build-and-push-by-using-gitlab-cicd) The registry is what users use to store their own Docker images. The bundled registry uses NGINX as a load balancer and GitLab as an authentication manager. @@ -603,8 +679,30 @@ For monitoring deployed apps, see the [Sentry integration docs](../operations/er Sidekiq is a Ruby background job processor that pulls jobs from the Redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background. +#### Puma + +NOTE: **Note:** +Starting with GitLab 13.0, Puma is the default web server and Unicorn has been +disabled by default. + +- [Project page](https://gitlab.com/gitlab-org/gitlab/blob/master/README.md) +- Configuration: + - [Omnibus](https://docs.gitlab.com/omnibus/settings/puma.html) + - [Charts](https://docs.gitlab.com/charts/charts/gitlab/webservice/) + - [Source](../install/installation.md#configure-it) + - [GDK](https://gitlab.com/gitlab-org/gitlab/blob/master/config/gitlab.yml.example) +- Layer: Core Service (Processor) +- Process: `puma` +- GitLab.com: [Puma](../user/gitlab_com/index.md#puma) + +[Puma](https://puma.io/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version. + #### Unicorn +NOTE: **Note:** +Starting with GitLab 13.0, Puma is the default web server and Unicorn has been +disabled by default. + - [Project page](https://gitlab.com/gitlab-org/gitlab/blob/master/README.md) - Configuration: - [Omnibus](https://docs.gitlab.com/omnibus/settings/unicorn.html) @@ -669,7 +767,7 @@ You can install them after you create a cluster. This includes: - [JupyterHub](https://jupyter.org) - [Knative](https://cloud.google.com/knative/) -## GitLab by Request Type +## GitLab by request type GitLab provides two "interfaces" for end users to access the service: @@ -678,20 +776,20 @@ GitLab provides two "interfaces" for end users to access the service: It's important to understand the distinction as some processes are used in both and others are exclusive to a specific request type. -### GitLab Web HTTP Request Cycle +### GitLab Web HTTP request cycle When making a request to an HTTP Endpoint (think `/users/sign_in`) the request will take the following path through the GitLab Service: - NGINX - Acts as our first line reverse proxy. -- GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn. -- Unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn. +- GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Puma. +- Puma - Since this is a web request, and it needs to access the application it will go to Puma. - PostgreSQL/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data. -### GitLab Git Request Cycle +### GitLab Git request cycle Below we describe the different paths that HTTP vs. SSH Git requests will take. There is some overlap with the Web Request Cycle but also some differences. -### Web Request (80/443) +### Web request (80/443) Git operations over HTTP use the stateless "smart" protocol described in the [Git documentation](https://git-scm.com/docs/http-protocol), but responsibility @@ -736,7 +834,7 @@ sequenceDiagram The sequence is similar for `git push`, except `git-receive-pack` is used instead of `git-upload-pack`. -### SSH Request (22) +### SSH request (22) Git operations over SSH can use the stateful protocol described in the [Git documentation](https://git-scm.com/docs/pack-protocol#_ssh_transport), but @@ -801,7 +899,7 @@ except there is no round-trip into Gitaly - Rails performs the action as part of the [internal API](internal_api.md) call, and GitLab Shell streams the response back to the user directly. -## System Layout +## System layout When referring to `~git` in the pictures it means the home directory of the Git user which is typically `/home/git`. @@ -811,9 +909,9 @@ The bare repositories are located in `/home/git/repositories`. GitLab is a Ruby To serve repositories over SSH there's an add-on application called GitLab Shell which is installed in `/home/git/gitlab-shell`. -### Installation Folder Summary +### Installation folder summary -To summarize here's the [directory structure of the `git` user home directory](../install/structure.md). +To summarize here's the [directory structure of the `git` user home directory](../install/installation.md#gitlab-directory-structure). ### Processes @@ -823,12 +921,12 @@ ps aux | grep '^git' GitLab has several components to operate. It requires a persistent database (PostgreSQL) and Redis database, and uses Apache `httpd` or NGINX to proxypass -Unicorn. All these components should run as different system users to GitLab -(e.g., `postgres`, `redis` and `www-data`, instead of `git`). +Puma. All these components should run as different system users to GitLab +(for example, `postgres`, `redis`, and `www-data`, instead of `git`). -As the `git` user it starts Sidekiq and Unicorn (a simple Ruby HTTP server +As the `git` user it starts Sidekiq and Puma (a simple Ruby HTTP server running on port `8080` by default). Under the GitLab user there are normally 4 -processes: `unicorn_rails master` (1 process), `unicorn_rails worker` +processes: `puma master` (1 process), `puma cluster worker` (2 processes), `sidekiq` (1 process). ### Repository access @@ -841,7 +939,7 @@ See the README for more information. ### Init scripts of the services -The GitLab init script starts and stops Unicorn and Sidekiq: +The GitLab init script starts and stops Puma and Sidekiq: ```plaintext /etc/init.d/gitlab @@ -881,9 +979,9 @@ Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [v ### Log locations of the services -GitLab (includes Unicorn and Sidekiq logs): +GitLab (includes Puma and Sidekiq logs): -- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `git_json.log` and `unicorn.stderr.log` normally. +- `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `puma.stdout.log`, `git_json.log` and `puma.stderr.log` normally. GitLab Shell: @@ -914,17 +1012,18 @@ PostgreSQL: ### GitLab specific configuration files -GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced config files include: +GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly referenced +configuration files include: -- `gitlab.yml` - GitLab configuration. -- `unicorn.rb` - Unicorn web server settings. -- `database.yml` - Database connection settings. +- `gitlab.yml` - GitLab configuration +- `puma.rb` - Puma web server settings +- `database.yml` - Database connection settings GitLab Shell has a configuration file at `/home/git/gitlab-shell/config.yml`. -### Maintenance Tasks +### Maintenance tasks -[GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master) provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance Rake tasks](../raketasks/maintenance.md). +[GitLab](https://gitlab.com/gitlab-org/gitlab/tree/master) provides Rake tasks with which you see version information and run a quick check on your configuration to ensure it is configured properly within the application. See [maintenance Rake tasks](../administration/raketasks/maintenance.md). In a nutshell, do the following: ```shell @@ -934,7 +1033,8 @@ bundle exec rake gitlab:env:info RAILS_ENV=production bundle exec rake gitlab:check RAILS_ENV=production ``` -Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While the sudo commands provided by GitLab work in Ubuntu they do not always work in RHEL. +Note: It is recommended to log into the `git` user using `sudo -i -u git` or `sudo su - git`. While +the `sudo` commands provided by GitLab work in Ubuntu they do not always work in RHEL. ## GitLab.com diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 4b58758b5c7..e5cc2ae4d1d 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -1,4 +1,11 @@ -# Background Migrations +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Background migrations Background migrations can be used to perform data migrations that would otherwise take a very long time (hours, days, years, etc) to complete. For @@ -92,6 +99,20 @@ bulk_migrate_async( ) ``` +Note that this will queue a Sidekiq job immediately: if you have a large number +of records, this may not be what you want. You can use the function +`queue_background_migration_jobs_by_range_at_intervals` to split the job into +batches: + +```ruby +queue_background_migration_jobs_by_range_at_intervals( + ClassName, + BackgroundMigrationClassName, + 2.minutes, + batch_size: 10_000 + ) +``` + You'll also need to make sure that newly created data is either migrated, or saved in both the old and new version upon creation. For complex and time consuming migrations it's best to schedule a background job using an diff --git a/doc/development/changelog.md b/doc/development/changelog.md index f57e666540c..922c4814d91 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -31,14 +31,16 @@ the `author` field. GitLab team members **should not**. - Any change that introduces a database migration, whether it's regular, post, or data migration, **must** have a changelog entry, even if it is behind a - disabled feature flag. + disabled feature flag. Since the migration is executed on [GitLab FOSS](https://gitlab.com/gitlab-org/gitlab-foss/), + the changelog for database schema changes should be written to the + `changelogs/unreleased/` directory, even when other elements of that change affect only GitLab EE. + - [Security fixes](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md) **must** have a changelog entry, without `merge_request` value and with `type` set to `security`. -- Any user-facing change **should** have a changelog entry. Example: "GitLab now - uses system fonts for all text." +- Any user-facing change **should** have a changelog entry. This includes both visual changes (regardless of how minor), and changes to the rendered DOM which impact how a screen reader may announce the content. - Performance improvements **should** have a changelog entry. -- Changes that need to be documented in the Telemetry [Event Dictionary](telemetry/event_dictionary.md) +- Changes that need to be documented in the Product Analytics [Event Dictionary](product_analytics/event_dictionary.md) also require a changelog entry. - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. @@ -46,6 +48,7 @@ the `author` field. GitLab team members **should not**. - Any docs-only changes **should not** have a changelog entry. - Any change behind a disabled feature flag **should not** have a changelog entry. - Any change behind an enabled feature flag **should** have a changelog entry. +- Any change that adds new usage data metrics and changes that needs to be documented in Product Analytics [Event Dictionary](telemetry/event_dictionary.md) **should** have a changelog entry. - A change that [removes a feature flag](feature_flags/development.md) **should** have a changelog entry - only if the feature flag did not default to true already. - A fix for a regression introduced and then fixed in the same release (i.e., diff --git a/doc/development/chatops_on_gitlabcom.md b/doc/development/chatops_on_gitlabcom.md index 3c1c7750842..c64766af589 100644 --- a/doc/development/chatops_on_gitlabcom.md +++ b/doc/development/chatops_on_gitlabcom.md @@ -22,9 +22,7 @@ To request access to Chatops on GitLab.com: 1. Log into <https://ops.gitlab.net/users/sign_in> **using the same username** as for GitLab.com (you may have to rename it). 1. You could also use the "Sign in with" Google button to sign in, with your GitLab.com email address. 1. Ask one of your team members to add you to the `chatops` project in Ops. They can do it by running `/chatops run member add <username> gitlab-com/chatops --ops` command in the `#chat-ops-test` Slack channel. - -NOTE: **Note:** -If you had to change your username for GitLab.com on the first step, make sure [to reflect this information](https://gitlab.com/gitlab-com/www-gitlab-com#adding-yourself-to-the-team-page) on [the team page](https://about.gitlab.com/company/team/). +1. If you had to change your username for GitLab.com on the first step, make sure [to reflect this information](https://gitlab.com/gitlab-com/www-gitlab-com#adding-yourself-to-the-team-page) on [the team page](https://about.gitlab.com/company/team/). ## See also diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 77cedc9814e..7d522a55559 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -13,15 +13,16 @@ This document explains how to develop [GitLab CI/CD templates](../../ci/examples All template files reside in the `lib/gitlab/ci/templates` directory, and are categorized by the following sub-directories: -| Sub-directory | Content | [Selectable in UI](#make-sure-the-new-template-can-be-selected-in-ui) | -|----------------|--------------------------------------------------------------|-----------------------------------------------------------------------| -| `/AWS/*` | Cloud Deployment (AWS) related jobs | No | -| `/Jobs/*` | Auto DevOps related jobs | No | -| `/Pages/*` | Static site generators for GitLab Pages (for example Jekyll) | Yes | -| `/Security/*` | Security related jobs | Yes | -| `/Verify/*` | Verify/testing related jobs | Yes | -| `/Workflows/*` | Common uses of the `workflow:` keyword | No | -| `/*` (root) | General templates | Yes | +| Sub-directory | Content | [Selectable in UI](#make-sure-the-new-template-can-be-selected-in-ui) | +|----------------|----------------------------------------------------|-----------------------------------------------------------------------| +| `/AWS/*` | Cloud Deployment (AWS) related jobs | No | +| `/Jobs/*` | Auto DevOps related jobs | No | +| `/Pages/*` | Static site generators for GitLab Pages (for example Jekyll) | Yes | +| `/Security/*` | Security related jobs | Yes | +| `/Terraform/*` | Infrastructure as Code related templates | No | +| `/Verify/*` | Verify/testing related jobs | Yes | +| `/Workflows/*` | Common uses of the `workflow:` keyword | No | +| `/*` (root) | General templates | Yes | ## Criteria @@ -110,7 +111,7 @@ to include older template versions. If other templates are included with `includ they can be combined with the `include: remote`: ```yaml -# To use the v13 stable template, which is not included in v14, fetch the specifc +# To use the v13 stable template, which is not included in v14, fetch the specific # template from the remote template repository with the `include:remote:` keyword. # If you fetch from the GitLab canonical project, use the following URL format: # https://gitlab.com/gitlab-org/gitlab/-/raw/<version>/lib/gitlab/ci/templates/<template-name> diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 2e319efa5f3..3ff802d3b23 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -67,6 +67,10 @@ page, with these behaviors: contains the string 'OOO', or the emoji is `:palm_tree:` or `:beach:`. 1. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) are three times as likely to be picked as other reviewers. +1. People whose [GitLab status](../user/profile/index.md#current-status) emoji + is `:large_blue_circle:` are more likely to be picked. This applies to both reviewers and trainee maintainers. + - Reviewers with `:large_blue_circle:` are two times as likely to be picked as other reviewers. + - Trainee maintainers with `:large_blue_circle:` are four times as likely to be picked as other reviewers. 1. It always picks the same reviewers and maintainers for the same branch name (unless their OOO status changes, as in point 1). It removes leading `ce-` and `ee-`, and trailing `-ce` and `-ee`, so @@ -100,6 +104,7 @@ with [domain expertise](#domain-experts). by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**. 1. If your merge request only includes end-to-end changes (*3*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** 1. If your merge request includes a new or updated [application limit](https://about.gitlab.com/handbook/product/product-processes/#introducing-application-limits), it must be **approved by a [product manager](https://about.gitlab.com/company/team/)**. +1. If your merge request includes Product Analytics (telemetry) changes, it should be reviewed and approved by a [Product analytics engineer](https://gitlab.com/gitlab-org/growth/product-analytics/engineers). - (*1*): Please note that specs other than JavaScript specs are considered backend code. - (*2*): We encourage you to seek guidance from a database maintainer if your merge diff --git a/doc/development/contributing/community_roles.md b/doc/development/contributing/community_roles.md index 7d2d1b77a0e..d880361e3aa 100644 --- a/doc/development/contributing/community_roles.md +++ b/doc/development/contributing/community_roles.md @@ -1,3 +1,9 @@ +--- +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Community members & roles GitLab community members and their privileges/responsibilities. diff --git a/doc/development/contributing/design.md b/doc/development/contributing/design.md index 352392931c0..891c764f07f 100644 --- a/doc/development/contributing/design.md +++ b/doc/development/contributing/design.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Implement design & UI elements For guidance on UX implementation at GitLab, please refer to our [Design System](https://design.gitlab.com/). diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 7550fe69546..6cbe57bf926 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Contribute to GitLab Thank you for your interest in contributing to GitLab. This guide details how diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index bb7b4713a5e..b7c05a369f0 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Issues workflow ## Issue tracker guidelines @@ -320,6 +327,11 @@ look for issues labeled `~"Accepting merge requests"` with a [weight of 1](https More experienced contributors are very welcome to tackle [any of them](https://gitlab.com/groups/gitlab-org/-/issues?state=opened&label_name[]=Accepting+merge+requests&assignee_id=None). +For more complex features that have a weight of 2 or more and clear scope, we recommend looking at issues +with the [label `~"Community Challenge"`](https://gitlab.com/gitlab-org/gitlab/-/issues?scope=all&utf8=%E2%9C%93&state=opened&label_name[]=Accepting%20merge%20requests&label_name[]=Community%20challenge). +If your MR for the `~"Community Challenge"` issue gets merged, you will also have a chance to win a custom +GitLab merchandise. + If you've decided that you would like to work on an issue, please @-mention the [appropriate product manager](https://about.gitlab.com/handbook/product/#who-to-talk-to-for-what) as soon as possible. The product manager will then pull in appropriate GitLab team diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index d88b159b666..31f59ad875c 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Merge requests workflow We welcome merge requests from everyone, with fixes and improvements @@ -218,7 +225,7 @@ requirements. 1. [Secure coding guidelines](https://gitlab.com/gitlab-com/gl-security/security-guidelines) have been followed. 1. [Documented](../documentation/index.md) in the `/doc` directory. 1. [Changelog entry added](../changelog.md), if necessary. -1. Reviewed by relevant (UX/FE/BE/tech writing) reviewers and all concerns are addressed. +1. Reviewed by relevant reviewers and all concerns are addressed for Availability, Regressions, Security. Documentation reviews should take place as soon as possible, but they should not block a merge request. 1. Merged by a project maintainer. 1. Create an issue in the [infrastructure issue tracker](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues) to inform the Infrastructure department when your contribution is changing default settings or introduces a new setting, if relevant. 1. Confirmed to be working in the [Canary stage](https://about.gitlab.com/handbook/engineering/#canary-testing) or on GitLab.com once the contribution is deployed. diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index f6e64c1f1e6..773c1a771cd 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Style guides ## Editor/IDE styling standardization diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index 1b41a52c95e..85411ff9aa7 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Adding foreign key constraint to an existing column Foreign keys help ensure consistency between related database tables. The current database review process **always** encourages you to add [foreign keys](../foreign_keys.md) when creating tables that reference records from other tables. @@ -103,7 +109,7 @@ class RemoveRecordsWithoutUserFromEmailsTable < ActiveRecord::Migration[5.2] end def down - # Can be a no-op when data inconsistency is not affecting the pre and post deploymnet version of the application. + # Can be a no-op when data inconsistency is not affecting the pre and post deployment version of the application. # In this case we might have records in the `emails` table where the associated record in the `users` table is not there anymore. end end diff --git a/doc/development/database/client_side_connection_pool.md b/doc/development/database/client_side_connection_pool.md new file mode 100644 index 00000000000..1a30d2d73a3 --- /dev/null +++ b/doc/development/database/client_side_connection_pool.md @@ -0,0 +1,63 @@ +--- +type: dev, reference +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Client-side connection-pool + +Ruby processes accessing the database through +ActiveRecord, automatically calculate the connection-pool size for the +process based on the concurrency. + +Because of the way [Ruby on Rails manages database +connections](#connection-lifecycle), it is important that we have at +least as many connections as we have threads. While there is a 'pool' +setting in [`database.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/database.yml.postgresql), it is not very practical because you need to +maintain it in tandem with the number of application threads. For this +reason, we override the number of allowed connections in the database +connection-pool based on the configured number of application threads. + +`Gitlab::Runtime.max_threads` is the number of user-facing +application threads the process has been configured with. We also have +auxiliary threads that use database connections. As it isn't +straightforward to keep an accurate count of the number of auxiliary threads as +the application evolves over time, we just add a fixed headroom to the +number of user-facing threads. It is OK if this number is too large +because connections are instantiated lazily. + +## Troubleshooting connection-pool issues + +The connection-pool usage can be seen per environment in the [connection-pool +saturation +dashboard](https://dashboards.gitlab.net/d/alerts-sat_rails_db_connection_pool/alerts-rails_db_connection_pool-saturation-detail?orgId=1). + +If the connection-pool is too small, this would manifest in +`ActiveRecord::ConnectionTimeoutError`s from the application. Because we alert +when almost all connections are used, we should know this before +timeouts occur. If this happens we can remediate by setting the +`DB_POOL_HEADROOM` environment variable to something bigger than the +hardcoded value (10). + +At this point, we need to investigate what is using more connections +than we anticipated. To do that, we can use the +`gitlab_ruby_threads_running_threads` metric. For example, [this +graph](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum%20by%20(thread_name)%20(%20gitlab_ruby_threads_running_threads%7Buses_db_connection%3D%22yes%22%7D%20)&g0.tab=0) +shows all running threads that connect to the database by their +name. Threads labeled `puma worker` or `sidekiq_worker_thread` are +the threads that define `Gitlab::Runtime.max_threads` so those are +accounted for. If there's more than 10 other threads running, we could +consider raising the default headroom. + +## Connection lifecycle + +For web requests, a connection is obtained from the pool at the first +time a database query is made. The connection is returned to the pool +after the request completes. + +For background jobs, the behavior is very similar. The thread obtains +a connection for the first query, and returns it after the job is +finished. + +This is managed by Rails internally. diff --git a/doc/development/database/database_reviewer_guidelines.md b/doc/development/database/database_reviewer_guidelines.md index 6cb061f9959..3345df8b46b 100644 --- a/doc/development/database/database_reviewer_guidelines.md +++ b/doc/development/database/database_reviewer_guidelines.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Database Reviewer Guidelines This page includes introductory material for new database reviewers. diff --git a/doc/development/database/index.md b/doc/development/database/index.md index 9ea5b6fcaac..4bcefefe7a7 100644 --- a/doc/development/database/index.md +++ b/doc/development/database/index.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Database guides ## Database Reviews @@ -24,6 +30,7 @@ - [Background migrations](../background_migrations.md) - [Swapping tables](../swapping_tables.md) - [Deleting migrations](../deleting_migrations.md) +- [Partitioning tables](table_partitioning.md) ## Debugging @@ -49,6 +56,8 @@ - [Database Debugging and Troubleshooting](../database_debugging.md) - [Query Count Limits](../query_count_limits.md) - [Creating enums](../creating_enums.md) +- [Client-side connection-pool](client_side_connection_pool.md) +- [Updating multiple values](./setting_multiple_values.md) ## Case studies diff --git a/doc/development/database/not_null_constraints.md b/doc/development/database/not_null_constraints.md index e4dec2afa10..96271863d94 100644 --- a/doc/development/database/not_null_constraints.md +++ b/doc/development/database/not_null_constraints.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # `NOT NULL` constraints > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/38358) in GitLab 13.0. @@ -33,7 +39,7 @@ end ## Add a `NOT NULL` column to an existing table -With PostgreSQL 11 being the minimum version since GitLab 13.0, adding columns with `NULL` and/or +With PostgreSQL 11 being the minimum version in GitLab 13.0 and later, adding columns with `NULL` and/or default values has become much easier and the standard `add_column` helper should be used in all cases. For example, consider a migration that adds a new `NOT NULL` column `active` to table `db_guides`, diff --git a/doc/development/database/setting_multiple_values.md b/doc/development/database/setting_multiple_values.md new file mode 100644 index 00000000000..5569a0e10b7 --- /dev/null +++ b/doc/development/database/setting_multiple_values.md @@ -0,0 +1,103 @@ +# Setting Multiple Values + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32921) in GitLab 13.5. + +Frequently, we will want to update multiple objects with new values for one +or more columns. The obvious way to do this is using `Relation#update_all`: + +```ruby +user.issues.open.update_all(due_date: 7.days.from_now) # (1) +user.issues.update_all('relative_position = relative_position + 1') # (2) +``` + +But what do you do if you cannot express the update as either a static value (1) +or as a calculation (2)? + +Thankfully we can use `UPDATE FROM` to express the need to update multiple rows +with distinct values in a single query. One can either use a temporary table, or +a Common Table Expression (CTE), and then use that as the source of the updates: + +```sql +with updates(obj_id, new_title, new_weight) as ( + values (1 :: integer, 'Very difficult issue' :: text, 8 :: integer), + (2, 'Very easy issue', 1) +) +update issues + set title = new_title, weight = new_weight + from updates + where id = obj_id +``` + +The bad news: There is no way to express this in ActiveRecord or even dropping +down to ARel - the `UpdateManager` just does not support `update from`, so this +is not expressible. + +The good news: We supply an abstraction to help you generate these kinds of +updates, called `Gitlab::Database::BulkUpdate`. This constructs queries such as the +above, and uses binding parameters to avoid SQL injection. + +## Usage + +To use this, we need: + +- the list of columns to update +- a mapping from object/ID to the new values to set for that object +- a way to determine the table for each object + +So for example, we can express the query above as: + +```ruby +issue_a = Issue.find(..) +issue_b = Issue.find(..) + +# Issues a single query: +::Gitlab::Database::BulkUpdate.execute(%i[title weight], { + issue_a => { title: 'Very difficult issue', weight: 8 }, + issue_b => { title: 'Very easy issue', weight: 1 } +}) +``` + +Here the table can be determined automatically, from calling +`object.class.table_name`, so we don't need to provide anything. + +We can even pass heterogeneous sets of objects, if the updates all make sense +for them: + +```ruby +issue_a = Issue.find(..) +issue_b = Issue.find(..) +merge_request = MergeRequest.find(..) + +# Issues two queries +::Gitlab::Database::BulkUpdate.execute(%i[title], { + issue_a => { title: 'A' }, + issue_b => { title: 'B' }, + merge_request => { title: 'B' } +}) +``` + +If your objects do not return the correct model class (perhaps because they are +part of a union), then we need to specify this explicitly in a block: + +```ruby +bazzes = params +objects = Foo.from_union([ + Foo.select("id, 'foo' as object_type").where(quux: true), + Bar.select("id, 'bar' as object_type").where(wibble: true) + ]) +# At this point, all the objects are instances of Foo, even the ones from the +# Bar table +mapping = objects.to_h { |obj| [obj, bazzes[obj.id] } + +# Issues at most 2 queries +::Gitlab::Database::BulkUpdate.execute(%i[baz], mapping) do |obj| + obj.object_type.constantize +end +``` + +## Caveats + +Note that this is a **very low level** tool, and operates on the raw column +values. Enumerations and state fields must be translated into their underlying +representations, for example, and nested associations are not supported. No +validations or hooks will be called. diff --git a/doc/development/database/strings_and_the_text_data_type.md b/doc/development/database/strings_and_the_text_data_type.md index b73dfa859fb..fe8cfa5cd22 100644 --- a/doc/development/database/strings_and_the_text_data_type.md +++ b/doc/development/database/strings_and_the_text_data_type.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Strings and the Text data type > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30453) in GitLab 13.0. diff --git a/doc/development/database/table_partitioning.md b/doc/development/database/table_partitioning.md new file mode 100644 index 00000000000..30d0b0a2f5b --- /dev/null +++ b/doc/development/database/table_partitioning.md @@ -0,0 +1,259 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Database table partitioning + +Table partitioning is a powerful database feature that allows a table's +data to be split into smaller physical tables that act as a single large +table. If the application is designed to work with partitioning in mind, +there can be multiple benefits, such as: + +- Query performance can be improved greatly, because the database can +cheaply eliminate much of the data from the search space, while still +providing full SQL capabilities. + +- Bulk deletes can be achieved with minimal impact on the database by +dropping entire partitions. This is a natural fit for features that need +to periodically delete data that falls outside the retention window. + +- Administrative tasks like `VACUUM` and index rebuilds can operate on +individual partitions, rather than across a single massive table. + +Unfortunately, not all models fit a partitioning scheme, and there are +significant drawbacks if implemented incorrectly. Additionally, tables +can only be partitioned at their creation, making it nontrivial to apply +partitioning to a busy database. A suite of migration tools are available +to enable backend developers to partition existing tables, but the +migration process is rather heavy, taking multiple steps split across +several releases. Due to the limitations of partitioning and the related +migrations, you should understand how partitioning fits your use case +before attempting to leverage this feature. + +## Determining when to use partitioning + +While partitioning can be very useful when properly applied, it's +imperative to identify if the data and workload of a table naturally fit a +partitioning scheme. There are a few details you'll have to understand +in order to decide if partitioning is a good fit for your particular +problem. + +First, a table is partitioned on a partition key, which is a column or +set of columns which determine how the data will be split across the +partitions. The partition key is used by the database when reading or +writing data, to decide which partition(s) need to be accessed. The +partition key should be a column that would be included in a `WHERE` +clause on almost all queries accessing that table. + +Second, it's necessary to understand the strategy the database will +use to split the data across the partitions. The scheme supported by the +GitLab migration helpers is date-range partitioning, where each partition +in the table contains data for a single month. In this case, the partitioning +key would need to be a timestamp or date column. In order for this type of +partitioning to work well, most queries would need to access data within a +certain date range. + +For a more concrete example, the `audit_events` table can be used, which +was the first table to be partitioned in the application database +(scheduled for deployment with the GitLab 13.5 release). This +table tracks audit entries of security events that happen in the +application. In almost all cases, users want to see audit activity that +occurs in a certain timeframe. As a result, date-range partitioning +was a natural fit for how the data would be accessed. + +To look at this in more detail, imagine a simplified `audit_events` schema: + +```sql +CREATE TABLE audit_events ( + id SERIAL NOT NULL PRIMARY KEY, + author_id INT NOT NULL, + details jsonb NOT NULL, + created_at timestamptz NOT NULL); +``` + +Now imagine typical queries in the UI would display the data within a +certain date range, like a single week: + +```sql +SELECT * +FROM audit_events +WHERE created_at >= '2020-01-01 00:00:00' + AND created_at < '2020-01-08 00:00:00' +ORDER BY created_at DESC +LIMIT 100 +``` + +If the table is partitioned on the `created_at` column the base table would +look like: + +```sql +CREATE TABLE audit_events ( + id SERIAL NOT NULL, + author_id INT NOT NULL, + details jsonb NOT NULL, + created_at timestamptz NOT NULL, + PRIMARY KEY (id, created_at)) +PARTITION BY RANGE(created_at); +``` + +NOTE: **Note:** +The primary key of a partitioned table must include the partition key as +part of the primary key definition. + +And we might have a list of partitions for the table, such as: + +```sql +audit_events_202001 FOR VALUES FROM ('2020-01-01') TO ('2020-02-01') +audit_events_202002 FOR VALUES FROM ('2020-02-01') TO ('2020-03-01') +audit_events_202003 FOR VALUES FROM ('2020-03-01') TO ('2020-04-01') +``` + +Each partition is a separate physical table, with the same structure as +the base `audit_events` table, but contains only data for rows where the +partition key falls in the specified range. For example, the partition +`audit_events_202001` contains rows where the `created_at` column is +greater than or equal to `2020-01-01` and less than `2020-02-01`. + +Now, if we look at the previous example query again, the database can +use the `WHERE` to recognize that all matching rows will be in the +`audit_events_202001` partition. Rather than searching all of the data +in all of the partitions, it can search only the single month's worth +of data in the appropriate partition. In a large table, this can +dramatically reduce the amount of data the database needs to access. +However, imagine a query that does not filter based on the partitioning +key, such as: + +```sql +SELECT * +FROM audit_events +WHERE author_id = 123 +ORDER BY created_at DESC +LIMIT 100 +``` + +In this example, the database can't prune any partitions from the search, +because matching data could exist in any of them. As a result, it has to +query each partition individually, and aggregate the rows into a single result +set. Since `author_id` would be indexed, the performance impact could +likely be acceptable, but on more complex queries the overhead can be +substantial. Partitioning should only be leveraged if the access patterns +of the data support the partitioning strategy, otherwise performance will +suffer. + +## Partitioning a table + +Unfortunately, tables can only be partitioned at their creation, making +it nontrivial to apply to a busy database. A suite of migration +tools have been developed to enable backend developers to partition +existing tables. This migration process takes multiple steps which must +be split across several releases. + +### Caveats + +The partitioning migration helpers work by creating a partitioned duplicate +of the original table and using a combination of a trigger and a background +migration to copy data into the new table. Changes to the original table +schema can be made in parallel with the partitioning migration, but they +must take care to not break the underlying mechanism that makes the migration +work. For example, if a column is added to the table that is being +partitioned, both the partitioned table and the trigger definition need to +be updated to match. + +### Step 1: Creating the partitioned copy (Release N) + +The first step is to add a migration to create the partitioned copy of +the original table. This migration will also create the appropriate +partitions based on the data in the original table, and install a +trigger that will sync writes from the original table into the +partitioned copy. + +An example migration of partitioning the `audit_events` table by its +`created_at` column would look like: + +```ruby +class PartitionAuditEvents < ActiveRecord::Migration[6.0] + include Gitlab::Database::PartitioningMigrationHelpers + + def up + partition_table_by_date :audit_events, :created_at + end + + def down + drop_partitioned_table_for :audit_events + end +end +``` + +Once this has executed, any inserts, updates or deletes in the +original table will also be duplicated in the new table. For updates and +deletes, the operation will only have an effect if the corresponding row +exists in the partitioned table. + +### Step 2: Backfill the partitioned copy (Release N) + +The second step is to add a post-deployment migration that will schedule +the background jobs that will backfill existing data from the original table +into the partitioned copy. + +Continuing the above example, the migration would look like: + +```ruby +class BackfillPartitionAuditEvents < ActiveRecord::Migration[6.0] + include Gitlab::Database::PartitioningMigrationHelpers + + def up + enqueue_partitioning_data_migration :audit_events + end + + def down + cleanup_partitioning_data_migration :audit_events + end +end +``` + +This step uses the same mechanism as any background migration, so you +may want to read the [Background Migration](../background_migrations.md) +guide for details on that process. Background jobs are scheduled every +2 minutes and copy `50_000` records at a time, which can be used to +estimate the timing of the background migration portion of the +partitioning migration. + +### Step 3: Post-backfill cleanup (Release N+1) + +The third step must occur at least one release after the release that +includes the background migration. This gives time for the background +migration to execute properly in self-managed installations. In this step, +add another post-deployment migration that will cleanup after the +background migration. This includes forcing any remaining jobs to +execute, and copying data that may have been missed, due to dropped or +failed jobs. + +Once again, continuing the example, this migration would look like: + +```ruby +class CleanupPartitionedAuditEventsBackfill < ActiveRecord::Migration[6.0] + include Gitlab::Database::PartitioningMigrationHelpers + + def up + finalize_backfilling_partitioned_table :audit_events + end + + def down + # no op + end +end +``` + +After this migration has completed, the original table and partitioned +table should contain identical data. The trigger installed on the +original table guarantees that the data will remain in sync going +forward. + +### Step 4: Swap the partitioned and non-partitioned tables (Release N+1) + +The final step of the migration will make the partitioned table ready +for use by the application. This section will be updated when the +migration helper is ready, for now development can be followed in the +[Tracking Issue](https://gitlab.com/gitlab-org/gitlab/-/issues/241267). diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index 61e8ac60bfe..ebd57df498e 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Troubleshooting and Debugging Database This section is to help give some copy-pasta you can use as a reference when you @@ -19,7 +25,7 @@ If you just want to delete everything and start over with an empty DB (approxima bundle exec rake db:reset RAILS_ENV=development ``` -If you just want to delete everything and start over with dummy data (approximately 4 minutes). This +If you just want to delete everything and start over with sample data (approximately 4 minutes). This also does `db:reset` and runs DB-specific migrations: ```shell diff --git a/doc/development/database_query_comments.md b/doc/development/database_query_comments.md index 77943f2b261..ccaaadef4f4 100644 --- a/doc/development/database_query_comments.md +++ b/doc/development/database_query_comments.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Database query comments with Marginalia The [Marginalia gem](https://github.com/basecamp/marginalia) is used to add diff --git a/doc/development/database_review.md b/doc/development/database_review.md index f56ffdbad21..3f5f36b0b6e 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Database Review Guidelines This page is specific to database reviews. Please refer to our @@ -21,7 +27,7 @@ A database review is required for: database review. - Changes in usage data metrics that use `count` and `distinct_count`. These metrics could have complex queries over large tables. - See the [Telemetry Guide](telemetry/usage_ping.md#implementing-usage-ping) + See the [Product Analytics Guide](product_analytics/usage_ping.md#implementing-usage-ping) for implementation details. A database reviewer is expected to look out for obviously complex @@ -184,10 +190,6 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac - [Check query plans](understanding_explain_plans.md) and suggest improvements to queries (changing the query, schema or adding indexes and similar) - General guideline is for queries to come in below 100ms execution time - - If queries rely on prior migrations that are not present yet on production - (eg indexes, columns), you can use a [one-off instance from the restore - pipeline](https://ops.gitlab.net/gitlab-com/gl-infra/gitlab-restore/postgres-gprd) - in order to establish a proper testing environment. If you don't have access to this project, reach out to #database on Slack to get advice on how to proceed. - Avoid N+1 problems and minimalize the [query count](merge_request_performance_guidelines.md#query-counts). ### Timing guidelines for migrations diff --git a/doc/development/db_dump.md b/doc/development/db_dump.md index 4095932e44c..b16cd4b08d1 100644 --- a/doc/development/db_dump.md +++ b/doc/development/db_dump.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Importing a database dump into a staging environment Sometimes it is useful to import the database from a production environment diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md index 9a4c15c5c19..952435150d6 100644 --- a/doc/development/distributed_tracing.md +++ b/doc/development/distributed_tracing.md @@ -1,15 +1,12 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- # Distributed Tracing - development guidelines -NOTE: **Note:** -Distributed Tracing in GitLab is currently considered **experimental**, as it has not yet been tested at scale on GitLab.com. - -GitLab is instrumented for distributed tracing. +GitLab is instrumented for distributed tracing. Distributed Tracing in GitLab is currently considered **experimental**, as it has not yet been tested at scale on GitLab.com. According to [Open Tracing](https://opentracing.io/docs/overview/what-is-tracing/): diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index a4a6ee2fa0f..0f03ceeb4b5 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -52,7 +52,7 @@ For feature flags disabled by default, if they can be used by end users: do not say anything about it. - Say whether it's recommended for production use. - Document how to enable and disable it. -- Add a warning to the user saying that the feature is disabled. +- Add a warning to the user saying that the feature might be disabled. For example, for a feature disabled by default, disabled on GitLab.com, cannot be enabled for a single project, and is not ready for production use: @@ -250,7 +250,7 @@ be enabled by project, and is ready for production use: > - [Introduced](link-to-issue) in GitLab 12.0. > - It's [deployed behind a feature flag](<replace with path to>/user/feature_flags.md), enabled by default. > - It's enabled on GitLab.com. -> - It can be enabled or disable for a single project. +> - It can be enabled or disabled for a single project. > - It's recommended for production use. > - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#anchor-to-section). **(CORE ONLY)** diff --git a/doc/development/documentation/graphql_styleguide.md b/doc/development/documentation/graphql_styleguide.md new file mode 100644 index 00000000000..11e6b159359 --- /dev/null +++ b/doc/development/documentation/graphql_styleguide.md @@ -0,0 +1,92 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +description: "Writing styles, markup, formatting, and other standards for GraphQL API's GitLab Documentation." +--- + +# GraphQL API + +GraphQL APIs are different from [RESTful APIs](restful_api_styleguide.md). Reference +information is generated in our [GraphQL reference](../../api/graphql/reference/index.md). + +However, it's helpful to include examples on how to use GraphQL for different +*use cases*, with samples that readers can use directly in the +[GraphiQL explorer](../api_graphql_styleguide.md#graphiql). + +This section describes the steps required to add your GraphQL examples to +GitLab documentation. + +## Add a dedicated GraphQL page + +To create a dedicated GraphQL page, create a new `.md` file in the +`doc/api/graphql/` directory. Give that file a functional name, such as +`import_from_specific_location.md`. + +## Start the page with an explanation + +Include a page title that describes the GraphQL functionality in a few words, +such as: + +```markdown +# Search for [substitute kind of data] +``` + +Describe the search. One sentence may be all you need. More information may +help readers learn how to use the example for their GitLab deployments. + +## Include a procedure using the GraphiQL explorer + +The GraphiQL explorer can help readers test queries with working deployments. +Set up the section with the following: + +- Use the following title: + + ```markdown + ## Set up the GraphiQL explorer + ``` + +- Include a code block with the query that anyone can include in their + instance of the GraphiQL explorer: + + ````markdown + ```graphql + query { + <insert queries here> + } + ``` + ```` + +- Tell the user what to do: + + ```markdown + 1. Open the GraphiQL explorer tool in the following URL: `https://gitlab.com/-/graphql-explorer`. + 1. Paste the `query` listed above into the left window of your GraphiQL explorer tool. + 1. Select **Play** to get the result shown here: + ``` + +- Include a screenshot of the result in the GraphiQL explorer. Follow the naming + convention described in the [Save the image](styleguide.md#save-the-image) section of the Documentation style guide. +- Follow up with an example of what you can do with the output. Make sure the + example is something that readers can do on their own deployments. +- Include a link to the [GraphQL API resources](../../api/graphql/reference/index.md). + +## Add the GraphQL example to the global navigation + +You should include a link for your new document in the global navigation (the list on the +left side of the documentation website). To do so, open a second MR, against the +[GitLab documentation repository](https://gitlab.com/gitlab-org/gitlab-docs/). + +We store our global navigation in the [`default-nav.yaml`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/content/_data/default-nav.yaml) file, in the +`content/_data` subdirectory. You can find the GraphQL section under the +following line: + +```yaml +- category_title: GraphQL +``` + +Be aware that CI tests for that second MR will fail with a bad link until the +main MR that adds the new GraphQL page is merged. Therefore, only merge the MR against the +[`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs) repository after the content has +been merged and live on `docs.gitlab.com`. diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index d6f24d6374d..e51f966ee6e 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -1,4 +1,7 @@ --- +stage: none +group: Documentation Guidelines +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: Learn how to contribute to GitLab Documentation. --- @@ -52,9 +55,9 @@ docs-only merge requests using the following guide: [Contributions to GitLab docs](workflow.md) are welcome from the entire GitLab community. -To ensure that GitLab docs are current, there are special processes and responsibilities for all [feature changes](feature-change-workflow.md), that is development work that impacts the appearance, usage, or administration of a feature. +To ensure that GitLab docs are current, there are special processes and responsibilities for all [feature changes](workflow.md), that is development work that impacts the appearance, usage, or administration of a feature. -However, anyone can contribute [documentation improvements](improvement-workflow.md) that are not associated with a feature change. For example, adding a new doc on how to accomplish a use case that's already possible with GitLab or with third-party tools and GitLab. +However, anyone can contribute [documentation improvements](workflow.md) that are not associated with a feature change. For example, adding a new doc on how to accomplish a use case that's already possible with GitLab or with third-party tools and GitLab. ## Markdown and styles @@ -128,7 +131,7 @@ The following metadata should be added when a page is moved to another location: - `redirect_to`: The relative path and filename (with an `.md` extension) of the location to which visitors should be redirected for a moved page. - [Learn more](#changing-document-location). + [Learn more](#move-or-rename-a-page). - `disqus_identifier`: Identifier for Disqus commenting system. Used to keep comments with a page that's been moved to a new URL. [Learn more](#redirections-for-pages-with-disqus-comments). @@ -156,17 +159,18 @@ Nanoc layout), which will be displayed at the top of the page if defined: [algorithm](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/lib/helpers/reading_time.rb) to calculate the reading time based on the number of words. -## Changing document location +## Move or rename a page + +Moving or renaming a document is the same as changing its location. This process +requires specific steps to ensure that visitors can find the new +documentation page, whether they're using `/help` from a GitLab instance or by +visiting <https://docs.gitlab.com>. -Changing a document's location requires specific steps to ensure that -users can seamlessly access the new doc page, whether they are accessing content -on a GitLab instance domain at `/help` or at <https://docs.gitlab.com>. Be sure to assign a -technical writer if you have any questions during the process (such as -whether the move is necessary), and ensure that a technical writer reviews this -change prior to merging. +Be sure to assign a technical writer to a page move or rename MR. Technical +Writers can help with any questions and can review your change. -If you indeed need to change a document's location, do not remove the old -document, but instead replace all of its content with the following: +To change a document's location, don't remove the old document, but instead +replace all of its content with the following: ```markdown --- @@ -176,14 +180,18 @@ redirect_to: '../path/to/file/index.md' This document was moved to [another location](../path/to/file/index.md). ``` -Where `../path/to/file/index.md` is usually the relative path to the old document. +Replace `../path/to/file/index.md` with the relative path to the old document. + +The `redirect_to` variable supports both full and relative URLs; for example: -The `redirect_to` variable supports both full and relative URLs, for example -`https://docs.gitlab.com/ee/path/to/file.html`, `../path/to/file.html`, `path/to/file.md`. -It ensures that the redirect will work for <https://docs.gitlab.com> and any `*.md` paths -will be compiled to `*.html`. -The new line underneath the front matter informs the user that the document -changed location and is useful for someone that browses that file from the repository. +- `https://docs.gitlab.com/ee/path/to/file.html` +- `../path/to/file.html` +- `path/to/file.md` + +The redirect works for <https://docs.gitlab.com>, and any `*.md` paths are +changed to `*.html`. The description line following the `redirect_to` code +informs the visitor that the document changed location if the redirect process +doesn't complete successfully. For example, if you move `doc/workflow/lfs/index.md` to `doc/administration/lfs.md`, then the steps would be: @@ -208,9 +216,8 @@ For example, if you move `doc/workflow/lfs/index.md` to git grep -n "lfs/lfs_administration" ``` -NOTE: **Note:** -If the document being moved has any Disqus comments on it, there are extra steps -to follow documented just [below](#redirections-for-pages-with-disqus-comments). +1. If the document being moved has any Disqus comments on it, follow the steps + described in [Redirections for pages with Disqus comments](#redirections-for-pages-with-disqus-comments). Things to note: @@ -241,7 +248,8 @@ using the old URL as value. For example, let's say we moved the document available under `https://docs.gitlab.com/my-old-location/README.html` to a new location, `https://docs.gitlab.com/my-new-location/index.html`. -Into the **new document** front matter, we add the following: +Into the **new document** front matter, we add the following information. You must +include the file name in the `disqus_identifier` URL, even if it's `index.html` or `README.html`. ```yaml --- @@ -249,9 +257,6 @@ disqus_identifier: 'https://docs.gitlab.com/my-old-location/README.html' --- ``` -Note: it is necessary to include the file name in the `disqus_identifier` URL, -even if it's `index.html` or `README.html`. - ## Merge requests for GitLab documentation Before getting started, make sure you read the introductory section @@ -267,9 +272,8 @@ represents a good-faith effort to follow the template and style standards, and is believed to be accurate. Further needs for what would make the doc even better should be immediately addressed -in a follow-up MR or issue. +in a follow-up merge request or issue. -NOTE: **Note:** If the release version you want to add the documentation to has already been frozen or released, use the label `~"Pick into X.Y"` to get it merged into the correct release. Avoid picking into a past release as much as you can, as @@ -392,8 +396,7 @@ You will need at least Maintainer permissions to be able to run it.  -NOTE: **Note:** -You will need to push a branch to those repositories, it doesn't work for forks. +You must push a branch to those repositories, as it doesn't work for forks. The `review-docs-deploy*` job will: @@ -410,17 +413,16 @@ minutes and it should appear online, otherwise you can check the status of the remote pipeline from the link in the merge request's job output. If the pipeline failed or got stuck, drop a line in the `#docs` chat channel. -TIP: **Tip:** -Someone with no merge rights to the GitLab projects (think of forks from -contributors) cannot run the manual job. In that case, you can -ask someone from the GitLab team who has the permissions to do that for you. - -NOTE: **Note:** Make sure that you always delete the branch of the merge request you were working on. If you don't, the remote docs branch won't be removed either, and the server where the Review Apps are hosted will eventually be out of disk space. +TIP: **Tip:** +Someone with no merge rights to the GitLab projects (think of forks from +contributors) cannot run the manual job. In that case, you can +ask someone from the GitLab team who has the permissions to do that for you. + ### Troubleshooting review apps In case the review app URL returns 404, follow these steps to debug: @@ -462,7 +464,7 @@ If you want to know the in-depth details, here's what's really happening: The following GitLab features are used among others: - [Manual actions](../../ci/yaml/README.md#whenmanual) -- [Multi project pipelines](../../ci/multi_project_pipeline_graphs.md) +- [Multi project pipelines](../../ci/multi_project_pipelines.md) - [Review Apps](../../ci/review_apps/index.md) - [Artifacts](../../ci/yaml/README.md#artifacts) - [Specific runner](../../ci/runners/README.md#prevent-a-specific-runner-from-being-enabled-for-other-projects) @@ -668,7 +670,7 @@ build pipelines: ``` We recommend installing the version of `markdownlint-cli` currently used in the documentation - linting [Docker image](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/dockerfiles/Dockerfile.gitlab-docs-lint#L38). + linting [Docker image](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/.gitlab-ci.yml#L420). 1. Install [`vale`](https://github.com/errata-ai/vale/releases). For example, to install using `brew` for macOS, run: @@ -678,7 +680,7 @@ build pipelines: ``` We recommend installing the version of Vale currently used in the documentation linting - [Docker image](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/dockerfiles/Dockerfile.gitlab-docs-lint#L16). + [Docker image](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/master/.gitlab-ci.yml#L419). In addition to using markdownlint and Vale at the command line, these tools can be [integrated with your code editor](#configure-editors). diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md new file mode 100644 index 00000000000..b12578b5d98 --- /dev/null +++ b/doc/development/documentation/restful_api_styleguide.md @@ -0,0 +1,180 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +description: "Writing styles, markup, formatting, and other standards for GitLab's RESTful APIs." +--- + +# RESTful API + +REST API resources are documented in Markdown under +[`/doc/api`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api). Each +resource has its own Markdown file, which is linked from `api_resources.md`. + +When modifying the Markdown, also update the corresponding +[OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api/openapi) +if one exists for the resource. If not, consider creating one. Match the latest +[OpenAPI 3.0.x specification](https://swagger.io/specification/). (For more +information, see the discussion in this +[issue](https://gitlab.com/gitlab-org/gitlab/-/issues/16023#note_370901810).) + +In the Markdown doc for a resource (AKA endpoint): + +- Every method must have the REST API request. For example: + + ```plaintext + GET /projects/:id/repository/branches + ``` + +- Every method must have a detailed [description of the parameters](#method-description). +- Every method must have a cURL example. +- Every method must have a response body (in JSON format). + +## API topic template + +The following can be used as a template to get started: + +````markdown +## Descriptive title + +One or two sentence description of what endpoint does. + +```plaintext +METHOD /endpoint +``` + +| Attribute | Type | Required | Description | +|:------------|:---------|:---------|:----------------------| +| `attribute` | datatype | yes/no | Detailed description. | +| `attribute` | datatype | yes/no | Detailed description. | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/endpoint?parameters" +``` + +Example response: + +```json +[ + { + } +] +``` +```` + +## Method description + +Use the following table headers to describe the methods. Attributes should +always be in code blocks using backticks (`` ` ``). + +```markdown +| Attribute | Type | Required | Description | +|:----------|:-----|:---------|:------------| +``` + +Rendered example: + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:--------------------| +| `user` | string | yes | The GitLab username. | + +## cURL commands + +- Use `https://gitlab.example.com/api/v4/` as an endpoint. +- Wherever needed use this personal access token: `<your_access_token>`. +- Always put the request first. `GET` is the default so you don't have to + include it. +- Wrap the URL in double quotes (`"`). +- Prefer to use examples using the personal access token and don't pass data of + username and password. + +| Methods | Description | +|:------------------------------------------- |:------------------------------------------------------| +| `--header "PRIVATE-TOKEN: <your_access_token>"` | Use this method as is, whenever authentication needed. | +| `--request POST` | Use this method when creating new objects | +| `--request PUT` | Use this method when updating existing objects | +| `--request DELETE` | Use this method when removing existing objects | + +## cURL Examples + +The following sections include a set of [cURL](https://curl.haxx.se) examples +you can use in the API documentation. + +CAUTION: **Caution:** +Do not use information for real users, URLs, or tokens. For documentation, refer to our +relevant style guide sections on [Fake user information](styleguide.md#fake-user-information), +[Fake URLs](styleguide.md#fake-urls), and [Fake tokens](styleguide.md#fake-tokens). + +### Simple cURL command + +Get the details of a group: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/gitlab-org" +``` + +### cURL example with parameters passed in the URL + +Create a new project under the authenticated user's namespace: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects?name=foo" +``` + +### Post data using cURL's `--data` + +Instead of using `--request POST` and appending the parameters to the URI, you +can use cURL's `--data` option. The example below will create a new project +`foo` under the authenticated user's namespace. + +```shell +curl --data "name=foo" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects" +``` + +### Post data using JSON content + +NOTE: **Note:** +In this example we create a new group. Watch carefully the single and double +quotes. + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' "https://gitlab.example.com/api/v4/groups" +``` + +### Post data using form-data + +Instead of using JSON or urlencode you can use multipart/form-data which +properly handles data encoding: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "title=ssh-key" --form "key=ssh-rsa AAAAB3NzaC1yc2EA..." "https://gitlab.example.com/api/v4/users/25/keys" +``` + +The above example is run by and administrator and will add an SSH public key +titled `ssh-key` to user's account which has an ID of 25. + +### Escape special characters + +Spaces or slashes (`/`) may sometimes result to errors, thus it is recommended +to escape them when possible. In the example below we create a new issue which +contains spaces in its title. Observe how spaces are escaped using the `%20` +ASCII code. + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/issues?title=Hello%20Dude" +``` + +Use `%2F` for slashes (`/`). + +### Pass arrays to API calls + +The GitLab API sometimes accepts arrays of strings or integers. For example, to +exclude specific users when requesting a list of users for a project, you would +do something like this: + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "skip_users[]=<user_id>" --data "skip_users[]=<user_id>" "https://gitlab.example.com/api/v4/projects/<project_id>/users" +``` diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 22d97a9e2cf..9fce9b4e4b3 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -324,7 +324,6 @@ There are three main considerations on the logic built for the nav: - `https://docs.gitlab.com/ee/` - `https://docs.gitlab.com/omnibus/` - `https://docs.gitlab.com/runner/` - - `https://docs.gitlab.com/debug/` - `https://docs.gitlab.com/*` - [EE-only](#ee-only-docs): documentation only available in `/ee/`, not on `/ce/`, e.g.: - `https://docs.gitlab.com/ee/user/group/epics/` @@ -342,8 +341,8 @@ all the nav links to other pages: <% dir = @item.identifier.to_s[%r{(?<=/)[^/]+}] %> ``` -For instance, for `https://docs.gitlab.com/ce/user/index.html`, -`dir` == `ce`, and for `https://docs.gitlab.com/omnibus/README.html`, +For instance, for `https://docs.gitlab.com/ee/user/index.html`, +`dir` == `ee`, and for `https://docs.gitlab.com/omnibus/README.html`, `dir` == `omnibus`. #### Default URL diff --git a/doc/development/documentation/site_architecture/release_process.md b/doc/development/documentation/site_architecture/release_process.md index 98bb116aba6..d04d34ff786 100644 --- a/doc/development/documentation/site_architecture/release_process.md +++ b/doc/development/documentation/site_architecture/release_process.md @@ -121,10 +121,11 @@ versions (stable branches `X.Y` of the `gitlab-docs` project): pipelines succeed: NOTE: **Note:** - The `release-X-Y` branch needs to be present locally, otherwise the Rake - task will abort. + The `release-X-Y` branch needs to be present locally, + and you need to have switched to it, otherwise the Rake task will fail. ```shell + git checkout release-X-Y ./bin/rake release:dropdowns ``` diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index 984c64b9e9e..6075124ef40 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -251,7 +251,7 @@ The table below shows what kind of documentation goes where. | `doc/legal/` | Legal documents about contributing to GitLab. | | `doc/install/` | Contains instructions for installing GitLab. | | `doc/update/` | Contains instructions for updating GitLab. | -| `doc/topics/` | Indexes per topic (`doc/topics/topic-name/index.md`): all resources for that topic. | +| `doc/topics/` | Indexes per topic (`doc/topics/topic_name/index.md`): all resources for that topic. | ### Work with directories and files @@ -287,9 +287,8 @@ The table below shows what kind of documentation goes where. 1. The `doc/topics/` directory holds topic-related technical content. Create `doc/topics/topic_name/subtopic_name/index.md` when subtopics become necessary. General user- and admin- related documentation, should be placed accordingly. -1. The directories `/workflow/`, `/university/`, and `/articles/` have been - *deprecated* and the majority their documentation has been moved to their - correct location in small iterations. +1. The `/university/` directory is *deprecated* and the majority of its documentation + has been moved. If you are unsure where to place a document or a content addition, this should not stop you from authoring and contributing. You can use your best judgment and @@ -369,7 +368,7 @@ create an issue or an MR to propose a change to the user interface text. - milestones - reorder issues - runner, runners, shared runners - - a to-do, to-dos + - a to-do item, to dos - *Some features are capitalized*, typically nouns naming GitLab-specific capabilities or tools. For example: - GitLab CI/CD @@ -410,7 +409,7 @@ Use forms of *sign in*, instead of *log in* or *login*. For example: Exceptions to this rule include the concept of *single sign-on* and references to user interface elements. For example: -- To sign in to product X, enter your credentials, and then click **Log in**. +- To sign in to product X, enter your credentials, and then select **Log in**. ### Inclusive language @@ -418,8 +417,11 @@ We strive to create documentation that is inclusive. This section includes guidance and examples in the following categories: - [Gender-specific wording](#avoid-gender-specific-wording). + (Tested in [`InclusionGender.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionGender.yml).) - [Ableist language](#avoid-ableist-language). + (Tested in [`InclusionAbleism.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionAbleism.yml).) - [Cultural sensitivity](#culturally-sensitive-language). + (Tested in [`InclusionCultural.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/.vale/gitlab/InclusionCultural.yml).) We write our developer documentation with inclusivity and diversity in mind. This page is not an exhaustive reference, but describes some general guidelines and @@ -433,12 +435,16 @@ a gender-neutral pronoun. Avoid the use of gender-specific pronouns, unless referring to a specific person. +<!-- vale gitlab.InclusionGender = NO --> + | Use | Avoid | |-----------------------------------|---------------------------------| | People, humanity | Mankind | | GitLab Team Members | Manpower | | You can install; They can install | He can install; She can install | +<!-- vale gitlab.InclusionGender = YES --> + If you need to set up [Fake user information](#fake-user-information), use diverse or non-gendered names with common surnames. @@ -446,6 +452,8 @@ diverse or non-gendered names with common surnames. Avoid terms that are also used in negative stereotypes for different groups. +<!-- vale gitlab.InclusionAbleism = NO --> + | Use | Avoid | |------------------------|----------------------| | Check for completeness | Sanity check | @@ -455,6 +463,8 @@ Avoid terms that are also used in negative stereotypes for different groups. | Active/Inactive | Enabled/Disabled | | On/Off | Enabled/Disabled | +<!-- vale gitlab.InclusionAbleism = YES --> + Credit: [Avoid ableist language](https://developers.google.com/style/inclusive-documentation#ableist-language) in the Google Developer Style Guide. @@ -464,13 +474,57 @@ Avoid terms that reflect negative cultural stereotypes and history. In most cases, you can replace terms such as `master` and `slave` with terms that are more precise and functional, such as `primary` and `secondary`. +<!-- vale gitlab.InclusionCultural = NO --> + | Use | Avoid | |----------------------|-----------------------| | Primary / secondary | Master / slave | | Allowlist / denylist | Blacklist / whitelist | +<!-- vale gitlab.InclusionCultural = YES --> + For more information see the following [Internet Draft specification](https://tools.ietf.org/html/draft-knodel-terminology-02). +### Fake user information + +You may need to include user information in entries such as a REST call or user profile. +**Do not** use real user information or email addresses in GitLab documentation. For email +addresses and names, do use: + +- **Email addresses**: Use an email address ending in `example.com`. +- **Names**: Use strings like `example_username`. Alternatively, use diverse or + non-gendered names with common surnames, such as `Sidney Jones`, `Zhang Wei`, + or `Alex Garcia`. + +### Fake URLs + +When including sample URLs in the documentation, use: + +- `example.com` when the domain name is generic. +- `gitlab.example.com` when referring to self-managed instances of GitLab. + +### Fake tokens + +There may be times where a token is needed to demonstrate an API call using +cURL or a variable used in CI. It is strongly advised not to use real tokens in +documentation even if the probability of a token being exploited is low. + +You can use the following fake tokens as examples: + +| Token type | Token value | +|:----------------------|:-------------------------------------------------------------------| +| Private user token | `<your_access_token>` | +| Personal access token | `n671WNGecHugsdEDPsyo` | +| Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | +| Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | +| CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` | +| Specific runner token | `yrnZW46BrtBFqM7xDzE7dddd` | +| Shared runner token | `6Vk7ZsosqQyfreAxXTZr` | +| Trigger token | `be20d8dcc028677c931e04f3871a9b` | +| Webhook secret token | `6XhDroRcYPM5by_h-HLY` | +| Health check token | `Tu7BgjR9qeZTEyRzGG2P` | +| Request profile token | `7VgpS4Ax5utVD2esNstz` | + ### Language to avoid When creating documentation, limit or avoid the use of the following verb @@ -529,6 +583,7 @@ tenses, words, and phrases: - Avoid the use of [racially-insensitive terminology or phrases](https://www.marketplace.org/2020/06/17/tech-companies-update-language-to-avoid-offensive-terms/). For example: - Use *primary* and *secondary* for database and server relationships. - Use *allowlist* and *denylist* to describe access control lists. +- Avoid the word *please*. For details, see [the Microsoft style guide](https://docs.microsoft.com/en-us/style-guide/a-z-word-list-term-collections/p/please). ### Word usage clarifications @@ -832,8 +887,10 @@ When creating tables of lists of features (such as whether or not features are available to certain roles on the [Permissions](../../user/permissions.md#project-members-permissions) page), use the following phrases (based on the SVG icons): -- *No*: **{dotted-circle}** No -- *Yes*: **{check-circle}** Yes +| Option | Markdown | Displayed result | +|--------|--------------------------|------------------------| +| No | `**{dotted-circle}** No` | **{dotted-circle}** No | +| Yes | `**{check-circle}** Yes` | **{check-circle}** Yes | ## Quotes @@ -890,8 +947,8 @@ search engine optimization (SEO), use the imperative, where possible. For guidelines on capitalizing headings, see the section on [capitalization](#capitalization). NOTE: **Note:** -If you change an existing title, be careful. Any such changes may affect not -only [links](#anchor-links) within the page, but may also affect links to the +If you change an existing title, be careful. These changes might affect not +only [links](#anchor-links) within the page, but might also affect links to the GitLab documentation from both the GitLab application and external sites. ### Anchor links @@ -1095,14 +1152,26 @@ document to ensure it links to the most recent version of the file. ## Navigation -To indicate the steps of navigation through the user interface: +When documenting navigation through the user interface: -- Use the exact word as shown in the UI, including any capital letters as-is. +- Use the exact wording as shown in the UI, including any capital letters as-is. - Use bold text for navigation items and the char "greater than" (`>`) as a - separator (for example, `Navigate to your project's **Settings > CI/CD**` ). + separator. For example: `Navigate to your project's **Settings > CI/CD**`. - If there are any expandable menus, make sure to mention that the user needs to - expand the tab to find the settings you're referring to (for example, - `Navigate to your project's **Settings > CI/CD** and expand **General pipelines**`). + expand the tab to find the settings you're referring to. For example: + `Navigate to your project's **Settings > CI/CD** and expand **General pipelines**`. + +### Navigational elements + +Use the following terms when referring to the main GitLab user interface +elements: + +- **Top menu**: This is the top menu that spans the width of the user interface. + It includes the GitLab logo, search field, counters, and the user's avatar. +- **Left sidebar**: This is the navigation sidebar on the left of the user + interface, specific to the project or group. +- **Right sidebar**: This is the navigation sidebar on the right of the user + interface, specific to the open issue, merge request, or epic. ## Images @@ -1162,9 +1231,6 @@ that: - Are accurate, succinct, and unique. - Don't use *image of …* or *graphic of…* to describe the image. -Also, if a heading immediately follows an image, be sure to add three dashes -(`---`) between the image and the heading. - ### Remove image shadow All images displayed on the [GitLab documentation site](https://docs.gitlab.com) @@ -1249,7 +1315,7 @@ reviewed and approved by a technical writer. 1. In YouTube, visit the video URL you want to display. Copy the regular URL from your browser (`https://www.youtube.com/watch?v=VIDEO-ID`) and replace the video title and link in the line under `<div class="video-fallback">`. -1. In YouTube, click **Share**, and then click **Embed**. +1. In YouTube, select **Share**, and then select **Embed**. 1. Copy the `<iframe>` source (`src`) **URL only** (`https://www.youtube.com/embed/VIDEO-ID`), and paste it, replacing the content of the `src` field in the @@ -1292,6 +1358,9 @@ the documentation site, but will be displayed on GitLab's `/help`. added to code blocks. To make things easier for the user, always add a full code block for things that can be useful to copy and paste, as they can easily do it with the button on code blocks. +- HTTP methods (`HTTP POST`) and HTTP status codes, both full (`404 File Not Found`) + and abbreviated (`404`), should be wrapped in inline code blocks when used in sentences. + For example: Send a `DELETE` request to delete the runner. Send a `POST` request to create one. - Add a blank line above and below code blocks. - When providing a shell command and its output, prefix the shell command with `$` and leave a blank line between the command and the output. @@ -1416,17 +1485,17 @@ interface: Use an icon when you find yourself having to describe an interface element. For example: -- Do: Click the Admin Area icon ( **{admin}** ). -- Don't: Click the Admin Area icon (the wrench icon). +- Do: Select the Admin Area icon ( **{admin}** ). +- Don't: Select the Admin Area icon (the wrench icon). ## Alert boxes When you need to call special attention to particular sentences, use the following markup to create highlighted alert boxes. -Note that the alert boxes only work for one paragraph only. Multiple paragraphs, -lists, headers and so on, will not render correctly. For multiple lines, use -[blockquotes](#blockquotes) instead. +Alert boxes work for one paragraph only. Multiple paragraphs, lists, and headers +won't render correctly. For multiple lines, use [blockquotes](#blockquotes) +instead. Alert boxes render only on the GitLab documentation site (<https://docs.gitlab.com>). Within GitLab itself, they will appear as plain Markdown text (like the examples @@ -1444,23 +1513,20 @@ guidelines, but for consistency you should try to use these values: ### Note -Notes catch the eye of most readers, and therefore should be used very sparingly. -In most cases, content considered for a note should be included: +Notes indicate additional information that is of special use to the reader. +Notes are most effective when used _sparingly_. -- As just another sentence in the previous paragraph or the most-relevant - paragraph. -- As its own standalone paragraph. -- As content under a new subheading that introduces the topic, making it more - visible or findable. +Try to avoid them. Too many notes can impact the scannability of a topic and +create an overly busy page. -#### When to use +Instead of adding a note, try one of these alternatives: -Use a note when there is a reason that most or all readers who browse the -section should see the content. That is, if missed, it’s likely to cause major -trouble for a minority of users or significant trouble for a majority of users. +- Re-write the sentence as part of the most-relevant paragraph. +- Put the information into its own standalone paragraph. +- Put the content under a new subheading that introduces the topic, which makes + it more visible. -Weigh the costs of distracting users to whom the content is not relevant against -the cost of users missing the content if it were not expressed as a note. +If you must use a note, use the following formatting: ```markdown NOTE: **Note:** @@ -1582,12 +1648,11 @@ application: The following are recommended verbs for specific uses with user interface elements: -| Recommended | Used for | Replaces | -|:--------------------|:---------------------------|:---------------------------| -| *click* | buttons, links, menu items | "hit", "press", "select" | -| *select* or *clear* | checkboxes | "enable", "click", "press" | -| *select* | dropdowns | "pick" | -| *expand* | expandable sections | "open" | +| Recommended | Used for | Replaces | +|:--------------------|:--------------------------------------|:---------------------------| +| *select* | buttons, links, menu items, dropdowns | "click, "press," "hit" | +| *select* or *clear* | checkboxes | "enable", "click", "press" | +| *expand* | expandable sections | "open" | ### Other Verbs @@ -1614,6 +1679,15 @@ heading level. ### Text for documentation requiring version text +When a feature is new or updated, you can add version information as a bulleted +item in the **Version history**, or as an inline reference with related text. + +#### Version text in the **Version History** + +If all content in a section is related, add version text in a bulleted list +following the heading for the section. To render correctly, it must be on its +own line and surrounded by blank lines. + - For features that need to declare the GitLab version that the feature was introduced. Text similar to the following should be added immediately below the heading as a blockquote: @@ -1663,9 +1737,20 @@ heading level. and replaced by [Feature name](link-to-feature-documentation). ``` -NOTE: **Note:** -Version text must be on its own line and surrounded by blank lines to render -correctly. +#### Inline version text + +If you're adding content to an existing topic, you can add version information +inline with the existing text. + +In this case, add `([introduced/deprecated](<link-to-issue>) in GitLab X.X)`. +If applicable, include the paid tier: `([introduced/deprecated](<link-to-issue>) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4)` + +Including the issue link is encouraged, but isn't a requirement. For example: + +```markdown +The voting strategy (introduced in GitLab 13.4) requires +the primary and secondary voters to agree. +``` ### Versions in the past or future @@ -1807,7 +1892,7 @@ for the changes to take effect. If the document you are editing resides in a place other than the GitLab CE/EE `doc/` directory, instead of the relative link, use the full path: -`https://docs.gitlab.com/ce/administration/restart_gitlab.html`. Replace +`https://docs.gitlab.com/ee/administration/restart_gitlab.html`. Replace `reconfigure` with `restart` where appropriate. ### Installation guide @@ -1894,216 +1979,9 @@ steps aren't required, consider setting up a [table](#tables) with headers of Learn how to [document features deployed behind flags](feature_flags.md). For guidance on developing GitLab with feature flags, see [Feature flags in development of GitLab](../feature_flags/index.md). -## RESTful API - -REST API resources are documented in Markdown under -[`/doc/api`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api). Each -resource has its own Markdown file, which is linked from `api_resources.md`. - -When modifying the Markdown, also update the corresponding -[OpenAPI definition](https://gitlab.com/gitlab-org/gitlab/-/tree/master/doc/api/openapi) -if one exists for the resource. If not, consider creating one. Match the latest -[OpenAPI 3.0.x specification](https://swagger.io/specification/). (For more -information, see the discussion in this -[issue](https://gitlab.com/gitlab-org/gitlab/-/issues/16023#note_370901810).) - -In the Markdown doc for a resource (AKA endpoint): - -- Every method must have the REST API request. For example: - - ```plaintext - GET /projects/:id/repository/branches - ``` - -- Every method must have a detailed [description of the parameters](#method-description). -- Every method must have a cURL example. -- Every method must have a response body (in JSON format). - -### API topic template - -The following can be used as a template to get started: - -````markdown -## Descriptive title - -One or two sentence description of what endpoint does. - -```plaintext -METHOD /endpoint -``` - -| Attribute | Type | Required | Description | -|:------------|:---------|:---------|:----------------------| -| `attribute` | datatype | yes/no | Detailed description. | -| `attribute` | datatype | yes/no | Detailed description. | - -Example request: - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/endpoint?parameters" -``` - -Example response: - -```json -[ - { - } -] -``` -```` - -### Fake user information - -You may need to demonstrate an API call or a cURL command that includes the name -and email address of a user. Don't use real user information in API calls: - -- **Email addresses**: Use an email address ending in `example.com`. -- **Names**: Use strings like `Example Username`. Alternatively, use diverse or - non-gendered names with common surnames, such as `Sidney Jones`, `Zhang Wei`, - or `Maria Garcia`. - -### Fake URLs - -When including sample URLs in the documentation, use: - -- `example.com` when the domain name is generic. -- `gitlab.example.com` when referring to self-managed instances of GitLab. - -### Fake tokens - -There may be times where a token is needed to demonstrate an API call using -cURL or a variable used in CI. It is strongly advised not to use real tokens in -documentation even if the probability of a token being exploited is low. - -You can use the following fake tokens as examples: - -| Token type | Token value | -|:----------------------|:-------------------------------------------------------------------| -| Private user token | `<your_access_token>` | -| Personal access token | `n671WNGecHugsdEDPsyo` | -| Application ID | `2fcb195768c39e9a94cec2c2e32c59c0aad7a3365c10892e8116b5d83d4096b6` | -| Application secret | `04f294d1eaca42b8692017b426d53bbc8fe75f827734f0260710b83a556082df` | -| CI/CD variable | `Li8j-mLUVA3eZYjPfd_H` | -| Specific runner token | `yrnZW46BrtBFqM7xDzE7dddd` | -| Shared runner token | `6Vk7ZsosqQyfreAxXTZr` | -| Trigger token | `be20d8dcc028677c931e04f3871a9b` | -| Webhook secret token | `6XhDroRcYPM5by_h-HLY` | -| Health check token | `Tu7BgjR9qeZTEyRzGG2P` | -| Request profile token | `7VgpS4Ax5utVD2esNstz` | - -### Method description - -Use the following table headers to describe the methods. Attributes should -always be in code blocks using backticks (`` ` ``). - -```markdown -| Attribute | Type | Required | Description | -|:----------|:-----|:---------|:------------| -``` - -Rendered example: - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:--------------------| -| `user` | string | yes | The GitLab username | - -### cURL commands - -- Use `https://gitlab.example.com/api/v4/` as an endpoint. -- Wherever needed use this personal access token: `<your_access_token>`. -- Always put the request first. `GET` is the default so you don't have to - include it. -- Wrap the URL in double quotes (`"`). -- Prefer to use examples using the personal access token and don't pass data of - username and password. - -| Methods | Description | -|:------------------------------------------- |:------------------------------------------------------| -| `--header "PRIVATE-TOKEN: <your_access_token>"` | Use this method as is, whenever authentication needed | -| `--request POST` | Use this method when creating new objects | -| `--request PUT` | Use this method when updating existing objects | -| `--request DELETE` | Use this method when removing existing objects | - -### cURL Examples - -The following sections include a set of [cURL](https://curl.haxx.se) examples -you can use in the API documentation. - -#### Simple cURL command - -Get the details of a group: - -```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/gitlab-org" -``` - -#### cURL example with parameters passed in the URL - -Create a new project under the authenticated user's namespace: - -```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects?name=foo" -``` - -#### Post data using cURL's `--data` - -Instead of using `--request POST` and appending the parameters to the URI, you -can use cURL's `--data` option. The example below will create a new project -`foo` under the authenticated user's namespace. - -```shell -curl --data "name=foo" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects" -``` - -#### Post data using JSON content - -NOTE: **Note:** -In this example we create a new group. Watch carefully the single and double -quotes. - -```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" --data '{"path": "my-group", "name": "My group"}' "https://gitlab.example.com/api/v4/groups" -``` - -#### Post data using form-data - -Instead of using JSON or urlencode you can use multipart/form-data which -properly handles data encoding: - -```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --form "title=ssh-key" --form "key=ssh-rsa AAAAB3NzaC1yc2EA..." "https://gitlab.example.com/api/v4/users/25/keys" -``` - -The above example is run by and administrator and will add an SSH public key -titled `ssh-key` to user's account which has an ID of 25. - -#### Escape special characters - -Spaces or slashes (`/`) may sometimes result to errors, thus it is recommended -to escape them when possible. In the example below we create a new issue which -contains spaces in its title. Observe how spaces are escaped using the `%20` -ASCII code. - -```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/42/issues?title=Hello%20Dude" -``` - -Use `%2F` for slashes (`/`). - -#### Pass arrays to API calls - -The GitLab API sometimes accepts arrays of strings or integers. For example, to -exclude specific users when requesting a list of users for a project, you would -do something like this: - -```shell -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" --data "skip_users[]=<user_id>" --data "skip_users[]=<user_id>" "https://gitlab.example.com/api/v4/projects/<project_id>/users" -``` - ## GraphQL API -GraphQL APIs are different from [RESTful APIs](#restful-api). Reference +GraphQL APIs are different from [RESTful APIs](restful_api_styleguide.md). Reference information is generated in our [GraphQL reference](../../api/graphql/reference/index.md). However, it's helpful to include examples on how to use GraphQL for different @@ -2158,7 +2036,7 @@ Set up the section with the following: ```markdown 1. Open the GraphiQL explorer tool in the following URL: `https://gitlab.com/-/graphql-explorer`. 1. Paste the `query` listed above into the left window of your GraphiQL explorer tool. - 1. Click Play to get the result shown here: + 1. Select Play to get the result shown here: ``` - Include a screenshot of the result in the GraphiQL explorer. Follow the naming diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index e70cf456101..639759e5014 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w This area is to maintain a compendium of useful information when working with Elasticsearch. Information on how to enable Elasticsearch and perform the initial indexing is in -the [Elasticsearch integration documentation](../integration/elasticsearch.md#enabling-elasticsearch). +the [Elasticsearch integration documentation](../integration/elasticsearch.md#enabling-advanced-search). ## Deep Dive diff --git a/doc/development/emails.md b/doc/development/emails.md index cf7f49ee834..de9607fef64 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -27,7 +27,7 @@ Please note that [S/MIME signed](../administration/smime_signing_email.md) email ## Mailer previews Rails provides a way to preview our mailer templates in HTML and plaintext using -dummy data. +sample data. The previews live in [`app/mailers/previews`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/app/mailers/previews) and can be viewed at [`/rails/mailers`](http://localhost:3000/rails/mailers). diff --git a/doc/development/event_tracking/backend.md b/doc/development/event_tracking/backend.md index 93e135772ef..79ea80a52ea 100644 --- a/doc/development/event_tracking/backend.md +++ b/doc/development/event_tracking/backend.md @@ -1,5 +1,5 @@ --- -redirect_to: '../telemetry/index.md' +redirect_to: '../product_analytics/index.md' --- -This document was moved to [another location](../telemetry/index.md). +This document was moved to [another location](../product_analytics/index.md). diff --git a/doc/development/event_tracking/frontend.md b/doc/development/event_tracking/frontend.md index 93e135772ef..79ea80a52ea 100644 --- a/doc/development/event_tracking/frontend.md +++ b/doc/development/event_tracking/frontend.md @@ -1,5 +1,5 @@ --- -redirect_to: '../telemetry/index.md' +redirect_to: '../product_analytics/index.md' --- -This document was moved to [another location](../telemetry/index.md). +This document was moved to [another location](../product_analytics/index.md). diff --git a/doc/development/event_tracking/index.md b/doc/development/event_tracking/index.md index 93e135772ef..79ea80a52ea 100644 --- a/doc/development/event_tracking/index.md +++ b/doc/development/event_tracking/index.md @@ -1,5 +1,5 @@ --- -redirect_to: '../telemetry/index.md' +redirect_to: '../product_analytics/index.md' --- -This document was moved to [another location](../telemetry/index.md). +This document was moved to [another location](../product_analytics/index.md). diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 07b803603a5..18b606450c2 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -16,46 +16,97 @@ In either case, an outcome of the experiment should be posted to the issue with ## Code reviews -Since the code of experiments will not be part of the codebase for a long time and we want to iterate fast to retrieve data, the code quality of experiments might sometimes not fulfill our standards but should not negatively impact the availability of GitLab whether the experiment is running or not. -Initially experiments will only be deployed to a fraction of users but we still want a flawless experience for those users. Therefore, experiments still require tests. - -For reviewers and maintainers: if you find code that would usually not make it through the review, but is temporarily acceptable, please mention your concerns but note that it's not necessary to change. -The author then adds a comment to this piece of code and adds a link to the issue that resolves the experiment. If the experiment is successful and becomes part of the product these follow up issues should be addressed. +Experiments' code quality can fail our standards for several reasons. These +reasons can include not being added to the codebase for a long time, or because +of fast iteration to retrieve data. However, having the experiment run (or not +run) shouldn't impact GitLab's availability. To avoid or identify issues, +experiments are initially deployed to a small number of users. Regardless, +experiments still need tests. + +If, as a reviewer or maintainer, you find code that would usually fail review +but is acceptable for now, mention your concerns with a note that there's no +need to change the code. The author can then add a comment to this piece of code +and link to the issue that resolves the experiment. If the experiment is +successful and becomes part of the product, any follow up issues should be +addressed. ## How to create an A/B test -- Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb): - - ```ruby - EXPERIMENTS = { - other_experiment: { - #... - }, - # Add your experiment here: - signup_flow: { - environment: ::Gitlab.dev_env_or_com?, # Target environment, defaults to enabled for development and GitLab.com - tracking_category: 'Growth::Acquisition::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data - } - }.freeze - ``` - -- Use the experiment in a controller: - - ```ruby - class RegistrationController < ApplicationController - def show - # experiment_enabled?(:feature_name) is also available in views and helpers - if experiment_enabled?(:signup_flow) - # render the experiment - else - # render the original version - end - end - end - ``` - -- Track necessary events. See the [telemetry guide](../telemetry/index.md) for details. -- After the merge request is merged, use [`chatops`](../../ci/chatops/README.md) in the +### Implementation + +1. Add the experiment to the `Gitlab::Experimentation::EXPERIMENTS` hash in [`experimentation.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib%2Fgitlab%2Fexperimentation.rb): + + ```ruby + EXPERIMENTS = { + other_experiment: { + #... + }, + # Add your experiment here: + signup_flow: { + environment: ::Gitlab.dev_env_or_com?, # Target environment, defaults to enabled for development and GitLab.com + tracking_category: 'Growth::Acquisition::Experiment::SignUpFlow' # Used for providing the category when setting up tracking data + } + }.freeze + ``` + +1. Use the experiment in the code. + + - Use this standard for the experiment in a controller: + + ```ruby + class RegistrationController < ApplicationController + def show + # experiment_enabled?(:experiment_key) is also available in views and helpers + if experiment_enabled?(:signup_flow) + # render the experiment + else + # render the original version + end + end + end + ``` + + - Make the experiment available to the frontend in a controller: + + ```ruby + before_action do + push_frontend_experiment(:signup_flow) + end + ``` + + The above will check whether the experiment is enabled and push the result to the frontend. + + You can check the state of the feature flag in JavaScript: + + ```javascript + import { isExperimentEnabled } from '~/experimentation'; + + if ( isExperimentEnabled('signupFlow') ) { + // ... + } + ``` + + - It is also possible to run an experiment outside of the controller scope, for example in a worker: + + ```ruby + class SomeWorker + def perform + # Check if the experiment is enabled at all (the percentage_of_time_value > 0) + return unless Gitlab::Experimentation.enabled?(:experiment_key) + + # Since we cannot access cookies in a worker, we need to bucket models based on a unique, unchanging attribute instead. + # Use the following method to check if the experiment is enabled for a certain attribute, for example a username or email address: + if Gitlab::Experimentation.enabled_for_attribute?(:experiment_key, some_attribute) + # execute experimental code + else + # execute control code + end + end + end + ``` + +1. Track necessary events. See the [product analytics guide](../product_analytics/index.md) for details. +1. After the merge request is merged, use [`chatops`](../../ci/chatops/README.md) in the [appropriate channel](../feature_flags/controls.md#communicate-the-change) to start the experiment for 10% of the users. The feature flag should have the name of the experiment with the `_experiment_percentage` suffix appended. For visibility, please also share any commands run against production in the `#s_growth` channel: @@ -69,3 +120,19 @@ For visibility, please also share any commands run against production in the `#s ```shell /chatops run feature delete signup_flow_experiment_percentage ``` + +### Tests and test helpers + +Use the following in Jest to test the experiment is enabled. + +```javascript +import { withGonExperiment } from 'helpers/experimentation_helper'; + +describe('given experiment is enabled', () => { + withGonExperiment('signupFlow'); + + it('should do the experimental thing', () => { + expect(wrapper.find('.js-some-experiment-triggered-element')).toEqual(expect.any(Element)); + }); +}); +``` diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index 3d27f67a8a6..c7e1ba59f23 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -15,5 +15,5 @@ You can find the Frontend Architecture experts on the [team page](https://about. ## Examples -You can find documentation about the desired architecture for a new feature -built with Vue.js [here](vue.md). +You can find [documentation about the desired architecture](vue.md) for a new +feature built with Vue.js. diff --git a/doc/development/fe_guide/dependencies.md b/doc/development/fe_guide/dependencies.md index 7f078df887d..ba97e7e39be 100644 --- a/doc/development/fe_guide/dependencies.md +++ b/doc/development/fe_guide/dependencies.md @@ -18,9 +18,9 @@ automatically create merge requests for updating dependencies of several project up-to-date list of projects managed by the renovate bot in the project’s README. Some key dependencies updated using renovate are: -- [`@gitlab/ui`](https://gitlab.com/gitlab-org/gitlab-ui/) -- [`@gitlab/svgs`](https://gitlab.com/gitlab-org/gitlab-svgs/) -- [`@gitlab/eslint-config`](https://gitlab.com/gitlab-org/gitlab-eslint-config) +- [`@gitlab/ui`](https://gitlab.com/gitlab-org/gitlab-ui) +- [`@gitlab/svgs`](https://gitlab.com/gitlab-org/gitlab-svgs) +- [`@gitlab/eslint-plugin`](https://gitlab.com/gitlab-org/frontend/eslint-plugin) ### Blocked dependencies diff --git a/doc/development/fe_guide/event_tracking.md b/doc/development/fe_guide/event_tracking.md index 93e135772ef..79ea80a52ea 100644 --- a/doc/development/fe_guide/event_tracking.md +++ b/doc/development/fe_guide/event_tracking.md @@ -1,5 +1,5 @@ --- -redirect_to: '../telemetry/index.md' +redirect_to: '../product_analytics/index.md' --- -This document was moved to [another location](../telemetry/index.md). +This document was moved to [another location](../product_analytics/index.md). diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 82cd19dce4f..ad3958d4496 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -187,7 +187,7 @@ As shown in the code example by using `produce`, we can perform any kind of dire `draftState`. Besides, `immer` guarantees that a new state which includes the changes to `draftState` will be generated. Finally, to verify whether the immutable cache update is working properly, we need to change -`assumeImmutableResults` to `true` in the `default client config` (see [Apollo Client](#apollo-client) for more info). +`assumeImmutableResults` to `true` in the `default client config` (see [Apollo Client](#apollo-client) for more info). If everything is working properly `assumeImmutableResults` should remain set to `true`. @@ -308,7 +308,7 @@ const resolvers = { export default resolvers; ``` -We need to pass resolvers object to our existing Apollo Client: +We need to pass a resolvers object to our existing Apollo Client: ```javascript // graphql.js @@ -319,13 +319,13 @@ import resolvers from './graphql/resolvers'; const defaultClient = createDefaultClient(resolvers); ``` -Now every single time on attempt to fetch a version, our client will fetch `id` and `sha` from the remote API endpoint and will assign our hardcoded values to `author` and `createdAt` version properties. With this data, frontend developers are able to work on UI part without being blocked by backend. When actual response is added to the API, a custom local resolver can be removed fast and the only change to query/fragment is `@client` directive removal. +For each attempt to fetch a version, our client will fetch `id` and `sha` from the remote API endpoint and will assign our hardcoded values to the `author` and `createdAt` version properties. With this data, frontend developers are able to work on their UI without being blocked by backend. When the actual response is added to the API, our custom local resolver can be removed and the only change to the query/fragment is to remove the `@client` directive. Read more about local state management with Apollo in the [Vue Apollo documentation](https://vue-apollo.netlify.app/guide/local-state.html#local-state). ### Using with Vuex -When Apollo Client is used within Vuex and fetched data is stored in the Vuex store, there is no need in keeping Apollo Client cache enabled. Otherwise we would have data from the API stored in two places - Vuex store and Apollo Client cache. More to say, with Apollo's default settings, a subsequent fetch from the GraphQL API could result in fetching data from Apollo cache (in the case where we have the same query and variables). To prevent this behavior, we need to disable Apollo Client cache passing a valid `fetchPolicy` option to its constructor: +When Apollo Client is used within Vuex and fetched data is stored in the Vuex store, there is no need to keep Apollo Client cache enabled. Otherwise we would have data from the API stored in two places - Vuex store and Apollo Client cache. With Apollo's default settings, a subsequent fetch from the GraphQL API could result in fetching data from Apollo cache (in the case where we have the same query and variables). To prevent this behavior, we need to disable Apollo Client cache by passing a valid `fetchPolicy` option to its constructor: ```javascript import fetchPolicies from '~/graphql_shared/fetch_policy_constants'; diff --git a/doc/development/fe_guide/icons.md b/doc/development/fe_guide/icons.md index b539293e9cf..67add5709d9 100644 --- a/doc/development/fe_guide/icons.md +++ b/doc/development/fe_guide/icons.md @@ -1,9 +1,10 @@ # Icons and SVG Illustrations -We manage our own Icon and Illustration library in the [`gitlab-svgs`](https://gitlab.com/gitlab-org/gitlab-svgs) repository. -This repository is published on [npm](https://www.npmjs.com/package/@gitlab/svgs) and managed as a dependency via yarn. -You can browse all available Icons and Illustrations [here](https://gitlab-org.gitlab.io/gitlab-svgs). -To upgrade to a new version run `yarn upgrade @gitlab/svgs`. +We manage our own icon and illustration library in the [`gitlab-svgs`](https://gitlab.com/gitlab-org/gitlab-svgs) +repository. This repository is published on [npm](https://www.npmjs.com/package/@gitlab/svgs), +and is managed as a dependency with yarn. You can browse all available +[icons and illustrations](https://gitlab-org.gitlab.io/gitlab-svgs). To upgrade +to a new version run `yarn upgrade @gitlab/svgs`. ## Icons @@ -21,10 +22,11 @@ To use a sprite Icon in HAML or Rails we use a specific helper function : sprite_icon(icon_name, size: nil, css_class: '') ``` -- **icon_name** Use the icon_name that you can find in the SVG Sprite - ([Overview is available here](https://gitlab-org.gitlab.io/gitlab-svgs)). -- **size (optional)** Use one of the following sizes : 16, 24, 32, 48, 72 (this will be translated into a `s16` class) -- **css_class (optional)** If you want to add additional CSS classes +- **icon_name**: Use the icon_name for the SVG sprite in the list of + ([GitLab SVGs](https://gitlab-org.gitlab.io/gitlab-svgs)). +- **size (optional)**: Use one of the following sizes : 16, 24, 32, 48, 72 (this + will be translated into a `s16` class) +- **css_class (optional)**: If you want to add additional CSS classes. **Example** @@ -66,10 +68,12 @@ export default { </template> ``` -- **name** Name of the Icon in the SVG Sprite ([Overview is available here](https://gitlab-org.gitlab.io/gitlab-svgs)). -- **size (optional)** Number value for the size which is then mapped to a specific CSS class - (Available Sizes: 8, 12, 16, 18, 24, 32, 48, 72 are mapped to `sXX` CSS classes) -- **class (optional)** Additional CSS Classes to add to the SVG tag. +- **name**: Name of the icon of the SVG sprite, as listed in the + ([GitLab SVG Previewer](https://gitlab-org.gitlab.io/gitlab-svgs)). +- **size: (optional)** Number value for the size which is then mapped to a + specific CSS class (Available sizes: 8, 12, 16, 18, 24, 32, 48, 72 are mapped + to `sXX` CSS classes) +- **class (optional)**: Additional CSS classes to add to the SVG tag. ### Usage in HTML/JS diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index ef23b6c4ed2..f909866d44e 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -76,6 +76,10 @@ How we use SVG for our [Icons and Illustrations](icons.md). General information about frontend [dependencies](dependencies.md) and how we manage them. +## Keyboard Shortcuts + +How we implement [keyboard shortcuts](keyboard_shortcuts.md) that can be customized and disabled. + ## Frontend FAQ Read the [frontend's FAQ](frontend_faq.md) for common small pieces of helpful information. diff --git a/doc/development/fe_guide/keyboard_shortcuts.md b/doc/development/fe_guide/keyboard_shortcuts.md new file mode 100644 index 00000000000..da9b3702a8d --- /dev/null +++ b/doc/development/fe_guide/keyboard_shortcuts.md @@ -0,0 +1,98 @@ +# Implementing keyboard shortcuts + +We use [Mousetrap](https://craig.is/killing/mice) to implement keyboard +shortcuts in GitLab. + +Mousetrap provides an API that allows keyboard shortcut strings (like +`mod+shift+p` or `p b`) to be bound to a JavaScript handler: + +```javascript +// Don't do this; see note below +Mousetrap.bind('p b', togglePerformanceBar) +``` + +However, associating a hard-coded key sequence to a handler (as shown above) +prevents these keyboard shortcuts from being customized or disabled by users. + +To allow keyboard shortcuts to be customized, commands are defined in +`~/behaviors/shortcuts/keybindings.js`. The `keysFor` method is responsible for +returning the correct key sequence for the provided command: + +```javascript +import { keysFor, TOGGLE_PERFORMANCE_BAR } from '~/behaviors/shortcuts/keybindings' + +Mousetrap.bind(keysFor(TOGGLE_PERFORMANCE_BAR), togglePerformanceBar); +``` + +## Shortcut customization + +`keybindings.js` stores keyboard shortcut customizations as a JSON string in +`localStorage`. When `keybindings.js` is first imported, it fetches any +customizations from `localStorage` and merges these customizations into the +default set of keybindings. There is no UI to edit these customizations. + +## Adding new shortcuts + +Because keyboard shortcuts can be customized or disabled by end users, +developers are encouraged to build _lots_ of keyboard shortcuts into GitLab. +Shortcuts that are less likely to be used should be +[disabled](#disabling-shortcuts) by default. + +To add a new shortcut, define and export a new command string in +`keybindings.js`: + +```javascript +export const MAKE_COFFEE = 'foodAndBeverage.makeCoffee'; +``` + +Next, add a new command definition under the appropriate group in the +`keybindingGroups` array: + +```javascript +{ + description: s__('KeyboardShortcuts|Make coffee'), + command: MAKE_COFFEE, + defaultKeys: ['mod+shift+c'], + customKeys: customizations[MAKE_COFFEE], +} +``` + +Finally, in the application code, import the `keysFor` function and the new +command and bind the shortcut to the handler using Mousetrap: + +```javascript +import { keysFor, MAKE_COFFEE } from '~/behaviors/shortcuts/keybindings' + +Mousetrap.bind(keysFor(MAKE_COFFEE), makeCoffee); +``` + +See the existing the command definitions in `keybindings.js` for more examples. + +## Disabling shortcuts + +A shortcut can be disabled, also known as _unassigned_, by assigning the +shortcut to an empty array `[]`. For example, to introduce a new shortcut that +is disabled by default, a command can be defined like this: + +```javascript +export const MAKE_MOCHA = 'foodAndBeverage.makeMocha'; + +{ + description: s__('KeyboardShortcuts|Make a mocha'), + command: MAKE_MOCHA, + defaultKeys: [], + customKeys: customizations[MAKE_MOCHA], +} +``` + +## Make cross-platform shortcuts + +It's difficult to make shortcuts that work well in all platforms and browsers. +This is one of the reasons that being able to customize and disable shortcuts is +so important. + +One important way to make keyboard shortcuts more portable is to use the `mod` +shortcut string, which resolves to `command` on Mac and `ctrl` otherwise. + +See [Mousetrap's documentation](https://craig.is/killing/mice#api.bind.combo) +for more information. diff --git a/doc/development/fe_guide/tooling.md b/doc/development/fe_guide/tooling.md index 5685ac5abcd..b93c0a9736b 100644 --- a/doc/development/fe_guide/tooling.md +++ b/doc/development/fe_guide/tooling.md @@ -101,7 +101,10 @@ Our code is automatically formatted with [Prettier](https://prettier.io) to foll ### Editor -The easiest way to include prettier in your workflow is by setting up your preferred editor (all major editors are supported) accordingly. We suggest setting up prettier to run automatically when each file is saved. Find [here](https://prettier.io/docs/en/editors.html) the best way to set it up in your preferred editor. +The recommended method to include Prettier in your workflow is to set up your +preferred editor (all major editors are supported) accordingly. We suggest +setting up Prettier to run when each file is saved. For instructions about using +Prettier in your preferred editor, see the [Prettier documentation](https://prettier.io/docs/en/editors.html). Please take care that you only let Prettier format the same file types as the global Yarn script does (`.js`, `.vue`, `.graphql`, and `.scss`). In VSCode by example you can easily exclude file formats in your settings file: diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 4badf3f0845..528dcb3b7f4 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -32,8 +32,9 @@ When using Vuex at GitLab, separate these concerns into different files to impro └── mutation_types.js # mutation types ``` -The following example shows an application that lists and adds users to the state. -(For a more complex example implementation take a look at the security applications store in [here](https://gitlab.com/gitlab-org/gitlab/tree/master/ee/app/assets/javascripts/vue_shared/security_reports/store)) +The following example shows an application that lists and adds users to the +state. (For a more complex example implementation, review the security +applications stored in this [repository](https://gitlab.com/gitlab-org/gitlab/tree/master/ee/app/assets/javascripts/vue_shared/security_reports/store)). ### `index.js` diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md index 3fd402ebe84..57e0ad8b772 100644 --- a/doc/development/feature_categorization/index.md +++ b/doc/development/feature_categorization/index.md @@ -23,9 +23,9 @@ product categories. When this occurs, you can automatically update and generate a new version of the file, which needs to be committed to the repository. -The [Scalabilitity +The [Scalability team](https://about.gitlab.com/handbook/engineering/infrastructure/team/scalability/) -currently maintains the `stages.yml` file. They will automatically be +currently maintains the `feature_categories.yml` file. They will automatically be notified on Slack when the file becomes outdated. ## Sidekiq workers @@ -75,38 +75,23 @@ A feature category can be specified on an entire controller using: ```ruby -class Projects::MergeRequestsController < ApplicationController - feature_category :source_code_management +class Boards::ListsController < ApplicationController + feature_category :kanban_boards end ``` The feature category can be limited to a list of actions using the -`only` argument, actions can be excluded using the `except` argument. +second argument: ```ruby -class Projects::MergeRequestsController < ApplicationController - feature_category :code_testing, only: [:metrics_reports] - feature_category :source_code_management, except: [:test_reports, :coverage_reports] +class DashboardController < ApplicationController + feature_category :issue_tracking, [:issues, :issues_calendar] + feature_category :code_review, [:merge_requests] end ``` -`except` and `only` arguments can not be combined. - -When specifying `except` all other actions will get the specified -category assigned. - -The assignment can also be scoped using `if` and `unless` procs: - -```ruby -class Projects::MergeRequestsController < ApplicationController - feature_category :source_code_management, - unless: -> (action) { action.include?("reports") } - if: -> (action) { action.include?("widget") } -end -``` - -In this case, both procs need to be satisfied for the action to get -the category assigned. +These forms cannot be mixed: if a controller has more than one category, +every single action must be listed. ### Excluding controller actions from feature categorization @@ -125,6 +110,5 @@ The `spec/controllers/every_controller_spec.rb` will iterate over all defined routes, and check the controller to see if a category is assigned to all actions. -The spec also validates if the used feature categories are known. And -if the actions used in `only` and `except` configuration still exist -as routes. +The spec also validates if the used feature categories are known. And if +the actions used in configuration still exist as routes. diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 605b5919e0b..ef38a85bec0 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -88,15 +88,13 @@ parts of the company. The developer responsible needs to determine whether this is necessary and the appropriate level of communication. This depends on the feature and what sort of impact it might have. -As a guideline: +Guidelines: -- For simple features that are low-risk, and easily rolled back, then - just proceed to [enabling the feature in `#production`](#process). -- For features that will impact user experience consider notifying +1. If the feature meets the requirements for creating a [Change Management](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process) issue, create a Change Management issue per [criticality guidelines](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows). +1. For simple, low-risk, easily reverted features, proceed and [enable the feature in `#production`](#process). +1. For features that impact the user experience, consider notifying + `#support_gitlab-com` first. `#support_gitlab-com` beforehand. -- For features with significant downstream effects (e.g.: turning on/off - Elasticsearch indexing) consider coordinating with `#production` - beforehand. #### Process @@ -250,13 +248,26 @@ Changes to the issue format can be submitted in the ## Cleaning up -Once the change is deemed stable, submit a new merge request to remove the -feature flag. This ensures the change is available to all users and self-managed -instances. Make sure to add the ~"feature flag" label to this merge request so -release managers are aware the changes are hidden behind a feature flag. If the -merge request has to be picked into a stable branch, make sure to also add the -appropriate `~"Pick into X.Y"` label (e.g. `~"Pick into 13.0"`). -See [the process document](process.md#including-a-feature-behind-feature-flag-in-the-final-release) for further details. +A feature flag should be removed as soon as it is no longer needed. Each additional +feature flag in the codebase increases the complexity of the application +and reduces confidence in our testing suite covering all possible combinations. +Additionally, a feature flag overwritten in some of the environments can result +in undefined and untested system behavior. + +To remove a feature flag: + +1. Open a new merge request with the ~"feature flag" label so + release managers are aware the changes are hidden behind a feature flag. +1. If the merge request has to be picked into a stable branch, add the + appropriate `~"Pick into X.Y"` label, for example `~"Pick into 13.0"`. + See [the feature flag process](process.md#including-a-feature-behind-feature-flag-in-the-final-release) + for further details. +1. Remove all references to the feature flag from the codebase. +1. Remove the YAML definition for the feature from the repository. +1. Clean up the feature flag from all environments with `/chatops run feature delete some_feature`. +1. Close the rollout issue for the feature flag after the feature flag is removed from the codebase. + +### Cleanup ChatOps When a feature gate has been removed from the code base, the feature record still exists in the database that the flag was deployed too. diff --git a/doc/development/feature_flags/development.md b/doc/development/feature_flags/development.md index 29bd0ca0a7e..067e480f6f8 100644 --- a/doc/development/feature_flags/development.md +++ b/doc/development/feature_flags/development.md @@ -52,10 +52,10 @@ invocations: ```ruby # Check if feature flag is enabled -Feature.enabled?(:my_ops_flag, project, type: ops) +Feature.enabled?(:my_ops_flag, project, type: :ops) # Check if feature flag is disabled -Feature.disabled?(:my_ops_flag, project, type: ops) +Feature.disabled?(:my_ops_flag, project, type: :ops) # Push feature flag to Frontend push_frontend_feature_flag(:my_ops_flag, project, type: :ops) @@ -153,6 +153,11 @@ default_enabled: false TIP: **Tip:** To create a feature flag that is only used in EE, add the `--ee` flag: `bin/feature-flag --ee` +## Delete a feature flag + +See [cleaning up feature flags](controls.md#cleaning-up) for more information about +deleting feature flags. + ## Develop with a feature flag There are two main ways of using Feature Flags in the GitLab codebase: @@ -440,6 +445,21 @@ Feature.enabled?(:my_feature2) # => false Feature.enabled?(:my_feature2, project1) # => true ``` +### `have_pushed_frontend_feature_flags` + +Use `have_pushed_frontend_feature_flags` to test if [`push_frontend_feature_flag`](#frontend) +has added the feature flag to the HTML. + +For example, + +```ruby +stub_feature_flags(value_stream_analytics_path_navigation: false) + +visit group_analytics_cycle_analytics_path(group) + +expect(page).to have_pushed_frontend_feature_flags(valueStreamAnalyticsPathNavigation: false) +``` + ### `stub_feature_flags` vs `Feature.enable*` It is preferred to use `stub_feature_flags` to enable feature flags @@ -496,6 +516,14 @@ Feature.enabled?(:ci_live_trace) # => false Feature.enabled?(:ci_live_trace, gate) # => true ``` +You can also disable a feature flag for a specific actor: + +```ruby +gate = stub_feature_flag_gate('CustomActor') + +stub_feature_flags(ci_live_trace: false, thing: gate) +``` + ### Controlling feature flags engine in tests Our Flipper engine in the test environment works in a memory mode `Flipper::Adapters::Memory`. diff --git a/doc/development/frontend.md b/doc/development/frontend.md index f46cc377f95..8bb5cf7af62 100644 --- a/doc/development/frontend.md +++ b/doc/development/frontend.md @@ -1,4 +1,5 @@ +--- +redirect_to: 'fe_guide/index.md' +--- -# Frontend Development Guidelines - -This page has moved [here](fe_guide/index.md). +This document was moved to [another location](fe_guide/index.md). diff --git a/doc/development/geo.md b/doc/development/geo.md index 5b4af1c9931..06b032a8f66 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Geo (development) **(PREMIUM ONLY)** Geo connects GitLab instances together. One GitLab instance is @@ -420,7 +426,7 @@ We switch and filter from each event by the `event_name` field. ### Geo Log Cursor (GitLab 10.0 and up) -Since GitLab 10.0, [System Webhooks](#system-hooks-gitlab-87-to-95) are no longer +In GitLab 10.0 and later, [System Webhooks](#system-hooks-gitlab-87-to-95) are no longer used and Geo Log Cursor is used instead. The Log Cursor traverses the `Geo::EventLog` rows to see if there are changes since the last time the log was checked and will handle repository updates, deletes, diff --git a/doc/development/geo/framework.md b/doc/development/geo/framework.md index b720a6ca47e..55f4be07bb4 100644 --- a/doc/development/geo/framework.md +++ b/doc/development/geo/framework.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Geo self-service framework (alpha) NOTE: **Note:** @@ -91,8 +97,6 @@ module Geo ::Packages::PackageFile end - # Change this to `true` to release replication of this model. Then remove - # this override in the next release. # The feature flag follows the format `geo_#{replicable_name}_replication`, # so here it would be `geo_package_file_replication` def self.replication_enabled_by_default? @@ -193,7 +197,9 @@ For example, to add support for files referenced by a `Widget` model with a file_store == ObjectStorage::Store::LOCAL end - def self.replicables_for_geo_node + # @param primary_key_in [Range, Widget] arg to pass to primary_key_in scope + # @return [ActiveRecord::Relation<Widget>] everything that should be synced to this node, restricted by primary key + def self.replicables_for_current_secondary(primary_key_in) # Should be implemented. The idea of the method is to restrict # the set of synced items depending on synchronization settings end @@ -220,8 +226,6 @@ For example, to add support for files referenced by a `Widget` model with a model_record.file end - # Change this to `true` to release replication of this model. Then remove - # this override in the next release. # The feature flag follows the format `geo_#{replicable_name}_replication`, # so here it would be `geo_widget_replication` def self.replication_enabled_by_default? @@ -637,7 +641,7 @@ the Admin Area UI, and Prometheus! include ::Types::Geo::RegistryType graphql_name 'WidgetRegistry' - description 'Represents the sync and verification state of a widget' + description 'Represents the Geo sync and verification state of a widget' field :widget_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Widget' end @@ -676,6 +680,12 @@ the Admin Area UI, and Prometheus! } ``` +1. Update the GraphQL reference documentation: + + ```shell + bundle exec rake gitlab:graphql:compile_docs + ``` + Individual widget synchronization and verification data should now be available via the GraphQL API! @@ -693,3 +703,33 @@ To do: This should be done as part of Widget sync and verification data (aggregate and individual) should now be available in the Admin UI! + +#### Releasing the feature + +1. In `ee/app/replicators/geo/widget_replicator.rb`, delete the `self.replication_enabled_by_default?` method: + + ```ruby + module Geo + class WidgetReplicator < Gitlab::Geo::Replicator + ... + + # REMOVE THIS METHOD + def self.replication_enabled_by_default? + false + end + # REMOVE THIS METHOD + + ... + end + end + ``` + +1. In `ee/app/graphql/types/geo/geo_node_type.rb`, remove the `feature_flag` option for the released type: + + ```ruby + field :widget_registries, ::Types::Geo::WidgetRegistryType.connection_type, + null: true, + resolver: ::Resolvers::Geo::WidgetRegistriesResolver, + description: 'Find widget registries on this Geo node', + feature_flag: :geo_widget_replication # REMOVE THIS LINE + ``` diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 6954b2bd6dc..15d25d2d1ed 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -130,7 +130,7 @@ become available, you will be able to share job templates like this Dependencies should be kept to the minimum. The introduction of a new dependency should be argued in the merge request, as per our [Approval Guidelines](../code_review.md#approval-guidelines). Both [License -Management](../../user/compliance/license_compliance/index.md) +Scanning](../../user/compliance/license_compliance/index.md) **(ULTIMATE)** and [Dependency Scanning](../../user/application_security/dependency_scanning/index.md) **(ULTIMATE)** should be activated on all projects to ensure new dependencies @@ -138,7 +138,7 @@ security status and license compatibility. ### Modules -Since Go 1.11, a standard dependency system is available behind the name [Go +In Go 1.11 and later, a standard dependency system is available behind the name [Go Modules](https://github.com/golang/go/wiki/Modules). It provides a way to define and lock dependencies for reproducible builds. It should be used whenever possible. @@ -472,7 +472,7 @@ In case we want to drop support for `go 1.11` in GitLab `12.10`, we need to veri We will not consider the active milestone, `12.10`, because a backport for `12.7` will be required in case of a critical security release. -1. If both [Omnibus and CNG](#updating-go-version) were using Go `1.12` since GitLab `12.7`, then we safely drop support for `1.11`. +1. If both [Omnibus and CNG](#updating-go-version) were using Go `1.12` in GitLab `12.7` and later, then we safely drop support for `1.11`. 1. If Omnibus or CNG were using `1.11` in GitLab `12.7`, then we still need to keep support for Go `1.11` for easier backporting of security fixes. ## Secure Team standards and style guidelines diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index f7b44e74c17..cc3db267d53 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -3,6 +3,21 @@ The purpose of this guide is to document potential "gotchas" that contributors might encounter or should avoid during development of GitLab CE and EE. +## Do not read files from app/assets directory + +In GitLab 10.8 and later, Omnibus has [dropped the `app/assets` directory](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2456), +after asset compilation. The `ee/app/assets`, `vendor/assets` directories are dropped as well. + +This means that reading files from that directory will fail in Omnibus-installed GitLab instances: + +```ruby +file = Rails.root.join('app/assets/images/logo.svg') + +# This file does not exist, read will fail with: +# Errno::ENOENT: No such file or directory @ rb_sysopen +File.read(file) +``` + ## Do not assert against the absolute value of a sequence-generated attribute Consider the following factory: diff --git a/doc/development/graphql_guide/index.md b/doc/development/graphql_guide/index.md index 62650f7cd3f..9d7fb5ba0a8 100644 --- a/doc/development/graphql_guide/index.md +++ b/doc/development/graphql_guide/index.md @@ -8,6 +8,6 @@ feedback, and suggestions. - [GraphQL API development style guide](../api_graphql_styleguide.md): development style guide for GraphQL. -- [GraphQL API documentation style guide](../documentation/styleguide.md#graphql-api): documentation +- [GraphQL API documentation style guide](../documentation/graphql_styleguide.md): documentation style guide for GraphQL. - [GraphQL API](../../api/graphql/index.md): user documentation for the GitLab GraphQL API. diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index b5c5a199b1e..59399e54c3e 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -205,7 +205,7 @@ For the static text strings we suggest two patterns for using these translations When possible, you should opt for this pattern, as this allows you to import these strings directly into your component specs for re-use during testing. -- Internal component `$options` object `: +- Internal component `$options` object: ```javascript <script> @@ -432,7 +432,7 @@ To avoid this error, use the applicable HTML entity code (`<` or `>`) inst - In JavaScript: ```javascript - import { sanitize } from 'dompurify'; + import { sanitize } from '~/lib/dompurify'; const i18n = { LESS_THAN_ONE_HOUR: sanitize(__('In < 1 hour'), { ALLOWED_TAGS: [] }) }; diff --git a/doc/development/i18n/index.md b/doc/development/i18n/index.md index 929eded3f8e..2d84fe4536f 100644 --- a/doc/development/i18n/index.md +++ b/doc/development/i18n/index.md @@ -38,10 +38,10 @@ Voting for translations is also valuable, helping to confirm good and flag inacc See [Translation guidelines](translation.md). -### Proof reading +### Proofreading -Proof reading helps ensure the accuracy and consistency of translations. All -translations are proof read before being accepted. If a translations requires +Proofreading helps ensure the accuracy and consistency of translations. All +translations are proofread before being accepted. If a translations requires changes, you will be notified with a comment explaining why. See [Proofreading Translations](proofreader.md) for more information on who's diff --git a/doc/development/img/architecture_simplified.png b/doc/development/img/architecture_simplified.png Binary files differindex 46ae2b3c055..72d00b91129 100644 --- a/doc/development/img/architecture_simplified.png +++ b/doc/development/img/architecture_simplified.png diff --git a/doc/development/instrumentation.md b/doc/development/instrumentation.md index e420ae0c54f..bdbcd52eb61 100644 --- a/doc/development/instrumentation.md +++ b/doc/development/instrumentation.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -11,7 +11,7 @@ blocks of Ruby code. Method instrumentation is the primary form of instrumentation with block-based instrumentation only being used when we want to drill down to specific regions of code within a method. -Please refer to [Telemetry](telemetry/index.md) if you are tracking product usage patterns. +Please refer to [Product Analytics](product_analytics/index.md) if you are tracking product usage patterns. ## Instrumenting Methods diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index dcfd0f40bf0..1094074cab6 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -374,12 +374,19 @@ which is shared by the analyzers that GitLab maintains. You can [contribute](htt new generic identifiers to if needed. Analyzers may also produce vendor-specific or product-specific identifiers, which don't belong in the [common library](https://gitlab.com/gitlab-org/security-products/analyzers/common). -The first item of the `identifiers` array is called the primary identifier. +The first item of the `identifiers` array is called the [primary +identifier](../../user/application_security/terminology/#primary-identifier). The primary identifier is particularly important, because it is used to [track vulnerabilities](#tracking-and-merging-vulnerabilities) as new commits are pushed to the repository. Identifiers are also used to [merge duplicate vulnerabilities](#tracking-and-merging-vulnerabilities) reported for the same commit, except for `CWE` and `WASC`. +Not all vulnerabilities have CVEs, and a CVE can be identified multiple times. As a result, a CVE +isn't a stable identifier and you shouldn't assume it as such when tracking vulnerabilities. + +The maximum number of identifiers for a vulnerability is set as 20. If a vulnerability has more than 20 identifiers, +the system will save only the first 20 of them. + ### Location The `location` indicates where the vulnerability has been detected. diff --git a/doc/development/integrations/secure_partner_integration.md b/doc/development/integrations/secure_partner_integration.md index 830cb84e257..19fd86f4bf6 100644 --- a/doc/development/integrations/secure_partner_integration.md +++ b/doc/development/integrations/secure_partner_integration.md @@ -36,7 +36,7 @@ best place to integrate your own product and its results into GitLab. - Pipeline jobs serve a variety of purposes. Jobs can do scanning for and have implications for app security, corporate policy, or compliance. When complete, the job reports back on its status and creates a - [job artifact](../../user/project/pipelines/job_artifacts.md) as a result. + [job artifact](../../ci/pipelines/job_artifacts.md) as a result. - The [Merge Request Security Widget](../../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports) displays the results of the pipeline's security checks and the developer can review them. The developer can review both a summary and a detailed version @@ -44,7 +44,7 @@ best place to integrate your own product and its results into GitLab. - If certain policies (such as [merge request approvals](../../user/project/merge_requests/merge_request_approvals.md)) are in place for a project, developers must resolve specific findings or get an approval from a specific list of people. -- The [security dashboard](../../user/application_security/security_dashboard/index.md#gitlab-security-dashboard) +- The [security dashboard](../../user/application_security/security_dashboard/index.md) also shows results which can developers can use to quickly see all the vulnerabilities that need to be addressed in the code. - When the developer reads the details about a vulnerability, they are @@ -78,7 +78,7 @@ and complete an integration with the Secure stage. to successfully display your own product's results with the rest of GitLab. - See detailed [technical directions](secure.md) for this step. - Read more about [job report artifacts](../../ci/pipelines/job_artifacts.md#artifactsreports). - - Read about [job artifacts](../../user/project/pipelines/job_artifacts.md). + - Read about [job artifacts](../../ci/pipelines/job_artifacts.md). - Your report artifact must be in one of our currently supported formats. For more information, see the [documentation on reports](secure.md#report). - Documentation for [SAST reports](../../user/application_security/sast/index.md#reports-json-format). @@ -89,7 +89,7 @@ and complete an integration with the Secure stage. and add the label `devops::secure`. - Once the job is completed, the data can be seen: - In the [Merge Request Security Report](../../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports) ([MR Security Report data flow](https://gitlab.com/snippets/1910005#merge-request-view)). - - While [browsing a Job Artifact](../../user/project/pipelines/job_artifacts.md). + - While [browsing a Job Artifact](../../ci/pipelines/job_artifacts.md). - In the [Security Dashboard](../../user/application_security/security_dashboard/index.md) ([Dashboard data flow](https://gitlab.com/snippets/1910005#project-and-group-dashboards)). 1. Optional: Provide a way to interact with results as Vulnerabilities: - Users can interact with the findings from your artifact within their workflow. They can dismiss the findings or accept them and create a backlog issue. diff --git a/doc/development/internal_api.md b/doc/development/internal_api.md index 4b46787c2c3..e25feda356d 100644 --- a/doc/development/internal_api.md +++ b/doc/development/internal_api.md @@ -52,7 +52,7 @@ POST /internal/allowed | `username` | string | no | Username from the certificate used to connect to GitLab Shell | | `project` | string | no (if `gl_repository` is passed) | Path to the project | | `gl_repository` | string | no (if `project` is passed) | Repository identifier (e.g. `project-7`) | -| `protocol` | string | yes | SSH when called from GitLab-shell, HTTP or SSH when called from Gitaly | +| `protocol` | string | yes | SSH when called from GitLab Shell, HTTP or SSH when called from Gitaly | | `action` | string | yes | Git command being run (`git-upload-pack`, `git-receive-pack`, `git-upload-archive`) | | `changes` | string | yes | `<oldrev> <newrev> <refname>` when called from Gitaly, the magic string `_any` when called from GitLab Shell | | `check_ip` | string | no | IP address from which call to GitLab Shell was made | @@ -223,6 +223,7 @@ Example response: - GitLab Geo - GitLab Shell's `bin/check` +- Gitaly ## Get new 2FA recovery codes using an SSH key diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index 64089d51c7b..a54798b4c35 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -15,14 +15,14 @@ This document provides various guidelines when developing for GitLab's Some Kubernetes operations, such as creating restricted project namespaces are performed on the GitLab Rails application. These -operations are performed using a [client library](#client-library). -These operations will carry an element of risk as the operations will be -run as the same user running the GitLab Rails application, see the -[security](#security) section below. +operations are performed using a [client library](#client-library), +and carry an element of risk. The operations are +run as the same user running the GitLab Rails application. For more information, +read the [security](#security) section below. Some Kubernetes operations, such as installing cluster applications are performed on one-off pods on the Kubernetes cluster itself. These -installation pods are currently named `install-<application_name>` and +installation pods are named `install-<application_name>` and are created within the `gitlab-managed-apps` namespace. In terms of code organization, we generally add objects that represent @@ -33,33 +33,32 @@ Kubernetes resources in We use the [`kubeclient`](https://rubygems.org/gems/kubeclient) gem to perform Kubernetes API calls. As the `kubeclient` gem does not support -different API Groups (e.g. `apis/rbac.authorization.k8s.io`) from a +different API Groups (such as `apis/rbac.authorization.k8s.io`) from a single client, we have created a wrapper class, [`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb) -that will enable you to achieve this. +that enable you to achieve this. -Selected Kubernetes API groups are currently supported. Do add support +Selected Kubernetes API groups are supported. Do add support for new API groups or methods to [`Gitlab::Kubernetes::KubeClient`](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/kubernetes/kube_client.rb) if you need to use them. New API groups or API group versions can be -added to `SUPPORTED_API_GROUPS` - internally, this will create an +added to `SUPPORTED_API_GROUPS` - internally, this creates an internal client for that group. New methods can be added as a delegation to the relevant internal client. ### Performance considerations -All calls to the Kubernetes API must be in a background process. Do not -perform Kubernetes API calls within a web request as this will block -Unicorn and can easily lead to a Denial Of Service (DoS) attack in GitLab as +All calls to the Kubernetes API must be in a background process. Don't +perform Kubernetes API calls within a web request. This blocks +Unicorn, and can lead to a denial-of-service (DoS) attack in GitLab as the Kubernetes cluster response times are outside of our control. The easiest way to ensure your calls happen a background process is to delegate any such work to happen in a [Sidekiq worker](sidekiq_style_guide.md). -There are instances where you would like to make calls to Kubernetes and -return the response and as such a background worker does not seem to be -a good fit. For such cases you should make use of [reactive -caching](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb). +You may want to make calls to Kubernetes and return the response, but a background +worker isn't a good fit. Consider using +[reactive caching](https://gitlab.com/gitlab-org/gitlab/blob/master/app/models/concerns/reactive_caching.rb). For example: ```ruby @@ -76,7 +75,7 @@ For example: ### Testing -We have some Webmock stubs in +We have some WebMock stubs in [`KubernetesHelpers`](https://gitlab.com/gitlab-org/gitlab/blob/master/spec/support/helpers/kubernetes_helpers.rb) which can help with mocking out calls to Kubernetes API in your tests. @@ -86,8 +85,8 @@ This section outlines the process for allowing a GitLab instance to create EKS c The following prerequisites are required: -A `Customer` AWS account. This is the account in which the -EKS cluster will be created. The following resources must be present: +A `Customer` AWS account. The EKS cluster is created in this account. The following +resources must be present: - A provisioning role that has permissions to create the cluster and associated resources. It must list the `GitLab` AWS account @@ -114,7 +113,7 @@ The process for creating a cluster is as follows: which takes somewhere between 10 and 15 minutes in most cases. 1. When the stack is ready, GitLab stores the cluster details and generates another set of temporary credentials, this time to allow connecting to - the cluster via Kubeclient. These credentials are valid for one minute. + the cluster via `kubeclient`. These credentials are valid for one minute. 1. GitLab configures the worker nodes so that they are able to authenticate to the cluster, and creates a service account for itself for future operations. 1. Credentials that are no longer required are removed. This deletes the following @@ -126,7 +125,7 @@ The process for creating a cluster is as follows: ## Security -### SSRF +### Server Side Request Forgery (SSRF) attacks As URLs for Kubernetes clusters are user controlled it is easily susceptible to Server Side Request Forgery (SSRF) attacks. You should @@ -153,11 +152,11 @@ Mitigation strategies include: app.make_errored!("Kubernetes error: #{e.error_code}") ``` -## Debugging +## Debugging Kubernetes integrations Logs related to the Kubernetes integration can be found in [`kubernetes.log`](../administration/logs.md#kuberneteslog). On a local -GDK install, this will be present in `log/kubernetes.log`. +GDK install, these logs are present in `log/kubernetes.log`. Some services such as [`Clusters::Applications::InstallService`](https://gitlab.com/gitlab-org/gitlab/blob/master/app/services/clusters/applications/install_service.rb#L18) @@ -176,7 +175,9 @@ kubectl logs <pod_name> --follow -n gitlab-managed-apps ## GitLab Managed Apps -GitLab provides [GitLab Managed Apps](../user/clusters/applications.md), a one-click install for various applications which can be added directly to your configured cluster. +GitLab provides [GitLab Managed Apps](../user/clusters/applications.md), a one-click +install for various applications which can be added directly to your configured cluster. **<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview of how to add a new GitLab-managed app, see [How to add GitLab-managed-apps to Kubernetes integration](https://youtu.be/mKm-jkranEk).** +For an overview of how to add a new GitLab-managed app, see +[How to add GitLab-managed-apps to Kubernetes integration](https://youtu.be/mKm-jkranEk).** diff --git a/doc/development/logging.md b/doc/development/logging.md index 474a500da61..14812978f2d 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -287,7 +287,6 @@ method or variable shouldn't be evaluated right away) See our [HOWTO: Use Sidekiq metadata logs](https://www.youtube.com/watch?v=_wDllvO_IY0) for further knowledge on creating visualizations in Kibana. -NOTE: **Note:** The fields of the context are currently only logged for Sidekiq jobs triggered through web requests. See the [follow-up work](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/68) diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 207dd02d258..71191d1d871 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -295,13 +295,16 @@ end Adding foreign key to `projects`: +We can use the `add_concurrenct_foreign_key` method in this case, as this helper method +has the lock retries built into it. + ```ruby include Gitlab::Database::MigrationHelpers +disable_ddl_transaction! + def up - with_lock_retries do - add_foreign_key :imports, :projects, column: :project_id, on_delete: :cascade # rubocop:disable Migration/AddConcurrentForeignKey - end + add_concurrent_foreign_key :imports, :projects, column: :project_id, on_delete: :cascade end def down @@ -316,10 +319,10 @@ Adding foreign key to `users`: ```ruby include Gitlab::Database::MigrationHelpers +disable_ddl_transaction! + def up - with_lock_retries do - add_foreign_key :imports, :users, column: :user_id, on_delete: :cascade # rubocop:disable Migration/AddConcurrentForeignKey - end + add_concurrent_foreign_key :imports, :users, column: :user_id, on_delete: :cascade end def down @@ -331,7 +334,7 @@ end **Usage with `disable_ddl_transaction!`** -Generally the `with_lock_retries` helper should work with `disabled_ddl_transaction!`. A custom RuboCop rule ensures that only allowed methods can be placed within the lock retries block. +Generally the `with_lock_retries` helper should work with `disable_ddl_transaction!`. A custom RuboCop rule ensures that only allowed methods can be placed within the lock retries block. ```ruby disable_ddl_transaction! @@ -348,7 +351,7 @@ end The RuboCop rule generally allows standard Rails migration methods, listed below. This example will cause a Rubocop offense: ```ruby -disabled_ddl_transaction! +disable_ddl_transaction! def up with_lock_retries do @@ -364,17 +367,7 @@ standard Rails migration helper methods. Calling more than one migration helper is not a problem if they're executed on the same table. Using the `with_lock_retries` helper method is advised when a database -migration involves one of the high-traffic tables: - -- `users` -- `projects` -- `namespaces` -- `gitlab_subscriptions` -- `issues` -- `merge_requests` -- `ci_pipelines` -- `ci_builds` -- `notes` +migration involves one of the [high-traffic tables](https://gitlab.com/gitlab-org/gitlab/-/blob/master/rubocop/rubocop-migrations.yml#L3). Example changes: @@ -612,7 +605,7 @@ See the style guide on [`NOT NULL` constraints](database/not_null_constraints.md ## Adding Columns With Default Values -With PostgreSQL 11 being the minimum version since GitLab 13.0, adding columns with default values has become much easier and +With PostgreSQL 11 being the minimum version in GitLab 13.0 and later, adding columns with default values has become much easier and the standard `add_column` helper should be used in all cases. Before PostgreSQL 11, adding a column with a default was problematic as it would @@ -647,7 +640,7 @@ tables: `namespaces`. This can be translated to: ```sql ALTER TABLE namespaces ALTER COLUMN request_access_enabled -DEFAULT false +SET DEFAULT false ``` In this particular case, the default value exists and we're just changing the metadata for diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index 5ca43b9b818..714be296b40 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -122,7 +122,7 @@ If we look at this schema from a database point of view, we can see two deployme And these deployments align perfectly with application changes. 1. At the beginning we have `Version N` on `Schema A`. -1. Then we have a _long_ transition periond with both `Version N` and `Version N+1` on `Schema B`. +1. Then we have a _long_ transition period with both `Version N` and `Version N+1` on `Schema B`. 1. When we only have `Version N+1` on `Schema B` the schema changes again. 1. Finally we have `Version N+1` on `Schema C`. diff --git a/doc/development/new_fe_guide/development/accessibility.md b/doc/development/new_fe_guide/development/accessibility.md index b9ee5c3a549..9c63ccad6e1 100644 --- a/doc/development/new_fe_guide/development/accessibility.md +++ b/doc/development/new_fe_guide/development/accessibility.md @@ -12,7 +12,7 @@ WAI-ARIA (the Accessible Rich Internet Applications specification) defines a way The `role` attribute describes the role the element plays in the context of the document. -Check the list of WAI-ARIA roles [here](https://www.w3.org/TR/wai-aria-1.1/#landmark_roles) +Review the list of [WAI-ARIA roles](https://www.w3.org/TR/wai-aria-1.1/#landmark_roles). ## Icons diff --git a/doc/development/new_fe_guide/development/index.md b/doc/development/new_fe_guide/development/index.md index 5dced3dc466..119dbc58012 100644 --- a/doc/development/new_fe_guide/development/index.md +++ b/doc/development/new_fe_guide/development/index.md @@ -12,6 +12,6 @@ Learn how to implement an accessible frontend. Learn how to keep our frontend performant. -## [Testing](testing.md) +## [Testing](../../testing_guide/frontend_testing.md) Learn how to keep our frontend tested. diff --git a/doc/development/packages.md b/doc/development/packages.md index 55e22d4bb5f..9eae99ff890 100644 --- a/doc/development/packages.md +++ b/doc/development/packages.md @@ -64,15 +64,16 @@ will have to be managed. Using instance level endpoints requires [stricter namin The current state of existing package registries availability is: -| Repository Type | Project Level | Group Level | Instance Level | -|-----------------|---------------|-------------|----------------| -| Maven | Yes | Yes | Yes | -| Conan | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/11679) | Yes | -| NPM | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36853) | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36853) | -| NuGet | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36423) | No | -| PyPI | Yes | No | No | -| Go | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213900) | No - [open-issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213902) | -| Composer | Yes | Yes | No | +| Repository Type | Project Level | Group Level | Instance Level | +|------------------|---------------|-------------|----------------| +| Maven | Yes | Yes | Yes | +| Conan | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/11679) | Yes | +| NPM | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36853) | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36853) | +| NuGet | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36423) | No | +| PyPI | Yes | No | No | +| Go | Yes | No - [open issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213900) | No - [open-issue](https://gitlab.com/gitlab-org/gitlab/-/issues/213902) | +| Composer | Yes | Yes | No | +| Generic | Yes | No | No | NOTE: **Note:** NPM is currently a hybrid of the instance level and group level. @@ -87,7 +88,7 @@ Composer package naming scope is Instance Level. To avoid name conflict for instance-level endpoints you will need to define a package naming convention that gives a way to identify the project that the package belongs to. This generally involves using the project ID or full project path in the package name. See -[Conan's naming convention](../user/packages/conan_repository/index.md#package-recipe-naming-convention-for-instance-level-remote) as an example. +[Conan's naming convention](../user/packages/conan_repository/index.md#package-recipe-naming-convention-for-instance-remotes) as an example. For group and project-level endpoints, naming can be less constrained and it will be up to the group and project members to be certain that there is no conflict between two package names. However, the system should prevent diff --git a/doc/development/performance.md b/doc/development/performance.md index 1dc7538c55b..3b59393bae6 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -247,8 +247,12 @@ The following configuration options can be configured: - `STACKPROF_ENABLED`: Enables stackprof signal handler on SIGUSR2 signal. Defaults to `false`. -- `STACKPROF_INTERVAL_US`: Sampling interval in microseconds. Defaults to - `10000` μs (100hz). +- `STACKPROF_MODE`: See [sampling modes](https://github.com/tmm1/stackprof#sampling). + Defaults to `cpu`. +- `STACKPROF_INTERVAL`: Sampling interval. Unit semantics depend on `STACKPROF_MODE`. + For `object` mode this is a per-event interval (every `n`th event will be sampled) + and defaults to `1000`. + For other modes such as `cpu` this is a frequency and defaults to `10000` μs (100hz). - `STACKPROF_FILE_PREFIX`: File path prefix where profiles are stored. Defaults to `$TMPDIR` (often corresponds to `/tmp`). - `STACKPROF_TIMEOUT_S`: Profiling timeout in seconds. Profiling will diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 7756ef376fc..d12220d8c95 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -122,7 +122,6 @@ graph RL; click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0" subgraph "Needs `setup-test-env` & `compile-test-assets`"; 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3; - 2_2-3 --> 1-6 & 1-4; end 2_3-1["build-assets-image (2.5 minutes)"]; @@ -228,7 +227,6 @@ graph RL; click 2_2-5 "https://app.periscopedata.com/app/gitlab/652085/Engineering-Productivity---Pipeline-Build-Durations?widget=8404303&udv=0" subgraph "Needs `setup-test-env` & `compile-test-assets`"; 2_2-2 & 2_2-4 & 2_2-5 --> 1-6 & 1-3; - 2_2-3 --> 1-6 & 1-4; end 2_3-1["build-assets-image (2.5 minutes)"]; diff --git a/doc/development/policies.md b/doc/development/policies.md index 8f05948cb41..8dfd4763551 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -171,7 +171,7 @@ class ParentPolicy < BasePolicy condition(:speaks_spanish) { @subject.spoken_languages.include?(:es) } condition(:has_license) { @subject.driving_license.present? } condition(:enjoys_broccoli) { @subject.enjoyment_of(:broccoli) > 0 } - + rule { speaks_spanish }.enable :read_spanish rule { has_license }.enable :drive_car rule { enjoys_broccoli }.enable :eat_broccoli @@ -190,7 +190,7 @@ child policy, for example: ```ruby class ChildPolicy < BasePolicy delegate { @subject.parent } - + rule { default }.prevent :drive_car end ``` @@ -211,11 +211,11 @@ The solution it to override the `:eat_broccoli` ability in the child policy: ```ruby class ChildPolicy < BasePolicy delegate { @subject.parent } - + overrides :eat_broccoli - + condition(:good_kid) { @subject.behavior_level >= Child::GOOD } - + rule { good_kid }.enable :eat_broccoli end ``` diff --git a/doc/development/product_analytics/event_dictionary.md b/doc/development/product_analytics/event_dictionary.md new file mode 100644 index 00000000000..b049db21c30 --- /dev/null +++ b/doc/development/product_analytics/event_dictionary.md @@ -0,0 +1,32 @@ +--- +stage: Growth +group: Product Analytics +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Event Dictionary + +**Note: We've temporarily moved the Event Dictionary to a [Google Sheet](https://docs.google.com/spreadsheets/d/1VzE8R72Px_Y_LlE3Z05LxUlG_dumWe3vl-HeUo70TPw/edit?usp=sharing)**. The previous Markdown table exceeded 600 rows making it difficult to manage. In the future, our intention is to move this back into our docs using a [YAML file](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/823). + +The event dictionary is a single source of truth for the metrics and events we collect for product usage data. The Event Dictionary lists all the metrics and events we track, why we're tracking them, and where they are tracked. + +This is a living document that is updated any time a new event is planned or implemented. It includes the following information. + +- Section, stage, or group +- Description +- Implementation status +- Availability by plan type +- Code path + +We're currently focusing our Event Dictionary on [Usage Ping](usage_ping.md). In the future, we will also include [Snowplow](snowplow.md). We currently have an initiative across the entire product organization to complete the [Event Dictionary for Usage Ping](https://gitlab.com/groups/gitlab-org/-/epics/4174). + +## Instructions + +1. Open the Event Dictionary and fill in all the **PM to edit** columns highlighted in yellow. +1. Check that all the metrics and events are assigned to the correct section, stage, or group. If a metric is used across many groups, assign it to the stage. If a metric is used across many stages, assign it to the section. If a metric is incorrectly assigned to another section, stage, or group, let the PM know you have reassigned it. If your group has no assigned metrics and events, check that your metrics and events are not incorrectly assigned to another PM. +1. Add descriptions of what your metrics and events are tracking. Work with your Engineering team or the Product Analytics team if you need help understanding this. +1. Add what plans this metric is available on. Work with your Engineering team or the Product Analytics team if you need help understanding this. + +## Planned metrics and events + +For future metrics and events you plan to track, please add them to the Event Dictionary and note the status as `Planned`, `In Progress`, or `Implemented`. Once you have confirmed the metric has been implemented and have confirmed the metric data is in our data warehouse, change the status to **Data Available**. diff --git a/doc/development/product_analytics/index.md b/doc/development/product_analytics/index.md new file mode 100644 index 00000000000..ab76d6f0561 --- /dev/null +++ b/doc/development/product_analytics/index.md @@ -0,0 +1,182 @@ +--- +stage: Growth +group: Product Analytics +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Product Analytics Guide + +At GitLab, we collect product usage data for the purpose of helping us build a better product. Data helps GitLab understand which parts of the product need improvement and which features we should build next. Product usage data also helps our team better understand the reasons why people use GitLab. With this knowledge we are able to make better product decisions. + +We encourage users to enable tracking, and we embrace full transparency with our tracking approach so it can be easily understood and trusted. + +By enabling tracking, users can: + +- Contribute back to the wider community. +- Help GitLab improve on the product. + +## Our tracking tools + +We use three methods to gather product usage data: + +- [Snowplow](#snowplow) +- [Usage Ping](#usage-ping) +- [Database import](#database-import) + +### Snowplow + +Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way +users engage with our website and application. + +Snowplow consists of two components: + +- [Snowplow JS](https://github.com/snowplow/snowplow/wiki/javascript-tracker) tracks client-side + events. +- [Snowplow Ruby](https://github.com/snowplow/snowplow/wiki/ruby-tracker) tracks server-side events. + +For more details, read the [Snowplow](snowplow.md) guide. + +### Usage Ping + +Usage Ping is a method for GitLab Inc to collect usage data on a GitLab instance. Usage Ping is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. This high-level data is used to help our product, support, and sales teams. + +For more details, read the [Usage Ping](usage_ping.md) guide. + +### Database import + +Database imports are full imports of data into GitLab's data warehouse. For GitLab.com, the PostgreSQL database is loaded into Snowflake data warehouse every 6 hours. For more details, see the [data team handbook](https://about.gitlab.com/handbook/business-ops/data-team/platform/#extract-and-load). + +## What data can be tracked + +Our different tracking tools allows us to track different types of events. The event types and examples of what data can be tracked are outlined below. + +The availability of event types and their tracking tools varies by segment. For example, on Self-Managed Users, we only have reporting using Database records via Usage Ping. + +| Event Types | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | +|----------------------------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| +| Snowplow (JS Pageview events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | +| Snowplow (JS UI events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | +| Snowplow (Ruby Pageview events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | +| Snowplow (Ruby CRUD / API events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | +| Usage Ping (Redis UI counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | +| Usage Ping (Redis Pageview counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | +| Usage Ping (Redis CRUD / API counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | +| Usage Ping (Database counters) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | +| Usage Ping (Instance settings) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | +| Usage Ping (Integration settings) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | +| Database import (Database records) | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | + +[Source file](https://docs.google.com/spreadsheets/d/1e8Afo41Ar8x3JxAXJF3nL83UxVZ3hPIyXdt243VnNuE/edit?usp=sharing) + +**Legend** + +✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible + +SaaS = GitLab.com. SM = Self-Managed instance + +### Pageview events + +- Number of sessions that visited the /dashboard/groups page + +### UI events + +- Number of sessions that clicked on a button or link +- Number of sessions that closed a modal + +UI events are any interface-driven actions from the browser including click data. + +### CRUD or API events + +- Number of Git pushes +- Number of GraphQL queries +- Number of requests to a Rails action or controller + +These are backend events that include the creation, read, update, deletion of records, and other events that might be triggered from layers other than those available in the interface. + +### Database records + +These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql). + +### Instance settings + +These are settings of your instance such as the instance's Git version and if certain features are enabled such as `container_registry_enabled`. + +### Integration settings + +These are integrations your GitLab instance interacts with such as an [external storage provider](../../administration/static_objects_external_storage.md) or an [external container registry](../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). These services must be able to send data back into a GitLab instance for data to be tracked. + +## Reporting level + +Our reporting levels of aggregate or individual reporting varies by segment. For example, on Self-Managed Users, we can report at an aggregate user level using Usage Ping but not on an Individual user level. + +| Aggregated Reporting | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | +|----------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| +| Snowplow | ✅ | 📅 | 📅 | ✅ | 📅 | ✅ | 📅 | 📅 | ✅ | 📅 | +| Usage Ping | ✅ | 🔄 | 📅 | 📅 | ✅ | ✅ | ✅ | ✅ | 📅 | ✅ | +| Database import | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | + +| Identifiable Reporting | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | +|------------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| +| Snowplow | ✅ | 📅 | 📅 | ✅ | 📅 | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | +| Usage Ping | ✅ | 🔄 | 📅 | ✖️ | ✖️ | ✅ | ✅ | ✖️ | ✖️ | ✖️ | +| Database import | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | + +**Legend** + +✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible + +SaaS = GitLab.com. SM = Self-Managed instance + +## Reporting time period + +Our reporting time periods varies by segment. For example, on Self-Managed Users, we can report all time counts and 28 day counts in Usage Ping. + +| Reporting Time Period | All Time | 28 Days | 7 Days | Daily | +|-----------------------|----------|---------|--------|-------| +| Snowplow | ✅ | ✅ | ✅ | ✅ | +| Usage Ping | ✅ | ✅ | 📅 | ✖️ | +| Database import | ✅ | ✅ | ✅ | ✅ | + +**Legend** + +✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible + +## Systems overview + +The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed instances. + + + +[Source file](https://app.diagrams.net/#G13DVpN-XnhWGz9tqReIj8pp1UE4ehk_EC) + +### GitLab Inc + +For Product Analytics purposes, GitLab Inc has three major components: + +1. [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/): This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors [GitLab.com's Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/snowplow/) and GitLab's Versions Application. +1. GitLab.com: This is the production GitLab application which is made up of a Client and Server. On the Client or browser side, a Snowplow JS Tracker (Frontend) is used to track client-side events. On the Server or application side, a Snowplow Ruby Tracker (Backend) is used to track server-side events. The server also contains Usage Ping which leverages a PostgreSQL database and a Redis in-memory data store to report on usage data. Lastly, the server also contains System Logs which are generated from running the GitLab application. +1. [Monitoring infrastructure](https://about.gitlab.com/handbook/engineering/monitoring/): This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana. + +### Self-managed + +For Product Analytics purposes, self-managed instances have two major components: + +1. Data infrastructure: Having a data infrastructure setup is optional on self-managed instances. If you'd like to collect Snowplow tracking events for your self-managed instance, you can setup your own self-managed Snowplow collector and configure your Snowplow events to point to your own collector. +1. GitLab: A self-managed GitLab instance contains all of the same components as GitLab.com mentioned above. + +### Differences between GitLab Inc and Self-managed + +As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Usage Ping, and PostgreSQL database imports all flow into GitLab Inc's data infrastructure. However, on self-managed, only Usage Ping flows into GitLab Inc's data infrastructure. + +As shown by the green lines, on GitLab.com system logs flow into GitLab Inc's monitoring infrastructure. On self-managed, there are no logs sent to GitLab Inc's monitoring infrastructure. + +Note (1): Snowplow JS and Snowplow Ruby are available on self-managed, however, the Snowplow Collector endpoint is set to a self-managed Snowplow Collector which GitLab Inc does not have access to. + +## Additional information + +More useful links: + +- [Product Analytics Direction](https://about.gitlab.com/direction/product-analytics/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) diff --git a/doc/development/product_analytics/snowplow.md b/doc/development/product_analytics/snowplow.md new file mode 100644 index 00000000000..21d92566ffd --- /dev/null +++ b/doc/development/product_analytics/snowplow.md @@ -0,0 +1,429 @@ +--- +stage: Growth +group: Product Analytics +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Snowplow Guide + +This guide provides an overview of how Snowplow works, and implementation details. + +For more information about Product Analytics, see: + +- [Product Analytics Guide](index.md) +- [Usage Ping Guide](usage_ping.md) + +More useful links: + +- [Product Analytics Direction](https://about.gitlab.com/direction/product-analytics/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) + +## What is Snowplow + +Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. + +[Snowplow](https://github.com/snowplow/snowplow) consists of the following loosely-coupled sub-systems: + +- **Trackers** fire Snowplow events. Snowplow has 12 trackers, covering web, mobile, desktop, server, and IoT. +- **Collectors** receive Snowplow events from trackers. We have three different event collectors, synchronizing events either to Amazon S3, Apache Kafka, or Amazon Kinesis. +- **Enrich** cleans up the raw Snowplow events, enriches them and puts them into storage. We have an Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process. +- **Storage** is where the Snowplow events live. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases. +- **Data modeling** is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker. +- **Analytics** are performed on the Snowplow events or on the aggregate tables. + + + +## Snowplow schema + +We have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/-/issues/207930) including the following definitions: + +- Frontend and backend taxonomy as listed below +- [Structured event taxonomy](#structured-event-taxonomy) +- [Self describing events](https://github.com/snowplow/snowplow/wiki/Custom-events#self-describing-events) +- [Iglu schema](https://gitlab.com/gitlab-org/iglu/) +- [Snowplow authored events](https://github.com/snowplow/snowplow/wiki/Snowplow-authored-events) + +## Enabling Snowplow + +Tracking can be enabled at: + +- The instance level, which enables tracking on both the frontend and backend layers. +- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. + +We utilize Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: + +- **Admin Area > Settings > General** in the UI. +- `admin/application_settings/integrations` in your browser. + +The following configuration is required: + +| Name | Value | +|---------------|---------------------------| +| Collector | `snowplow.trx.gitlab.net` | +| Site ID | `gitlab` | +| Cookie domain | `.gitlab.com` | + +## Snowplow request flow + +The following example shows a basic request/response flow between the following components: + +- Snowplow JS / Ruby Trackers on GitLab.com +- [GitLab.com Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/library/snowplow/index.md) +- GitLab's S3 Bucket +- GitLab's Snowflake Data Warehouse +- Sisense: + +```mermaid +sequenceDiagram + participant Snowplow JS (Frontend) + participant Snowplow Ruby (Backend) + participant GitLab.com Snowplow Collector + participant S3 Bucket + participant Snowflake DW + participant Sisense Dashboards + Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event + Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event + loop Process using Kinesis Stream + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Enrich events + GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk + end + GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose + S3 Bucket->>Snowflake DW: Import data + Snowflake DW->>Snowflake DW: Transform data using dbt + Snowflake DW->>Sisense Dashboards: Data available for querying +``` + +## Structured event taxonomy + +When adding new click events, we should add them in a way that's internally consistent. If we don't, it'll be very painful to perform analysis across features since each feature will be capturing events differently. + +The current method provides several attributes that are sent on each click event. Please try to follow these guidelines when specifying events to capture: + +| attribute | type | required | description | +| --------- | ------- | -------- | ----------- | +| category | text | true | The page or backend area of the application. Unless infeasible, please use the Rails page attribute by default in the frontend, and namespace + classname on the backend. | +| action | text | true | The action the user is taking, or aspect that's being instrumented. The first word should always describe the action or aspect: clicks should be `click`, activations should be `activate`, creations should be `create`, etc. Use underscores to describe what was acted on; for example, activating a form field would be `activate_form_input`. An interface action like clicking on a dropdown would be `click_dropdown`, while a behavior like creating a project record from the backend would be `create_project` | +| label | text | false | The specific element, or object that's being acted on. This is either the label of the element (e.g. a tab labeled 'Create from template' may be `create_from_template`) or a unique identifier if no text is available (e.g. closing the Groups dropdown in the top navbar might be `groups_dropdown_close`), or it could be the name or title attribute of a record being created. | +| property | text | false | Any additional property of the element, or object being acted on. | +| value | decimal | false | Describes a numeric value or something directly related to the event. This could be the value of an input (e.g. `10` when clicking `internal` visibility). | + +## Implementing Snowplow JS (Frontend) tracking + +GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilize tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Structured event taxonomy](#structured-event-taxonomy). + +| field | type | default value | description | +|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `category` | string | document.body.dataset.page | Page or subsection of a page that events are being captured within. | +| `action` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | +| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | + +### Tracking in HAML (or Vue Templates) + +When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-event` attribute automatically have event tracking bound on clicks. + +Below is an example of `data-track-*` attributes assigned to a button: + +```haml +%button.btn{ data: { track: { event: "click_button", label: "template_preview", property: "my-template" } } } +``` + +```html +<button class="btn" + data-track-event="click_button" + data-track-label="template_preview" + data-track-property="my-template" +/> +``` + +Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you need to implement your own listeners and follow the instructions in [Tracking in raw JavaScript](#tracking-in-raw-javascript). + +Below is a list of supported `data-track-*` attributes: + +| attribute | required | description | +|:----------------------|:---------|:------------| +| `data-track-event` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. | +| `data-track-label` | false | The `label` as described in our [Structured event taxonomy](#structured-event-taxonomy). | +| `data-track-property` | false | The `property` as described in our [Structured event taxonomy](#structured-event-taxonomy). | +| `data-track-value` | false | The `value` as described in our [Structured event taxonomy](#structured-event-taxonomy). If omitted, this is the element's `value` property or an empty string. For checkboxes, the default value is the element's checked attribute or `false` when unchecked. | +| `data-track-context` | false | The `context` as described in our [Structured event taxonomy](#structured-event-taxonomy). | + +### Tracking within Vue components + +There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin. + +```javascript +import Tracking from '~/tracking'; +const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); +``` + +You can provide default options that are passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. + +You can then use the mixin normally in your component with the `mixin` Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These override any defaults and allow the values to be dynamic from props, or based on state. + +```javascript +export default { + mixins: [trackingMixin], + // ...[component implementation]... + data() { + return { + expanded: false, + tracking: { + label: 'left_sidebar' + } + }; + }, +} +``` + +The mixin provides a `track` method that can be called within the template, or from component methods. An example of the whole implementation might look like the following. + +```javascript +export default { + mixins: [Tracking.mixin({ label: 'right_sidebar' })], + data() { + return { + expanded: false, + }; + }, + methods: { + toggle() { + this.expanded = !this.expanded; + this.track('click_toggle', { value: this.expanded }) + } + } +}; +``` + +And if needed within the template, you can use the `track` method directly as well. + +```html +<template> + <div> + <a class="toggle" @click.prevent="toggle">Toggle</a> + <div v-if="expanded"> + <p>Hello world!</p> + <a @click.prevent="track('click_action')">Track an event</a> + </div> + </div> +</template> +``` + +### Tracking in raw JavaScript + +Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually. + +```javascript +import Tracking from '~/tracking'; + +const button = document.getElementById('create_from_template_button'); +button.addEventListener('click', () => { + Tracking.event('dashboard:projects:index', 'click_button', { + label: 'create_from_template', + property: 'template_preview', + value: 'rails', + }); +}) +``` + +### Tests and test helpers + +In Jest particularly in Vue tests, you can use the following: + +```javascript +import { mockTracking } from 'helpers/tracking_helper'; + +describe('MyTracking', () => { + let spy; + + beforeEach(() => { + spy = mockTracking('_category_', wrapper.element, jest.spyOn); + }); + + it('tracks an event when clicked on feedback', () => { + wrapper.find('.discover-feedback-icon').trigger('click'); + + expect(spy).toHaveBeenCalledWith('_category_', 'click_button', { + label: 'security-discover-feedback-cta', + property: '0', + }); + }); +}); +``` + +In obsolete Karma tests it's used as below: + +```javascript +import { mockTracking, triggerEvent } from 'spec/helpers/tracking_helper'; + +describe('my component', () => { + let trackingSpy; + + beforeEach(() => { + trackingSpy = mockTracking('_category_', vm.$el, spyOn); + }); + + const triggerEvent = () => { + // action which should trigger a event + }; + + it('tracks an event when toggled', () => { + expect(trackingSpy).not.toHaveBeenCalled(); + + triggerEvent('a.toggle'); + + expect(trackingSpy).toHaveBeenCalledWith('_category_', 'click_edit_button', { + label: 'right_sidebar', + property: 'confidentiality', + }); + }); +}); +``` + +## Implementing Snowplow Ruby (Backend) tracking + +GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events. + +Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: + +| argument | type | default value | description | +|:-----------|:-------|:--------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `category` | string | 'application' | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | +| `action` | string | 'generic' | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | +| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described in [Structured event taxonomy](#structured-event-taxonomy). These are set as empty strings if you don't provide them. | + +Tracking can be viewed as either tracking user behavior, or can be utilized for instrumentation to monitor and visualize performance over time in an area or aspect of code. + +For example: + +```ruby +class Projects::CreateService < BaseService + def execute + project = Project.create(params) + + Gitlab::Tracking.event('Projects::CreateService', 'create_project', + label: project.errors.full_messages.to_sentence, + value: project.valid? + ) + end +end +``` + +### Unit testing + +Use the `expect_snowplow_event` helper when testing backend Snowplow events. See [testing best practices]( +https://docs.gitlab.com/ee/development/testing_guide/best_practices.html#test-snowplow-events) for details. + +### Performance + +We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. + +## Developing and testing Snowplow + +There are several tools for developing and testing Snowplow Event + +| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | +|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| +| Snowplow Analytics Debugger Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| Snowplow Inspector Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| Snowplow Micro | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | +| Snowplow Mini | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{status_preparing}** | **{status_preparing}** | + +**Legend** + +**{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned + +### Preparing your MR for Review + +1. For frontend events, in the MR description section, add a screenshot of the event's relevant section using the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. +1. For backend events, please use Snowplow Micro and add the output of the Snowplow Micro good events `GET http://localhost:9090/micro/good`. + +### Snowplow Analytics Debugger Chrome Extension + +Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. + +1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. +1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. +1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html). + +### Snowplow Inspector Chrome Extension + +Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments. + +1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en). +1. Open the Chrome extension by pressing the Snowplow Inspector icon beside the address bar. +1. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window. + +### Snowplow Micro + +Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. + +Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. + +- Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) +- Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) +- Watch our [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) + +1. Install [Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro): + + ```shell + docker run --mount type=bind,source=$(pwd)/example,destination=/config -p 9090:9090 snowplow/snowplow-micro:latest --collector-config /config/micro.conf --iglu /config/iglu.json + ``` + +1. Install Snowplow Micro by cloning the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration): + + ```shell + git clone git@gitlab.com:gitlab-org/snowplow-micro-configuration.git + ./snowplow-micro.sh + ``` + +1. Update port in SQL to set `9090`: + + ```shell + gdk psql -d gitlabhq_development + update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; + ``` + +1. Update `app/assets/javascripts/tracking.js` to [remove this line](https://gitlab.com/snippets/1918635): + + ```javascript + forceSecureTracker: true + ``` + +1. Update `lib/gitlab/tracking.rb` to [add these lines](https://gitlab.com/snippets/1918635): + + ```ruby + protocol: 'http', + port: 9090, + ``` + +1. Update `lib/gitlab/tracking.rb` to [change async emitter from https to http](https://gitlab.com/snippets/1918635): + + ```ruby + SnowplowTracker::AsyncEmitter.new(Gitlab::CurrentSettings.snowplow_collector_hostname, protocol: 'http'), + ``` + +1. Enable Snowplow in the admin area, Settings::Integrations::Snowplow to point to: + `http://localhost:3000/admin/application_settings/integrations#js-snowplow-settings`. + +1. Restart GDK: + + ```shell + `gdk restart` + ``` + +1. Send a test Snowplow event from the Rails console: + + ```ruby + Gitlab::Tracking.self_describing_event('iglu:com.gitlab/pageview_context/jsonschema/1-0-0', { page_type: 'MY_TYPE' }, context: nil ) + ``` + +### Snowplow Mini + +[Snowplow Mini](https://github.com/snowplow/snowplow-mini) is an easily-deployable, single-instance version of Snowplow. + +Snowplow Mini can be used for testing frontend and backend events on a production, staging and local development environment. + +For GitLab.com, we're setting up a [QA and Testing environment](https://gitlab.com/gitlab-org/telemetry/-/issues/266) using Snowplow Mini. diff --git a/doc/development/product_analytics/usage_ping.md b/doc/development/product_analytics/usage_ping.md new file mode 100644 index 00000000000..d482af77d8a --- /dev/null +++ b/doc/development/product_analytics/usage_ping.md @@ -0,0 +1,937 @@ +--- +stage: Growth +group: Product Analytics +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Usage Ping Guide + +> - Introduced in GitLab Enterprise Edition 8.10. +> - More statistics were added in GitLab Enterprise Edition 8.12. +> - Moved to GitLab Core in 9.1. +> - More statistics were added in GitLab Ultimate 11.2. + +This guide describes Usage Ping's purpose and how it's implemented. + +For more information about Product Analytics, see: + +- [Product Analytics Guide](index.md) +- [Snowplow Guide](snowplow.md) + +More useful links: + +- [Product Analytics Direction](https://about.gitlab.com/direction/product-analytics/) +- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) +- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) +- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) + +## What is Usage Ping? + +- GitLab sends a weekly payload containing usage data to GitLab Inc. Usage Ping provides high-level data to help our product, support, and sales teams. It does not send any project names, usernames, or any other specific data. The information from the usage ping is not anonymous, it is linked to the hostname of the instance. Sending usage ping is optional, and any instance can disable analytics. +- The usage data is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. In addition to counts, other facts + that help us classify and understand GitLab installations are collected. +- Usage ping is important to GitLab as we use it to calculate our Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features. +- While usage ping is enabled, GitLab will gather data from the other instances and will be able to show usage statistics of your instance to your users. + +### Why should we enable Usage Ping? + +- The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. +- As a benefit of having the usage ping active, GitLab lets you analyze the users’ activities over time of your GitLab installation. +- As a benefit of having the usage ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance’s adoption of Concurrent DevOps from planning to monitoring. +- You will get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) +- You will get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? +- You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. +- Usage Ping is enabled by default. To disable it, see [Disable Usage Ping](#disable-usage-ping). + +### Limitations + +- Usage Ping does not track frontend events things like page views, link clicks, or user sessions, and only focuses on aggregated backend events. +- Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Usage Ping to track aggregated backend events on self-managed. + +## Usage Ping payload + +You can view the exact JSON payload sent to GitLab Inc. in the administration panel. To view the payload: + +1. Navigate to **Admin Area > Settings > Metrics and profiling**. +1. Expand the **Usage statistics** section. +1. Click the **Preview payload** button. + +For an example payload, see [Example Usage Ping payload](#example-usage-ping-payload). + +## Disable Usage Ping + +To disable Usage Ping in the GitLab UI, go to the **Settings** page of your administration panel and uncheck the **Usage Ping** checkbox. + +To disable Usage Ping and prevent it from being configured in the future through the administration panel, Omnibus installs can set the following in [`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options): + +```ruby +gitlab_rails['usage_ping_enabled'] = false +``` + +Source installations can set the following in `gitlab.yml`: + +```yaml +production: &base + # ... + gitlab: + # ... + usage_ping_enabled: false +``` + +## Usage Ping request flow + +The following example shows a basic request/response flow between a GitLab instance, the Versions Application, the License Application, Salesforce, GitLab's S3 Bucket, GitLab's Snowflake Data Warehouse, and Sisense: + +```mermaid +sequenceDiagram + participant GitLab Instance + participant Versions Application + participant Licenses Application + participant Salesforce + participant S3 Bucket + participant Snowflake DW + participant Sisense Dashboards + GitLab Instance->>Versions Application: Send Usage Ping + loop Process usage data + Versions Application->>Versions Application: Parse usage data + Versions Application->>Versions Application: Write to database + Versions Application->>Versions Application: Update license ping time + end + loop Process data for Salesforce + Versions Application-xLicenses Application: Request Zuora subscription id + Licenses Application-xVersions Application: Zuora subscription id + Versions Application-xSalesforce: Request Zuora account id by Zuora subscription id + Salesforce-xVersions Application: Zuora account id + Versions Application-xSalesforce: Usage data for the Zuora account + end + Versions Application->>S3 Bucket: Export Versions database + S3 Bucket->>Snowflake DW: Import data + Snowflake DW->>Snowflake DW: Transform data using dbt + Snowflake DW->>Sisense Dashboards: Data available for querying + Versions Application->>GitLab Instance: DevOps Report (Conversational Development Index) +``` + +## How Usage Ping works + +1. The Usage Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_usage_ping_worker.rb#L30) is set in Sidekiq to run weekly. +1. When the cron job runs, it calls [`GitLab::UsageData.to_json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L22). +1. `GitLab::UsageData.to_json` [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L22) to ~400+ other counter method calls. +1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L14) into a single JSON payload in `GitLab::UsageData.to_json`. +1. The JSON payload is then [posted to the Versions application]( https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L20). + +## Implementing Usage Ping + +Usage Ping consists of two kinds of data, counters and observations. Counters track how often a certain event +happened over time, such as how many CI pipelines have run. They are monotonic and always trend up. +Observations are facts collected from one or more GitLab instances and can carry arbitrary data. There are no +general guidelines around how to collect those, due to the individual nature of that data. + +There are several types of counters which are all found in `usage_data.rb`: + +- **Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation +- **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation on given column +- **Sum Batch Counters:** Sum the values of a given ActiveRecord_Relation on given column +- **Alternative Counters:** Used for settings and configurations +- **Redis Counters:** Used for in-memory counts. + +NOTE: **Note:** +Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Usage Ping. + +### Why batch counting + +For large tables, PostgreSQL can take a long time to count rows due to MVCC [(Multi-version Concurrency Control)](https://en.wikipedia.org/wiki/Multiversion_concurrency_control). Batch counting is a counting method where a single large query is broken into multiple smaller queries. For example, instead of a single query querying 1,000,000 records, with batch counting, you can execute 100 queries of 10,000 records each. Batch counting is useful for avoiding database timeouts as each batch query is significantly shorter than one single long running query. + +For GitLab.com, there are extremely large tables with 15 second query timeouts, so we use batch counting to avoid encountering timeouts. Here are the sizes of some GitLab.com tables: + +| Table | Row counts in millions | +|------------------------------|------------------------| +| `merge_request_diff_commits` | 2280 | +| `ci_build_trace_sections` | 1764 | +| `merge_request_diff_files` | 1082 | +| `events` | 514 | + +There are two batch counting methods provided, `Ordinary Batch Counters` and `Distinct Batch Counters`. Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, a specialized index may need to be added on the columns involved in a counter. + +### Ordinary Batch Counters + +Handles `ActiveRecord::StatementInvalid` error + +Simple count of a given ActiveRecord_Relation, does a non-distinct batch count, smartly reduces batch_size and handles errors. + +Method: `count(relation, column = nil, batch: true, start: nil, finish: nil)` + +Arguments: + +- `relation` the ActiveRecord_Relation to perform the count +- `column` the column to perform the count on, by default is the primary key +- `batch`: default `true` in order to use batch counting +- `start`: custom start of the batch counting in order to avoid complex min calculations +- `end`: custom end of the batch counting in order to avoid complex min calculations + +Examples: + +```ruby +count(User.active) +count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) +count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id)) +``` + +### Distinct Batch Counters + +Handles `ActiveRecord::StatementInvalid` error + +Distinct count of a given ActiveRecord_Relation on given column, a distinct batch count, smartly reduces batch_size and handles errors. + +Method: `distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil)` + +Arguments: + +- `relation` the ActiveRecord_Relation to perform the count +- `column` the column to perform the distinct count, by default is the primary key +- `batch`: default `true` in order to use batch counting +- `batch_size`: if none set it will use default value 10000 from `Gitlab::Database::BatchCounter` +- `start`: custom start of the batch counting in order to avoid complex min calculations +- `end`: custom end of the batch counting in order to avoid complex min calculations + +Examples: + +```ruby +distinct_count(::Project, :creator_id) +distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) +distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') +``` + +### Sum Batch Counters + +Handles `ActiveRecord::StatementInvalid` error + +Sum the values of a given ActiveRecord_Relation on given column and handles errors. + +Method: `sum(relation, column, batch_size: nil, start: nil, finish: nil)` + +Arguments: + +- `relation` the ActiveRecord_Relation to perform the operation +- `column` the column to sum on +- `batch_size`: if none set it will use default value 1000 from `Gitlab::Database::BatchCounter` +- `start`: custom start of the batch counting in order to avoid complex min calculations +- `end`: custom end of the batch counting in order to avoid complex min calculations + +Examples: + +```ruby +sum(JiraImportState.finished, :imported_issues_count) +``` + +### Grouping & Batch Operations + +The `count`, `distinct_count`, and `sum` batch counters can accept an `ActiveRecord::Relation` +object, which groups by a specified column. With a grouped relation, the methods do batch counting, +handle errors, and returns a hash table of key-value pairs. + +Examples: + +```ruby +count(Namespace.group(:type)) +# returns => {nil=>179, "Group"=>54} + +distinct_count(Project.group(:visibility_level), :creator_id) +# returns => {0=>1, 10=>1, 20=>11} + +sum(Issue.group(:state_id), :weight)) +# returns => {1=>3542, 2=>6820} +``` + +### Redis Counters + +Handles `::Redis::CommandError` and `Gitlab::UsageDataCounters::BaseCounter::UnknownEvent` +returns -1 when a block is sent or hash with all values -1 when a `counter(Gitlab::UsageDataCounters)` is sent +different behavior due to 2 different implementations of Redis counter + +Method: `redis_usage_data(counter, &block)` + +Arguments: + +- `counter`: a counter from `Gitlab::UsageDataCounters`, that has `fallback_totals` method implemented +- or a `block`: which is evaluated + +#### Ordinary Redis Counters + +Examples of implementation: + +- Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb) +- Using Redis methods [`HINCRBY`](https://redis.io/commands/hincrby), [`HGETALL`](https://redis.io/commands/hgetall), and [`Gitlab::UsageCounters::PodLogs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_counters/pod_logs.rb) + +#### Redis HLL Counters + +With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values. + +Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount). + +##### Adding new events + +1. Define events in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml). + + Example event: + + ```yaml + - name: i_compliance_credential_inventory + category: compliance + redis_slot: compliance + expiry: 42 # 6 weeks + aggregation: weekly + ``` + + Keys: + + - `name`: unique event name. + + Name format `<prefix>_<redis_slot>_name`. + + Use one of the following prefixes for the event's name: + + - `g_` for group, as an event which is tracked for group. + - `p_` for project, as an event which is tracked for project. + - `i_` for instance, as an event which is tracked for instance. + - `a_` for events encompassing all `g_`, `p_`, `i_`. + - `o_` for other. + + Consider including in the event's name the Redis slot in order to be able to count totals for a specific category. + + Example names: `i_compliance_credential_inventory`, `g_analytics_contribution`. + + - `category`: event category. Used for getting total counts for events in a category, for easier + access to a group of events. + - `redis_slot`: optional Redis slot; default value: event name. Used if needed to calculate totals + for a group of metrics. Ensure keys are in the same slot. For example: + `i_compliance_credential_inventory` with `redis_slot: 'compliance'` will build Redis key + `i_{compliance}_credential_inventory-2020-34`. If `redis_slot` is not defined the Redis key will + be `{i_compliance_credential_inventory}-2020-34`. + - `expiry`: expiry time in days. Default: 29 days for daily aggregation and 6 weeks for weekly + aggregation. + - `aggregation`: aggregation `:daily` or `:weekly`. The argument defines how we build the Redis + keys for data storage. For `daily` we keep a key for metric per day of the year, for `weekly` we + keep a key for metric per week of the year. + +1. Track event in controller using `RedisTracking` module with `track_redis_hll_event(*controller_actions, name:, feature:, feature_default_enabled: false)`. + + Arguments: + + - `controller_actions`: controller actions we want to track. + - `name`: event name. + - `feature`: feature name, all metrics we track should be under feature flag. + - `feature_default_enabled`: feature flag is disabled by default, set to `true` for it to be enabled by default. + + Example usage: + + ```ruby + # controller + class ProjectsController < Projects::ApplicationController + include RedisTracking + + skip_before_action :authenticate_user!, only: :show + track_redis_hll_event :index, :show, name: 'g_compliance_example_feature_visitors', feature: :compliance_example_feature, feature_default_enabled: true + + def index + render html: 'index' + end + + def new + render html: 'new' + end + + def show + render html: 'show' + end + end + ``` + +1. Track event in API using `increment_unique_values(event_name, values)` helper method. + + In order to be able to track the event, Usage Ping must be enabled and the event feature `usage_data_<event_name>` must be enabled. + + Arguments: + + - `event_name`: event name. + - `values`: values counted, one value or array of values. + + Example usage: + + ```ruby + get ':id/registry/repositories' do + repositories = ContainerRepositoriesFinder.new( + user: current_user, subject: user_group + ).execute + + increment_unique_values('i_list_repositories', current_user.id) + + present paginate(repositories), with: Entities::ContainerRegistry::Repository, tags: params[:tags], tags_count: params[:tags_count] + end + ``` + +1. Track event using `track_usage_event(event_name, values) in services and graphql + + Increment unique values count using Redis HLL, for given event name. + + Example: + + [Track usage event for incident created in service](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/issues/update_service.rb) + + [Track usage event for incident created in graphql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/mutations/alert_management/update_alert_status.rb) + + ```ruby + track_usage_event(:incident_management_incident_created, current_user.id) + ``` + +1. Track event using `UsageData` API + + Increment unique users count using Redis HLL, for given event name. + + Tracking events using the `UsageData` API requires the `usage_data_api` feature flag to be enabled, which is enabled by default. + + API requests are protected by checking for a valid CSRF token. + + In order to be able to increment the values the related feature `usage_data<event_name>` should be enabled. + + ```plaintext + POST /usage_data/increment_unique_users + ``` + + | Attribute | Type | Required | Description | + | :-------- | :--- | :------- | :---------- | + | `event` | string | yes | The event name it should be tracked | + + Response +w + Return 200 if tracking failed for any reason. + + - `200` if event was tracked or any errors + - `400 Bad request` if event parameter is missing + - `401 Unauthorized` if user is not authenticated + - `403 Forbidden` for invalid CSRF token provided + +1. Track events using JavaScript/Vue API helper which calls the API above + + Example usage for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml): + + Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled in order to be able to track events + + ```javascript + import api from '~/api'; + + api.trackRedisHllUserEvent('my_already_defined_event_name'), + ``` + +1. Track event using base module `Gitlab::UsageDataCounters::HLLRedisCounter.track_event(entity_id, event_name)`. + + Arguments: + + - `entity_id`: value we count. For example: user_id, visitor_id. + - `event_name`: event name. + +1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date)`. + + Arguments: + + - `event_names`: the list of event names. + - `start_date`: start date of the period for which we want to get event data. + - `end_date`: end date of the period for which we want to get event data. + +Recommendations: + +- Key should expire in 29 days for daily and 42 days for weekly. +- If possible, data granularity should be a week. For example a key could be composed from the + metric's name and week of the year, `2020-33-{metric_name}`. +- Use a [feature flag](../../operations/feature_flags.md) to have a control over the impact when + adding new metrics. + +##### Known events in usage data payload + +All events added in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml) are automatically added to usage data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). +For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: + +- `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#adding-new-events) events and data for the last complete week for weekly [aggregation](#adding-new-events) events. +- `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#adding-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#adding-new-events) events. +- `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. +- `#{category}_total_unique_counts_monthly`: Total unique counts for events in same category for the last 28 days or the last 4 complete weeks, if events are in the same Redis slot and we have more than one metric. + +Example of `redis_hll_counters` data: + +```ruby +{:redis_hll_counters=> + {"compliance"=> + {"g_compliance_dashboard_weekly"=>0, + "g_compliance_dashboard_monthly"=>0, + "g_compliance_audit_events_weekly"=>0, + "g_compliance_audit_events_monthly"=>0, + "compliance_total_unique_counts_weekly"=>0, + "compliance_total_unique_counts_monthly"=>0}, + "analytics"=> + {"g_analytics_contribution_weekly"=>0, + "g_analytics_contribution_monthly"=>0, + "g_analytics_insights_weekly"=>0, + "g_analytics_insights_monthly"=>0, + "analytics_total_unique_counts_weekly"=>0, + "analytics_total_unique_counts_monthly"=>0}, + "ide_edit"=> + {"g_edit_by_web_ide_weekly"=>0, + "g_edit_by_web_ide_monthly"=>0, + "g_edit_by_sfe_weekly"=>0, + "g_edit_by_sfe_monthly"=>0, + "ide_edit_total_unique_counts_weekly"=>0, + "ide_edit_total_unique_counts_monthly"=>0}, + "search"=> + {"i_search_total_weekly"=>0, "i_search_total_monthly"=>0, "i_search_advanced_weekly"=>0, "i_search_advanced_monthly"=>0, "i_search_paid_weekly"=>0, "i_search_paid_monthly"=>0, "search_total_unique_counts_weekly"=>0, "search_total_unique_counts_monthly"=>0}, + "source_code"=>{"wiki_action_weekly"=>0, "wiki_action_monthly"=>0} + } +``` + +Example usage: + +```ruby +# Redis Counters +redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) +redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] } + +# Define events in known_events.yml https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml + +# Tracking events +Gitlab::UsageDataCounters::HLLRedisCounter.track_event(visitor_id, 'expand_vulnerabilities') + +# Get unique events for metric +redis_usage_data { Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'expand_vulnerabilities', start_date: 28.days.ago, end_date: Date.current) } +``` + +### Alternative Counters + +Handles `StandardError` and fallbacks into -1 this way not all measures fail if we encounter one exception. +Mainly used for settings and configurations. + +Method: `alt_usage_data(value = nil, fallback: -1, &block)` + +Arguments: + +- `value`: a simple static value in which case the value is simply returned. +- or a `block`: which is evaluated +- `fallback: -1`: the common value used for any metrics that are failing. + +Example of usage: + +```ruby +alt_usage_data { Gitlab::VERSION } +alt_usage_data { Gitlab::CurrentSettings.uuid } +alt_usage_data(999) +``` + +### Prometheus Queries + +In those cases where operational metrics should be part of Usage Ping, a database or Redis query is unlikely +to provide useful data. Instead, Prometheus might be more appropriate, since most of GitLab's architectural +components publish metrics to it that can be queried back, aggregated, and included as usage data. + +NOTE: **Note:** +Prometheus as a data source for Usage Ping is currently only available for single-node Omnibus installations +that are running the [bundled Prometheus](../../administration/monitoring/prometheus/index.md) instance. + +In order to query Prometheus for metrics, a helper method is available that will `yield` a fully configured +`PrometheusClient`, given it is available as per the note above: + +```ruby +with_prometheus_client do |client| + response = client.query('<your query>') + ... +end +``` + +Please refer to [the `PrometheusClient` definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/prometheus_client.rb) +for how to use its API to query for data. + +## Developing and testing Usage Ping + +### 1. Use your Rails console to manually test counters + +```ruby +# count +Gitlab::UsageData.count(User.active) +Gitlab::UsageData.count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) + +# count distinct +Gitlab::UsageData.distinct_count(::Project, :creator_id) +Gitlab::UsageData.distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) +``` + +### 2. Generate the SQL query + +Your Rails console will return the generated SQL queries. + +Example: + +```ruby +pry(main)> Gitlab::UsageData.count(User.active) + (2.6ms) SELECT "features"."key" FROM "features" + (15.3ms) SELECT MIN("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) + (2.4ms) SELECT MAX("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) + (1.9ms) SELECT COUNT("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) AND "users"."id" BETWEEN 1 AND 100000 +``` + +### 3. Optimize queries with #database-lab + +Paste the SQL query into `#database-lab` to see how the query performs at scale. + +- `#database-lab` is a Slack channel which uses a production-sized environment to test your queries. +- GitLab.com’s production database has a 15 second timeout. +- Any single query must stay below 1 second execution time with cold caches. +- Add a specialized index on columns involved to reduce the execution time. + +In order to have an understanding of the query's execution we add in the MR description the following information: + +- For counters that have a `time_period` test we add information for both cases: + - `time_period = {}` for all time periods + - `time_period = { created_at: 28.days.ago..Time.current }` for last 28 days period +- Execution plan and query time before and after optimization +- Query generated for the index and time +- Migration output for up and down execution + +We also use `#database-lab` and [explain.depesz.com](https://explain.depesz.com/). For more details, see the [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries). + +#### Optimization recommendations and examples + +- Use specialized indexes [example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871), [example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445). +- Use defined `start` and `finish`, and simple queries, because these values can be memoized and reused, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37155). +- Avoid joins and write the queries as simply as possible, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36316). +- Set a custom `batch_size` for `distinct_count`, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38000). + +### 4. Add the metric definition + +When adding, changing, or updating metrics, please update the [Event Dictionary's **Usage Ping** table](event_dictionary.md). + +### 5. Add new metric to Versions Application + +Check if new metrics need to be added to the Versions Application. See `usage_data` [schema](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L147) and usage data [parameters accepted](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/app/services/usage_ping.rb). Any metrics added under the `counts` key are saved in the `stats` column. + +### 6. Add the feature label + +Add the `feature` label to the Merge Request for new Usage Ping metrics. These are user-facing changes and are part of expanding the Usage Ping feature. + +### 7. Add a changelog file + +Ensure you comply with the [Changelog entries guide](../changelog.md). + +### 8. Ask for a Product Analytics Review + +On GitLab.com, we have DangerBot setup to monitor Product Analytics related files and DangerBot will recommend a Product Analytics review. Mention `@gitlab-org/growth/product_analytics/engineers` in your MR for a review. + +### 9. Verify your metric + +On GitLab.com, the Product Analytics team regularly monitors Usage Ping. They may alert you that your metrics need further optimization to run quicker and with greater success. You may also use the [Usage Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" & "Saas" and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you may re-optimize your metric. + +### Optional: Test Prometheus based Usage Ping + +If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) that you would like to inspect and verify, +then you need to ensure that a Prometheus server is running locally, and that furthermore the respective GitLab components +are exporting metrics to it. If you do not need to test data coming from Prometheus, no further action +is necessary, since Usage Ping should degrade gracefully in the absence of a running Prometheus server. + +There are currently three kinds of components that may export data to Prometheus, and which are included in Usage Ping: + +- [`node_exporter`](https://github.com/prometheus/node_exporter) - Exports node metrics from the host machine +- [`gitlab-exporter`](https://gitlab.com/gitlab-org/gitlab-exporter) - Exports process metrics from various GitLab components +- various GitLab services such as Sidekiq and the Rails server that export their own metrics + +#### Test with an Omnibus container + +This is the recommended approach to test Prometheus based Usage Ping. + +The easiest way to verify your changes is to build a new Omnibus image from your code branch via CI, then download the image +and run a local container instance: + +1. From your merge request, click on the `qa` stage, then trigger the `package-and-qa` job. This job will trigger an Omnibus +build in a [downstream pipeline of the `omnibus-gitlab-mirror` project](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines). +1. In the downstream pipeline, wait for the `gitlab-docker` job to finish. +1. Open the job logs and locate the full container name including the version. It will take the following form: `registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>`. +1. On your local machine, make sure you are logged in to the GitLab Docker registry. You can find the instructions for this in +[Authenticate to the GitLab Container Registry](../../user/packages/container_registry/index.md#authenticate-with-the-container-registry). +1. Once logged in, download the new image via `docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>` +1. For more information about working with and running Omnibus GitLab containers in Docker, please refer to [GitLab Docker images](https://docs.gitlab.com/omnibus/docker/README.html) in the Omnibus documentation. + +#### Test with GitLab development toolkits + +This is the less recommended approach, since it comes with a number of difficulties when emulating a real GitLab deployment. + +The [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) is not currently set up to run a Prometheus server or `node_exporter` alongside other GitLab components. If you would +like to do so, [Monitoring the GDK with Prometheus](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/prometheus/index.md#monitoring-the-gdk-with-prometheus) is a good start. + +The [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit) has limited support for testing Prometheus based Usage Ping. +By default, it already comes with a fully configured Prometheus service that is set up to scrape a number of components, +but with the following limitations: + +- It does not currently run a `gitlab-exporter` instance, so several `process_*` metrics from services such as Gitaly may be missing. +- While it runs a `node_exporter`, `docker-compose` services emulate hosts, meaning that it would normally report itself to not be associated +with any of the other services that are running. That is not how node metrics are reported in a production setup, where `node_exporter` +always runs as a process alongside other GitLab components on any given node. From Usage Ping's perspective none of the node data would therefore +appear to be associated to any of the services running, since they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics will appear in Usage Ping. + +## Example Usage Ping payload + +The following is example content of the Usage Ping payload. + +```json +{ + "uuid": "0000000-0000-0000-0000-000000000000", + "hostname": "example.com", + "version": "12.10.0-pre", + "installation_type": "omnibus-gitlab", + "active_user_count": 999, + "recorded_at": "2020-04-17T07:43:54.162+00:00", + "edition": "EEU", + "license_md5": "00000000000000000000000000000000", + "license_id": null, + "historical_max_users": 999, + "licensee": { + "Name": "ABC, Inc.", + "Email": "email@example.com", + "Company": "ABC, Inc." + }, + "license_user_count": 999, + "license_starts_at": "2020-01-01", + "license_expires_at": "2021-01-01", + "license_plan": "ultimate", + "license_add_ons": { + }, + "license_trial": false, + "counts": { + "assignee_lists": 999, + "boards": 999, + "ci_builds": 999, + ... + }, + "container_registry_enabled": true, + "dependency_proxy_enabled": false, + "gitlab_shared_runners_enabled": true, + "gravatar_enabled": true, + "influxdb_metrics_enabled": true, + "ldap_enabled": false, + "mattermost_enabled": false, + "omniauth_enabled": true, + "prometheus_enabled": false, + "prometheus_metrics_enabled": false, + "reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com", + "signup_enabled": true, + "web_ide_clientside_preview_enabled": true, + "ingress_modsecurity_enabled": true, + "projects_with_expiration_policy_disabled": 999, + "projects_with_expiration_policy_enabled": 999, + ... + "elasticsearch_enabled": true, + "license_trial_ends_on": null, + "geo_enabled": false, + "git": { + "version": { + "major": 2, + "minor": 26, + "patch": 1 + } + }, + "gitaly": { + "version": "12.10.0-rc1-93-g40980d40", + "servers": 56, + "clusters": 14, + "filesystems": [ + "EXT_2_3_4" + ] + }, + "gitlab_pages": { + "enabled": true, + "version": "1.17.0" + }, + "container_registry_server": { + "vendor": "gitlab", + "version": "2.9.1-gitlab" + }, + "database": { + "adapter": "postgresql", + "version": "9.6.15", + "pg_system_id": 6842684531675334351 + }, + "avg_cycle_analytics": { + "issue": { + "average": 999, + "sd": 999, + "missing": 999 + }, + "plan": { + "average": null, + "sd": 999, + "missing": 999 + }, + "code": { + "average": null, + "sd": 999, + "missing": 999 + }, + "test": { + "average": null, + "sd": 999, + "missing": 999 + }, + "review": { + "average": null, + "sd": 999, + "missing": 999 + }, + "staging": { + "average": null, + "sd": 999, + "missing": 999 + }, + "production": { + "average": null, + "sd": 999, + "missing": 999 + }, + "total": 999 + }, + "analytics_unique_visits": { + "g_analytics_contribution": 999, + ... + }, + "usage_activity_by_stage": { + "configure": { + "project_clusters_enabled": 999, + ... + }, + "create": { + "merge_requests": 999, + ... + }, + "manage": { + "events": 999, + ... + }, + "monitor": { + "clusters": 999, + ... + }, + "package": { + "projects_with_packages": 999 + }, + "plan": { + "issues": 999, + ... + }, + "release": { + "deployments": 999, + ... + }, + "secure": { + "user_container_scanning_jobs": 999, + ... + }, + "verify": { + "ci_builds": 999, + ... + } + }, + "usage_activity_by_stage_monthly": { + "configure": { + "project_clusters_enabled": 999, + ... + }, + "create": { + "merge_requests": 999, + ... + }, + "manage": { + "events": 999, + ... + }, + "monitor": { + "clusters": 999, + ... + }, + "package": { + "projects_with_packages": 999 + }, + "plan": { + "issues": 999, + ... + }, + "release": { + "deployments": 999, + ... + }, + "secure": { + "user_container_scanning_jobs": 999, + ... + }, + "verify": { + "ci_builds": 999, + ... + } + }, + "topology": { + "duration_s": 0.013836685999194742, + "application_requests_per_hour": 4224, + "query_apdex_weekly_average": 0.996, + "failures": [], + "nodes": [ + { + "node_memory_total_bytes": 33269903360, + "node_memory_utilization": 0.35, + "node_cpus": 16, + "node_cpu_utilization": 0.2, + "node_uname_info": { + "machine": "x86_64", + "sysname": "Linux", + "release": "4.19.76-linuxkit" + }, + "node_services": [ + { + "name": "web", + "process_count": 16, + "process_memory_pss": 233349888, + "process_memory_rss": 788220927, + "process_memory_uss": 195295487, + "server": "puma" + }, + { + "name": "sidekiq", + "process_count": 1, + "process_memory_pss": 734080000, + "process_memory_rss": 750051328, + "process_memory_uss": 731533312 + }, + ... + ], + ... + }, + ... + ] + } +} +``` + +## Notable changes + +In GitLab 13.5, `pg_system_id` was added to send the [PostgreSQL system identifier](https://www.2ndquadrant.com/en/blog/support-for-postgresqls-system-identifier-in-barman/). + +## Exporting Usage Ping SQL queries and definitions + +Two Rake tasks exist to export Usage Ping definitions. + +- The Rake tasks export the raw SQL queries for `count`, `distinct_count`, `sum`. +- The Rake tasks export the Redis counter class or the line of the Redis block for `redis_usage_data`. +- The Rake tasks calculate the `alt_usage_data` metrics. + +In the home directory of your local GitLab installation run the following Rake tasks for the YAML and JSON versions respectively: + +```shell +# for YAML export +bin/rake gitlab:usage_data:dump_sql_in_yaml + +# for JSON export +bin/rake gitlab:usage_data:dump_sql_in_json + +# You may pipe the output into a file +bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml +``` diff --git a/doc/development/profiling.md b/doc/development/profiling.md index f7ead94d725..f5a4d1edb92 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -91,9 +91,6 @@ that builds on this to add some additional niceties, such as allowing configuration with a single YAML file for multiple URLs, and uploading of the profile and log output to S3. -For GitLab.com, you can find the latest results here (restricted to GitLab Team members only): -`https://redash.gitlab.com/dashboard/gitlab-profiler-statistics` - ## Sherlock Sherlock is a custom profiling tool built into GitLab. Sherlock is _only_ diff --git a/doc/development/prometheus.md b/doc/development/prometheus.md index 902d4e6a1d0..3e7acaf6d94 100644 --- a/doc/development/prometheus.md +++ b/doc/development/prometheus.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -47,3 +47,13 @@ using a Prometheus managed application in Kubernetes: ``` 1. Open `localhost:9090` in your browser to display the Prometheus user interface. + +## Script access to Prometheus + +You can script the access to Prometheus, extracting the name of the pod automatically like this: + +```shell +POD_INFORMATION=$(kubectl get pods -n gitlab-managed-apps | grep 'prometheus-prometheus-server') +POD_NAME=$(echo $POD_INFORMATION | awk '{print $1;}') +kubectl port-forward $POD_NAME 9090:9090 -n gitlab-managed-apps +``` diff --git a/doc/development/prometheus_metrics.md b/doc/development/prometheus_metrics.md index a39d19d8750..8fcc025b35b 100644 --- a/doc/development/prometheus_metrics.md +++ b/doc/development/prometheus_metrics.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: APM +group: Health info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -35,11 +35,6 @@ After you add or change an existing common metric, you must [re-run the import s Or, you can create a database migration: -NOTE: **Note:** -If a query metric (which is identified by `id:`) is removed it will not be removed from database by default. -You might want to add additional database migration that makes a decision what to do with removed one. -For example: you might be interested in migrating all dependent data to a different metric. - ```ruby class ImportCommonMetrics < ActiveRecord::Migration[4.2] include Gitlab::Database::MigrationHelpers @@ -56,6 +51,10 @@ class ImportCommonMetrics < ActiveRecord::Migration[4.2] end ``` +If a query metric (which is identified by `id:`) is removed it will not be removed from database by default. +You might want to add additional database migration that makes a decision what to do with removed one. +For example: you might be interested in migrating all dependent data to a different metric. + ## GitLab Prometheus metrics GitLab provides [Prometheus metrics](../administration/monitoring/prometheus/gitlab_metrics.md) diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index f3386305e93..cf125c46565 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -85,9 +85,7 @@ The ReactiveCaching concern can be used in models as well as `project_services` 1. Implement the `calculate_reactive_cache` method in your model/service. 1. Call `with_reactive_cache` in your model/service where the cached value is needed. -1. If the `calculate_reactive_cache` method above submits requests to external services -(e.g. Prometheus, K8s), make sure to change the -[`reactive_cache_work_type` accordingly](#selfreactive_cache_work_type). +1. Set the [`reactive_cache_work_type` accordingly](#selfreactive_cache_work_type). ### In controllers @@ -252,7 +250,7 @@ self.reactive_cache_hard_limit = 5.megabytes - This is the type of work performed by the `calculate_reactive_cache` method. Based on this attribute, it's able to pick the right worker to process the caching job. Make sure to set it as `:external_dependency` if the work performs any external request -(e.g. Kubernetes, Sentry). +(e.g. Kubernetes, Sentry); otherwise set it to `:no_dependency`. #### `self.reactive_cache_worker_finder` diff --git a/doc/development/redis.md b/doc/development/redis.md index d205082b9c6..a0ae84beb8d 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -68,6 +68,10 @@ which is enabled for the `cache` and `shared_state` ## Redis in structured logging +For GitLab Team Members: There are [basic](https://www.youtube.com/watch?v=Uhdj19Dc6vU) and +[advanced](https://youtu.be/jw1Wv2IJxzs) videos that show how you can work with the Redis +structured logging fields on GitLab.com. + Our [structured logging](logging.md#use-structured-json-logging) for web requests and Sidekiq jobs contains fields for the duration, call count, bytes written, and bytes read per Redis instance, along with a total for @@ -96,10 +100,14 @@ requests that read the most data from the cache, we can just sort by ### The slow log +TIP: **Tip:** +There is a [video showing how to see the slow log](https://youtu.be/BBI68QuYRH8) (GitLab internal) +on GitLab.com + On GitLab.com, entries from the [Redis slow log](https://redis.io/commands/slowlog) are available in the `pubsub-redis-inf-gprd*` index with the [`redis.slowlog` -tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). +tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time_s),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). This shows commands that have taken a long time and may be a performance concern. @@ -113,7 +121,7 @@ passing to fluentd (and ultimately Elasticsearch). The [Redis Keyspace Analyzer](https://gitlab.com/gitlab-com/gl-infra/redis-keyspace-analyzer) project contains tools for dumping the full key list and memory usage of a Redis -instance, and then analyzing those lists while elimating potentially sensitive +instance, and then analyzing those lists while eliminating potentially sensitive data from the results. It can be used to find the most frequent key patterns, or those that use the most memory. diff --git a/doc/development/reference_processing.md b/doc/development/reference_processing.md index 527fb94f228..cf587043cae 100644 --- a/doc/development/reference_processing.md +++ b/doc/development/reference_processing.md @@ -77,6 +77,22 @@ a minimum implementation of `AbstractReferenceFilter` should define: and an identifier, find the object. For example, this in a reference filter for merge requests, this might be `project.merge_requests.where(iid: iid)`. +### Add a new reference prefix and filter + +For reference filters for new objects, use a prefix format following the pattern +`^<object_type>#`, because: + +1. Varied single-character prefixes are hard for users to track. Especially for + lower-use object types, this can diminish value for the feature. +1. Suitable single-character prefixes are limited. +1. Following a consistent pattern allows users to infer the existence of new features. + +To add a reference prefix for a new object `apple`,which has both a name and ID, +format the reference as: + +- `^apple#123` for identification by ID. +- `^apple#"Granny Smith"` for identification by name. + ### Performance #### Find object optimization diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 1961d1dcc34..e35bda82aaa 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -1,3 +1,10 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + # Secure Coding Guidelines This document contains descriptions and guidelines for addressing security @@ -82,7 +89,8 @@ This Ruby Regex specialty can have security impact, as often regular expressions #### Examples -GitLab specific examples can be found [here](https://gitlab.com/gitlab-org/gitlab/-/issues/36029#note_251262187) and [there](https://gitlab.com/gitlab-org/gitlab/-/issues/33569). +GitLab-specific examples can be found in the following [path traversal](https://gitlab.com/gitlab-org/gitlab/-/issues/36029#note_251262187) +and [open redirect](https://gitlab.com/gitlab-org/gitlab/-/issues/33569) issues. Another example would be this fictional Ruby on Rails controller: @@ -392,5 +400,34 @@ In order to prevent Path Traversal vulnerabilities, user-controlled filenames or #### GitLab specific validations -- [`Gitlab::Utils.check_path_traversal`](https://gitlab.com/gitlab-org/security/gitlab/-/blob/master/lib/gitlab/utils.rb#L12-24) can be used to validate user input against Path Traversal vulnerabilities. Remember to add further validation when setting the `allowed_absolute` option to `true`. -- [`file_path` API validator](https://gitlab.com/gitlab-org/security/gitlab/-/blob/master/lib/api/validations/validators/file_path.rb) to validate user input when working with the Grape gem. +The methods `Gitlab::Utils.check_path_traversal!()` and `Gitlab::Utils.check_allowed_absolute_path!()` +can be used to validate user-supplied paths and prevent vulnerabilities. +`check_path_traversal!()` will detect their Path Traversal payloads and accepts URL-encoded paths. +`check_allowed_absolute_path!()` will check if a path is absolute and whether it is inside the allowed path list. By default, absolute +paths are not allowed, so you need to pass a list of allowed absolute paths to the `path_allowlist` +parameter when using `check_allowed_absolute_path!()`. + +To use a combination of both checks, follow the example below: + +```ruby +path = Gitlab::Utils.check_path_traversal!(path) +Gitlab::Utils.check_allowed_absolute_path!(path, path_allowlist) +``` + +In the REST API, we have the [`FilePath`](https://gitlab.com/gitlab-org/security/gitlab/-/blob/master/lib/api/validations/validators/file_path.rb) +validator that can be used to perform the checking on any file path argument the endpoints have. +It can be used as follows: + +```ruby +requires :file_path, type: String, file_path: { allowlist: ['/foo/bar/', '/home/foo/', '/app/home'] } +``` + +The Path Traversal check can also be used to forbid any absolute path: + +```ruby +requires :file_path, type: String, file_path: true +``` + +NOTE: **Note:** +Absolute paths are not allowed by default. If allowing an absolute path is required, you +need to provide an array of paths to the parameter `allowlist`. diff --git a/doc/development/shell_scripting_guide/index.md b/doc/development/shell_scripting_guide/index.md index 622c90d7a97..704b46c7efa 100644 --- a/doc/development/shell_scripting_guide/index.md +++ b/doc/development/shell_scripting_guide/index.md @@ -1,7 +1,5 @@ # Shell scripting standards and style guidelines -## Overview - GitLab consists of many various services and sub-projects. The majority of their backend code is written in [Ruby](https://www.ruby-lang.org) and [Go](https://golang.org). However, some of them use shell scripts for @@ -15,7 +13,7 @@ should be eventually harmonized with this guide. If there are any per-project deviations from this guide, they should be described in the `README.md` or `PROCESS.md` file for such a project. -### Avoid using shell scripts +## Avoid using shell scripts CAUTION: **Caution:** This is a must-read section. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index fdea27f43ca..24570cfc07b 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -215,6 +215,85 @@ From the rails console: Feature.enable!(:disable_authorized_projects_deduplication) ``` +## Limited capacity worker + +It is possible to limit the number of concurrent running jobs for a worker class +by using the `LimitedCapacity::Worker` concern. + +The worker must implement three methods: + +- `perform_work` - the concern implements the usual `perform` method and calls +`perform_work` if there is any capacity available. +- `remaining_work_count` - number of jobs that will have work to perform. +- `max_running_jobs` - maximum number of jobs allowed to run concurrently. + +```ruby +class MyDummyWorker + include ApplicationWorker + include LimitedCapacity::Worker + + def perform_work(*args) + end + + def remaining_work_count(*args) + 5 + end + + def max_running_jobs + 25 + end +end +``` + +Additional to the regular worker, a cron worker must be defined as well to +backfill the queue with jobs. the arguments passed to `perform_with_capacity` +will be passed along to the `perform_work` method. + +```ruby +class ScheduleMyDummyCronWorker + include ApplicationWorker + include CronjobQueue + + def perform(*args) + MyDummyWorker.perform_with_capacity(*args) + end +end +``` + +### How many jobs are running? + +It will be running `max_running_jobs` at almost all times. + +The cron worker will check the remaining capacity on each execution and it +will schedule at most `max_running_jobs` jobs. Those jobs on completion will +re-enqueue themselves immediately, but not on failure. The cron worker is in +charge of replacing those failed jobs. + +### Handling errors and idempotence + +This concern disables Sidekiq retries, logs the errors, and sends the job to the +dead queue. This is done to have only one source that produces jobs and because +the retry would occupy a slot with a job that will be performed in the distant future. + +We let the cron worker enqueue new jobs, this could be seen as our retry and +back off mechanism because the job might fail again if executed immediately. +This means that for every failed job, we will be running at a lower capacity +until the cron worker fills the capacity again. If it is important for the +worker not to get a backlog, exceptions must be handled in `#perform_work` and +the job should not raise. + +The jobs are deduplicated using the `:none` strategy, but the worker is not +marked as `idempotent!`. + +### Metrics + +This concern exposes three Prometheus metrics of gauge type with the worker class +name as label: + +- `limited_capacity_worker_running_jobs` +- `limited_capacity_worker_max_running_jobs` +- `limited_capacity_worker_remaining_work_count` + ## Job urgency Jobs can have an `urgency` attribute set, which can be `:high`, @@ -617,7 +696,7 @@ during deployment before all Rails and Sidekiq nodes have the updated code. #### Deprecate and remove an argument -**Before you remove arguments from the `perform_async` and `perform` methods.**, deprecate them. The +**Before you remove arguments from the `perform_async` and `perform` methods.**, deprecate them. The following example deprecates and then removes `arg2` from the `perform_async` method: 1. Provide a default value (usually `nil`) and use a comment to mark the diff --git a/doc/development/sql.md b/doc/development/sql.md index 3b969c7d27a..55a8192578b 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -220,7 +220,7 @@ Project.select(:id, :user_id).joins(:merge_requests) Never use ActiveRecord's `pluck` to pluck a set of values into memory only to use them as an argument for another query. For example, this will execute an -extra unecessary database query and load a lot of unecessary data into memory: +extra unnecessary database query and load a lot of unnecessary data into memory: ```ruby projects = Project.all.pluck(:id) diff --git a/doc/development/swapping_tables.md b/doc/development/swapping_tables.md index 5c62900dbff..f633721ea9d 100644 --- a/doc/development/swapping_tables.md +++ b/doc/development/swapping_tables.md @@ -1,3 +1,9 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Swapping Tables Sometimes you need to replace one table with another. For example, when diff --git a/doc/development/telemetry/event_dictionary.md b/doc/development/telemetry/event_dictionary.md index d8cc32ea8d0..53b7ddea095 100644 --- a/doc/development/telemetry/event_dictionary.md +++ b/doc/development/telemetry/event_dictionary.md @@ -1,32 +1,5 @@ --- -stage: Growth -group: Telemetry -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: '../product_analytics/event_dictionary.md' --- -# Event Dictionary - -**Note: We've temporarily moved the Event Dictionary to a [Google Sheet](https://docs.google.com/spreadsheets/d/1VzE8R72Px_Y_LlE3Z05LxUlG_dumWe3vl-HeUo70TPw/edit?usp=sharing)**. The previous Markdown table exceeded 600 rows making it difficult to manage. In the future, our intention is to move this back into our docs using a [YAML file](https://gitlab.com/gitlab-org/gitlab-docs/-/issues/823). - -The event dictionary is a single source of truth for the metrics and events we collect for product usage data. The Event Dictionary lists all the metrics and events we track, why we're tracking them, and where they are tracked. - -This is a living document that is updated any time a new event is planned or implemented. It includes the following information. - -- Section, stage, or group -- Description -- Implementation status -- Availability by plan type -- Code path - -We're currently focusing our Event Dictionary on [Usage Ping](usage_ping.md). In the future, we will also include [Snowplow](snowplow.md). We currently have an initiative across the entire product organization to complete the [Event Dictionary for Usage Ping](https://gitlab.com/groups/gitlab-org/-/epics/4174). - -## Instructions - -1. Open the Event Dictionary and fill in all the **PM to edit** columns highlighted in yellow. -1. Check that all the metrics and events are assigned to the correct section, stage, or group. If a metric is used across many groups, assign it to the stage. If a metric is used across many stages, assign it to the section. If a metric is incorrectly assigned to another section, stage, or group, let the PM know you have reassigned it. If your group has no assigned metrics and events, check that your metrics and events are not incorrectly assigned to another PM. -1. Add descriptions of what your metrics and events are tracking. Work with your Engineering team or the Telemetry team if you need help understanding this. -1. Add what plans this metric is available on. Work with your Engineering team or the Telemetry team if you need help understanding this. - -## Planned metrics and events - -For future metrics and events you plan to track, please add them to the Event Dictionary and note the status as `Planned`, `In Progress`, or `Implemented`. Once you have confirmed the metric has been implemented and have confirmed the metric data is in our data warehouse, change the status to **Data Available**. +This document was moved to [another location](../product_analytics/event_dictionary.md). diff --git a/doc/development/telemetry/index.md b/doc/development/telemetry/index.md index b5032ce3730..79ea80a52ea 100644 --- a/doc/development/telemetry/index.md +++ b/doc/development/telemetry/index.md @@ -1,182 +1,5 @@ --- -stage: Growth -group: Telemetry -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: '../product_analytics/index.md' --- -# Telemetry Guide - -At GitLab, we collect product usage data for the purpose of helping us build a better product. Data helps GitLab understand which parts of the product need improvement and which features we should build next. Product usage data also helps our team better understand the reasons why people use GitLab. With this knowledge we are able to make better product decisions. - -We encourage users to enable tracking, and we embrace full transparency with our tracking approach so it can be easily understood and trusted. - -By enabling tracking, users can: - -- Contribute back to the wider community. -- Help GitLab improve on the product. - -## Our tracking tools - -We use three methods to gather product usage data: - -- [Snowplow](#snowplow) -- [Usage Ping](#usage-ping) -- [Database import](#database-import) - -### Snowplow - -Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way -users engage with our website and application. - -Snowplow consists of two components: - -- [Snowplow JS](https://github.com/snowplow/snowplow/wiki/javascript-tracker) tracks client-side - events. -- [Snowplow Ruby](https://github.com/snowplow/snowplow/wiki/ruby-tracker) tracks server-side events. - -For more details, read the [Snowplow](snowplow.md) guide. - -### Usage Ping - -Usage Ping is a method for GitLab Inc to collect usage data on a GitLab instance. Usage Ping is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. This high-level data is used to help our product, support, and sales teams. - -For more details, read the [Usage Ping](usage_ping.md) guide. - -### Database import - -Database imports are full imports of data into GitLab's data warehouse. For GitLab.com, the PostgreSQL database is loaded into Snowflake data warehouse every 6 hours. For more details, see the [data team handbook](https://about.gitlab.com/handbook/business-ops/data-team/platform/#extract-and-load). - -## What data can be tracked - -Our different tracking tools allows us to track different types of events. The event types and examples of what data can be tracked are outlined below. - -The availability of event types and their tracking tools varies by segment. For example, on Self-Managed Users, we only have reporting using Database records via Usage Ping. - -| Event Types | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | -|----------------------------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| -| Snowplow (JS Pageview events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | -| Snowplow (JS UI events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | -| Snowplow (Ruby Pageview events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | -| Snowplow (Ruby CRUD / API events) | ✅ | 📅 | 📅 | ✅ | 📅 | 📅 | 📅 | 📅 | 📅 | 📅 | -| Usage Ping (Redis UI counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | -| Usage Ping (Redis Pageview counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | -| Usage Ping (Redis CRUD / API counters) | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | 🔄 | 🔄 | 🔄 | ✖️ | 🔄 | -| Usage Ping (Database counters) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | -| Usage Ping (Instance settings) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | -| Usage Ping (Integration settings) | ✅ | 🔄 | 📅 | ✖️ | ✅ | ✅ | ✅ | ✅ | ✖️ | ✅ | -| Database import (Database records) | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | - -[Source file](https://docs.google.com/spreadsheets/d/1e8Afo41Ar8x3JxAXJF3nL83UxVZ3hPIyXdt243VnNuE/edit?usp=sharing) - -**Legend** - -✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible - -SaaS = GitLab.com. SM = Self-Managed instance - -### Pageview events - -- Number of sessions that visited the /dashboard/groups page - -### UI events - -- Number of sessions that clicked on a button or link -- Number of sessions that closed a modal - -UI events are any interface-driven actions from the browser including click data. - -### CRUD or API events - -- Number of Git pushes -- Number of GraphQL queries -- Number of requests to a Rails action or controller - -These are backend events that include the creation, read, update, deletion of records, and other events that might be triggered from layers other than those available in the interface. - -### Database records - -These are raw database records which can be explored using business intelligence tools like Sisense. The full list of available tables can be found in [structure.sql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/structure.sql). - -### Instance settings - -These are settings of your instance such as the instance's Git version and if certain features are enabled such as `container_registry_enabled`. - -### Integration settings - -These are integrations your GitLab instance interacts with such as an [external storage provider](../../administration/static_objects_external_storage.md) or an [external container registry](../../administration/packages/container_registry.md#use-an-external-container-registry-with-gitlab-as-an-auth-endpoint). These services must be able to send data back into a GitLab instance for data to be tracked. - -## Reporting level - -Our reporting levels of aggregate or individual reporting varies by segment. For example, on Self-Managed Users, we can report at an aggregate user level using Usage Ping but not on an Individual user level. - -| Aggregated Reporting | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | -|----------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| -| Snowplow | ✅ | 📅 | 📅 | ✅ | 📅 | ✅ | 📅 | 📅 | ✅ | 📅 | -| Usage Ping | ✅ | 🔄 | 📅 | 📅 | ✅ | ✅ | ✅ | ✅ | 📅 | ✅ | -| Database import | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | - -| Identifiable Reporting | SaaS Instance | SaaS Plan | SaaS Group | SaaS Session | SaaS User | SM Instance | SM Plan | SM Group | SM Session | SM User | -|------------------------|---------------|-----------|------------|--------------|-----------|-------------|---------|----------|------------|---------| -| Snowplow | ✅ | 📅 | 📅 | ✅ | 📅 | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | -| Usage Ping | ✅ | 🔄 | 📅 | ✖️ | ✖️ | ✅ | ✅ | ✖️ | ✖️ | ✖️ | -| Database import | ✅ | ✅ | ✅ | ✖️ | ✅ | ✖️ | ✖️ | ✖️ | ✖️ | ✖️ | - -**Legend** - -✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible - -SaaS = GitLab.com. SM = Self-Managed instance - -## Reporting time period - -Our reporting time periods varies by segment. For example, on Self-Managed Users, we can report all time counts and 28 day counts in Usage Ping. - -| Reporting Time Period | All Time | 28 Days | 7 Days | Daily | -|-----------------------|----------|---------|--------|-------| -| Snowplow | ✅ | ✅ | ✅ | ✅ | -| Usage Ping | ✅ | ✅ | 📅 | ✖️ | -| Database import | ✅ | ✅ | ✅ | ✅ | - -**Legend** - -✅ Available, 🔄 In Progress, 📅 Planned, ✖️ Not Possible - -## Systems overview - -The systems overview is a simplified diagram showing the interactions between GitLab Inc and self-managed instances. - - - -[Source file](https://app.diagrams.net/#G13DVpN-XnhWGz9tqReIj8pp1UE4ehk_EC) - -### GitLab Inc - -For Telemetry purposes, GitLab Inc has three major components: - -1. [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/): This contains everything managed by our data team including Sisense Dashboards for visualization, Snowflake for Data Warehousing, incoming data sources such as PostgreSQL Pipeline and S3 Bucket, and lastly our data collectors [GitLab.com's Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/tree/master/library/snowplow/) and GitLab's Versions Application. -1. GitLab.com: This is the production GitLab application which is made up of a Client and Server. On the Client or browser side, a Snowplow JS Tracker (Frontend) is used to track client-side events. On the Server or application side, a Snowplow Ruby Tracker (Backend) is used to track server-side events. The server also contains Usage Ping which leverages a PostgreSQL database and a Redis in-memory data store to report on usage data. Lastly, the server also contains System Logs which are generated from running the GitLab application. -1. [Monitoring infrastructure](https://about.gitlab.com/handbook/engineering/monitoring/): This is the infrastructure used to ensure GitLab.com is operating smoothly. System Logs are sent from GitLab.com to our monitoring infrastructure and collected by a FluentD collector. From FluentD, logs are either sent to long term Google Cloud Services cold storage via Stackdriver, or, they are sent to our Elastic Cluster via Cloud Pub/Sub which can be explored in real-time using Kibana. - -### Self-managed - -For Telemetry purposes, self-managed instances have two major components: - -1. Data infrastructure: Having a data infrastructure setup is optional on self-managed instances. If you'd like to collect Snowplow tracking events for your self-managed instance, you can setup your own self-managed Snowplow collector and configure your Snowplow events to point to your own collector. -1. GitLab: A self-managed GitLab instance contains all of the same components as GitLab.com mentioned above. - -### Differences between GitLab Inc and Self-managed - -As shown by the orange lines, on GitLab.com Snowplow JS, Snowplow Ruby, Usage Ping, and PostgreSQL database imports all flow into GitLab Inc's data infrastructure. However, on self-managed, only Usage Ping flows into GitLab Inc's data infrastructure. - -As shown by the green lines, on GitLab.com system logs flow into GitLab Inc's monitoring infrastructure. On self-managed, there are no logs sent to GitLab Inc's monitoring infrastructure. - -Note (1): Snowplow JS and Snowplow Ruby are available on self-managed, however, the Snowplow Collector endpoint is set to a self-managed Snowplow Collector which GitLab Inc does not have access to. - -## Additional information - -More useful links: - -- [Telemetry Direction](https://about.gitlab.com/direction/telemetry/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) +This document was moved to [another location](../product_analytics/index.md). diff --git a/doc/development/telemetry/snowplow.md b/doc/development/telemetry/snowplow.md index f427d7d1488..fd75d29286b 100644 --- a/doc/development/telemetry/snowplow.md +++ b/doc/development/telemetry/snowplow.md @@ -1,410 +1,5 @@ --- -stage: Growth -group: Telemetry -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: '../product_analytics/snowplow.md' --- -# Snowplow Guide - -This guide provides an overview of how Snowplow works, and implementation details. - -For more information about Telemetry, see: - -- [Telemetry Guide](index.md) -- [Usage Ping Guide](usage_ping.md) - -More useful links: - -- [Telemetry Direction](https://about.gitlab.com/direction/telemetry/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) - -## What is Snowplow - -Snowplow is an enterprise-grade marketing and product analytics platform which helps track the way users engage with our website and application. - -[Snowplow](https://github.com/snowplow/snowplow) consists of the following loosely-coupled sub-systems: - -- **Trackers** fire Snowplow events. Snowplow has 12 trackers, covering web, mobile, desktop, server, and IoT. -- **Collectors** receive Snowplow events from trackers. We have three different event collectors, synchronizing events either to Amazon S3, Apache Kafka, or Amazon Kinesis. -- **Enrich** cleans up the raw Snowplow events, enriches them and puts them into storage. We have an Hadoop-based enrichment process, and a Kinesis-based or Kafka-based process. -- **Storage** is where the Snowplow events live. We store the Snowplow events in a flat file structure on S3, and in the Redshift and PostgreSQL databases. -- **Data modeling** is where event-level data is joined with other data sets and aggregated into smaller data sets, and business logic is applied. This produces a clean set of tables which make it easier to perform analysis on the data. We have data models for Redshift and Looker. -- **Analytics** are performed on the Snowplow events or on the aggregate tables. - - - -## Snowplow schema - -We have many definitions of Snowplow's schema. We have an active issue to [standardize this schema](https://gitlab.com/gitlab-org/gitlab/-/issues/207930) including the following definitions: - -- Frontend and backend taxonomy as listed below -- [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy) -- [Self describing events](https://github.com/snowplow/snowplow/wiki/Custom-events#self-describing-events) -- [Iglu schema](https://gitlab.com/gitlab-org/iglu/) -- [Snowplow authored events](https://github.com/snowplow/snowplow/wiki/Snowplow-authored-events) - -## Enabling Snowplow - -Tracking can be enabled at: - -- The instance level, which enables tracking on both the frontend and backend layers. -- User level, though user tracking can be disabled on a per-user basis. GitLab tracking respects the [Do Not Track](https://www.eff.org/issues/do-not-track) standard, so any user who has enabled the Do Not Track option in their browser is not tracked at a user level. - -We utilize Snowplow for the majority of our tracking strategy and it is enabled on GitLab.com. On a self-managed instance, Snowplow can be enabled by navigating to: - -- **Admin Area > Settings > Integrations** in the UI. -- `admin/application_settings/integrations` in your browser. - -The following configuration is required: - -| Name | Value | -|---------------|---------------------------| -| Collector | `snowplow.trx.gitlab.net` | -| Site ID | `gitlab` | -| Cookie domain | `.gitlab.com` | - -## Snowplow request flow - -The following example shows a basic request/response flow between the following components: - -- Snowplow JS / Ruby Trackers on GitLab.com -- [GitLab.com Snowplow Collector](https://gitlab.com/gitlab-com/gl-infra/readiness/-/blob/master/library/snowplow/index.md) -- GitLab's S3 Bucket -- GitLab's Snowflake Data Warehouse -- Sisense: - -```mermaid -sequenceDiagram - participant Snowplow JS (Frontend) - participant Snowplow Ruby (Backend) - participant GitLab.com Snowplow Collector - participant S3 Bucket - participant Snowflake DW - participant Sisense Dashboards - Snowplow JS (Frontend) ->> GitLab.com Snowplow Collector: FE Tracking event - Snowplow Ruby (Backend) ->> GitLab.com Snowplow Collector: BE Tracking event - loop Process using Kinesis Stream - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Log raw events - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Enrich events - GitLab.com Snowplow Collector ->> GitLab.com Snowplow Collector: Write to disk - end - GitLab.com Snowplow Collector ->> S3 Bucket: Kinesis Firehose - S3 Bucket->>Snowflake DW: Import data - Snowflake DW->>Snowflake DW: Transform data using dbt - Snowflake DW->>Sisense Dashboards: Data available for querying -``` - -## Implementing Snowplow JS (Frontend) tracking - -GitLab provides `Tracking`, an interface that wraps the [Snowplow JavaScript Tracker](https://github.com/snowplow/snowplow/wiki/javascript-tracker) for tracking custom events. There are a few ways to utilize tracking, but each generally requires at minimum, a `category` and an `action`. Additional data can be provided that adheres to our [Feature instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). - -| field | type | default value | description | -|:-----------|:-------|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `category` | string | document.body.dataset.page | Page or subsection of a page that events are being captured within. | -| `action` | string | 'generic' | Action the user is taking. Clicks should be `click` and activations should be `activate`, so for example, focusing a form field would be `activate_form_input`, and clicking a button would be `click_button`. | -| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). | - -### Tracking in HAML (or Vue Templates) - -When working within HAML (or Vue templates) we can add `data-track-*` attributes to elements of interest. All elements that have a `data-track-event` attribute automatically have event tracking bound on clicks. - -Below is an example of `data-track-*` attributes assigned to a button: - -```haml -%button.btn{ data: { track: { event: "click_button", label: "template_preview", property: "my-template" } } } -``` - -```html -<button class="btn" - data-track-event="click_button" - data-track-label="template_preview" - data-track-property="my-template" -/> -``` - -Event listeners are bound at the document level to handle click events on or within elements with these data attributes. This allows them to be properly handled on re-rendering and changes to the DOM. Note that because of the way these events are bound, click events should not be stopped from propagating up the DOM tree. If for any reason click events are being stopped from propagating, you need to implement your own listeners and follow the instructions in [Tracking in raw JavaScript](#tracking-in-raw-javascript). - -Below is a list of supported `data-track-*` attributes: - -| attribute | required | description | -|:----------------------|:---------|:------------| -| `data-track-event` | true | Action the user is taking. Clicks must be prepended with `click` and activations must be prepended with `activate`. For example, focusing a form field would be `activate_form_input` and clicking a button would be `click_button`. | -| `data-track-label` | false | The `label` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). | -| `data-track-property` | false | The `property` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). | -| `data-track-value` | false | The `value` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). If omitted, this is the element's `value` property or an empty string. For checkboxes, the default value is the element's checked attribute or `false` when unchecked. | -| `data-track-context` | false | The `context` as described [in our Feature Instrumentation taxonomy](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). | - -### Tracking within Vue components - -There's a tracking Vue mixin that can be used in components if more complex tracking is required. To use it, first import the `Tracking` library and request a mixin. - -```javascript -import Tracking from '~/tracking'; -const trackingMixin = Tracking.mixin({ label: 'right_sidebar' }); -``` - -You can provide default options that are passed along whenever an event is tracked from within your component. For instance, if all events within a component should be tracked with a given `label`, you can provide one at this time. Available defaults are `category`, `label`, `property`, and `value`. If no category is specified, `document.body.dataset.page` is used as the default. - -You can then use the mixin normally in your component with the `mixin` Vue declaration. The mixin also provides the ability to specify tracking options in `data` or `computed`. These override any defaults and allow the values to be dynamic from props, or based on state. - -```javascript -export default { - mixins: [trackingMixin], - // ...[component implementation]... - data() { - return { - expanded: false, - tracking: { - label: 'left_sidebar' - } - }; - }, -} -``` - -The mixin provides a `track` method that can be called within the template, or from component methods. An example of the whole implementation might look like the following. - -```javascript -export default { - mixins: [Tracking.mixin({ label: 'right_sidebar' })], - data() { - return { - expanded: false, - }; - }, - methods: { - toggle() { - this.expanded = !this.expanded; - this.track('click_toggle', { value: this.expanded }) - } - } -}; -``` - -And if needed within the template, you can use the `track` method directly as well. - -```html -<template> - <div> - <a class="toggle" @click.prevent="toggle">Toggle</a> - <div v-if="expanded"> - <p>Hello world!</p> - <a @click.prevent="track('click_action')">Track an event</a> - </div> - </div> -</template> -``` - -### Tracking in raw JavaScript - -Custom event tracking and instrumentation can be added by directly calling the `Tracking.event` static function. The following example demonstrates tracking a click on a button by calling `Tracking.event` manually. - -```javascript -import Tracking from '~/tracking'; - -const button = document.getElementById('create_from_template_button'); -button.addEventListener('click', () => { - Tracking.event('dashboard:projects:index', 'click_button', { - label: 'create_from_template', - property: 'template_preview', - value: 'rails', - }); -}) -``` - -### Tests and test helpers - -In Jest particularly in Vue tests, you can use the following: - -```javascript -import { mockTracking } from 'helpers/tracking_helper'; - -describe('MyTracking', () => { - let spy; - - beforeEach(() => { - spy = mockTracking('_category_', wrapper.element, jest.spyOn); - }); - - it('tracks an event when clicked on feedback', () => { - wrapper.find('.discover-feedback-icon').trigger('click'); - - expect(spy).toHaveBeenCalledWith('_category_', 'click_button', { - label: 'security-discover-feedback-cta', - property: '0', - }); - }); -}); -``` - -In obsolete Karma tests it's used as below: - -```javascript -import { mockTracking, triggerEvent } from 'spec/helpers/tracking_helper'; - -describe('my component', () => { - let trackingSpy; - - beforeEach(() => { - trackingSpy = mockTracking('_category_', vm.$el, spyOn); - }); - - const triggerEvent = () => { - // action which should trigger a event - }; - - it('tracks an event when toggled', () => { - expect(trackingSpy).not.toHaveBeenCalled(); - - triggerEvent('a.toggle'); - - expect(trackingSpy).toHaveBeenCalledWith('_category_', 'click_edit_button', { - label: 'right_sidebar', - property: 'confidentiality', - }); - }); -}); -``` - -## Implementing Snowplow Ruby (Backend) tracking - -GitLab provides `Gitlab::Tracking`, an interface that wraps the [Snowplow Ruby Tracker](https://github.com/snowplow/snowplow/wiki/ruby-tracker) for tracking custom events. - -Custom event tracking and instrumentation can be added by directly calling the `GitLab::Tracking.event` class method, which accepts the following arguments: - -| argument | type | default value | description | -|:-----------|:-------|:--------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `category` | string | 'application' | Area or aspect of the application. This could be `HealthCheckController` or `Lfs::FileTransformer` for instance. | -| `action` | string | 'generic' | The action being taken, which can be anything from a controller action like `create` to something like an Active Record callback. | -| `data` | object | {} | Additional data such as `label`, `property`, `value`, and `context` as described in [Instrumentation at GitLab](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/#sts=Taxonomy). These are set as empty strings if you don't provide them. | - -Tracking can be viewed as either tracking user behavior, or can be utilized for instrumentation to monitor and visualize performance over time in an area or aspect of code. - -For example: - -```ruby -class Projects::CreateService < BaseService - def execute - project = Project.create(params) - - Gitlab::Tracking.event('Projects::CreateService', 'create_project', - label: project.errors.full_messages.to_sentence, - value: project.valid? - ) - end -end -``` - -### Performance - -We use the [AsyncEmitter](https://github.com/snowplow/snowplow/wiki/Ruby-Tracker#52-the-asyncemitter-class) when tracking events, which allows for instrumentation calls to be run in a background thread. This is still an active area of development. - -## Developing and testing Snowplow - -There are several tools for developing and testing Snowplow Event - -| Testing Tool | Frontend Tracking | Backend Tracking | Local Development Environment | Production Environment | Production Environment | -|----------------------------------------------|--------------------|---------------------|-------------------------------|------------------------|------------------------| -| Snowplow Analytics Debugger Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Inspector Chrome Extension | **{check-circle}** | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| Snowplow Micro | **{check-circle}** | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{dotted-circle}** | -| Snowplow Mini | **{check-circle}** | **{check-circle}** | **{dotted-circle}** | **{status_preparing}** | **{status_preparing}** | - -**Legend** - -**{check-circle}** Available, **{status_preparing}** In progress, **{dotted-circle}** Not Planned - -### Preparing your MR for Review - -1. For frontend events, in the MR description section, add a screenshot of the event's relevant section using the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. -1. For backend events, please use Snowplow Micro and add the output of the Snowplow Micro good events `GET http://localhost:9090/micro/good`. - -### Snowplow Analytics Debugger Chrome Extension - -Snowplow Analytics Debugger is a browser extension for testing frontend events. This works on production, staging and local development environments. - -1. Install the [Snowplow Analytics Debugger](https://chrome.google.com/webstore/detail/snowplow-analytics-debugg/jbnlcgeengmijcghameodeaenefieedm) Chrome browser extension. -1. Open Chrome DevTools to the Snowplow Analytics Debugger tab. -1. Learn more at [Igloo Analytics](https://www.iglooanalytics.com/blog/snowplow-analytics-debugger-chrome-extension.html). - -### Snowplow Inspector Chrome Extension - -Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works on production, staging and local development environments. - -1. Install [Snowplow Inspector](https://chrome.google.com/webstore/detail/snowplow-inspector/maplkdomeamdlngconidoefjpogkmljm?hl=en). -1. Open the Chrome extension by pressing the Snowplow Inspector icon beside the address bar. -1. Click around on a webpage with Snowplow and you should see JavaScript events firing in the inspector window. - -### Snowplow Micro - -Snowplow Micro is a very small version of a full Snowplow data collection pipeline: small enough that it can be launched by a test suite. Events can be recorded into Snowplow Micro just as they can a full Snowplow pipeline. Micro then exposes an API that can be queried. - -Snowplow Micro is a Docker-based solution for testing frontend and backend events in a local development environment. You need to modify GDK using the instructions below to set this up. - -- Read [Introducing Snowplow Micro](https://snowplowanalytics.com/blog/2019/07/17/introducing-snowplow-micro/) -- Look at the [Snowplow Micro repository](https://github.com/snowplow-incubator/snowplow-micro) -- Watch our [installation guide recording](https://www.youtube.com/watch?v=OX46fo_A0Ag) - -1. Install [Snowplow Micro](https://github.com/snowplow-incubator/snowplow-micro): - - ```shell - docker run --mount type=bind,source=$(pwd)/example,destination=/config -p 9090:9090 snowplow/snowplow-micro:latest --collector-config /config/micro.conf --iglu /config/iglu.json - ``` - -1. Install snowplow micro by cloning the settings in [this project](https://gitlab.com/a_akgun/snowplow-micro): - - ```shell - git clone git@gitlab.com:a_akgun/snowplow-micro.git - ./snowplow-micro.sh - ``` - -1. Update port in SQL to set `9090`: - - ```shell - gdk psql -d gitlabhq_development - update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; - ``` - -1. Update `app/assets/javascripts/tracking.js` to [remove this line](https://gitlab.com/snippets/1918635): - - ```javascript - forceSecureTracker: true - ``` - -1. Update `lib/gitlab/tracking.rb` to [add these lines](https://gitlab.com/snippets/1918635): - - ```ruby - protocol: 'http', - port: 9090, - ``` - -1. Update `lib/gitlab/tracking.rb` to [change async emitter from https to http](https://gitlab.com/snippets/1918635): - - ```ruby - SnowplowTracker::AsyncEmitter.new(Gitlab::CurrentSettings.snowplow_collector_hostname, protocol: 'http'), - ``` - -1. Enable Snowplow in the admin area, Settings::Integrations::Snowplow to point to: - `http://localhost:3000/admin/application_settings/integrations#js-snowplow-settings`. - -1. Restart GDK: - - ```shell - `gdk restart` - ``` - -1. Send a test Snowplow event from the Rails console: - - ```ruby - Gitlab::Tracking.self_describing_event('iglu:com.gitlab/pageview_context/jsonschema/1-0-0', { page_type: 'MY_TYPE' }, context: nil ) - ``` - -### Snowplow Mini - -[Snowplow Mini](https://github.com/snowplow/snowplow-mini) is an easily-deployable, single-instance version of Snowplow. - -Snowplow Mini can be used for testing frontend and backend events on a production, staging and local development environment. - -For GitLab.com, we're setting up a [QA and Testing environment](https://gitlab.com/gitlab-org/telemetry/-/issues/266) using Snowplow Mini. +This document was moved to [another location](../product_analytics/snowplow.md). diff --git a/doc/development/telemetry/usage_ping.md b/doc/development/telemetry/usage_ping.md index ff4e7e0797b..1026e7a5a66 100644 --- a/doc/development/telemetry/usage_ping.md +++ b/doc/development/telemetry/usage_ping.md @@ -1,887 +1,5 @@ --- -stage: Growth -group: Telemetry -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +redirect_to: '../product_analytics/usage_ping.md' --- -# Usage Ping Guide - -> - Introduced in GitLab Enterprise Edition 8.10. -> - More statistics were added in GitLab Enterprise Edition 8.12. -> - Moved to GitLab Core in 9.1. -> - More statistics were added in GitLab Ultimate 11.2. - -This guide describes Usage Ping's purpose and how it's implemented. - -For more information about Telemetry, see: - -- [Telemetry Guide](index.md) -- [Snowplow Guide](snowplow.md) - -More useful links: - -- [Telemetry Direction](https://about.gitlab.com/direction/telemetry/) -- [Data Analysis Process](https://about.gitlab.com/handbook/business-ops/data-team/#data-analysis-process/) -- [Data for Product Managers](https://about.gitlab.com/handbook/business-ops/data-team/programs/data-for-product-managers/) -- [Data Infrastructure](https://about.gitlab.com/handbook/business-ops/data-team/platform/infrastructure/) - -## What is Usage Ping? - -- GitLab sends a weekly payload containing usage data to GitLab Inc. Usage Ping provides high-level data to help our product, support, and sales teams. It does not send any project names, usernames, or any other specific data. The information from the usage ping is not anonymous, it is linked to the hostname of the instance. Sending usage ping is optional, and any instance can disable analytics. -- The usage data is primarily composed of row counts for different tables in the instance’s database. By comparing these counts month over month (or week over week), we can get a rough sense for how an instance is using the different features within the product. In addition to counts, other facts - that help us classify and understand GitLab installations are collected. -- Usage ping is important to GitLab as we use it to calculate our Stage Monthly Active Users (SMAU) which helps us measure the success of our stages and features. -- Once usage ping is enabled, GitLab will gather data from the other instances and will be able to show usage statistics of your instance to your users. - -### Why should we enable Usage Ping? - -- The main purpose of Usage Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. -- As a benefit of having the usage ping active, GitLab lets you analyze the users’ activities over time of your GitLab installation. -- As a benefit of having the usage ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance’s adoption of Concurrent DevOps from planning to monitoring. -- You will get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) -- You will get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? -- You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. -- Usage Ping is enabled by default. To disable it, see [Disable Usage Ping](#disable-usage-ping). - -### Limitations - -- Usage Ping does not track frontend events things like page views, link clicks, or user sessions, and only focuses on aggregated backend events. -- Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Usage Ping to track aggregated backend events on self-managed. - -## Usage Ping payload - -You can view the exact JSON payload sent to GitLab Inc. in the administration panel. To view the payload: - -1. Navigate to **Admin Area > Settings > Metrics and profiling**. -1. Expand the **Usage statistics** section. -1. Click the **Preview payload** button. - -For an example payload, see [Example Usage Ping payload](#example-usage-ping-payload). - -## Disable Usage Ping - -To disable Usage Ping in the GitLab UI, go to the **Settings** page of your administration panel and uncheck the **Usage Ping** checkbox. - -To disable Usage Ping and prevent it from being configured in the future through the administration panel, Omnibus installs can set the following in [`gitlab.rb`](https://docs.gitlab.com/omnibus/settings/configuration.html#configuration-options): - -```ruby -gitlab_rails['usage_ping_enabled'] = false -``` - -Source installations can set the following in `gitlab.yml`: - -```yaml -production: &base - # ... - gitlab: - # ... - usage_ping_enabled: false -``` - -## Usage Ping request flow - -The following example shows a basic request/response flow between a GitLab instance, the Versions Application, the License Application, Salesforce, GitLab's S3 Bucket, GitLab's Snowflake Data Warehouse, and Sisense: - -```mermaid -sequenceDiagram - participant GitLab Instance - participant Versions Application - participant Licenses Application - participant Salesforce - participant S3 Bucket - participant Snowflake DW - participant Sisense Dashboards - GitLab Instance->>Versions Application: Send Usage Ping - loop Process usage data - Versions Application->>Versions Application: Parse usage data - Versions Application->>Versions Application: Write to database - Versions Application->>Versions Application: Update license ping time - end - loop Process data for Salesforce - Versions Application-xLicenses Application: Request Zuora subscription id - Licenses Application-xVersions Application: Zuora subscription id - Versions Application-xSalesforce: Request Zuora account id by Zuora subscription id - Salesforce-xVersions Application: Zuora account id - Versions Application-xSalesforce: Usage data for the Zuora account - end - Versions Application->>S3 Bucket: Export Versions database - S3 Bucket->>Snowflake DW: Import data - Snowflake DW->>Snowflake DW: Transform data using dbt - Snowflake DW->>Sisense Dashboards: Data available for querying - Versions Application->>GitLab Instance: DevOps Report (Conversational Development Index) -``` - -## How Usage Ping works - -1. The Usage Ping [cron job](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/gitlab_usage_ping_worker.rb#L30) is set in Sidekiq to run weekly. -1. When the cron job runs, it calls [`GitLab::UsageData.to_json`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L22). -1. `GitLab::UsageData.to_json` [cascades down](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L22) to ~400+ other counter method calls. -1. The response of all methods calls are [merged together](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data.rb#L14) into a single JSON payload in `GitLab::UsageData.to_json`. -1. The JSON payload is then [posted to the Versions application]( https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/submit_usage_ping_service.rb#L20). - -## Implementing Usage Ping - -Usage Ping consists of two kinds of data, counters and observations. Counters track how often a certain event -happened over time, such as how many CI pipelines have run. They are monotonic and always trend up. -Observations are facts collected from one or more GitLab instances and can carry arbitrary data. There are no -general guidelines around how to collect those, due to the individual nature of that data. - -There are four types of counters which are all found in `usage_data.rb`: - -- **Ordinary Batch Counters:** Simple count of a given ActiveRecord_Relation -- **Distinct Batch Counters:** Distinct count of a given ActiveRecord_Relation on given column -- **Alternative Counters:** Used for settings and configurations -- **Redis Counters:** Used for in-memory counts. - -NOTE: **Note:** -Only use the provided counter methods. Each counter method contains a built in fail safe to isolate each counter to avoid breaking the entire Usage Ping. - -### Why batch counting - -For large tables, PostgreSQL can take a long time to count rows due to MVCC [(Multi-version Concurrency Control)](https://en.wikipedia.org/wiki/Multiversion_concurrency_control). Batch counting is a counting method where a single large query is broken into multiple smaller queries. For example, instead of a single query querying 1,000,000 records, with batch counting, you can execute 100 queries of 10,000 records each. Batch counting is useful for avoiding database timeouts as each batch query is significantly shorter than one single long running query. - -For GitLab.com, there are extremely large tables with 15 second query timeouts, so we use batch counting to avoid encountering timeouts. Here are the sizes of some GitLab.com tables: - -| Table | Row counts in millions | -|------------------------------|------------------------| -| `merge_request_diff_commits` | 2280 | -| `ci_build_trace_sections` | 1764 | -| `merge_request_diff_files` | 1082 | -| `events` | 514 | - -There are two batch counting methods provided, `Ordinary Batch Counters` and `Distinct Batch Counters`. Batch counting requires indexes on columns to calculate max, min, and range queries. In some cases, a specialized index may need to be added on the columns involved in a counter. - -### Ordinary Batch Counters - -Handles `ActiveRecord::StatementInvalid` error - -Simple count of a given ActiveRecord_Relation, does a non-distinct batch count, smartly reduces batch_size and handles errors. - -Method: `count(relation, column = nil, batch: true, start: nil, finish: nil)` - -Arguments: - -- `relation` the ActiveRecord_Relation to perform the count -- `column` the column to perform the count on, by default is the primary key -- `batch`: default `true` in order to use batch counting -- `start`: custom start of the batch counting in order to avoid complex min calculations -- `end`: custom end of the batch counting in order to avoid complex min calculations - -Examples: - -```ruby -count(User.active) -count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) -count(::Clusters::Cluster.aws_installed.enabled, :cluster_id, start: ::Clusters::Cluster.minimum(:id), finish: ::Clusters::Cluster.maximum(:id)) -``` - -### Distinct Batch Counters - -Handles `ActiveRecord::StatementInvalid` error - -Distinct count of a given ActiveRecord_Relation on given column, a distinct batch count, smartly reduces batch_size and handles errors. - -Method: `distinct_count(relation, column = nil, batch: true, batch_size: nil, start: nil, finish: nil)` - -Arguments: - -- `relation` the ActiveRecord_Relation to perform the count -- `column` the column to perform the distinct count, by default is the primary key -- `batch`: default `true` in order to use batch counting -- `batch_size`: if none set it will use default value 10000 from `Gitlab::Database::BatchCounter` -- `start`: custom start of the batch counting in order to avoid complex min calculations -- `end`: custom end of the batch counting in order to avoid complex min calculations - -Examples: - -```ruby -distinct_count(::Project, :creator_id) -distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) -distinct_count(::Clusters::Applications::CertManager.where(time_period).available.joins(:cluster), 'clusters.user_id') -``` - -### Redis Counters - -Handles `::Redis::CommandError` and `Gitlab::UsageDataCounters::BaseCounter::UnknownEvent` -returns -1 when a block is sent or hash with all values -1 when a `counter(Gitlab::UsageDataCounters)` is sent -different behavior due to 2 different implementations of Redis counter - -Method: `redis_usage_data(counter, &block)` - -Arguments: - -- `counter`: a counter from `Gitlab::UsageDataCounters`, that has `fallback_totals` method implemented -- or a `block`: which is evaluated - -#### Ordinary Redis Counters - -Examples of implementation: - -- Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb) -- Using Redis methods [`HINCRBY`](https://redis.io/commands/hincrby), [`HGETALL`](https://redis.io/commands/hgetall), and [`Gitlab::UsageCounters::PodLogs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_counters/pod_logs.rb) - -#### Redis HLL Counters - -With `Gitlab::UsageDataCounters::HLLRedisCounter` we have available data structures used to count unique values. - -Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PFCOUNT](https://redis.io/commands/pfcount). - -##### Adding new events - -1. Define events in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml). - - Example event: - - ```yaml - - name: i_compliance_credential_inventory - category: compliance - redis_slot: compliance - expiry: 42 # 6 weeks - aggregation: weekly - ``` - - Keys: - - - `name`: unique event name. - - Name format `<prefix>_<redis_slot>_name`. - - Use one of the following prefixes for the event's name: - - - `g_` for group, as an event which is tracked for group. - - `p_` for project, as an event which is tracked for project. - - `i_` for instance, as an event which is tracked for instance. - - `a_` for events encompassing all `g_`, `p_`, `i_`. - - `o_` for other. - - Consider including in the event's name the Redis slot in order to be able to count totals for a specific category. - - Example names: `i_compliance_credential_inventory`, `g_analytics_contribution`. - - - `category`: event category. Used for getting total counts for events in a category, for easier - access to a group of events. - - `redis_slot`: optional Redis slot; default value: event name. Used if needed to calculate totals - for a group of metrics. Ensure keys are in the same slot. For example: - `i_compliance_credential_inventory` with `redis_slot: 'compliance'` will build Redis key - `i_{compliance}_credential_inventory-2020-34`. If `redis_slot` is not defined the Redis key will - be `{i_compliance_credential_inventory}-2020-34`. - - `expiry`: expiry time in days. Default: 29 days for daily aggregation and 6 weeks for weekly - aggregation. - - `aggregation`: aggregation `:daily` or `:weekly`. The argument defines how we build the Redis - keys for data storage. For `daily` we keep a key for metric per day of the year, for `weekly` we - keep a key for metric per week of the year. - -1. Track event in controller using `RedisTracking` module with `track_redis_hll_event(*controller_actions, name:, feature:, feature_default_enabled: false)`. - - Arguments: - - - `controller_actions`: controller actions we want to track. - - `name`: event name. - - `feature`: feature name, all metrics we track should be under feature flag. - - `feature_default_enabled`: feature flag is disabled by default, set to `true` for it to be enabled by default. - - Example usage: - - ```ruby - # controller - class ProjectsController < Projects::ApplicationController - include RedisTracking - - skip_before_action :authenticate_user!, only: :show - track_redis_hll_event :index, :show, name: 'i_analytics_dev_ops_score', feature: :g_compliance_dashboard_feature, feature_default_enabled: true - - def index - render html: 'index' - end - - def new - render html: 'new' - end - - def show - render html: 'show' - end - end - ``` - -1. Track event in API using `increment_unique_values(event_name, values)` helper method. - - In order to be able to track the event, Usage Ping must be enabled and the event feature `usage_data_<event_name>` must be enabled. - - Arguments: - - - `event_name`: event name. - - `values`: values counted, one value or array of values. - - Example usage: - - ```ruby - get ':id/registry/repositories' do - repositories = ContainerRepositoriesFinder.new( - user: current_user, subject: user_group - ).execute - - increment_unique_values('i_list_repositories', current_user.id) - - present paginate(repositories), with: Entities::ContainerRegistry::Repository, tags: params[:tags], tags_count: params[:tags_count] - end - ``` - -1. Track event using `track_usage_event(event_name, values) in services and graphql - - Increment unique values count using Redis HLL, for given event name. - - Example: - - [Track usage event for incident created in service](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/issues/update_service.rb) - - [Track usage event for incident created in graphql](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/graphql/mutations/alert_management/update_alert_status.rb) - - ```ruby - track_usage_event(:incident_management_incident_created, current_user.id) - ``` - -1. Track event using `UsageData` API - - Increment unique users count using Redis HLL, for given event name. - - Tracking events using the `UsageData` API requires the `usage_data_api` feature flag to be enabled, which is disabled by default. - - API requests are protected by checking for a valid CSRF token. - - In order to be able to increment the values the related feature `usage_data<event_name>` should be enabled. - - ```plaintext - POST /usage_data/increment_unique_users - ``` - - | Attribute | Type | Required | Description | - | :-------- | :--- | :------- | :---------- | - | `event` | string | yes | The event name it should be tracked | - - Response - - Return 200 if tracking failed for any reason. - - - `200` if event was tracked or any errors - - `400 Bad request` if event parameter is missing - - `401 Unauthorized` if user is not authenticated - - `403 Forbidden` for invalid CSRF token provided - -1. Track event using base module `Gitlab::UsageDataCounters::HLLRedisCounter.track_event(entity_id, event_name)`. - - Arguments: - - - `entity_id`: value we count. For example: user_id, visitor_id. - - `event_name`: event name. - -1. Get event data using `Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names:, start_date:, end_date)`. - - Arguments: - - - `event_names`: the list of event names. - - `start_date`: start date of the period for which we want to get event data. - - `end_date`: end date of the period for which we want to get event data. - -Recommendations: - -- Key should expire in 29 days for daily and 42 days for weekly. -- If possible, data granularity should be a week. For example a key could be composed from the - metric's name and week of the year, `2020-33-{metric_name}`. -- Use a [feature flag](../../operations/feature_flags.md) to have a control over the impact when - adding new metrics. - -##### Known events in usage data payload - -All events added in [`known_events.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml) are automatically added to usage data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). -For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: - -- `#{event_name}_weekly` data for 7 days for daily [aggregation](#adding-new-events) events and data for last complete week for weekly [aggregation](#adding-new-events) events. -- `#{event_name}_monthly` data for 28 days for daily [aggregation](#adding-new-events) events and data for last 4 complete weeks for weekly [aggregation](#adding-new-events) events. -- `#{category}_total_unique_counts_weekly` total unique counts for events in same category for last 7 days or last complete week, if events are in the same Redis slot and if we have more than one metric. -- `#{event_name}_weekly` - Data for 7 days for daily [aggregation](#adding-new-events) events and data for the last complete week for weekly [aggregation](#adding-new-events) events. -- `#{event_name}_monthly` - Data for 28 days for daily [aggregation](#adding-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#adding-new-events) events. -- `#{category}_total_unique_counts_weekly` - Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. -- `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#adding-new-events) events and data for last complete week for weekly [aggregation](#adding-new-events) events. -- `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#adding-new-events) events and data for last 4 complete weeks for weekly [aggregation](#adding-new-events) events. -- `#{category}_total_unique_counts_weekly` total unique counts for events in same category for last 7 days or last complete week, if events are in the same Redis slot and if we have more than one metric. -- `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#adding-new-events) events and data for the last complete week for weekly [aggregation](#adding-new-events) events. -- `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#adding-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#adding-new-events) events. -- `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. -- `#{category}_total_unique_counts_monthly`: Total unique counts for events in same category for the last 28 days or the last 4 complete weeks, if events are in the same Redis slot and we have more than one metric. - -Example of `redis_hll_counters` data: - -```ruby -{:redis_hll_counters=> - {"compliance"=> - {"g_compliance_dashboard_weekly"=>0, - "g_compliance_dashboard_monthly"=>0, - "g_compliance_audit_events_weekly"=>0, - "g_compliance_audit_events_monthly"=>0, - "compliance_total_unique_counts_weekly"=>0, - "compliance_total_unique_counts_monthly"=>0}, - "analytics"=> - {"g_analytics_contribution_weekly"=>0, - "g_analytics_contribution_monthly"=>0, - "g_analytics_insights_weekly"=>0, - "g_analytics_insights_monthly"=>0, - "analytics_total_unique_counts_weekly"=>0, - "analytics_total_unique_counts_monthly"=>0}, - "ide_edit"=> - {"g_edit_by_web_ide_weekly"=>0, - "g_edit_by_web_ide_monthly"=>0, - "g_edit_by_sfe_weekly"=>0, - "g_edit_by_sfe_monthly"=>0, - "ide_edit_total_unique_counts_weekly"=>0, - "ide_edit_total_unique_counts_monthly"=>0}, - "search"=> - {"i_search_total_weekly"=>0, "i_search_total_monthly"=>0, "i_search_advanced_weekly"=>0, "i_search_advanced_monthly"=>0, "i_search_paid_weekly"=>0, "i_search_paid_monthly"=>0, "search_total_unique_counts_weekly"=>0, "search_total_unique_counts_monthly"=>0}, - "source_code"=>{"wiki_action_weekly"=>0, "wiki_action_monthly"=>0} - } -``` - -Example usage: - -```ruby -# Redis Counters -redis_usage_data(Gitlab::UsageDataCounters::WikiPageCounter) -redis_usage_data { ::Gitlab::UsageCounters::PodLogs.usage_totals[:total] } - -# Define events in known_events.yml https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events.yml - -# Tracking events -Gitlab::UsageDataCounters::HLLRedisCounter.track_event(visitor_id, 'expand_vulnerabilities') - -# Get unique events for metric -redis_usage_data { Gitlab::UsageDataCounters::HLLRedisCounter.unique_events(event_names: 'expand_vulnerabilities', start_date: 28.days.ago, end_date: Date.current) } -``` - -### Alternative Counters - -Handles `StandardError` and fallbacks into -1 this way not all measures fail if we encounter one exception. -Mainly used for settings and configurations. - -Method: `alt_usage_data(value = nil, fallback: -1, &block)` - -Arguments: - -- `value`: a simple static value in which case the value is simply returned. -- or a `block`: which is evaluated -- `fallback: -1`: the common value used for any metrics that are failing. - -Example of usage: - -```ruby -alt_usage_data { Gitlab::VERSION } -alt_usage_data { Gitlab::CurrentSettings.uuid } -alt_usage_data(999) -``` - -### Prometheus Queries - -In those cases where operational metrics should be part of Usage Ping, a database or Redis query is unlikely -to provide useful data. Instead, Prometheus might be more appropriate, since most of GitLab's architectural -components publish metrics to it that can be queried back, aggregated, and included as usage data. - -NOTE: **Note:** -Prometheus as a data source for Usage Ping is currently only available for single-node Omnibus installations -that are running the [bundled Prometheus](../../administration/monitoring/prometheus/index.md) instance. - -In order to query Prometheus for metrics, a helper method is available that will `yield` a fully configured -`PrometheusClient`, given it is available as per the note above: - -```ruby -with_prometheus_client do |client| - response = client.query('<your query>') - ... -end -``` - -Please refer to [the `PrometheusClient` definition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/prometheus_client.rb) -for how to use its API to query for data. - -## Developing and testing Usage Ping - -### 1. Use your Rails console to manually test counters - -```ruby -# count -Gitlab::UsageData.count(User.active) -Gitlab::UsageData.count(::Clusters::Cluster.aws_installed.enabled, :cluster_id) - -# count distinct -Gitlab::UsageData.distinct_count(::Project, :creator_id) -Gitlab::UsageData.distinct_count(::Note.with_suggestions.where(time_period), :author_id, start: ::User.minimum(:id), finish: ::User.maximum(:id)) -``` - -### 2. Generate the SQL query - -Your Rails console will return the generated SQL queries. - -Example: - -```ruby -pry(main)> Gitlab::UsageData.count(User.active) - (2.6ms) SELECT "features"."key" FROM "features" - (15.3ms) SELECT MIN("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) - (2.4ms) SELECT MAX("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) - (1.9ms) SELECT COUNT("users"."id") FROM "users" WHERE ("users"."state" IN ('active')) AND ("users"."user_type" IS NULL OR "users"."user_type" IN (6, 4)) AND "users"."id" BETWEEN 1 AND 100000 -``` - -### 3. Optimize queries with #database-lab - -Paste the SQL query into `#database-lab` to see how the query performs at scale. - -- `#database-lab` is a Slack channel which uses a production-sized environment to test your queries. -- GitLab.com’s production database has a 15 second timeout. -- Any single query must stay below 1 second execution time with cold caches. -- Add a specialized index on columns involved to reduce the execution time. - -In order to have an understanding of the query's execution we add in the MR description the following information: - -- For counters that have a `time_period` test we add information for both cases: - - `time_period = {}` for all time periods - - `time_period = { created_at: 28.days.ago..Time.current }` for last 28 days period -- Execution plan and query time before and after optimization -- Query generated for the index and time -- Migration output for up and down execution - -We also use `#database-lab` and [explain.depesz.com](https://explain.depesz.com/). For more details, see the [database review guide](../database_review.md#preparation-when-adding-or-modifying-queries). - -#### Optimization recommendations and examples - -- Use specialized indexes [example 1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26871), [example 2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26445). -- Use defined `start` and `finish`, and simple queries, because these values can be memoized and reused, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37155). -- Avoid joins and write the queries as simply as possible, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36316). -- Set a custom `batch_size` for `distinct_count`, [example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38000). - -### 4. Add the metric definition - -When adding, changing, or updating metrics, please update the [Event Dictionary's **Usage Ping** table](event_dictionary.md). - -### 5. Add new metric to Versions Application - -Check if new metrics need to be added to the Versions Application. See `usage_data` [schema](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L147) and usage data [parameters accepted](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/app/services/usage_ping.rb). Any metrics added under the `counts` key are saved in the `counts` column. - -### 6. Add the feature label - -Add the `feature` label to the Merge Request for new Usage Ping metrics. These are user-facing changes and are part of expanding the Usage Ping feature. - -### 7. Add a changelog file - -Ensure you comply with the [Changelog entries guide](../changelog.md). - -### 8. Ask for a Telemetry Review - -On GitLab.com, we have DangerBot setup to monitor Telemetry related files and DangerBot will recommend a Telemetry review. Mention `@gitlab-org/growth/telemetry/engineers` in your MR for a review. - -### 9. Verify your metric - -On GitLab.com, the Product Analytics team regularly monitors Usage Ping. They may alert you that your metrics need further optimization to run quicker and with greater success. You may also use the [Usage Ping QA dashboard](https://app.periscopedata.com/app/gitlab/632033/Usage-Ping-QA) to check how well your metric performs. The dashboard allows filtering by GitLab version, by "Self-managed" & "Saas" and shows you how many failures have occurred for each metric. Whenever you notice a high failure rate, you may re-optimize your metric. - -### Optional: Test Prometheus based Usage Ping - -If the data submitted includes metrics [queried from Prometheus](#prometheus-queries) that you would like to inspect and verify, -then you need to ensure that a Prometheus server is running locally, and that furthermore the respective GitLab components -are exporting metrics to it. If you do not need to test data coming from Prometheus, no further action -is necessary, since Usage Ping should degrade gracefully in the absence of a running Prometheus server. - -There are currently three kinds of components that may export data to Prometheus, and which are included in Usage Ping: - -- [`node_exporter`](https://github.com/prometheus/node_exporter) - Exports node metrics from the host machine -- [`gitlab-exporter`](https://gitlab.com/gitlab-org/gitlab-exporter) - Exports process metrics from various GitLab components -- various GitLab services such as Sidekiq and the Rails server that export their own metrics - -#### Test with an Omnibus container - -This is the recommended approach to test Prometheus based Usage Ping. - -The easiest way to verify your changes is to build a new Omnibus image from your code branch via CI, then download the image -and run a local container instance: - -1. From your merge request, click on the `qa` stage, then trigger the `package-and-qa` job. This job will trigger an Omnibus -build in a [downstream pipeline of the `omnibus-gitlab-mirror` project](https://gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/-/pipelines). -1. In the downstream pipeline, wait for the `gitlab-docker` job to finish. -1. Open the job logs and locate the full container name including the version. It will take the following form: `registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>`. -1. On your local machine, make sure you are logged in to the GitLab Docker registry. You can find the instructions for this in -[Authenticating to the GitLab Container Registry](../../user/packages/container_registry/index.md#authenticating-to-the-gitlab-container-registry). -1. Once logged in, download the new image via `docker pull registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee:<VERSION>` -1. For more information about working with and running Omnibus GitLab containers in Docker, please refer to [GitLab Docker images](https://docs.gitlab.com/omnibus/docker/README.html) in the Omnibus documentation. - -#### Test with GitLab development toolkits - -This is the less recommended approach, since it comes with a number of difficulties when emulating a real GitLab deployment. - -The [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) is not currently set up to run a Prometheus server or `node_exporter` alongside other GitLab components. If you would -like to do so, [Monitoring the GDK with Prometheus](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/master/doc/howto/prometheus/index.md#monitoring-the-gdk-with-prometheus) is a good start. - -The [GCK](https://gitlab.com/gitlab-org/gitlab-compose-kit) has limited support for testing Prometheus based Usage Ping. -By default, it already comes with a fully configured Prometheus service that is set up to scrape a number of components, -but with the following limitations: - -- It does not currently run a `gitlab-exporter` instance, so several `process_*` metrics from services such as Gitaly may be missing. -- While it runs a `node_exporter`, `docker-compose` services emulate hosts, meaning that it would normally report itself to not be associated -with any of the other services that are running. That is not how node metrics are reported in a production setup, where `node_exporter` -always runs as a process alongside other GitLab components on any given node. From Usage Ping's perspective none of the node data would therefore -appear to be associated to any of the services running, since they all appear to be running on different hosts. To alleviate this problem, the `node_exporter` in GCK was arbitrarily "assigned" to the `web` service, meaning only for this service `node_*` metrics will appear in Usage Ping. - -## Example Usage Ping payload - -The following is example content of the Usage Ping payload. - -```json -{ - "uuid": "0000000-0000-0000-0000-000000000000", - "hostname": "example.com", - "version": "12.10.0-pre", - "installation_type": "omnibus-gitlab", - "active_user_count": 999, - "recorded_at": "2020-04-17T07:43:54.162+00:00", - "edition": "EEU", - "license_md5": "00000000000000000000000000000000", - "license_id": null, - "historical_max_users": 999, - "licensee": { - "Name": "ABC, Inc.", - "Email": "email@example.com", - "Company": "ABC, Inc." - }, - "license_user_count": 999, - "license_starts_at": "2020-01-01", - "license_expires_at": "2021-01-01", - "license_plan": "ultimate", - "license_add_ons": { - }, - "license_trial": false, - "counts": { - "assignee_lists": 999, - "boards": 999, - "ci_builds": 999, - ... - }, - "container_registry_enabled": true, - "dependency_proxy_enabled": false, - "gitlab_shared_runners_enabled": true, - "gravatar_enabled": true, - "influxdb_metrics_enabled": true, - "ldap_enabled": false, - "mattermost_enabled": false, - "omniauth_enabled": true, - "prometheus_enabled": false, - "prometheus_metrics_enabled": false, - "reply_by_email_enabled": "incoming+%{key}@incoming.gitlab.com", - "signup_enabled": true, - "web_ide_clientside_preview_enabled": true, - "ingress_modsecurity_enabled": true, - "projects_with_expiration_policy_disabled": 999, - "projects_with_expiration_policy_enabled": 999, - ... - "elasticsearch_enabled": true, - "license_trial_ends_on": null, - "geo_enabled": false, - "git": { - "version": { - "major": 2, - "minor": 26, - "patch": 1 - } - }, - "gitaly": { - "version": "12.10.0-rc1-93-g40980d40", - "servers": 56, - "clusters": 14, - "filesystems": [ - "EXT_2_3_4" - ] - }, - "gitlab_pages": { - "enabled": true, - "version": "1.17.0" - }, - "container_registry_server": { - "vendor": "gitlab", - "version": "2.9.1-gitlab" - }, - "database": { - "adapter": "postgresql", - "version": "9.6.15" - }, - "avg_cycle_analytics": { - "issue": { - "average": 999, - "sd": 999, - "missing": 999 - }, - "plan": { - "average": null, - "sd": 999, - "missing": 999 - }, - "code": { - "average": null, - "sd": 999, - "missing": 999 - }, - "test": { - "average": null, - "sd": 999, - "missing": 999 - }, - "review": { - "average": null, - "sd": 999, - "missing": 999 - }, - "staging": { - "average": null, - "sd": 999, - "missing": 999 - }, - "production": { - "average": null, - "sd": 999, - "missing": 999 - }, - "total": 999 - }, - "analytics_unique_visits": { - "g_analytics_contribution": 999, - ... - }, - "usage_activity_by_stage": { - "configure": { - "project_clusters_enabled": 999, - ... - }, - "create": { - "merge_requests": 999, - ... - }, - "manage": { - "events": 999, - ... - }, - "monitor": { - "clusters": 999, - ... - }, - "package": { - "projects_with_packages": 999 - }, - "plan": { - "issues": 999, - ... - }, - "release": { - "deployments": 999, - ... - }, - "secure": { - "user_container_scanning_jobs": 999, - ... - }, - "verify": { - "ci_builds": 999, - ... - } - }, - "usage_activity_by_stage_monthly": { - "configure": { - "project_clusters_enabled": 999, - ... - }, - "create": { - "merge_requests": 999, - ... - }, - "manage": { - "events": 999, - ... - }, - "monitor": { - "clusters": 999, - ... - }, - "package": { - "projects_with_packages": 999 - }, - "plan": { - "issues": 999, - ... - }, - "release": { - "deployments": 999, - ... - }, - "secure": { - "user_container_scanning_jobs": 999, - ... - }, - "verify": { - "ci_builds": 999, - ... - } - }, - "topology": { - "duration_s": 0.013836685999194742, - "application_requests_per_hour": 4224, - "query_apdex_weekly_average": 0.996, - "failures": [], - "nodes": [ - { - "node_memory_total_bytes": 33269903360, - "node_memory_utilization": 0.35, - "node_cpus": 16, - "node_cpu_utilization": 0.2, - "node_uname_info": { - "machine": "x86_64", - "sysname": "Linux", - "release": "4.19.76-linuxkit" - }, - "node_services": [ - { - "name": "web", - "process_count": 16, - "process_memory_pss": 233349888, - "process_memory_rss": 788220927, - "process_memory_uss": 195295487, - "server": "puma" - }, - { - "name": "sidekiq", - "process_count": 1, - "process_memory_pss": 734080000, - "process_memory_rss": 750051328, - "process_memory_uss": 731533312 - }, - ... - ], - ... - }, - ... - ] - } -} -``` - -## Exporting Usage Ping SQL queries and definitions - -Two Rake tasks exist to export Usage Ping definitions. - -- The Rake tasks export the raw SQL queries for `count`, `distinct_count`, `sum`. -- The Rake tasks export the Redis counter class or the line of the Redis block for `redis_usage_data`. -- The Rake tasks calculate the `alt_usage_data` metrics. - -In the home directory of your local GitLab installation run the following Rake tasks for the YAML and JSON versions respectively: - -```shell -# for YAML export -bin/rake gitlab:usage_data:dump_sql_in_yaml - -# for JSON export -bin/rake gitlab:usage_data:dump_sql_in_json - -# You may pipe the output into a file -bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml -``` +This document was moved to [another location](../product_analytics/usage_ping.md). diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 6ef9be381b4..2c1d70a005e 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -81,7 +81,7 @@ browser is much slower than parsing the HTML response from the app. A common cause of slow tests is excessive creation of objects, and thus computation and DB time. Factories are essential to development, but they can -make inserting data into the DB so easy that we may be able to optimize. +make inserting data into the DB so easy that we may be able to optimize. The two basic techniques to bear in mind here are: @@ -151,11 +151,14 @@ In order to reuse a single object for all calls to a named factory in implicit p can be used: ```ruby +RSpec.describe API::Search, factory_default: :keep do let_it_be(:namespace) { create_default(:namespace) } ``` Then every project we create will use this `namespace`, without us having to pass -it as `namespace: namespace`. +it as `namespace: namespace`. In order to make it work along with `let_it_be`, `factory_default: :keep` +must be explicitly specified. That will keep the default factory for every example in a suite instead of +recreating it for each example. Maybe we don't need to create 208 different projects - we can create one and reuse it. In addition, we can see that only about 1/3 of the @@ -237,7 +240,7 @@ end it 'schedules a background job' do expect(BackgroundJob).to receive(:perform_async) - + subject.execute end ``` @@ -249,7 +252,7 @@ combining the examples: ```ruby it 'performs the expected side-effects' do expect(BackgroundJob).to receive(:perform_async) - + expect { subject.execute } .to change(Event, :count).by(1) .and change { arg_0.frobulance }.to('wibble') @@ -481,26 +484,30 @@ This will result in only one `Project`, `User`, and `ProjectMember` created for is handled automatically using a transaction rollback. Note that if you modify an object defined inside a `let_it_be` block, -then you will need to reload the object as needed, or specify the `reload` -option to reload for every example. +then you must do one of the following: + +- Reload the object as needed. +- Use the `let_it_be_with_reload` alias. +- Specify the `reload` option to reload for every example. ```ruby +let_it_be_with_reload(:project) { create(:project) } let_it_be(:project, reload: true) { create(:project) } ``` -You can also specify the `refind` option as well to completely load a -new object. +You can also use the `let_it_be_with_refind` alias, or specify the `refind` +option as well to completely load a new object. ```ruby +let_it_be_with_refind(:project) { create(:project) } let_it_be(:project, refind: true) { create(:project) } ``` ### Time-sensitive tests -[Timecop](https://github.com/travisjeffery/timecop) is available in our -Ruby-based tests for verifying things that are time-sensitive. Any test that -exercises or verifies something time-sensitive should make use of Timecop to -prevent transient test failures. +[`ActiveSupport::Testing::TimeHelpers`](https://api.rubyonrails.org/v6.0.3.1/classes/ActiveSupport/Testing/TimeHelpers.html) +can be used to verify things that are time-sensitive. Any test that exercises or verifies something time-sensitive +should make use of these helpers to prevent transient test failures. Example: @@ -508,7 +515,7 @@ Example: it 'is overdue' do issue = build(:issue, due_date: Date.tomorrow) - Timecop.freeze(3.days.from_now) do + travel_to(3.days.from_now) do expect(issue).to be_overdue end end @@ -888,6 +895,10 @@ GitLab uses [factory_bot](https://github.com/thoughtbot/factory_bot) as a test f resulting record to pass validation. - When instantiating from a factory, don't supply attributes that aren't required by the test. +- Prefer [implicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#implicit-definition) + or [explicit](https://github.com/thoughtbot/factory_bot/blob/master/GETTING_STARTED.md#explicit-definition) + association definitions instead of using `create` / `build` for association setup. + See [issue #262624](https://gitlab.com/gitlab-org/gitlab/-/issues/262624) for further context. - Factories don't have to be limited to `ActiveRecord` objects. [See example](https://gitlab.com/gitlab-org/gitlab-foss/commit/0b8cefd3b2385a21cfed779bd659978c0402766d). diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 6917639454c..8091142410c 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -28,9 +28,6 @@ After that, the next pipeline will use the up-to-date `knapsack/report-master.js The GitLab test suite is [monitored](../performance.md#rspec-profiling) for the `master` branch, and any branch that includes `rspec-profile` in their name. -A [public dashboard](https://redash.gitlab.com/public/dashboards/l1WhHXaxrCWM5Ai9D7YDqHKehq6OU3bx5gssaiWe?org_slug=default) is available for everyone to see. Feel free to look at the -slowest test files and try to improve them. - ## CI setup - Rails logging to `log/test.log` is disabled by default in CI [for diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index c552c44c864..a1883f44170 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -196,8 +196,8 @@ end **What do we test?** -1. Can we log in? -1. Can we log out? +1. Can we sign in? +1. Can we sign out? **How do we test?** diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 36cb49256a6..58bae749dc5 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -241,7 +241,11 @@ All tests expect to be able to log in at the start of the test. For an example see: <https://gitlab.com/gitlab-org/gitlab/-/issues/34736> -Ideally, any actions performed in an `after(:context)` (or [`before(:context)`](#limit-the-use-of-the-ui-in-beforecontext-and-after-hooks)) block would be performed via the API. But if it's necessary to do so via the UI (e.g., if API functionality doesn't exist), make sure to log out at the end of the block. +Ideally, actions performed in an `after(:context)` (or +[`before(:context)`](#limit-the-use-of-the-ui-in-beforecontext-and-after-hooks)) +block are performed using the API. If it's necessary to do so with the user +interface (for example, if API functionality doesn't exist), be sure to sign +out at the end of the block. ```ruby after(:all) do @@ -310,3 +314,56 @@ end # Using native mouse click events in the case of a mask/overlay click_element_coordinates(:title) ``` + +## Ensure `expect` statements wait efficiently + +In general, we use an `expect` statement to check that something _is_ as we expect it. For example: + +```ruby +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).to have_job("a_job") +end +``` + +### Ensure `expect` checks for negation efficiently + +However, sometimes we want to check that something is _not_ as we _don't_ want it to be. In other +words, we want to make sure something is absent. In such a case we should use an appropriate +predicate method that returns quickly, rather than waiting for a state that won't appear. + +It's most efficient to use a predicate method that returns immediately when there is no job, or waits +until it disappears: + +```ruby +# Good +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).to have_no_job("a_job") +end +``` + +### Problematic alternatives + +Alternatively, if we want to check that a job doesn't exist it might be tempting to use `not_to`: + +```ruby +# Bad +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).not_to have_job("a_job") +end +``` + +For this statement to pass, `have_job("a_job")` has to return `false` so that `not_to` can negate it. +The problem is that `have_job("a_job")` waits up to ten seconds for `"a job"` to appear before +returning `false`. Under the expected condition this test will take ten seconds longer than it needs to. + +Instead, we could force no wait: + +```ruby +# Not as bad but potentially flaky +Page::Project::Pipeline::Show.perform do |pipeline| + expect(pipeline).not_to have_job("a_job", wait: 0) +end +``` + +The problem is that if `"a_job"` is present and we're waiting for it to disappear, this statement +will fail. diff --git a/doc/development/testing_guide/end_to_end/environment_selection.md b/doc/development/testing_guide/end_to_end/environment_selection.md index 9eb7db72a51..325f251b280 100644 --- a/doc/development/testing_guide/end_to_end/environment_selection.md +++ b/doc/development/testing_guide/end_to_end/environment_selection.md @@ -1,7 +1,7 @@ # Environment selection -Some tests are designed to be run against specific environments. We can specify -what environments to run tests against using the `only` metadata. +Some tests are designed to be run against specific environments or [pipelines](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#scheduled-qa-test-pipelines). +We can specify what environments or pipelines to run tests against using the `only` metadata. ## Available switches @@ -11,15 +11,16 @@ what environments to run tests against using the `only` metadata. | `subdomain` | Set the subdomain matcher | `Array` or `String` | | `domain` | Set the domain matcher | `String` | | `production` | Match against production | `Static` | +| `pipeline` | Match against a pipeline | `Array` or `Static`| CAUTION: **Caution:** -You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously. +You cannot specify `:production` and `{ <switch>: 'value' }` simultaneously. These options are mutually exclusive. If you want to specify production, you can control the `tld` and `domain` independently. ## Examples -| Environment | Key | Matches (regex) | +| Environment or pipeline | Key | Matches (regex for environments, string matching for pipelines) | | ---------------- | --- | --------------- | | `any` | `` | `.+.com` | | `gitlab.com` | `only: :production` | `gitlab.com` | @@ -27,18 +28,24 @@ can control the `tld` and `domain` independently. | `gitlab.com and staging.gitlab.com` | `only: { subdomain: /(staging.)?/, domain: 'gitlab' }` | `(staging.)?gitlab.com` | | `dev.gitlab.org` | `only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' }` | `(dev).gitlab.org` | | `staging.gitlab.com & domain.gitlab.com` | `only: { subdomain: %i[staging domain] }` | `(staging|domain).+.com` | +| `nightly` | `only: { pipeline: :nightly }` | "nightly" | +| `nightly`, `canary` | `only_run_in_pipeline: [:nightly, :canary]` | ["nightly"](https://gitlab.com/gitlab-org/quality/nightly) and ["canary"](https://gitlab.com/gitlab-org/quality/canary) | ```ruby RSpec.describe 'Area' do - it 'runs in any environment' do; end + it 'runs in any environment or pipeline' do; end - it 'runs only in production', only: :production do; end + it 'runs only in production environment', only: :production do; end - it 'runs only in staging', only: { subdomain: :staging } do; end + it 'runs only in staging environment', only: { subdomain: :staging } do; end - it 'runs in dev', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end + it 'runs in dev environment', only: { tld: '.org', domain: 'gitlab', subdomain: 'dev' } do; end - it 'runs in prod and staging', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {} + it 'runs in prod and staging environments', only: { subdomain: /(staging.)?/, domain: 'gitlab' } {} + + it 'runs only in nightly pipeline', only: { pipeline: :nightly } do; end + + it 'runs in nightly and canary pipelines', only: { pipeline: [:nightly, :canary] } do; end end ``` @@ -46,6 +53,8 @@ NOTE: **Note:** If the test has a `before` or `after`, you must add the `only` metadata to the outer `RSpec.describe`. +If you want to run an `only: { :pipeline }` tagged test on your local GDK make sure either the `CI_PROJECT_NAME` environment variable is unset, or that the `CI_PROJECT_NAME` environment variable matches the specified pipeline in the `only: { :pipeline }` tag, or just delete the `only: { :pipeline }` tag. + ## Quarantining a test for a specific environment Similarly to specifying that a test should only run against a specific environment, it's also possible to quarantine a diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 87a9738b313..e571774167d 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -1,29 +1,70 @@ # Testing with feature flags -To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to enable and disable feature flags ([via the API](../../../api/features.md)). +To run a specific test with a feature flag enabled you can use the `QA::Runtime::Feature` class to +enable and disable feature flags ([via the API](../../../api/features.md)). -Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` will automatically authenticate as an administrator as long as you provide an appropriate access token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` and `GITLAB_ADMIN_PASSWORD`. +Note that administrator authorization is required to change feature flags. `QA::Runtime::Feature` +will automatically authenticate as an administrator as long as you provide an appropriate access +token via `GITLAB_QA_ADMIN_ACCESS_TOKEN` (recommended), or provide `GITLAB_ADMIN_USERNAME` +and `GITLAB_ADMIN_PASSWORD`. -Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments where admin access is not available. +Please be sure to include the tag `:requires_admin` so that the test can be skipped in environments +where admin access is not available. + +CAUTION: **Caution:** +You are strongly advised to [enable feature flags only for a group, project, user](../../feature_flags/development.md#feature-actors), +or [feature group](../../feature_flags/development.md#feature-groups). This makes it possible to +test a feature in a shared environment without affecting other users. + +For example, the code below would enable a feature flag named `:feature_flag_name` for the project +created by the test: ```ruby RSpec.describe "with feature flag enabled", :requires_admin do + let(:project) { Resource::Project.fabricate_via_api! } + before do - Runtime::Feature.enable('feature_flag_name') + Runtime::Feature.enable(:feature_flag_name, project: project) end it "feature flag test" do - # Execute a test with a feature flag enabled + # Execute the test with the feature flag enabled. + # It will only affect the project created in this test. end after do - Runtime::Feature.disable('feature_flag_name') + Runtime::Feature.disable(:feature_flag_name, project: project) end end ``` +Note that the `enable` and `disable` methods first set the flag and then check that the updated +value is returned by the API. + +Similarly, you can enable a feature for a group, user, or feature group: + +```ruby +group = Resource::Group.fabricate_via_api! +Runtime::Feature.enable(:feature_flag_name, group: group) + +user = Resource::User.fabricate_via_api! +Runtime::Feature.enable(:feature_flag_name, user: user) + +feature_group = "a_feature_group" +Runtime::Feature.enable(:feature_flag_name, feature_group: feature_group) +``` + +If no scope is provided, the feature flag will be set instance-wide: + +```ruby +# This will affect all users! +Runtime::Feature.enable(:feature_flag_name) +``` + ## Running a scenario with a feature flag enabled -It's also possible to run an entire scenario with a feature flag enabled, without having to edit existing tests or write new ones. +It's also possible to run an entire scenario with a feature flag enabled, without having to edit +existing tests or write new ones. -Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) for details. +Please see the [QA README](https://gitlab.com/gitlab-org/gitlab/tree/master/qa#running-tests-with-a-feature-flag-enabled) +for details. diff --git a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md index a9f54b53e5a..3a1303d9c0c 100644 --- a/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md +++ b/doc/development/testing_guide/end_to_end/rspec_metadata_tests.md @@ -11,7 +11,7 @@ This is a partial list of the [RSpec metadata](https://relishapp.com/rspec/rspec | `:gitaly_cluster` | The test will run against a GitLab instance where repositories are stored on redundant Gitaly nodes behind a Praefect node. All nodes are [separate containers](../../../administration/gitaly/praefect.md#requirements-for-configuring-a-gitaly-cluster). Tests that use this tag have a longer setup time since there are three additional containers that need to be started. | | `:jira` | The test requires a Jira Server. [GitLab-QA](https://gitlab.com/gitlab-org/gitlab-qa) will provision the Jira Server in a Docker container when the `Test::Integration::Jira` test scenario is run. | `:kubernetes` | The test includes a GitLab instance that is configured to be run behind an SSH tunnel, allowing a TLS-accessible GitLab. This test will also include provisioning of at least one Kubernetes cluster to test against. *This tag is often be paired with `:orchestrated`.* | -| `:only` | The test is only to be run against specific environments. See [Environment selection](environment_selection.md) for more information. | +| `:only` | The test is only to be run against specific environments or pipelines. See [Environment selection](environment_selection.md) for more information. | | `:orchestrated` | The GitLab instance under test may be [configured by `gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/-/blob/master/docs/what_tests_can_be_run.md#orchestrated-tests) to be different to the default GitLab configuration, or `gitlab-qa` may launch additional services in separate Docker containers, or both. Tests tagged with `:orchestrated` are excluded when testing environments where we can't dynamically modify GitLab's configuration (for example, Staging). | | `:quarantine` | The test has been [quarantined](https://about.gitlab.com/handbook/engineering/quality/guidelines/debugging-qa-test-failures/#quarantining-tests), will run in a separate job that only includes quarantined tests, and is allowed to fail. The test will be skipped in its regular job so that if it fails it will not hold up the pipeline. Note that you can also [quarantine a test only when it runs against specific environment](environment_selection.md#quarantining-a-test-for-a-specific-environment). | | `:reliable` | The test has been [promoted to a reliable test](https://about.gitlab.com/handbook/engineering/quality/guidelines/reliable-tests/#promoting-an-existing-test-to-reliable) meaning it passes consistently in all pipelines, including merge requests. | diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index 7ac0a00fcff..658839fcf96 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -274,7 +274,7 @@ CHROME_HEADLESS=false bundle exec bin/qa QA::EE::Scenario::Test::Geo --primary-a ### Using Geo in Docker -You can use [GitLab-QA Orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) to orchestrate two GitLab containers and configure them as a Geo setup. +You can use [GitLab-QA Orchestrator](https://gitlab.com/gitlab-org/gitlab-qa) to orchestrate two GitLab containers and configure them as a Geo setup. Geo requires an EE license. To visit the Geo sites in your browser, you will need a reverse proxy server (for example, [NGINX](https://www.nginx.com/)). @@ -319,7 +319,7 @@ Geo requires an EE license. To visit the Geo sites in your browser, you will nee _Map the hostnames to the local IP in `/etc/hosts` file on your machine:_ ```plaintext - 127.0.0.1 gitlab-primary.geo gitlab-secondary.geo + 127.0.0.1 gitlab-primary.geo gitlab-secondary.geo ``` _Note the assigned ports:_ diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 7e9f097f624..645c2633831 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -141,7 +141,7 @@ end ``` ```ruby -Page::Project::New.peform do |new_page| +Page::Project::New.perform do |new_page| new_page.do_something end ``` @@ -155,7 +155,7 @@ end ``` ```ruby -Page::Project::New.peform do |page| +Page::Project::New.perform do |page| page.do_something end ``` diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 30e78766dde..730f8d5ad7d 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -229,7 +229,7 @@ beforeEach(() => { it('exists', () => { // Best - // NOTE: both mount and shallowMount work as long as a DOM element is available + // NOTE: both mount and shallowMount work as long as a DOM element is available // Finds a properly formatted link with an accessible name of "Click Me" getByRole(el, 'link', { name: /Click Me/i }) getByRole(el, 'link', { name: 'Click Me' }) diff --git a/doc/development/what_requires_downtime.md b/doc/development/what_requires_downtime.md index 7c99bcde413..9063fb867e2 100644 --- a/doc/development/what_requires_downtime.md +++ b/doc/development/what_requires_downtime.md @@ -176,7 +176,7 @@ class ChangeUsersUsernameStringToText < ActiveRecord::Migration[4.2] end def down - cleanup_concurrent_column_type_change :users, :username + undo_change_column_type_concurrently :users, :username end end ``` @@ -197,7 +197,7 @@ class ChangeUsersUsernameStringToTextCleanup < ActiveRecord::Migration[4.2] end def down - change_column_type_concurrently :users, :username, :string + undo_cleanup_concurrent_column_type_change :users, :username, :string end end ``` diff --git a/doc/development/wikis.md b/doc/development/wikis.md new file mode 100644 index 00000000000..9a436762645 --- /dev/null +++ b/doc/development/wikis.md @@ -0,0 +1,94 @@ +--- +type: reference, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +description: "GitLab's development guidelines for Wikis" +--- + +# Wikis development guide + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227027) in GitLab 13.5. + +## Overview + +The wiki functionality in GitLab is based on [Gollum 4.x](https://github.com/gollum/gollum/), +which is used in [Gitaly's](gitaly.md) Ruby service and accessed from the Rails app through Gitaly RPC calls. + +Wikis use Git repositories as storage backend, and can be accessed through: + +- The [Web UI](../user/project/wiki/index.md) +- The [REST API](../api/wikis.md) +- [Git itself](../user/project/wiki/#adding-and-editing-wiki-pages-locally) + +[Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2214) in GitLab 13.5, wikis are also available +for groups, in addition to projects. + +## Involved Gems + +Some notable gems that are used for wikis are: + +| Component | Description | Gem name | GitLab project | Upstream project | +|:--------------|:-----------------------------------------------|:-------------------------------|:--------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------| +| `gitlab` | Markup renderer, depends on various other gems | `gitlab-markup` | [`gitlab-org/gitlab-markup`](https://gitlab.com/gitlab-org/gitlab-markup) | [`github/markup`](https://github.com/github/markup) | +| `gitaly-ruby` | Main Gollum library | `gitlab-gollum-lib` | [`gitlab-org/gollum-lib`](https://gitlab.com/gitlab-org/gollum-lib) | [`gollum/gollum-lib`](https://github.com/gollum/gollum-lib) | +| | Gollum Git adapter for Rugged | `gitlab-gollum-rugged_adapter` | [`gitlab-org/gitlab-gollum-rugged_adapter`](https://gitlab.com/gitlab-org/gitlab-gollum-rugged_adapter) | [`gollum/rugged_adapter`](https://github.com/gollum/rugged_adapter) | +| | Rugged (also used in Gitaly itself) | `rugged` | - | [`libgit2/rugged`](https://github.com/libgit2/rugged) | + +### Notes on Gollum + +We only use Gollum as a storage abstraction layer, to handle the mapping between wiki page slugs and files in the repository. + +When rendering wiki pages, we don't use Gollum at all and instead go through a +[custom Banzai pipeline](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/banzai/pipeline/wiki_pipeline.rb). +This adds some [wiki-specific markup](../user/markdown.md#wiki-specific-markdown), such as Gollum's `[[link]]` syntax. + +Since we do not make use of most of Gollum's features, we plan to move away from it entirely at some point. +[See this epic](https://gitlab.com/groups/gitlab-org/-/epics/2381) for reference. + +## Model classes + +The `Wiki` class is the main abstraction around a wiki repository, it needs to be initialized +with a container which can be either a `Project` or `Group`: + +```mermaid +classDiagram + Wiki --> ProjectWiki + Wiki --> GroupWiki + + class Wiki { + #container + #repository + } + + class ProjectWiki { + #project → #container + } + + class GroupWiki { + #group → #container + } +``` + +Some models wrap similar classes from Gitaly and Gollum: + +| Rails Model | Gitaly Class | Gollum | +|:------------|:--------------------------------------------------------|:---------------| +| `Wiki` | `Gitlab::Git::Wiki` | `Gollum::Wiki` | +| `WikiPage` | `Gitlab::Git::WikiPage`, `Gitlab::Git::WikiPageVersion` | `Gollum::Page` | +| | `Gitlab::Git::WikiFile` | `Gollum::File` | + +Only some data is persisted in the database: + +| Model | Description | +|:----------------------|:-----------------------------------------| +| `WikiPage::Meta` | Metadata for wiki pages | +| `WikiPage::Slug` | Current and previous slugs of wiki pages | +| `ProjectRepository` | Gitaly storage data for project wikis | +| `GroupWikiRepository` | Gitaly storage data for group wikis | + +## Attachments + +The web UI uploads attachments through the REST API, which stores the files as commits in the wiki repository. + +Prior to GitLab 11.3 attachments were stored outside of the repository, [see this issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33475). |
