diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-11-18 13:16:36 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-11-18 13:16:36 +0000 |
| commit | 311b0269b4eb9839fa63f80c8d7a58f32b8138a0 (patch) | |
| tree | 07e7870bca8aed6d61fdcc810731c50d2c40af47 /doc/development | |
| parent | 27909cef6c4170ed9205afa7426b8d3de47cbb0c (diff) | |
| download | gitlab-ce-311b0269b4eb9839fa63f80c8d7a58f32b8138a0.tar.gz | |
Add latest changes from gitlab-org/gitlab@14-5-stable-eev14.5.0-rc42
Diffstat (limited to 'doc/development')
79 files changed, 3173 insertions, 1449 deletions
diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 9ca08ab1dc2..571d2f353d4 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -312,3 +312,16 @@ def down remove_concurrent_index_by_name :ci_builds, INDEX_NAME end ``` + +## Test database index changes locally + +You must test the database index changes locally before creating a merge request. + +### Verify indexes created asynchronously + +Use the asynchronous index helpers on your local environment to test changes for creating an index: + +1. Enable the feature flags by running `Feature.enable(:database_async_index_creation)` and `Feature.enable(:database_reindexing)` in the Rails console. +1. Run `bundle exec rails db:migrate` so that it creates an entry in the `postgres_async_indexes` table. +1. Run `bundle exec rails gitlab:db:reindex` so that the index is created asynchronously. +1. To verify the index, open the PostgreSQL console using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md) command `gdk psql` and run the command `\d <index_name>` to check that your newly created index exists. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 73814399d2e..df2f3c337cd 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -240,7 +240,7 @@ it's own file in the [`validators`](https://gitlab.com/gitlab-org/gitlab/-/blob/ ## Internal API -The [internal API](internal_api.md) is documented for internal use. Please keep it up to date so we know what endpoints +The [internal API](internal_api/index.md) is documented for internal use. Please keep it up to date so we know what endpoints different components are making use of. ## Avoiding N+1 problems diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md index c1d7ac9fa0c..5bc6fffdb48 100644 --- a/doc/development/application_slis/index.md +++ b/doc/development/application_slis/index.md @@ -43,7 +43,7 @@ example: ```ruby Gitlab::Metrics::Sli.initialize_sli(:received_email, [ { - feature_category: :issue_tracking, + feature_category: :team_planning, email_type: :create_issue }, { @@ -110,21 +110,79 @@ will also be incremented: gitlab_sli:received_email:success_total{ feature_category='service_desk', email_type='service_desk' } ``` +So far, only tracking `apdex` using a success rate is supported. If you +need to track errors this way, please upvote +[this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1395) +and leave a comment so we can prioritize this. + ## Using the SLI in service monitoring and alerts -When the application is emitting metrics for the new SLI, those need -to be consumed in the service catalog to result in alerts, and be -included in the error budget for stage groups and GitLab.com's overall -availability. +When the application is emitting metrics for a new SLI, they need +to be consumed from the [metrics catalog](https://gitlab.com/gitlab-com/runbooks/-/tree/master/metrics-catalog) +to result in alerts, and included in the error budget for stage +groups and GitLab.com's overall availability. + +Start by adding the new SLI to the +[Application-SLI library](https://gitlab.com/gitlab-com/runbooks/-/blob/d109886dfd5170793eeb8de3d69aafd4a9da78f6/metrics-catalog/gitlab-slis/library.libsonnet#L4). +After that, add the following information: + +- `name`: the name of the SLI as defined in code. For example + `received_email`. +- `significantLabels`: an array of Prometheus labels that belong to the + metrics. For example: `["email_type"]`. If the significant labels + for the SLI include `feature_category`, the metrics will also + feed into the + [error budgets for stage groups](../stage_group_dashboards.md#error-budget). +- `featureCategory`: if the SLI applies to a single feature category, + you can specify it statically through this field to feed the SLI + into the error budgets for stage groups. +- `description`: a Markdown string explaining the SLI. It will + be shown on dashboards and alerts. +- `kind`: the kind of indicator. Only `sliDefinition.apdexKind` is supported at the moment. + Reach out in + [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1395) + if you want to implement an SLI for success or error rates. + +When done, run `make generate` to generate recording rules for +the new SLI. This command creates recordings for all services +emitting these metrics aggregated over `significantLabels`. + +Open up a merge request with these changes and request review from a Scalability +team member. + +When these changes are merged, and the aggregations in +[Thanos](https://thanos.gitlab.net) recorded, query Thanos to see +the success ratio of the new aggregated metrics. For example: + +```prometheus +sum by (environment, stage, type)(gitlab_sli_aggregation:rails_request_apdex:apdex:success:rate_1h) +/ +sum by (environment, stage, type)(gitlab_sli_aggregation:rails_request_apdex:apdex:weight:rate_1h) +``` + +This shows the success ratio, which can guide you to set an +appropriate SLO when adding this SLI to a service. + +Then, add the SLI to the appropriate service +catalog file. For example, the [`web` service](https://gitlab.com/gitlab-com/runbooks/-/blob/2b7be37a006c236bd684a4e6a1fbf4c66158292a/metrics-catalog/services/web.jsonnet#L198): + +```jsonnet +rails_requests: + sliLibrary.get('rails_request_apdex') + .generateServiceLevelIndicator({ job: 'gitlab-rails' }) +``` + +To pass extra selectors and override properties of the SLI, see the +[service monitoring documentation](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/README.md). -This is currently being worked on in [this -project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573). As -part of [this -issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1307) -we will update the documentation. +SLIs with statically defined feature categories can already receive +alerts about the SLI in specified Slack channels. For more information, read the +[alert routing documentation](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/alert-routing.md). +In [this project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/614) +we are extending this so alerts for SLIs with a `feature_category` +label in the souce metrics can also be routed. -For any question, please don't hesitate to createan issue in [the -Scalability issue -tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues) +For any question, please don't hesitate to create an issue in +[the Scalability issue tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues) or come find us in [#g_scalability](https://gitlab.slack.com/archives/CMMF8TKR9) on Slack. diff --git a/doc/development/application_slis/rails_request_apdex.md b/doc/development/application_slis/rails_request_apdex.md index e1ab5368578..b31c7d8756b 100644 --- a/doc/development/application_slis/rails_request_apdex.md +++ b/doc/development/application_slis/rails_request_apdex.md @@ -4,184 +4,171 @@ group: Scalability info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Rails request apdex SLI +# Rails request Apdex SLI > [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525) in GitLab 14.4 NOTE: -This SLI is not yet used in [error budgets for stage -groups](../stage_group_dashboards.md#error-budget) or service -monitoring. This is being worked on in [this -project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573). +This SLI is used for service monitoring. But not for [error budgets for stage groups](../stage_group_dashboards.md#error-budget) +by default. You can [opt in](#error-budget-attribution-and-ownership). -The request apdex SLI is [an SLI defined in the application](index.md) -that measures the duration of successful requests as an indicator for +The request Apdex SLI (Service Level Indicator) is [an SLI defined in the application](index.md). +It measures the duration of successful requests as an indicator for application performance. This includes the REST and GraphQL API, and the regular controller endpoints. It consists of these counters: 1. `gitlab_sli:rails_request_apdex:total`: This counter gets incremented for every request that did not result in a response - with a 5xx status code. This means that slow failures don't get - counted twice: The request is already counted in the error-SLI. + with a `5xx` status code. It ensures slow failures are not + counted twice, because the request is already counted in the error SLI. 1. `gitlab_sli:rails_request_apdex:success_total`: This counter gets incremented for every successful request that performed faster than - the [defined target duration depending on the endpoint's - urgency](#adjusting-request-urgency). + the [defined target duration depending on the endpoint's urgency](#adjusting-request-urgency). Both these counters are labeled with: 1. `endpoint_id`: The identification of the Rails Controller or the - Grape-API endpoint + Grape-API endpoint. 1. `feature_category`: The feature category specified for that controller or API endpoint. ## Request Apdex SLO -These counters can be combined into a success ratio, the objective for -this ratio is defined in the service catalog per service: +These counters can be combined into a success ratio. The objective for +this ratio is defined in the service catalog per service. For this SLI to meet SLO, +the ratio recorded must be higher than: -1. [Web: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/web.jsonnet#L19) -1. [API: 0.995](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/api.jsonnet#L19) -1. [Git: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/git.jsonnet#L22) - -This means that for this SLI to meet SLO, the ratio recorded needs to -be higher than those defined above. +- [Web: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/web.jsonnet#L19) +- [API: 0.995](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/api.jsonnet#L19) +- [Git: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/git.jsonnet#L22) For example: for the web-service, we want at least 99.8% of requests to be faster than their target duration. -These are the targets we use for alerting and service montoring. So -durations should be set keeping those into account. So we would not -cause alerts. But the goal would be to set the urgency to a target -that users would be satisfied with. +We use these targets for alerting and service monitoring. Set durations taking +these targets into account, so we don't cause alerts. The goal, however, is to +set the urgency to a target that satisfies our users. -Both successful measurements and unsuccessful ones have an impact on the +Both successful measurements and unsuccessful ones affect the error budget for stage groups. ## Adjusting request urgency Not all endpoints perform the same type of work, so it is possible to -define different urgencies for different endpoints. An endpoint with a -lower urgency can have a longer request duration than endpoints that -are high urgency. - -Long-running requests are more expensive for our -infrastructure: while one request is being served, the thread remains -occupied for the duration of that request. So nothing else can be handled by that -thread. Because of Ruby's Global VM Lock, the thread might keep the +define different urgency levels for different endpoints. An endpoint with a +lower urgency can have a longer request duration than endpoints with high urgency. + +Long-running requests are more expensive for our infrastructure. While serving +one request, the thread remains occupied for the duration of that request. The thread +can handle nothing else. Due to Ruby's Global VM Lock, the thread might keep the lock and stall other requests handled by the same Puma worker -process. The request is in fact a noisy neighbor for other requests -handled by the worker. This is why the upper bound for a target -duration is capped at 5 seconds. +process. The request is, in fact, a noisy neighbor for other requests +handled by the worker. We cap the upper bound for a target duration at 5 seconds +for this reason. ## Decreasing the urgency (setting a higher target duration) -Increasing the urgency on an existing endpoint can be done on -a case-by-case basis. Please take the following into account: +You can decrease the urgency on an existing endpoint on +a case-by-case basis. Take the following into account: -1. Apdex is about perceived performance, if a user is actively waiting +1. Apdex is about perceived performance. If a user is actively waiting for the result of a request, waiting 5 seconds might not be - acceptable. While if the endpoint is used by an automation - requiring a lot of data, 5 seconds could be okay. + acceptable. However, if the endpoint is used by an automation + requiring a lot of data, 5 seconds could be acceptable. A product manager can help to identify how an endpoint is used. 1. The workload for some endpoints can sometimes differ greatly depending on the parameters specified by the caller. The urgency - needs to accomodate that. In some cases, it might be interesting to + needs to accommodate those differences. In some cases, you could define a separate [application SLI](index.md#defining-a-new-sli) for what the endpoint is doing. When the endpoints in certain cases turn into no-ops, making them very fast, we should ignore these fast requests when setting the target. For example, if the `MergeRequests::DraftsController` is - hit for every merge request being viewed, but doesn't need to - render anything in most cases, then we should pick the target that - would still accomodate the endpoint performing work. + hit for every merge request being viewed, but rarely renders + anything, then we should pick the target that + would still accommodate the endpoint performing work. 1. Consider the dependent resources consumed by the endpoint. If the endpoint - loads a lot of data from Gitaly or the database and this is causing - it to not perform satisfactory. It could be better to optimize the + loads a lot of data from Gitaly or the database, and this causes + unsatisfactory performance, consider optimizing the way the data is loaded rather than increasing the target duration by lowering the urgency. - In cases like this, it might be appropriate to temporarily decrease + In these cases, it might be appropriate to temporarily decrease urgency to make the endpoint meet SLO, if this is bearable for the - infrastructure. In such cases, please link an issue from a code - comment. + infrastructure. In such cases, create a code comment linking to an issue. If the endpoint consumes a lot of CPU time, we should also consider this: these kinds of requests are the kind of noisy neighbors we should try to keep as short as possible. -1. Traffic characteristics should also be taken into account: if the - trafic to the endpoint is bursty, like CI traffic spinning up a +1. Traffic characteristics should also be taken into account. If the + traffic to the endpoint is bursty, like CI traffic spinning up a big batch of jobs hitting the same endpoint, then having these - endpoints take 5s is not acceptable from an infrastructure point of - view. We cannot scale up the fleet fast enough to accomodate for + endpoints take five seconds is unacceptable from an infrastructure point of + view. We cannot scale up the fleet fast enough to accommodate for the incoming slow requests alongside the regular traffic. When lowering the urgency for an existing endpoint, please involve a [Scalability team member](https://about.gitlab.com/handbook/engineering/infrastructure/team/scalability/#team-members) in the review. We can use request rates and durations available in the -logs to come up with a recommendation. Picking a threshold can be done -using the same process as for [increasing -urgency](#increasing-urgency-setting-a-lower-target-duration), picking -a duration that is higher than the SLO for the service. +logs to come up with a recommendation. You can pick a threshold +using the same process as for +[increasing urgency](#increasing-urgency-setting-a-lower-target-duration), +picking a duration that is higher than the SLO for the service. We shouldn't set the longest durations on endpoints in the merge -requests that introduces them, since we don't yet have data to support +requests that introduces them, because we don't yet have data to support the decision. ## Increasing urgency (setting a lower target duration) -When decreasing the target duration, we need to make sure the endpoint +When increasing the urgency, we must make sure the endpoint still meets SLO for the fleet that handles the request. You can use the -information in the logs to determine this: +information in the logs to check: -1. Open [this table in - Kibana](https://log.gprd.gitlab.net/goto/bbb6465c68eb83642269e64a467df3df) +1. Open [this table in Kibana](https://log.gprd.gitlab.net/goto/bbb6465c68eb83642269e64a467df3df) 1. The table loads information for the busiest endpoints by - default. You can speed things up by adding a filter for - `json.caller_id.keyword` and adding the identifier you're intersted - in (for example: `Projects::RawController#show`). + default. To speed the response, add both: + - A filter for `json.caller_id.keyword`. + - The identifier you're interested in, such as `Projects::RawController#show`. 1. Check the [appropriate percentile duration](#request-apdex-slo) for - the service the endpoint is handled by. The overall duration should - be lower than the target you intend to set. + the service handling the endpoint. The overall duration should + be lower than your intended target. -1. If the overall duration is below the intended targed. Please also - check the peaks over time in [this - graph](https://log.gprd.gitlab.net/goto/9319c4a402461d204d13f3a4924a89fc) +1. If the overall duration is below the intended target, check the peaks over time + in [this graph](https://log.gprd.gitlab.net/goto/9319c4a402461d204d13f3a4924a89fc) in Kibana. Here, the percentile in question should not peak above the target duration we want to set. -Since decreasing a threshold too much could result in alerts for the -apdex degradation, please also involve a Scalability team member in -the merge reqeust. +As decreasing a threshold too much could result in alerts for the +Apdex degradation, please also involve a Scalability team member in +the merge request. ## How to adjust the urgency -The urgency can be specified similar to how endpoints [get a feature -category](../feature_categorization/index.md). - -For endpoints that don't have a specific target, the default urgency (1s duration) will be used. +You can specify urgency similar to how endpoints +[get a feature category](../feature_categorization/index.md). Endpoints without a +specific target use the default urgency: 1s duration. These configurations +are available: -The following configurations are available: - -| Urgency | Duration in seconds | Notes | -|----------|---------------------|-----------------------------------------------| -| :high | 0.25s | | -| :medium | 0.5s | | -| :default | 1s | This is the default when nothing is specified | -| :low | 5s | | +| Urgency | Duration in seconds | Notes | +|------------|---------------------|-----------------------------------------------| +| `:high` | 0.25s | | +| `:medium` | 0.5s | | +| `:default` | 1s | The default when nothing is specified. | +| `:low` | 5s | | ### Rails controller -An urgency can be specified for all actions in a controller like this: +An urgency can be specified for all actions in a controller: ```ruby class Boards::ListsController < ApplicationController @@ -189,8 +176,7 @@ class Boards::ListsController < ApplicationController end ``` -To specify the urgency also for certain actions in a controller, they -can be specified like this: +To also specify the urgency for certain actions in a controller: ```ruby class Boards::ListsController < ApplicationController @@ -200,8 +186,7 @@ end ### Grape endpoints -To specify the urgency for an entire API class, this can be done as -follows: +To specify the urgency for an entire API class: ```ruby module API @@ -211,8 +196,7 @@ module API end ``` -To specify the urgency also for certain actions in a API class, they -can be specified like this: +To specify the urgency also for certain actions in a API class: ```ruby module API @@ -232,3 +216,33 @@ get 'client/features', urgency: :low do # endpoint logic end ``` + +### Error budget attribution and ownership + +This SLI is used for service level monitoring. It feeds into the +[error budget for stage +groups](../stage_group_dashboards.md#error-budget). For this +particular SLI, we have opted everyone out by default to give time to +set the correct urgencies on endpoints before it affects a group's +error budget. + +To include this SLI in the error budget, remove the `rails_requests` +from the `ignored_components` array in the entry for your group. Read +more about what is configurable in the +[runbooks documentation](https://gitlab.com/gitlab-com/runbooks/-/tree/master/services#teamsyml). + +For more information, read the epic for +[defining custom SLIs and incorporating them into error budgets](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525)). +The endpoints for the SLI feed into a group's error budget based on the +[feature category declared on it](../feature_categorization/index.md). + +To know which endpoints are included for your group, you can see the +request rates on the +[group dashboard for your group](https://dashboards.gitlab.net/dashboards/f/stage-groups/stage-groups). +In the **Budget Attribution** row, the **Puma Apdex** log link shows you +how many requests are not meeting a 1s or 5s target. + +Learn more about the content of the dashboard in the documentation for +[Dashboards for stage groups](../stage_group_dashboards.md). For more information +on our exploration of the error budget itself, read the infrastructure issue +[Stage group error budget exploration dashboard](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1365). diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 9accd4a3595..38d0d5d7843 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -348,7 +348,6 @@ Component statuses are linked to configuration documentation for each component. | [Gitaly](#gitaly) | Git RPC service for handling all Git calls made by GitLab | ✅ | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | | [GitLab Exporter](#gitlab-exporter) | Generates a variety of GitLab metrics | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE | | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ❌ | ✅ | ❌ | ⚙ | EE Only | -| [GitLab Kubernetes Agent](#gitlab-kubernetes-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | | [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE | | [GitLab Kubernetes Agent](#gitlab-kubernetes-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE | @@ -977,12 +976,12 @@ in Rails, scheduled to run whenever an SSH key is modified by a user. instead of keys. In this case, `AuthorizedKeysCommand` is replaced with an `AuthorizedPrincipalsCommand`. This extracts a username from the certificate without using the Rails internal API, which is used instead of `key_id` in the -[`/api/internal/allowed`](internal_api.md) call later. +[`/api/internal/allowed`](internal_api/index.md) call later. GitLab Shell also has a few operations that do not involve Gitaly, such as resetting two-factor authentication codes. These are handled in the same way, except there is no round-trip into Gitaly - Rails performs the action as part -of the [internal API](internal_api.md) call, and GitLab Shell streams the +of the [internal API](internal_api/index.md) call, and GitLab Shell streams the response back to the user directly. ## System layout diff --git a/doc/development/avoiding_downtime_in_migrations.md b/doc/development/avoiding_downtime_in_migrations.md index 9418eafa487..a5fc1909551 100644 --- a/doc/development/avoiding_downtime_in_migrations.md +++ b/doc/development/avoiding_downtime_in_migrations.md @@ -377,7 +377,181 @@ ensures that no downtime is needed. This operation does not require downtime. -## Data Migrations +## Migrating `integer` primary keys to `bigint` + +To [prevent the overflow risk](https://gitlab.com/groups/gitlab-org/-/epics/4785) for some tables +with `integer` primary key (PK), we have to migrate their PK to `bigint`. The process to do this +without downtime and causing too much load on the database is described below. + +### Initialize the conversion and start migrating existing data (release N) + +To start the process, add a regular migration to create the new `bigint` columns. Use the provided +`initialize_conversion_of_integer_to_bigint` helper. The helper also creates a database trigger +to keep in sync both columns for any new records ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072312_initialize_conversion_of_ci_stages_to_bigint.rb)): + +```ruby +class InitializeConversionOfCiStagesToBigint < ActiveRecord::Migration[6.1] + include Gitlab::Database::MigrationHelpers + + TABLE = :ci_stages + COLUMNS = %i(id) + + def up + initialize_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end + + def down + revert_initialize_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end +end +``` + +Ignore the new `bigint` columns: + +```ruby +module Ci + class Stage < Ci::ApplicationRecord + include IgnorableColumns + ignore_column :id_convert_to_bigint, remove_with: '14.2', remove_after: '2021-08-22' + end +``` + +To migrate existing data, we introduced new type of _batched background migrations_. +Unlike the classic background migrations, built on top of Sidekiq, batched background migrations +don't have to enqueue and schedule all the background jobs at the beginning. +They also have other advantages, like automatic tuning of the batch size, better progress visibility, +and collecting metrics. To start the process, use the provided `backfill_conversion_of_integer_to_bigint` +helper ([example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072346_backfill_ci_stages_for_bigint_conversion.rb)): + +```ruby +class BackfillCiStagesForBigintConversion < ActiveRecord::Migration[6.1] + include Gitlab::Database::MigrationHelpers + + TABLE = :ci_stages + COLUMNS = %i(id) + + def up + backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end + + def down + revert_backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end +end +``` + +### Monitor the background migration + +Check how the migration is performing while it's running. Multiple ways to do this are described below. + +#### High-level status of batched background migrations + +See how to [check the status of batched background migrations](../update/index.md#checking-for-background-migrations-before-upgrading). + +#### Query the database + +We can query the related database tables directly. Requires access to read-only replica. +Example queries: + +```sql +-- Get details for batched background migration for given table +SELECT * FROM batched_background_migrations WHERE table_name = 'namespaces'\gx + +-- Get count of batched background migration jobs by status for given table +SELECT + batched_background_migrations.id, batched_background_migration_jobs.status, COUNT(*) +FROM + batched_background_migrations + JOIN batched_background_migration_jobs ON batched_background_migrations.id = batched_background_migration_jobs.batched_background_migration_id +WHERE + table_name = 'namespaces' +GROUP BY + batched_background_migrations.id, batched_background_migration_jobs.status; + +-- Batched background migration progress for given table (based on estimated total number of tuples) +SELECT + m.table_name, + LEAST(100 * sum(j.batch_size) / pg_class.reltuples, 100) AS percentage_complete +FROM + batched_background_migrations m + JOIN batched_background_migration_jobs j ON j.batched_background_migration_id = m.id + JOIN pg_class ON pg_class.relname = m.table_name +WHERE + j.status = 3 AND m.table_name = 'namespaces' +GROUP BY m.id, pg_class.reltuples; +``` + +#### Sidekiq logs + +We can also use the Sidekiq logs to monitor the worker that executes the batched background +migrations: + +1. Sign in to [Kibana](https://log.gprd.gitlab.net) with a `@gitlab.com` email address. +1. Change the index pattern to `pubsub-sidekiq-inf-gprd*`. +1. Add filter for `json.queue: cronjob:database_batched_background_migration`. + +#### PostgerSQL slow queries log + +Slow queries log keeps track of low queries that took above 1 second to execute. To see them +for batched background migration: + +1. Sign in to [Kibana](https://log.gprd.gitlab.net) with a `@gitlab.com` email address. +1. Change the index pattern to `pubsub-postgres-inf-gprd*`. +1. Add filter for `json.endpoint_id.keyword: Database::BatchedBackgroundMigrationWorker`. +1. Optional. To see only updates, add a filter for `json.command_tag.keyword: UPDATE`. +1. Optional. To see only failed statements, add a filter for `json.error_severiry.keyword: ERROR`. +1. Optional. Add a filter by table name. + +#### Grafana dashboards + +To monitor the health of the database, use these additional metrics: + +- [PostgreSQL Tuple Statistics](https://dashboards.gitlab.net/d/000000167/postgresql-tuple-statistics?orgId=1&refresh=1m): if you see high rate of updates for the tables being actively converted, or increasing percentage of dead tuples for this table, it might mean that autovacuum cannot keep up. +- [PostgreSQL Overview](https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1): if you see high system usage or transactions per second (TPS) on the primary database server, it might mean that the migration is causing problems. + +### Prometheus metrics + +Number of [metrics](https://gitlab.com/gitlab-org/gitlab/-/blob/294a92484ce4611f660439aa48eee4dfec2230b5/lib/gitlab/database/background_migration/batched_migration_wrapper.rb#L90-128) +for each batched background migration are published to Prometheus. These metrics can be searched for and +visualized in Thanos ([see an example](https://thanos-query.ops.gitlab.net/graph?g0.expr=sum%20(rate(batched_migration_job_updated_tuples_total%7Benv%3D%22gprd%22%7D%5B5m%5D))%20by%20(migration_id)%20&g0.tab=0&g0.stacked=0&g0.range_input=3d&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2021-06-13%2012%3A18%3A24&g0.moment_input=2021-06-13%2012%3A18%3A24)). + +### Swap the columns (release N + 1) + +After the background is completed and the new `bigint` columns are populated for all records, we can +swap the columns. Swapping is done with post-deployment migration. The exact process depends on the +table being converted, but in general it's done in the following steps: + +1. Using the provided `ensure_batched_background_migration_is_finished` helper, make sure the batched +migration has finished ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L13-18)). +If the migration has not completed, the subsequent steps fail anyway. By checking in advance we +aim to have more helpful error message. +1. Create indexes using the `bigint` columns that match the existing indexes using the `integer` +column ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L28-34)). +1. Create foreign keys (FK) using the `bigint` columns that match the existing FKs using the +`integer` column. Do this both for FK referencing other tables, and FKs that reference the table +that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L36-43)). +1. Inside a transaction, swap the columns: + 1. Lock the tables involved. To reduce the chance of hitting a deadlock, we recommended to do this in parent to child order ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L47)). + 1. Rename the columns to swap names ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L49-54)) + 1. Reset the trigger function ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L56-57)). + 1. Swap the defaults ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L59-62)). + 1. Swap the PK constraint (if any) ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L64-68)). + 1. Remove old indexes and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L70-72)). + 1. Remove old FKs (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). + +See example [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66088), and [migration](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb). + +### Remove the trigger and old `integer` columns (release N + 2) + +Using post-deployment migration and the provided `cleanup_conversion_of_integer_to_bigint` helper, +drop the database trigger and the old `integer` columns ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69714)). + +### Remove ignore rules (release N + 3) + +In the next release after the columns were dropped, remove the ignore rules as we do not need them +anymore ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71161)). + +## Data migrations Data migrations can be tricky. The usual approach to migrate data is to take a 3 step approach: diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index b0de00648ba..838235fd795 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -37,7 +37,7 @@ For example, when using the [GDK](https://gitlab.com/gitlab-org/gitlab-developme this can be done with the following command: ```shell -GITLAB_CHAOS_SECRET=secret gdk run +GITLAB_CHAOS_SECRET=secret gdk start ``` Replace `secret` with your own secret token. diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index b74a1d0d58a..46442aa6106 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -60,7 +60,7 @@ don't have any other `.gitlab-ci.yml` files. When authoring pipeline templates: - Place any [global keywords](../../ci/yaml/index.md#global-keywords) like `image` - or `before_script` in a [`default`](../../ci/yaml/index.md#custom-default-keyword-values) + or `before_script` in a [`default`](../../ci/yaml/index.md#default) section at the top of the template. - Note clearly in the [code comments](#explain-the-template-with-comments) if the template is designed to be used with the `includes` keyword in an existing @@ -77,7 +77,7 @@ other pipeline configuration. When authoring job templates: -- Do not use [global](../../ci/yaml/index.md#global-keywords) or [`default`](../../ci/yaml/index.md#custom-default-keyword-values) +- Do not use [global](../../ci/yaml/index.md#global-keywords) or [`default`](../../ci/yaml/index.md#default) keywords. When a root `.gitlab-ci.yml` includes a template, global or default keywords might be overridden and cause unexpected behavior. If a job template requires a specific stage, explain in the code comments that users must manually add the stage diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 89516c2168b..7e797309a26 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -152,7 +152,7 @@ See the [test engineering process](https://about.gitlab.com/handbook/engineering 1. For the code that this change impacts, I believe that the automated tests ([Testing Guide](testing_guide/index.md)) validate functionality that is highly important to users (including consideration of [all test levels](testing_guide/testing_levels.md)). 1. If the existing automated tests do not cover the above functionality, I have added the necessary additional tests or added an issue to describe the automation testing gap and linked it to this MR. 1. I have considered the technical aspects of this change's impact on GitLab.com hosted customers and self-managed customers. -1. I have considered the impact of this change on the frontend, backend, and database portions of the system where appropriate and applied the `~frontend`, `~backend`, and `~database` labels accordingly. +1. I have considered the impact of this change on the frontend, backend, and database portions of the system where appropriate and applied the `~ux`, `~frontend`, `~backend`, and `~database` labels accordingly. 1. I have tested this MR in [all supported browsers](../install/requirements.md#supported-web-browsers), or determined that this testing is not needed. 1. I have confirmed that this change is [backwards compatible across updates](multi_version_compatibility.md), or I have decided that this does not apply. 1. I have properly separated EE content from FOSS, or this MR is FOSS only. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index d0f107ba98a..6a7bbb2e302 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -45,7 +45,7 @@ scheduling into milestones. Labeling is a task for everyone. (For some projects, Most issues will have labels for at least one of the following: -- Type. For example: `~feature`, `~bug`, `~tooling`, or `~documentation`. +- Type. For example: `~"type::feature"`, `~"type::bug"`, or `~"type::tooling"`. - Stage. For example: `~"devops::plan"` or `~"devops::create"`. - Group. For example: `~"group::source code"`, `~"group::knowledge"`, or `~"group::editor"`. - Category. For example: `~"Category:Code Analytics"`, `~"Category:DevOps Reports"`, or `~"Category:Templates"`. @@ -70,12 +70,12 @@ issue should have one and only one. The current type labels are: -- `~feature` +- `~"type::feature"` - `~"feature::addition"` - `~"feature::enhancement"` - - `~"feature::maintenance"` -- `~bug` -- `~tooling` +- `~"type::maintenance"` +- `~"type::bug"` +- `~"type::tooling"` - `~"tooling::pipelines"` - `~"tooling::workflow"` - `~"support request"` @@ -168,7 +168,7 @@ their color is `#428BCA`. `<Category Name>` is the category name as it is in the single source of truth for categories at <https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml>. -For instance, the "DevOps Report" category is represented by the +For instance, the "DevOps Reports" category is represented by the ~"Category:DevOps Reports" label in the `gitlab-org` group since its `devops_reports.name` value is "DevOps Reports". @@ -342,11 +342,11 @@ To create a feature proposal, open an issue on the [issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). In order to help track the feature proposals, we have created a -[`feature`](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=feature) label. +[`~"type::feature"`](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=type::feature) label. For the time being, users that are not members of the project cannot add labels. You can instead ask one of the [core team](https://about.gitlab.com/community/core-team/) -members to add the label `~feature` to the issue or add the following -code snippet right after your description in a new line: `~feature`. +members to add the label `~"type::feature"` to the issue or add the following +code snippet right after your description in a new line: `~"type::feature"`. Please keep feature proposals as small and simple as possible, complex ones might be edited to make them small and simple. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index a521d89db2b..82fd62d8d79 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -83,6 +83,24 @@ request is as follows: migrations on a fresh database before the MR is reviewed. If the review leads to large changes in the MR, execute the migrations again once the review is complete. 1. Write tests for more complex migrations. +1. If your merge request adds new validations to existing models, to make sure the + data processing is backwards compatible: + + - Ask in the [`#database`](https://gitlab.slack.com/archives/CNZ8E900G) Slack channel + for assistance to execute the database query that checks the existing rows to + ensure existing rows aren't impacted by the change. + - Add the necessary validation with a feature flag to be gradually rolled out + following [the rollout steps](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#rollout). + + If this merge request is urgent, the code owners should make the final call on + whether reviewing existing rows should be included as an immediate follow-up task + to the merge request. + + NOTE: + There isn't a way to know anything about our customers' data on their + [self-managed instances](../../subscriptions/self_managed/index.md), so keep + that in mind for any data implications with your merge request. + 1. Merge requests **must** adhere to the [merge request performance guidelines](../merge_request_performance_guidelines.md). 1. For tests that use Capybara, read [how to write reliable, asynchronous integration tests](https://thoughtbot.com/blog/write-reliable-asynchronous-integration-tests-with-capybara). @@ -240,6 +258,7 @@ requirements. 1. Working and clean code that is commented where needed. 1. [Unit, integration, and system tests](../testing_guide/index.md) that all pass on the CI server. +1. Peer member testing is optional but recommended when the risk of a change is high. This includes when the changes are [far-reaching](https://about.gitlab.com/handbook/engineering/development/#reducing-the-impact-of-far-reaching-work) or are for [components critical for security](../code_review.md#security). 1. Regressions and bugs are covered with tests that reduce the risk of the issue happening again. 1. [Performance guidelines](../merge_request_performance_guidelines.md) have been followed. diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index aca37e2182a..829f6af76be 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -118,6 +118,23 @@ However, you can speed these cycles up somewhat by emptying the `.gitlab/ci/rails.gitlab-ci.yml` file in your merge request. Just don't forget to revert the change before merging! +#### Adding labels via Danger + +NOTE: +This is currently applicable to the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) +project only. + +Danger is often used to improve MR hygiene by adding labels. Instead of calling the +API directly in your `Dangerfile`, add the labels to the `project_helper.labels_to_add` array. +The main `Dangerfile` will then take care of adding the labels to the MR with a single API call. + +#### Shared rules and plugins + +If the rule or plugin you implement can be useful for other projects, think about +upstreaming them to the [`gitlab-org/gitlab-dangerfiles`](https://gitlab.com/gitlab-org/gitlab-dangerfiles) project. + +#### Enable Danger on a project + To enable the Dangerfile on another existing GitLab project, run the following extra steps: diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md new file mode 100644 index 00000000000..157c1284512 --- /dev/null +++ b/doc/development/database/loose_foreign_keys.md @@ -0,0 +1,182 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Loose foreign keys + +## Problem statement + +In relational databases (including PostgreSQL), foreign keys provide a way to link +two database tables together, and ensure data-consistency between them. In GitLab, +[foreign keys](../foreign_keys.md) are vital part of the database design process. +Most of our database tables have foreign keys. + +With the ongoing database [decomposition work](https://gitlab.com/groups/gitlab-org/-/epics/6168), +linked records might be present on two different database servers. Ensuring data consistency +between two databases is not possible with standard PostgreSQL foreign keys. PostgreSQL +does not support foreign keys operating within a single database server, defining +a link between two database tables in two different database servers over the network. + +Example: + +- Database "Main": `projects` table +- Database "CI": `ci_pipelines` table + +A project can have many pipelines. When a project is deleted, the associated `ci_pipeline` (via the +`project_id` column) records must be also deleted. + +With a multi-database setup, this cannot be achieved with foreign keys. + +## Asynchronous approach + +Our preferred approach to this problem is eventual consistency. With the loose foreign keys +feature, we can configure delayed association cleanup without negatively affecting the +application performance. + +### How it works + +In the previous example, a record in the `projects` table can have multiple `ci_pipeline` +records. To keep the cleanup process separate from the actual parent record deletion, +we can: + +1. Create a `DELETE` trigger on the `projects` table. + Record the deletions in a separate table (`deleted_records`). +1. A job checks the `deleted_records` table every 5 minutes. +1. For each record in the table, delete the associated `ci_pipelines` records + using the `project_id` column. + +NOTE: +For this procedure to work, we must register which tables to clean up asynchronously. + +## Example migration and configuration + +### Configure the model + +First, tell the application that the `projects` table has a new loose foreign key. +You can do this in the `Project` model: + +```ruby +class Project < ApplicationRecord + # ... + + include LooseForeignKey + + loose_foreign_key :ci_pipelines, :project_id, on_delete: :async_delete # or async_nullify + + # ... +end +``` + +This instruction ensures the asynchronous cleanup process knows about the association, and the +how to do the cleanup. In this case, the associated `ci_pipelines` records are deleted. + +### Track record changes + +To know about deletions in the `projects` table, configure a `DELETE` trigger using a database +migration (post-migration). The trigger needs to be configured only once. If the model already has +at least one `loose_foreign_key` definition, then this step can be skipped: + +```ruby +class TrackProjectRecordChanges < Gitlab::Database::Migration[1.0] + include Gitlab::Database::MigrationHelpers::LooseForeignKeyHelpers + + enable_lock_retries! + + def up + track_record_deletions(:projects) + end + + def down + untrack_record_deletions(:projects) + end +end +``` + +### Remove the foreign key + +If there is an existing foreign key, then it can be removed from the database. As of GitLab 14.5, +the following foreign key describes the link between the `projects` and `ci_pipelines` tables: + +```sql +ALTER TABLE ONLY ci_pipelines +ADD CONSTRAINT fk_86635dbd80 +FOREIGN KEY (project_id) +REFERENCES projects(id) +ON DELETE CASCADE; +``` + +The migration should run after the `DELETE` trigger is installed. If the foreign key is deleted +earlier, there is a good chance of introducing data inconsistency which needs manual cleanup: + +```ruby +class RemoveProjectsCiPipelineFk < Gitlab::Database::Migration[1.0] + enable_lock_retries! + + def up + remove_foreign_key_if_exists(:ci_pipelines, :projects, name: "fk_86635dbd80") + end + + def down + add_concurrent_foreign_key(:ci_pipelines, :projects, name: "fk_86635dbd80", column: :project_id, target_column: :id, on_delete: "cascade") + end +end +``` + +At this point, the setup phase is concluded. The deleted `projects` records should be automatically +picked up by the scheduled cleanup worker job. + +## Caveats of loose foreign keys + +### Record creation + +The feature provides an efficient way of cleaning up associated records after the parent record is +deleted. Without foreign keys, it's the application's responsibility to validate if the parent record +exists when a new associated record is created. + +A bad example: record creation with the given ID (`project_id` comes from user input). +In this example, nothing prevents us from passing a random project ID: + +```ruby +Ci::Pipeline.create!(project_id: params[:project_id]) +``` + +A good example: record creation with extra check: + +```ruby +project = Project.find(params[:project_id]) +Ci::Pipeline.create!(project_id: project.id) +``` + +### Association lookup + +Consider the following HTTP request: + +```plaintext +GET /projects/5/pipelines/100 +``` + +The controller action ignores the `project_id` parameter and finds the pipeline using the ID: + +```ruby + def show + # bad, avoid it + pipeline = Ci::Pipeline.find(params[:id]) # 100 +end +``` + +This endpoint still works when the parent `Project` model is deleted. This can be considered a +a data leak which should not happen under normal circumstances: + +```ruby +def show + # good + project = Project.find(params[:project_id]) + pipeline = project.pipelines.find(params[:pipeline_id]) # 100 +end +``` + +NOTE: +This example is unlikely in GitLab, because we usually look up the parent models to perform +permission checks. diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index 0ba752ba3a6..a17ad798305 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -88,16 +88,6 @@ test: &test statement_timeout: 120s ``` -### Migrations - -Place any migrations that affect `Ci::CiDatabaseRecord` models -and their tables in two directories: - -- `db/migrate` -- `db/ci_migrate` - -We aim to keep the schema for both tables the same across both databases. - <!-- NOTE: The `validate_cross_joins!` method in `spec/support/database/prevent_cross_joins.rb` references the following heading in the code, so if you make a change to this heading, make sure to update @@ -272,6 +262,62 @@ logic to delete these rows if or whenever necessary in your domain. Finally, this de-normalization and new query also improves performance because it does less joins and needs less filtering. +##### Remove a redundant join + +Sometimes there are cases where a query is doing excess (or redundant) joins. + +A common example occurs where a query is joining from `A` to `C`, via some +table with both foreign keys, `B`. +When you only care about counting how +many rows there are in `C` and if there are foreign keys and `NOT NULL` constraints +on the foreign keys in `B`, then it might be enough to count those rows. +For example, in +[MR 71811](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71811), it was +previously doing `project.runners.count`, which would produce a query like: + +```sql +select count(*) from projects +inner join ci_runner_projects on ci_runner_projects.project_id = projects.id +where ci_runner_projects.runner_id IN (1, 2, 3) +``` + +This was changed to avoid the cross-join by changing the code to +`project.runner_projects.count`. It produces the same response with the +following query: + +```sql +select count(*) from ci_runner_projects +where ci_runner_projects.runner_id IN (1, 2, 3) +``` + +Another common redundant join is joining all the way to another table, +then filtering by primary key when you could have instead filtered on a foreign +key. See an example in +[MR 71614](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71614). The previous +code was `joins(scan: :build).where(ci_builds: { id: build_ids })`, which +generated a query like: + +```sql +select ... +inner join security_scans +inner join ci_builds on security_scans.build_id = ci_builds.id +where ci_builds.id IN (1, 2, 3) +``` + +However, as `security_scans` already has a foreign key `build_id`, the code +can be changed to `joins(:scan).where(security_scans: { build_id: build_ids })`, +which produces the same response with the following query: + +```sql +select ... +inner join security_scans +where security_scans.build_id IN (1, 2, 3) +``` + +Both of these examples of removing redundant joins remove the cross-joins, +but they have the added benefit of producing simpler and faster +queries. + ##### Use `disable_joins` for `has_one` or `has_many` `through:` relations Sometimes a join query is caused by using `has_one ... through:` or `has_many diff --git a/doc/development/database_review.md b/doc/development/database_review.md index dcd5baab177..45be797b720 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -172,7 +172,9 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac `gitlab-org/gitlab-foss` (`project_id = 13083`) or the `gitlab-org/gitlab` (`project_id = 278964`) projects provide enough data to serve as a good example. - That means that no query plan should return 0 records or less records than the provided limit (if a limit is included). If a query is used in batching, a proper example batch with adequate included results should be identified and provided. - - If your queries belong to a new feature in GitLab.com and thus they don't return data in production, it's suggested to analyze the query and to provide the plan from a local environment. + - If your queries belong to a new feature in GitLab.com and thus they don't return data in production: + - You may analyze the query and to provide the plan from a local environment. + - `#database-lab` and [postgres.ai](https://postgres.ai/) both allow updates to data (`exec UPDATE issues SET ...`) and creation of new tables and columns (`exec ALTER TABLE issues ADD COLUMN ...`). - More information on how to find the number of actual returned records in [Understanding EXPLAIN plans](understanding_explain_plans.md) - For query changes, it is best to provide both the SQL queries along with the plan _before_ and _after_ the change. This helps spot differences quickly. diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index c169af1958e..8fc6f2e2641 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -71,8 +71,14 @@ Possible version history entries are: > - [Enabled on GitLab.com](issue-link) in GitLab X.X. > - [Enabled on GitLab.com](issue-link) in GitLab X.X. Available to GitLab.com administrators only. > - [Enabled on self-managed](issue-link) in GitLab X.X. -> - [Feature flag <flag name> removed](issue-link) in GitLab X.X. -> - [Generally available](issue-link) in GitLab X.X. +> - [Generally available](issue-link) in GitLab X.Y. [Feature flag <flag name>](issue-link) removed. +``` + +You can combine entries if they happened in the same release: + +```markdown +> - Introduced in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. ``` ## Feature flag documentation examples @@ -84,7 +90,7 @@ The following examples show the progression of a feature flag. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the featured flag](../administration/feature_flags.md) named `forti_token_cloud`. +ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `forti_token_cloud`. The feature is not ready for production use. ``` @@ -105,6 +111,5 @@ And, when the feature is done and fully available to all users: > - Introduced in GitLab 13.7 [with a flag](../../administration/feature_flags.md) named `forti_token_cloud`. Disabled by default. > - [Enabled on self-managed](https://gitlab.com/issue/etc) GitLab 13.8. > - [Enabled on GitLab.com](https://gitlab.com/issue/etc) in GitLab 13.9. -> - [Feature flag removed](https://gitlab.com/issue/etc) in GitLab 14.0. -> - [Generally available](issue-link) in GitLab 14.0. +> - [Generally available](issue-link) in GitLab 14.0. [Feature flag <flag name>](issue-link) removed. ``` diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 75538fe1fe7..776e5aefd00 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -170,22 +170,33 @@ it increases the work of the release managers. ## GitLab `/help` -Every GitLab instance includes the documentation, which is available at `/help` -(`https://gitlab.example.com/help`). For example, <https://gitlab.com/help>. - -The documentation available online on <https://docs.gitlab.com> is deployed every -four hours from the `main` branch of GitLab, Omnibus, and Runner. Therefore, -after a merge request gets merged, it is available online on the same day. -However, it's shipped (and available on `/help`) within the milestone assigned -to the MR. - -For example, let's say your merge request has a milestone set to 11.3, which -a release date of 2018-09-22. If it gets merged on 2018-09-15, it is -available online on 2018-09-15, but, as the feature freeze date has passed, if -the MR does not have a `~"Pick into 11.3"` label, the milestone has to be changed -to 11.4 and it ships with all GitLab packages only on 2018-10-22, -with GitLab 11.4. Meaning, it's available only with `/help` from GitLab -11.4 onward, but available on <https://docs.gitlab.com/> on the same day it was merged. +Every GitLab instance includes documentation at `/help` (`https://gitlab.example.com/help`) +that matches the version of the instance. For example, <https://gitlab.com/help>. + +The documentation available online at <https://docs.gitlab.com> is deployed every +four hours from the default branch of [GitLab, Omnibus, Runner, and Charts](#source-files-and-rendered-web-locations). +After a merge request that updates documentation is merged, it is available online +in 4 hours or less. + +However, it's only available at `/help` on self-managed instances in the next released +version. The date an update is merged can impact which self-managed release the update +is present in. + +For example: + +1. A merge request in `gitlab` updates documentation. It has a milestone of 14.4, + with an expected release date of 2021-10-22. +1. It is merged on 2021-10-19 and available online the same day at <https://docs.gitlab.com>. +1. GitLab 14.4 is released on 2021-10-22, based on the `gitlab` codebase from 2021-10-18 + (one day *before* the update was merged). +1. The change shows up in the 14.5 self-managed release, due to missing the release cutoff + for 14.4. + +The exact cutoff date for each release is flexible, and can be earlier or later +than expected due to holidays, weekends, or other events. In general, MRs merged +by the 17th should be present in the release on the 22nd, though it is not guaranteed. +If it is important that a documentation update is present in that month's release, +merge it as early as possible. ### Linking to `/help` diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index b3d3e2641b7..03980a42381 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -49,8 +49,8 @@ METHOD /endpoint Supported attributes: -| Attribute | Type | Required | Description | -|:------------|:---------|:---------|:----------------------| +| Attribute | Type | Required | Description | +| :---------- | :------- | :--------------------- | :-------------------- | | `attribute` | datatype | **{check-circle}** Yes | Detailed description. | | `attribute` | datatype | **{dotted-circle}** No | Detailed description. | | `attribute` | datatype | **{dotted-circle}** No | Detailed description. | @@ -80,16 +80,23 @@ to describe the GitLab release that introduced the API call. Use the following table headers to describe the methods. Attributes should always be in code blocks using backticks (`` ` ``). +Sort the attributes in the table: first, required, then alphabetically. + ```markdown -| Attribute | Type | Required | Description | -|:----------|:-----|:---------|:------------| +| Attribute | Type | Required | Description | +| :------------- | :------------ | :--------------------- | :--------------------------------------------------- | +| `user` | string | **{check-circle}** Yes | The GitLab username. | +| `assignee_ids` | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | ``` Rendered example: -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:--------------------| -| `user` | string | yes | The GitLab username. | +| Attribute | Type | Required | Description | +| :------------- | :------------ | :--------------------- | :--------------------------------------------------- | +| `user` | string | **{check-circle}** Yes | The GitLab username. | +| `assignee_ids` | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | ## cURL commands @@ -101,12 +108,12 @@ Rendered example: - Prefer to use examples using the personal access token and don't pass data of username and password. -| Methods | Description | -|:------------------------------------------- |:------------------------------------------------------| +| Methods | Description | +| :---------------------------------------------- | :----------------------------------------------------- | | `--header "PRIVATE-TOKEN: <your_access_token>"` | Use this method as is, whenever authentication needed. | -| `--request POST` | Use this method when creating new objects | -| `--request PUT` | Use this method when updating existing objects | -| `--request DELETE` | Use this method when removing existing objects | +| `--request POST` | Use this method when creating new objects | +| `--request PUT` | Use this method when updating existing objects | +| `--request DELETE` | Use this method when removing existing objects | ## cURL Examples diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index 1b764ada87b..c038ee96dbf 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -4,26 +4,158 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Documentation deployment process +# Documentation deployments + +The documentation [release process](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/releases.md) +involves: + +- Merge requests, to make changes to the `main` and relevant stable branches. +- Pipelines, to build and deploy Docker images to the [`gitlab-docs` container registry](https://gitlab.com/gitlab-org/gitlab-docs/container_registry) + for the relevant stable branches. +- Docker images used to build and deploy all the online documentation, including stable versions and the latest documentation. + +Documentation deployments have dependencies on pipelines and Docker images as follows: + +- The latest documentation pipelines and images depend on the stable documentation pipelines and images. +- The Pages deployment pipelines depend on the latest documentation images (which, in turn, depend on the stable + pipelines and images.) + +For general information on using Docker with CI/CD pipelines, see [Docker integration](../../../ci/docker/index.md). + +## Stable branches + +Stable branches for documentation include the relevant stable branches of all the projects required to publish the entire +documentation suite. For example, the stable version of documentation for version `14.4` includes: + +- The [`14.4`](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/14.4) branch of the `gitlab-docs` project. +- The [`14-4-stable-ee`](https://gitlab.com/gitlab-org/gitlab/-/tree/14-4-stable-ee) branch of the `gitlab` project. +- The [`14-4-stable`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/14-4-stable) branch of the `gitlab-runner` project. +- The [`14-4-stable`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/14-4-stable) branch of the `omnibus-gitlab` project. +- The [`5-4-stable`](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/5-4-stable) branch of the `charts/gitlab` project. + `charts/gitlab` versions are [mapped](https://docs.gitlab.com/charts/installation/version_mappings.html) to GitLab + versions. + +The Technical Writing team +[creates the stable branch](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/releases.md#create-stable-branch-and-docker-image-for-release) +for the `gitlab-docs` project, which makes use of the stable branches created by other teams. + +## Stable documentation + +When merge requests are merged that target stable branches of `gitlab-docs`, a pipeline builds +that stable documentation and deploys it to the registry. For example: + +- [14.4 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/394459635). +- [14.3 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/393774811). +- [14.2 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/393774758). +- [13.12 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/395365202). +- [12.10 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/395365405). + +In particular, the [`image:docs-single` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L655) in each pipeline +takes what is built, and pushes it to the [container registry](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). + +```mermaid +graph TD + A["14.4 MR merged"] + B["14.3 MR merged"] + C["14.2 MR merged"] + D["13.12 MR merged"] + E["12.10 MR merged"] + F{{"Container registry on `gitlab-docs` project"}} + A--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:14.4` image"-->F + B--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:14.3` image"-->F + C--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:14.2` image"-->F + D--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:13.12` image"-->F + E--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:12.10` image"-->F +``` -The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/) -contains all needed Dockerfiles to build and deploy <https://docs.gitlab.com>. It -is heavily inspired by Docker's -[Dockerfile](https://github.com/docker/docker.github.io/blob/06ed03db13895bfe867761b6fc2ad40acf6026dd/Dockerfile). +### Rebuild stable documentation images + +To rebuild any of the stable documentation images, create a [new pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/new) +for the stable branch of the image to rebuild. You might do this: + +- To include new documentation changes from an upstream stable branch into a stable version Docker image. For example, + rebuild the `14.4` Docker image to include changes subsequently merged in the `gitlab` project's + [`14-4-stable-ee`](https://gitlab.com/gitlab-org/gitlab/-/tree/14-4-stable-ee) branch. +- To incorporate changes made to the `gitlab-docs` project itself to a stable branch. For example: + - CSS style changes. + - Changes to the [version menu for a new release](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/releases.md#update-dropdown-for-online-versions). + +## Latest documentation + +A Docker image (tagged `latest`) is built that contains: + +- The latest online version of the documentation. +- The documentation from the stable branches of upstream projects. + +The [`image:docs-latest` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L678): + +- Pulls the latest documentation from the default branches of the relevant upstream projects. +- Pulls the Docker images previously built by the `image:docs-single` jobs. + +For example, [a pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/399233948) containing the +[`image:docs-latest` job](https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/1733948330): + +```mermaid +graph TD + A["Latest `gitlab`, `gitlab-runner`<br>`omnibus-gitlab`, and `charts`"] + subgraph "Container registry on `gitlab-docs` project" + B["14.4 versioned docs<br>`gitlab-docs:14.4`"] + C["14.3 versioned docs<br>`gitlab-docs:14.3`"] + D["14.2 versioned docs<br>`gitlab-docs:14.2`"] + E["13.12 versioned docs<br>`gitlab-docs:13.12`"] + F["12.10 versioned docs<br>`gitlab-docs:12.10`"] + end + G[["Scheduled pipeline<br>`image:docs-latest` job<br>combines all these"]] + A--"Default branches<br>pulled down"-->G + B--"`gitlab-docs:14.4` image<br>pulled down"-->G + C--"`gitlab-docs:14.3` image<br>pulled down"-->G + D--"`gitlab-docs:14.2` image<br>pulled down"-->G + E--"`gitlab-docs:13.12` image<br>pulled down"-->G + F--"`gitlab-docs:12.10` image<br>pulled down"-->G + H{{"Container registry on gitlab-docs project"}} + G--"Latest `gitlab-docs:latest` image<br>pushed up"-->H +``` + +## Documentation Pages deployment + +[GitLab Docs](https://docs.gitlab.com) is a [Pages site](../../../user/project/pages/index.md) and documentation updates +for it must be deployed to become available. + +The [`pages`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L491) +job runs the necessary commands to combine: + +- A very up-to-date build of the `gitlab-docs` site code. +- The latest docs from the default branches of the upstream projects. +- The documentation from `image:docs-latest`. + +For example, [a pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/399233948) containing the +[`pages` job](https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/1733948332). + +```mermaid +graph LR + A{{"Container registry on gitlab-docs project"}} + B[["Scheduled pipeline<br>`pages` and<br>`pages:deploy` job"]] + C([docs.gitlab.com]) + A--"`gitlab-docs:latest`<br>pulled"-->B + B--"Unpacked documentation uploaded"-->C +``` -The following Dockerfiles are used. +## Docker files -| Dockerfile | Docker image | Description | -| ---------- | ------------ | ----------- | -| [`Dockerfile.bootstrap`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.bootstrap) | `gitlab-docs:bootstrap` | Contains all the dependencies that are needed to build the website. If the gems are updated and `Gemfile{,.lock}` changes, the image must be rebuilt. | -| [`Dockerfile.builder.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.builder.onbuild) | `gitlab-docs:builder-onbuild` | Base image to build the docs website. It uses `ONBUILD` to perform all steps and depends on `gitlab-docs:bootstrap`. | -| [`Dockerfile.nginx.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.nginx.onbuild) | `gitlab-docs:nginx-onbuild` | Base image to use for building documentation archives. It uses `ONBUILD` to perform all required steps to copy the archive, and relies upon its parent `Dockerfile.builder.onbuild` that is invoked when building single documentation archives (see the `Dockerfile` of each branch. | -| [`Dockerfile.archives`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.archives) | `gitlab-docs:archives` | Contains all the versions of the website in one archive. It copies all generated HTML files from every version in one location. | +The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/) contains all needed +Dockerfiles to build and deploy <https://docs.gitlab.com>. It is heavily inspired by Docker's +[Dockerfile](https://github.com/docker/docker.github.io/blob/06ed03db13895bfe867761b6fc2ad40acf6026dd/Dockerfile). + +| Dockerfile | Docker image | Description | +|:---------------------------------------------------------------------------------------------------------------------------|:------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [`Dockerfile.bootstrap`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.bootstrap) | `gitlab-docs:bootstrap` | Contains all the dependencies that are needed to build the website. If the gems are updated and `Gemfile{,.lock}` changes, the image must be rebuilt. | +| [`Dockerfile.builder.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.builder.onbuild) | `gitlab-docs:builder-onbuild` | Base image to build the docs website. It uses `ONBUILD` to perform all steps and depends on `gitlab-docs:bootstrap`. | +| [`Dockerfile.nginx.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.nginx.onbuild) | `gitlab-docs:nginx-onbuild` | Base image to use for building documentation archives. It uses `ONBUILD` to perform all required steps to copy the archive, and relies upon its parent `Dockerfile.builder.onbuild` that is invoked when building single documentation archives (see the `Dockerfile` of each branch) | +| [`Dockerfile.archives`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.archives) | `gitlab-docs:archives` | Contains all the versions of the website in one archive. It copies all generated HTML files from every version in one location. | -## How to build the images +### How to build the images -Although build images are built automatically via GitLab CI/CD, you can build -and tag all tooling images locally: +Although build images are built automatically via GitLab CI/CD, you can build and tag all tooling images locally: 1. Make sure you have [Docker installed](https://docs.docker.com/install/). 1. Make sure you're in the `dockerfiles/` directory of the `gitlab-docs` repository. @@ -37,28 +169,3 @@ and tag all tooling images locally: For each image, there's a manual job under the `images` stage in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/.gitlab-ci.yml) which can be invoked at any time. - -## Update an old Docker image with new upstream docs content - -If there are any changes to any of the stable branches of the products that are -not included in the single Docker image, just rerun the pipeline (`https://gitlab.com/gitlab-org/gitlab-docs/pipelines/new`) -for the version in question. - -## Porting new website changes to old versions - -WARNING: -Porting changes to older branches can have unintended effects as we're constantly -changing the backend of the website. Use only when you know what you're doing -and make sure to test locally. - -The website keeps changing and being improved. In order to consolidate -those changes to the stable branches, we'd need to pick certain changes -from time to time. - -If this is not possible or there are many changes, merge main into them: - -```shell -git branch 12.0 -git fetch origin main -git merge origin/main -``` diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 3845586ce60..6d2b93b9462 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -97,20 +97,13 @@ mechanics of what is required is [documented below](#data-file) but, in principl The global nav has five levels: -- **Section** +- Section - Category - Doc - Doc - Doc -The majority of the links available on the nav were added according to the UI. -The match is not perfect, as for some UI nav items the documentation doesn't -apply, and there are also other links to help the new users to discover the -documentation. The docs under **Administration** are ordered alphabetically -for clarity. - -To see the improvements planned, check the -[global nav epic](https://gitlab.com/groups/gitlab-org/-/epics/1599). +You can view this structure in [the navigation.yml file](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/navigation.yaml). **Do not** [add items](#add-a-navigation-entry) to the global nav without the consent of one of the technical writers. diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index d1736e10000..60894430d15 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -124,12 +124,12 @@ Every four hours a scheduled pipeline builds and deploys the docs site. The pipe fetches the current docs from the main project's main branch, builds it with Nanoc and deploys it to <https://docs.gitlab.com>. -If you need to build and deploy the site immediately (must have maintainer level permissions): +To build and deploy the site immediately (must have the Maintainer role): 1. In [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs), go to **{rocket}** **CI/CD > Schedules**. 1. For the `Build docs.gitlab.com every 4 hours` scheduled pipeline, click the **play** (**{play}**) button. -Read more about the [deployment process](deployment_process.md). +Read more about [documentation deployments](deployment_process.md). ## Using YAML data files @@ -163,6 +163,35 @@ We can then loop over the `versions` array with something like: Note that the data file must have the `yaml` extension (not `yml`) and that we reference the array with a symbol (`:versions`). +## Archived documentation banner + +A banner is displayed on archived documentation pages with the text `This is archived documentation for +GitLab. Go to the latest.` when either: + +- The version of the documentation displayed is not the first version entry in `online` in + `content/_data/versions.yaml`. +- The documentation was built from the default branch (`main`). + +For example, if the `online` entries for `content/_data/versions.yaml` are: + +```yaml +online: + - "14.4" + - "14.3" + - "14.2" +``` + +In this case, the archived documentation banner isn't displayed: + +- For 14.4, the docs built from the `14.4` branch. The branch name is the first entry in `online`. +- For 14.5-pre, the docs built from the default project branch (`main`). + +The archived documentation banner is displayed: + +- For 14.3. +- For 14.2. +- For any other version. + ## Bumping versions of CSS and JavaScript Whenever the custom CSS and JavaScript files under `content/assets/` change, diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index 229dbb077fe..fac83af89f4 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -234,9 +234,21 @@ If you need to add more than one task, consider using subsections for each distinct task. ``` +### Related topics + +If inline links are not sufficient, you can create a topic called **Related topics** +and include a bulleted list of related topics. This topic should be above the Troubleshooting section. + +```markdown +# Related topics + +- [Configure your pipeline](link-to-topic) +- [Trigger a pipeline manually](link-to-topic) +``` + ### Topics and resources pages -This is a page with a list of links that point to important sections +This page has a list of links that point to important sections of documentation for a specific GitLab feature or tool. We do not encourage the use of these types of pages. diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 72491ab3a33..1382ec263f2 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -317,8 +317,8 @@ create an issue or an MR to propose a change to the user interface text. #### Feature names -- *Feature names are typically lowercase*. -- *Some features are capitalized*, typically nouns naming GitLab-specific +- Feature names are typically lowercase. +- Some features are capitalized, typically nouns that name GitLab-specific capabilities or tools. See the [word list](word_list.md) for details. @@ -404,13 +404,13 @@ Some contractions, however, should be avoided: | Do | Don't | |------------------------------------------|-----------------------------------------| - | Do *not* install X with Y. | *Don't* install X with Y. | + | Do not install X with Y. | Don't install X with Y. | - Do not use contractions in reference documentation. For example: | Do | Don't | |------------------------------------------|-----------------------------------------| - | Do *not* set a limit greater than 1000. | *Don't* set a limit greater than 1000. | + | Do not set a limit greater than 1000. | Don't set a limit greater than 1000. | | For `parameter1`, the default is 10. | For `parameter1`, the default's 10. | - Avoid contractions in error messages. Examples: @@ -701,7 +701,7 @@ that's best described by a matrix, tables are the best choice. To keep tables accessible and scannable, tables should not have any empty cells. If there is no otherwise meaningful value for a cell, consider entering -*N/A* (for 'not applicable') or *none*. +**N/A** for 'not applicable' or **None**. To help tables be easier to maintain, consider adding additional spaces to the column widths to make them consistent. For example: @@ -1026,18 +1026,13 @@ document to ensure it links to the most recent version of the file. When documenting how to navigate through the GitLab UI: - Always use location, then action. - - `From the **Visibility** list,` (location) `select **Public**.` (action) + - From the **Visibility** dropdown list (location), select **Public** (action). - Be brief and specific. For example: - - Avoid: `Select **Save** for the changes to take effect.` - - Use instead: `Select **Save**.` -- When selecting from high-level UI elements, use the word **on**. - - Avoid: `From the left sidebar...` or `In the left sidebar...` - - Use instead: `On the left sidebar...` -- If a step must include a reason, start the step with it. - - Avoid: `Select the link in the merge request to view the changes.` - - Use instead: `To view the changes, select the link in the merge request.` -- If a step is optional, start the step with the word `Optional` followed by a period. - - `1. Optional. Enter a name for the dog.` + - Do: Select **Save**. + - Do not: Select **Save** for the changes to take effect. +- If a step must include a reason, start the step with it. This helps the user scan more quickly. + - Do: To view the changes, in the merge request, select the link. + - Do not: Select the link in the merge request to view the changes. ### Names for menus @@ -1060,15 +1055,19 @@ Guidance for each individual UI element is in [the word list](word_list.md). To be consistent, use this format when you write navigation steps in a task topic. +```markdown 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **General pipelines**. +``` Another example: +```markdown 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **General pipelines**. +``` An Admin Area example: @@ -1082,6 +1081,42 @@ To select your avatar: 1. On the top bar, in the top right corner, select your avatar. ``` +### Optional steps + +If a step is optional, start the step with the word `Optional` followed by a period. + +For example: + +```markdown +1. Optional. Enter a description for the job. +``` + +### Documenting multiple fields at once + +If the UI text sufficiently explains the fields in a section, do not include a task step for every field. +Instead, summarize multiple fields in a single task step. + +Use the phrase **Complete the fields**. + +For example: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Push rules**. +1. Complete the fields. + +If you are documenting multiple fields and only one field needs explanation, do it in the same step: + +1. Expand **Push rules**. +1. Complete the fields. **Branch name** must be a regular expression. + +To describe multiple fields, use bullets: + +1. Expand **General pipelines**. +1. Complete the fields. + - **Branch name** must be a regular expression. + - **User** must be a user with at least the **Maintainer** role. + ## Images Images, including screenshots, can help a reader better understand a concept. diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index f1e6a147571..595dab09bf5 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -259,6 +259,16 @@ Use **box** instead of **field** or **text box**. - Do: In the **Variable name** box, enter `my text`. - Do not: In the **Variable name** field, enter `my text`. +However, you can make an exception when you are writing a task and you need to refer to all +of the fields at once. For example: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Complete the fields. + +Learn more about [documenting multiple fields at once](index.md#documenting-multiple-fields-at-once). + ## foo Do not use **foo** in product documentation. You can use it in our API and contributor documentation, but try to use a clearer and more meaningful example instead. @@ -454,6 +464,13 @@ Do not use **note that** because it's wordy. - Do: You can change the settings. - Do not: Note that you can change the settings. +## on + +When documenting how to select high-level UI elements, use the word **on**. + +- Do: `On the left sidebar...` +- Do not: `From the left sidebar...` or `In the left sidebar...` + ## once The word **once** means **one time**. Don't use it to mean **after** or **when**. @@ -524,8 +541,8 @@ Use lowercase for **runners**. These are the agents that run CI/CD jobs. See als Do not use **(s)** to make a word optionally plural. It can slow down comprehension. For example: -Do: Select the jobs you want. -Do not: Select the job(s) you want. +- Do: Select the jobs you want. +- Do not: Select the job(s) you want. If you can select multiples of something, then write the word as plural. @@ -555,6 +572,10 @@ or collapsing a section, don't include the word **section**. Use **select** with buttons, links, menu items, and lists. **Select** applies to more devices, while **click** is more specific to a mouse. +## Service Desk + +Use title case for **Service Desk**. + ## setup, set up Use **setup** as a noun, and **set up** as a verb. For example: diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 90c1137e5c5..782cd3411b1 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -39,7 +39,7 @@ The following are also added by members of the Technical Writing team: - The `~Technical Writing` [team label](../contributing/issue_workflow.md#team-labels). Documentation changes that are not associated with the release of a new or updated feature -do not take the `~feature` label, but still need the `~documentation` label. +do not take the `~"type::feature"` label, but still need the `~documentation` label. They may include: diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 68d8b424331..7c67b3495ba 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -250,6 +250,11 @@ the migration runs and set it back to that value when the migration is completed will halt the migration if the storage required is not available when the migration runs. The migration must provide the space required in bytes by defining a `space_required_bytes` method. +- `retry_on_failure` - Enable the retry on failure feature. By default, it retries + the migration 30 times. After it runs out of retries, the migration is marked as halted. + To customize the number of retries, pass the `max_attempts` argument: + `retry_on_failure max_attempts: 10` + ```ruby # frozen_string_literal: true @@ -259,6 +264,7 @@ class BatchedMigrationName < Elastic::Migration throttle_delay 10.minutes pause_indexing! space_requirements! + retry_on_failure # ... end @@ -359,17 +365,15 @@ being upgraded to, we do the following: ### Prometheus -GitLab exports [Prometheus -metrics](../administration/monitoring/prometheus/gitlab_metrics.md) relating to -the number of requests and timing for all web/API requests and Sidekiq jobs, +GitLab exports [Prometheus metrics](../administration/monitoring/prometheus/gitlab_metrics.md) +relating to the number of requests and timing for all web/API requests and Sidekiq jobs, which can help diagnose performance trends and compare how Elasticsearch timing is impacting overall performance relative to the time spent doing other things. #### Indexing queues -GitLab also exports [Prometheus -metrics](../administration/monitoring/prometheus/gitlab_metrics.md) for -indexing queues, which can help diagnose performance bottlenecks and determine +GitLab also exports [Prometheus metrics](../administration/monitoring/prometheus/gitlab_metrics.md) +for indexing queues, which can help diagnose performance bottlenecks and determine whether or not your GitLab instance or Elasticsearch server can keep up with the volume of updates. diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index 1cd3fefb7cf..af4512dcde0 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -92,7 +92,7 @@ end ``` When this code executes, the experiment is run, a variant is assigned, and (if within a -controller or view) a `window.gon.experiment.pill_color` object will be available in the +controller or view) a `window.gl.experiments.pill_color` object will be available in the client layer, with details like: - The assigned variant. @@ -522,14 +522,14 @@ shared example: [tracks assignment and records the subject](https://gitlab.com/g This is in flux as of GitLab 13.10, and can't be documented just yet. -Any experiment that's been run in the request lifecycle surfaces in `window.gon.experiment`, +Any experiment that's been run in the request lifecycle surfaces in and `window.gl.experiments`, and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0) so you can use it when resolving some concepts around experimentation in the client layer. ### Use experiments in Vue With the `gitlab-experiment` component, you can define slots that match the name of the -variants pushed to `window.gon.experiment`. For example, if we alter the `pill_color` +variants pushed to `window.gl.experiments`. For example, if we alter the `pill_color` experiment to just use the default variants of `control` and `candidate` like so: ```ruby @@ -587,7 +587,51 @@ For example, the Vue component for the previously-defined `pill_color` experimen ``` NOTE: -When there is no experiment data in the `window.gon.experiment` object for the given experiment name, the `control` slot will be used, if it exists. +When there is no experiment data in the `window.gl.experiments` object for the given experiment name, the `control` slot will be used, if it exists. + +## Test with Jest + +### Stub Helpers + +You can stub experiments using the `stubExperiments` helper defined in `spec/frontend/__helpers__/experimentation_helper.js`. + +```javascript +import { stubExperiments } from 'helpers/experimentation_helper'; +import { getExperimentData } from '~/experimentation/utils'; + +describe('when my_experiment is enabled', () => { + beforeEach(() => { + stubExperiments({ my_experiment: 'candidate' }); + }); + + it('sets the correct data', () => { + expect(getExperimentData('my_experiment')).toEqual({ experiment: 'my_experiment', variant: 'candidate' }); + }); +}); +``` + +NOTE: +This method of stubbing in Jest specs will not automatically un-stub itself at the end of the test. We merge our stubbed experiment in with all the other global data in `window.gl`. If you need to remove the stubbed experiment(s) after your test or ensure a clean global object before your test, you'll need to manage the global object directly yourself: + +```javascript +desribe('tests that care about global state', () => { + const originalObjects = []; + + beforeEach(() => { + // For backwards compatibility for now, we're using both window.gon & window.gl + originalObjects.push(window.gon, window.gl); + }); + + afterEach(() => { + [window.gon, window.gl] = originalObjects; + }); + + it('stubs experiment in fresh global state', () => { + stubExperiment({ my_experiment: 'candidate' }); + // ... + }); +}) +``` ## Notes on feature flags diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index c4ebef4c289..e71e414002a 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -23,6 +23,17 @@ they found that "ARIA correlated to higher detectable errors". It is likely that *misuse* of ARIA is a big cause of increased errors, so when in doubt don't use `aria-*`, `role`, and `tabindex` and stick with semantic HTML. +## Enable keyboard navigation on macOS + +By default, macOS limits the <kbd>tab</kbd> key to **Text boxes and lists only**. To enable full keyboard navigation: + +1. Open **System Preferences**. +1. Select **Keyboard**. +1. Open the **Shortcuts** tab. +1. Enable the setting **Use keyboard navigation to move focus between controls**. + +You can read more about enabling browser-specific keyboard navigation on [a11yproject](https://www.a11yproject.com/posts/2017-12-29-macos-browser-keyboard-navigation/). + ## Quick checklist - [Text](#text-inputs-with-accessible-names), diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md index 4e50621add4..cd82a7dadc3 100644 --- a/doc/development/fe_guide/development_process.md +++ b/doc/development/fe_guide/development_process.md @@ -56,7 +56,7 @@ Please use your best judgment when to use it and please contribute new points th - [ ] **Performance** Have you checked performance? For example do the same thing with 500 comments instead of 1. Document the tests and possible findings in the MR so a reviewer can directly see it. - [ ] Have you tested with a variety of our [supported browsers](../../install/requirements.md#supported-web-browsers)? You can use [browserstack](https://www.browserstack.com/) to be able to access a wide variety of browsers and operating systems. - [ ] Did you check the mobile view? -- [ ] Check the built webpack bundle (For the report run `WEBPACK_REPORT=true gdk run`, then open `webpack-report/index.html`) if we have unnecessary bloat due to wrong references, including libraries multiple times, etc.. If you need help contact the webpack [domain expert](https://about.gitlab.com/handbook/engineering/frontend/#frontend-domain-experts) +- [ ] Check the built webpack bundle (For the report run `WEBPACK_REPORT=true gdk start`, then open `webpack-report/index.html`) if we have unnecessary bloat due to wrong references, including libraries multiple times, etc.. If you need help contact the webpack [domain expert](https://about.gitlab.com/handbook/engineering/frontend/#frontend-domain-experts) - [ ] **Tests** Not only greenfield tests - Test also all bad cases that come to your mind. - [ ] If you have multiple MR's then also smoke test against the final merge. - [ ] Are there any big changes on how and especially how frequently we use the API then let production know about it diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 0229aa0123a..44d43a32803 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -103,7 +103,6 @@ Default client accepts two parameters: `resolvers` and `config`. - `config` parameter takes an object of configuration settings: - `cacheConfig` field accepts an optional object of settings to [customize Apollo cache](https://www.apollographql.com/docs/react/caching/cache-configuration/#configuring-the-cache) - `baseUrl` allows us to pass a URL for GraphQL endpoint different from our main endpoint (for example, `${gon.relative_url_root}/api/graphql`) - - `assumeImmutableResults` (set to `false` by default) - this setting, when set to `true`, assumes that every single operation on updating Apollo Cache is immutable. It also sets `freezeResults` to `true`, so any attempt on mutating Apollo Cache throws a console warning in development environment. Please ensure you're following the immutability pattern on cache update operations before setting this option to `true`. - `fetchPolicy` determines how you want your component to interact with the Apollo cache. Defaults to "cache-first". ### Multiple client queries for the same object @@ -179,7 +178,7 @@ with a **new and updated** object. To facilitate the process of updating the cache and returning the new object we use the library [Immer](https://immerjs.github.io/immer/). -When possible, follow these conventions: +Please, follow these conventions: - The updated cache is named `data`. - The original cache data is named `sourceData`. @@ -204,11 +203,6 @@ client.writeQuery({ As shown in the code example by using `produce`, we can perform any kind of direct manipulation of the `draftState`. Besides, `immer` guarantees that a new state which includes the changes to `draftState` is generated. -Finally, to verify whether the immutable cache update is working properly, we need to change -`assumeImmutableResults` to `true` in the default client configuration. See [Apollo Client](#apollo-client) for more information. - -If everything is working properly `assumeImmutableResults` should remain set to `true`. - ## Usage in Vue To use Vue Apollo, import the [Vue Apollo](https://github.com/vuejs/vue-apollo) plugin as well @@ -1106,17 +1100,15 @@ To test the logic of Apollo cache updates, we might want to mock an Apollo Clien To separate tests with mocked client from 'usual' unit tests, create an additional factory and pass the created `mockApollo` as an option to the `createComponent`-factory. This way we only create Apollo Client instance when it's necessary. -We need to inject `VueApollo` to the Vue local instance and, likewise, it is recommended to call `localVue.use()` in `createMockApolloProvider()` to only load it when it is necessary. +We need to inject `VueApollo` into the Vue instance by calling `Vue.use(VueApollo)`. This will install `VueApollo` globally for all the tests in the file. It is recommended to call `Vue.use(VueApollo)` just after the imports. ```javascript import VueApollo from 'vue-apollo'; -import { createLocalVue } from '@vue/test-utils'; +import Vue from 'vue'; -const localVue = createLocalVue(); +Vue.use(VueApollo); function createMockApolloProvider() { - localVue.use(VueApollo); - return createMockApollo(requestHandlers); } @@ -1124,7 +1116,6 @@ function createComponent(options = {}) { const { mockApollo } = options; ... return shallowMount(..., { - localVue, apolloProvider: mockApollo, ... }); @@ -1194,7 +1185,6 @@ describe('Some component', () => { const { mockApollo } = options; return shallowMount(Index, { - localVue, apolloProvider: mockApollo, }); } @@ -1281,7 +1271,6 @@ function createComponent(options = {}) { const { mockApollo } = options; return shallowMount(Index, { - localVue, apolloProvider: mockApollo, }); } @@ -1414,7 +1403,6 @@ const createComponentWithApollo = ({ props = {} } = {}) => { mockApollo = createMockApollo([], mockResolvers); // resolvers are the second parameter wrapper = shallowMount(MyComponent, { - localVue, propsData: {}, apolloProvider: mockApollo, // ... @@ -1482,7 +1470,6 @@ function createComponent(options = {}) { const { mockApollo } = options; return shallowMount(Index, { - localVue, apolloProvider: mockApollo, }); } diff --git a/doc/development/fe_guide/storybook.md b/doc/development/fe_guide/storybook.md index a46157d2cad..9c4bcf02389 100644 --- a/doc/development/fe_guide/storybook.md +++ b/doc/development/fe_guide/storybook.md @@ -47,8 +47,9 @@ To add a story: 1. Write the story as per the [official Storybook instructions](https://storybook.js.org/docs/vue/writing-stories/introduction/) Notes: - - Specify the `title` field of the story as the component's file path from the `javascripts/` directory, - e.g. if the component is located at `app/assets/javascripts/vue_shared/components/sidebar/todo_toggle/todo_button.vue`, specify the story `title` as `vue_shared/components/sidebar/todo_toggle/todo_button`. This will ensure the Storybook navigation maps closely to our internal directory structure. + - Specify the `title` field of the story as the component's file path from the `javascripts/` directory. + + For example, if the component is located at `app/assets/javascripts/vue_shared/components/sidebar/todo_toggle/todo_button.vue`, specify the story `title` as `vue_shared/components/sidebar/todo_toggle/todo_button`. This will ensure the Storybook navigation maps closely to our internal directory structure. ## Mock backend APIs diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index ffaaa3e87c7..5d5b250e9a9 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -133,6 +133,40 @@ Before adding a new variable for a color or a size, guarantee: - There isn't an existing one. - There isn't a similar one we can use instead. +### Using `extend` at-rule + +Usage of the `extend` at-rule is prohibited due to [memory leaks](https://gitlab.com/gitlab-org/gitlab/-/issues/323021) and [the rule doesn't work as it should to](https://sass-lang.com/documentation/breaking-changes/extend-compound). Use mixins instead: + +```scss +// Bad +.gl-pt-3 { + padding-top: 12px; +} + +.my-element { + @extend .gl-pt-3; +} + +// compiles to +.gl-pt-3, .my-element { + padding-top: 12px; +} + +// Good +@mixing gl-pt-3 { + padding-top: 12px; +} + +.my-element { + @include gl-pt-3; +} + +// compiles to +.my-element { + padding-top: 12px; +} +``` + ## Linting We use [stylelint](https://stylelint.io) to check for style guide conformity. It uses the diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 509e2f4b688..5d5d37e0398 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -73,9 +73,8 @@ component, is that you avoid the need to create a fixture or an HTML element in ##### provide/inject Vue supports dependency injection through [provide/inject](https://vuejs.org/v2/api/#provide-inject). -Values passed to the component through `provide` can be accessed in the component the `inject` configuration. -In the following example of a Vue app initialization, a value from HAML is passed to the component -through the `provide` configuration: +In the component the `inject` configuration accesses the values `provide` passes down. +This example of a Vue app initialization shows how the `provide` configuration passes a value from HAML to the component: ```javascript #js-vue-app{ data: { endpoint: 'foo' }} @@ -251,7 +250,7 @@ return new Vue({ render(createElement) { return createElement('my-component', { props: { - username: gon.current_username, + avatarUrl: gl.avatarUrl, }, }); }, diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index 7da462a5f89..c6f480deb22 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -29,6 +29,12 @@ Component's computed properties / methods or external helpers. `$on`, `$once`, and `$off` methods [are removed](https://github.com/vuejs/rfcs/blob/master/active-rfcs/0020-events-api-change.md) from the Vue instance, so in Vue 3 it can't be used to create an event hub. +**When to use** + +If you are in a Vue app that doesn't use any event hub, try to avoid adding a new one unless absolutely necessary. For example, if you need a child component to react to its parent's event, it's preferred to pass a prop down. Then, use the watch property on that prop in the child component to create the desired side effect. + +If you need cross-component communication (between different Vue apps), then perhaps introducing a hub is the right decision. + **What to use instead** Vue documentation recommends using the [mitt](https://github.com/developit/mitt) library. It's relatively small (200 bytes, compressed) and has a clear API: diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md index 20325facc75..d6b64001e13 100644 --- a/doc/development/feature_categorization/index.md +++ b/doc/development/feature_categorization/index.md @@ -96,7 +96,7 @@ second argument: ```ruby class DashboardController < ApplicationController - feature_category :issue_tracking, [:issues, :issues_calendar] + feature_category :team_planning, [:issues, :issues_calendar] feature_category :code_review, [:merge_requests] end ``` @@ -137,7 +137,7 @@ Grape API endpoints can use the `feature_category` class method, like ```ruby module API class Issues < ::API::Base - feature_category :issue_tracking + feature_category :team_planning end end ``` diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 35dbc2703f9..abb100c659e 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -90,9 +90,9 @@ This depends on the feature and what sort of impact it might have. Guidelines: -1. If the feature meets the requirements for creating a [Change Management](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process) issue, create a Change Management issue per [criticality guidelines](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows). -1. For simple, low-risk, easily reverted features, proceed and [enable the feature in `#production`](#process). -1. For features that impact the user experience, consider notifying `#support_gitlab-com` beforehand. +- Consider notifying `#support_gitlab-com` beforehand. So in case if the feature has any side effects on user experience, they can mitigate and disable the feature flag to reduce some impact. +- If the feature meets the requirements for creating a [Change Management](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process) issue, create a Change Management issue per [criticality guidelines](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows). +- For simple, low-risk, easily reverted features, proceed and [enable the feature in `#production`](#process). #### Process diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 987ff7c9fe8..86dc4c73062 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -535,7 +535,7 @@ to ensure the feature works properly. When using the testing environment, all feature flags are enabled by default. WARNING: -This does not apply to end-to-end (QA) tests, which [do not disable feature flags by default](#end-to-end-qa-tests). There is a different [process for using feature flags in end-to-end tests](../testing_guide/end_to_end/feature_flags.md). +This does not apply to end-to-end (QA) tests, which [do not enable feature flags by default](#end-to-end-qa-tests). There is a different [process for using feature flags in end-to-end tests](../testing_guide/end_to_end/feature_flags.md). To disable a feature flag in a test, use the `stub_feature_flags` helper. For example, to globally disable the `ci_live_trace` feature diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index e071aa4ffd9..5ff1bc7b127 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -55,8 +55,10 @@ to a gem, go through these steps: - For an example, see the [merge request !57805](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57805). 1. Once the gem is stable - we have been using it in production for a while with few, if any, changes - extract to its own project under - the `gitlab-org` namespace. - 1. When creating the project, follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/#creating-a-new-project). + the [`gitlab-org/ruby/gems` namespace](https://gitlab.com/gitlab-org/ruby/gems/). + + - To create this project: + 1. Follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/#creating-a-new-project). 1. Follow the instructions for setting up a [CI/CD configuration](https://about.gitlab.com/handbook/engineering/#cicd-configuration). 1. Follow the instructions for [publishing a project](https://about.gitlab.com/handbook/engineering/#publishing-a-project). - See [issue diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index fa4fa59e35d..5a7a5a6abcb 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -53,7 +53,7 @@ immediately after the Gitaly one. This allows you to test your changes before they are merged. - See [below](#running-tests-with-a-locally-modified-version-of-gitaly) for instructions on running GitLab tests with a modified version of Gitaly. -- In GDK run `gdk install` and restart `gdk run` (or `gdk run app`) to use a locally modified Gitaly version for development +- In GDK run `gdk install` and restart GDK using `gdk restart` to use a locally modified Gitaly version for development ### `gitaly-ruby` diff --git a/doc/development/go_guide/go_upgrade.md b/doc/development/go_guide/go_upgrade.md index 75a093a55ac..53f2d7d176a 100644 --- a/doc/development/go_guide/go_upgrade.md +++ b/doc/development/go_guide/go_upgrade.md @@ -89,10 +89,14 @@ if you need help finding the correct person or labels: 1. Schedule an update with the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/-/issues): - Title the issue `Support using Go version <VERSION_NUMBER>`. - Set the issue as related to every issue created in the previous step. -1. Schedule one issue per Secure Stage team and add the `devops::secure` label to each: +1. Schedule one issue per Sec Section team that maintains Go based Security Analyzers and add the `section::sec` label to each: - [Static Analysis tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). - [Composition Analysis tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). - [Container Security tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). + + NOTE: + Updates to these Security analyzers should not block upgrades to Charts or Omnibus since + the analyzers are built independently as separate container images. 1. Schedule builder updates with Distribution projects: - Dependency and GitLab Development Kit issues created in previous steps should be set as blockers. - Each issue should have the title `Support building with Go <VERSION_NUMBER>` and description as noted: diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 44f8924be04..9bf8b7ef89a 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -364,8 +364,7 @@ JSON format, as all our infrastructure assumes that. When using [Logrus](https://github.com/sirupsen/logrus) you can turn on structured logging simply by using the build in [JSON formatter](https://github.com/sirupsen/logrus#formatters). This follows the -same logging type we use in our [Ruby -applications](../logging.md#use-structured-json-logging). +same logging type we use in our [Ruby applications](../logging.md#use-structured-json-logging). #### How to use Logrus @@ -451,6 +450,78 @@ The conventional Secure [analyzer](https://gitlab.com/gitlab-org/security-produc If the scanner report is small, less than 35 lines, then feel free to [inline the report](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow/-/blob/8bd2428a/convert/convert_test.go#L13-77) rather than use a `testdata` directory. +#### Test Diffs + +The [go-cmp]<https://github.com/google/go-cmp> package should be used when comparing large structs in tests. It makes it possible to output a specific diff where the two structs differ, rather than seeing the whole of both structs printed out in the test logs. Here is a small example: + +```golang +package main + +import ( + "reflect" + "testing" + + "github.com/google/go-cmp/cmp" +) + +type Foo struct { + Desc Bar + Point Baz +} + +type Bar struct { + A string + B string +} + +type Baz struct { + X int + Y int +} + +func TestHelloWorld(t *testing.T) { + want := Foo{ + Desc: Bar{A: "a", B: "b"}, + Point: Baz{X: 1, Y: 2}, + } + + got := Foo{ + Desc: Bar{A: "a", B: "b"}, + Point: Baz{X: 2, Y: 2}, + } + + t.Log("reflect comparison:") + if !reflect.DeepEqual(got, want) { + t.Errorf("Wrong result. want:\n%v\nGot:\n%v", want, got) + } + + t.Log("cmp comparison:") + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("Wrong result. (-want +got):\n%s", diff) + } +} +``` + +The output demonstrates why `go-cmp` is far superior when comparing large structs. Even though you could spot the difference with this small difference, it quickly gets unwieldy as the data grows. + +```plaintext + main_test.go:36: reflect comparison: + main_test.go:38: Wrong result. want: + {{a b} {1 2}} + Got: + {{a b} {2 2}} + main_test.go:41: cmp comparison: + main_test.go:43: Wrong result. (-want +got): + main.Foo{ + Desc: {A: "a", B: "b"}, + Point: main.Baz{ + - X: 1, + + X: 2, + Y: 2, + }, + } +``` + --- [Return to Development documentation](../index.md). diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 34b7f5e8763..04a9f68abec 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -48,10 +48,10 @@ Be sure to check the following guidelines before you translate any strings. ### Namespaced strings -When an externalized string is prepended with a namespace (for example, -`s_('OpenedNDaysAgo|Opened')`), the namespace should be removed from the final translation. For -example, in French, `OpenedNDaysAgo|Opened` is translated to `Ouvert•e`, not -`OpenedNDaysAgo|Ouvert•e`. +A namespace precedes the string and is separated from it by a `|` (`namespace|string`). When you see +a namespace before an externalized string, you should remove the namespace from the final +translation. For example, in `OpenedNDaysAgo|Opened`, remove `OpenedNDaysAgo|`. If translating to +French, translate `OpenedNDaysAgo|Opened` to `Ouvert•e`, not `OpenedNDaysAgo|Ouvert•e`. ### Technical terms @@ -61,6 +61,12 @@ should always be in English are noted in the glossary when using This helps maintain a logical connection and consistency between tools (for example, a Git client) and GitLab. +To find the list of technical terms: + +1. Go to [`translate.gitlab.com`](https://translate.gitlab.com). +1. Select the language to translate. +1. Select **Glossary**. + ### Formality The level of formality used in software varies by language: diff --git a/doc/development/index.md b/doc/development/index.md index 3780c986367..fa49d43d46c 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -122,7 +122,7 @@ In these cases, use the following workflow: - [User Experience (UX)](https://about.gitlab.com/handbook/engineering/ux/) - [Security](https://about.gitlab.com/handbook/engineering/security/) - [Quality](https://about.gitlab.com/handbook/engineering/quality/) - - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity-team/) + - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/) - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/) - [Technical Writing](https://about.gitlab.com/handbook/engineering/ux/technical-writing/) @@ -181,6 +181,7 @@ the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/rev - [Avoid modules with instance variables](module_with_instance_variables.md), if possible - [Guidelines for reusing abstractions](reusing_abstractions.md) +- [Ruby 3 gotchas](ruby3_gotchas.md) ### Rails Framework related diff --git a/doc/development/internal_api.md b/doc/development/internal_api.md index 660d8c60ba8..7790a5e23e6 100644 --- a/doc/development/internal_api.md +++ b/doc/development/internal_api.md @@ -1,829 +1,9 @@ --- -stage: Create -group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, api +redirect_to: 'internal_api/index.md' +remove_date: '2022-02-09' --- -# Internal API **(FREE)** +This document was moved to [another location](internal_api/index.md). -The internal API is used by different GitLab components, it can not be -used by other consumers. This documentation is intended for people -working on the GitLab codebase. - -This documentation does not yet include the internal API used by -GitLab Pages. - -## Adding new endpoints - -API endpoints should be externally accessible by default, with proper authentication and authorization. -Before adding a new internal endpoint, consider if the API would potentially be -useful to the wider GitLab community and can be made externally accessible. - -One reason we might favor internal API endpoints sometimes is when using such an endpoint requires -internal data that external actors can not have. For example, in the internal Pages API we might use -a secret token that identifies a request as internal or sign a request with a public key that is -not available to a wider community. - -Another reason to separate something into an internal API is when request to such API endpoint -should never go through an edge (public) load balancer. This way we can configure different rate -limiting rules and policies around how the endpoint is being accessed, because we know that only -internal requests can be made to that endpoint going through an internal load balancer. - -## Authentication - -These methods are all authenticated using a shared secret. This secret -is stored in a file at the path configured in `config/gitlab.yml` by -default this is in the root of the rails app named -`.gitlab_shell_secret` - -To authenticate using that token, clients read the contents of that -file, and include the token Base64 encoded in a `secret_token` parameter -or in the `Gitlab-Shared-Secret` header. - -NOTE: -The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (`kas`) uses JSON Web Token (JWT) -authentication, which is different from GitLab Shell. - -## Git Authentication - -This is called by [Gitaly](https://gitlab.com/gitlab-org/gitaly) and -[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) to check access to a -repository. - -- **When called from GitLab Shell**: No changes are passed, and the internal - API replies with the information needed to pass the request on to Gitaly. -- **When called from Gitaly in a `pre-receive` hook**: The changes are passed - and validated to determine if the push is allowed. - -Calls are limited to 50 seconds each. - -```plaintext -POST /internal/allowed -``` - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key_id` | string | no | ID of the SSH-key used to connect to GitLab Shell | -| `username` | string | no | Username from the certificate used to connect to GitLab Shell | -| `project` | string | no (if `gl_repository` is passed) | Path to the project | -| `gl_repository` | string | no (if `project` is passed) | Repository identifier, such as `project-7` | -| `protocol` | string | yes | SSH when called from GitLab Shell, HTTP or SSH when called from Gitaly | -| `action` | string | yes | Git command being run (`git-upload-pack`, `git-receive-pack`, `git-upload-archive`) | -| `changes` | string | yes | `<oldrev> <newrev> <refname>` when called from Gitaly, the magic string `_any` when called from GitLab Shell | -| `check_ip` | string | no | IP address from which call to GitLab Shell was made | - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ - --data "key_id=11&project=gnuwget/wget2&action=git-upload-pack&protocol=ssh" \ - "http://localhost:3001/api/v4/internal/allowed" -``` - -Example response: - -```json -{ - "status": true, - "gl_repository": "project-3", - "gl_project_path": "gnuwget/wget2", - "gl_id": "user-1", - "gl_username": "root", - "git_config_options": [], - "gitaly": { - "repository": { - "storage_name": "default", - "relative_path": "@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git", - "git_object_directory": "", - "git_alternate_object_directories": [], - "gl_repository": "project-3", - "gl_project_path": "gnuwget/wget2" - }, - "address": "unix:/Users/bvl/repos/gitlab/gitaly.socket", - "token": null - }, - "gl_console_messages": [] -} -``` - -### Known consumers - -- Gitaly -- GitLab Shell - -## LFS Authentication - -This is the endpoint that gets called from GitLab Shell to provide -information for LFS clients when the repository is accessed over SSH. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key_id` | string | no | ID of the SSH-key used to connect to GitLab Shell | -| `username`| string | no | Username from the certificate used to connect to GitLab Shell | -| `project` | string | no | Path to the project | - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ - --data "key_id=11&project=gnuwget/wget2" "http://localhost:3001/api/v4/internal/lfs_authenticate" -``` - -```json -{ - "username": "root", - "lfs_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkYXRhIjp7ImFjdG9yIjoicm9vdCJ9LCJqdGkiOiIyYWJhZDcxZC0xNDFlLTQ2NGUtOTZlMi1mODllYWRiMGVmZTYiLCJpYXQiOjE1NzAxMTc2NzYsIm5iZiI6MTU3MDExNzY3MSwiZXhwIjoxNTcwMTE5NDc2fQ.g7atlBw1QMY7QEBVPE0LZ8ZlKtaRzaMRmNn41r2YITM", - "repository_http_path": "http://localhost:3001/gnuwget/wget2.git", - "expires_in": 1800 -} -``` - -### Known consumers - -- GitLab Shell - -## Authorized Keys Check - -This endpoint is called by the GitLab Shell authorized keys -check. Which is called by OpenSSH for [fast SSH key -lookup](../administration/operations/fast_ssh_key_lookup.md). - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key` | string | yes | SSH key as passed by OpenSSH to GitLab Shell | - -```plaintext -GET /internal/authorized_keys -``` - -Example request: - -```shell -curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/authorized_keys?key=<key as passed by OpenSSH>" -``` - -Example response: - -```json -{ - "id": 11, - "title": "admin@example.com", - "key": "ssh-rsa ...", - "created_at": "2019-06-27T15:29:02.219Z" -} -``` - -### Known consumers - -- GitLab Shell - -## Get user for user ID or key - -This endpoint is used when a user performs `ssh git@gitlab.com`. It -discovers the user associated with an SSH key. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | -| `username` | string | no | Username of the user being looked up, used by GitLab Shell when authenticating using a certificate | - -```plaintext -GET /internal/discover -``` - -Example request: - -```shell -curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/discover?key_id=7" -``` - -Example response: - -```json -{ - "id": 7, - "name": "Dede Eichmann", - "username": "rubi" -} -``` - -### Known consumers - -- GitLab Shell - -## Instance information - -This gets some generic information about the instance. This is used -by Geo nodes to get information about each other. - -```plaintext -GET /internal/check -``` - -Example request: - -```shell -curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/check" -``` - -Example response: - -```json -{ - "api_version": "v4", - "gitlab_version": "12.3.0-pre", - "gitlab_rev": "d69c988e6a6", - "redis": true -} -``` - -### Known consumers - -- GitLab Geo -- GitLab Shell's `bin/check` -- Gitaly - -## Get new 2FA recovery codes using an SSH key - -This is called from GitLab Shell and allows users to get new 2FA -recovery codes based on their SSH key. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | -| `user_id` | integer | no | **Deprecated** User_id for which to generate new recovery codes | - -```plaintext -GET /internal/two_factor_recovery_codes -``` - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ - --data "key_id=7" "http://localhost:3001/api/v4/internal/two_factor_recovery_codes" -``` - -Example response: - -```json -{ - "success": true, - "recovery_codes": [ - "d93ee7037944afd5", - "19d7b84862de93dd", - "1e8c52169195bf71", - "be50444dddb7ca84", - "26048c77d161d5b7", - "482d5c03d1628c47", - "d2c695e309ce7679", - "dfb4748afc4f12a7", - "0e5f53d1399d7979", - "af04d5622153b020" - ] -} -``` - -### Known consumers - -- GitLab Shell - -## Get new personal access-token - -This is called from GitLab Shell and allows users to generate a new -personal access token. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `name` | string | yes | The name of the new token | -| `scopes` | string array | yes | The authorization scopes for the new token, these must be valid token scopes | -| `expires_at` | string | no | The expiry date for the new token | -| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | -| `user_id` | integer | no | User\_id for which to generate the new token | - -```plaintext -POST /internal/personal_access_token -``` - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ - --data "user_id=29&name=mytokenname&scopes[]=read_user&scopes[]=read_repository&expires_at=2020-07-24" \ - "http://localhost:3001/api/v4/internal/personal_access_token" -``` - -Example response: - -```json -{ - "success": true, - "token": "Hf_79B288hRv_3-TSD1R", - "scopes": ["read_user","read_repository"], - "expires_at": "2020-07-24" -} -``` - -### Known consumers - -- GitLab Shell - -## Incrementing counter on pre-receive - -This is called from the Gitaly hooks increasing the reference counter -for a push that might be accepted. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `gl_repository` | string | yes | repository identifier for the repository receiving the push | - -```plaintext -POST /internal/pre_receive -``` - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ - --data "gl_repository=project-7" "http://localhost:3001/api/v4/internal/pre_receive" -``` - -Example response: - -```json -{ - "reference_counter_increased": true -} -``` - -## PostReceive - -Called from Gitaly after a receiving a push. This triggers the -`PostReceive`-worker in Sidekiq, processes the passed push options and -builds the response including messages that need to be displayed to -the user. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `identifier` | string | yes | `user-[id]` or `key-[id]` Identifying the user performing the push | -| `gl_repository` | string | yes | identifier of the repository being pushed to | -| `push_options` | string array | no | array of push options | -| `changes` | string | no | refs to be updated in the push in the format `oldrev newrev refname\n`. | - -```plaintext -POST /internal/post_receive -``` - -Example Request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ - --data "gl_repository=project-7" --data "identifier=user-1" \ - --data "changes=0000000000000000000000000000000000000000 fd9e76b9136bdd9fe217061b497745792fe5a5ee gh-pages\n" \ - "http://localhost:3001/api/v4/internal/post_receive" -``` - -Example response: - -```json -{ - "messages": [ - { - "message": "Hello from post-receive", - "type": "alert" - } - ], - "reference_counter_decreased": true -} -``` - -## Kubernetes agent endpoints - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41045) in GitLab 13.4. -> - This feature is not deployed on GitLab.com -> - It's not recommended for production use. - -The following endpoints are used by the GitLab Kubernetes Agent Server (`kas`) -for various purposes. - -These endpoints are all authenticated using JWT. The JWT secret is stored in a file -specified in `config/gitlab.yml`. By default, the location is in the root of the -GitLab Rails app in a file called `.gitlab_kas_secret`. - -WARNING: -The Kubernetes agent is under development and is not recommended for production use. - -### Kubernetes agent information - -Called from GitLab Kubernetes Agent Server (`kas`) to retrieve agent -information for the given agent token. This returns the Gitaly connection -information for the agent's project in order for `kas` to fetch and update -the agent's configuration. - -```plaintext -GET /internal/kubernetes/agent_info -``` - -Example Request: - -```shell -curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ - --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" -``` - -### Kubernetes agent project information - -Called from GitLab Kubernetes Agent Server (`kas`) to retrieve project -information for the given agent token. This returns the Gitaly -connection for the requested project. GitLab `kas` uses this to configure -the agent to fetch Kubernetes resources from the project repository to -sync. - -Only public projects are supported. For private projects, the ability for the -agent to be authorized is [not yet implemented](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../api/index.md#namespaced-path-encoding) | - -```plaintext -GET /internal/kubernetes/project_info -``` - -Example Request: - -```shell -curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ - --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" -``` - -### Kubernetes agent usage metrics - -Called from GitLab Kubernetes Agent Server (`kas`) to increase the usage -metric counters. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `gitops_sync_count` | integer| no | The number to increase the `gitops_sync_count` counter by | -| `k8s_api_proxy_request_count` | integer| no | The number to increase the `k8s_api_proxy_request_count` counter by | - -```plaintext -POST /internal/kubernetes/usage_metrics -``` - -Example Request: - -```shell -curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Content-Type: application/json" \ - --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" -``` - -### Kubernetes agent alert metrics - -Called from GitLab Kubernetes Agent Server (KAS) to save alerts derived from Cilium on Kubernetes -Cluster. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `alert` | Hash | yes | Alerts detail. Same format as [3rd party alert](../operations/incident_management/integrations.md#customize-the-alert-payload-outside-of-gitlab). | - -```plaintext -POST internal/kubernetes/modules/cilium_alert -``` - -Example Request: - -```shell -curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ - --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ - --data '"{\"alert\":{\"title\":\"minimal\",\"message\":\"network problem\",\"evalMatches\":[{\"value\":1,\"metric\":\"Count\",\"tags\":{}}]}}"' \ - "http://localhost:3000/api/v4/internal/kubernetes/modules/cilium_alert" -``` - -### Create Starboard vulnerability - -Called from the GitLab Kubernetes Agent Server (`kas`) to create a security vulnerability -from a Starboard vulnerability report. This request is idempotent. Multiple requests with the same data -create a single vulnerability. - -| Attribute | Type | Required | Description | -|:----------------|:-------|:---------|:------------| -| `vulnerability` | Hash | yes | Vulnerability data matching the security report schema [`vulnerability` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | -| `scanner` | Hash | yes | Scanner data matching the security report schema [`scanner` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | - -```plaintext -PUT internal/kubernetes/modules/starboard_vulnerability -``` - -Example Request: - -```shell -curl --request PUT --header "Gitlab-Kas-Api-Request: <JWT token>" \ - --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ - --url "http://localhost:3000/api/v4/internal/kubernetes/modules/starboard_vulnerability" \ - --data '{ - "vulnerability": { - "name": "CVE-123-4567 in libc", - "severity": "high", - "confidence": "unknown", - "location": { - "kubernetes_resource": { - "namespace": "production", - "kind": "deployment", - "name": "nginx", - "container": "nginx" - } - }, - "identifiers": [ - { - "type": "cve", - "name": "CVE-123-4567", - "value": "CVE-123-4567" - } - ] - }, - "scanner": { - "id": "starboard_trivy", - "name": "Trivy (via Starboard Operator)", - "vendor": "GitLab" - } -}' -``` - -## Subscriptions - -The subscriptions endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) -in order to apply subscriptions including trials, and add-on purchases, for personal namespaces or top-level groups within GitLab.com. - -### Creating a subscription - -Use a POST to create a subscription. - -```plaintext -POST /namespaces/:id/gitlab_subscription -``` - -| Attribute | Type | Required | Description | -|:------------|:--------|:---------|:------------| -| `start_date` | date | yes | Start date of subscription | -| `end_date` | date | no | End date of subscription | -| `plan_code` | string | no | Subscription tier code | -| `seats` | integer | no | Number of seats in subscription | -| `max_seats_used` | integer | no | Highest number of active users in the last month | -| `auto_renew` | boolean | no | Whether subscription auto-renews on end date | -| `trial` | boolean | no | Whether subscription is a trial | -| `trial_starts_on` | date | no | Start date of trial | -| `trial_ends_on` | date | no | End date of trial | - -Example request: - -```shell -curl --request POST --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription?start_date="2020-07-15"&plan="premium"&seats=10" -``` - -Example response: - -```json -{ - "plan": { - "code":"premium", - "name":"premium", - "trial":false, - "auto_renew":null, - "upgradable":false - }, - "usage": { - "seats_in_subscription":10, - "seats_in_use":1, - "max_seats_used":0, - "seats_owed":0 - }, - "billing": { - "subscription_start_date":"2020-07-15", - "subscription_end_date":null, - "trial_ends_on":null - } -} -``` - -### Updating a subscription - -Use a PUT command to update an existing subscription. - -```plaintext -PUT /namespaces/:id/gitlab_subscription -``` - -| Attribute | Type | Required | Description | -|:------------|:--------|:---------|:------------| -| `start_date` | date | no | Start date of subscription | -| `end_date` | date | no | End date of subscription | -| `plan_code` | string | no | Subscription tier code | -| `seats` | integer | no | Number of seats in subscription | -| `max_seats_used` | integer | no | Highest number of active users in the last month | -| `auto_renew` | boolean | no | Whether subscription auto-renews on end date | -| `trial` | boolean | no | Whether subscription is a trial | -| `trial_starts_on` | date | no | Start date of trial. Required if trial is true. | -| `trial_ends_on` | date | no | End date of trial | - -Example request: - -```shell -curl --request PUT --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription?max_seats_used=0" -``` - -Example response: - -```json -{ - "plan": { - "code":"premium", - "name":"premium", - "trial":false, - "auto_renew":null, - "upgradable":false - }, - "usage": { - "seats_in_subscription":80, - "seats_in_use":82, - "max_seats_used":0, - "seats_owed":2 - }, - "billing": { - "subscription_start_date":"2020-07-15", - "subscription_end_date":"2021-07-15", - "trial_ends_on":null - } -} -``` - -### Retrieving a subscription - -Use a GET command to view an existing subscription. - -```plaintext -GET /namespaces/:id/gitlab_subscription -``` - -Example request: - -```shell -curl --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription" -``` - -Example response: - -```json -{ - "plan": { - "code":"premium", - "name":"premium", - "trial":false, - "auto_renew":null, - "upgradable":false - }, - "usage": { - "seats_in_subscription":80, - "seats_in_use":82, - "max_seats_used":82, - "seats_owed":2 - }, - "billing": { - "subscription_start_date":"2020-07-15", - "subscription_end_date":"2021-07-15", - "trial_ends_on":null - } -} -``` - -### Known consumers - -- CustomersDot - -## CI minute provisioning - -The CI Minute endpoints are used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) -to apply additional packs of CI minutes, for personal namespaces or top-level groups within GitLab.com. - -### Creating an additional pack - -Use a POST to create additional packs. - -```plaintext -POST /namespaces/:id/minutes -``` - -| Attribute | Type | Required | Description | -|:------------|:--------|:---------|:------------| -| `packs` | array | yes | An array of purchased minutes packs | -| `packs[expires_at]` | date | yes | Expiry date of the purchased pack| -| `packs[number_of_minutes]` | integer | yes | Number of additional minutes | -| `packs[purchase_xid]` | string | yes | The unique ID of the purchase | - -Example request: - -```shell -curl --request POST \ - --url "http://localhost:3000/api/v4/namespaces/123/minutes" \ - --header 'Content-Type: application/json' \ - --header 'PRIVATE-TOKEN: <admin access token>' \ - --data '{ - "packs": [ - { - "number_of_minutes": 10000, - "expires_at": "2022-01-01", - "purchase_xid": "46952fe69bebc1a4de10b2b4ff439d0c" - } - ] - }' -``` - -Example response: - -```json -[ - { - "namespace_id": 123, - "expires_at": "2022-01-01", - "number_of_minutes": 10000, - "purchase_xid": "46952fe69bebc1a4de10b2b4ff439d0c" - } -] -``` - -### Moving additional packs - -Use a PATCH to move additional packs from one namespace to another. - -```plaintext -PATCH /namespaces/:id/minutes/move/:target_id -``` - -| Attribute | Type | Required | Description | -|:------------|:--------|:---------|:------------| -| `id` | string | yes | The ID of the namespace to transfer packs from | -| `target_id` | string | yes | The ID of the target namespace to transfer the packs to | - -Example request: - -```shell -curl --request PATCH \ - --url "http://localhost:3000/api/v4/namespaces/123/minutes/move/321" \ - --header 'PRIVATE-TOKEN: <admin access token>' -``` - -Example response: - -```json -{ - "message": "202 Accepted" -} -``` - -### Known consumers - -- CustomersDot - -## Upcoming reconciliations - -The `upcoming_reconciliations` endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) -to update upcoming reconciliations for namespaces. - -### Updating `upcoming_reconciliations` - -Use a PUT command to update `upcoming_reconciliations`. - -```plaintext -PUT /internal/upcoming_reconciliations -``` - -| Attribute | Type | Required | Description | -|:-------------------|:-----------|:---------|:------------| -| `upcoming_reconciliations` | array | yes | Array of upcoming reconciliations | - -Each array element contains: - -| Attribute | Type | Required | Description | -|:-------------------|:-----------|:---------|:------------| -| `namespace_id` | integer | yes | ID of the namespace to be reconciled | -| `next_reconciliation_date` | date | yes | Date when next reconciliation will happen | -| `display_alert_from` | date | yes | Start date to display alert of upcoming reconciliation | - -Example request: - -```shell -curl --request PUT --header "PRIVATE-TOKEN: <admin_access_token>" --header "Content-Type: application/json" \ - --data '{"upcoming_reconciliations": [{"namespace_id": 127, "next_reconciliation_date": "13 Jun 2021", "display_alert_from": "06 Jun 2021"}, {"namespace_id": 129, "next_reconciliation_date": "12 Jun 2021", "display_alert_from": "05 Jun 2021"}]}' \ - "https://gitlab.com/api/v4/internal/upcoming_reconciliations" -``` - -Example response: - -```plaintext -200 -``` - -### Known consumers - -- CustomersDot +<!-- This redirect file can be deleted after <2022-02-09>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md new file mode 100644 index 00000000000..0fe9a5362cf --- /dev/null +++ b/doc/development/internal_api/index.md @@ -0,0 +1,831 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: reference, api +--- + +# Internal API **(FREE)** + +The internal API is used by different GitLab components, it can not be +used by other consumers. This documentation is intended for people +working on the GitLab codebase. + +This documentation does not yet include the internal API used by +GitLab Pages. + +## Adding new endpoints + +API endpoints should be externally accessible by default, with proper authentication and authorization. +Before adding a new internal endpoint, consider if the API would potentially be +useful to the wider GitLab community and can be made externally accessible. + +One reason we might favor internal API endpoints sometimes is when using such an endpoint requires +internal data that external actors can not have. For example, in the internal Pages API we might use +a secret token that identifies a request as internal or sign a request with a public key that is +not available to a wider community. + +Another reason to separate something into an internal API is when request to such API endpoint +should never go through an edge (public) load balancer. This way we can configure different rate +limiting rules and policies around how the endpoint is being accessed, because we know that only +internal requests can be made to that endpoint going through an internal load balancer. + +## Authentication + +These methods are all authenticated using a shared secret. This secret +is stored in a file at the path configured in `config/gitlab.yml` by +default this is in the root of the rails app named +`.gitlab_shell_secret` + +To authenticate using that token, clients read the contents of that +file, and include the token Base64 encoded in a `secret_token` parameter +or in the `Gitlab-Shared-Secret` header. + +NOTE: +The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (`kas`) uses JSON Web Token (JWT) +authentication, which is different from GitLab Shell. + +## Git Authentication + +This is called by [Gitaly](https://gitlab.com/gitlab-org/gitaly) and +[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) to check access to a +repository. + +- **When called from GitLab Shell**: No changes are passed, and the internal + API replies with the information needed to pass the request on to Gitaly. +- **When called from Gitaly in a `pre-receive` hook**: The changes are passed + and validated to determine if the push is allowed. + +Calls are limited to 50 seconds each. + +This endpoint is covered in more detail on [its own page](internal_api_allowed.md), due to the scope of what it covers. + +```plaintext +POST /internal/allowed +``` + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key_id` | string | no | ID of the SSH-key used to connect to GitLab Shell | +| `username` | string | no | Username from the certificate used to connect to GitLab Shell | +| `project` | string | no (if `gl_repository` is passed) | Path to the project | +| `gl_repository` | string | no (if `project` is passed) | Repository identifier, such as `project-7` | +| `protocol` | string | yes | SSH when called from GitLab Shell, HTTP or SSH when called from Gitaly | +| `action` | string | yes | Git command being run (`git-upload-pack`, `git-receive-pack`, `git-upload-archive`) | +| `changes` | string | yes | `<oldrev> <newrev> <refname>` when called from Gitaly, the magic string `_any` when called from GitLab Shell | +| `check_ip` | string | no | IP address from which call to GitLab Shell was made | + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ + --data "key_id=11&project=gnuwget/wget2&action=git-upload-pack&protocol=ssh" \ + "http://localhost:3001/api/v4/internal/allowed" +``` + +Example response: + +```json +{ + "status": true, + "gl_repository": "project-3", + "gl_project_path": "gnuwget/wget2", + "gl_id": "user-1", + "gl_username": "root", + "git_config_options": [], + "gitaly": { + "repository": { + "storage_name": "default", + "relative_path": "@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git", + "git_object_directory": "", + "git_alternate_object_directories": [], + "gl_repository": "project-3", + "gl_project_path": "gnuwget/wget2" + }, + "address": "unix:/Users/bvl/repos/gitlab/gitaly.socket", + "token": null + }, + "gl_console_messages": [] +} +``` + +### Known consumers + +- Gitaly +- GitLab Shell + +## LFS Authentication + +This is the endpoint that gets called from GitLab Shell to provide +information for LFS clients when the repository is accessed over SSH. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key_id` | string | no | ID of the SSH-key used to connect to GitLab Shell | +| `username`| string | no | Username from the certificate used to connect to GitLab Shell | +| `project` | string | no | Path to the project | + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ + --data "key_id=11&project=gnuwget/wget2" "http://localhost:3001/api/v4/internal/lfs_authenticate" +``` + +```json +{ + "username": "root", + "lfs_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkYXRhIjp7ImFjdG9yIjoicm9vdCJ9LCJqdGkiOiIyYWJhZDcxZC0xNDFlLTQ2NGUtOTZlMi1mODllYWRiMGVmZTYiLCJpYXQiOjE1NzAxMTc2NzYsIm5iZiI6MTU3MDExNzY3MSwiZXhwIjoxNTcwMTE5NDc2fQ.g7atlBw1QMY7QEBVPE0LZ8ZlKtaRzaMRmNn41r2YITM", + "repository_http_path": "http://localhost:3001/gnuwget/wget2.git", + "expires_in": 1800 +} +``` + +### Known consumers + +- GitLab Shell + +## Authorized Keys Check + +This endpoint is called by the GitLab Shell authorized keys +check. Which is called by OpenSSH for [fast SSH key +lookup](../../administration/operations/fast_ssh_key_lookup.md). + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key` | string | yes | SSH key as passed by OpenSSH to GitLab Shell | + +```plaintext +GET /internal/authorized_keys +``` + +Example request: + +```shell +curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/authorized_keys?key=<key as passed by OpenSSH>" +``` + +Example response: + +```json +{ + "id": 11, + "title": "admin@example.com", + "key": "ssh-rsa ...", + "created_at": "2019-06-27T15:29:02.219Z" +} +``` + +### Known consumers + +- GitLab Shell + +## Get user for user ID or key + +This endpoint is used when a user performs `ssh git@gitlab.com`. It +discovers the user associated with an SSH key. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | +| `username` | string | no | Username of the user being looked up, used by GitLab Shell when authenticating using a certificate | + +```plaintext +GET /internal/discover +``` + +Example request: + +```shell +curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/discover?key_id=7" +``` + +Example response: + +```json +{ + "id": 7, + "name": "Dede Eichmann", + "username": "rubi" +} +``` + +### Known consumers + +- GitLab Shell + +## Instance information + +This gets some generic information about the instance. This is used +by Geo nodes to get information about each other. + +```plaintext +GET /internal/check +``` + +Example request: + +```shell +curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/check" +``` + +Example response: + +```json +{ + "api_version": "v4", + "gitlab_version": "12.3.0-pre", + "gitlab_rev": "d69c988e6a6", + "redis": true +} +``` + +### Known consumers + +- GitLab Geo +- GitLab Shell's `bin/check` +- Gitaly + +## Get new 2FA recovery codes using an SSH key + +This is called from GitLab Shell and allows users to get new 2FA +recovery codes based on their SSH key. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | +| `user_id` | integer | no | **Deprecated** User_id for which to generate new recovery codes | + +```plaintext +GET /internal/two_factor_recovery_codes +``` + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "key_id=7" "http://localhost:3001/api/v4/internal/two_factor_recovery_codes" +``` + +Example response: + +```json +{ + "success": true, + "recovery_codes": [ + "d93ee7037944afd5", + "19d7b84862de93dd", + "1e8c52169195bf71", + "be50444dddb7ca84", + "26048c77d161d5b7", + "482d5c03d1628c47", + "d2c695e309ce7679", + "dfb4748afc4f12a7", + "0e5f53d1399d7979", + "af04d5622153b020" + ] +} +``` + +### Known consumers + +- GitLab Shell + +## Get new personal access-token + +This is called from GitLab Shell and allows users to generate a new +personal access token. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `name` | string | yes | The name of the new token | +| `scopes` | string array | yes | The authorization scopes for the new token, these must be valid token scopes | +| `expires_at` | string | no | The expiry date for the new token | +| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | +| `user_id` | integer | no | User ID for which to generate the new token | + +```plaintext +POST /internal/personal_access_token +``` + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "user_id=29&name=mytokenname&scopes[]=read_user&scopes[]=read_repository&expires_at=2020-07-24" \ + "http://localhost:3001/api/v4/internal/personal_access_token" +``` + +Example response: + +```json +{ + "success": true, + "token": "Hf_79B288hRv_3-TSD1R", + "scopes": ["read_user","read_repository"], + "expires_at": "2020-07-24" +} +``` + +### Known consumers + +- GitLab Shell + +## Incrementing counter on pre-receive + +This is called from the Gitaly hooks increasing the reference counter +for a push that might be accepted. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `gl_repository` | string | yes | repository identifier for the repository receiving the push | + +```plaintext +POST /internal/pre_receive +``` + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "gl_repository=project-7" "http://localhost:3001/api/v4/internal/pre_receive" +``` + +Example response: + +```json +{ + "reference_counter_increased": true +} +``` + +## PostReceive + +Called from Gitaly after a receiving a push. This triggers the +`PostReceive`-worker in Sidekiq, processes the passed push options and +builds the response including messages that need to be displayed to +the user. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `identifier` | string | yes | `user-[id]` or `key-[id]` Identifying the user performing the push | +| `gl_repository` | string | yes | identifier of the repository being pushed to | +| `push_options` | string array | no | array of push options | +| `changes` | string | no | refs to be updated in the push in the format `oldrev newrev refname\n`. | + +```plaintext +POST /internal/post_receive +``` + +Example Request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "gl_repository=project-7" --data "identifier=user-1" \ + --data "changes=0000000000000000000000000000000000000000 fd9e76b9136bdd9fe217061b497745792fe5a5ee gh-pages\n" \ + "http://localhost:3001/api/v4/internal/post_receive" +``` + +Example response: + +```json +{ + "messages": [ + { + "message": "Hello from post-receive", + "type": "alert" + } + ], + "reference_counter_decreased": true +} +``` + +## Kubernetes agent endpoints + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41045) in GitLab 13.4. +> - This feature is not deployed on GitLab.com +> - It's not recommended for production use. + +The following endpoints are used by the GitLab Kubernetes Agent Server (`kas`) +for various purposes. + +These endpoints are all authenticated using JWT. The JWT secret is stored in a file +specified in `config/gitlab.yml`. By default, the location is in the root of the +GitLab Rails app in a file called `.gitlab_kas_secret`. + +WARNING: +The Kubernetes agent is under development and is not recommended for production use. + +### Kubernetes agent information + +Called from GitLab Kubernetes Agent Server (`kas`) to retrieve agent +information for the given agent token. This returns the Gitaly connection +information for the agent's project in order for `kas` to fetch and update +the agent's configuration. + +```plaintext +GET /internal/kubernetes/agent_info +``` + +Example Request: + +```shell +curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" +``` + +### Kubernetes agent project information + +Called from GitLab Kubernetes Agent Server (`kas`) to retrieve project +information for the given agent token. This returns the Gitaly +connection for the requested project. GitLab `kas` uses this to configure +the agent to fetch Kubernetes resources from the project repository to +sync. + +Only public projects are supported. For private projects, the ability for the +agent to be authorized is [not yet implemented](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../api/index.md#namespaced-path-encoding) | + +```plaintext +GET /internal/kubernetes/project_info +``` + +Example Request: + +```shell +curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" +``` + +### Kubernetes agent usage metrics + +Called from GitLab Kubernetes Agent Server (`kas`) to increase the usage +metric counters. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `gitops_sync_count` | integer| no | The number to increase the `gitops_sync_count` counter by | +| `k8s_api_proxy_request_count` | integer| no | The number to increase the `k8s_api_proxy_request_count` counter by | + +```plaintext +POST /internal/kubernetes/usage_metrics +``` + +Example Request: + +```shell +curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Content-Type: application/json" \ + --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" +``` + +### Kubernetes agent alert metrics + +Called from GitLab Kubernetes Agent Server (KAS) to save alerts derived from Cilium on Kubernetes +Cluster. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `alert` | Hash | yes | Alerts detail. Same format as [3rd party alert](../../operations/incident_management/integrations.md#customize-the-alert-payload-outside-of-gitlab). | + +```plaintext +POST internal/kubernetes/modules/cilium_alert +``` + +Example Request: + +```shell +curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ + --data '"{\"alert\":{\"title\":\"minimal\",\"message\":\"network problem\",\"evalMatches\":[{\"value\":1,\"metric\":\"Count\",\"tags\":{}}]}}"' \ + "http://localhost:3000/api/v4/internal/kubernetes/modules/cilium_alert" +``` + +### Create Starboard vulnerability + +Called from the GitLab Kubernetes Agent Server (`kas`) to create a security vulnerability +from a Starboard vulnerability report. This request is idempotent. Multiple requests with the same data +create a single vulnerability. + +| Attribute | Type | Required | Description | +|:----------------|:-------|:---------|:------------| +| `vulnerability` | Hash | yes | Vulnerability data matching the security report schema [`vulnerability` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | +| `scanner` | Hash | yes | Scanner data matching the security report schema [`scanner` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | + +```plaintext +PUT internal/kubernetes/modules/starboard_vulnerability +``` + +Example Request: + +```shell +curl --request PUT --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ + --url "http://localhost:3000/api/v4/internal/kubernetes/modules/starboard_vulnerability" \ + --data '{ + "vulnerability": { + "name": "CVE-123-4567 in libc", + "severity": "high", + "confidence": "unknown", + "location": { + "kubernetes_resource": { + "namespace": "production", + "kind": "deployment", + "name": "nginx", + "container": "nginx" + } + }, + "identifiers": [ + { + "type": "cve", + "name": "CVE-123-4567", + "value": "CVE-123-4567" + } + ] + }, + "scanner": { + "id": "starboard_trivy", + "name": "Trivy (via Starboard Operator)", + "vendor": "GitLab" + } +}' +``` + +## Subscriptions + +The subscriptions endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) +in order to apply subscriptions including trials, and add-on purchases, for personal namespaces or top-level groups within GitLab.com. + +### Creating a subscription + +Use a POST to create a subscription. + +```plaintext +POST /namespaces/:id/gitlab_subscription +``` + +| Attribute | Type | Required | Description | +|:------------|:--------|:---------|:------------| +| `start_date` | date | yes | Start date of subscription | +| `end_date` | date | no | End date of subscription | +| `plan_code` | string | no | Subscription tier code | +| `seats` | integer | no | Number of seats in subscription | +| `max_seats_used` | integer | no | Highest number of active users in the last month | +| `auto_renew` | boolean | no | Whether subscription auto-renews on end date | +| `trial` | boolean | no | Whether subscription is a trial | +| `trial_starts_on` | date | no | Start date of trial | +| `trial_ends_on` | date | no | End date of trial | + +Example request: + +```shell +curl --request POST --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription?start_date="2020-07-15"&plan="premium"&seats=10" +``` + +Example response: + +```json +{ + "plan": { + "code":"premium", + "name":"premium", + "trial":false, + "auto_renew":null, + "upgradable":false + }, + "usage": { + "seats_in_subscription":10, + "seats_in_use":1, + "max_seats_used":0, + "seats_owed":0 + }, + "billing": { + "subscription_start_date":"2020-07-15", + "subscription_end_date":null, + "trial_ends_on":null + } +} +``` + +### Updating a subscription + +Use a PUT command to update an existing subscription. + +```plaintext +PUT /namespaces/:id/gitlab_subscription +``` + +| Attribute | Type | Required | Description | +|:------------|:--------|:---------|:------------| +| `start_date` | date | no | Start date of subscription | +| `end_date` | date | no | End date of subscription | +| `plan_code` | string | no | Subscription tier code | +| `seats` | integer | no | Number of seats in subscription | +| `max_seats_used` | integer | no | Highest number of active users in the last month | +| `auto_renew` | boolean | no | Whether subscription auto-renews on end date | +| `trial` | boolean | no | Whether subscription is a trial | +| `trial_starts_on` | date | no | Start date of trial. Required if trial is true. | +| `trial_ends_on` | date | no | End date of trial | + +Example request: + +```shell +curl --request PUT --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription?max_seats_used=0" +``` + +Example response: + +```json +{ + "plan": { + "code":"premium", + "name":"premium", + "trial":false, + "auto_renew":null, + "upgradable":false + }, + "usage": { + "seats_in_subscription":80, + "seats_in_use":82, + "max_seats_used":0, + "seats_owed":2 + }, + "billing": { + "subscription_start_date":"2020-07-15", + "subscription_end_date":"2021-07-15", + "trial_ends_on":null + } +} +``` + +### Retrieving a subscription + +Use a GET command to view an existing subscription. + +```plaintext +GET /namespaces/:id/gitlab_subscription +``` + +Example request: + +```shell +curl --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription" +``` + +Example response: + +```json +{ + "plan": { + "code":"premium", + "name":"premium", + "trial":false, + "auto_renew":null, + "upgradable":false + }, + "usage": { + "seats_in_subscription":80, + "seats_in_use":82, + "max_seats_used":82, + "seats_owed":2 + }, + "billing": { + "subscription_start_date":"2020-07-15", + "subscription_end_date":"2021-07-15", + "trial_ends_on":null + } +} +``` + +### Known consumers + +- CustomersDot + +## CI minute provisioning + +The CI Minute endpoints are used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) +to apply additional packs of CI minutes, for personal namespaces or top-level groups within GitLab.com. + +### Creating an additional pack + +Use a POST to create additional packs. + +```plaintext +POST /namespaces/:id/minutes +``` + +| Attribute | Type | Required | Description | +|:------------|:--------|:---------|:------------| +| `packs` | array | yes | An array of purchased minutes packs | +| `packs[expires_at]` | date | yes | Expiry date of the purchased pack| +| `packs[number_of_minutes]` | integer | yes | Number of additional minutes | +| `packs[purchase_xid]` | string | yes | The unique ID of the purchase | + +Example request: + +```shell +curl --request POST \ + --url "http://localhost:3000/api/v4/namespaces/123/minutes" \ + --header 'Content-Type: application/json' \ + --header 'PRIVATE-TOKEN: <admin access token>' \ + --data '{ + "packs": [ + { + "number_of_minutes": 10000, + "expires_at": "2022-01-01", + "purchase_xid": "46952fe69bebc1a4de10b2b4ff439d0c" + } + ] + }' +``` + +Example response: + +```json +[ + { + "namespace_id": 123, + "expires_at": "2022-01-01", + "number_of_minutes": 10000, + "purchase_xid": "46952fe69bebc1a4de10b2b4ff439d0c" + } +] +``` + +### Moving additional packs + +Use a PATCH to move additional packs from one namespace to another. + +```plaintext +PATCH /namespaces/:id/minutes/move/:target_id +``` + +| Attribute | Type | Required | Description | +|:------------|:--------|:---------|:------------| +| `id` | string | yes | The ID of the namespace to transfer packs from | +| `target_id` | string | yes | The ID of the target namespace to transfer the packs to | + +Example request: + +```shell +curl --request PATCH \ + --url "http://localhost:3000/api/v4/namespaces/123/minutes/move/321" \ + --header 'PRIVATE-TOKEN: <admin access token>' +``` + +Example response: + +```json +{ + "message": "202 Accepted" +} +``` + +### Known consumers + +- CustomersDot + +## Upcoming reconciliations + +The `upcoming_reconciliations` endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) +to update upcoming reconciliations for namespaces. + +### Updating `upcoming_reconciliations` + +Use a PUT command to update `upcoming_reconciliations`. + +```plaintext +PUT /internal/upcoming_reconciliations +``` + +| Attribute | Type | Required | Description | +|:-------------------|:-----------|:---------|:------------| +| `upcoming_reconciliations` | array | yes | Array of upcoming reconciliations | + +Each array element contains: + +| Attribute | Type | Required | Description | +|:-------------------|:-----------|:---------|:------------| +| `namespace_id` | integer | yes | ID of the namespace to be reconciled | +| `next_reconciliation_date` | date | yes | Date when next reconciliation will happen | +| `display_alert_from` | date | yes | Start date to display alert of upcoming reconciliation | + +Example request: + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <admin_access_token>" --header "Content-Type: application/json" \ + --data '{"upcoming_reconciliations": [{"namespace_id": 127, "next_reconciliation_date": "13 Jun 2021", "display_alert_from": "06 Jun 2021"}, {"namespace_id": 129, "next_reconciliation_date": "12 Jun 2021", "display_alert_from": "05 Jun 2021"}]}' \ + "https://gitlab.com/api/v4/internal/upcoming_reconciliations" +``` + +Example response: + +```plaintext +200 +``` + +### Known consumers + +- CustomersDot diff --git a/doc/development/internal_api/internal_api_allowed.md b/doc/development/internal_api/internal_api_allowed.md new file mode 100644 index 00000000000..83dbb54babd --- /dev/null +++ b/doc/development/internal_api/internal_api_allowed.md @@ -0,0 +1,109 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: reference, api +--- + +# Internal allowed API + +The `internal/allowed` endpoint assesses whether a user has permission to perform +certain operations on the Git repository. It performs multiple checks, such as: + +- Ensuring the branch or tag name is acceptable. +- Whether or not the user has the authority to perform that action. + +## Endpoint definition + +The internal API endpoints are defined under +[`lib/api/internal`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/api/internal), +and the `/allowed` path is in +[`lib/api/internal/base.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/base.rb). + +## Use the endpoint + +`internal/allowed` is called when you: + +- Push to the repository. +- Perform actions on the repository through the GitLab user interface, such as + applying suggestions or using the GitLab IDE. + +Gitaly typically calls this endpoint. It is only called internally (by other parts +of the application) rather than by external users of the API. + +## Push checks + +A key part of the `internal/allowed` flow is the call to +`EE::Gitlab::Checks::PushRuleCheck`, which can perform the following checks: + +- `EE::Gitlab::Checks::PushRules::CommitCheck` +- `EE::Gitlab::Checks::PushRules::TagCheck` +- `EE::Gitlab::Checks::PushRules::BranchCheck` +- `EE::Gitlab::Checks::PushRules::FileSizeCheck` + +## Recursion + +Some of the Gitaly RPCs called by `internal/allowed` then, themselves, make calls +back to `internal/allowed`. These calls are now correlated with the original request. +Gitaly relies on the Rails application for authorization, and maintains no permissions model itself. + +These calls show up in the logs differently to the initial requests. {example} + +Because this endpoint can be called recursively, slow performance on this endpoint can result in an exponential performance impact. This documentation is in fact adapted from [the investigation into its performance](https://gitlab.com/gitlab-org/gitlab/-/issues/222247). + +## Known performance issues + +### Refs + +The number of [`refs`](https://git-scm.com/book/en/v2/Git-Internals-Git-References) +on the Git repository have a notable effect on the performance of `git` commands +called upon it. Gitaly RPCs are similarly affected. Certain `git` commands scan +through all refs, causing a notable impact on the speed of those commands. + +On the `internal/allowed` endpoint, the recursive nature of RPC calls mean the +ref counts have an exponential effect on performance. + +#### Environment refs + +[Stale environment refs](https://gitlab.com/gitlab-org/gitlab/-/issues/296625) +are a common example of excessive refs causing performance issues. Stale environment +refs can number into the tens of thousands on busy repositories, as they aren't +cleared up automatically. + +#### Dangling refs + +Dangling refs are created to prevent accidental deletion of objects from object pools. +Large numbers of these refs can exist, which may have potential performance implications. +For existing discussion around this issue, read +[`gitaly#1900`](https://gitlab.com/gitlab-org/gitaly/-/issues/1900). This issue +appears to have less effect than stale environment refs. + +### Pool repositories + +When a fork is created on GitLab, a central pool repository is created and the forks +are linked to it. This pool repository prevents duplication of data by storing +data common to other forks. However, the pool repository is not cleaned up in the +same manner as the standard repositories, and is more prone to the refs issue. + +## Feature flags + +### Parallel push checks + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `parallel_push_checks`. +On GitLab.com, by default this feature is not available. To make it available +per project, ask GitLab.com administrator to +[enable the feature flag](../../administration/feature_flags.md) named `parallel_push_checks`. +You should not use this feature for production environments. + +This experimental feature flag enables the endpoint to run multiple RPCs simultaneously, +reducing the overall time taken by roughly half. This time savings is achieved through +threading, and has potential side effects at large scale. On GitLab.com, this feature flag +is enabled only for `gitlab-org/gitlab` and `gitlab-com/www-gitlab-com` projects. +Without it, those projects routinely time out requests to the endpoint. When this +feature was deployed to all of GitLab.com, some pushes failed, presumably due to +exhausting resources like database connection pools. + +We recommend you enable this feature flag only if you are experiencing timeouts, and +only enable it for that specific project. diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md index fba6f2f616d..9ea7fd3dc29 100644 --- a/doc/development/iterating_tables_in_batches.md +++ b/doc/development/iterating_tables_in_batches.md @@ -142,7 +142,7 @@ database query: SELECT "users"."id" FROM "users" ORDER BY "users"."id" ASC LIMIT 1 ``` - + Notice that the query only reads data from the index (`INDEX ONLY SCAN`), the table is not accessed. Database indexes are sorted so taking out the first item is a very cheap operation. @@ -155,7 +155,7 @@ to get a "shifted" `id` value. SELECT "users"."id" FROM "users" WHERE "users"."id" >= 1 ORDER BY "users"."id" ASC LIMIT 1 OFFSET 5 ``` - + Again, the query only looks into the index. The `OFFSET 5` takes out the sixth `id` value: this query reads a maximum of six items from the index regardless of the table size or the iteration @@ -181,7 +181,7 @@ previous iteration in order to find out the next end `id` value. SELECT "users"."id" FROM "users" WHERE "users"."id" >= 302 ORDER BY "users"."id" ASC LIMIT 1 OFFSET 5 ``` - + Now we can easily construct the `users` query for the second iteration. diff --git a/doc/development/jh_features_review.md b/doc/development/jh_features_review.md index cb0f8ddbd13..858048c1e7c 100644 --- a/doc/development/jh_features_review.md +++ b/doc/development/jh_features_review.md @@ -23,6 +23,14 @@ We have two kinds of changes related to JH: If needed, review the corresponding JH merge request located at [JH repository](https://gitlab.com/gitlab-jh/gitlab) +## When to merge files to the GitLab Inc. repository + +Files that are added to the `gitlab-jh` repository outside of `jh/` must be mirrored in the GitLab Inc. repository. + +If code that is added to the GitLab Inc. repository references (for example, `render_if_exists`) any `gitlab-jh` file that does not +exist in the GitLab Inc. codebase, add a comment with a link to the JiHu merge request or file. This is to prevent +the reference from being misidentified as a missing partial and subsequently deleted in the `gitlab` codebase. + ## Process overview See the [merge request process](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/#merge-request-process) diff --git a/doc/development/maintenance_mode.md b/doc/development/maintenance_mode.md index e308ab26c27..c2fd4bab605 100644 --- a/doc/development/maintenance_mode.md +++ b/doc/development/maintenance_mode.md @@ -13,7 +13,7 @@ GitLab Maintenance Mode **only** blocks writes from HTTP and SSH requests at the - [the read-only database method](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/lib/ee/gitlab/database.rb#L13), which toggles special behavior when we are not allowed to write to the database. [Search the codebase for `Gitlab::Database.read_only?`.](https://gitlab.com/search?search=Gitlab%3A%3ADatabase.read_only%3F&group_id=9970&project_id=278964&scope=blobs&search_code=false&snippets=false&repository_ref=) - [the read-only middleware](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/middleware/read_only/controller.rb), where HTTP requests that cause database writes are blocked, unless explicitly allowed. -- [Git push access via SSH is denied](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/lib/ee/gitlab/git_access.rb#L13) by returning 401 when `gitlab-shell` POSTs to [`/internal/allowed`](internal_api.md) to [check if access is allowed](internal_api.md#git-authentication). +- [Git push access via SSH is denied](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/lib/ee/gitlab/git_access.rb#L13) by returning 401 when `gitlab-shell` POSTs to [`/internal/allowed`](internal_api/index.md) to [check if access is allowed](internal_api/index.md#git-authentication). - [Container registry authentication service](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/app/services/ee/auth/container_registry_authentication_service.rb#L12), where updates to the container registry are blocked. The database itself is not in read-only mode (except in a Geo secondary site) and can be written by sources other than the ones blocked. diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index e03b96a0e14..0bd9979e790 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -135,6 +135,7 @@ various database operations, such as: - [dropping and renaming columns](avoiding_downtime_in_migrations.md#dropping-columns) - [changing column constraints and types](avoiding_downtime_in_migrations.md#changing-column-constraints) - [adding and dropping indexes, tables, and foreign keys](avoiding_downtime_in_migrations.md#adding-indexes) +- [migrating `integer` primary keys to `bigint`](avoiding_downtime_in_migrations.md#migrating-integer-primary-keys-to-bigint) and explains how to perform them without requiring downtime. @@ -312,12 +313,8 @@ A better strategy is to split the migration, so that we only need to acquire one ```ruby enable_lock_retries! -def up - remove_column :users, :full_name -end - -def down - add_column :users, :full_name, :string +def change + remove_column :users, :full_name, :string end ``` @@ -587,8 +584,6 @@ class like so: ```ruby class MyMigration < Gitlab::Database::Migration[1.0] - include Gitlab::Database::MigrationHelpers - disable_ddl_transaction! INDEX_NAME = 'index_name' @@ -632,8 +627,6 @@ be used with a name option. For example: ```ruby class MyMigration < Gitlab::Database::Migration[1.0] - include Gitlab::Database::MigrationHelpers - INDEX_NAME = 'index_name' def up diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index f834f4f4ee3..27a7cd6e85e 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -49,7 +49,15 @@ Is it ok that some nodes have the new Rails version, but some nodes have the old ## A walkthrough of an update -Backwards compatibility problems during updates are often very subtle. This is why it is worth familiarizing yourself with [update instructions](../update/index.md), [reference architectures](../administration/reference_architectures/index.md), and [GitLab.com's architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/). But to illustrate how these problems arise, take a look at this example of a simple update. +Backward compatibility problems during updates are often very subtle. This is why it is worth +familiarizing yourself with: + +- [Update instructions](../update/index.md) +- [Reference architectures](../administration/reference_architectures/index.md) +- [GitLab.com's architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/) +- [GitLab.com's upgrade pipeline](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/gitlab-com-deployer.md#upgrade-pipeline-default) + +To illustrate how these problems arise, take a look at this example: - 🚢 New version - 🙂 Old version diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md index c6af0ddf9ea..e5263288210 100644 --- a/doc/development/namespaces_storage_statistics.md +++ b/doc/development/namespaces_storage_statistics.md @@ -53,7 +53,10 @@ SELECT split_part("rs".path, '/', 1) as root_path, COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, - COALESCE(SUM(ps.packages_size), 0) AS packages_size + COALESCE(SUM(ps.pipeline_artifacts_size), 0) AS pipeline_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size, + COALESCE(SUM(ps.snippets_size), 0) AS snippets_size, + COALESCE(SUM(ps.uploads_size), 0) AS uploads_size FROM "projects" INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' INNER JOIN project_statistics ps ON ps.project_id = projects.id @@ -83,7 +86,10 @@ WITH refresh AS ( COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, - COALESCE(SUM(ps.packages_size), 0) AS packages_size + COALESCE(SUM(ps.pipeline_artifacts_size), 0) AS pipeline_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size, + COALESCE(SUM(ps.snippets_size), 0) AS snippets_size, + COALESCE(SUM(ps.uploads_size), 0) AS uploads_size FROM "projects" INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' INNER JOIN project_statistics ps ON ps.project_id = projects.id @@ -94,7 +100,10 @@ SET storage_size = refresh.storage_size, wiki_size = refresh.wiki_size, lfs_objects_size = refresh.lfs_objects_size, build_artifacts_size = refresh.build_artifacts_size, - packages_size = refresh.packages_size + pipeline_artifacts_size = refresh.pipeline_artifacts_size, + packages_size = refresh.packages_size, + snippets_size = refresh.snippets_size, + uploads_size = refresh.uploads_size FROM refresh INNER JOIN routes rs ON rs.path = refresh.root_path AND rs.source_type = 'Namespace' WHERE namespace_storage_statistics.namespace_id = rs.source_id diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 45982d6075b..71a11d2024c 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -23,6 +23,30 @@ as much as possible. After a merge request has been approved, the pipeline would contain the full RSpec & Jest tests. This will ensure that all tests have been run before a merge request is merged. +### Overview of the GitLab project test dependency + +To understand how the minimal test jobs are executed, we need to understand the dependency between +GitLab code (frontend and backend) and the respective tests (Jest and RSpec). +This dependency can be visualized in the following diagram: + +```mermaid +flowchart LR + subgraph frontend + fe["Frontend code"]--tested with-->jest + end + subgraph backend + be["Backend code"]--tested with-->rspec + end + + be--generates-->fixtures["frontend fixtures"] + fixtures--used in-->jest +``` + +In summary: + +- RSpec tests are dependent on the backend code. +- Jest tests are dependent on both frontend and backend code, the latter through the frontend fixtures. + ### RSpec minimal jobs #### Determining related RSpec test files in a merge request @@ -57,7 +81,7 @@ In this mode, `jest` would resolve all the dependencies of related to the change In addition, there are a few circumstances where we would always run the full Jest tests: -- when the `pipeline:run-all-rspec` label is set on the merge request +- when the `pipeline:run-all-jest` label is set on the merge request - when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) - when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) - when any frontend "core" file is changed (i.e. `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) @@ -142,6 +166,13 @@ Our current RSpec tests parallelization setup is as follows: After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file. +### Flaky tests + +Tests that are [known to be flaky](testing_guide/flaky_tests.md#automatic-retries-and-flaky-tests-detection) are: + +- skipped if the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is set to `true` (`false` by default) +- run if `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is not set to `true` or if the `~"pipeline:run-flaky-tests"` label is set on the MR + ### Monitoring The GitLab test suite is [monitored](performance.md#rspec-profiling) for the `main` branch, and any branch @@ -157,6 +188,8 @@ that includes `rspec-profile` in their name. Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. +If you want to force a Review App to be deployed regardless of your changes, you can add the `pipeline:run-review-app` label to the merge request. + ## As-if-FOSS jobs The `* as-if-foss` jobs run the GitLab test suite "as if FOSS", meaning as if the jobs would run in the context @@ -590,7 +623,8 @@ The current stages are: - `build-images`: This stage includes jobs that prepare Docker images that are needed by jobs in subsequent stages or downstream pipelines. - `fixtures`: This stage includes jobs that prepare fixtures needed by frontend tests. -- `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs. +- `lint`: This stage includes linting and static analysis jobs. +- `test`: This stage includes most of the tests, and DB/migration jobs. - `post-test`: This stage includes jobs that build reports or gather data from the `test` stage's jobs (for example, coverage, Knapsack metadata, and so on). - `review-prepare`: This stage includes a job that build the CNG images that are @@ -664,7 +698,7 @@ then included in individual jobs via [`extends`](../ci/yaml/index.md#extends). The `rules` definitions are composed of `if:` conditions and `changes:` patterns, which are also defined in [`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) -and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#anchors) +and included in `rules` definitions via [YAML anchors](../ci/yaml/yaml_optimization.md#anchors) #### `if:` conditions @@ -696,7 +730,6 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#ancho | `if-dot-com-gitlab-org-and-security-merge-request` | Limit jobs creation to merge requests for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-gitlab-org-and-security-tag` | Limit jobs creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | -| `if-cache-credentials-schedule` | Limits jobs to scheduled pipelines with the `$CI_REPO_CACHE_CREDENTIALS` variable set. | | | `if-security-pipeline-merge-result` | Matches if the pipeline is for a security merge request triggered by `@gitlab-release-tools-bot`. | | <!-- vale gitlab.Substitutions = YES --> @@ -741,8 +774,10 @@ request, be sure to start the `dont-interrupt-me` job before pushing. [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml), with fixed keys: - `.setup-test-env-cache` + - `.ruby-cache` - `.rails-cache` - `.static-analysis-cache` + - `.rubocop-cache` - `.coverage-cache` - `.danger-review-cache` - `.qa-cache` @@ -752,7 +787,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing. 1. Only the following jobs, running in 2-hourly scheduled pipelines, are pushing (that is, updating) to the caches: - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-rubocop-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). @@ -764,71 +799,28 @@ request, be sure to start the `dont-interrupt-me` job before pushing. We limit the artifacts that are saved and retrieved by jobs to the minimum in order to reduce the upload/download time and costs, as well as the artifacts storage. -### Pre-clone step - -The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/-/issues/39134) -to seed the project with a recent archive of the repository. This is done for -several reasons: - -- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone. -- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. - -The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable -[defined by GitLab.com shared runners](../ci/runners/build_cloud/linux_build_cloud.md#pre-clone-script). - -The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: - -```shell -( - echo "Downloading archived master..." - wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz - - if [ ! -f /tmp/gitlab.tar.gz ]; then - echo "Repository cache not available, cloning a new directory..." - exit - fi - - rm -rf $CI_PROJECT_DIR - echo "Extracting tarball into $CI_PROJECT_DIR..." - mkdir -p $CI_PROJECT_DIR - cd $CI_PROJECT_DIR - tar xzf /tmp/gitlab.tar.gz - rm -f /tmp/gitlab.tar.gz - chmod a+w $CI_PROJECT_DIR -) -``` - -The first step of the script downloads `gitlab-master.tar.gz` from -Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) -that is responsible for keeping that archive up-to-date. Every two hours -on a scheduled pipeline, it does the following: +### Git fetch caching -1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. -1. Saves the data as a `.tar.gz`. -1. Uploads it into the Google Cloud Storage bucket. +Because GitLab.com uses the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache), +concurrent Git fetches of the same pipeline ref are deduplicated on +the Gitaly server (always) and served from cache (when available). -When a CI job runs with this configuration, the output looks something like this: +This works well for the following reasons: -```shell -$ eval "$CI_PRE_CLONE_SCRIPT" -Downloading archived master... -Extracting tarball into /builds/group/project... -Fetching changes... -Reinitialized existing Git repository in /builds/group/project/.git/ -``` +- The pack-objects cache is enabled on all Gitaly servers on GitLab.com. +- The CI/CD [Git strategy setting](../ci/pipelines/settings.md#choose-the-default-git-strategy) for `gitlab-org/gitlab` is **Git clone**, + causing all jobs to fetch the same data, which maximizes the cache hit ratio. +- We use [shallow clone](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git + history for every job. -Note that the `Reinitialized existing Git repository` message shows that -the pre-clone step worked. The runner runs `git init`, which -overwrites the Git configuration with the appropriate settings to fetch -from the GitLab repository. +### Pre-clone step -`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account -JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you're a -GitLab Team Member, find credentials in the -[GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). +NOTE: +We no longer use this optimization for `gitlab-org/gitlab` because the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache) +allows Gitaly to serve the full CI/CD fetch traffic now. See [Git fetch caching](#git-fetch-caching). -Note that this bucket should be located in the same continent as the -runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). +The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable +[defined by GitLab.com shared runners](../ci/runners/runner_cloud/linux_runner_cloud.md#pre-clone-script). --- diff --git a/doc/development/policies.md b/doc/development/policies.md index 8ad3d3f42ba..e5191f00d9a 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -111,7 +111,7 @@ Each line represents a rule that was evaluated. There are a few things to note: Here you can see that the first four rules were evaluated `false` for which user and subject. For example, you can see in the last line that -the rule was activated because the user `john` had Reporter access to +the rule was activated because the user `john` had the Reporter [role](../user/permissions.md) on `Project/4`. When a policy is asked whether a particular ability is allowed diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index 3c0a1419604..5aaf4c72e64 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -74,7 +74,7 @@ For more information, read the internal issue ### In models and integrations -The ReactiveCaching concern can be used in models as well as `integrations` +The ReactiveCaching concern can be used in models as well as integrations (`app/models/integrations`). 1. Include the concern in your model or integration. @@ -88,7 +88,7 @@ The ReactiveCaching concern can be used in models as well as `integrations` To include the concern in an integration: ```ruby - include ReactiveService + include Integrations::ReactivelyCached ``` 1. Implement the `calculate_reactive_cache` method in your model or integration. @@ -201,15 +201,15 @@ There are some `class_attribute` options which can be tweaked. and `"ExampleModel:1:arg1:arg2:alive"` respectively, where `ExampleModel` is the name of the model, `1` is the ID of the record, `arg1` and `arg2` are parameters passed to `with_reactive_cache`. -- If you're including this concern in a service instead, you must override - the default by adding the following to your service: +- If you're including this concern in an integration (`app/models/integrations/`) instead, you must override + the default by adding the following to your integration: ```ruby - self.reactive_cache_key = ->(service) { [service.class.model_name.singular, service.project_id] } + self.reactive_cache_key = ->(integration) { [integration.class.model_name.singular, integration.project_id] } ``` If your reactive_cache_key is exactly like the above, you can use the existing - `ReactiveService` concern instead. + `Integrations::ReactivelyCached` concern instead. #### `self.reactive_cache_lease_timeout` diff --git a/doc/development/redis.md b/doc/development/redis.md index fa07cebdc61..75170b8c746 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -21,8 +21,7 @@ GitLab uses [Redis](https://redis.io) for the following distinct purposes: In most environments (including the GDK), all of these point to the same Redis instance. -On GitLab.com, we use [separate Redis -instances](../administration/redis/replication_and_failover.md#running-multiple-redis-clusters). +On GitLab.com, we use [separate Redis instances](../administration/redis/replication_and_failover.md#running-multiple-redis-clusters). See the [Redis SRE guide](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/redis/redis-survival-guide-for-sres.md) for more details on our setup. @@ -121,8 +120,7 @@ on GitLab.com On GitLab.com, entries from the [Redis slow log](https://redis.io/commands/slowlog) are available in the -`pubsub-redis-inf-gprd*` index with the [`redis.slowlog` -tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time_s),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). +`pubsub-redis-inf-gprd*` index with the [`redis.slowlog` tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time_s),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). This shows commands that have taken a long time and may be a performance concern. diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 568e8a9d123..ccf82dc6c77 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -133,8 +133,20 @@ Everything in `lib/api`. Everything that resides in `app/services`. +Services should consider inheriting from: + +- `BaseContainerService` for services scoped by container (project or group) +- `BaseProjectService` for services scoped to projects +- `BaseGroupService` for services scoped to groups + +or, create a new base class and update the list above. + +Legacy classes inherited from `BaseService` for historical reasons. + In Service classes the use of `execute` and `#execute` is preferred over `call` and `#call`. +Classes that are not service objects should be [created elsewhere](directory_structure.md#use-namespaces-to-define-bounded-contexts), such as in `lib`. + #### ServiceResponse Service classes usually have an `execute` method, which can return a diff --git a/doc/development/ruby3_gotchas.md b/doc/development/ruby3_gotchas.md new file mode 100644 index 00000000000..e4ed5039e3c --- /dev/null +++ b/doc/development/ruby3_gotchas.md @@ -0,0 +1,140 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Ruby 3 gotchas + +This section documents several problems we found while working on [Ruby 3 support](https://gitlab.com/groups/gitlab-org/-/epics/5149) +and which led to subtle bugs or test failures that were difficult to understand. We encourage every GitLab contributor +who writes Ruby code on a regular basis to familiarize themselves with these issues. + +To find the complete list of changes to the Ruby 3 language and standard library, see +[Ruby Changes](https://rubyreferences.github.io/rubychanges/3.0.html). + +## `Hash#each` consistently yields a 2-element array to lambdas + +Consider the following code snippet: + +```ruby +def foo(a, b) + p [a, b] +end + +def bar(a, b = 2) + p [a, b] +end + +foo_lambda = method(:foo).to_proc +bar_lambda = method(:bar).to_proc + +{ a: 1 }.each(&foo_lambda) +{ a: 1 }.each(&bar_lambda) +``` + +In Ruby 2.7, the output of this program suggests that yielding hash entries to lambdas behaves +differently depending on how many required arguments there are: + +```ruby +# Ruby 2.7 +{ a: 1 }.each(&foo_lambda) # prints [:a, 1] +{ a: 1 }.each(&bar_lambda) # prints [[:a, 1], 2] +``` + +Ruby 3 makes this behavior consistent and always attempts to yield hash entries as a single `[key, value]` array: + +```ruby +# Ruby 3.0 +{ a: 1 }.each(&foo_lambda) # `foo': wrong number of arguments (given 1, expected 2) (ArgumentError) +{ a: 1 }.each(&bar_lambda) # prints [[:a, 1], 2] +``` + +To write code that works under both 2.7 and 3.0, consider the following options: + +- Always pass the lambda body as a block: `{ a: 1 }.each { |a, b| p [a, b] }`. +- Deconstruct the lambda arguments: `{ a: 1 }.each(&->((a, b)) { p [a, b] })`. + +We recommend always passing the block explicitly, and prefer two required arguments as block parameters. + +To learn more, see [Ruby issue 12706](https://bugs.ruby-lang.org/issues/12706). + +## `Symbol#to_proc` returns signature metadata consistent with lambdas + +A common idiom in Ruby is to obtain `Proc` objects using the `&:<symbol>` shorthand and +pass them to higher-order functions: + +```ruby +[1, 2, 3].each(&:to_s) +``` + +Ruby desugars `&:<symbol>` to `Symbol#to_proc`. We can call it with +the method _receiver_ as its first argument (here: `Integer`), and all method _arguments_ +(here: none) as its remaining arguments. + +This behaves the same in both Ruby 2.7 and Ruby 3. Where Ruby 3 diverges is when capturing +this `Proc` object and inspecting its call signature. +This is often done when writing DSLs or using other forms of meta-programming: + +```ruby +p = :foo.to_proc # This usually happens via a conversion through `&:foo` + +# Ruby 2.7: prints [[:rest]] (-1) +# Ruby 3.0: prints [[:req], [:rest]] (-2) +puts "#{p.parameters} (#{p.arity})" +``` + +Ruby 2.7 reports zero required and one optional parameter for this `Proc` object, while Ruby 3 reports one required +and one optional parameter. Ruby 2.7 is incorrect: the first argument must +always be passed, as it is the receiver of the method the `Proc` object represents, and methods cannot be +called without a receiver. + +Ruby 3 corrects this: the code that tests `Proc` object arity or parameter lists might now break and +has to be updated. + +To learn more, see [Ruby issue 16260](https://bugs.ruby-lang.org/issues/16260). + +## `OpenStruct` does not evaluate fields lazily + +The `OpenStruct` implementation has undergone a partial rewrite in Ruby 3, resulting in +behavioral changes. In Ruby 2.7, `OpenStruct` defines methods lazily, when the method is first accessed. +In Ruby 3.0, it defines these methods eagerly in the initializer, which can break classes that inherit from `OpenStruct` +and override these methods. + +Don't inherit from `OpenStruct` for these reasons; ideally, don't use it at all. +`OpenStruct` is [considered problematic](https://ruby-doc.org/stdlib-3.0.2/libdoc/ostruct/rdoc/OpenStruct.html#class-OpenStruct-label-Caveats). +When writing new code, prefer a `Struct` instead, which is simpler in implementation, although less flexible. + +## `Regexp` and `Range` instances are frozen + +It is not necessary anymore to explicitly freeze `Regexp` or `Range` instances because Ruby 3 freezes +them automatically upon creation. + +This has a subtle side-effect: Tests that stub method calls on these types now fail with an error because +RSpec cannot stub frozen objects: + +```ruby +# Ruby 2.7: works +# Ruby 3.0: error: "can't modify frozen object" +allow(subject.function_returning_range).to receive(:max).and_return(42) +``` + +Rewrite affected tests by not stubbing method calls on frozen objects. The example above can be rewritten as: + +```ruby +# Works with any Ruby version +allow(subject).to receive(:function_returning_range).and_return(1..42) +``` + +## Table tests fail with Ruby 3.0.2 + +Ruby 3.0.2 has a known bug that causes [table tests](testing_guide/best_practices.md#table-based--parameterized-tests) +to fail when table values consist of integer values. +The reasons are documented in [issue 337614](https://gitlab.com/gitlab-org/gitlab/-/issues/337614). +This problem has been fixed in Ruby and the fix is expected to be included in Ruby 3.0.3. + +The problem only affects users who run an unpatched Ruby 3.0.2. This is likely the case when you +installed Ruby manually or via tools like `asdf`. Users of the `gitlab-development-kit (GDK)` +are also affected by this problem. + +Build images are not affected because they include the patch set addressing this bug. diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index ad6bff8499a..f9816986b2d 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -137,7 +137,7 @@ This is typically necessary, since gems or Ruby applications that we maintain ou update these repositories for the GitLab Rails application to work with a new Ruby, it is good practice to keep Ruby versions in lock-step across all our repositories. For minor and major upgrades, add new CI/CD jobs to these repositories using the new Ruby. -A [build matrix definition](../ci/yaml/index.md#parallel-matrix-jobs) can do this efficiently. +A [build matrix definition](../ci/yaml/index.md#parallelmatrix) can do this efficiently. #### Decide which repositories to update diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 0f13b8ecae9..65fdb815f87 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -540,6 +540,94 @@ out, _ = exec.Command("sh", "-c", "echo 1 | cat /etc/passwd").Output() This outputs `1` followed by the content of `/etc/passwd`. +## General recommendations + +### TLS minimum recommended version + +As we have [moved away from supporting TLS 1.0 and 1.1](https://about.gitlab.com/blog/2018/10/15/gitlab-to-deprecate-older-tls/), we should only use TLS 1.2 and above. + +#### Ciphers + +We recommend using the ciphers that Mozilla is providing in their [recommended SSL configuration generator](https://ssl-config.mozilla.org/#server=go&version=1.17&config=intermediate&guideline=5.6) for TLS 1.2: + +- `ECDHE-ECDSA-AES128-GCM-SHA256` +- `ECDHE-RSA-AES128-GCM-SHA256` +- `ECDHE-ECDSA-AES256-GCM-SHA384` +- `ECDHE-RSA-AES256-GCM-SHA384` +- `ECDHE-ECDSA-CHACHA20-POLY1305` +- `ECDHE-RSA-CHACHA20-POLY1305` + +And the following cipher suites (according to the [RFC 8446](https://datatracker.ietf.org/doc/html/rfc8446#appendix-B.4)) for TLS 1.3: + +- `TLS_AES_128_GCM_SHA256` +- `TLS_AES_256_GCM_SHA384` +- `TLS_CHACHA20_POLY1305_SHA256` + +*Note*: **Golang** does [not support](https://github.com/golang/go/blob/go1.17/src/crypto/tls/cipher_suites.go#L676) all cipher suites with TLS 1.3. + +##### Implementation examples + +##### TLS 1.3 + +For TLS 1.3, **Golang** only supports [3 cipher suites](https://github.com/golang/go/blob/go1.17/src/crypto/tls/cipher_suites.go#L676), as such we only need to set the TLS version: + +```golang +cfg := &tls.Config{ + MinVersion: tls.VersionTLS13, +} +``` + +For **Ruby**, you can use [HTTParty](https://github.com/jnunemaker/httparty) and specify TLS 1.3 version as well as ciphers: + +Whenever possible this example should be **avoided** for security purposes: + +```ruby +response = HTTParty.get('https://gitlab.com', ssl_version: :TLSv1_3, ciphers: ['TLS_AES_128_GCM_SHA256', 'TLS_AES_256_GCM_SHA384', 'TLS_CHACHA20_POLY1305_SHA256']) +``` + +When using [`GitLab::HTTP`](#gitlab-http-library), the code looks like: + +This is the **recommended** implementation to avoid security issues such as SSRF: + +```ruby +response = GitLab::HTTP.perform_request(Net::HTTP::Get, 'https://gitlab.com', ssl_version: :TLSv1_3, ciphers: ['TLS_AES_128_GCM_SHA256', 'TLS_AES_256_GCM_SHA384', 'TLS_CHACHA20_POLY1305_SHA256']) +``` + +##### TLS 1.2 + +**Golang** does support multiple cipher suites that we do not want to use with TLS 1.2. We need to explicitly list authorized ciphers: + +```golang +func secureCipherSuites() []uint16 { + return []uint16{ + tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, + tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, + tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, + tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, + tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, + tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, + } +``` + +And then use `secureCipherSuites()` in `tls.Config`: + +```golang +tls.Config{ + (...), + CipherSuites: secureCipherSuites(), + MinVersion: tls.VersionTLS12, + (...), +} +``` + +This example was taken [here](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/871b52dc700f1a66f6644fbb1e78a6d463a6ff83/internal/tool/tlstool/tlstool.go#L72). + +For **Ruby**, you can use again [HTTParty](https://github.com/jnunemaker/httparty) and specify this time TLS 1.2 version alongside with the recommended ciphers: + +```ruby +response = GitLab::HTTP.perform_request(Net::HTTP::Get, 'https://gitlab.com', ssl_version: :TLSv1_2, ciphers: ['ECDHE-ECDSA-AES128-GCM-SHA256', 'ECDHE-RSA-AES128-GCM-SHA256', 'ECDHE-ECDSA-AES256-GCM-SHA384', 'ECDHE-RSA-AES256-GCM-SHA384', 'ECDHE-ECDSA-CHACHA20-POLY1305', 'ECDHE-RSA-CHACHA20-POLY1305']) +``` + ## GitLab Internal Authorization ### Introduction @@ -592,3 +680,90 @@ In order to prevent this from happening, it is recommended to use the method `us forbidden!(api_access_denied_message(user)) end ``` + +## Guidelines when defining missing methods with metaprogramming + +Metaprogramming is a way to define methods **at runtime**, instead of at the time of writing and deploying the code. It is a powerful tool, but can be dangerous if we allow untrusted actors (like users) to define their own arbitrary methods. For example, imagine we accidentally let an attacker overwrite an access control method to always return true! It can lead to many classes of vulnerabilities such as access control bypass, information disclosure, arbitrary file reads, and remote code execution. + +Key methods to watch out for are `method_missing`, `define_method`, `delegate`, and similar methods. + +### Insecure metaprogramming example + +This example is adapted from an example submitted by [@jobert](https://hackerone.com/jobert?type=user) through our HackerOne bug bounty program. +Thank you for your contribution! + +Before Ruby 2.5.1, you could implement delegators using the `delegate` or `method_missing` methods. For example: + +```ruby +class User + def initialize(attributes) + @options = OpenStruct.new(attributes) + end + + def is_admin? + name.eql?("Sid") # Note - never do this! + end + + def method_missing(method, *args) + @options.send(method, *args) + end +end +``` + +When a method was called on a `User` instance that didn't exist, it passed it along to the `@options` instance variable. + +```ruby +User.new({name: "Jeeves"}).is_admin? +# => false + +User.new(name: "Sid").is_admin? +# => true + +User.new(name: "Jeeves", "is_admin?" => true).is_admin? +# => false +``` + +Because the `is_admin?` method is already defined on the class, its behavior is not overridden when passing `is_admin?` to the initializer. + +This class can be refactored to use the `Forwardable` method and `def_delegators`: + +```ruby +class User + extend Forwardable + + def initialize(attributes) + @options = OpenStruct.new(attributes) + + self.class.instance_eval do + def_delegators :@options, *attributes.keys + end + end + + def is_admin? + name.eql?("Sid") # Note - never do this! + end +end +``` + +It might seem like this example has the same behavior as the first code example. However, there's one crucial difference: **because the delegators are meta-programmed after the class is loaded, it can overwrite existing methods**: + +```ruby +User.new({name: "Jeeves"}).is_admin? +# => false + +User.new(name: "Sid").is_admin? +# => true + +User.new(name: "Jeeves", "is_admin?" => true).is_admin? +# => true +# ^------------------ The method is overwritten! Sneaky Jeeves! +``` + +In the example above, the `is_admin?` method is overwritten when passing it to the initializer. + +### Best practices + +- Never pass user-provided details into method-defining metaprogramming methods. + - If you must, be **very** confident that you've sanitized the values correctly. + Consider creating an allowlist of values, and validating the user input against that. +- When extending classes that use metaprogramming, make sure you don't inadvertently override any method definition safety checks. diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 34a845147dc..65a8b4c1cad 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -295,19 +295,22 @@ Examples of implementation: - Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb) - Using Redis methods [`HINCRBY`](https://redis.io/commands/hincrby), [`HGETALL`](https://redis.io/commands/hgetall), and [`Gitlab::UsageCounters::PodLogs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_counters/pod_logs.rb) -##### UsageData API tracking +##### `UsageData` API -<!-- There's nearly identical content in `##### Adding new events`. If you fix errors here, you may need to fix the same errors in the other location. --> +You can use the `UsageData` API to track events. +To track events, the `usage_data_api` feature flag must +be enabled (set to `default_enabled: true`). +Enabled by default in GitLab 13.7 and later. -1. Track event using `UsageData` API +##### UsageData API tracking - Increment event count using ordinary Redis counter, for given event name. +1. Track events using the [`UsageData` API](#usagedata-api). - Tracking events using the `UsageData` API requires the `usage_data_api` feature flag to be enabled, which is enabled by default. + Increment event count using an ordinary Redis counter, for a given event name. API requests are protected by checking for a valid CSRF token. - To be able to increment the values, the related feature `usage_data_<event_name>` should be enabled. + To increment the values, the related feature `usage_data_<event_name>` must be enabled. ```plaintext POST /usage_data/increment_counter @@ -315,18 +318,18 @@ Examples of implementation: | Attribute | Type | Required | Description | | :-------- | :--- | :------- | :---------- | - | `event` | string | yes | The event name it should be tracked | + | `event` | string | yes | The event name to track. | Response: - - `200` if event was tracked - - `400 Bad request` if event parameter is missing - - `401 Unauthorized` if user is not authenticated - - `403 Forbidden` for invalid CSRF token provided + - `200` if the event was tracked. + - `400 Bad request` if the event parameter is missing. + - `401 Unauthorized` if the user is not authenticated. + - `403 Forbidden` if an invalid CSRF token is provided. -1. Track events using JavaScript/Vue API helper which calls the API above +1. Track events using the JavaScript/Vue API helper which calls the [`UsageData` API](#usagedata-api). - Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled to be able to track events + To track events, `usage_data_api` and `usage_data_#{event_name}` must be enabled. ```javascript import api from '~/api'; @@ -460,14 +463,10 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF track_usage_event(:incident_management_incident_created, current_user.id) ``` - - Using the `UsageData` API. - <!-- There's nearly identical content in `##### UsageData API Tracking`. If you find / fix errors here, you may need to fix errors in that section too. --> + - Using the [`UsageData` API](#usagedata-api). Increment unique users count using Redis HLL, for a given event name. - To track events using the `UsageData` API, ensure the `usage_data_api` feature flag - is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later. - API requests are protected by checking for a valid CSRF token. ```plaintext @@ -485,10 +484,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF - `401 Unauthorized` if the user is not authenticated. - `403 Forbidden` if an invalid CSRF token is provided. - - Using the JavaScript/Vue API helper, which calls the `UsageData` API. - - To track events using the `UsageData` API, ensure the `usage_data_api` feature flag - is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later. + - Using the JavaScript/Vue API helper, which calls the [`UsageData` API](#usagedata-api). Example for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/): @@ -559,13 +555,18 @@ We can also disable tracking completely by using the global flag: ##### Known events are added automatically in Service Data payload -All events added in [`known_events/common.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml) are automatically added to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). +Service Ping adds all events [`known_events/*.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events) to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: - `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#add-new-events) events and data for the last complete week for weekly [aggregation](#add-new-events) events. - `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#add-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#add-new-events) events. -Redis HLL implementation calculates automatic total metrics, if there are more than one metric for the same category, aggregation, and Redis slot. +Redis HLL implementation calculates total metrics when both of these conditions are met: + +- The category is manually included in [CATEGORIES_FOR_TOTALS](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/hll_redis_counter.rb#L21). +- There is more than one metric for the same category, aggregation, and Redis slot. + +We add total unique counts for the weekly and monthly time frames where applicable: - `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. - `#{category}_total_unique_counts_monthly`: Total unique counts for events in same category for the last 28 days or the last 4 complete weeks, if events are in the same Redis slot and we have more than one metric. diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 19bf7446da9..6ddbe2f9646 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -50,35 +50,57 @@ We use the following terminology to describe the Service Ping components: - **Metrics**: primarily made up of row counts for different tables in an instance's database. Each metric has a corresponding [metric definition](metrics_dictionary.md#metrics-definition-and-validation) in a YAML file. +- **MAU**: monthly active users. +- **WAU**: weekly active users. ### Why should we enable Service Ping? - The main purpose of Service Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. - As a benefit of having Service Ping active, GitLab lets you analyze the users' activities over time of your GitLab installation. -- As a benefit of having Service Ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. +- As a benefit of having Service Ping active, GitLab provides you with [DevOps Score](../../user/admin_area/analytics/dev_ops_report.md#devops-score), which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. - You get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) - You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. - Service Ping is enabled by default. To disable it, see [Disable Service Ping](#disable-service-ping). - When Service Ping is enabled, you have the option to participate in our [Registration Features Program](#registration-features-program) and receive free paid features. -#### Registration Features Program +### Limitations + +- Service Ping does not track frontend events things like page views, link clicks, or user sessions. +- Service Ping focuses only on aggregated backend events. + +Because of these limitations we recommend you: + +- Instrument your products with Snowplow for more detailed analytics on GitLab.com. +- Use Service Ping to track aggregated backend events on self-managed instances. + +### Registration Features Program > Introduced in GitLab 14.1. -Starting with GitLab version 14.1, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data via [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. +In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. -##### Features available in 14.1 and later +#### Features available in 14.1 and later 1. [Email from GitLab](../../tools/email.md). +#### Features available in 14.4 and later + +1. [Repository size limit](../../user/admin_area/settings/account_and_limit_settings.md#repository-size-limit). +1. [Restrict group access by IP address](../../user/group/index.md#restrict-group-access-by-ip-address). + NOTE: Registration is not yet required for participation, but will be added in a future milestone. -### Limitations +#### Enable Registration Features -- Service Ping does not track frontend events things like page views, link clicks, or user sessions, and only focuses on aggregated backend events. -- Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Service Ping to track aggregated backend events on self-managed. +1. Sign in as a user with the [Administrator](../../user/permissions.md) role. +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling**. +1. Expand the **Usage statistics** section. +1. If not enabled, select the **Enable Service Ping** checkbox. +1. Select the **Enable Registration Features** checkbox. +1. Select **Save changes**. ## View the Service Ping payload **(FREE SELF)** @@ -180,7 +202,7 @@ sequenceDiagram S3 Bucket->>Snowflake DW: Import data Snowflake DW->>Snowflake DW: Transform data using dbt Snowflake DW->>Sisense Dashboards: Data available for querying - Versions Application->>GitLab Instance: DevOps Report (Conversational Development Index) + Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index) ``` ## How Service Ping works @@ -209,10 +231,6 @@ We also collect metrics specific to [Geo](../../administration/geo/index.md) sec "repositories_replication_enabled"=>true, "repositories_synced_count"=>24, "repositories_failed_count"=>0, - "attachments_replication_enabled"=>true, - "attachments_count"=>1, - "attachments_synced_count"=>1, - "attachments_failed_count"=>0, "git_fetch_event_count_weekly"=>nil, "git_push_event_count_weekly"=>nil, ... other geo node status fields @@ -450,27 +468,56 @@ bin/rake gitlab:usage_data:dump_sql_in_json bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml ``` -## Generating and troubleshooting Service Ping - -This activity is to be done via a detached screen session on a remote server. +## Generate Service Ping -Before you begin these steps, make sure the key is added to the SSH agent locally -with the `ssh-add` command. +To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a detached screen session on a remote server. ### Triggering -1. Connect to bastion with agent forwarding: `$ ssh -A lb-bastion.gprd.gitlab.com` -1. Create named screen: `$ screen -S <username>_usage_ping_<date>` -1. Connect to console host: `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal` -1. Run `SubmitUsagePingService.new.execute` -1. Detach from screen: `ctrl + a, ctrl + d` -1. Exit from bastion: `$ exit` +#### Trigger Service Ping with Teleport + +1. Request temporary [access](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to the required environment. +1. After your approval is issued, [access the Rails console](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#access-approval). +1. Run `ServicePing::SubmitService.new.execute`. + +#### Trigger Service Ping with a detached screen session + +1. Connect to bastion with agent forwarding: + + `$ ssh -A lb-bastion.gprd.gitlab.com`. +1. Create named screen: + + `$ screen -S <username>_usage_ping_<date>`. +1. Connect to console host: + + `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal`. +1. Run `ServicePing::SubmitService.new.execute` +1. Detach from screen: + + `ctrl + a, ctrl + d` +1. Exit from bastion: + + `$ exit` ### Verification (After approx 30 hours) -1. Reconnect to bastion: `$ ssh -A lb-bastion.gprd.gitlab.com` -1. Find your screen session: `$ screen -ls` -1. Attach to your screen session: `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22` +#### Verify with a detached screen session + +1. Follow [the steps](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to request a new access to the required environment and connect to the Rails console +1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` +1. Check the when the payload was sent: `RawUsageData.last.sent_at` + +#### Verify using detached screen session + +1. Reconnect to bastion: + + `$ ssh -A lb-bastion.gprd.gitlab.com` +1. Find your screen session: + + `$ screen -ls` +1. Attach to your screen session: + + `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22` 1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` 1. Check the when the payload was sent: `RawUsageData.last.sent_at` diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index c1478e6290e..b3c404de150 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -197,18 +197,28 @@ tier: The GitLab codebase provides a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_definition_generator.rb) to create new metric definitions. -For uniqueness, the generated file includes a timestamp prefix, in ISO 8601 format. +For uniqueness, the generated files include a timestamp prefix in ISO 8601 format. -The generator takes the key path argument and 2 options and creates the metric YAML definition in corresponding location: +The generator takes a list of key paths and 2 options as arguments. It creates metric YAML definitions in the corresponding location: - `--ee`, `--no-ee` Indicates if metric is for EE. -- `--dir=DIR` indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`. +- `--dir=DIR` Indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`. + +**Single metric example** ```shell bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d create config/metrics/counts_7d/issues.yml ``` +**Multiple metrics example** + +```shell +bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d +create config/metrics/counts_7d/issues.yml +create config/metrics/counts_7d/users.yml +``` + NOTE: To create a metric definition used in EE, add the `--ee` flag. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index d45e2073fe7..e28a328888d 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -255,6 +255,11 @@ module Ci end ``` +Also, you can pass `if_deduplicated: :reschedule_once` option to re-run a job once after +the currently running job finished and deduplication happened at least once. +This ensures that the latest result is always produced even if a race condition +happened. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342123) for more information. + #### Scheduling jobs in the future GitLab doesn't skip jobs scheduled in the future, as we assume that @@ -279,6 +284,36 @@ module AuthorizedProjectUpdate end ``` +### Setting the deduplication time-to-live (TTL) + +Deduplication depends on an idempotency key that is stored in Redis. This is normally +cleared by the configured deduplication strategy. + +However, the key can remain until its TTL in certain cases like: + +1. `until_executing` is used but the job was never enqueued or executed after the Sidekiq + client middleware was run. + +1. `until_executed` is used but the job fails to finish due to retry exhaustion, gets + interrupted the maximum number of times, or gets lost. + +The default value is 6 hours. During this time, jobs won't be enqueued even if the first +job never executed or finished. + +The TTL can be configured with: + +```ruby +class ProjectImportScheduleWorker + include ApplicationWorker + + idempotent! + deduplicate :until_executing, ttl: 5.minutes +end +``` + +Duplicate jobs can happen when the TTL is reached, so make sure you lower this only for jobs +that can tolerate some duplication. + ### Deduplication with load balancing > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6763) in GitLab 14.4. @@ -740,8 +775,7 @@ We use the following approach to determine whether a worker is CPU-bound: ## Feature category -All Sidekiq workers must define a known [feature -category](feature_categorization/index.md#sidekiq-workers). +All Sidekiq workers must define a known [feature category](feature_categorization/index.md#sidekiq-workers). ## Job weights diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index 0d81b442850..fe1de789eae 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -102,14 +102,12 @@ track_action: "click_button" }) ### Implement Vue component tracking For custom event tracking, use a Vue `mixin` in components. Vue `mixin` exposes the `Tracking.event` -static method and the `track` method called from components or templates. You can specify tracking -options in `data` or `computed`. These options override any defaults and allow the values to be dynamic -from props or based on state. +static method and the `track` method. You can specify tracking options in `data` or `computed`. +These options override any defaults and allow the values to be dynamic from props or based on state. -Default options are passed when an event is tracked from the component. If you don't specify an option, -the default `document.body.dataset.page` is used. The default options are: +Several default options are passed when an event is tracked from the component: -- `category` +- `category`: If you don't specify, by default `document.body.dataset.page` is used. - `label` - `property` - `value` @@ -426,7 +424,7 @@ records the same events as the full Snowplow pipeline. To query events, use the To install and run Snowplow Micro, complete these steps to modify the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit): -1. Ensure Docker is installed and running. +1. Ensure [Docker is installed](https://docs.docker.com/get-docker/) and running. 1. To install Snowplow Micro, clone the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration). @@ -438,73 +436,18 @@ To install and run Snowplow Micro, complete these steps to modify the ./snowplow-micro.sh ``` -1. Use GDK to start the PostgreSQL terminal and connect - to the `gitlabhq_development` database: +1. Set the environment variable to tell the GDK to use Snowplow Micro in development. This overrides two `application_settings` options: + - `snowplow_enabled` setting will instead return `true` from `Gitlab::Tracking.enabled?` + - `snowplow_collector_hostname` setting will instead always return `localhost:9090` (or whatever is set for `SNOWPLOW_MICRO_URI`) from `Gitlab::Tracking.collector_hostname`. ```shell - gdk psql -d gitlabhq_development + export SNOWPLOW_MICRO_ENABLE=1 ``` -1. Update your instance's settings to enable Snowplow events and - point to the Snowplow Micro collector: + Optionally, you can set the URI for you Snowplow Micro instance as well (the default value is `http://localhost:9090`): ```shell - update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; - ``` - -1. Update `DEFAULT_SNOWPLOW_OPTIONS` in `app/assets/javascripts/tracking/constants.js` to remove `forceSecureTracker: true`: - - ```diff - diff --git a/app/assets/javascripts/tracking/constants.js b/app/assets/javascripts/tracking/constants.js - index 598111e4086..eff38074d4c 100644 - --- a/app/assets/javascripts/tracking/constants.js - +++ b/app/assets/javascripts/tracking/constants.js - @@ -7,7 +7,6 @@ export const DEFAULT_SNOWPLOW_OPTIONS = { - appId: '', - userFingerprint: false, - respectDoNotTrack: true, - - forceSecureTracker: true, - eventMethod: 'post', - contexts: { webPage: true, performanceTiming: true }, - formTracking: false, - ``` - -1. Update `options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: - - ```diff - diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb - index 618e359211b..e9084623c43 100644 - --- a/lib/gitlab/tracking.rb - +++ b/lib/gitlab/tracking.rb - @@ -41,7 +41,9 @@ def options(group) - cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain, - app_id: Gitlab::CurrentSettings.snowplow_app_id, - form_tracking: additional_features, - - link_click_tracking: additional_features - + link_click_tracking: additional_features, - + protocol: 'http', - + port: 9090 - }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym } - end - ``` - -1. Update `emitter` in `lib/gitlab/tracking/destinations/snowplow.rb` to change `protocol`: - - ```diff - diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb - index 4fa844de325..5dd9d0eacfb 100644 - --- a/lib/gitlab/tracking/destinations/snowplow.rb - +++ b/lib/gitlab/tracking/destinations/snowplow.rb - @@ -40,7 +40,7 @@ def tracker - def emitter - SnowplowTracker::AsyncEmitter.new( - Gitlab::CurrentSettings.snowplow_collector_hostname, - - protocol: 'https' - + protocol: 'http' - ) - end - end - + export SNOWPLOW_MICRO_URI=https://127.0.0.1:8080 ``` 1. Restart GDK: diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index b8b35857adf..f4f41221699 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -6,10 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Snowplow -This page provides an overview of how Snowplow works and how to enable it. - -## What is Snowplow - Snowplow is an enterprise-grade marketing and Product Intelligence platform that tracks how users engage with our website and application. [Snowplow](https://snowplowanalytics.com) consists of several loosely-coupled sub-systems: @@ -23,12 +19,6 @@ Snowplow is an enterprise-grade marketing and Product Intelligence platform that  -### Useful links - -- [Snowplow data structure](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/) -- [Our Iglu schema registry](https://gitlab.com/gitlab-org/iglu) -- [List of events used in our codebase (Event Dictionary)](https://metrics.gitlab.com/snowplow.html) - ## Enable Snowplow tracking Tracking can be enabled at: @@ -165,6 +155,9 @@ Snowplow JavaScript adds [web-specific parameters](https://docs.snowplowanalytic ## Related topics +- [Snowplow data structure](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/) +- [Our Iglu schema registry](https://gitlab.com/gitlab-org/iglu) +- [List of events used in our codebase (Event Dictionary)](https://metrics.gitlab.com/snowplow.html) - [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) - [Service Ping Guide](../service_ping/index.md) - [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 52e89a10556..6a739d9e1a5 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -50,6 +50,23 @@ bundle exec guard When using spring and guard together, use `SPRING=1 bundle exec guard` instead to make use of spring. +### Eager loading the application code + +By default, the application code: + +- Isn't eagerly loaded in the `test` environment. +- Is eagerly loaded in CI/CD (when `ENV['CI'].present?`) to surface any potential loading issues. + +If you need to enable eager loading when executing tests, +use the `GITLAB_TEST_EAGER_LOAD` environment variable: + +```shell +GITLAB_TEST_EAGER_LOAD=1 bin/rspec spec/models/project_spec.rb +``` + +If your test depends on all the application code that is being loaded, add the `:eager_load` tag. +This ensures that the application code is eagerly loaded before the test execution. + ### Ruby warnings > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47767) in GitLab 13.7. diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index 7370cc5771b..27a87d25170 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -350,7 +350,7 @@ GITLAB_PASSWORD=<GDK root password> bundle exec bin/qa Test::Instance::All http: Where `<test_file>` is: - `qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb` when running the Login example. -- `qa/specs/features/browser_ui/2_plan/issues/create_issue_spec.rb` when running the Issue example. +- `qa/specs/features/browser_ui/2_plan/issue/create_issue_spec.rb` when running the Issue example. ## End-to-end test merge request template diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 994ee3f253c..52212410cc6 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -79,11 +79,21 @@ for details. End-to-end tests should pass with a feature flag enabled before it is enabled on Staging or on GitLab.com. Tests that need to be updated should be identified as part of [quad-planning](https://about.gitlab.com/handbook/engineering/quality/quad-planning/). The relevant [counterpart Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors) is responsible for updating the tests or assisting another engineer to do so. However, if a change does not go through quad-planning and a required test update is not made, test failures could block deployment. -If a test enables a feature flag as describe above, it is sufficient to run the `package-and-qa` job in a merge request containing the relevant changes. -Or, if the feature flag and relevant changes have already been merged, you can confirm that the tests -pass on `main`. The end-to-end tests run on `main` every two hours, and the results are posted to a [Test -Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amaster). +### Automatic test execution when a feature flag definition changes -If the relevant tests do not enable the feature flag themselves, you can check if the tests will need -to be updated by opening a draft merge request that enables the flag by default and then running the `package-and-qa` job. +If a merge request adds or edits a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation), +two `package-and-qa` jobs will be included automatically in the merge request pipeline. One job will enable the defined +feature flag and the other will disable it. The jobs execute the same suite of tests to confirm that they pass with if +the feature flag is either enabled or disabled. + +### Test execution during feature development + +If an end-to-end test enables a feature flag, the end-to-end test suite can be used to test changes in a merge request +by running the `package-and-qa` job in the merge request pipeline. If the feature flag and relevant changes have already been merged, you can confirm that the tests +pass on the default branch. The end-to-end tests run on the default branch every two hours, and the results are posted to a [Test +Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amain). + +If the relevant tests do not enable the feature flag themselves, you can check if the tests will need to be updated by opening +a draft merge request that enables the flag by default via a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation). +That will [automatically execute the end-to-end test suite](#automatic-test-execution-when-a-feature-flag-definition-changes). The merge request can be closed once the tests pass. If you need assistance to update the tests, please contact the relevant [stable counterpart in the Quality department](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), or any Software Engineer in Test if there is no stable counterpart for your group. diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index b097d6b0729..57c8c3bf59c 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -147,7 +147,7 @@ as well as these: | Variable | Description | |-|-| | `QA_SCENARIO` | The scenario to run (default `Test::Instance::Image`) | -| `QA_TESTS` | The test(s) to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, e.g., `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. | +| `QA_TESTS` | The test(s) to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, for example, `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. | | `QA_RSPEC_TAGS` | The RSpec tags to add (no default) | For now [manual jobs with custom variables don't use the same variable diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index eadd0ef49a0..95984a701e7 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -165,20 +165,20 @@ QA_DEBUG=true WEBDRIVER_HEADLESS=false GITLAB_ADMIN_USERNAME=rootusername GITLAB The following includes more information on the command: --`QA_DEBUG` - Set to `true` to verbosely log page object actions. --`WEBDRIVER_HEADLESS` - When running locally, set to `false` to allow browser tests to be visible - watch your tests being run. --`GITLAB_ADMIN_USERNAME` - Administrator username to use when adding a license. --`GITLAB_ADMIN_PASSWORD` - Administrator password to use when adding a license. --`GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with administrator access. Used for API access as an administrator during tests. --`CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps. --`https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK --`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs --`--tag` - the meta-tags used to filter the specs correctly +- `QA_DEBUG` - Set to `true` to verbosely log page object actions. +- `WEBDRIVER_HEADLESS` - When running locally, set to `false` to allow browser tests to be visible - watch your tests being run. +- `GITLAB_ADMIN_USERNAME` - Administrator username to use when adding a license. +- `GITLAB_ADMIN_PASSWORD` - Administrator password to use when adding a license. +- `GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with administrator access. Used for API access as an administrator during tests. +- `CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps. +- `https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK +- `qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs +- `--tag` - the meta-tags used to filter the specs correctly At the moment of this writing, there are two specs which run monitor tests: --`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Free --`qa/specs/features/ee/browser_ui/8_monitor/all_monitor_features_spec.rb` - has the specs of features for paid GitLab (Enterprise Edition) +- `qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Free +- `qa/specs/features/ee/browser_ui/8_monitor/all_monitor_features_spec.rb` - has the specs of features for paid GitLab (Enterprise Edition) ### How to debug @@ -485,3 +485,109 @@ To run the LDAP tests on your local with TLS disabled, follow these steps: ```shell GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_DEBUG=true WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb ``` + +## Guide to the mobile suite + +### What are mobile tests + +Tests that are tagged with `:mobile` can be run against specified mobile devices using cloud emulator/simulator services. + +### How to run mobile tests with Sauce Labs + +Running directly against an environment like staging is not recommended because Sauce Labs test logs expose credentials. Therefore, it is best practice and the default to use a tunnel. + +Tunnel installation instructions are here [https://docs.saucelabs.com/secure-connections/sauce-connect/installation]. To start the tunnel, after following the installation above, copy the run command in Sauce Labs > Tunnels (must be logged in to Sauce Labs with the credentials found in 1Password) and run in terminal. + +NOTE: +It is highly recommended to use `GITLAB_QA_ACCESS_TOKEN` to speed up tests and reduce flakiness. + +`QA_REMOTE_MOBILE_DEVICE_NAME` can be any device name listed in [https://saucelabs.com/platform/supported-browsers-devices] under Emulators/simulators and the latest versions of Android or iOS. `QA_BROWSER` must be set to `safari` for iOS devices and `chrome` for Android devices. + +1. To test against a local instance with a tunnel running, in `gitlab/qa` run: + +```shell +$ QA_BROWSER="safari" \ + QA_REMOTE_MOBILE_DEVICE_NAME="iPhone 12 Simulator" \ + QA_REMOTE_GRID="ondemand.saucelabs.com:80" \ + QA_REMOTE_GRID_USERNAME="gitlab-sl" \ + QA_REMOTE_GRID_ACCESS_KEY="<found in Sauce Lab account>" \ + GITLAB_QA_ACCESS_TOKEN="<token>" \ + bundle exec bin/qa Test::Instance::All http://<local_ip>:3000 -- <relative_spec_path> +``` + +Results can be watched in real time while logged into Sauce Labs under AUTOMATED > Test Results. + +### How to add an existing test to the mobile suite + +The main reason a test might fail when adding the `:mobile` tag is navigation differences in desktop vs mobile layouts, therefore the test needs to be updated to use mobile navigation when running mobile tests. + +If an existing method needs to be changed or a new one created, a new mobile page object should be created in `qa/qa/mobile/page/` and it should be prepended in the original page object by adding: + +```ruby +prepend Mobile::Page::NewPageObject if Runtime::Env.mobile_layout? +``` + +For example to change an existing method when running mobile tests: + +New mobile page object: + +```ruby +module QA + module Mobile + module Page + module Project + module Show + extend QA::Page::PageConcern + + def self.prepended(base) + super + + base.class_eval do + prepend QA::Mobile::Page::Main::Menu + + view 'app/assets/javascripts/nav/components/top_nav_new_dropdown.vue' do + element :new_issue_mobile_button + end + end + end + + def go_to_new_issue + open_mobile_new_dropdown + + click_element(:new_issue_mobile_button) + end + end + end + end + end +end +``` + +Original page object prepending the new mobile if there's a mobile layout: + +```ruby +module QA + module Page + module Project + class Show < Page::Base + prepend Mobile::Page::Project::Show if Runtime::Env.mobile_layout? + + view 'app/views/layouts/header/_new_dropdown.html.haml' do + element :new_menu_toggle + end + + view 'app/helpers/nav/new_dropdown_helper.rb' do + element :new_issue_link + end + + def go_to_new_issue + click_element(:new_menu_toggle) + click_element(:new_issue_link) + end + end + end + end +end +``` + +When running mobile tests for phone layouts, both `remote_mobile_device_name` and `mobile_layout` are `true` but when using a tablet layout, only `remote_mobile_device_name` is true. This is because phone layouts have more menus closed by default such as how both tablets and phones have the left nav closed but unlike phone layouts, tablets have the regular top navigation bar, not the mobile one. So in the case where the navigation being edited needs to be used in tablet layouts as well, use `remote_mobile_device_name` instead of `mobile_layout?` when prepending so it will use it if it's a tablet layout as well. diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 22ddae6e836..a9571df1188 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -14,7 +14,7 @@ This document describes the conventions used at GitLab for writing End-to-end (E When clicking in a single link to navigate, use `click_`. -E.g.: +For example: ```ruby def click_ci_cd_pipelines @@ -33,7 +33,7 @@ From a testing perspective, if we want to check that clicking a link, or a butto When interacting with multiple elements to go to a page, use `go_to_`. -E.g.: +For example: ```ruby def go_to_operations_environments @@ -63,12 +63,12 @@ We follow a simple formula roughly based on Hungarian notation. - `type`: A generic control on the page that can be seen by a user. - `_button` - `_checkbox` - - `_container`: an element that includes other elements, but doesn't present visible content itself. E.g., an element that has a third-party editor inside it, but which isn't the editor itself and so doesn't include the editor's content. + - `_container`: an element that includes other elements, but doesn't present visible content itself. For example, an element that has a third-party editor inside it, but which isn't the editor itself and so doesn't include the editor's content. - `_content`: any element that contains text, images, or any other content displayed to the user. - `_dropdown` - `_field`: a text input element. - `_link` - - `_modal`: a popup modal dialog, e.g., a confirmation prompt. + - `_modal`: a popup modal dialog, for example, a confirmation prompt. - `_placeholder`: a temporary element that appears while content is loading. For example, the elements that are displayed instead of discussions while the discussions are being fetched. - `_radio` - `_tab` @@ -116,7 +116,7 @@ we use the name of the page object in [snake_case](https://en.wikipedia.org/wiki (all lowercase, with words separated by an underscore). See good and bad examples below. While we prefer to follow the standard in most cases, it is also acceptable to -use common abbreviations (e.g., `mr`) or other alternatives, as long as +use common abbreviations (for example, `mr`) or other alternatives, as long as the name is not ambiguous. This can include appending `_page` if it helps to avoid confusion or make the code more readable. For example, if a page object is named `New`, it could be confusing to name the block argument `new` because that diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 0e721ba2760..3096386d7c3 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -423,7 +423,8 @@ it('passes', () => { ### Waiting in tests Sometimes a test needs to wait for something to happen in the application before it continues. -Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) +Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/setTimeout) + because it makes the reason for waiting unclear. Instead use one of the following approaches. #### Promises and Ajax calls diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 4091f213a8f..3190ab6c899 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -16,6 +16,7 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be - for merge requests with frontend changes - for merge requests with QA changes - for scheduled pipelines +- the MR has the `pipeline:run-review-app` label set ## QA runs on Review Apps @@ -37,6 +38,15 @@ browser performance testing using a ## How to +### Redeploy Review App from a clean slate + +To reset Review App and redeploy from a clean slate, do the following: + +1. Run `review-stop` job. +1. Re-deploy by running or retrying `review-deploy` job. + +Doing this will remove all existing data from a previously deployed Review App. + ### Get access to the GCP Review Apps cluster You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/-/issues/new) @@ -180,7 +190,7 @@ subgraph "CNG-mirror pipeline" **Additional notes:** - If the `review-deploy` job keeps failing (and a manual retry didn't help), - please post a message in the `#g_qe_engineering_productivity` channel and/or create a `~"Engineering Productivity"` `~"ep::review apps"` `~bug` + please post a message in the `#g_qe_engineering_productivity` channel and/or create a `~"Engineering Productivity"` `~"ep::review apps"` `~"type::bug"` issue with a link to your merge request. Note that the deployment failure can reveal an actual problem introduced in your merge request (i.e. this isn't necessarily a transient failure)! @@ -189,7 +199,7 @@ subgraph "CNG-mirror pipeline" your merge request. You can also download the artifacts to see screenshots of the page at the time the failures occurred. If you don't find the cause of the failure or if it seems unrelated to your change, please post a message in the - `#quality` channel and/or create a ~Quality ~bug issue with a link to your + `#quality` channel and/or create a ~Quality ~"type::bug" issue with a link to your merge request. - The manual `review-stop` can be used to stop a Review App manually, and is also started by GitLab once a merge @@ -238,6 +248,16 @@ Leading indicators may be health check failures leading to restarts or majority The [Review Apps Overview dashboard](https://console.cloud.google.com/monitoring/classic/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d) aids in identifying load spikes on the cluster, and if nodes are problematic or the entire cluster is trending towards unhealthy. +### Database related errors in `review-deploy` or `review-qa-smoke` + +Occasionally the state of a Review App's database could diverge from the database schema. This could be caused by +changes to migration files or schema, such as a migration being renamed or deleted. This typically manifest in migration errors such as: + +- migration job failing with a column that already exists +- migration job failing with a column that does not exist + +To recover from this, please attempt to [redeploy Review App from a clean slate](#redeploy-review-app-from-a-clean-slate) + ### Release failed with `ImagePullBackOff` **Potential cause:** diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md index 757a70fb4e0..4092c1a2f6d 100644 --- a/doc/development/testing_guide/testing_migrations_guide.md +++ b/doc/development/testing_guide/testing_migrations_guide.md @@ -38,7 +38,7 @@ ensures proper isolation. ## Testing an `ActiveRecord::Migration` class -To test an `ActiveRecord::Migration` class (i.e., a +To test an `ActiveRecord::Migration` class (for example, a regular migration `db/migrate` or a post-migration `db/post_migrate`), you must load the migration file by using the `require_migration!` helper method because it is not autoloaded by Rails. @@ -61,14 +61,35 @@ Since the migration files are not autoloaded by Rails, you must manually load the migration file. To do so, you can use the `require_migration!` helper method which can automatically load the correct migration file based on the spec filename. -For example, if your spec file is named as `populate_foo_column_spec.rb` then the -helper method tries to load `${schema_version}_populate_foo_column.rb` migration file. +In GitLab 14.4 and later, you can use `require_migration!` to load migration files from spec files +that contain the schema version in the filename (for example, +`2021101412150000_populate_foo_column_spec.rb`). -In case there is no pattern between your spec file and the actual migration file, -you can provide the migration filename without the schema version, like so: +```ruby +# frozen_string_literal: true + +require 'spec_helper' +require_migration! + +RSpec.describe PopulateFooColumn do + ... +end +``` + +In some cases, you must require multiple migration files to use them in your specs. Here, there's no +pattern between your spec file and the other migration file. You can provide the migration filename +like so: ```ruby -require_migration!('populate_foo_column') +# frozen_string_literal: true + +require 'spec_helper' +require_migration! +require_migration!('populate_bar_column') + +RSpec.describe PopulateFooColumn do + ... +end ``` #### `table` @@ -213,54 +234,65 @@ Migration tests depend on what the migration does exactly, the most common types #### Example of a data migration test This spec tests the -[`db/post_migrate/20170526185842_migrate_pipeline_stages.rb`](https://gitlab.com/gitlab-org/gitlab-foss/blob/v11.6.5/db/post_migrate/20170526185842_migrate_pipeline_stages.rb) +[`db/post_migrate/20200723040950_migrate_incident_issues_to_incident_type.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20200723040950_migrate_incident_issues_to_incident_type.rb) migration. You can find the complete spec in -[`spec/migrations/migrate_pipeline_stages_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/blob/v11.6.5/spec/migrations/migrate_pipeline_stages_spec.rb). +[`spec/migrations/migrate_incident_issues_to_incident_type_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/migrations/migrate_incident_issues_to_incident_type_spec.rb). ```ruby -require 'spec_helper' +# frozen_string_literal: true +require 'spec_helper' require_migration! -RSpec.describe MigratePipelineStages do - # Create test data - pipeline and CI/CD jobs. - let(:jobs) { table(:ci_builds) } - let(:stages) { table(:ci_stages) } - let(:pipelines) { table(:ci_pipelines) } +RSpec.describe MigrateIncidentIssuesToIncidentType do + let(:migration) { described_class.new } + let(:projects) { table(:projects) } + let(:namespaces) { table(:namespaces) } + let(:labels) { table(:labels) } + let(:issues) { table(:issues) } + let(:label_links) { table(:label_links) } + let(:label_props) { IncidentManagement::CreateIncidentLabelService::LABEL_PROPERTIES } + + let(:namespace) { namespaces.create!(name: 'foo', path: 'foo') } + let!(:project) { projects.create!(namespace_id: namespace.id) } + let(:label) { labels.create!(project_id: project.id, **label_props) } + let!(:incident_issue) { issues.create!(project_id: project.id) } + let!(:other_issue) { issues.create!(project_id: project.id) } + + # Issue issue_type enum + let(:issue_type) { 0 } + let(:incident_type) { 1 } before do - projects.create!(id: 123, name: 'gitlab1', path: 'gitlab1') - pipelines.create!(id: 1, project_id: 123, ref: 'master', sha: 'adf43c3a') - jobs.create!(id: 1, commit_id: 1, project_id: 123, stage_idx: 2, stage: 'build') - jobs.create!(id: 2, commit_id: 1, project_id: 123, stage_idx: 1, stage: 'test') + label_links.create!(target_id: incident_issue.id, label_id: label.id, target_type: 'Issue') end - # Test just the up migration. - it 'correctly migrates pipeline stages' do - expect(stages.count).to be_zero - - migrate! + describe '#up' do + it 'updates the incident issue type' do + expect { migrate! } + .to change { incident_issue.reload.issue_type } + .from(issue_type) + .to(incident_type) - expect(stages.count).to eq 2 - expect(stages.all.pluck(:name)).to match_array %w[test build] + expect(other_issue.reload.issue_type).to eql(issue_type) + end end - # Test a reversible migration. - it 'correctly migrates up and down pipeline stages' do - reversible_migration do |migration| - # Expectations will run before the up migration, - # and then again after the down migration - migration.before -> { - expect(stages.count).to be_zero - } - - # Expectations will run after the up migration. - migration.after -> { - expect(stages.count).to eq 2 - expect(stages.all.pluck(:name)).to match_array %w[test build] - } + describe '#down' do + let!(:incident_issue) { issues.create!(project_id: project.id, issue_type: issue_type) } + + it 'updates the incident issue type' do + migration.up + + expect { migration.down } + .to change { incident_issue.reload.issue_type } + .from(incident_type) + .to(issue_type) + + expect(other_issue.reload.issue_type).to eql(issue_type) end + end end ``` @@ -336,41 +368,62 @@ end ### Example background migration test This spec tests the -[`lib/gitlab/background_migration/archive_legacy_traces.rb`](https://gitlab.com/gitlab-org/gitlab-foss/blob/v11.6.5/lib/gitlab/background_migration/archive_legacy_traces.rb) +[`lib/gitlab/background_migration/backfill_draft_status_on_merge_requests.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/background_migration/backfill_draft_status_on_merge_requests.rb) background migration. You can find the complete spec on -[`spec/lib/gitlab/background_migration/archive_legacy_traces_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/blob/v11.6.5/spec/lib/gitlab/background_migration/archive_legacy_traces_spec.rb) +[`spec/lib/gitlab/background_migration/backfill_draft_status_on_merge_requests_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/background_migration/backfill_draft_status_on_merge_requests_spec.rb) ```ruby +# frozen_string_literal: true + require 'spec_helper' -describe Gitlab::BackgroundMigration::ArchiveLegacyTraces, schema: 20180529152628 do - include TraceHelpers +RSpec.describe Gitlab::BackgroundMigration::BackfillDraftStatusOnMergeRequests do + let(:namespaces) { table(:namespaces) } + let(:projects) { table(:projects) } + let(:merge_requests) { table(:merge_requests) } - let(:namespaces) { table(:namespaces) } - let(:projects) { table(:projects) } - let(:builds) { table(:ci_builds) } - let(:job_artifacts) { table(:ci_job_artifacts) } + let(:group) { namespaces.create!(name: 'gitlab', path: 'gitlab') } + let(:project) { projects.create!(namespace_id: group.id) } - before do - namespaces.create!(id: 123, name: 'gitlab1', path: 'gitlab1') - projects.create!(id: 123, name: 'gitlab1', path: 'gitlab1', namespace_id: 123) - @build = builds.create!(id: 1, project_id: 123, status: 'success', type: 'Ci::Build') + let(:draft_prefixes) { ["[Draft]", "(Draft)", "Draft:", "Draft", "[WIP]", "WIP:", "WIP"] } + + def create_merge_request(params) + common_params = { + target_project_id: project.id, + target_branch: 'feature1', + source_branch: 'master' + } + + merge_requests.create!(common_params.merge(params)) end - context 'when trace file exists at the right place' do + context "for MRs with #draft? == true titles but draft attribute false" do + let(:mr_ids) { merge_requests.all.collect(&:id) } + before do - create_legacy_trace(@build, 'trace in file') + draft_prefixes.each do |prefix| + (1..4).each do |n| + create_merge_request( + title: "#{prefix} This is a title", + draft: false, + state_id: n + ) + end + end end - it 'correctly archive legacy traces' do - expect(job_artifacts.count).to eq(0) - expect(File.exist?(legacy_trace_path(@build))).to be_truthy + it "updates all open draft merge request's draft field to true" do + mr_count = merge_requests.all.count + + expect { subject.perform(mr_ids.first, mr_ids.last) } + .to change { MergeRequest.where(draft: false).count } + .from(mr_count).to(mr_count - draft_prefixes.length) + end - described_class.new.perform(1, 1) + it "marks successful slices as completed" do + expect(subject).to receive(:mark_job_as_succeeded).with(mr_ids.first, mr_ids.last) - expect(job_artifacts.count).to eq(1) - expect(File.exist?(legacy_trace_path(@build))).to be_falsy - expect(File.read(archived_trace_path(job_artifacts.first))).to eq('trace in file') + subject.perform(mr_ids.first, mr_ids.last) end end end diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index c3dfeaa6b92..e06ece38135 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -172,7 +172,7 @@ cost describing how expensive the entire node was. In general: the greater the values, the more expensive the node. When using `EXPLAIN ANALYZE`, these statistics will also include the actual time -(in milliseconds) spent, and other runtime statistics (e.g. the actual number of +(in milliseconds) spent, and other runtime statistics (for example, the actual number of produced rows): ```sql @@ -555,7 +555,7 @@ There are two terms commonly used for databases: cardinality, and selectivity. Cardinality refers to the number of unique values in a particular column in a table. -Selectivity is the number of unique values produced by an operation (e.g. an +Selectivity is the number of unique values produced by an operation (for example, an index scan or filter), relative to the total number of rows. The higher the selectivity, the more likely PostgreSQL is able to use an index. @@ -675,8 +675,8 @@ best avoided in most cases, such as: - Sequential scans on large tables - Filters that remove a lot of rows -- Performing a certain step (e.g. an index scan) that requires _a lot_ of - buffers (e.g. more than 512 MB for GitLab.com). +- Performing a certain step that requires _a lot_ of + buffers (for example, an index scan for GitLab.com that requires more than 512 MB). As a general guideline, aim for a query that: diff --git a/doc/development/uploads.md b/doc/development/uploads.md index 9c5686db0f6..6d8b951be83 100644 --- a/doc/development/uploads.md +++ b/doc/development/uploads.md @@ -7,12 +7,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Uploads development documentation [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) has special rules for handling uploads. -To prevent occupying a Ruby process on I/O operations, we process the upload in workhorse, where is cheaper. +We process the upload in Workhorse to prevent occupying a Ruby process on I/O operations and because it is cheaper. This process can also directly upload to object storage. ## The problem description -The following graph explains machine boundaries in a scalable GitLab installation. Without any workhorse optimization in place, we can expect incoming requests to follow the numbers on the arrows. +The following graph explains machine boundaries in a scalable GitLab installation. Without any Workhorse optimization in place, we can expect incoming requests to follow the numbers on the arrows. ```mermaid graph TB @@ -27,10 +27,10 @@ graph TB subgraph "redis cluster" r(persisted redis) end - LB-- 1 -->workhorse + LB-- 1 -->Workhorse subgraph "web or API fleet" - workhorse-- 2 -->rails + Workhorse-- 2 -->rails end rails-- "3 (write files)" -->nfs rails-- "4 (schedule a job)" -->r @@ -63,12 +63,12 @@ graph TB subgraph "redis cluster" r(persisted redis) end - LB-- 1 -->workhorse + LB-- 1 -->Workhorse subgraph "web or API fleet" - workhorse-- "3 (without files)" -->rails + Workhorse-- "3 (without files)" -->rails end - workhorse -- "2 (write files)" -->nfs + Workhorse -- "2 (write files)" -->nfs rails-- "4 (schedule a job)" -->r subgraph sidekiq @@ -120,7 +120,7 @@ We have three kinds of file encoding in our uploads: 1. <i class="fa fa-check-circle"></i> **multipart**: `multipart/form-data` is the most common, a file is encoded as a part of a multipart encoded request. 1. <i class="fa fa-check-circle"></i> **body**: some APIs uploads files as the whole request body. -1. <i class="fa fa-times-circle"></i> **JSON**: some JSON API uploads files as base64 encoded strings. This requires a change to GitLab Workhorse, +1. <i class="fa fa-times-circle"></i> **JSON**: some JSON APIs upload files as base64-encoded strings. This requires a change to GitLab Workhorse, which is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325068). ## Uploading technologies @@ -131,9 +131,9 @@ GitLab supports 3 kinds of uploading technologies, here follows a brief descript ### Rack Multipart upload -This is the default kind of upload, and it's most expensive in terms of resources. +This is the default kind of upload, and it's the most expensive in terms of resources. -In this case, workhorse is unaware of files being uploaded and acts as a regular proxy. +In this case, Workhorse is unaware of files being uploaded and acts as a regular proxy. When a multipart request reaches the rails application, `Rack::Multipart` leaves behind temporary files in `/tmp` and uses valuable Ruby process time to copy files around. @@ -213,7 +213,7 @@ sequenceDiagram This is the more advanced acceleration technique we have in place. -Workhorse asks rails for temporary pre-signed object storage URLs and directly uploads to object storage. +Workhorse asks Rails for temporary pre-signed object storage URLs and directly uploads to object storage. In this setup, an extra Rails route must be implemented in order to handle authorization. Examples of this can be found in: @@ -221,7 +221,7 @@ In this setup, an extra Rails route must be implemented in order to handle autho and [its routes](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/config/routes/git_http.rb#L31-32). - [API endpoints for uploading packages](packages.md#file-uploads). -This falls back to _disk buffered upload_ when `direct_upload` is disabled inside the [object storage setting](../administration/uploads.md#object-storage-settings). +Direct upload falls back to _disk buffered upload_ when `direct_upload` is disabled inside the [object storage setting](../administration/uploads.md#object-storage-settings). The answer to the `/authorize` call contains only a file system path. ```mermaid @@ -275,7 +275,7 @@ sequenceDiagram In this section, we describe how to add a new upload route [accelerated](#uploading-technologies) by Workhorse for [body and multipart](#upload-encodings) encoded uploads. -Uploads routes belong to one of these categories: +Upload routes belong to one of these categories: 1. Rails controllers: uploads handled by Rails controllers. 1. Grape API: uploads handled by a Grape API endpoint. @@ -289,7 +289,7 @@ GraphQL uploads do not support [direct upload](#direct-upload) yet. Depending on For both the Rails controller and Grape API uploads, Workhorse has to be updated in order to get the support for the new upload route. -1. Open an new issue in the [Workhorse tracker](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/new) describing precisely the new upload route: +1. Open a new issue in the [Workhorse tracker](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/new) describing precisely the new upload route: - The route's URL. - The [upload encoding](#upload-encodings). - If possible, provide a dump of the upload request. diff --git a/doc/development/verifying_database_capabilities.md b/doc/development/verifying_database_capabilities.md index c5e854701c2..bda9c68eae5 100644 --- a/doc/development/verifying_database_capabilities.md +++ b/doc/development/verifying_database_capabilities.md @@ -12,13 +12,13 @@ necessary to add database (version) specific behavior. To facilitate this we have the following methods that you can use: -- `Gitlab::Database.main.version`: returns the PostgreSQL version number as a string +- `ApplicationRecord.database.version`: returns the PostgreSQL version number as a string in the format `X.Y.Z`. This allows you to write code such as: ```ruby -if Gitlab::Database.main.version.to_f >= 11.7 +if ApplicationRecord.database.version.to_f >= 11.7 run_really_fast_query else run_fast_query diff --git a/doc/development/workspaces/index.md b/doc/development/workspaces/index.md new file mode 100644 index 00000000000..b36c79e84d7 --- /dev/null +++ b/doc/development/workspaces/index.md @@ -0,0 +1,120 @@ +--- +comments: false +type: index, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +description: "Development Guidelines: learn about workspaces when developing GitLab." +--- + +# Workspaces + +The [Workspaces initiative](../../user/workspace/index.md) focuses on reaching feature parity between +SaaS and self-managed installations. + +## Consolidate groups and projects + +- [Architecture blueprint](../../architecture/blueprints/consolidating_groups_and_projects/index.md) + +One facet of the workspaces initiative is to consolidate groups and projects, +addressing the feature disparity between them. Some features, such as epics, are +only available at the group level. Some features, such as issues, are only available +at the project level. Other features, such as milestones, are available to both groups +and projects. + +We receive many requests to add features either to the group or project level. +Moving features around to different levels is problematic on multiple levels: + +- It requires engineering time to move the features. +- It requires UX overhead to maintain mental models of feature availability. +- It creates redundant code. + +When features are copied from one level (project, group, or instance) to another, +the copies often have small, nuanced differences between them. These nuances cause +extra engineering time when fixes are needed, because the fix must be copied to +several locations. These nuances also create different user experiences when the +feature is used in different places. + +A solution for this problem is to consolidate groups and projects into a single +entity, `namespace`. The work on this solution is split into several phases and +is tracked in [epic 6473](https://gitlab.com/groups/gitlab-org/-/epics/6473). + +### Phase 1 + +- [Phase 1 epic](https://gitlab.com/groups/gitlab-org/-/epics/6697). +- **Goals**: + 1. Ensure every project receives a corresponding record in the `namespaces` + table with `type='Project'`. + 1. For user namespaces, the type changes from `NULL` to `User`. + +We should make sure that projects, and the project namespace, are equivalent: + +- **Create project:** use Rails callbacks to ensure a new project namespace is + created for each project. Project namespace records should contain `created_at` and + `updated_at` attributes equal to the project's `created_at`/`updated_at` attributes. +- **Update project:** use the `after_save` callback in Rails to ensure some + attributes are kept in sync between project and project namespaces. + Read [`project#after_save`](https://gitlab.com/gitlab-org/gitlab/blob/6d26634e864d7b748dda0e283eb2477362263bc3/app/models/project.rb#L101-L101) + for more information. +- **Delete project:** use FKs cascade delete or Rails callbacks to ensure when a `Project` + or its `ProjectNamespace` is removed, its corresponding `ProjectNamespace` or `Project` + is also removed. +- **Transfer project to a different group:** make sure that when a project is transferred, + its corresponding project namespace is transferred to the same group. +- **Transfer group:** make sure when transferring a group that all of its sub-projects, + either direct or through descendant groups, have their corresponding project + namespaces transferred correctly as well. +- **Export or import project** + - **Export project** continues to export only the project, and not its project namespace, + in this phase. The project namespace does not contain any specific information + to export at this point. Eventually we want the project namespace to be exported as well. + - **Import project** creates a new project, so the project namespace is created through + Rails `after_save` callback on the project model. +- **Export or import group:** when importing or exporting a `Group`, projects are not + included in the operation. If that feature is changed to include `Project` when its group is + imported or exported, the logic must include their corresponding project namespaces + in the import or export. + +After ensuring these points, run a database migration to create a `ProjectNamespace` +record for every `Project`. Project namespace records created during the migration +should have `created_at` and `updated_at` attributes set to the migration runtime. +The project namespaces' `created_at` and `updated_at` attributes would not match +their corresponding project's `created_at` and `updated_at` attributes. We want +the different dates to help audit any of the created project namespaces, in case we need it. +After this work completes, we must migrate data as described in +[Backfill `ProjectNamespace` for every Project](https://gitlab.com/gitlab-org/gitlab/-/issues/337100). + +### Phase 2 + +- [Phase 2 epic](https://gitlab.com/groups/gitlab-org/-/epics/6768). +- **Goal**: Make `ProjectNamespace` the front entity to interact with instead of `Project`. + +In this phase: + +- Communicate the changes company-wide at the engineering level. We want to make + engineers aware of the upcoming changes, even though teams are not expected to + collaborate actively until phase 3. +- Raise awareness to avoid regressions, and conflicting or duplicate work that + can be dealt with before phase 3. + +Problems to solve as part of this phase: + +- Routes handling through `ProjectNamespace` rather than `Project`. +- Memberships handling. +- Policies handling. +- Import and export. +- Other interactions between project namespace and project models. + +### Phase 3 + +- [Phase 3 epic](https://gitlab.com/groups/gitlab-org/-/epics/6585). +- **Goal**: Feature parity between the namespace types. + +Phase 3 is when the active migration of features from `Project` to `ProjectNamespace`, +or directly to `Namespace`, happens. + +## Related topics + +- [Consolidating groups and projects](../../architecture/blueprints/consolidating_groups_and_projects/index.md) + architecture documentation +- [Workspace user documentation](../../user/workspace/index.md) |