diff options
Diffstat (limited to 'doc/architecture')
39 files changed, 2545 insertions, 167 deletions
diff --git a/doc/architecture/blueprints/_template.md b/doc/architecture/blueprints/_template.md index 798d51a97ad..e39c2b51a5b 100644 --- a/doc/architecture/blueprints/_template.md +++ b/doc/architecture/blueprints/_template.md @@ -1,6 +1,6 @@ --- status: proposed -creation-date: yyyy-mm-dd +creation-date: "yyyy-mm-dd" authors: [ "@username" ] coach: "@username" approvers: [ "@product-manager", "@engineering-manager" ] @@ -10,7 +10,8 @@ participating-stages: [] <!-- **Note:** Please remove comment blocks for sections you've filled in. -When your blueprint is complete, all of these comment blocks should be removed. +When your blueprint ready for review, all of these comment blocks should be +removed. To get started with a blueprint you can use this template to inform you about what you may want to document in it at the beginning. This content will change diff --git a/doc/architecture/blueprints/ci_data_decay/index.md b/doc/architecture/blueprints/ci_data_decay/index.md index b7c3bdde2f8..6df37e28992 100644 --- a/doc/architecture/blueprints/ci_data_decay/index.md +++ b/doc/architecture/blueprints/ci_data_decay/index.md @@ -23,7 +23,7 @@ builds [continues to grow exponentially](../ci_scale/index.md). GitLab CI/CD has come a long way since the initial release, but the design of the data storage for pipeline builds remains almost the same since 2012. In 2021 we started working on database decomposition and extracting CI/CD data to -ia separate database. Now we want to improve the architecture of GitLab CI/CD +a separate database. Now we want to improve the architecture of GitLab CI/CD product to enable further scaling. ## Goals @@ -78,7 +78,7 @@ pipeline processing in such pipeline. It means that all the metadata, we store in PostgreSQL, that is needed to efficiently and reliably process builds can be safely moved to a different data store. -Currently, storing pipeline processing data is expensive as this kind of CI/CD +Storing pipeline processing data is expensive as this kind of CI/CD data represents a significant portion of data stored in CI/CD tables. Once we restrict access to processing archived pipelines, we can move this metadata to a different place - preferably object storage - and make it accessible on diff --git a/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md index d61412ae1ed..261390d1d14 100644 --- a/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md +++ b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md @@ -75,7 +75,7 @@ incidents, over the last couple of months, for example: - S2: 2022-04-12 [Transactions detected that have been running for more than 10m](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6821) - S2: 2022-04-06 [Database contention plausibly caused by excessive `ci_builds` reads](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6773) - S2: 2022-03-18 [Unable to remove a foreign key on `ci_builds`](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6642) -- S2: 2022-10-10 [The queuing_queries_duration SLI apdex violating SLO](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7852#note_1130123525) +- S2: 2022-10-10 [The `queuing_queries_duration` SLI apdex violating SLO](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/7852#note_1130123525) We have approximately 50 `ci_*` prefixed database tables, and some of them would benefit from partitioning. @@ -180,6 +180,49 @@ respective database tables. Using `RANGE` partitioning works similarly to using of `partition_id` values, using `RANGE` partitioning might be a better strategy. +### Multi-project pipelines + +Parent-child pipeline will always be part of the same partition because child +pipelines are considered a resource of the parent pipeline. They can't be +viewed individually in the project pipeline list page. + +On the other hand, multi-project pipelines can be viewed in the pipeline list page. +They can also be accessed from the pipeline graph as downstream/upstream links +when created via the `trigger` token or the API using a job token. +They can also be created from other pipelines by using trigger tokens, but in this +case we don't store the source pipeline. + +While partitioning `ci_builds` we need to update the foreign keys to the +`ci_sources_pipelines` table: + +```plain +Foreign-key constraints: + "fk_be5624bf37" FOREIGN KEY (source_job_id) REFERENCES ci_builds(id) ON DELETE CASCADE + "fk_d4e29af7d7" FOREIGN KEY (source_pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE + "fk_e1bad85861" FOREIGN KEY (pipeline_id) REFERENCES ci_pipelines(id) ON DELETE CASCADE +``` + +A `ci_sources_pipelines` record references two `ci_pipelines` rows (parent and +the child). Our usual strategy has been to add a `partition_id` to the +table, but if we do it here we will force all multi-project +pipelines to be part of the same partition. + +We should add two `partition_id` columns for this table, a +`partition_id` and a `source_partition_id`: + +```plain +Foreign-key constraints: + "fk_be5624bf37" FOREIGN KEY (source_job_id, source_partition_id) REFERENCES ci_builds(id, source_partition_id) ON DELETE CASCADE + "fk_d4e29af7d7" FOREIGN KEY (source_pipeline_id, source_partition_id) REFERENCES ci_pipelines(id, source_partition_id) ON DELETE CASCADE + "fk_e1bad85861" FOREIGN KEY (pipeline_id, partition_id) REFERENCES ci_pipelines(id, partition_id) ON DELETE CASCADE +``` + +This solution is the closest to a two way door decision because: + +- We retain the ability to reference pipelines in different partitions. +- If we later decide that we want to force multi-project pipelines in the same partition + we could add a constraint to validate that both columns have the same value. + ## Why do we want to use explicit logical partition ids? Partitioning CI/CD data using a logical `partition_id` has several benefits. We @@ -248,10 +291,9 @@ smart enough to move rows between partitions on its own. A partitioned table is called a **routing** table and it will use the `p_` prefix which should help us with building automated tooling for query analysis. -A table partition will be called **partition** and it can use the a -physical partition ID as suffix, leaded by a `p` letter, for example -`ci_builds_p101`. Existing CI tables will become **zero partitions** of the -new routing tables. Depending on the chosen +A table partition will be called **partition** and it can use the a physical +partition ID as suffix, for example `ci_builds_101`. Existing CI tables will +become **zero partitions** of the new routing tables. Depending on the chosen [partitioning strategy](#how-do-we-want-to-partition-cicd-data) for a given table, it is possible to have many logical partitions per one physical partition. @@ -273,6 +315,37 @@ during a low traffic period([after `00:00 UTC`](https://dashboards.gitlab.net/d/ See an example of this strategy in our [partition tooling](../../../development/database/table_partitioning.md#step-6---create-parent-table-and-attach-existing-table-as-the-initial-partition)). +### Partitioning steps + +The database [partition tooling](../../../development/database/table_partitioning.md#partitioning-a-table-list) +docs contain a list of steps to partition a table, but the steps are not enough +for our iterative strategy. As our dataset continues to grow we want to take +advantage of partitioning performance right away and not wait until all tables +are partitioned. For example, after partitioning the `ci_builds_metadata` table +we want to start writing and reading data to/from a new partition. This means +that we will increase the `partition_id` value from `100`, the default value, +to `101`. Now all of the new resources for the pipeline hierarchy will be +persisted with `partition_id = 101`. We can continue following the database +tooling instructions for the next table that will be partitioned, but we require +a few extra steps: + +- add `partition_id` column for the FK references with default value of `100` + since the majority of records should have that value. +- change application logic to cascade the `partition_id` value +- correct `partition_id` values for recent records with a post deploy/background + migration, similar to this: + + ```sql + UPDATE ci_pipeline_metadata + SET partition_id = ci_pipelines.partition_id + FROM ci_pipelines + WHERE ci_pipelines.id = ci_pipeline_metadata.pipeline_id + AND ci_pipelines.partition_id in (101, 102); + ``` + +- change the foreign key definitions +- ... + ## Storing partitions metadata in the database To build an efficient mechanism that will be responsible for creating @@ -297,7 +370,7 @@ system - any letter from `g` to `z` in Latin alphabet, for example `x`. In that case an example of an URI would look like `1e240x5ba0`. If we decide to update the primary identifier of a partitioned resource (today it is just a big integer) it is important to design a system that is resilient to migrating data -between partitions, to avoid changing idenfiers when rebalancing happens. +between partitions, to avoid changing identifiers when rebalancing happens. `ci_partitions` table will store information about a partition identifier, pipeline ids range it is valid for and whether the partitions have been diff --git a/doc/architecture/blueprints/ci_pipeline_components/index.md b/doc/architecture/blueprints/ci_pipeline_components/index.md index a3c72227f3e..100c9e67fda 100644 --- a/doc/architecture/blueprints/ci_pipeline_components/index.md +++ b/doc/architecture/blueprints/ci_pipeline_components/index.md @@ -3,7 +3,7 @@ status: proposed creation-date: "2022-09-14" authors: [ "@fabio", "@grzesiek" ] coach: "@kamil" -approvers: [ "@dov" ] +approvers: [ "@dhershkovitch", "@marknuzzo" ] owning-stage: "~devops::verify" participating-stages: [] --- @@ -110,7 +110,8 @@ identifying abstract concepts and are subject to changes as we refine the design ## Definition of pipeline component -A pipeline component is a reusable single-purpose building block that abstracts away a single pipeline configuration unit. Components are used to compose a part or entire pipeline configuration. +A pipeline component is a reusable single-purpose building block that abstracts away a single pipeline configuration unit. +Components are used to compose a part or entire pipeline configuration. It can optionally take input parameters and set output data to be adaptable and reusable in different pipeline contexts, while encapsulating and isolating implementation details. @@ -133,27 +134,319 @@ For best experience with any systems made of components it's fundamental that co The version identifies the exact interface and behavior of the component. - **Resolvable**: when a component depends on another component, this dependency must be explicit and trackable. -## Proposal +## Structure of a component -Prerequisites to create a component: +A pipeline component is identified by the path to a repository or directory that defines it +and a specific version: `<component-path>@<version>`. -- Create a project. Description and avatar are highly recommended to improve discoverability. -- Add a `README.md` in the top level directory that documents the component. - What it does, how to use it, how to contribute, etc. - This file is mandatory. -- Add a `.gitlab-ci.yml` in the top level directory to test that the components works as expected. - This file is highly recommended. +For example: `gitlab-org/dast@1.0`. -Characteristics of a component: +### The component path -- It must have a **name** to be referenced to and **description** for extra details. -- It must specify its **type** which defines how it can be used (raw configuration to be `include`d, child pipeline workflow, job step). -- It must define its **content** based on the type. -- It must specify **input parameters** that it accepts. Components should depend on input parameters for dynamic values and not environment variables. -- It can optionally define **output data** that it returns. -- Its YAML specification should be **validated statically** (for example: using JSON schema validators). -- It should be possible to use specific **versions** of a component by referencing official releases and SHA. -- It should be possible to use components defined locally in the same repository. +A component path must contain at least the metadata YAML and optionally a related `README.md` documentation file. + +The component path can be: + +- A path to a project: `gitlab-org/dast`. In this case the 2 files are defined in the root directory of the repository. +- A path to a project subdirectory: `gitlab-org/dast/api-scan`. In this case the 2 files are defined in the `api-scan` directory. +- A path to a local directory: `/path/to/component`. This path must contain the metadata YAML that defines the component. + The path must start with `/` to indicate a full path in the repository. + +The metadata YAML file follows the filename convention `gitlab-<component-type>.yml` where component type is one of: + +| Component type | Context | +| -------------- | ------- | +| `template` | For components used under `include:` keyword | +| `step` | For components used under `steps:` keyword | +| `workflow` | For components used under `trigger:` keyword | + +Based on the context where the component is used we fetch the correct YAML file. +For example, if we are including a component `gitlab-org/dast@1.0` we expect a YAML file named `gitlab-template.yml` in the +top level directory of `gitlab-org/dast` repository. + +A `gitlab-<component-type>.yml` file: + +- Must have a **name** to be referenced to and **description** for extra details. +- Must specify its **type** in the filename, which defines how it can be used (raw configuration to be `include`d, child pipeline workflow, job step). +- Must define its **content** based on the type. +- Must specify **input parameters** that it accepts. Components should depend on input parameters for dynamic values and not environment variables. +- Can optionally define **output data** that it returns. +- Should be **validated statically** (for example: using JSON schema validators). + +```yaml +spec: + inputs: + website: + environment: + default: test + test_run: + options: + - unit + - integration + - system +content: { ... } +``` + +Components that are released in the catalog must have a `README.md` file in the same directory as the +metadata YAML file. The `README.md` represents the documentation for the specific component, hence it's recommended +even when not releasing versions in the catalog. + +### The component version + +The version of the component can be (in order of highest priority first): + +1. A commit SHA - For example: `gitlab-org/dast@e3262fdd0914fa823210cdb79a8c421e2cef79d8` +1. A released tag - For example: `gitlab-org/dast@1.0` +1. A special moving target version that points to the most recent released tag - For example: `gitlab-org/dast@~latest` +1. An unreleased tag - For example: `gitlab-org/dast@rc-1.0` +1. A branch name - For example: `gitlab-org/dast@master` + +If a tag and branch exist with the same name, the tag takes precedence over the branch. +Similarly, if a tag is named `e3262fdd0914fa823210cdb79a8c421e2cef79d8`, a commit SHA (if exists) +takes precedence over the tag. + +As we want to be able to reference any revisions (even those not released), a component must be defined in a Git repository. + +NOTE: +When referencing a component by local path (for example `./path/to/component`), its version is implicit and matches +the commit SHA of the current pipeline context. + +## Components project + +A components project is a GitLab project/repository that exclusively hosts one or more pipeline components. + +For components projects it's highly recommended to set an appropriate avatar and project description +to improve discoverability in the catalog. + +### Structure of a components project + +A project can host one or more components depending on whether the author wants to define a single component +per project or include multiple cohesive components under the same project. + +Let's imagine we are developing a component that runs RSpec tests for a Rails app. We create a component project +called `myorg/rails-rspec`. + +The following directory structure would support 1 component per project: + +```plaintext +. +├── gitlab-<type>.yml +├── README.md +└── .gitlab-ci.yml +``` + +The `.gitlab-ci.yml` is recommended for the project to ensure changes are verified accordingly. + +The component is now identified by the path `myorg/rails-rspec`. In other words, this means that +the `gitlab-<type>.yml` and `README.md` are located in the root directory of the repository. + +The following directory structure would support multiple components per project: + +```plaintext +. +├── .gitlab-ci.yml +├── unit/ +│ ├── gitlab-workflow.yml +│ └── README.md +├── integration/ +│ ├── gitlab-workflow.yml +│ └── README.md +└── feature/ + ├── gitlab-workflow.yml + └── README.md +``` + +In this example we are defining multiple test profiles that are executed with RSpec. +The user could choose to use one or more of these. + +Each of these components are identified by their path `myorg/rails-rspec/unit`, `myorg/rails-rspec/integration` +and `myorg/rails-rspec/feature`. + +This directory structure could also support both strategies: + +```plaintext +. +├── gitlab-template.yml # myorg/rails-rspec +├── README.md +├── .gitlab-ci.yml +├── unit/ +│ ├── gitlab-workflow.yml # myorg/rails-rspec/unit +│ └── README.md +├── integration/ +│ ├── gitlab-workflow.yml # myorg/rails-rspec/integration +│ └── README.md +└── feature/ + ├── gitlab-workflow.yml # myorg/rails-rspec/feature + └── README.md +``` + +With the above structure we could have a top-level component that can be used as the +default component. For example, `myorg/rails-rspec` could run all the test profiles together. +However, more specific test profiles could be used separately (for example `myorg/rails-rspec/integration`). + +NOTE: +Any nesting more than 1 level is initially not permitted. +This limitation encourages cohesion at project level and keeps complexity low. + +## Input parameters `spec:inputs:` parameters + +If the component takes any input parameters they must be specified according to the following schema: + +```yaml +spec: + inputs: + website: # by default all declared inputs are mandatory. + environment: + default: test # apply default if not provided. This makes the input optional. + test_run: + options: # a choice must be made from the list since there is no default value. + - unit + - integration + - system +``` + +When using the component we pass the input parameters as follows: + +```yaml +include: + - component: org/my-component@1.0 + with: + website: ${MY_WEBSITE} # variables expansion + test_run: system + environment: $[[ inputs.environment ]] # interpolation of upstream inputs +``` + +Variables expansion must be supported for `with:` syntax as well as interpolation of +possible [inputs provided upstream](#input-parameters-for-pipelines). + +Input parameters are validated as soon as possible: + +1. Read the file `gitlab-template.yml` inside `org/my-component`. +1. Parse `spec:inputs` and validate the parameters against this schema. +1. If successfully validated, proceed with parsing `content:`. Return an error otherwise. +1. Interpolate input parameters inside the component's `content:`. + +```yaml +spec: + inputs: + environment: + options: [test, staging, production] +content: + "run-tests-$[[ inputs.environment ]]": + script: ./run-test + + scan-website: + script: ./scan-website $[[ inputs.environment ]] + rules: + - if: $[[ inputs.environment ]] == 'staging' + - if: $[[ inputs.environment ]] == 'production' +``` + +With `$[[ inputs.XXX ]]` inputs are interpolated immediately after parsing the `content:`. + +### Why input parameters and not environment variables? + +Until today we have been leveraging environment variables to pass information around. +For example, we use environment variables to pass information from an upstream pipeline to a +downstream pipeline. + +Using environment variables for passing information to a component is like declaring global +variables in programming languages. The more variables we declare the more we risk variable +conflicts and increase variables scope. + +Input parameters are like variables passed to the component which exist inside a specific +scope and they don't leak to the outside. +Inputs are not inherited from upstream `include`s. They must be passed explicitly. + +This paradigm allows to build more robust and isolated components as well as declare and +enforce contracts. + +### Input parameters for existing `include:` syntax + +Because we are adding input parameters to components used via `include:component` we have an opportunity to +extend it to other `include:` types support inputs via `with:` syntax: + +```yaml +include: + - component: org/my-component@1.0 + with: + foo: bar + - local: path/to/file.yml + with: + foo: bar + - project: org/another + file: .gitlab-ci.yml + with: + foo: bar + - remote: http://example.com/ci/config + with: + foo: bar + - template: Auto-DevOps.gitlab-ci.yml + with: + foo: bar +``` + +Then the configuration being included must specify the inputs: + +```yaml +spec: + inputs: + foo: + +# rest of the configuration +``` + +If a YAML includes content using `with:` but the including YAML doesn't specify `inputs:`, an error should be raised. + +|`with:`| `inputs:` | result | +| --- | --- | --- | +| specified | | raise error | +| specified | specified | validate inputs | +| | specified | use defaults | +| | | legacy `include:` without input passing | + +### Input parameters for pipelines + +Inputs can also be used to pass parameters to a pipeline when triggered and benefit from immediate validation. + +Today we have different use cases where using explicit input parameters would be beneficial: + +1. `Run Pipeline` UI form. + - **Problem today**: We are using top-level variables with `variables:*:description` to surface environment variables to the UI. + The problem with this is the mix of responsibilities as well as the jump in [precedence](../../../ci/variables/index.md#cicd-variable-precedence) + that a variable gets (from a YAML variable to a pipeline variable). + Building validation and features on top of this solution is challenging and complex. +1. Trigger a pipeline via API. For example `POST /projects/:id/pipelines/trigger` with `{ inputs: { provider: 'aws' } }` +1. Trigger a pipeline via `trigger:` syntax. + +```yaml +deploy-app: + trigger: + project: org/deployer + with: + provider: aws + deploy_environment: staging +``` + +To solve the problem of `Run Pipeline` UI form we could fully leverage the `spec:inputs` schema: + +```yaml +spec: + inputs: + concurrency: + default: 10 # displayed as default value in the input box + provider: # can enforce `required` in the form validation + description: Deployment provider # optional: render as input label. + deploy_environment: + options: # render a selectbox with options in order of how they are defined below + - staging # 1st option + - canary # 2nd option + - production # 3rd option + default: staging # selected by default in the UI. + # if `default:` is not specified, the user must explicitly select + # an option. + description: Deployment environment # optional: render as input label. +``` ## Limits @@ -188,3 +481,34 @@ Some limits we could consider adding: - Allow self-managed administrators to populate their self-managed catalog by importing/updating components from GitLab.com or from repository exports. - Iterate on feedback. + +## Who + +Proposal: + +<!-- vale gitlab.Spelling = NO --> + +| Role | Who +|--------------------------------|-------------------------| +| Author | Fabio Pitino | +| Engineering Leaders | Cheryl Li, Mark Nuzzo | +| Product Manager | Dov Hershkovitch | +| Architecture Evolution Coaches | Kamil Trzciński, Grzegorz Bizon | + +DRIs: + +| Role | Who +|------------------------------|------------------------| +| Leadership | Mark Nuzzo | +| Product | Dov Hershkovitch | +| Engineering | Fabio Pitino | +| UX | Kevin Comoli (interim), Sunjung Park | + +Domain experts: + +| Area | Who +|------------------------------|------------------------| +| Verify / Pipeline authoring | Avielle Wolfe | +| Verify / Pipeline authoring | Laura Montemayor-Rodriguez | + +<!-- vale gitlab.Spelling = YES --> diff --git a/doc/architecture/blueprints/ci_scale/index.md b/doc/architecture/blueprints/ci_scale/index.md index c02fb35974b..574a79c86a5 100644 --- a/doc/architecture/blueprints/ci_scale/index.md +++ b/doc/architecture/blueprints/ci_scale/index.md @@ -1,8 +1,10 @@ --- -stage: none -group: unassigned -comments: false -description: 'Improve scalability of GitLab CI/CD' +status: in progress +creation-date: "2021-01-21" +authors: [ "@grzesiek" ] +coach: "@grzesiek" +approvers: [ "@cheryl.li", "@jreporter" ] +owning-stage: "~devops::verify" --- # CI/CD Scaling diff --git a/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md index 53f38fa85fd..3ef98c33035 100644 --- a/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md +++ b/doc/architecture/blueprints/composable_codebase_using_rails_engines/index.md @@ -74,7 +74,7 @@ This blueprint explicitly talks about **horizontal** split and **Application Lay The Bounded Contexts is a topic that was discussed extensively number of times for a couple of years. Reflected in number of issues: -- [Create new models / classes within a module / namespace](https://gitlab.com/gitlab-org/gitlab/-/issues/212156) +- [Create new models / classes in a module / namespace](https://gitlab.com/gitlab-org/gitlab/-/issues/212156) - [Make teams to be maintainers of their code](https://gitlab.com/gitlab-org/gitlab/-/issues/25872) - [Use nested structure to organize CI classes](https://gitlab.com/gitlab-org/gitlab/-/issues/209745) - [WIP: Make it simple to build and use "Decoupled Services"](https://gitlab.com/gitlab-org/gitlab/-/issues/31121) @@ -86,7 +86,7 @@ We are partially executing a **Bounded Contexts** idea: - Since we use namespaces, individual contributor or reviewer can know who to reach from domain experts about help with the given context -The module namespaces are actively being used today to model codebase around team boundaries. Currently, the most +The module namespaces are actively being used today to model codebase around team boundaries. The most prominent namespaces being used today are `Ci::` and `Packages::`. They provide a good way to contain the code owned by a group in a well-defined structure. @@ -125,7 +125,7 @@ application layers. This list is not exhaustive, but shows a general list of the - Web Packages API: provide a REST API compatible with the packaging tools: Debian, Maven, Container Registry Proxy, etc. - Git nodes: all code required to authorize `git pull/push` over `SSH` or `HTTPS` - Sidekiq: run background jobs -- Services/Models/DB: all code required to maintain our database structure, data validation, business logic and policies models that needs to be shared with other components +- Services/Models/DB: all code required to maintain our database structure, data validation, business logic, and policies models that needs to be shared with other components The best way to likely describe how the actual GitLab Rails split would look like. It is a satellite model. Where we have a single core, that is shared across all satellite components. The design of that implies @@ -301,7 +301,7 @@ All work can be found in these merge requests: - [Draft: PoC - Move GraphQL to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180) - [Draft: PoC - Move Controllers and Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53720) - [Draft: PoC - Move only Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53982) -- [Measure performance impact for proposed web_engine](https://gitlab.com/gitlab-org/gitlab/-/issues/300548) +- [Measure performance impact for proposed `web_engine`](https://gitlab.com/gitlab-org/gitlab/-/issues/300548) What was done? @@ -328,8 +328,8 @@ What was done? 1. Move gems to the `engines/web_engine/` - - We moved all GraphQL gems to the actual web_engine Gemfile - - We moved Grape API gem to the actual web_engine Gemfile + - We moved all GraphQL gems to the actual `web_engine` Gemfile + - We moved Grape API gem to the actual `web_engine` Gemfile ```ruby Gem::Specification.new do |spec| @@ -344,9 +344,9 @@ What was done? 1. Move routes to the `engines/web_engine/config/routes.rb` file - - We moved GraphQL routes to the web_engine routes. - - We moved API routes to the web_engine routes. - - We moved most of the controller routes to the web_engine routes. + - We moved GraphQL routes to the `web_engine` routes. + - We moved API routes to the `web_engine` routes. + - We moved most of the controller routes to the `web_engine` routes. ```ruby Rails.application.routes.draw do @@ -367,7 +367,7 @@ What was done? 1. Connect GitLab application with the WebEngine - In GitLab Gemfile.rb, add web_engine to the engines group + In GitLab Gemfile.rb, add `web_engine` to the engines group ```ruby # Gemfile @@ -376,7 +376,7 @@ What was done? end ``` - Since the gem is inside :engines group, it will not be automatically required by default. + Since the gem is inside :engines group, it is not automatically required by default. 1. Configure GitLab when to load the engine. @@ -432,7 +432,7 @@ What was done? - We control specs from main application using environment variable `TEST_WEB_ENGINE` - We added new CI job that will run `engines/web_engine/spec` tests separately using `TEST_WEB_ENGINE` environment variable. - We added new CI job that will run `engines/web_engine/ee/spec` tests separately using `TEST_WEB_ENGINE` environment variable. - - We are running all whitebox frontend tests with `TEST_WEB_ENGINE=true` + - We are running all white box frontend tests with `TEST_WEB_ENGINE=true` #### Results @@ -451,7 +451,7 @@ Savings on Sidekiq `start-up` event, for a single Sidekiq cluster without GraphQ - Boot-up time was reduced from 45.31 to 21.80 seconds. It was 23.51 seconds faster (51.89%) - We have 805,772 less live objects, 4,587,535 less allocated objects, 2,866 less allocated pages and 3.65 MB less allocated space for objects outside of the heap - We loaded 2,326 less code files (15.64%) -- We reduced the duration of a single full GC cycle from 0.80s to 0.70 (12.64%) +- We reduced the duration of a single full GC cycle from 0.80 seconds to 0.70 seconds (12.64%) Puma single, showed very little difference as expected. @@ -461,20 +461,20 @@ More details can be found in the [issue](https://gitlab.com/gitlab-org/gitlab/-/ Estimating the results for the scale of running GitLab.com, today we use: -- Currently individual GC cycle takes around [130ms for Web](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22web%22%7D%5B5m%5D)%2Frate(ruby_gc_duration_seconds_count%5B5m%5D))&g0.tab=0) - and [200ms for Sidekiq](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22sidekiq%22%7D%5B5m%5D)%2Frate(ruby_gc_duration_seconds_count%5B5m%5D))&g0.tab=0) on GitLab.com +- Individual GC cycle takes around [130 ms for Web](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22web%22%7D%5B5m%5D)%2Frate(ruby_gc_duration_seconds_count%5B5m%5D))&g0.tab=0) + and [200 ms for Sidekiq](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22sidekiq%22%7D%5B5m%5D)%2Frate(ruby_gc_duration_seconds_count%5B5m%5D))&g0.tab=0) on GitLab.com - On average we do around [2 GC cycles per-second](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.end_input=2021-02-17%2017%3A56&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_count%7Bstage%3D%22main%22%2Ctype%3D%22web%22%7D%5B5m%5D))&g0.tab=0) or [0.12 cycles per second for Sidekiq](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.end_input=2021-02-17%2017%3A56&g0.max_source_resolution=0s&g0.expr=avg(rate(ruby_gc_duration_seconds_count%7Bstage%3D%22main%22%2Ctype%3D%22sidekiq%22%7D%5B5m%5D))&g0.tab=0) - This translates to using [around 9.5 vCPUs per-second for Web](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22web%22%7D%5B5m%5D))&g0.tab=0) and [around 8 vCPUs per-second for Sidekiq](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(rate(ruby_gc_duration_seconds_sum%7Bstage%3D%22main%22%2Ctype%3D%22sidekiq%22%7D%5B5m%5D))&g0.tab=0) of spend on GC alone -- Sidekiq [uses 2.1GB on average](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=max(ruby_process_unique_memory_bytes%7Btype%3D%22sidekiq%22%7D)%2F1024%2F1024%2F1024&g0.tab=1) - or [550GB in total](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_unique_memory_bytes%7Btype%3D%22sidekiq%22%7D)%2F1024%2F1024%2F1024&g0.tab=0) of memory on GitLab.com +- Sidekiq [uses 2.1 GB on average](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=max(ruby_process_unique_memory_bytes%7Btype%3D%22sidekiq%22%7D)%2F1024%2F1024%2F1024&g0.tab=1) + or [550 GB in total](https://thanos-query.ops.gitlab.net/graph?g0.range_input=1h&g0.max_source_resolution=0s&g0.expr=sum(ruby_process_unique_memory_bytes%7Btype%3D%22sidekiq%22%7D)%2F1024%2F1024%2F1024&g0.tab=0) of memory on GitLab.com We estimate the possible maximum savings for introducing `web_engine`: -- Reduce a GC cycle time by 20%, from to 200ms to 160ms +- Reduce a GC cycle time by 20%, from to 200 ms to 160 ms - The amount of GC cycles per-second would stay the same, but due to GC cycle time reduction we would use around 6 vCPUs instead of 8 vCPUs -- In the best case we would be looking at Sidekiq alone we would be estimating to save up-to 137GB of memory on GitLab.com +- In the best case we would be looking at Sidekiq alone we would be estimating to save up-to 137 GB of memory on GitLab.com This model could be extended to introduce `sidekiq_engine` giving a similar benefits (even more important due to visible impact on users) for Web nodes. @@ -498,14 +498,14 @@ Cons: - It is harder to implement GraphQL subscriptions as in case of Sidekiq as we need another way to pass subscriptions - `api_v4` paths can be used in some services that are used by Sidekiq (for example `api_v4_projects_path`) -- url_helpers paths are used in models and services, that could be used by Sidekiq (for example `Gitlab::Routing.url_helpers.project_pipelines_path` is used by [ExpirePipelineCacheService](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/expire_pipeline_cache_service.rb#L20) in [ExpirePipelineCacheWorker](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/expire_pipeline_cache_worker.rb#L18)) +- `url_helpers` paths are used in models and services, that could be used by Sidekiq (for example `Gitlab::Routing.url_helpers.project_pipelines_path` is used by [ExpirePipelineCacheService](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/services/ci/expire_pipeline_cache_service.rb#L20) in [ExpirePipelineCacheWorker](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/expire_pipeline_cache_worker.rb#L18)) #### Example: GraphQL [Draft: PoC - Move GraphQL to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180) - The [99% of changes](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180/diffs?commit_id=49c9881c6696eb620dccac71532a3173f5702ea8) as visible in the above MRs is moving files as-is. -- The [actual work](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180/diffs?commit_id=1d9a9edfa29ea6638e7d8a6712ddf09f5be77a44) on fixing cross-dependencies, specs, and configuring web_engine +- The [actual work](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180/diffs?commit_id=1d9a9edfa29ea6638e7d8a6712ddf09f5be77a44) on fixing cross-dependencies, specs, and configuring `web_engine` - We [adapted](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180/diffs?commit_id=d7f862cc209ce242000b2aec88ff7f4485acdd92) CI to test `engines/web_engine/` as a self-sufficient component of stack Today, loading GraphQL requires a bunch of [dependencies](https://gitlab.com/gitlab-org/gitlab/-/issues/288044): @@ -581,7 +581,7 @@ to be created to ensure that we do not have explosion of engines. - [Draft: PoC - Move GraphQL to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50180) - [Draft: PoC - Move Controllers and Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53720) - [Draft: PoC - Move only Grape API:API to the WebEngine](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53982) -- [Measure performance impact for proposed web_engine](https://gitlab.com/gitlab-org/gitlab/-/issues/300548) +- [Measure performance impact for proposed `web_engine`](https://gitlab.com/gitlab-org/gitlab/-/issues/300548) - [Create new models / classes within a module / namespace](https://gitlab.com/gitlab-org/gitlab/-/issues/212156) - [Make teams to be maintainers of their code](https://gitlab.com/gitlab-org/gitlab/-/issues/25872) - [Use nested structure to organize CI classes](https://gitlab.com/gitlab-org/gitlab/-/issues/209745) diff --git a/doc/architecture/blueprints/container_registry_metadata_database/index.md b/doc/architecture/blueprints/container_registry_metadata_database/index.md index 63e27286756..f3bcf1e4e59 100644 --- a/doc/architecture/blueprints/container_registry_metadata_database/index.md +++ b/doc/architecture/blueprints/container_registry_metadata_database/index.md @@ -258,7 +258,7 @@ The expected registry behavior will be covered with integration tests, using a p If unable to pull a connection from the pool to serve a given request, the registry will timeout and return an HTTP `500 Internal Server Error` error to the client and report the error to Sentry. These issues should trigger a development escalation to investigate why the pool is being exhausted. There might be too much load for the preconfigured pool size, or there might be transactions holding on to connections for too long. -Prometheus metrics should be used to create alerts to act on a possible saturation before the application starts erroring. Special attention will be paid to these scenarios during the gradual migration of the GitLab.com registry, where we will have limited, gradual, and controlled exposure on the new registry nodes. During that process, we can identify usage patterns, observe metrics, and fine tune both infrastructure and application settings accordingly as the load increases. If needed, a rate limiting algorithm may be applied to limit impact. Decisions will be based on real data to avoid overly restrictive measures and premature optimizations. +Prometheus metrics should be used to create alerts to act on a possible saturation before the application starts returning errors. Special attention will be paid to these scenarios during the gradual migration of the GitLab.com registry, where we will have limited, gradual, and controlled exposure on the new registry nodes. During that process, we can identify usage patterns, observe metrics, and fine tune both infrastructure and application settings accordingly as the load increases. If needed, a rate limiting algorithm may be applied to limit impact. Decisions will be based on real data to avoid overly restrictive measures and premature optimizations. The expected registry behavior will be covered with integration tests by manipulating the pool size and spawning multiple concurrent requests against the API, putting pressure on the pool and eventually exhausting its capacity. @@ -294,7 +294,7 @@ Together, these resources should provide an adequate level of insight into the r GitLab ships with the GitLab Container Registry by default, but it's also compatible with third-party registries, as long as they comply with the [Docker Distribution V2 Specification](https://docs.docker.com/registry/spec/api/), now superseded by the [Open Container Initiative (OCI) Image Specification](https://github.com/opencontainers/image-spec/blob/master/spec.md). -So far, we strived to maintain full compatibility with third-party registries when adding new features. For example, in 12.8, we introduced a new [tag delete feature](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/23325) to delete a single tag without deleting the underlying manifest. Because this feature is not part of the Docker or OCI specifications, we have kept the previous behavior as a fallback option to maintain compatibility with third-party registries. +So far, we have tried to maintain full compatibility with third-party registries when adding new features. For example, in 12.8, we introduced a new [tag delete feature](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/23325) to delete a single tag without deleting the underlying manifest. Because this feature is not part of the Docker or OCI specifications, we have kept the previous behavior as a fallback option to maintain compatibility with third-party registries. However, this will likely change in the future. Apart from online garbage collection, and as described in [challenges](#challenges), the metadata database will unblock the implementation of many requested features for the GitLab Container Registry in the mid/long term. Most of these features will only be available for instances using the GitLab Container Registry. They are not part of the Docker Distribution or OCI specifications, neither we will be able to provide a compatible fallback option. @@ -316,7 +316,7 @@ All GitLab Rails features dependent on a specific version of the registry should This is already done to determine whether a tag should be deleted using the new tag delete feature (only available in the GitLab Container Registry v2.8.1+) or the old method. In this case, GitLab Rails sends an `OPTIONS` request to the registry tag route to determine whether the `DELETE` method is supported or not. -Alternatively, and as the universal long-term solution, we need to determine the registry vendor, version, and supported features (the last two are only applicable if the vendor is GitLab) and persist it in the GitLab Rails database. This information can then be used in realtime to toggle features or fallback to alternative methods, if possible. The initial implementation of this approach was introduced as part of [#204839](https://gitlab.com/gitlab-org/gitlab/-/issues/204839). Currently, it's only used for metrics purposes. Further improvements are required to guarantee that the version information is kept up to date in self-managed instances, where the registry may be hot swapped. +Alternatively, and as the universal long-term solution, we need to determine the registry vendor, version, and supported features (the last two are only applicable if the vendor is GitLab) and persist it in the GitLab Rails database. This information can then be used in real time to toggle features or fallback to alternative methods, if possible. The initial implementation of this approach was introduced as part of [#204839](https://gitlab.com/gitlab-org/gitlab/-/issues/204839). Currently, it's only used for metrics purposes. Further improvements are required to guarantee that the version information is kept up to date in self-managed instances, where the registry may be hot swapped. ##### Release and Deployment diff --git a/doc/architecture/blueprints/database/scalability/patterns/time_decay.md b/doc/architecture/blueprints/database/scalability/patterns/time_decay.md index 93f5dffd3f5..2b36a43a6db 100644 --- a/doc/architecture/blueprints/database/scalability/patterns/time_decay.md +++ b/doc/architecture/blueprints/database/scalability/patterns/time_decay.md @@ -154,7 +154,7 @@ factors: The perfect partitioning scheme keeps **all queries over a dataset almost always over a single partition**, with some cases going over two partitions and seldom over multiple partitions being an acceptable balance. We should also target for **partitions that are as small as possible**, below -5-10M records and/or 10GB each maximum. +5-10M records and/or 10 GB each maximum. Partitioning can be combined with other strategies to either prune (drop) old partitions, move them to cheaper storage inside the database or move them outside of the database (archive or use of other @@ -241,7 +241,7 @@ Related epic: [Partitioning: `web_hook_logs` table](https://gitlab.com/groups/gi The important characteristics of `web_hook_logs` are the following: 1. Size of the dataset: it is a really large table. At the moment we decided to - partition it (`2021-03-01`), it had roughly 527M records and a total size of roughly 1TB + partition it (`2021-03-01`), it had roughly 527M records and a total size of roughly 1 TB - Table: `web_hook_logs` - Rows: approximately 527M @@ -261,7 +261,7 @@ As a result, on March 2021 there were still not deleted records since July 2020 increasing in size by more than 2 million records per day instead of staying at a more or less stable size. -Finally, the rate of inserts has grown to more than 170GB of data per month by March 2021 and keeps +Finally, the rate of inserts has grown to more than 170 GB of data per month by March 2021 and keeps on growing, so the only viable solution to pruning old data was through partitioning. Our approach was to partition the table per month as it aligned with the 90 days retention policy. diff --git a/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md b/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md new file mode 100644 index 00000000000..a6efe68310e --- /dev/null +++ b/doc/architecture/blueprints/gitlab_observability_backend/metrics/index.md @@ -0,0 +1,688 @@ +--- +status: proposed +creation-date: "2022-11-09" +authors: [ "@ankitbhatnagar" ] +coach: "@mappelman" +approvers: [ "@sebastienpahl", "@nicholasklick" ] +owning-stage: "~monitor::observability" +participating-stages: [] +--- + +# GitLab Observability Backend - Metrics + +## Summary + +Developing a multi-user system to store & query observability data typically formatted in widely accepted, industry-standard formats using Clickhouse as underlying storage, with support for long-term data retention and aggregation. + +## Motivation + +From the six pillars of Observability, commonly abbreviated as `TEMPLE` - Traces, Events, Metrics, Profiles, Logs & Errors, Metrics constitute one of the most important pillars of observability data for modern day systems, helping their users gather insights about their operational posture. + +Metrics which are commonly structured as timeseries data have the following characteristics: + +- indexed by their corresponding timestamps; +- continuously expanding in size; +- usually aggregated, down-sampled, and queried in ranges; and +- have very write-intensive requirements. + +Within GitLab Observability Backend, we aim to add the support for our customers to ingest and query observability data around their systems & applications, helping them improve the operational health of their systems. + +### Goals + +With the development of the proposed system, we have the following goals: + +- Scalable, low latency & cost-effective monitoring system backed by Clickhouse whose performance has been proven via repeatable benchmarks. + +- Support for long-term storage for Prometheus/OpenTelemetry formatted metrics, ingested via Prometheus remote_write API and queried via Prometheus remote_read API, PromQL or SQL with support for metadata and exemplars. + +The aformentioned goals can further be broken down into the following four sub-goals: + +#### Ingesting data + +- For the system to be capable of ingesting large volumes of writes and reads, we aim to ensure that it must be horizontally scalable & provide durability guarantees to ensure no writes are dropped once ingested. + +#### Persisting data + +- We aim to support ingesting telemetry/data sent using Prometheus `remote_write` protocol. Any persistence we design for our dataset must be multi-tenant by default, ensuring we can store observability data for multiple tenants/groups/projects within the same storage backend. + +- We aim to develop a test suite for data correctness, seeking inspiration from how Prometheus compliance test suite checks the correctness of a given Metrics implementation and running it as a part of our CI setup. + +NOTE: +Although remote_write_sender does not test the correctness of a remote write receiver itself as is our case, it does bring some inspiration to implement/develop one within the scope of this project. + +- We aim to also ensure compatibility for special Prometheus data types, e.g. Prometheus histogram(s), summary(s). + +#### Reading data + +- We aim to support querying data using PromQL which means translating PromQL queries into Clickhouse SQL. To do this, [PromQL](https://github.com/prometheus/prometheus/tree/main/promql/parser) or [MetricsQL](https://github.com/VictoriaMetrics/metricsql) parsers are good alternatives. + +- We aim to provide additional value by exposing all ingested data via the native Clickhouse SQL interface subject to the following reliability characteristics: + - query validation, sanitation + - rate limiting + - resource limiting - memory, cpu, network bandwidth + +- We aim to pass Prometheus test suits for correctness via the [Prometheus Compliance test suite](https://github.com/prometheus/compliance/tree/main/promql) with a target goal of 100% success rate. + +#### Deleting data + +- We aim to support being able to delete any ingested data should such a need arise. This is also in addition to us naturally deleting data when a configured TTL expires and/or respective retention policies are enforced. We must, within our schemas, build a way to delete data by labels OR their content, also add to our offering the necessary tooling to do so. + +### Non-Goals + +With the goals established above, we also want to establish what specific things are non-goals with the current proposal. They are: + +- We do not aim to support ingestion using OpenTelemetry/OpenMetrics formats with our first iteration, though our users can still use the Opentelemetry exporters(s) internally consuming the standard Prometheus `remote_write` protocol. More information [here](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/prometheusremotewriteexporter). + +- We do not aim to support ingesting Prometheus exemplars in our first iteration, though we do aim to account for them in our design from the beginning. + +NOTE: +Worth noting that we intend to model exemplars the same way we’re modeling metric-labels, so building on top of the same data structure should help implementt support for metadata/exemplars rather easily. + +## Proposal + +We intend to use GitLab Observability Backend as a framework for the Metrics implementation so that its lifecycle is also managed via already existing Kubernetes controllers e.g. scheduler, tenant-operator. + + + +From a development perspective, what’s been marked as our “Application Server” above needs to be developed as a part of this proposal while the remaining peripheral components either already exist or can be provisioned via existing code in `scheduler`/`tenant-operator`. + +**On the write path**, we expect to receive incoming data via `HTTP`/`gRPC` `Ingress` similar to what we do for our existing services, e.g. errortracking, tracing. + +NOTE: +Additionally, since we intend to ingest data via Prometheus `remote_write` API, the received data will be Protobuf-encoded, Snappy-compressed. All received data therefore needs to be decompressed & decoded to turn it into a set of `prompb.TimeSeries` objects, which the rest of our components interact with. + +We also need to make sure to avoid writing a lot of small writes into Clickhouse, therefore it’d be prudent to batch data before writing it into Clickhouse. + +We must also make sure ingestion remains decoupled with `Storage` so as to reduce undue dependence on a given storage implementation. While we do intend to use Clickhouse as our backing storage for any foreseeable future, this ensures we do not tie ourselves in into Clickhouse too much should future business requirements warrant the usage of a different backend/technology. A good way to implement this in Golang would be our implementations adhering to a standard interface, the following for example: + +```golang +type Storage interface { + Read( + ctx context.Context, + request *prompb.ReadRequest + ) (*prompb.ReadResponse, error) + Write( + ctx context.Context, + request *prompb.WriteRequest + ) error +} +``` + +NOTE: +We understand this couples the implementation with Prometheus data format/request types, but adding methods to the interface to support more data formats should be trivial looking forward with minimal changes to code. + +**On the read path**, we aim to allow our users to use the Prometheus `remote_read` API and be able to query ingested data via PromQL & SQL. Support for `remote_read` API should be trivial to implement, while supporting PromQL would need translating it into SQL. We can however employ the usage of already existing [PromQL](https://github.com/prometheus/prometheus/tree/main/promql/parser) parsing libraries. + +We aim to focus on implementing query validation & sanitation, rate-limiting and regulating resource-consumption to ensure underlying systems, esp. storage, remain in good operational health at all times. + +### Supported deployments + +In this first iteration of the metrics backend, we intend to support a generic deployment model that makes sure we can capture as much usage as possible and begin dogfooding the product as soon as possible. This is well illustrated in the [aforementioned architecture diagram](#proposal). + +In its most vanilla form, metrics support in GitLab Observability Backend can be used via the Prometheus remote read & write APIs. If a user already uses Prometheus as their monitoring abstraction, it can be configured to use this backend directly. + +- remote_write: [configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_write) +- remote_read: [configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#remote_read) + +For users of the system that do not use a Prometheus instance for scraping their telemetry data, they can export their metrics via a multitude of collectors/agents such as the OpenTelemetry collector or the Prometheus Agent for example, all of which can be configured to use our remote_write endpoint. For reads however, we intend to run a Prometheus within GOB (alongside the application server) itself, then hook it up automatically with the GitLab Observability UI (GOUI) preconfigured to consume our remote_read endpoint. + +Notably, the ability to use a GOB-run Prometheus instance is applicable while we can only support remote_read API for running queries. Looking forward towards our next iteration, we should be able to get rid of this additional component altogether when we have full support for executing PromQL and/or SQL queries directly from GOUI. + +**Per-group deployments**: From a scalability perspective, we deploy an instance of Ingress, a Prometheus instance & the application server per group to make sure we can scale them subject to traffic volumes of the respective tenant. It also helps isolate resource consumption across tenants in an otherwise multi-tenant system. + +### Metric collection and storage + +It is important to separate metric collection on the client side with the storage we provision at our end. + +### State of the art for storage + +Existing long-term Prometheus compatible metrics vendors provide APIs that are compatible with Prometheus remote_write. + +### State of the art for Prometheus clients + +Metric collection clients such as Prometheus itself, Grafana Cloud Agent, Datadog Agent, etc. will scrape metrics endpoints typically from within a firewalled environment, store locally scraped metrics in a [Write Ahead Log (WAL)](https://en.wikipedia.org/wiki/Write-ahead_logging) and then batch send them to an external environment (i.e. the vendor or an internally managed system like Thanos) via the Prometheus `remote_write` protocol. + +- A client-side collector is an important part of the overall architecture, though it's owned by the customer/user since it needs to run in their environment. This gives the end user full control over their data because they control how it is collected and to where it is delivered. + +- It's **not** feasible to provide an external vendor with credentials to access and scrape endpoints within a user's firewalled environment. + +- It's also critically important that our `remote_write` APIs respond correctly with the appropriate rate-limiting status codes so that Prometheus Clients can respect them. + +[Here](https://grafana.com/blog/2021/05/26/the-future-of-prometheus-remote-write/) is a good background/history on Prometheus `remote_write` and its importance in Prometheus based observability. + +## Design and implementation details + +Following are details of how we aim to design & implement the proposed solution. To that end, a reference implementation was also developed to understand the scope of the problem and provide early data to ensure our proposal was drafted around informed decisions and/or results of our experimentation. + +## Reference implementation(s) + +- [Application server](https://gitlab.com/gitlab-org/opstrace/opstrace/-/merge_requests/1823) +- [Metrics generator](https://gitlab.com/ankitbhatnagar/metrics-gen/-/blob/main/main.go) + +## Target environments + +Keeping inline with our current operational structure, we intend to deploy the metrics offering as a part of GitLab Observability Backend, deployed on the following two target environments: + +- kind cluster (for local development) +- GKE cluster (for staging/production environments) + +## Schema Design + +### **Proposed solution**: Fully normalized tables for decreased redundancy & increased read performance + +### primary, denormalized data table + +```sql +CREATE TABLE IF NOT EXISTS samples ON CLUSTER '{cluster}' ( + series_id UUID, + timestamp DateTime64(3, ‘UTC’) CODEC(Delta(4), ZSTD), + value Float64 CODEC(Gorilla, ZSTD) +) ENGINE = ReplicatedMergeTree() +PARTITION BY toYYYYMMDD(timestamp) +ORDER BY (series_id, timestamp) +``` + +### metadata table to support timeseries metadata/exemplars + +```sql +CREATE TABLE IF NOT EXISTS samples_metadata ON CLUSTER '{cluster}' ( + series_id UUID, + timestamp DateTime64(3, ‘UTC’) CODEC(Delta(4), ZSTD), + metadata Map(String, String) CODEC(ZSTD), +) ENGINE = ReplicatedMergeTree() +PARTITION BY toYYYYMMDD(timestamp) +ORDER BY (series_id, timestamp) +``` + +### lookup table(s) + +```sql +CREATE TABLE IF NOT EXISTS labels_to_series ON CLUSTER '{cluster}' ( + labels Map(String, String) CODEC(ZSTD) + series_id UUID +) engine=ReplicatedMergeTree +PRIMARY KEY (labels, series_id) +``` + +```sql +CREATE TABLE IF NOT EXISTS group_to_series ON CLUSTER ‘{cluster}’ ( + group_id Uint64, + series_id UUID, +) ORDER BY (group_id, series_id) +``` + +### Refinements + +- sharding considerations for a given tenant when ingesting/persisting data if we intend to co-locate data specific to multiple tenants within the same database tables. To simplify things, segregating tenant-specific data to their own dedicated set of tables would make a lot of sense. + +- structural considerations for “timestamps” when ingesting data across tenants. + +- creation_time vs ingestion_time + +- No support for transactions in the native client yet, to be able to effectively manage writes across multiple tables. + +NOTE: +Slightly non-trivial but we can potentially investigate the possibility of using ClickHouse/ch-go directly, it supposedly promises a better performance profile too. + +### Pros - multiple tables + +- Normalised data structuring allows for efficient storage of data, removing any redundancy across multiple samples for a given timeseries. Evidently, for the “samples” schema, we expect to store 32 bytes of data per metric point. + +- Better search complexity when filtering timeseries by labels/metadata, via the use of better indexed columns. + +- All data is identifiable via a unique identifier, which can be used to maintain data consistency across tables. + +### Cons - multiple tables + +- Writes are trivially expensive considering writes across multiple tables. + +- Writes across tables also need to be implemented as a transaction to guarantee consistency when ingesting data. + +### Operational characteristics - multiple tables + +### Storage - multiple tables + +A major portion of our writes are made into the `samples` schema which contains a tuple containing three data points per metric point written: +|column|data type|byte size| +|---|---|---| +|series_id|UUID|16 bytes| +|timestamp|DateTime64|8 bytes| +|value|Float64|8 bytes| + +Therefore, we estimate to use 32 bytes per sample ingested. + +### Compression - multiple tables + +Inspecting the amount of compression we’re able to get with the given design on our major schemas, we see it as a good starting point. Following measurements for both primary tables: + +**Schema**: `labels_to_series` containing close to 12k unique `series_id`, each mapping to a set of 10-12 label string pairs + +```sql +SELECT + table, + column, + formatReadableSize(sum(data_compressed_bytes) AS x) AS compressedsize, + formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed +FROM system.parts_columns +WHERE table LIKE 'labels_to_series_1' +GROUP BY + database, + table, + column +ORDER BY x ASC + +Query id: 723b4145-14f7-4e74-9ada-01c17c2f1fd5 + +┌─table──────────────┬─column────┬─compressedsize─┬─uncompressed─┐ +│ labels_to_series_1 │ labels │ 586.66 KiB │ 2.42 MiB │ +│ labels_to_series_1 │ series_id │ 586.66 KiB │ 2.42 MiB │ +└────────────────────┴───────────┴────────────────┴──────────────┘ +``` + +**Schema**: `samples` containing about 20k metric samples each containing a tuple comprising `series_id` (16 bytes), `timestamp` (8 bytes) and `value` (8 bytes). + +```sql +SELECT + table, + column, + formatReadableSize(sum(data_compressed_bytes) AS x) AS compressedsize, + formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed +FROM system.parts_columns +WHERE table LIKE 'samples_1' +GROUP BY + database, + table, + column +ORDER BY x ASC + +Query id: 04219cea-06ea-4c5f-9287-23cb23c023d2 + +┌─table─────┬─column────┬─compressedsize─┬─uncompressed─┐ +│ samples_1 │ value │ 373.21 KiB │ 709.78 KiB │ +│ samples_1 │ timestamp │ 373.21 KiB │ 709.78 KiB │ +│ samples_1 │ series_id │ 373.21 KiB │ 709.78 KiB │ +└───────────┴───────────┴────────────────┴──────────────┘ +``` + +### Performance - multiple tables + +From profiling our reference implementation, it can also be noted that most of our time right now is spent in the application writing data to Clickhouse and/or its related operations. A “top” pprof profile sampled from the implementation looked like: + +```shell +(pprof) top +Showing nodes accounting for 42253.20kB, 100% of 42253.20kB total +Showing top 10 nodes out of 58 + flat flat% sum% cum cum% +13630.30kB 32.26% 32.26% 13630.30kB 32.26% github.com/ClickHouse/clickhouse-go/v2/lib/compress.NewWriter (inline) +11880.92kB 28.12% 60.38% 11880.92kB 28.12% github.com/ClickHouse/clickhouse-go/v2/lib/compress.NewReader (inline) + 5921.37kB 14.01% 74.39% 5921.37kB 14.01% bufio.NewReaderSize (inline) + 5921.37kB 14.01% 88.41% 5921.37kB 14.01% bufio.NewWriterSize (inline) + 1537.69kB 3.64% 92.04% 1537.69kB 3.64% runtime.allocm + 1040.73kB 2.46% 94.51% 1040.73kB 2.46% github.com/aws/aws-sdk-go/aws/endpoints.init + 1024.41kB 2.42% 96.93% 1024.41kB 2.42% runtime.malg + 768.26kB 1.82% 98.75% 768.26kB 1.82% go.uber.org/zap/zapcore.newCounters + 528.17kB 1.25% 100% 528.17kB 1.25% regexp.(*bitState).reset + 0 0% 100% 5927.73kB 14.03% github.com/ClickHouse/clickhouse-go/v2.(*clickhouse).Ping +``` + +As is evident above from our preliminary analysis, writing data into Clickhouse can be a potential bottleneck. Therefore, on the write path, it'd be prudent to batch our writes into Clickhouse so as to reduce the amount of work the application server ends up doing making the ingestion path more efficient. + +On the read path, it’s also possible to parallelize reads for the samples table either by series_id(s) OR by blocks of time between the queried start and end timestamps. + +### Caveats + +- When dropping labels from already existing metrics, we treat their new counterparts as completely new series and hence attribute them to a new series_id. This avoids having to merge series data and/or values. The old series, if not actively written into, should eventually fall off their retention and get deleted. + +- We have not yet accounted for any data aggregation. Our assumption is that the backing store (in Clickhouse) should allow us to keep a “sufficient” amount of data in its raw form and that we should be able to query against it within our query latency SLOs. + +### **Rejected alternative**: Single, centralized table + +### single, centralized data table + +```sql +CREATE TABLE IF NOT EXISTS metrics ON CLUSTER ‘{cluster}’ ( + group_id UInt64, + name LowCardinality(String) CODEC(ZSTD), + labels Map(String, String) CODEC(ZSTD), + metadata Map(String, String) CODEC(ZSTD), + value Float64 CODEC (Gorilla, ZSTD), + timestamp DateTime64(3, ‘UTC’) CODEC(Delta(4),ZSTD) +) ENGINE = ReplicatedMergeTree() +PARTITION BY toYYYYMMDD(timestamp) +ORDER BY (group_id, name, timestamp); +``` + +### Pros - single table + +- Single source of truth, so all metrics data lives in one big table. + +- Querying data is easier to express in terms of writing SQL queries without having to query data across multiple tables. + +### Cons - single table + +- Huge redundancy built into the data structure since attributes such as name, labels, metadata are stored repeatedly for each sample collected. + +- Non-trivial complexity to search timeseries with values for labels/metadata given how they’re stored when backed by Maps/Arrays. + +- High query latencies by virtue of having to scan large amounts of data per query made. + +### Operational Characteristics - single table + +### Storage - single table + +|column|data type|byte size| +|---|---|---| +|group_id|UUID|16 bytes| +|name|String|-| +|labels|Map(String, String)|-| +|metadata|Map(String, String)|-| +|value|Float64|8 bytes| +|timestamp|DateTime64|8 bytes| + +NOTE: +Strings are of an arbitrary length, the length is not limited. Their value can contain an arbitrary set of bytes, including null bytes. We will need to regulate what we write into these columns application side. + +### Compression - single table + +**Schema**: `metrics` containing about 20k metric samples each consisting of a `group_id`, `metric name`, `labels`, `metadata`, `timestamp` & corresponding `value`. + +```sql +SELECT count(*) +FROM metrics_1 + +Query id: e580f20b-b422-4d93-bb1f-eb1435761604 + +┌─count()─┐ +│ 12144 │ + + +SELECT + table, + column, + formatReadableSize(sum(data_compressed_bytes) AS x) AS compressedsize, + formatReadableSize(sum(data_uncompressed_bytes)) AS uncompressed +FROM system.parts_columns +WHERE table LIKE 'metrics_1' +GROUP BY + database, + table, + column +ORDER BY x ASC + +Query id: b2677493-3fbc-46c1-a9a7-4524a7a86cb4 + +┌─table─────┬─column────┬─compressedsize─┬─uncompressed─┐ +│ metrics_1 │ labels │ 283.02 MiB │ 1.66 GiB │ +│ metrics_1 │ metadata │ 283.02 MiB │ 1.66 GiB │ +│ metrics_1 │ group_id │ 283.02 MiB │ 1.66 GiB │ +│ metrics_1 │ value │ 283.02 MiB │ 1.66 GiB │ +│ metrics_1 │ name │ 283.02 MiB │ 1.66 GiB │ +│ metrics_1 │ timestamp │ 283.02 MiB │ 1.66 GiB │ +└───────────┴───────────┴────────────────┴──────────────┘ +``` + +Though we see a good compression factor for the aforementioned schema, the amount of storage needed to store the corresponding dataset is approximately 300MiB. We also expect to see this footprint increase linearly given the redundancy baked into the schema design itself, also one of the reasons we intend **not** to proceed with this design further. + +### Performance - single table + +```shell +(pprof) top +Showing nodes accounting for 12844.95kB, 100% of 12844.95kB total +Showing top 10 nodes out of 40 + flat flat% sum% cum cum% + 2562.81kB 19.95% 19.95% 2562.81kB 19.95% runtime.allocm + 2561.90kB 19.94% 39.90% 2561.90kB 19.94% github.com/aws/aws-sdk-go/aws/endpoints.init + 2374.91kB 18.49% 58.39% 2374.91kB 18.49% github.com/ClickHouse/clickhouse-go/v2/lib/compress.NewReader (inline) + 1696.32kB 13.21% 71.59% 1696.32kB 13.21% bufio.NewWriterSize (inline) + 1184.27kB 9.22% 80.81% 1184.27kB 9.22% bufio.NewReaderSize (inline) + 1184.27kB 9.22% 90.03% 1184.27kB 9.22% github.com/ClickHouse/clickhouse-go/v2/lib/compress.NewWriter (inline) + 768.26kB 5.98% 96.01% 768.26kB 5.98% go.uber.org/zap/zapcore.newCounters + 512.20kB 3.99% 100% 512.20kB 3.99% runtime.malg + 0 0% 100% 6439.78kB 50.13% github.com/ClickHouse/clickhouse-go/v2.(*clickhouse).Ping + 0 0% 100% 6439.78kB 50.13% github.com/ClickHouse/clickhouse-go/v2.(*clickhouse).acquire +``` + +Writes against this schema perform much better in terms of compute, given it's concentrated on one table and does not need looking up `series_id` from a side table. + +### General storage considerations - Clickhouse + +The following sections intend to deep-dive into specific characteristics of our schema design and/or their interaction with Clickhouse - the database system. + +- table engines + + - [MergeTree](https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree/) + - [S3 Table Engine](https://clickhouse.com/docs/en/engines/table-engines/integrations/s3/) + +- efficient partitioning and/or sharding + + - Configuring our schemas with the right partitioning keys so as to have the least amount of blocks scanned when reading back the data. + - Sharding here would refer to how we design our data placement strategy to make sure the cluster remains optimally balanced at all times. + +- data compression + +As is visible from the aforementioned preliminary results, we see good compression results with dictionary and delta encoding for strings and floats respectively. When storing labels with a `Map` of `LowCardinality(String)`s, we were able to pack data efficiently. + +- materialized views + +Can be updated dynamically as the need be, help make read paths performant + +- async inserts + +- batch inserts + +- retention/TTLs + +We should only store data for a predetermined period of time, post which we either delete data, aggregate it or ship it to an archival store to reduce operational costs of having to store data for longer periods of time. + +- data aggregation/rollups + +- index granularity + +- skip indexes + +- `max_server_memory_usage_to_ram_ratio` + +### Data access via SQL + +While our corpus of data is PromQL-queryable, it would be prudent to make sure we make the SQL interface “generally available” as well. This capability opens up multiple possibilities to query resident data and allows our users to slice and dice their datasets whichever way they prefer to and/or need to. + +#### Challenges + +- Resource/cost profiling. +- Query validation and sanitation. + +### Illustrative example(s) of data access + +### Writes + +On the write path, we first ensure registering a given set labels to a unique `series_id` and/or re-using one should we have seen the timeseries already in the past. For example: + +```plaintext +redis{region="us-east-1",'os':'Ubuntu15.10',...} <TIMESTAMP> <VALUE> +``` + +**Schema**: labels_to_series + +```sql +SELECT * +FROM labels_to_series_1 +WHERE series_id = '6d926ae8-c3c3-420e-a9e2-d91aff3ac125' +FORMAT Vertical + +Query id: dcbc4bd8-0bdb-4c35-823a-3874096aab6e + +Row 1: +────── +labels: {'arch':'x64','service':'1','__name__':'redis','region':'us-east-1','os':'Ubuntu15.10','team':'LON','service_environment':'production','rack':'36','service_version':'0','measurement':'pubsub_patterns','hostname':'host_32','datacenter':'us-east-1a'} +series_id: 6d926ae8-c3c3-420e-a9e2-d91aff3ac125 + +1 row in set. Elapsed: 0.612 sec. +``` + +Post which, we register each metric point in the `samples` table attributing it to the corresponding `series_id`. + +**Schema**: samples + +```sql +SELECT * +FROM samples_1 +WHERE series_id = '6d926ae8-c3c3-420e-a9e2-d91aff3ac125' +LIMIT 1 +FORMAT Vertical + +Query id: f3b410af-d831-4859-8828-31c89c0385b5 + +Row 1: +────── +series_id: 6d926ae8-c3c3-420e-a9e2-d91aff3ac125 +timestamp: 2022-11-10 12:59:14.939 +value: 0 +``` + +### Reads + +On the read path, we first query all timeseries identifiers by searching for the labels under consideration. Once we have all the `series_id`(s), we then look up all corresponding samples between the query start timestamp and end timestamp. + +For e.g. + +```plaintext +kernel{service_environment=~"prod.*", measurement="boot_time"} +``` + +which gets translated into first looking for all related timeseries: + +```sql +SELECT * +FROM labels_to_series +WHERE +((labels['__name__']) = 'kernel') AND +match(labels['service_environment'], 'prod.*') AND +((labels['measurement']) = 'boot_time'); +``` + +yielding a bunch of `series_id`(s) corresponding to the labels just looked up. + +**Sidenote**, this mostly-static dataset can also be cached and built up in-memory gradually to reduce paying the latency cost the second time, which should reduce the number of lookups considerably. + +To account for newer writes when maintaining this cache: + +- Have an out-of-band process/goroutine maintain this cache, so even if a few queries miss the most recent data, subsequent ones eventually catch up. + +- Have TTLs on the keys, jittered per key so as to rebuild them frequently enough to account for new writes. + +Once we know which timeseries we’re querying for, from there, we can easily look up all samples via the following query: + +```sql +SELECT * +FROM samples +WHERE series_id IN ( + 'a12544be-0a3a-4693-86b0-c61a4553aea3', + 'abd42fc4-74c7-4d80-9b6c-12f673db375d', + … +) +AND timestamp >= '1667546789' +AND timestamp <= '1667633189' +ORDER BY timestamp; +``` + +yielding all timeseries samples we were interested in. + +We then render these into an array of `prometheus.QueryResult` object(s) and return back to the caller as a `prometheus.ReadResponse` object. + +NOTE: +The queries have been broken down into multiple queries only during our early experimentation/iteration, it’d be prudent to use subqueries within the same roundtrip to the database going forward into production/benchmarking. + +## Production Readiness + +### Batching + +Considering we’ll need to batch data before ingesting large volumes of small writes into Clickhouse, the design must account for app-local persistence to allow it to locally batch incoming data before landing it into Clickhouse in batches of a predetermined size in order to increase performance and allow the table engine to continue to persist data successfully. + +We have considered the following alternatives to implement app-local batching: + +- In-memory - non durable +- BadgerDB - durable, embedded, performant +- Redis - trivial, external dependency +- Kafka - non-trivial, external dependency but it can augment multiple other use-cases and help other problem domains at GitLab. + +**Note**: Similar challenges have also surfaced with the CH interactions `errortracking` - the subsystem has in its current implementation. There have been multiple attempts to solve this problem domain in the past - [this MR](https://gitlab.com/gitlab-org/opstrace/opstrace/-/merge_requests/1660) implemented an in-memory alternative while [this one](https://gitlab.com/gitlab-org/opstrace/opstrace/-/merge_requests/1767) attempted an on-disk alternative. + +Any work done in this area of concern would also benefit other subsystems such as errortracking, logging, etc. + +### Scalability + +We intend to start testing the proposed implementation with 10K metric-points per second to test/establish our initial hypothesis, though ideally, we must design the underlying backend for 1M points ingested per second. + +### Benchmarking + +We propose the following three dimensions be tested while benchmarking the proposed implementation: + +- Data ingest performance +- On-disk storage requirements (accounting for replication if applicable) +- Mean query response times + +For understanding performance, we’ll need to first compile a list of such queries given the data we ingest for our tests. Clickhouse query logging is super helpful while doing this. + +NOTE: +Ideally, we aim to benchmark the system to be able to ingest >1M metric points/sec while consistently serving most queries under <1 sec. + +### Past work & references + +- [Benchmark ClickHouse for metrics](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1666) +- [Incubation:APM ClickHouse evaluation](https://gitlab.com/gitlab-org/incubation-engineering/apm/apm/-/issues/4) +- [Incubation:APM ClickHouse metrics schema](https://gitlab.com/gitlab-org/incubation-engineering/apm/apm/-/issues/10) +- [Our research around TimescaleDB](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/14137) +- [Current Workload on our Thanos-based setup](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/15420#current-workload) +- [Scaling-200m-series](https://opstrace.com/blog/scaling-200m-series) + +### Cost-estimation + +- We aim to make sure the system's not too expensive, especially given our biggest footprint is on Clickhouse and the underlying storage. + +- We must consider the usage of multiple storage medium(s), especially: + - Tiered storage + - Object storage + +### Tooling + +- We aim to building visibility into high cardinality metrics to be able to assist with keeping our databases healthy by pruning/dropping unused metrics. + +- Similarly, we aim to develop the ability to see unused metrics for the end-user, which can be easily & dynamically built into the system by parsing all read requests and building usage statistics. + +- We aim to add monitoring for per-metric scrape frequencies to make sure the end-user is not ingesting data at a volume they do not need and/or find useful. + +## Looking forward + +### Linkage across telemetry pillars, exemplars + +We must build the metrics system in a way to be able cross-reference ingested data with other telemetry pillars, such as traces, logs and errors, so as to provide a more holistic view of all instrumentation a system sends our way. + +### User-defined SQL queries to aggregate data and/or generate materialized views + +We should allow users of the system to be able to run user-defined, ad-hoc queries similar to how Prometheus recording rules help generate custom metrics from existing ones. + +### Write Ahead Logs (WALs) + +We believe that should we feel the need to start buffering data local to the ingestion application and/or move away from Clickhouse for persisting data, on-disk WALs would be a good direction to proceed into given their prevelant usage among other monitoring system. + +### Custom DSLs or query builders + +Using PromQL directly could be a steep learning curve for users. It would be really nice to have a query builder (as is common in Grafana) to allow building of the typical queries you'd expect to run and to allow exploration of the available metrics. It also serves as a way to learn the DSL, so more complex queries can be created later. + +## Roadmap & Next Steps + +The following section enlists how we intend to implement the aforementioned proposal around building Metrics support into GitLab Observability Service. Each corresponding document and/or issue contains further details of how each next step is planned to be executed. + +- **[DONE]** [Research & draft design proposal and/or requirements](https://docs.google.com/document/d/1kHyIoWEcs14sh3CGfKGiI8QbCsdfIHeYkzVstenpsdE/edit?usp=sharing) +- **[IN-PROGRESS]** [Submit system/schema designs (proposal) & gather feedback](https://docs.google.com/document/d/1kHyIoWEcs14sh3CGfKGiI8QbCsdfIHeYkzVstenpsdE/edit?usp=sharing) +- **[IN-PROGRESS]** [Develop table definitions and/or storage interfaces](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1666) +- **[IN-PROGRESS]** [Prototype reference implementation, instrument key metrics](https://gitlab.com/gitlab-org/opstrace/opstrace/-/merge_requests/1823) +- [Benchmark Clickhouse and/or proposed schemas, gather expert advice from Clickhouse Inc.](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1666) +- Develop write path(s) - `remote_write` API +- Develop read path(s) - `remote_read` API, `PromQL`-based querier. +- Setup testbed(s) for repeatable benchmarking/testing +- Schema design and/or application server improvements if needed +- Production Readiness v1.0-alpha/beta +- Implement vanguarded/staged rollouts +- Run extended alpha/beta testing +- Release v1.0 diff --git a/doc/architecture/blueprints/gitlab_observability_backend/metrics/supported-deployments.png b/doc/architecture/blueprints/gitlab_observability_backend/metrics/supported-deployments.png Binary files differnew file mode 100644 index 00000000000..54c4ed3b48f --- /dev/null +++ b/doc/architecture/blueprints/gitlab_observability_backend/metrics/supported-deployments.png diff --git a/doc/architecture/blueprints/image_resizing/index.md b/doc/architecture/blueprints/image_resizing/index.md index 948378d8834..99a7c4ae144 100644 --- a/doc/architecture/blueprints/image_resizing/index.md +++ b/doc/architecture/blueprints/image_resizing/index.md @@ -14,7 +14,7 @@ Currently, we are showing all uploaded images 1:1, which is of course not ideal. ## MVC Avatar Resizing -When implementing a dynamic image resizing solution, images should be resized and optimized on the fly so that if we define new targeted sizes later we can add them dynamically. This would mean a huge improvement in performance as some of the measurements suggest that we can save up to 95% of our current load size. Our initial investigations indicate that we have uploaded approximately 1.65 million avatars totaling approximately 80GB in size and averaging approximately 48kb each. Early measurements indicate we can reduce the most common avatar dimensions to between 1-3kb in size, netting us a greater than 90% size reduction. For the MVC we don't consider application level caching and rely purely on HTTP based caches as implemented in CDNs and browsers, but might revisit this decision later on. To easily mitigate performance issues with avatar resizing, especially in the case of self managed, an operations feature flag is implemented to disable dynamic image resizing. +When implementing a dynamic image resizing solution, images should be resized and optimized on the fly so that if we define new targeted sizes later we can add them dynamically. This would mean a huge improvement in performance as some of the measurements suggest that we can save up to 95% of our current load size. Our initial investigations indicate that we have uploaded approximately 1.65 million avatars totaling approximately 80 GB in size and averaging approximately 48 KB each. Early measurements indicate we can reduce the most common avatar dimensions to between 1-3 KB in size, netting us a greater than 90% size reduction. For the MVC we don't consider application level caching and rely purely on HTTP based caches as implemented in CDNs and browsers, but might revisit this decision later on. To mitigate performance issues with avatar resizing, especially in the case of self managed, an operations feature flag is implemented to disable dynamic image resizing. ```mermaid sequenceDiagram @@ -38,10 +38,10 @@ sequenceDiagram Content image resizing is a more complex problem to tackle. There are no set size restrictions and there are additional features or requirements to consider. - Dynamic WebP support - the WebP format typically achieves an average of 30% more compression than JPEG without the loss of image quality. More details are in [this Google Comparative Study](https://developers.google.com/speed/webp/docs/c_study) -- Extract first image of GIF's so we can prevent from loading 10MB pixels +- Extract first GIF image so we can prevent from loading 10 MB pixels - Check Device Pixel Ratio to deliver nice images on High DPI screens - Progressive image loading, similar to what is described in [this article about how to build a progressive image loader](https://www.sitepoint.com/how-to-build-your-own-progressive-image-loader/) -- Resizing recommendations (size, clarity, and so on) +- Resizing recommendations (for example, size and clarity) - Storage The MVC Avatar resizing implementation is integrated into Workhorse. With the extra requirements for content image resizing, this may require further use of GraphicsMagik (GM) or a similar library and breaking it out of Workhorse. diff --git a/doc/architecture/blueprints/object_storage/index.md b/doc/architecture/blueprints/object_storage/index.md index 61dc37d7706..950a5f13c38 100644 --- a/doc/architecture/blueprints/object_storage/index.md +++ b/doc/architecture/blueprints/object_storage/index.md @@ -62,7 +62,7 @@ This has led to increased complexity across the board, from development that would normally be "free". - In many cases, we copy around object storage files needlessly (for example, [issue #285597](https://gitlab.com/gitlab-org/gitlab/-/issues/285597)). - Large files (LFS, packages, and so on) are slow to finalize or don't work + Large files (for example, LFS and packages) are slow to finalize or don't work at all as a result. ## Improvements over the current situation @@ -113,7 +113,7 @@ Because every group of features requires its own bucket, we don't have direct upload enabled everywhere. Contributing a new upload requires coding it in both Ruby on Rails and Go. -Implementing a new feature that does not yet have a dedicated bucket +Implementing a new feature that does not have a dedicated bucket requires the developer to also create a merge request in Omnibus and CNG, as well as coordinate with SREs to configure the new bucket for our own environments. @@ -138,7 +138,7 @@ access to new features without infrastructure chores. Our implementation is built on top of a 3rd-party framework where every object storage client is a 3rd-party library. Unfortunately some of them are unmaintained. -[We have customers who cannot push 5GB Git LFS objects](https://gitlab.com/gitlab-org/gitlab/-/issues/216442), +[We have customers who cannot push 5 GB Git LFS objects](https://gitlab.com/gitlab-org/gitlab/-/issues/216442), but with such a vital feature implemented in 3rd-party libraries we are slowed down in fixing it, and we also rely on external maintainers to merge and release fixes. diff --git a/doc/architecture/blueprints/pods/images/pods-and-fulfillment.png b/doc/architecture/blueprints/pods/images/pods-and-fulfillment.png Binary files differindex aab8556a5d3..fea32d1800e 100644 --- a/doc/architecture/blueprints/pods/images/pods-and-fulfillment.png +++ b/doc/architecture/blueprints/pods/images/pods-and-fulfillment.png diff --git a/doc/architecture/blueprints/pods/index.md b/doc/architecture/blueprints/pods/index.md index 3ba319d169b..0a36de5790f 100644 --- a/doc/architecture/blueprints/pods/index.md +++ b/doc/architecture/blueprints/pods/index.md @@ -324,6 +324,18 @@ This is the list of known affected features with the proposed solutions. - [Pods: GraphQL](pods-feature-graphql.md) - [Pods: Organizations](pods-feature-organizations.md) - [Pods: Router Endpoints Classification](pods-feature-router-endpoints-classification.md) +- [Pods: Schema changes (Postgres and Elasticsearch migrations)](pods-feature-schema-changes.md) +- [Pods: Global Search](pods-feature-global-search.md) +- [Pods: CI Runners](pods-feature-ci-runners.md) +- [Pods: Admin Area](pods-feature-admin-area.md) +- [Pods: Container Registry](pods-feature-container-registry.md) +- [Pods: Contributions: Forks](pods-feature-contributions-forks.md) +- [Pods: Personal Namespaces](pods-feature-personal-namespaces.md) +- [Pods: Dashboard: Projects, Todos, Issues, Merge Requests, Activity, ...](pods-feature-dashboard.md) +- [Pods: Snippets](pods-feature-snippets.md) +- [Pods: Uploads](pods-feature-uploads.md) +- [Pods: GitLab Pages](pods-feature-gitlab-pages.md) +- [Pods: Agent for Kubernetes](pods-feature-agent-for-kubernetes.md) ## Links diff --git a/doc/architecture/blueprints/pods/pods-feature-admin-area.md b/doc/architecture/blueprints/pods/pods-feature-admin-area.md new file mode 100644 index 00000000000..7efaa383510 --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-admin-area.md @@ -0,0 +1,58 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Admin Area' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Admin Area + +In our Pods architecture proposal we plan to share all admin related tables in +GitLab. This allows simpler management of all Pods in one interface and reduces +the risk of settings diverging in different Pods. This introduces challenges +with admin pages that allow you to manage data that will be spread across all +Pods. + +## 1. Definition + +There are consequences for admin pages that contain data that spans "the whole +instance" as the Admin pages may be served by any Pod or possibly just 1 pod. +There are already many parts of the Admin interface that will have data that +spans many pods. For example lists of all Groups, Projects, Topics, Jobs, +Analytics, Applications and more. There are also administrative monitoring +capabilities in the Admin page that will span many pods such as the "Background +Jobs" and "Background Migrations" pages. + +## 2. Data flow + +## 3. Proposal + +We will need to decide how to handle these exceptions with a few possible +options: + +1. Move all these pages out into a dedicated per-pod Admin section. Probably + the URL will need to be routable to a single Pod like `/pods/<pod_id>/admin`, + then we can display this data per Pod. These pages will be distinct from + other Admin pages which control settings that are shared across all Pods. We + will also need to consider how this impacts self-managed customers and + whether, or not, this should be visible for single-pod instances of GitLab. +1. Build some aggregation interfaces for this data so that it can be fetched + from all Pods and presented in a single UI. This may be beneficial to an + administrator that needs to see and filter all data at a glance, especially + when they don't know which Pod the data is on. The downside, however, is + that building this kind of aggregation is very tricky when all the Pods are + designed to be totally independent, and it does also enforce more strict + requirements on compatibility between Pods. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-agent-for-kubernetes.md b/doc/architecture/blueprints/pods/pods-feature-agent-for-kubernetes.md new file mode 100644 index 00000000000..f390c751b8b --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-agent-for-kubernetes.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Agent for Kubernetes' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Agent for Kubernetes + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-ci-runners.md b/doc/architecture/blueprints/pods/pods-feature-ci-runners.md new file mode 100644 index 00000000000..b75515a916f --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-ci-runners.md @@ -0,0 +1,169 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: CI Runners' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: CI Runners + +GitLab in order to execute CI jobs [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/), +very often managed by customer in their infrastructure. + +All CI jobs created as part of CI pipeline are run in a context of project +it poses a challenge how to manage GitLab Runners. + +## 1. Definition + +There are 3 different types of runners: + +- instance-wide: runners that are registered globally with specific tags (selection criteria) +- group runners: runners that execute jobs from a given top-level group or subprojects of that group +- project runners: runners that execute jobs from projects or many projects: some runners might + have projects assigned from projects in different top-level groups. + +This alongside with existing data structure where `ci_runners` is a table describing +all types of runners poses a challenge how the `ci_runners` should be managed in a Pods environment. + +## 2. Data flow + +GitLab Runners use a set of globally scoped endpoints to: + +- registration of a new runner via registration token `https://gitlab.com/api/v4/runners` + ([subject for removal](../runner_tokens/index.md)) (`registration token`) +- requests jobs via an authenticated `https://gitlab.com/api/v4/jobs/request` endpoint (`runner token`) +- upload job status via `https://gitlab.com/api/v4/jobs/:job_id` (`build token`) +- upload trace via `https://gitlab.com/api/v4/jobs/:job_id/trace` (`build token`) +- download and upload artifacts via `https://gitlab.com/api/v4/jobs/:job_id/artifacts` (`build token`) + +Currently three types of authentication tokens are used: + +- runner registration token ([subject for removal](../runner_tokens/index.md)) +- runner token representing an registered runner in a system with specific configuration (`tags`, `locked`, etc.) +- build token representing an ephemeral token giving a limited access to updating a specific + job, uploading artifacts, downloading dependent artifacts, downloading and uploading + container registry images + +Each of those endpoints do receive an authentication token via header (`JOB-TOKEN` for `/trace`) +or body parameter (`token` all other endpoints). + +Since the CI pipeline would be created in a context of a specific Pod it would be required +that pick of a build would have to be processed by that particular Pod. This requires +that build picking depending on a solution would have to be either: + +- routed to correct Pod for a first time +- be made to be two phase: request build from global pool, claim build on a specific Pod using a Pod specific URL + +## 3. Proposal + +This section describes various proposals. Reader should consider that those +proposals do describe solutions for different problems. Many or some aspects +of those proposals might be the solution to the stated problem. + +### 3.1. Authentication tokens + +Even though the paths for CI Runners are not routable they can be made routable with +those two possible solutions: + +- The `https://gitlab.com/api/v4/jobs/request` uses a long polling mechanism with + a ticketing mechanism (based on `X-GitLab-Last-Update` header). Runner when first + starts sends a request to GitLab to which GitLab responds with either a build to pick + by runner. This value is completely controlled by GitLab. This allows GitLab + to use JWT or any other means to encode `pod` identifier that could be easily + decodable by Router. +- The majority of communication (in terms of volume) is using `build token` making it + the easiest target to change since GitLab is sole owner of the token that Runner later + uses for specific job. There were prior discussions about not storing `build token` + but rather using `JWT` token with defined scopes. Such token could encode the `pod` + to which router could easily route all requests. + +### 3.2. Request body + +- The most of used endpoints pass authentication token in request body. It might be desired + to use HTTP Headers as an easier way to access this information by Router without + a need to proxy requests. + +### 3.3. Instance-wide are Pod local + +We can pick a design where all runners are always registered and local to a given Pod: + +- Each Pod has it's own set of instance-wide runners that are updated at it's own pace +- The project runners can only be linked to projects from the same organization + creating strong isolation. +- In this model the `ci_runners` table is local to the Pod. +- In this model we would require the above endpoints to be scoped to a Pod in some way + or made routable. It might be via prefixing them, adding additional Pod parameter, + or providing much more robust way to decode runner token and match it to Pod. +- If routable token is used, we could move away from cryptographic random stored in + database to rather prefer to use JWT tokens that would encode +- The Admin Area showing registered Runners would have to be scoped to a Pod + +This model might be desired since it provides strong isolation guarantees. +This model does significantly increase maintenance overhead since each Pod is managed +separately. + +This model may require adjustments to runner tags feature so that projects have consistent runner experience across pods. + +### 3.4. Instance-wide are cluster-wide + +Contrary to proposal where all runners are Pod local, we can consider that runners +are global, or just instance-wide runners are global. + +However, this requires significant overhaul of system and to change the following aspects: + +- `ci_runners` table would likely have to be split decomposed into `ci_instance_runners`, ... +- all interfaces would have to be adopted to use correct table +- build queuing would have to be reworked to be two phase where each Pod would know of all pending + and running builds, but the actual claim of a build would happen against a Pod containing data +- likely `ci_pending_builds` and `ci_running_builds` would have to be made `cluster-wide` tables + increasing likelihood of creating hotspots in a system related to CI queueing + +This model makes it complex to implement from engineering side. Does make some data being shared +between Pods. Creates hotspots / scalability issues in a system (ex. during abuse) that +might impact experience of organizations on other Pods. + +### 3.5. GitLab CI Daemon + +Another potential solution to explore is to have a dedicated service responsible for builds queueing +owning it's database and working in a model of either sharded or podded service. There were prior +discussions about [CI/CD Daemon](https://gitlab.com/gitlab-org/gitlab/-/issues/19435). + +If the service would be sharded: + +- depending on a model if runners are cluster-wide or pod-local this service would have to fetch + data from all Pods +- if the sharded service would be used we could adapt a model of either sharing database containing + `ci_pending_builds/ci_running_builds` with the service +- if the sharded service would be used we could consider a push model where each Pod pushes to CI/CD Daemon + builds that should be picked by Runner +- the sharded service would be aware which Pod is responsible for processing the given build and could + route processing requests to designated Pod + +If the service would be podded: + +- all expectations of routable endpoints are still valid + +In general usage of CI Daemon does not help significantly with the stated problem. However, this offers +a few upsides related to more efficient processing and decoupling model: push model and it opens a way +to offer stateful communication with GitLab Runners (ex. gRPC or Websockets). + +## 4. Evaluation + +Considering all solutions it appears that solution giving the most promise is: + +- use "instance-wide are Pod local" +- refine endpoints to have routable identities (either via specific paths, or better tokens) + +Other potential upsides is to get rid of `ci_builds.token` and rather use a `JWT token` +that can much better and easier encode wider set of scopes allowed by CI runner. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-container-registry.md b/doc/architecture/blueprints/pods/pods-feature-container-registry.md new file mode 100644 index 00000000000..d47913fbc2a --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-container-registry.md @@ -0,0 +1,131 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Container Registry' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Container Registry + +GitLab Container Registry is a feature allowing to store Docker Container Images +in GitLab. You can read about GitLab integration [here](../../../user/packages/container_registry/index.md). + +## 1. Definition + +GitLab Container Registry is a complex service requiring usage of PostgreSQL, Redis +and Object Storage dependencies. Right now there's undergoing work to introduce +[Container Registry Metadata](../container_registry_metadata_database/index.md) +to optimize data storage and image retention policies of Container Registry. + +GitLab Container Registry is serving as a container for stored data, +but on it's own does not authenticate `docker login`. The `docker login` +is executed with user credentials (can be `personal access token`) +or CI build credentials (ephemeral `ci_builds.token`). + +Container Registry uses data deduplication. It means that the same blob +(image layer) that is shared between many projects is stored only once. +Each layer is hashed by `sha256`. + +The `docker login` does request JWT time-limited authentication token that +is signed by GitLab, but validated by Container Registry service. The JWT +token does store all authorized scopes (`container repository images`) +and operation types (`push` or `pull`). A single JWT authentication token +can be have many authorized scopes. This allows container registry and client +to mount existing blobs from another scopes. GitLab responds only with +authorized scopes. Then it is up to GitLab Container Registry to validate +if the given operation can be performed. + +The GitLab.com pages are always scoped to project. Each project can have many +container registry images attached. + +Currently in case of GitLab.com the actual registry service is served +via `https://registry.gitlab.com`. + +The main identifiable problems are: + +- the authentication request (`https://gitlab.com/jwt/auth`) that is processed by GitLab.com +- the `https://registry.gitlab.com` that is run by external service and uses it's own data store +- the data deduplication, the Pods architecture with registry run in a Pod would reduce + efficiency of data storage + +## 2. Data flow + +### 2.1. Authorization request that is send by `docker login` + +```shell +curl \ + --user "username:password" \ + "https://gitlab/jwt/auth?client_id=docker&offline_token=true&service=container_registry&scope=repository:gitlab-org/gitlab-build-images:push,pull" +``` + +Result is encoded and signed JWT token. Second base64 encoded string (split by `.`) contains JSON with authorized scopes. + +```json +{"auth_type":"none","access":[{"type":"repository","name":"gitlab-org/gitlab-build-images","actions":["pull"]}],"jti":"61ca2459-091c-4496-a3cf-01bac51d4dc8","aud":"container_registry","iss":"omnibus-gitlab-issuer","iat":1669309469,"nbf":166} +``` + +### 2.2. Docker client fetching tags + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/tags/list + +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/manifests/danger-ruby-2.6.6 +``` + +### 2.3. Docker client fetching blobs and manifests + +```shell +curl \ + -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \ + -H "Authorization: Bearer token" \ + https://registry.gitlab.com/v2/gitlab-org/gitlab-build-images/blobs/sha256:a3f2e1afa377d20897e08a85cae089393daa0ec019feab3851d592248674b416 +``` + +## 3. Proposal + +### 3.1. Shard Container Registry separately to Pods architecture + +Due to it's architecture it extensive architecture and in general highly scalable +horizontal architecture it should be evaluated if the GitLab Container Registry +should be run not in Pod, but in a Cluster and be scaled independently. + +This might be easier, but would definitely not offer the same amount of data isolation. + +### 3.2. Run Container Registry within a Pod + +It appears that except `/jwt/auth` which would likely have to be processed by Router +(to decode `scope`) the container registry could be run as a local service of a Pod. + +The actual data at least in case of GitLab.com is not forwarded via registry, +but rather served directly from Object Storage / CDN. + +Its design encodes container repository image in a URL that is easily routable. +It appears that we could re-use the same stateless Router service in front of Container Registry +to serve manifests and blobs redirect. + +The only downside is increased complexity of managing standalone registry for each Pod, +but this might be desired approach. + +## 4. Evaluation + +There do not seem any theoretical problems with running GitLab Container Registry in a Pod. +Service seems that can be easily made routable to work well. + +The practical complexities are around managing complex service from infrastructure side. + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-contributions-forks.md b/doc/architecture/blueprints/pods/pods-feature-contributions-forks.md new file mode 100644 index 00000000000..566ae50ec49 --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-contributions-forks.md @@ -0,0 +1,120 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Contributions: Forks' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Contributions: Forks + +[Forking workflow](../../../user/project/repository/forking_workflow.md) allows users +to copy existing project sources into their own namespace of choice (personal or group). + +## 1. Definition + +[Forking workflow](../../../user/project/repository/forking_workflow.md) is common workflow +with various usage patterns: + +- allows users to contribute back to upstream project +- persist repositories into their personal namespace +- copy to make changes and release as modified project + +Forks allow users not having write access to parent project to make changes. The forking workflow +is especially important for the Open Source community which is able to contribute back +to public projects. However, it is equally important in some companies which prefer the strong split +of responsibilites and tighter access control. The access to project is restricted +to designated list of developers. + +Forks enable: + +- tigther control of who can modify the upstream project +- split of the responsibilites: parent project might use CI configuration connecting to production systems +- run CI pipelines in context of fork in much more restrictive environment +- consider all forks to be unveted which reduces risks of leaking secrets, or any other information + tied with the project + +The forking model is problematic in Pods architecture for following reasons: + +- Forks are clones of existing repositories, forks could be created across different organizations, Pods and Gitaly shards. +- User can create merge request and contribute back to upstream project, this upstream project might in a different organization and Pod. +- The merge request CI pipeline is to executed in a context of source project, but presented in a context of target project. + +## 2. Data flow + +## 3. Proposals + +### 3.1. Intra-Cluster forks + +This proposal makes us to implement forks as a intra-ClusterPod forks where communication is done via API +between all trusted Pods of a cluster: + +- Forks when created, they are created always in context of user choice of group. +- Forks are isolated from Organization. +- Organization or group owner could disable forking across organizations or forking in general. +- When a Merge Request is created it is created in context of target project, referencing + external project on another Pod. +- To target project the merge reference is transfered that is used for presenting information + in context of target project. +- CI pipeline is fetched in context of source project as it-is today, the result is fetched into + Merge Request of target project. +- The Pod holding target project internally uses GraphQL to fetch status of source project + and include in context of the information for merge request. + +Upsides: + +- All existing forks continue to work as-is, as they are treated as intra-Cluster forks. + +Downsides: + +- The purpose of Organizations is to provide strong isolation between organizations + allowing to fork across does break security boundaries. +- However, this is no different to ability of users today to clone repository to local computer + and push it to any repository of choice. +- Access control of source project can be lower than those of target project. System today + requires that in order to contribute back the access level needs to be the same for fork and upstream. + +### 3.2. Forks are created in a personal namespace of the current organization + +Instead of creating projects across organizations, the forks are created in a user personal namespace +tied with the organization. Example: + +- Each user that is part of organization receives their personal namespace. For example for `GitLab Inc.` + it could be `gitlab.com/organization/gitlab-inc/@ayufan`. +- The user has to fork into it's own personal namespace of the organization. +- The user has that many personal namespaces as many organizations it belongs to. +- The personal namespace behaves similar to currently offered personal namespace. +- The user can manage and create projects within a personal namespace. +- The organization can prevent or disable usage of personal namespaces disallowing forks. +- All current forks are migrated into personal namespace of user in Organization. +- All forks are part of to the organization. +- The forks are not federated features. +- The personal namespace and forked project do not share configuration with parent project. + +### 3.3. Forks are created as internal projects under current project + +Instead of creating projects across organizations, the forks are attachments to existing projects. +Each user forking a project receives their unique project. Example: + +- For project: `gitlab.com/gitlab-org/gitlab`, forks would be created in `gitlab.com/gitlab-org/gitlab/@kamil-gitlab`. +- Forks are created in a context of current organization, they do not cross organization boundaries + and are managed by the organization. +- Tied to the user (or any other user-provided name of the fork). +- The forks are not federated features. + +Downsides: + +- Does not answer how to handle and migrate all exisiting forks. +- Might share current group / project settings - breaking some security boundaries. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-dashboard.md b/doc/architecture/blueprints/pods/pods-feature-dashboard.md new file mode 100644 index 00000000000..e63d912b4c9 --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-dashboard.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Dashboard' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Dashboard + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-data-migration.md b/doc/architecture/blueprints/pods/pods-feature-data-migration.md index fad6bca45fa..fbe97316dcc 100644 --- a/doc/architecture/blueprints/pods/pods-feature-data-migration.md +++ b/doc/architecture/blueprints/pods/pods-feature-data-migration.md @@ -26,6 +26,24 @@ we can document the reasons for not choosing this approach. It is essential for Pods architecture to provide a way to migrate data out of big Pods into smaller ones. This describes various approaches to provide this type of split. +We also need to handle for cases where data is already violating the expected +isolation constraints of Pods (ie. references cannot span multiple +organizations). We know that existing features like linked issues allowed users +to link issues across any projects regardless of their hierarchy. There are many +similar features. All of this data will need to be migrated in some way before +it can be split across different pods. This may mean some data needs to be +deleted, or the feature changed and modelled slightly differently before we can +properly split or migrate the organizations between pods. + +Having schema deviations across different Pods, which is a necessary +consequence of different databases, will also impact our ability to migrate +data between pods. Different schemas impact our ability to reliably replicate +data across pods and especially impact our ability to validate that the data is +correctly replicated. It might force us to only be able to move data between +pods when the schemas are all in sync (slowing down deployments and the +rebalancing process) or possibly only migrate from newer to older schemas which +would be complex. + ## 1. Definition ## 2. Data flow @@ -48,13 +66,35 @@ physical replication, etc. 1. The data of Pod 0 is live replicated to as many Pods it needs to be split. 1. Once consensus is achieved between Pod 0 and N-Pods the organizations to be migrated away are marked as read-only cluster-wide. -1. The `routes` is updated on for all organizations to be split to indicate an authorative +1. The `routes` is updated on for all organizations to be split to indicate an authoritative Pod holding the most recent data, like `gitlab-org` on `pod-100`. 1. The data for `gitlab-org` on Pod 0, and on other non-authoritative N-Pods are dormant and will be removed in the future. 1. All accesses to `gitlab-org` on a given Pod are validated about `pod_id` of `routes` to ensure that given Pod is authoritative to handle the data. +#### More challenges of this proposal + +1. There is no streaming replication capability for Elasticsearch, but you could + snapshot the whole Elasticsearch index and recreate, but this takes hours. + It could be handled by pausing Elasticsearch indexing on the initial pod during + the migration as indexing downtime is not a big issue, but this still needs + to be coordinated with the migration process +1. Syncing Redis, Gitaly, CI Postgres, Main Postgres, registry Postgres, other + new data stores snapshots in an online system would likely lead to gaps + without a long downtime. You need to choose a sync point and at the sync + point you need to stop writes to perform the migration. The more data stores + there are to migrate at the same time the longer the write downtime for the + failover. We would also need to find a reliable place in the application to + actually block updates to all these systems with a high degree of + confidence. In the past we've only been confident by shutting down all rails + services because any rails process could write directly to any of these at + any time due to async workloads or other surprising code paths. +1. How to efficiently delete all the orphaned data. Locating all `ci_builds` + associated with half the organizations would be very expensive if we have to + do joins. We haven't yet determined if we'd want to store an `organization_id` + column on every table, but this is the kind of thing it would be helpful for. + ### 3.2. Migrate organization from an existing Pod This is different to split, as we intend to perform logical and selective replication @@ -75,6 +115,14 @@ which Pod is authoritative for this organization. 1. It likely will require a full database structure analysis (more robust than project import/export) to perform selective PostgreSQL logical replication. +#### More challenges of this proposal + +1. Logical replication is still not performant enough to keep up with our + scale. Even if we could use logical replication we still don't have an + efficient way to filter data related to a single organization without + joining all the way to the `organizations` table which will slow down + logical replication dramatically. + ## 4. Evaluation ## 4.1. Pros diff --git a/doc/architecture/blueprints/pods/pods-feature-git-access.md b/doc/architecture/blueprints/pods/pods-feature-git-access.md index ae996281d46..9bda2d1de9c 100644 --- a/doc/architecture/blueprints/pods/pods-feature-git-access.md +++ b/doc/architecture/blueprints/pods/pods-feature-git-access.md @@ -15,7 +15,7 @@ we can document the reasons for not choosing this approach. # Pods: Git Access This document describes impact of Pods architecture on all Git access (over HTTPS and SSH) -patterns providing explanantion of how potentially those features should be changed +patterns providing explanation of how potentially those features should be changed to work well with Pods. ## 1. Definition @@ -130,7 +130,7 @@ sequenceDiagram ## 3. Proposal -The Pods stateless router proposal requires that any ambigious path (that is not routable) +The Pods stateless router proposal requires that any ambiguous path (that is not routable) will be made to be routable. It means that at least the following paths will have to be updated do introduce a routable entity (project, group, or organization). diff --git a/doc/architecture/blueprints/pods/pods-feature-gitlab-pages.md b/doc/architecture/blueprints/pods/pods-feature-gitlab-pages.md new file mode 100644 index 00000000000..932f996d8ba --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-gitlab-pages.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: GitLab Pages' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: GitLab Pages + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-global-search.md b/doc/architecture/blueprints/pods/pods-feature-global-search.md new file mode 100644 index 00000000000..5ea863ac646 --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-global-search.md @@ -0,0 +1,47 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Global search' +--- + +DISCLAIMER: +This page may contain information related to upcoming products, features and +functionality. It is important to note that the information presented is for +informational purposes only, so please do not rely on the information for +purchasing or planning purposes. Just like with all projects, the items +mentioned on the page are subject to change or delay, and the development, +release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Global search + +When we introduce multiple Pods we intend to isolate all services related to +those Pods. This will include Elasticsearch which means our current global +search functionality will not work. It may be possible to implement aggregated +search across all pods, but it is unlikely to be performant to do fan-out +searches across all pods especially once you start to do pagination which +requires setting the correct offset and page number for each search. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +Likely first versions of Pods will simply not support global searches and then +we may later consider if building global searches to support popular use cases +is worthwhile. + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-graphql.md b/doc/architecture/blueprints/pods/pods-feature-graphql.md index 5f8a39c0b3f..87c8391fbb3 100644 --- a/doc/architecture/blueprints/pods/pods-feature-graphql.md +++ b/doc/architecture/blueprints/pods/pods-feature-graphql.md @@ -23,7 +23,7 @@ we can document the reasons for not choosing this approach. # Pods: GraphQL -GitLab exensively uses GraphQL to perform efficient data query operations. +GitLab extensively uses GraphQL to perform efficient data query operations. GraphQL due to it's nature is not directly routable. The way how GitLab uses it calls the `/api/graphql` endpoint, and only query or mutation of body request might define where the data can be accessed. diff --git a/doc/architecture/blueprints/pods/pods-feature-personal-namespaces.md b/doc/architecture/blueprints/pods/pods-feature-personal-namespaces.md new file mode 100644 index 00000000000..f78044bb551 --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-personal-namespaces.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Personal Namespaces' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Personal Namespaces + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-router-endpoints-classification.md b/doc/architecture/blueprints/pods/pods-feature-router-endpoints-classification.md index c672342fff9..bf0969fcb38 100644 --- a/doc/architecture/blueprints/pods/pods-feature-router-endpoints-classification.md +++ b/doc/architecture/blueprints/pods/pods-feature-router-endpoints-classification.md @@ -29,7 +29,7 @@ hitting load balancer of a GitLab installation to a Pod that can serve it. Each Pod should be able to decode each request and classify for which Pod it belongs to. -GitLab currently implements houndreds of endpoints. This document tries +GitLab currently implements hundreds of endpoints. This document tries to describe various techniques that can be implemented to allow the Rails to provide this information efficiently. diff --git a/doc/architecture/blueprints/pods/pods-feature-schema-changes.md b/doc/architecture/blueprints/pods/pods-feature-schema-changes.md new file mode 100644 index 00000000000..ae7c288028d --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-schema-changes.md @@ -0,0 +1,55 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Schema changes' +--- + +DISCLAIMER: +This page may contain information related to upcoming products, features and +functionality. It is important to note that the information presented is for +informational purposes only, so please do not rely on the information for +purchasing or planning purposes. Just like with all projects, the items +mentioned on the page are subject to change or delay, and the development, +release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Schema changes + +When we introduce multiple Pods that own their own databases this will +complicate the process of making schema changes to Postgres and Elasticsearch. +Today we already need to be careful to make changes comply with our zero +downtime deployments. For example, +[when removing a column we need to make changes over 3 separate deployments](../../../development/database/avoiding_downtime_in_migrations.md#dropping-columns). +We have tooling like `post_migrate` that helps with these kinds of changes to +reduce the number of merge requests needed, but these will be complicated when +we are dealing with deploying multiple rails applications that will be at +different versions at any one time. This problem will be particularly tricky to +solve for shared databases like our plan to share the `users` related tables +among all Pods. + +A key benefit of Pods may be that it allows us to run different +customers on different versions of GitLab. We may choose to update our own pod +before all our customers giving us even more flexibility than our current +canary architecture. But doing this means that schema changes need to have even +more versions of backward compatibility support which could slow down +development as we need extra steps to make schema changes. + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-snippets.md b/doc/architecture/blueprints/pods/pods-feature-snippets.md new file mode 100644 index 00000000000..1bb866ca958 --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-snippets.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Snippets' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Snippets + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/pods-feature-uploads.md b/doc/architecture/blueprints/pods/pods-feature-uploads.md new file mode 100644 index 00000000000..634f3ef9560 --- /dev/null +++ b/doc/architecture/blueprints/pods/pods-feature-uploads.md @@ -0,0 +1,29 @@ +--- +stage: enablement +group: pods +comments: false +description: 'Pods: Uploads' +--- + +This document is a work-in-progress and represents a very early state of the +Pods design. Significant aspects are not documented, though we expect to add +them in the future. This is one possible architecture for Pods, and we intend to +contrast this with alternatives before deciding which approach to implement. +This documentation will be kept even if we decide not to implement this so that +we can document the reasons for not choosing this approach. + +# Pods: Uploads + +> TL;DR + +## 1. Definition + +## 2. Data flow + +## 3. Proposal + +## 4. Evaluation + +## 4.1. Pros + +## 4.2. Cons diff --git a/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md b/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md index 21aa72273fe..ab19b652f93 100644 --- a/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md +++ b/doc/architecture/blueprints/pods/proposal-stateless-router-with-buffering-requests.md @@ -26,7 +26,7 @@ monolith. This architecture also supports regions by allowing for low traffic databases to be replicated across regions. Users are not directly exposed to the concept of Pods but instead they see -different data dependent on their currently chosen "organization". +different data dependent on their chosen "organization". [Organizations](index.md#organizations) will be a new model introduced to enforce isolation in the application and allow us to decide which request route to which pod, since an organization can only be on a single pod. @@ -166,7 +166,7 @@ graph TD; 1. A new column `routes.pod_id` is added to `routes` table 1. A new Router service exists to choose which pod to route a request to. 1. A new concept will be introduced in GitLab called an organization and a user can select a "default organization" and this will be a user level setting. The default organization is used to redirect users away from ambiguous routes like `/dashboard` to organization scoped routes like `/organizations/my-organization/-/dashboard`. Legacy users will have a special default organization that allows them to keep using global resources on `Pod US0`. All existing namespaces will initially move to this public organization. -1. If a pod receives a request for a `routes.pod_id` that it does not own it returns a `302` with `X-Gitlab-Pod-Redirect` header so that the router can send the request to the correct pod. The correct pod can also set a header `X-Gitlab-Pod-Cache` which contains information about how this request should be cached to remember the pod. For example if the request was `/gitlab-org/gitlab` then the header would encode `/gitlab-org/* => Pod US0` (ie. any requests starting with `/gitlab-org/` can always be routed to `Pod US0` +1. If a pod receives a request for a `routes.pod_id` that it does not own it returns a `302` with `X-Gitlab-Pod-Redirect` header so that the router can send the request to the correct pod. The correct pod can also set a header `X-Gitlab-Pod-Cache` which contains information about how this request should be cached to remember the pod. For example if the request was `/gitlab-org/gitlab` then the header would encode `/gitlab-org/* => Pod US0` (for example, any requests starting with `/gitlab-org/` can always be routed to `Pod US0` 1. When the pod does not know (from the cache) which pod to send a request to it just picks a random pod within it's region 1. Writes to `gitlab_users` and `gitlab_routes` are sent to a primary PostgreSQL server in our `US` region but reads can come from replicas in the same region. This will add latency for these writes but we expect they are infrequent relative to the rest of GitLab. @@ -176,7 +176,7 @@ All users will get a new column `users.default_organization` which they can control in user settings. We will introduce a concept of the `GitLab.com Public` organization. This will be set as the default organization for all existing users. This organization will allow the user to see data from all namespaces in -`Pod US0` (ie. our original GitLab.com instance). This behavior can be invisible to +`Pod US0` (for example, our original GitLab.com instance). This behavior can be invisible to existing users such that they don't even get told when they are viewing a global page like `/dashboard` that it's even scoped to an organization. @@ -195,7 +195,7 @@ frustrating and painful so to avoid this we will decompose and share all Admin A settings in the `gitlab_admin` schema. This should be safe (similar to other shared schemas) because these receive very little write traffic. -In cases where different pods need different settings (eg. the +In cases where different pods need different settings (for example, the Elasticsearch URL), we will either decide to use a templated format in the relevant `application_settings` row which allows it to be dynamic per pod. Alternatively if that proves difficult we'll introduce a new table @@ -241,7 +241,7 @@ keeping settings in sync for all pods. 1. Data in `gitlab_users` and `gitlab_routes` databases must be replicated in all regions which may be an issue for certain types of compliance. 1. The router cache may need to be very large if we get a wide variety of URLs - (ie. long tail). In such a case we may need to implement a 2nd level of + (for example, long tail). In such a case we may need to implement a 2nd level of caching in user cookies so their frequently accessed pages always go to the right pod the first time. 1. Having shared database access for `gitlab_users` and `gitlab_routes` @@ -363,7 +363,7 @@ sequenceDiagram 1. User is in Europe so DNS resolves to the router in Europe 1. The router does not have `/my-company/*` cached yet so it chooses randomly `Pod EU1` 1. `Pod EU1` redirects them through a login flow -1. Stil they request `/my-company/my-project` without the router cache, so the router chooses a random pod `Pod EU1` +1. Still they request `/my-company/my-project` without the router cache, so the router chooses a random pod `Pod EU1` 1. `Pod EU1` does not have `/my-company`, but it knows that it lives in `Pod EU0` so it redirects the router to `Pod EU0` 1. `Pod EU0` returns the correct response as well as setting the cache headers for the router `/my-company/* => Pod EU0` 1. The router now caches and remembers any request paths matching `/my-company/*` should go to `Pod EU0` @@ -499,7 +499,7 @@ allowed to use legacy global functionality like `/dashboard` to see data across namespaces located on `Pod US0`. The rails backend also knows that the default pod to render any ambiguous routes like `/dashboard` is `Pod US0`. Lastly the user will be allowed to navigate to organizations on another pod like `/my-organization` but when they do the -user will see a message indicating that some data may be missing (eg. the +user will see a message indicating that some data may be missing (for example, the MRs/Issues/Todos) counts. #### Navigates to `/gitlab-org/gitlab` while not logged in @@ -615,9 +615,9 @@ Migrating data between pods will need to factor all data stores: ### Is it still possible to leak the existence of private groups via a timing attack? If you have router in EU, and you know that EU router by default redirects -to EU located Pods, you know their latency (lets assume 10ms). Now, if your +to EU located Pods, you know their latency (lets assume 10 ms). Now, if your request is bounced back and redirected to US which has different latency -(lets assume that roundtrip will be around 60ms) you can deduce that 404 was +(lets assume that roundtrip will be around 60 ms) you can deduce that 404 was returned by US Pod and know that your 404 is in fact 403. We may defer this until we actually implement a pod in a different region. Such timing attacks are already theoretically possible with the way we do permission checks today but the timing difference is probably too small to be able to detect. diff --git a/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md b/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md index e7520f3d6a8..c99b02a35e9 100644 --- a/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md +++ b/doc/architecture/blueprints/pods/proposal-stateless-router-with-routes-learning.md @@ -26,7 +26,7 @@ monolith. This architecture also supports regions by allowing for low traffic databases to be replicated across regions. Users are not directly exposed to the concept of Pods but instead they see -different data dependent on their currently chosen "organization". +different data dependent on their chosen "organization". [Organizations](index.md#organizations) will be a new model introduced to enforce isolation in the application and allow us to decide which request route to which pod, since an organization can only be on a single pod. @@ -639,9 +639,9 @@ Migrating data between pods will need to factor all data stores: ### Is it still possible to leak the existence of private groups via a timing attack? If you have router in EU, and you know that EU router by default redirects -to EU located Pods, you know their latency (lets assume 10ms). Now, if your +to EU located Pods, you know their latency (lets assume 10 ms). Now, if your request is bounced back and redirected to US which has different latency -(lets assume that roundtrip will be around 60ms) you can deduce that 404 was +(lets assume that roundtrip will be around 60 ms) you can deduce that 404 was returned by US Pod and know that your 404 is in fact 403. We may defer this until we actually implement a pod in a different region. Such timing attacks are already theoretically possible with the way we do permission checks today but the timing difference is probably too small to be able to detect. diff --git a/doc/architecture/blueprints/rate_limiting/index.md b/doc/architecture/blueprints/rate_limiting/index.md index ffe0712d69b..22709a90cee 100644 --- a/doc/architecture/blueprints/rate_limiting/index.md +++ b/doc/architecture/blueprints/rate_limiting/index.md @@ -50,7 +50,7 @@ vision of our next rate limiting and policies enforcement architecture. - Finding what limits are defined requires performing a codebase audit. - We don't have a good way to expose limits to satellite services like Registry. - We enforce a number of different policies via opaque external systems - (Pipeline Validation Service, Bouncer, Watchtower, Cloudflare, Haproxy). + (Pipeline Validation Service, Bouncer, Watchtower, Cloudflare, HAProxy). - There is not standardized way to define policies in a way consistent with defining limits. - It is difficult to understand when a user is approaching a limit threshold. - There is no way to automatically notify a user when they are approaching thresholds. @@ -103,16 +103,16 @@ quota and by a policy. risks to performance, stability, and security. - _Example:_ API calls per second for a given IP address - _Example:_ `git clone` events per minute for a given user - - _Example:_ maximum artifact upload size of 1GB + - _Example:_ maximum artifact upload size of 1 GB - **Quota:** A global constraint in application usage that is aggregated across an entire namespace over the duration of their billing cycle. - _Example:_ 400 CI/CD minutes per namespace per month - - _Example:_ 10GB transfer per namespace per month + - _Example:_ 10 GB transfer per namespace per month - **Policy:** A representation of business logic that is decoupled from application code. Decoupled policy definitions allow logic to be shared across multiple services and/or "hot-loaded" at runtime without releasing a new version of the application. - _Example:_ decode and verify a JWT, determine whether the user has access to the - given resource based on the JWT's scopes and claims + given resource based on the JWT scopes and claims - _Example:_ deny access based on group-level constraints (such as IP allowlist, SSO, and 2FA) across all services @@ -286,7 +286,7 @@ The GitLab Policy Service might be used in two different ways: 1. The policy service feature will be used as a backend to store policies defined by users. These are two slightly different use-cases: first one is about using -internally-defined policies to ensure the stability / availably of a GitLab +internally-defined policies to ensure the stability / availability of a GitLab instance (GitLab.com or self-managed instance). The second use-case is about making GitLab Policy Service a feature that users will be able to build on top of. @@ -303,7 +303,7 @@ the sections of this document above. It is possible that GitLab Policy Service and Decoupled Limits Service can actually be the same thing. It, however, depends on the implementation details that we can't predict yet, and the decision about merging these services -together will need to be informed by subsequent interations' feedback. +together will need to be informed by subsequent iterations' feedback. ## Hierarchical limits @@ -362,7 +362,7 @@ hierarchy. Choosing a proper solution will require a thoughtful research. b. Develop YAML model for limits. c. Build Rails SDK. d. Create examples showcasing usage of the new rate limits SDK. -**Phase 3**: Team Fanout of Rails SDK - Stage Groups +**Phase 3**: Team fan out of Rails SDK - Stage Groups a. Individual stage groups begin using the SDK built in Phase 2 for new limit and policies. b. Stage groups begin replacing historical adhoc limit implementations with the SDK. c. Provides means to monitor and observe the progress of the replacement effort. Ideally this is broken down to the `feature_category` level to drive group-level buy-in -- Owning Team. @@ -373,7 +373,7 @@ hierarchy. Choosing a proper solution will require a thoughtful research. **Phase 5**: SDK for Satellite Services - Owning Team a. Build Golang SDK. c. Create examples showcasing usage of the new rate limits SDK. -**Phase 6**: Team Fanout for Satellite Services - Stage Groups +**Phase 6**: Team fan out for Satellite Services - Stage Groups a. Individual stage groups being using the SDK built in Phase 5 for new limit and policies. b. Stage groups begin replacing historical adhoc limit implementations with the SDK. diff --git a/doc/architecture/blueprints/remote_development/img/remote_dev_15_7.png b/doc/architecture/blueprints/remote_development/img/remote_dev_15_7.png Binary files differnew file mode 100644 index 00000000000..d0849ded94f --- /dev/null +++ b/doc/architecture/blueprints/remote_development/img/remote_dev_15_7.png diff --git a/doc/architecture/blueprints/remote_development/img/remote_dev_15_7_1.png b/doc/architecture/blueprints/remote_development/img/remote_dev_15_7_1.png Binary files differnew file mode 100644 index 00000000000..330873380d4 --- /dev/null +++ b/doc/architecture/blueprints/remote_development/img/remote_dev_15_7_1.png diff --git a/doc/architecture/blueprints/remote_development/index.md b/doc/architecture/blueprints/remote_development/index.md new file mode 100644 index 00000000000..39ea2fb2948 --- /dev/null +++ b/doc/architecture/blueprints/remote_development/index.md @@ -0,0 +1,315 @@ +--- +status: proposed +creation-date: "2022-11-15" +authors: [ "@vtak" ] +coach: "@grzesiek" +approvers: [ "@ericschurter", "@oregand" ] +owning-stage: "~devops::create" +participating-stages: [] +--- + +# Remote Development + +## Summary + +Remote Development is a new architecture for our software-as-a-service platform that provides a more consistent user experience writing code hosted in GitLab. It may also provide additional features in the future, such as a purely browser-based workspace and the ability to connect to an already running VM/Container or to use a GitLab-hosted VM/Container. + +## Web IDE and Remote Development + +It is important to note that `Remote Development !== Web IDE`, and this is something we want to be explicit about in this document as the terms can become conflated when they shouldn't. Our new Web IDE is a separate ongoing effort that is running in parallel to Remote Development. + +These two separate categories do have some overlap as it is a goal to allow a user to connect a running workspace to the Web IDE, **but** this does not mean the two are dependent on one another. + +You can use the [Web IDE](../../../user/project/web_ide/index.md) to commit changes to a project directly from your web browser without installing any dependencies or cloning any repositories. The Web IDE, however, lacks a native runtime environment on which you would compile code, run tests, or generate real-time feedback in the IDE. For a more complete IDE experience, you can pair the Web IDE with a Remote Development workspace that has been properly configured to run as a host. + + + +## Long-term vision + +As a [new Software Developer to a team such as Sasha](https://about.gitlab.com/handbook/product/personas/#sasha-software-developer) with no local development environment, I should be able to: + +- Navigate to a repository on GitLab.com or self-managed. +- Click a button that will provide a list of current workspaces for this repository. +- Click a button that will create a new workspace or select an existing workspace from a list. +- Go through a configuration wizard that will let me select various options for my workspace (memory/CPU). +- Start up a workspace from the Web IDE and within a minute have a fully interactive terminal panel at my disposal. +- Make code changes, run tests, troubleshoot based on the terminal output, and commit new changes. +- Submit MRs of any kind without having to clone the repository locally or to manually update a local development environment. + +## User Flow Diagram + + + +## Terminology + +We use the following terms to describe components and properties of the Remote Development architecture. + +### Remote Development + +Remote Development allows you to use a secure development environment in the cloud that you can connect to from your local machine through a web browser or a client-based solution with the purpose of developing a software product there. + +#### Remote Development properties + +- Separate your development environment to avoid impacting your local machine configuration. +- Make it easy for new contributors to get started and keep everyone on a consistent environment. +- Use tools or runtimes not available on your local OS or manage multiple versions of them. +- Access an existing development environment from multiple machines or locations. + +Discouraged synonyms: VS Code for web, Remote Development Extension, browser-only WebIDE, Client only WebIDE + +### Workspace + +Container/VM-based developer machines providing all the tools and dependencies needed to code, build, test, run, and debug applications. + +#### Workspace properties + +- Workspaces should be isolated from each other by default and are responsible for managing the lifecycle of their components. This isolation can be multi-layered: namespace isolation, network isolation, resources isolation, node isolation, sandboxing containers, etc. ([reference](https://kubernetes.io/docs/concepts/security/multi-tenancy/)). +- A workspace should contain project components as well as editor components. +- A workspace should be a combination of resources that support cloud-based development environment. +- Workspaces are constrained by the amount of resources provided to them. + +### Legacy Web IDE + +The current production [Web IDE](../../../user/project/web_ide/index.md). + +#### Legacy Web IDE properties + +An advanced editor with commit staging that currently supports: + +- [Live Preview](../../../user/project/web_ide/index.md#live-preview) +- [Interactive Web Terminals](../../../user/project/web_ide/index.md#interactive-web-terminals-for-the-web-ide) + +### Web IDE + +VS Code for web - replacement of our current legacy Web IDE. + +#### Web IDE properties + +A package for bootstrapping GitLab context-aware Web IDE that: + +- Is built on top of Microsoft's VS Code. We customize and add VS Code features in the [GitLab fork of the VS Code project](https://gitlab.com/gitlab-org/gitlab-web-ide-vscode-fork). +- Can be configured in a way that it connects to the workspace rather than only using the browser. When connected to a workspace, a user should be able to do the following from the Web IDE: + - Edit, build, or debug on a different OS than they are running locally. + - Make use of larger or more specialized hardware than their local machine for development. + - Separate developer environments to avoid conflicts, improve security, and speed up onboarding. + +### Remote Development Extension for Desktop + +Something that plugs into the desktop IDE and connects you to the workspace. + +#### Remote Development Extension for Desktop properties + +- Allows you to open any folder in a workspace. +- Should be desktop IDE agnostic. +- Should have access to local files or APIs. + +## Goals + +### A consistent experience + +Organizations should have the same user experience on our SaaS platform as they do on a self-managed GitLab instance. We want to abstract away the user's development environment to avoid impacting their local machine configuration. We also want to provide support for developing on the same operating system you deploy to or use larger or more specialized hardware. + +A major goal is that each member of a development team should have the same development experience minus any specialized local configuration. This will also make it easy for new contributors to get started and keep everyone on a consistent environment. + +### Increased availability + +A workspace should allow access to an existing development environment from multiple machines and locations across a single or multiple teams. It should also allow a user to make use of tools or runtimes not available on their local OS or manage multiple versions of them. + +Additionally, Remote Development workspaces could provide a way to implement disaster recovery if we are able to leverage the capabilities of [Pods](../../../architecture/blueprints/pods/index.md). + +### Scalability + +As an organization begins to scale, they quickly realize the need to support additional types of projects that might require extensive workflows. Remote Development workspaces aim to solve that issue by abstracting away the burden of complex machine configuration, dependency management, and possible data-seeding issues. + +To facilitate working on different features across different projects, Remote Development should allow each user to provision multiple workspaces to enable quick context switching. + +Eventually, we should be able to allow users to vertically scale their workspaces with more compute cores, memory, and other resources. If a user is currently working against a 2 CPU and 4 GB RAM workspace but comes to find they need more CPU, they should be able to upgrade their compute layer to something more suitable with a click or CLI command within the workspace. + +### Provide built-in security and enterprise readiness + +As Remote Development becomes a viable replacement for Virtual Desktop Infrastructure solutions, they must be secure and support enterprise requirements, such as role-based access control and the ability to remove all source code from developer machines. + +### Accelerate project and developer onboarding + +As a zero-install development environment that runs in your browser, Remote Development makes it easy for anyone to join your team and contribute to a project. + +### Regions + +GitLab.com is only hosted within the United States of America. Organizations located in other regions have voiced demand for local SaaS offerings. BYO infrastructure helps work in conjunction with [GitLab Regions](https://gitlab.com/groups/gitlab-org/-/epics/6037) because a user's workspace may be deployed within different geographies. The ability to deploy workspaces to different geographies might also help to solve data residency and compliance problems. + +## High-level architecture problems to solve + +A number of technical issues need to be resolved to implement a stable Remote Development offering. This section will be expanded. + +- Who is our main persona for BYO infrastructure? +- How do users authenticate? +- How do we support more than one IDE? +- How are workspaces provisioned? +- How can workspaces implement disaster recovery capabilities? +- If we cannot use SSH, what are the viable alternatives for establishing a secure WebSocket connection? +- Are we running into any limitations in functionality with the Web IDE by not having it running in the container itself? For example, are we going to get code completion, linting, and language server type features to work with our approach? +- How will our environments be provisioned, managed, created, destroyed, etc.? +- To what extent do we need to provide the user with a UI to interact with the provisioned environments? +- How will the files inside the workspace get live updated based on changes in the Web IDE? Are we going to use a [CRDT](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)-like setup to patch files in a container? Are we going to generate a diff and send it though a WebSocket connection? + +## Iteration plan + +We can't ship the entire Remote Development architecture in one go - it is too large. Instead, we are adopting an iteration plan that provides value along the way. + +- Use GitLab Agent for Kubernetes Remote Development Module. +- Integrate Remote Development with the UI and Web IDE. +- Improve security and usability. + +### High-level approach + +The nuts and bolts are being worked out at [Remote Development GA4K Architecture](https://gitlab.com/gitlab-org/remote-development/gitlab-remote-development-docs/-/blob/main/doc/architecture.md) to keep a SSoT. Once we have hammered out the details, we'll replace this section with the diagram in the above repository. + +### Iteration 0: [GitLab Agent for Kubernetes Remote Development Module (plumbing)](https://gitlab.com/groups/gitlab-org/-/epics/9138) + +#### Goals + +- Use the [GitLab Agent](../../../user/clusters/agent/index.md) integration. +- Create a workspace in a Kubernetes cluster based on a `devfile` in a public repository. +- Install the IDE and dependencies as defined. +- Report the status of the environment (via the terminal or through an endpoint). +- Connect to an IDE in the workspace. + +#### Requirements + +- Remote environment running on a Kubernetes cluster based on a `devfile` in a repo. + +These are **not** part of Iteration 0: + +- Authentication/authorization with GitLab and a user. +- Integration of Remote Development with the GitLab UI and Web IDE. +- Using GA4K instead of an Ingress controller. + +#### Assumptions + +- We will use [`devworkspace-operator` v0.17.0 (latest version)](https://github.com/devfile/devworkspace-operator/releases/tag/v0.17.0). A prerequisite is [`cert-manager`](https://github.com/devfile/devworkspace-operator#with-yaml-resources). +- We have an Ingress controller ([Ingress-NGINX](https://github.com/kubernetes/ingress-nginx)), which is accessible over the network. +- The initial server is stubbed. + +#### Success criteria + +- Using GA4K to communicate with the Kubernetes API from the `remote_dev` agent module. +- All calls to the Kubernetes API are done through GA4K. +- A workspace in a Kubernetes cluster created using DevWorkspace Operator. + +### Iteration 1: [Rails endpoints, authentication, and authorization](https://gitlab.com/groups/gitlab-org/-/epics/9323) + +#### Goals + +- Add endpoints in Rails to accept work from a user. +- Poll Rails for work from KAS. +- Add authentication and authorization to the workspaces created in the Kubernetes cluster. +- Extend the GA4K `remote_dev` agent module to accept more types of work (get details of a workspace, list workspaces for a user, etc). +- Build an editor injector for the GitLab fork of VS Code. + +#### Requirements + +- [GitLab Agent for Kubernetes Remote Development Module (plumbing)](https://gitlab.com/groups/gitlab-org/-/epics/9138) is complete. + +These are **not** part of Iteration 1: + +- Integration of Remote Development with the GitLab UI and Web IDE. +- Using GA4K instead of an Ingress controller. + +#### Assumptions + +- TBA + +#### Success criteria + +- Poll Rails for work from KAS. +- Rails endpoints to create/delete/get/list workspaces. +- All requests are correctly authenticated and authorized except where the user has requested the traffic to be public (for example, opening a server while developing and making it public). +- A user can create a workspace, start a server on that workspace, and have that traffic become private/internal/public. +- We are using the GitLab fork of VS Code as an editor. + +### Iteration 2: [Integrate Remote Development with the UI and Web IDE](https://gitlab.com/groups/gitlab-org/-/epics/9169) + +#### Goals + +- Allow users full control of their workspaces via the GitLab UI. + +#### Requirements + +- [GitLab Agent for Kubernetes Remote Development Module](https://gitlab.com/groups/gitlab-org/-/epics/9138). + +These are **not** part of Iteration 2: + +- Usability improvements +- Security improvements + +#### Success criteria + +- Be able to list/create/delete/stop/start/restart workspaces from the UI. +- Be able to create workspaces for the user in the Web IDE. +- Allow the Web IDE terminal to connect to different containers in the workspace. +- Configure DevWorkspace Operator for user-expected configuration (30-minute workspace timeout, a separate persistent volume for each workspace that is deleted when the workspace is deleted, etc.). + +### Iteration 3: [Improve security and usability](https://gitlab.com/groups/gitlab-org/-/epics/9170) + +#### Goals + +- Improve security and usability of our Remote Development solution. + +#### Requirements + +- [Integrate Remote Development with the UI and Web IDE](https://gitlab.com/groups/gitlab-org/-/epics/9169) is complete. + +#### Assumptions + +- We are allowing for internal feedback and closed/early customer feedback that can be iterated on. +- We have explored or are exploring the feasibility of using GA4K with Ingresses in [Solving Ingress problems for Remote Development](https://gitlab.com/gitlab-org/gitlab/-/issues/378998). +- We have explored or are exploring Kata containers for providing root access to workspace users in [Investigate Kata Containers / Firecracker / gVisor](https://gitlab.com/gitlab-org/gitlab/-/issues/367043). +- We have explored or are exploring how Ingress/Egress requests cannot be misused from [resources within or outside the cluster](https://gitlab.com/gitlab-org/remote-development/gitlab-remote-development-docs/-/blob/main/doc/securing-the-workspace.md) (security hardening). + +#### Success criteria + +Add options to: + +- Create different classes of workspaces (1gb-2cpu, 4gb-8cpu, etc.). +- Vertically scale up workspace resources. +- Inject secrets from a GitLab user/group/repository. +- Configure timeouts of workspaces at multiple levels. +- Allow users to expose endpoints in their workspace (for example, not allow anyone in the organization to expose any endpoint publicly). + +## Market analysis + +We have conducted a market analysis to understand the broader market and what others can offer us by way of open-source libraries, integrations, or partnership opportunities. We have broken down the effort into a set of issues where we investigate each potential competitor/pathway/partnership as a spike. + +- [Market analysis](https://gitlab.com/groups/gitlab-org/-/epics/8131) +- [YouTube results](https://www.youtube.com/playlist?list=PL05JrBw4t0KrRQhnSYRNh1s1mEUypx67-) + +### Next Steps + +While our spike proved fruitful, we have paused this investigation until we reach our goals in [Viable Maturity](https://gitlab.com/groups/gitlab-org/-/epics/9190). + +## Che versus a custom-built solution + +After an investigation into using [Che](https://gitlab.com/gitlab-org/gitlab/-/issues/366052) as our backend to accelerate Remote Development, we ultimately opted to [write our own custom-built solution](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/97449#note_1131215629). + +Some advantages of us opting to write our own custom-built solution are: + +- We can still use the core DevWorkspace Operator and build on top of it. +- It is easier to add support for other configurations apart from `devfile` in the future if the need arises. +- We have the ability to choose which tech stack to use (for example, instead of using Traefik which is used in Che, explore NGINX itself or use GitLab Agent for Kubernetes). + +## Links + +- [Remote Development presentation](https://docs.google.com/presentation/d/1XHH_ZilZPufQoWVWViv3evipI-BnAvRQrdvzlhBuumw/edit#slide=id.g131f2bb72e4_0_8) +- [Category Strategy epic](https://gitlab.com/groups/gitlab-org/-/epics/7419) +- [Minimal Maturity epic](https://gitlab.com/groups/gitlab-org/-/epics/9189) +- [Viable Maturity epic](https://gitlab.com/groups/gitlab-org/-/epics/9190) +- [Complete Maturity epic](https://gitlab.com/groups/gitlab-org/-/epics/9191) +- [Bi-weekly sync](https://docs.google.com/document/d/1hWVvksIc7VzZjG-0iSlzBnLpyr-OjwBVCYMxsBB3h_E/edit#) +- [Market analysis and architecture](https://gitlab.com/groups/gitlab-org/-/epics/8131) +- [GA4K Architecture](https://gitlab.com/gitlab-org/remote-development/gitlab-remote-development-docs/-/blob/main/doc/architecture.md) +- [BYO infrastructure](https://gitlab.com/groups/gitlab-org/-/epics/8290) +- [Browser runtime](https://gitlab.com/groups/gitlab-org/-/epics/8291) +- [GitLab-hosted infrastructure](https://gitlab.com/groups/gitlab-org/-/epics/8292) +- [Browser runtime spike](https://gitlab.com/gitlab-org/gitlab-web-ide/-/merge_requests/58). +- [Remote Development direction](https://about.gitlab.com/direction/create/editor/remote_development) +- [Ideal user journey](https://about.gitlab.com/direction/create/editor/remote_development/#ideal-user-journey) diff --git a/doc/architecture/blueprints/runner_scaling/index.md b/doc/architecture/blueprints/runner_scaling/index.md index 24c6820f94a..8eb6bfd2551 100644 --- a/doc/architecture/blueprints/runner_scaling/index.md +++ b/doc/architecture/blueprints/runner_scaling/index.md @@ -234,7 +234,7 @@ them each separately. etc... This information is very provider specific. - **VM lifecycle management**. Multiple machines will be created and a system must keep track of which machines belong to this executor. Typically - a cloud provider will have a way to manage a set of homogenous machines. + a cloud provider will have a way to manage a set of homogeneous machines. E.g. GCE Instance Group. The basic operations are increase, decrease and usually delete a specific machine. - **VM autoscaling**. In addition to low-level lifecycle management, @@ -273,7 +273,7 @@ interfaces. Within the `docker+autoscaling` executor the [`machineExecutor`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/executors/docker/machine/machine.go#L19) type has a [`Machine`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/helpers/docker/machine.go#L7) -interface which it uses to aquire a VM during the common [`Prepare`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/executors/docker/machine/machine.go#L71) +interface which it uses to acquire a VM during the common [`Prepare`](https://gitlab.com/gitlab-org/gitlab-runner/-/blob/267f40d871cd260dd063f7fbd36a921fedc62241/executors/docker/machine/machine.go#L71) phase. This abstraction primarily creates, accesses and deletes VMs. There is no current abstraction for the VM autoscaling logic. It is tightly @@ -372,7 +372,7 @@ provide a context and an environment in which a build will be executed by one of the Custom Executors. There are multiple solutions to implementing a custom provider abstraction. We -can use raw Go plugins, Hashcorp's Go Plugin, HTTP interface or gRPC based +can use raw Go plugins, HashiCorp's Go Plugin, HTTP interface or gRPC based facade service. There are many solutions, and we want to choose the most optimal one. In order to do that, we will describe the solutions in a separate document, define requirements and score the solution accordingly. This will @@ -390,18 +390,18 @@ Rationale: [Description of the Custom Executor Provider proposal](https://gitlab We can introduce a more simple version of the `Machine` abstraction in the form of a "Fleeting" interface. Fleeting provides a low-level interface to -a homogenous VM group which allows increasing and decreasing the set size +a homogeneous VM group which allows increasing and decreasing the set size as well as consuming a VM from within the set. Plugins for cloud providers and other VM sources are implemented via the -Hashicorp go-plugin library. This is in practice gRPC over STDIN/STDOUT +HashiCorp go-plugin library. This is in practice gRPC over STDIN/STDOUT but other wire protocols can be used also. In order to make use of the new interface, the autoscaling logic is pulled out of the Docker Executor and placed into a new Taskscaler library. This places the concerns of VM lifecycle, VM shape and job routing within -the plugin. It also places the conern of VM autoscaling into a separate +the plugin. It also places the concern of VM autoscaling into a separate component so it can be used by multiple Runner Executors (not just `docker+autoscaling`). Rationale: [Description of the InstanceGroup / Fleeting proposal](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/28848#note_823430883) diff --git a/doc/architecture/blueprints/runner_tokens/index.md b/doc/architecture/blueprints/runner_tokens/index.md index 3f8a27e503d..7a282649c5c 100644 --- a/doc/architecture/blueprints/runner_tokens/index.md +++ b/doc/architecture/blueprints/runner_tokens/index.md @@ -14,7 +14,7 @@ CI/CD jobs in a reliable and concurrent environment. Ever since the beginnings of the service as a Ruby program, runners are registered in a GitLab instance with a registration token - a randomly generated string of text. The registration token is unique for its given scope (instance, group, or project). The registration token proves that the party that registers the runner has -administrator access to the instance, group, or project to which the runner is registered. +administrative access to the instance, group, or project to which the runner is registered. This approach has worked well in the initial years, but some major known issues started to become apparent as the target audience grew: @@ -37,49 +37,122 @@ We call this new mechanism the "next GitLab Runner Token architecture". The proposal addresses the issues of a _single token per scope_ and _token storage_ by eliminating the need for a registration token. Runner creation happens -in the GitLab Runners settings page for the given scope, in the context of the logged-in user -, which provides traceability. The page provides instructions to configure the newly-created -runner in supported environments. +in the GitLab Runners settings page for the given scope, in the context of the logged-in user, +which provides traceability. The page provides instructions to configure the newly-created +runner in supported environments using the existing `gitlab-runner register` command. -The runner configuration will be generated through a new `deploy` command, which will leverage -the `/runners/verify` REST endpoint to ensure the validity of the authentication token. The remaining concerns become non-issues due to the elimination of the registration token. -The configuration can be applied across many machines by reusing the same instructions. -A unique system identifier will be generated automatically if a value is missing from -the runner entry in the `config.toml` file. This allows differentiating systems sharing the same -runner token (for example, in auto-scaling scenarios), and is crucial for the proper functioning of our -long-polling mechanism when the same authentication token is shared across two or more runner managers. +### Using the authentication token in place of the registration token + +<!-- vale gitlab.Spelling = NO --> +In this proposal, runners created in the GitLab UI are assigned authentication tokens prefixed with +`glrt-` (**G**it**L**ab **R**unner **T**oken). +<!-- vale gitlab.Spelling = YES --> +The prefix allows the existing `register` command to use the authentication token _in lieu_ +of the current registration token (`--registration-token`), requiring minimal adjustments in +existing workflows. +The authentication token is shown to the user only once - after completing the creation flow - to +discourage unintended reuse. + +Given that the runner is pre-created through the GitLab UI, the `register` command fails if +provided with arguments that are exposed in the runner creation form. +Some examples are `--tag-list`, `--run-untagged`, `--locked`, or `--access-level` as these are +sensitive parameters that should be decided at creation time by an administrator/owner. +The runner configuration is generated through the existing `register` command, which can behave in +two different ways depending on whether it is supplied a registration token or an authentication +token in the `--registration-token` argument: + +| Token type | Behavior | +| ---------- | -------- | +| Registration token | Leverages the `POST /api/v4/runners` REST endpoint to create a new runner, creating a new entry in `config.toml`. | +| Authentication token | Leverages the `POST /api/v4/runners/verify` REST endpoint to ensure the validity of the authentication token. Creates an entry in `config.toml` file and a `system_id` value in a sidecar file if missing (`.runner_system_id`). | + +### Transition period + +During a transition period, legacy tokens ("registration tokens") continue to be shown on the +GitLab Runners settings page and to be accepted by the `gitlab-runner register` command. +The legacy workflow is nevertheless discouraged in the UI. +Users are steered towards the new flow consisting of creating the runner in the UI and using the +resulting authentication token with the `gitlab-runner register` command as they do today. +This approach reduces disruption to users responsible for deploying runners. + +### Reusing the runner authentication token across many machines + +In the existing model, a new runner is created whenever a new worker is required. This +has led to many situations where runners are left behind and become stale. + +In the proposed model, a `ci_runners` table entry describes a configuration that the user can reuse +across multiple machines. +A unique system identifier is [generated automatically](#generating-a-system_id-value) whenever the +runner application starts up or the configuration is saved. +This allows differentiating the context in which the runner is being used. + +The `system_id` value complements the short runner token that is currently used to identify a +runner in command line output, CI job logs, and GitLab UI. Given that the creation of runners involves user interaction, it should be possible to eventually lower the per-plan limit of CI runners that can be registered per scope. -### Auto-scaling scenarios (for example Helm chart) +#### Generating a `system_id` value -In the existing model, a new runner is created whenever a new worker is required. This -has led to many situations where runners are left behind and become stale. +We ensure that a unique system identifier is assigned at all times to a `gitlab-runner` +installation. +The ID is derived from an existing machine identifier such as `/etc/machine-id` (on Linux) and +hashed for privacy, in which case it is prefixed with `s_`. +If an ID is not available, a random string is used instead, in which case it is prefixed with `r_`. -In the proposed model, a `ci_runners` table entry describes a configuration, -which the runner could reuse across multiple machines. This allows differentiating the context in -which the runner is being used. In situations where we must differentiate between runners -that reuse the same configuration, we can use the unique system identifier to track all -unique "runners" that are executed in context of a single `ci_runners` model. This unique -system identifier would be present in the Runner's `config.toml` configuration file and -initially set when generating the new `[[runners]]` configuration by means of the `deploy` command. -Legacy files that miss values for unique system identifiers will get rewritten automatically with new values. +This unique ID identifies the `gitlab-runner` process and is sent +on `POST /api/v4/jobs` requests for all runners in the `config.toml` file. + +The ID is generated and saved both at `gitlab-runner` startup and whenever the configuration is +saved to disk. +Instead of saving the ID at the root of `config.toml` though, we save it to a new file that lives +next to it - `.runner_system_id`. The goal for this new file is to make it less likely that IDs +get reused due to manual copying of the `config.toml` file + +```plain +s_cpwhDr7zFz4xBJujFeEM +``` ### Runner identification in CI jobs -For users to identify the machine where the job was executed, the unique identifier will need to be visible in CI job contexts. +For users to identify the machine where the job was executed, the unique identifier needs to be +visible in CI job contexts. As a first iteration, GitLab Runner will include the unique system identifier in the build logs, wherever it publishes the short token SHA. -Given that the runner will potentially be reused with different unique system identifiers, -we can store the unique system ID. This ensures the unique system ID maps to a GitLab Runner's `config.toml` entry with -the runner token. The `ci_runner_machines` would hold information about each unique runner machine, -with information when runner last connected, and what type of runner it was. The relevant fields -will be moved from the `ci_runners`. -The `ci_builds_runner_session` (or `ci_builds` or `ci_builds_metadata`) will reference +Given that the runner can potentially be reused with different unique system identifiers, +we should store the unique system ID in the database. +This ensures the unique system ID maps to a GitLab Runner's `system_id` value with the runner token. +A new `ci_runner_machines` table holds information about each unique runner machine, +with information regarding when the runner last connected, and what type of runner it was. + +In the long term, the relevant fields are to be moved from the `ci_runners` into +`ci_runner_machines`. +Until the removal milestone though, they should be kept in the `ci_runners` as a fallback when a +matching `ci_runner_machines` record does not exist. +An expected scenario is the case when the table is created but the runner hasn't pinged the GitLab +instance (for example if the runner is offline). + +In addition, we should add the following columns to `ci_runners`: + +- a `user_id` column to keep track of who created a runner; +- a `registration_type` enum column to `ci_runners` to signal whether a runner has been created + using the legacy `register` method, or the new UI-based method. + Possible values are `:registration_token` and `:authenticated_user`. + This allows the stale runner cleanup service to determine which runners to clean up, and allows + future uses that may not be apparent. + +```sql +CREATE TABLE ci_runner ( + ... + user_id bigint + registration_type int8 +) +``` + +The `ci_builds_runner_session` (or `ci_builds` or `ci_builds_metadata`) shall reference `ci_runner_machines`. We might consider a more efficient way to store `contacted_at` than updating the existing record. @@ -110,7 +183,8 @@ CREATE TABLE ci_runner_machines ( - Runners can always be traced back to the user who created it, using the audit log; - The claims of a CI runner are known at creation time, and cannot be changed from the runner (for example, changing the `access_level`/`protected` flag). Authenticated users - may however still edit these settings through the GitLab UI. + may however still edit these settings through the GitLab UI; +- Easier cleanup of stale runners, which doesn't touch the `ci_runner` table. ## Details @@ -121,53 +195,95 @@ token to register new runners. The new workflow looks as follows: - 1. The user opens the Runners settings page; + 1. The user opens the Runners settings page (instance, group, or project level); 1. The user fills in the details regarding the new desired runner, namely description, tags, protected, locked, etc.; 1. The user clicks `Create`. That results in the following: - 1. Creates a new runner in the `ci_runners` table (and corresponding authentication token); + 1. Creates a new runner in the `ci_runners` table (and corresponding `glrt-` prefixed authentication token); 1. Presents the user with instructions on how to configure this new runner on a machine, with possibilities for different supported deployment scenarios (e.g. shell, `docker-compose`, Helm chart, etc.) - This information contains a token which will only be available to the user once, and the UI - will make it clear to the user that the value will not be shown again, as registering the same runner multiple times + This information contains a token which is available to the user only once, and the UI + makes it clear to the user that the value shall not be shown again, as registering the same runner multiple times is discouraged (though not impossible). - 1. The user copies and pastes the instructions for the intended deployment scenario (a `deploy` command), leading to the following actions: + 1. The user copies and pastes the instructions for the intended deployment scenario (a `register` command), leading to the following actions: - 1. Upon executing the new `gitlab-runner deploy` command in the instructions, `gitlab-runner` will perform - a call to the `POST /runners/verify` with the given runner token; - 1. If the `POST /runners/verify` GitLab endpoint validates the token, the `config.toml` file will be populated with the configuration. + 1. Upon executing the new `gitlab-runner register` command in the instructions, `gitlab-runner` performs + a call to the `POST /api/v4/runners/verify` with the given runner token; + 1. If the `POST /api/v4/runners/verify` GitLab endpoint validates the token, the `config.toml` + file is populated with the configuration; + 1. Whenever a runner pings for a job, the respective `ci_runner_machines` record is + ["upserted"](https://en.wiktionary.org/wiki/upsert) with the latest information about the + runner (with Redis cache in front of it like we do for Runner heartbeats). - The `gitlab-runner deploy` will also accept executor-specific arguments - currently present in the `register` command. +As part of the transition period, we provide admins and top-level group owners with an +instance/group-level setting (`allow_runner_registration_token`) to disable the legacy registration +token functionality and enforce using only the new workflow. +Any attempt by a `gitlab-runner register` command to hit the `POST /api/v4/runners` endpoint +to register a new runner with a registration token results in a `HTTP 410 Gone` status code. -As part of the transition period, we will provide admins and top-level group owners with a instance/group-level setting to disable -the legacy registration token functionality and enforce using only the new workflow. -Any attempt by a `gitlab-runner register` command to hit the `POST /runners` endpoint to register a new runner -will result in a `HTTP 410 - Gone` status code. The instance setting is inherited by the groups -, which means that if the legacy registration method is disabled at the instance method, the descendant groups/projects will also mandatorily -prevent the legacy registration method. +The instance setting is inherited by the groups. This means that if the legacy registration method +is disabled at the instance method, the descendant groups/projects mandatorily prevents the legacy +registration method. The registration token workflow is to be deprecated (with a deprecation notice printed by the `gitlab-runner register` command) and removed at a future major release after the concept is proven stable and customers have migrated to the new workflow. ### Handling of legacy runners -Legacy versions of GitLab Runner will not send the unique system identifier in its requests, and we +Legacy versions of GitLab Runner do not send the unique system identifier in its requests, and we will not change logic in Workhorse to handle unique system IDs. This can be improved upon in the -future once the legacy registration system is removed, and runners have been upgraded to newer +future after the legacy registration system is removed, and runners have been upgraded to newer versions. -Not using the unique system ID means that all connected runners with the same token will be +Job pings from such legacy runners results in a `ci_runner_machines` record containing a +`<legacy>` `machine_id` field value. + +Not using the unique system ID means that all connected runners with the same token are notified, instead of just the runner matching the exact system identifier. While not ideal, this is not an issue per-se. -### Helm chart +### `ci_runner_machines` record lifetime + +New records are created when the runner pings the GitLab instance for new jobs, if a record matching +the `token`+`system_id` does not already exist. + +Due to the time-decaying nature of the `ci_runner_machines` records, they are automatically +cleaned after 7 days after the last contact from the respective runner. + +### Required adaptations -The `runnerRegistrationToken` entry in the [`values.yaml` file](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/a70bc29a903b79d5675bb0c45d981adf8b7a8659/values.yaml#L52) -will be retired. The `runnerRegistrationToken` entry will be replaced by the existing `runnerToken` value, which will be passed -to the new `gitlab-runner deploy` command in [`configmap.yaml`](https://gitlab.com/gitlab-org/charts/gitlab-runner/-/blob/a70bc29a903b79d5675bb0c45d981adf8b7a8659/templates/configmap.yaml#L116). +#### Migration to `ci_runner_machines` table + +When details from `ci_runner_machines` are needed, we need to fall back to the existing fields in +`ci_runner` if a match is not found in `ci_runner_machines`. + +#### REST API + +API endpoints receiving runner tokens should be changed to also take an optional +`system_id` parameter, sent alongside with the runner token (most often as a JSON parameter on the +request body). + +#### GraphQL `CiRunner` type + +The [`CiRunner` type](../../../api/graphql/reference/index.md#cirunner) closely reflects the +`ci_runners` model. This means that machine information such as `ipAddress`, `architectureName`, +and `executorName` among others are no longer singular values in the proposed approach. +We can live with that fact for the time being and start returning lists of unique values, separated +by commas. +The respective `CiRunner` fields must return the values for the `ci_runner_machines` entries +(falling back to `ci_runner` record if non-existent). + +#### Stale runner cleanup + +The functionality to +[clean up stale runners](../../../ci/runners/configure_runners.md#clean-up-stale-runners) needs +to be adapted to clean up `ci_runner_machines` records instead of `ci_runners` records. + +At some point after the removal of the registration token support, we'll want to create a background +migration to clean up stale runners that have been created with a registration token (leveraging the +enum column created in the `ci_runners` table. ### Runner creation through API @@ -176,21 +292,66 @@ using PAT tokens for example - such that every runner is associated with an owne ## Implementation plan +### Stage 1 - Deprecations + +| Component | Milestone | Changes | +|------------------|----------:|---------| +| GitLab Rails app | `15.6` | Deprecate `POST /api/v4/runners` endpoint for `17.0`. This hinges on a [proposal](https://gitlab.com/gitlab-org/gitlab/-/issues/373774) to allow deprecating REST API endpoints for security reasons. | +| GitLab Runner | `15.6` | Add deprecation notice for `register` command for `17.0`. | +| GitLab Runner Helm Chart | `15.6` | Add deprecation notice for `runnerRegistrationToken` command for `17.0`. | +| GitLab Runner Operator | `15.6` | Add deprecation notice for `runner-registration-token` command for `17.0`. | +| GitLab Runner / GitLab Rails app | `15.7` | Add deprecation notice for registration token reset for `17.0`. | + +### Stage 2 - Prepare `gitlab-runner` for `system_id` + +| Component | Milestone | Changes | +|------------------|----------:|---------| +| GitLab Runner | `15.x` | Ensure a sidecar TOML file exists with a `system_id` value.<br/>Log new system ID values with `INFO` level as they get assigned. | +| GitLab Runner | `15.x` | Log unique system ID in the build logs. | +| GitLab Runner | `15.x` | Label Prometheus metrics with unique system ID. | +| GitLab Runner | `15.x` | Prepare `register` command to fail if runner server-side configuration options are passed together with a new `glrt-` token. | + +### Stage 3 - Database changes + +| Component | Milestone | Changes | +|------------------|----------:|---------| +| GitLab Rails app | | Create database migration to add columns to `ci_runners` table. | +| GitLab Rails app | | Create database migration to add `ci_runner_machines` table. | +| GitLab Rails app | | Create database migration to add `ci_runner_machines.machine_id` foreign key to `ci_builds_runner_session` table. | +| GitLab Rails app | | Create database migrations to add `allow_runner_registration_token` setting to `application_settings` and `namespace_settings` tables (default: `true`). | +| GitLab Runner | | Use runner token + `system_id` JSON parameters in `POST /jobs/request` request in the [heartbeat request](https://gitlab.com/gitlab-org/gitlab/blob/c73c96a8ffd515295842d72a3635a8ae873d688c/lib/api/ci/helpers/runner.rb#L14-20) to update the `ci_runner_machines` cache/table. | +| GitLab Runner | | Start sending `system_id` value in `POST /jobs/request` request and other follow-up requests that require identifying the unique system. | +| GitLab Rails app | | Create service similar to `StaleGroupRunnersPruneCronWorker` service to clean up `ci_runner_machines` records instead of `ci_runners` records.<br/>Existing service continues to exist but focuses only on legacy runners. | + +### Stage 4 - New UI + +| Component | Milestone | Changes | +|------------------|----------:|---------| +| GitLab Runner | | Implement new GraphQL user-authenticated API to create a new runner. | +| GitLab Runner | | [Add prefix to newly generated runner authentication tokens](https://gitlab.com/gitlab-org/gitlab/-/issues/383198). | +| GitLab Rails app | | Implement UI to create new runner. | +| GitLab Rails app | | GraphQL changes to `CiRunner` type. | +| GitLab Rails app | | UI changes to runner details view (listing of platform, architecture, IP address, etc.) (?) | + +### Stage 5 - Optional disabling of registration token + +| Component | Milestone | Changes | +|------------------|----------:|---------| +| GitLab Rails app | | Add UI to allow disabling use of registration tokens at project or group level. | +| GitLab Rails app | `16.0` | Introduce `:disable_runner_registration_tokens` feature flag (enabled by default) to control whether use of registration tokens is allowed. | +| GitLab Rails app | | Make [`POST /api/v4/runners` endpoint](../../../api/runners.md#register-a-new-runner) permanently return `HTTP 410 Gone` if either `allow_runner_registration_token` setting or `:disable_runner_registration_tokens` feature flag disables registration tokens.<br/>A future v5 version of the API should return `HTTP 404 Not Found`. | +| GitLab Rails app | | Start refusing job requests that don't include a unique ID, if either `allow_runner_registration_token` setting or `:disable_runner_registration_tokens` feature flag disables registration tokens. | +| GitLab Rails app | | Hide legacy UI showing registration with a registration token, if `:disable_runner_registration_tokens` feature flag disables registration tokens. | + +### Stage 6 - Removals + | Component | Milestone | Changes | -|------------------|-----------|---------| -| GitLab Rails app | `15.x` (latest at `15.6`) | Deprecate `POST /api/v4/runners` endpoint for `16.0`. This hinges on a [proposal](https://gitlab.com/gitlab-org/gitlab/-/issues/373774) to allow deprecating REST API endpoints for security reasons. | -| GitLab Runner | `15.x` (latest at `15.8`) | Add deprecation notice for `register` command for `16.0`. | -| GitLab Runner | `15.x` | Ensure all runner entries in `config.toml` have unique system identifier values assigned. Log new system ID values with `INFO` level as they get created. | -| GitLab Runner | `15.x` | Start additionally logging unique system ID anywhere we log the runner short SHA. | -| GitLab Rails app | `15.x` | Create database migrations to add settings from `application_settings` and `namaspace_settings` tables. | -| GitLab Runner | `15.x` | Start sending `unique_id` value in `POST /jobs/request` request and other follow-up requests that require identifying the unique system. | -| GitLab Runner | `15.x` | Implement new user-authenticated API (REST and GraphQL) to create a new runner. | -| GitLab Rails app | `15.x` | Implement UI to create new runner. | -| GitLab Runner | `16.0` | Remove `register` command and support for `POST /runners` endpoint. | -| GitLab Rails app | `16.0` | Remove legacy UI showing registration with a registration token. | -| GitLab Rails app | `16.0` | Create database migrations to remove settings from `application_settings` and `namaspace_settings` tables. | -| GitLab Rails app | `16.0` | Make [`POST /api/v4/runners` endpoint](../../../api/runners.md#register-a-new-runner-deprecated) permanently return `410 Gone`. A future v5 version of the API would return `404 Not Found`. | -| GitLab Rails app | `16.0` | Start refusing job requests that don't include a unique ID. | +|------------------|----------:|---------| +| GitLab Rails app | `17.0` | Remove legacy UI showing registration with a registration token. | +| GitLab Runner | `17.0` | Remove runner model arguments from `register` command (for example `--run-untagged`, `--tag-list`, etc.) | +| GitLab Rails app | `17.0` | Create database migrations to drop `allow_runner_registration_token` setting columns from `application_settings` and `namespace_settings` tables. | +| GitLab Rails app | `17.0` | Create database migrations to drop:<br/>- `runners_registration_token`/`runners_registration_token_encrypted` columns from `application_settings`;<br/>- `runners_token`/`runners_token_encrypted` from `namespaces` table;<br/>- `runners_token`/`runners_token_encrypted` from `projects` table. | +| GitLab Rails app | `17.0` | Remove `:disable_runner_registration_tokens` feature flag. | ## Status diff --git a/doc/architecture/blueprints/work_items/index.md b/doc/architecture/blueprints/work_items/index.md index 75a9d8d76ad..101fdbf4c2d 100644 --- a/doc/architecture/blueprints/work_items/index.md +++ b/doc/architecture/blueprints/work_items/index.md @@ -60,8 +60,8 @@ All Work Item types share the same pool of predefined widgets and are customized | assignees | | | description | | | hierarchy | | -| [iteration](https://gitlab.com/gitlab-org/gitlab/-/issues/367456) | work_items_mvc_2 | -| [milestone](https://gitlab.com/gitlab-org/gitlab/-/issues/367463) | work_items_mvc_2 | +| [iteration](https://gitlab.com/gitlab-org/gitlab/-/issues/367456) | | +| [milestone](https://gitlab.com/gitlab-org/gitlab/-/issues/367463) | | | labels | | | start and due date | | | status\* | | |
