diff options
Diffstat (limited to 'doc/architecture')
13 files changed, 76 insertions, 78 deletions
diff --git a/doc/architecture/blueprints/ci_data_decay/index.md b/doc/architecture/blueprints/ci_data_decay/index.md index 8808a526df0..7c0bdf299db 100644 --- a/doc/architecture/blueprints/ci_data_decay/index.md +++ b/doc/architecture/blueprints/ci_data_decay/index.md @@ -48,8 +48,8 @@ PostgreSQL database running on GitLab.com. This volume contributes to significant performance problems, development challenges and is often related to production incidents. -We also expect a [significant growth in the number of builds executed on -GitLab.com](../ci_scale/index.md) in the upcoming years. +We also expect a [significant growth in the number of builds executed on GitLab.com](../ci_scale/index.md) +in the upcoming years. ## Opportunity @@ -61,8 +61,8 @@ pipelines that are older than a few months might help us to move this data out of the primary database, to a different storage, that is more performant and cost effective. -It is already possible to prevent processing builds [that have been -archived](../../../user/admin_area/settings/continuous_integration.md#archive-jobs). +It is already possible to prevent processing builds +[that have been archived](../../../user/admin_area/settings/continuous_integration.md#archive-jobs). When a build gets archived it will not be possible to retry it, but we still do keep all the processing metadata in the database, and it consumes resources that are scarce in the primary database. diff --git a/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md index 60b20c50696..868dae4fc6c 100644 --- a/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md +++ b/doc/architecture/blueprints/ci_data_decay/pipeline_partitioning.md @@ -306,9 +306,19 @@ We also need to build a proof of concept for removing data on the PostgreSQL side (using foreign keys with `ON DELETE CASCADE`) and removing data through Rails associations, as this might be an important area of uncertainty. -We need to [better understand](https://gitlab.com/gitlab-org/gitlab/-/issues/360148) -how unique constraints we are currently using will perform when using the -partitioned schema. +We [learned](https://gitlab.com/gitlab-org/gitlab/-/issues/360148) that `PostgreSQL` +does not allow to create a single index (unique or otherwise) across all partitions of a table. + +One solution to solve this problem is to embed the partitioning key inside the uniqueness constraint. + +This might mean prepending the partition ID in a hexadecimal format before the token itself and storing +the concatenated string in a database. To do that we would need to reserve an appropriate number of +leading bytes in a token to accommodate for the maximum number of partitions we may have in the future. +It seems that reserving four characters, what would translate into 16-bits number in base-16, +might be sufficient. The maximum number we can encode this way would be FFFF, what is 65535 in decimal. + +This would provide a unique constraint per-partition which +is sufficient for global uniqueness. We have also designed a query analyzer that makes it possible to detect direct usage of zero partitions, legacy tables that have been attached as first diff --git a/doc/architecture/blueprints/ci_scale/index.md b/doc/architecture/blueprints/ci_scale/index.md index 1c3aee2f860..5822ae2b5ed 100644 --- a/doc/architecture/blueprints/ci_scale/index.md +++ b/doc/architecture/blueprints/ci_scale/index.md @@ -89,8 +89,8 @@ environment. We also expect a significant, exponential growth in the upcoming years. -One of the forecasts done using [Facebook's -Prophet](https://facebook.github.io/prophet/) shows that in the first half of +One of the forecasts done using [Facebook's Prophet](https://facebook.github.io/prophet/) +shows that in the first half of 2024 we expect seeing 20M builds created on GitLab.com each day. In comparison to around 2M we see created today, this is 10x growth our product might need to sustain in upcoming years. @@ -115,17 +115,14 @@ of the CI/CD Apdex score, and sometimes even causes a significant performance degradation in the production environment. There are multiple other strategies that can improve performance and -reliability. We can use [Redis -queuing](https://gitlab.com/gitlab-org/gitlab/-/issues/322972), or [a separate -table that will accelerate SQL queries used to build -queues](https://gitlab.com/gitlab-org/gitlab/-/issues/322766) and we want to -explore them. - -**Status**: As of October 2021 the new architecture [has been implemented on -GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/5909#note_680407908). -The following epic tracks making it generally available: [Make the new pending -builds architecture generally available]( -https://gitlab.com/groups/gitlab-org/-/epics/6954). +reliability. We can use [Redis queuing](https://gitlab.com/gitlab-org/gitlab/-/issues/322972), or +[a separate table that will accelerate SQL queries used to build queues](https://gitlab.com/gitlab-org/gitlab/-/issues/322766) +and we want to explore them. + +**Status**: As of October 2021 the new architecture +[has been implemented on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/5909#note_680407908). +The following epic tracks making it generally available: +[Make the new pending builds architecture generally available](https://gitlab.com/groups/gitlab-org/-/epics/6954). ### Moving big amounts of data is challenging @@ -135,8 +132,7 @@ stores more than 600 gigabytes of data, and `ci_builds.yaml_variables` more than 300 gigabytes (as of February 2021). It is a lot of data that needs to be reliably moved to a different place. -Unfortunately, right now, our [background -migrations](https://docs.gitlab.com/ee/development/background_migrations.html) +Unfortunately, right now, our [background migrations](../../../development/database/background_migrations.md) are not reliable enough to migrate this amount of data at scale. We need to build mechanisms that will give us confidence in moving this data between columns, tables, partitions or database shards. diff --git a/doc/architecture/blueprints/cloud_native_build_logs/index.md b/doc/architecture/blueprints/cloud_native_build_logs/index.md index 3aba10fc758..0c941e332cb 100644 --- a/doc/architecture/blueprints/cloud_native_build_logs/index.md +++ b/doc/architecture/blueprints/cloud_native_build_logs/index.md @@ -12,8 +12,8 @@ Cloud native and the adoption of Kubernetes has been recognised by GitLab to be one of the top two biggest tailwinds that are helping us grow faster as a company behind the project. -This effort is described in a more details [in the infrastructure team -handbook](https://about.gitlab.com/handbook/engineering/infrastructure/production/kubernetes/gitlab-com/). +This effort is described in a more details +[in the infrastructure team handbook](https://about.gitlab.com/handbook/engineering/infrastructure/production/kubernetes/gitlab-com/). ## Traditional build logs @@ -88,9 +88,8 @@ even tried to replace NFS with Since that time it has become apparent that the cost of operations and maintenance of a NFS cluster is significant and that if we ever decide to -migrate to Kubernetes [we need to decouple GitLab from a shared local storage -and -NFS](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/426#note_375646396). +migrate to Kubernetes +[we need to decouple GitLab from a shared local storage and NFS](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/426#note_375646396). 1. NFS might be a single point of failure 1. NFS can only be reliably scaled vertically @@ -112,12 +111,10 @@ of complexity, maintenance cost and enormous, negative impact on availability. 1. ✓ Rollout the feature into production environment incrementally The work needed to make the new architecture production ready and enabled on -GitLab.com had been tracked in [Cloud Native Build Logs on -GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/4275) epic. +GitLab.com had been tracked in [Cloud Native Build Logs on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/4275) epic. -Enabling this feature on GitLab.com is a subtask of [making the new -architecture generally -available](https://gitlab.com/groups/gitlab-org/-/epics/3791) for everyone. +Enabling this feature on GitLab.com is a subtask of +[making the new architecture generally available](https://gitlab.com/groups/gitlab-org/-/epics/3791) for everyone. ## Status diff --git a/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md b/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md index e545e8844ec..89c3a4cd6b4 100644 --- a/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md +++ b/doc/architecture/blueprints/cloud_native_gitlab_pages/index.md @@ -17,8 +17,8 @@ Cloud Native and the adoption of Kubernetes has been recognised by GitLab to be one of the top two biggest tailwinds that are helping us grow faster as a company behind the project. -This effort is described in more detail [in the infrastructure team handbook -page](https://about.gitlab.com/handbook/engineering/infrastructure/production/kubernetes/gitlab-com/). +This effort is described in more detail +[in the infrastructure team handbook page](https://about.gitlab.com/handbook/engineering/infrastructure/production/kubernetes/gitlab-com/). GitLab Pages is tightly coupled with NFS and in order to unblock Kubernetes migration a significant change to GitLab Pages' architecture is required. This @@ -55,9 +55,8 @@ even tried to replace NFS with Since that time it has become apparent that the cost of operations and maintenance of a NFS cluster is significant and that if we ever decide to -migrate to Kubernetes [we need to decouple GitLab from a shared local storage -and -NFS](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/426#note_375646396). +migrate to Kubernetes +[we need to decouple GitLab from a shared local storage and NFS](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/426#note_375646396). 1. NFS might be a single point of failure 1. NFS can only be reliably scaled vertically @@ -84,8 +83,8 @@ graph TD C -- Serves static content --> E(Visitors) ``` -This new architecture has been briefly described in [the blog -post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/) +This new architecture has been briefly described in +[the blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/) too. ## Iterations diff --git a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md index 53ea56b8724..df8686ed0aa 100644 --- a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md +++ b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md @@ -131,7 +131,7 @@ epic. The initial iteration will provide a framework to house features under `Namespaces`. Stage groups will eventually need to migrate their own features and functionality over to `Namespaces`. This may impact these features in unexpected ways. Therefore, to minimize UX debt and maintain product consistency, stage groups will have to consider a number of factors when migrating their features over to `Namespaces`: -1. **Conceptual model**: What are the current and future state conceptual models of these features ([see object modeling for designers](https://hpadkisson.medium.com/object-modeling-for-designers-an-introduction-7871bdcf8baf))? These should be documented in Pajamas (example: [merge requests](https://design.gitlab.com/objects/merge-request)). +1. **Conceptual model**: What are the current and future state conceptual models of these features ([see object modeling for designers](https://hpadkisson.medium.com/object-modeling-for-designers-an-introduction-7871bdcf8baf))? These should be documented in Pajamas (example: [merge requests](https://design.gitlab.com/objects/merge-request/)). 1. **Merge conflicts**: What inconsistencies are there across project, group, and administrator levels? How might these be addressed? For an example of how we rationalized this for labels, please see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338820). 1. **Inheritance & information flow**: How is information inherited across our container hierarchy currently? How might this be impacted if complying with the new [inheritance behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/343316) framework? 1. **Settings**: Where can settings for this feature be found currently? How will these be impacted by `Namespaces`? diff --git a/doc/architecture/blueprints/container_registry_metadata_database/index.md b/doc/architecture/blueprints/container_registry_metadata_database/index.md index 5105f2e08e2..9d40593d7ce 100644 --- a/doc/architecture/blueprints/container_registry_metadata_database/index.md +++ b/doc/architecture/blueprints/container_registry_metadata_database/index.md @@ -78,7 +78,7 @@ The single entrypoint for the registry is the [HTTP API](https://gitlab.com/gitl | Operation | UI | Background | Observations | | ------------------------------------------------------------ | ------------------ | ------------------------ | ------------------------------------------------------------ | | [Check API version](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/spec/api.md#api-version-check) | **{check-circle}** Yes | **{check-circle}** Yes | Used globally to ensure that the registry supports the Docker Distribution V2 API, as well as for identifying whether GitLab Rails is talking to the GitLab Container Registry or a third-party one (used to toggle features only available in the former). | -| [List repository tags](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/spec/api.md#listing-image-tags) | **{check-circle}** Yes | **{check-circle}** Yes | Used to list and show tags in the UI. Used to list tags in the background for [cleanup policies](../../../user/packages/container_registry/reduce_container_registry_storage.md#cleanup-policy) and [Geo replication](../../../administration/geo/replication/docker_registry.md). | +| [List repository tags](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/spec/api.md#listing-image-tags) | **{check-circle}** Yes | **{check-circle}** Yes | Used to list and show tags in the UI. Used to list tags in the background for [cleanup policies](../../../user/packages/container_registry/reduce_container_registry_storage.md#cleanup-policy) and [Geo replication](../../../administration/geo/replication/container_registry.md). | | [Check if manifest exists](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/spec/api.md#existing-manifests) | **{check-circle}** Yes | **{dotted-circle}** No | Used to get the digest of a manifest by tag. This is then used to pull the manifest and show the tag details in the UI. | | [Pull manifest](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/spec/api.md#pulling-an-image-manifest) | **{check-circle}** Yes | **{dotted-circle}** No | Used to show the image size and the manifest digest in the tag details UI. | | [Pull blob](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs/spec/api.md#pulling-a-layer) | **{check-circle}** Yes | **{dotted-circle}** No | Used to show the configuration digest and the creation date in the tag details UI. | @@ -148,7 +148,7 @@ The interaction between the registry and its clients, including GitLab Rails and Following the GitLab [Go standards and style guidelines](../../../development/go_guide), no ORM is used to manage the database, only the [`database/sql`](https://pkg.go.dev/database/sql) package from the Go standard library, a PostgreSQL driver ([`lib/pq`](https://pkg.go.dev/github.com/lib/pq?tab=doc)) and raw SQL queries, over a TCP connection pool. -The design and development of the registry database adhere to the GitLab [database guidelines](../../../development/database/). Being a Go application, the required tooling to support the database will have to be developed, such as for running database migrations. +The design and development of the registry database adhere to the GitLab [database guidelines](../../../development/database/index.md). Being a Go application, the required tooling to support the database will have to be developed, such as for running database migrations. Running *online* and [*post deployment*](../../../development/database/post_deployment_migrations.md) migrations is already supported by the registry CLI, as described in the [documentation](https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/database-migrations.md). @@ -160,7 +160,7 @@ Although blobs are shared across repositories, manifest and tag metadata are sco #### GitLab.com -Due to scale, performance and isolation concerns, for GitLab.com the registry database will be on a separate dedicated PostgreSQL cluster. Please see [#93](https://gitlab.com/gitlab-org/container-registry/-/issues/93) and [GitLab-com/gl-infra/infrastructure#10109](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10109) for additional context. +Due to scale, performance and isolation concerns, for GitLab.com the registry database will be on a separate dedicated PostgreSQL cluster. Please see [#93](https://gitlab.com/gitlab-org/container-registry/-/issues/93) and [GitLab-com/gl-infra/reliability#10109](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/10109) for additional context. The diagram below illustrates the architecture of the database cluster: @@ -272,7 +272,7 @@ The expected registry behavior will be covered with integration tests, using a p Apart from unusual network and systems conditions, problematic migrations and data failures can also affect the database availability and, as a consequence, the registry availability. -Database migrations will adhere to the same [development best practices](../../../development/database/) used for GitLab Rails, except Rails-specific methods and tools, as the registry is a Go application with no ORM, and migrations are therefore expressed with raw SQL statements. Regardless, all changes will require a review and approval from the Database team. +Database migrations will adhere to the same [development best practices](../../../development/database/index.md) used for GitLab Rails, except Rails-specific methods and tools, as the registry is a Go application with no ORM, and migrations are therefore expressed with raw SQL statements. Regardless, all changes will require a review and approval from the Database team. Database migrations will be idempotent, with guard clauses used whenever necessary. It is also intended to guarantee that they are forward compatible. For a clustered environment, a node running registry version `N` should not cause any issues when the database schema is from version `N`+1. @@ -280,9 +280,9 @@ Database migrations will be idempotent, with guard clauses used whenever necessa Adding one more component to the system makes it even more critical to guarantee that we have proper observability over the registry's behavior and its dependencies. It should be guaranteed that we have all the necessary tools in place before rolling out the new registry backed by the metadata database. -For this purpose, [error reporting with Sentry](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11297), [improved structured logging](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10933), and [improved HTTP metrics](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/10935) were already implemented and released. At the time of writing, the GitLab.com rollout is in progress. +For this purpose, [error reporting with Sentry](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/11297), [improved structured logging](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/10933), and [improved HTTP metrics](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/10935) were already implemented and released. At the time of writing, the GitLab.com rollout is in progress. -Additionally, the Prometheus metrics will be augmented with [details on the database connection pool](https://gitlab.com/gitlab-org/container-registry/-/issues/238). These will be added to the registry Grafana dashboards, joining the existing HTTP API, deployment, and storage metrics. The database cluster for the registry [will also have metrics and alerts](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11447). +Additionally, the Prometheus metrics will be augmented with [details on the database connection pool](https://gitlab.com/gitlab-org/container-registry/-/issues/238). These will be added to the registry Grafana dashboards, joining the existing HTTP API, deployment, and storage metrics. The database cluster for the registry [will also have metrics and alerts](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/11447). Together, these resources should provide an adequate level of insight into the registry's performance and behavior. @@ -343,7 +343,7 @@ A more detailed list of all tasks, as well as periodic progress updates can be f - [Proposal for continuous, on-demand online garbage collection](https://gitlab.com/gitlab-org/container-registry/-/issues/199) - [Gradual migration proposal for the GitLab.com container registry](https://gitlab.com/gitlab-org/container-registry/-/issues/191) - [Create a self-serve registry deployment](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/316) -- [Database cluster for container registry](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/11154) +- [Database cluster for container registry](https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/11154) ## Who diff --git a/doc/architecture/blueprints/database_scaling/size-limits.md b/doc/architecture/blueprints/database_scaling/size-limits.md index 375c82f8833..284f6402d3c 100644 --- a/doc/architecture/blueprints/database_scaling/size-limits.md +++ b/doc/architecture/blueprints/database_scaling/size-limits.md @@ -136,7 +136,7 @@ As such, the target size of a physical table after refactoring depends on the si There is no standard solution to reduce table sizes - there are many! 1. **Retention**: Delete unnecessary data, for example expire old and unneeded records. -1. **Remove STI**: We still use [single-table inheritance](../../../development/single_table_inheritance.md) in a few places, which is considered an anti-pattern. Redesigning this, we can split data into multiple tables. +1. **Remove STI**: We still use [single-table inheritance](../../../development/database/single_table_inheritance.md) in a few places, which is considered an anti-pattern. Redesigning this, we can split data into multiple tables. 1. **Index optimization**: Drop unnecessary indexes and consolidate overlapping indexes if possible. 1. **Optimise data types**: Review data type decisions and optimise data types where possible (example: use integer instead of text for an enum column) 1. **Partitioning**: Apply a partitioning scheme if there is a common access dimension. @@ -167,7 +167,7 @@ The [epic for `~group::database`](https://gitlab.com/groups/gitlab-org/-/epics/6 <!-- vale gitlab.Spelling = NO --> -Identifying solutions for offending tables is driven by the [GitLab Database Team](https://about.gitlab.com/handbook/engineering/development/enablement/database/) and respective stage groups. +Identifying solutions for offending tables is driven by the [GitLab Database Team](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/) and respective stage groups. | Role | Who |------------------------------|-------------------------| diff --git a/doc/architecture/blueprints/database_testing/index.md b/doc/architecture/blueprints/database_testing/index.md index 30ebd06c81f..5bc9528d568 100644 --- a/doc/architecture/blueprints/database_testing/index.md +++ b/doc/architecture/blueprints/database_testing/index.md @@ -127,7 +127,7 @@ An alternative approach we have discussed and abandoned is to "scrub" and anonym <!-- vale gitlab.Spelling = NO --> -This effort is owned and driven by the [GitLab Database Team](https://about.gitlab.com/handbook/engineering/development/enablement/database/) with support from the [GitLab.com Reliability Datastores](https://about.gitlab.com/handbook/engineering/infrastructure/team/reliability/) team. +This effort is owned and driven by the [GitLab Database Team](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/database/) with support from the [GitLab.com Reliability Datastores](https://about.gitlab.com/handbook/engineering/infrastructure/team/reliability/) team. | Role | Who |------------------------------|-------------------------| diff --git a/doc/architecture/blueprints/feature_flags_development/index.md b/doc/architecture/blueprints/feature_flags_development/index.md index 94d52ea3474..eaca7da6bd7 100644 --- a/doc/architecture/blueprints/feature_flags_development/index.md +++ b/doc/architecture/blueprints/feature_flags_development/index.md @@ -115,9 +115,9 @@ These are reason why these changes are needed: ## Iterations -This work is being done as part of dedicated epic: [Improve internal usage of -Feature Flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). This epic -describes a meta reasons for making these changes. +This work is being done as part of dedicated epic: +[Improve internal usage of Feature Flags](https://gitlab.com/groups/gitlab-org/-/epics/3551). +This epic describes a meta reasons for making these changes. ## Who diff --git a/doc/architecture/blueprints/graphql_api/index.md b/doc/architecture/blueprints/graphql_api/index.md index b3ba1ad1960..eb045de491e 100644 --- a/doc/architecture/blueprints/graphql_api/index.md +++ b/doc/architecture/blueprints/graphql_api/index.md @@ -20,8 +20,7 @@ GraphQL development and helped to surface the need of improving tooling we use to extend the new API. This document describes the work that is needed to build a stable foundation that -will support our development efforts and a large-scale usage of the [GraphQL -API](https://docs.gitlab.com/ee/api/graphql/index.html). +will support our development efforts and a large-scale usage of the [GraphQL API](../../../api/graphql/index.md). ## Summary @@ -45,12 +44,12 @@ It is an opportunity to learn from our experience in evolving the REST API, for the scale, and to apply this knowledge onto the GraphQL development efforts. We can do that by building query-to-feature correlation mechanisms, adding scalable state synchronization support and aligning GraphQL with other -architectural initiatives being executed in parallel, like [the support for -direct uploads](https://gitlab.com/gitlab-org/gitlab/-/issues/280819). +architectural initiatives being executed in parallel, like +[the support for direct uploads](https://gitlab.com/gitlab-org/gitlab/-/issues/280819). GraphQL should be secure by default. We can avoid common security mistakes by -building mechanisms that will help us to enforce [OWASP GraphQL -recommendations](https://cheatsheetseries.owasp.org/cheatsheets/GraphQL_Cheat_Sheet.html) +building mechanisms that will help us to enforce +[OWASP GraphQL recommendations](https://cheatsheetseries.owasp.org/cheatsheets/GraphQL_Cheat_Sheet.html) that are relevant to us. Understanding what are the needs of the wider community will also allow us to diff --git a/doc/architecture/blueprints/object_storage/index.md b/doc/architecture/blueprints/object_storage/index.md index 3420caa325f..b70339c8b8d 100644 --- a/doc/architecture/blueprints/object_storage/index.md +++ b/doc/architecture/blueprints/object_storage/index.md @@ -31,9 +31,9 @@ underlying implementation for shared, distributed, highly-available (HA) file storage. Over time, we have built support for object storage across the -application, solving specific problems in a [multitude of -iterations](https://about.gitlab.com/company/team/structure/working-groups/object-storage/#company-efforts-on-uploads). This -has led to increased complexity across the board, from development +application, solving specific problems in a +[multitude of iterations](https://about.gitlab.com/company/team/structure/working-groups/object-storage/#company-efforts-on-uploads). +This has led to increased complexity across the board, from development (new features and bug fixes) to installation: - New GitLab installations require the creation and configuration of @@ -67,10 +67,8 @@ has led to increased complexity across the board, from development The following is a brief description of the main directions we can take to remove the pain points affecting our object storage implementation. -This is also available as [a YouTube -video](https://youtu.be/X9V_w8hsM8E) recorded for the [Object Storage -Working -Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/). +This is also available as [a YouTube video](https://youtu.be/X9V_w8hsM8E) recorded for the +[Object Storage Working Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/). ### Simplify GitLab architecture by shipping MinIO @@ -80,8 +78,8 @@ local storage and object storage. With local storage, there is the assumption of a shared storage between components. This can be achieved by having a single box -installation, without HA, or with a NFS, which [we no longer -recommend](../../../administration/nfs.md). +installation, without HA, or with a NFS, which +[we no longer recommend](../../../administration/nfs.md). We have a testing gap on object storage. It also requires Workhorse and MinIO, which are not present in our pipelines, so too much is @@ -136,8 +134,8 @@ access to new features without infrastructure chores. Our implementation is built on top of a 3rd-party framework where every object storage client is a 3rd-party library. Unfortunately some -of them are unmaintained. [We have customers who cannot push 5GB Git -LFS objects](https://gitlab.com/gitlab-org/gitlab/-/issues/216442), +of them are unmaintained. +[We have customers who cannot push 5GB Git LFS objects](https://gitlab.com/gitlab-org/gitlab/-/issues/216442), but with such a vital feature implemented in 3rd-party libraries we are slowed down in fixing it, and we also rely on external maintainers to merge and release fixes. @@ -147,8 +145,7 @@ Before the introduction of direct upload, using the library, _"a gem that provides a simple and extremely flexible way to upload files from Ruby applications."_, was the boring solution. However this is no longer our use-case, as we upload files from -Workhorse, and we had to [patch CarrierWave's -internals](https://gitlab.com/gitlab-org/gitlab/-/issues/285597#note_452696638) +Workhorse, and we had to [patch CarrierWave's internals](https://gitlab.com/gitlab-org/gitlab/-/issues/285597#note_452696638) to support direct upload. A brief proposal covering CarrierWave removal and a new streamlined @@ -217,7 +214,7 @@ Proposal: DRIs: -The DRI for this blueprint is the [Object Storage Working -Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/). +The DRI for this blueprint is the +[Object Storage Working Group](https://about.gitlab.com/company/team/structure/working-groups/object-storage/). <!-- vale gitlab.Spelling = YES --> diff --git a/doc/architecture/blueprints/runner_scaling/index.md b/doc/architecture/blueprints/runner_scaling/index.md index c4bd8433ab3..494aaa6a641 100644 --- a/doc/architecture/blueprints/runner_scaling/index.md +++ b/doc/architecture/blueprints/runner_scaling/index.md @@ -33,8 +33,8 @@ This design choice was crucial for the GitLab Runner success. Since that time the auto-scaling feature has been used by many users and customers and enabled rapid growth of CI/CD adoption on GitLab.com. -We can not, however, continue using Docker Machine. Work on that project [was -paused in July 2018](https://github.com/docker/machine/issues/4537) and there +We can not, however, continue using Docker Machine. Work on that project +[was paused in July 2018](https://github.com/docker/machine/issues/4537) and there was no development made since that time (except for some highly important security fixes). In 2018, after Docker Machine entered the "maintenance mode", we decided to create [our own fork](https://gitlab.com/gitlab-org/ci-cd/docker-machine) @@ -76,8 +76,8 @@ mechanism with a reliable and flexible mechanism. We might be unable to build a drop-in replacement for Docker Machine, as there are presumably many reasons why it has been deprecated. It is very difficult to maintain compatibility with so many cloud providers, and it seems that Docker Machine has been deprecated -in favor of Docker Desktop, which is not a viable replacement for us. [This -issue](https://github.com/docker/roadmap/issues/245) contains a discussion +in favor of Docker Desktop, which is not a viable replacement for us. +[This issue](https://github.com/docker/roadmap/issues/245) contains a discussion about how people are using Docker Machine right now, and it seems that GitLab CI is one of the most frequent reasons for people to keep using Docker Machine. |