diff options
Diffstat (limited to 'doc/development')
23 files changed, 200 insertions, 172 deletions
diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 5cb2ddf6e52..2adca2dae28 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -12,21 +12,21 @@ New versions of GitLab are released in stable branches and the master branch is For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/release/docs/tree/master#gitlab-release-process). -Both EE and CE require some add-on components called gitlab-shell and Gitaly. These components are available from the [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. +Both EE and CE require some add-on components called GitLab Shell and Gitaly. These components are available from the [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [Gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. ## Components A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and a [GitLab API](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/api) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses redis as a non-persistent database backend for job information, meta data, and incoming jobs. -We also support deploying GitLab on Kubernetes using our [gitlab Helm chart](https://docs.gitlab.com/charts/). +We also support deploying GitLab on Kubernetes using our [GitLab Helm chart](https://docs.gitlab.com/charts/). -The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. +The GitLab web app uses PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare Git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. -When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects. +When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving Git objects. -The add-on component gitlab-shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. gitlab-shell accesses the bare repositories through Gitaly to serve git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. gitlab-shell queries the GitLab API to determine authorization and access. +The add-on component GitLab Shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. GitLab Shell accesses the bare repositories through Gitaly to serve Git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. GitLab Shell queries the GitLab API to determine authorization and access. -Gitaly executes git operations from gitlab-shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). +Gitaly executes Git operations from GitLab Shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from Git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). You may also be interested in the [production architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/). @@ -130,7 +130,7 @@ Component statuses are linked to configuration documentation for each component. | [NGINX](#nginx) | Routes requests to appropriate components, terminates SSL | [✅][nginx-omnibus] | [✅][nginx-charts] | [⚙][nginx-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⤓][nginx-source] | ❌ | CE & EE | | [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | [✅][unicorn-omnibus] | [✅][unicorn-charts] | [✅][unicorn-charts] | [✅](../user/gitlab_com/index.md#unicorn) | [⚙][unicorn-source] | [✅][gitlab-yml] | CE & EE | | [Sidekiq](#sidekiq) | Background jobs processor | [✅][sidekiq-omnibus] | [✅][sidekiq-charts] | [✅](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) | [✅](../user/gitlab_com/index.md#sidekiq) | [✅][gitlab-yml] | [✅][gitlab-yml] | CE & EE | -| [Gitaly](#gitaly) | Git RPC service for handling all git calls made by GitLab | [✅][gitaly-omnibus] | [✅][gitaly-charts] | [✅][gitaly-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][gitaly-source] | ✅ | CE & EE | +| [Gitaly](#gitaly) | Git RPC service for handling all Git calls made by GitLab | [✅][gitaly-omnibus] | [✅][gitaly-charts] | [✅][gitaly-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][gitaly-source] | ✅ | CE & EE | | [GitLab Workhorse](#gitlab-workhorse) | Smart reverse proxy, handles large HTTP requests | [✅][workhorse-omnibus] | [✅][workhorse-charts] | [✅][workhorse-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][workhorse-source] | ✅ | CE & EE | | [GitLab Shell](#gitlab-shell) | Handles `git` over SSH sessions | [✅][shell-omnibus] | [✅][shell-charts] | [✅][shell-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][shell-source] | [✅][gitlab-yml] | CE & EE | | [GitLab Pages](#gitlab-pages) | Hosts static websites | [⚙][pages-omnibus] | [❌][pages-charts] | [❌][pages-charts] | [✅](../user/gitlab_com/index.md#gitlab-pages) | [⚙][pages-source] | [⚙][pages-gdk] | CE & EE | @@ -185,7 +185,7 @@ GitLab can be considered to have two layers from a process perspective: - Layer: Monitoring - Process: `alertmanager` -[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by Prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. +[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by Prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue #45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. #### Certificate management @@ -223,12 +223,12 @@ Elasticsearch is a distributed RESTful search engine built for the cloud. Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly). -#### Gitlab Geo +#### GitLab Geo - Configuration: [Omnibus][geo-omnibus], [Charts][geo-charts], [GDK][geo-gdk] - Layer: Core Service (Processor) -#### Gitlab Monitor +#### GitLab Monitor - [Project page](https://gitlab.com/gitlab-org/gitlab-monitor) - Configuration: [Omnibus][gitlab-monitor-omnibus], [Charts][gitlab-monitor-charts] @@ -237,7 +237,7 @@ Gitaly is a service designed by GitLab to remove our need for NFS for Git storag GitLab Monitor is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor). -#### Gitlab Pages +#### GitLab Pages - Configuration: [Omnibus][pages-omnibus], [Charts][pages-charts], [Source][pages-source], [GDK][pages-gdk] - Layer: Core Service (Processor) @@ -246,7 +246,7 @@ GitLab Pages is a feature that allows you to publish static websites directly fr You can use it either for personal or business websites, such as portfolios, documentation, manifestos, and business presentations. You can also attribute any license to your content. -#### Gitlab Runner +#### GitLab Runner - [Project page](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/README.md) - Configuration: [Omnibus][runner-omnibus], [Charts][runner-charts], [Source][runner-source], [GDK][runner-gdk] @@ -256,7 +256,7 @@ GitLab Runner runs tests and sends the results to GitLab. GitLab CI is the open-source continuous integration service included with GitLab that coordinates the testing. The old name of this project was GitLab CI Multi Runner but please use "GitLab Runner" (without CI) from now on. -#### Gitlab Shell +#### GitLab Shell - [Project page](https://gitlab.com/gitlab-org/gitlab-shell/blob/master/README.md) - Configuration: [Omnibus][shell-omnibus], [Charts][shell-charts], [Source][shell-source], [GDK][gitlab-yml] @@ -264,7 +264,7 @@ GitLab CI is the open-source continuous integration service included with GitLab [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) is a program designed at GitLab to handle ssh-based `git` sessions, and modifies the list of authorized keys. GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh. -#### Gitlab Workhorse +#### GitLab Workhorse - [Project page](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md) - Configuration: [Omnibus][workhorse-omnibus], [Charts][workhorse-charts], [Source][workhorse-source] @@ -475,7 +475,7 @@ It's important to understand the distinction as some processes are used in both When making a request to an HTTP Endpoint (think `/users/sign_in`) the request will take the following path through the GitLab Service: - nginx - Acts as our first line reverse proxy. -- gitlab-workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn. +- GitLab Workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn. - unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn. - Postgres/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data. @@ -493,13 +493,13 @@ TODO ## System Layout -When referring to `~git` in the pictures it means the home directory of the git user which is typically `/home/git`. +When referring to `~git` in the pictures it means the home directory of the Git user which is typically `/home/git`. GitLab is primarily installed within the `/home/git` user home directory as `git` user. Within the home directory is where the gitlabhq server software resides as well as the repositories (though the repository location is configurable). The bare repositories are located in `/home/git/repositories`. GitLab is a ruby on rails application so the particulars of the inner workings can be learned by studying how a ruby on rails application works. -To serve repositories over SSH there's an add-on application called gitlab-shell which is installed in `/home/git/gitlab-shell`. +To serve repositories over SSH there's an add-on application called GitLab Shell which is installed in `/home/git/gitlab-shell`. ### Installation Folder Summary @@ -523,7 +523,7 @@ processes: `unicorn_rails master` (1 process), `unicorn_rails worker` ### Repository access -Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull utilizes the GitLab API and SSH cloning is handled by gitlab-shell (previously explained). +Repositories get accessed via HTTP or SSH. HTTP cloning/push/pull utilizes the GitLab API and SSH cloning is handled by GitLab Shell (previously explained). ## Troubleshooting @@ -531,28 +531,28 @@ See the README for more information. ### Init scripts of the services -The GitLab init script starts and stops Unicorn and Sidekiq. +The GitLab init script starts and stops Unicorn and Sidekiq: ``` /etc/init.d/gitlab Usage: service gitlab {start|stop|restart|reload|status} ``` -Redis (key-value store/non-persistent database) +Redis (key-value store/non-persistent database): ``` /etc/init.d/redis Usage: /etc/init.d/redis {start|stop|status|restart|condrestart|try-restart} ``` -SSH daemon +SSH daemon: ``` /etc/init.d/sshd Usage: /etc/init.d/sshd {start|stop|restart|reload|force-reload|condrestart|try-restart|status} ``` -Web server (one of the following) +Web server (one of the following): ``` /etc/init.d/httpd @@ -562,7 +562,7 @@ $ /etc/init.d/nginx Usage: nginx {start|stop|restart|reload|force-reload|status|configtest} ``` -Persistent database +Persistent database: ``` $ /etc/init.d/postgresql @@ -571,34 +571,34 @@ Usage: /etc/init.d/postgresql {start|stop|restart|reload|force-reload|status} [v ### Log locations of the services -gitlabhq (includes Unicorn and Sidekiq logs) +gitlabhq (includes Unicorn and Sidekiq logs): - `/home/git/gitlab/log/` contains `application.log`, `production.log`, `sidekiq.log`, `unicorn.stdout.log`, `git_json.log` and `unicorn.stderr.log` normally. -gitlab-shell +GitLab Shell: - `/home/git/gitlab-shell/gitlab-shell.log` -ssh +SSH: - `/var/log/auth.log` auth log (on Ubuntu). - `/var/log/secure` auth log (on RHEL). -nginx +nginx: - `/var/log/nginx/` contains error and access logs. -Apache httpd +Apache httpd: - [Explanation of Apache logs](https://httpd.apache.org/docs/2.2/logs.html). - `/var/log/apache2/` contains error and output logs (on Ubuntu). - `/var/log/httpd/` contains error and output logs (on RHEL). -redis +Redis: - `/var/log/redis/redis.log` there are also log-rotated logs there. -PostgreSQL +PostgreSQL: - `/var/log/postgresql/*` @@ -610,7 +610,7 @@ GitLab has configuration files located in `/home/git/gitlab/config/*`. Commonly - `unicorn.rb` - Unicorn web server settings. - `database.yml` - Database connection settings. -gitlab-shell has a configuration file at `/home/git/gitlab-shell/config.yml`. +GitLab Shell has a configuration file at `/home/git/gitlab-shell/config.yml`. ### Maintenance Tasks diff --git a/doc/development/automatic_ce_ee_merge.md b/doc/development/automatic_ce_ee_merge.md index 158606aa6a2..c2700461467 100644 --- a/doc/development/automatic_ce_ee_merge.md +++ b/doc/development/automatic_ce_ee_merge.md @@ -174,9 +174,9 @@ Now, every time you create an MR for CE and EE: ## How we run the Automatic CE->EE merge at GitLab At GitLab, we use the [Merge Train](https://gitlab.com/gitlab-org/merge-train) -project to keep our [gitlab-ee](https://gitlab.com/gitlab-org/gitlab-ee) +project to keep our [GitLab EE](https://gitlab.com/gitlab-org/gitlab-ee) repository updated with commits from -[gitlab-ce](https://gitlab.com/gitlab-org/gitlab-ce). +[GitLab CE](https://gitlab.com/gitlab-org/gitlab-ce). We have a mirror of the [Merge Train](https://gitlab.com/gitlab-org/merge-train) project [configured](https://ops.gitlab.net/gitlab-org/merge-train) to run an diff --git a/doc/development/background_migrations.md b/doc/development/background_migrations.md index 642dac614c7..3fd95537eaa 100644 --- a/doc/development/background_migrations.md +++ b/doc/development/background_migrations.md @@ -294,7 +294,7 @@ to migrate you database down and up, which can result in other background migrations being called. That means that using `spy` test doubles with `have_received` is encouraged, instead of using regular test doubles, because your expectations defined in a `it` block can conflict with what is being -called in RSpec hooks. See [gitlab-org/gitlab-ce#35351][issue-rspec-hooks] +called in RSpec hooks. See [issue #35351][issue-rspec-hooks] for more details. ## Best practices diff --git a/doc/development/build_test_package.md b/doc/development/build_test_package.md index c5f6adfeaeb..21891f70d73 100644 --- a/doc/development/build_test_package.md +++ b/doc/development/build_test_package.md @@ -3,7 +3,7 @@ While developing a new feature or modifying an existing one, it is helpful if an installable package (or a docker image) containing those changes is available for testing. For this very purpose, a manual job is provided in the GitLab CI/CD -pipeline that can be used to trigger a pipeline in the omnibus-gitlab repository +pipeline that can be used to trigger a pipeline in the Omnibus GitLab repository that will create: - A deb package for Ubuntu 16.04, available as a build artifact, and @@ -12,7 +12,7 @@ that will create: (images titled `gitlab-ce` and `gitlab-ee` respectively and image tag is the commit which triggered the pipeline). -When you push a commit to either the gitlab-ce or gitlab-ee project, the +When you push a commit to either the GitLab CE or GitLab EE project, the pipeline for that commit will have a `build-package` manual action you can trigger. @@ -30,9 +30,9 @@ branch `0-1-stable`, modify the content of `GITALY_SERVER_VERSION` to `0-1-stable` and push the commit. This will create a manual job that can be used to trigger the build. -## Specifying the branch in omnibus-gitlab repository +## Specifying the branch in Omnibus GitLab repository -In scenarios where a configuration change is to be introduced and omnibus-gitlab +In scenarios where a configuration change is to be introduced and Omnibus GitLab repository already has the necessary changes in a specific branch, you can build a package against that branch through an environment variable named `OMNIBUS_BRANCH`. To do this, specify that environment variable with the name of diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index eb3b227473b..6c9fa983c96 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -9,7 +9,7 @@ An easy first step is to search for your error in Slack or google "GitLab (my er Available `RAILS_ENV` -- `production` (generally not for your main GDK db, but you may need this for e.g. omnibus) +- `production` (generally not for your main GDK db, but you may need this for e.g. Omnibus) - `development` (this is your main GDK db) - `test` (used for tests like rspec) diff --git a/doc/development/database_review.md b/doc/development/database_review.md index 3f1b359cb0b..367a481ee11 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -91,7 +91,7 @@ and details for a database reviewer: concurrent index/foreign key helpers (with transactions disabled) - Check consistency with `db/schema.rb` and that migrations are [reversible](migration_style_guide.md#reversibility) - Check queries timing (If any): Queries executed in a migration - need to fit comfortable within `15s` - preferably much less than that - on GitLab.com. + need to fit comfortably within `15s` - preferably much less than that - on GitLab.com. - Check [background migrations](background_migrations.md): - For data migrations, establish a time estimate for execution - They should only be used when migrating data in larger tables. diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index a732a94b7c4..1358851f3cd 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -910,7 +910,7 @@ import bundle from 'ee_else_ce/protected_branches/protected_branches_bundle.js'; ``` See the frontend guide [performance section](fe_guide/performance.md) for -information on managing page-specific javascript within EE. +information on managing page-specific JavaScript within EE. ## Vue code in `assets/javascript` @@ -1057,7 +1057,7 @@ Here is a workflow to make sure those changes end up backported safely into CE t **Note:** regarding SCSS, make sure the files living outside `/ee/` don't diverge between CE and EE projects. -## gitlab-svgs +## GitLab-svgs Conflicts in `app/assets/images/icons.json` or `app/assets/images/icons.svg` can be resolved simply by regenerating those assets with diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 090e5235619..f2412c249c1 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -40,9 +40,11 @@ There is no need to install any plugins If you're interested on working with the new beta repo indexer, all you need to do is: -- git clone git@gitlab.com:gitlab-org/gitlab-elasticsearch-indexer.git -- make -- make install +```sh +git clone git@gitlab.com:gitlab-org/gitlab-elasticsearch-indexer.git +make +make install +``` this adds `gitlab-elasticsearch-indexer` to `$GOPATH/bin`, please make sure that is in your `$PATH`. After that GitLab will find it and you'll be able to enable it in the admin settings area. @@ -188,7 +190,7 @@ The global configurations per version are now in the `Elastic::(Version)::Config NOTE: **Note:** this is not applicable yet as multiple indices functionality is not fully implemented. -Folders like `ee/lib/elastic/v12p1` contain snapshots of search logic from different versions. To keep a continuous git history, the latest version lives under `ee/lib/elastic/latest`, but its classes are aliased under an actual version (e.g. `ee/lib/elastic/v12p3`). When referencing these classes, never use the `Latest` namespace directly, but use the actual version (e.g. `V12p3`). +Folders like `ee/lib/elastic/v12p1` contain snapshots of search logic from different versions. To keep a continuous Git history, the latest version lives under `ee/lib/elastic/latest`, but its classes are aliased under an actual version (e.g. `ee/lib/elastic/v12p3`). When referencing these classes, never use the `Latest` namespace directly, but use the actual version (e.g. `V12p3`). The version name basically follows GitLab's release version. If setting is changed in 12.3, we will create a new namespace called `V12p3` (p stands for "point"). Raise an issue if there is a need to name a version differently. diff --git a/doc/development/emails.md b/doc/development/emails.md index edec0f86989..5676c3b32f4 100644 --- a/doc/development/emails.md +++ b/doc/development/emails.md @@ -6,7 +6,7 @@ To view rendered emails "sent" in your development instance, visit [`/rails/letter_opener`](http://localhost:3000/rails/letter_opener). Please note that [S/MIME signed](../administration/smime_signing_email.md) emails -[cannot be currently previewed](https://github.com/fgrehm/letter_opener_web/issues/96) with +[cannot be currently previewed](https://github.com/fgrehm/letter_opener_web/issues/96) with `letter_opener`. ## Mailer previews diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 56872f8c075..f1374b9e280 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -8,5 +8,5 @@ disable those changes, without having to revert an entire release. Before using feature flags for GitLab's development, read through the following: - [Process for using features flags](process.md). -- [Developing with feature flags documentation](development.md). -- [Controlling feature flags documentation](controls.md). +- [Developing with feature flags](development.md). +- [Controlling feature flags](controls.md). diff --git a/doc/development/filtering_by_label.md b/doc/development/filtering_by_label.md index 5e7376db725..dd8944ff1c8 100644 --- a/doc/development/filtering_by_label.md +++ b/doc/development/filtering_by_label.md @@ -40,16 +40,14 @@ In particular, note that: This is more complicated than is ideal. It makes the query construction more prone to errors (such as -[gitlab-org/gitlab-ce#15557](https://gitlab.com/gitlab-org/gitlab-ce/issues/15557)). +[issue #15557](https://gitlab.com/gitlab-org/gitlab-ce/issues/15557)). ## Attempt A: WHERE EXISTS ### Attempt A1: use multiple subqueries with WHERE EXISTS -In -[gitlab-org/gitlab-ce#37137](https://gitlab.com/gitlab-org/gitlab-ce/issues/37137) -and its associated merge request -[gitlab-org/gitlab-ce!14022](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14022), +In [issue #37137](https://gitlab.com/gitlab-org/gitlab-ce/issues/37137) +and its associated [merge request](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14022), we tried to replace the `GROUP BY` with multiple uses of `WHERE EXISTS`. For the example above, this would give: @@ -81,12 +79,11 @@ it did not improve query performance. ## Attempt B: Denormalize using an array column -Having [removed MySQL support in GitLab -12.1](https://about.gitlab.com/2019/06/27/removing-mysql-support/), using -[Postgres's arrays](https://www.postgresql.org/docs/9.6/arrays.html) became more +Having [removed MySQL support in GitLab 12.1](https://about.gitlab.com/2019/06/27/removing-mysql-support/), +using [Postgres's arrays](https://www.postgresql.org/docs/9.6/arrays.html) became more tractable as we didn't have to support two databases. We discussed denormalizing the `label_links` table for querying in -[gitlab-org/gitlab-ce#49651](https://gitlab.com/gitlab-org/gitlab-ce/issues/49651), +[issue #49651](https://gitlab.com/gitlab-org/gitlab-ce/issues/49651), with two options: label IDs and titles. We can think of both of those as array columns on `issues`, `merge_requests`, @@ -150,8 +147,7 @@ WHERE label_titles @> ARRAY['Plan', 'backend'] ``` -And our [tests in -gitlab-org/gitlab-ce#49651](https://gitlab.com/gitlab-org/gitlab-ce/issues/49651#note_188777346) +And our [tests in issue #49651](https://gitlab.com/gitlab-org/gitlab-ce/issues/49651#note_188777346) showed that this could be fast. However, at present, the disadvantages outweigh the advantages. diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index ec9718cea71..8d93c52e7bc 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -3,9 +3,9 @@ When adding a new entry to `Gemfile` or upgrading an existing dependency pay attention to the following rules. -## No gems fetched from git repositories +## No gems fetched from Git repositories -We do not allow gems that are fetched from git repositories. All gems have +We do not allow gems that are fetched from Git repositories. All gems have to be available in the RubyGems index. We want to minimize external build dependencies and build times. diff --git a/doc/development/geo.md b/doc/development/geo.md index 24f16eae9fa..cc3e2d1ccc5 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -170,7 +170,7 @@ while `pull` requests will continue to be served by the **secondary** node for m HTTPS and SSH requests are handled differently: - With HTTPS, we will give the user a `HTTP 302 Redirect` pointing to the project on the **primary** node. - The git client is wise enough to understand that status code and process the redirection. + The Git client is wise enough to understand that status code and process the redirection. - With SSH, because there is no equivalent way to perform a redirect, we have to proxy the request. This is done inside [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell), by first translating the request to the HTTP protocol, and then proxying it to the **primary** node. diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index 5ce59891afa..4dd1edf9b5a 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -8,30 +8,6 @@ storage disk use. To counteract this problem, we are adding Git object deduplication for forks to GitLab. In this document, we will describe how GitLab implements Git object deduplication. -## Enabling Git object deduplication via feature flags - -As of GitLab 12.0, Git object deduplication in GitLab is still behind a -feature flag. In this document, you can read about the effects of -enabling the feature. Also, note that Git object deduplication is -limited to forks of public projects on hashed repository storage. - -You can enable deduplication globally by setting the `object_pools` -feature flag to `true`: - -``` {.ruby} -Feature.enable(:object_pools) -``` - -Or just for forks of a specific project: - -``` {.ruby} -fork_parent = Project.find(MY_PROJECT_ID) -Feature.enable(:object_pools, fork_parent) -``` - -To check if a project uses Git object deduplication, look in a Rails -console if `project.pool_repository` is present. - ## Pool repositories ### Understanding Git alternates @@ -193,7 +169,7 @@ There are three different things that can go wrong here. In this case, we miss out on disk space savings but all RPC's on A itself will function fine. The next time garbage collection runs on A, the alternates connection gets established in Gitaly. This is done by -`Projects::GitDeduplicationService` in gitlab-rails. +`Projects::GitDeduplicationService` in GitLab Rails. #### 2. SQL says repo A belongs to pool P1 but Gitaly says A has alternate objects in pool P2 diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 2ade59b76ed..592fc13873b 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -45,13 +45,13 @@ The process for adding new Gitaly features is: - release a new version of gitaly-proto - write implementation and tests for the RPC [in Gitaly](https://gitlab.com/gitlab-org/gitaly), in Go or Ruby - release a new version of Gitaly -- write client code in gitlab-ce/ee, gitlab-workhorse or gitlab-shell that calls the new Gitaly RPC +- write client code in GitLab CE/EE, GitLab Workhorse or GitLab Shell that calls the new Gitaly RPC These steps often overlap. It is possible to use an unreleased version of Gitaly and gitaly-proto during testing and development. - See the [Gitaly repo](https://gitlab.com/gitlab-org/gitaly/blob/master/CONTRIBUTING.md#development-and-testing-with-a-custom-gitaly-proto) for instructions on writing server side code with an unreleased protocol. -- See [below](#running-tests-with-a-locally-modified-version-of-gitaly) for instructions on running gitlab-ce tests with a modified version of Gitaly. +- See [below](#running-tests-with-a-locally-modified-version-of-gitaly) for instructions on running GitLab CE tests with a modified version of Gitaly. - In GDK run `gdk install` and restart `gdk run` (or `gdk run app`) to use a locally modified Gitaly version for development ### Gitaly-ruby @@ -146,7 +146,7 @@ Once the code is wrapped in this block, this code-path will be excluded from n+1 ## Request counts -Commits and other git data, is now fetched through Gitaly. These fetches can, +Commits and other Git data, is now fetched through Gitaly. These fetches can, much like with a database, be batched. This improves performance for the client and for Gitaly itself and therefore for the users too. To keep performance stable and guard performance regressions, Gitaly calls can be counted and the call count @@ -164,10 +164,10 @@ end ## Running tests with a locally modified version of Gitaly -Normally, gitlab-ce/ee tests use a local clone of Gitaly in +Normally, GitLab CE/EE tests use a local clone of Gitaly in `tmp/tests/gitaly` pinned at the version specified in `GITALY_SERVER_VERSION`. The `GITALY_SERVER_VERSION` file supports -`=my-branch` syntax to use a custom branch in gitlab-org/gitaly. If +`=my-branch` syntax to use a custom branch in <https://gitlab.com/gitlab-org/gitaly>. If you want to run tests locally against a modified version of Gitaly you can replace `tmp/tests/gitaly` with a symlink. This is much faster because the `=my-branch` syntax forces a Gitaly re-install each time @@ -276,9 +276,9 @@ Here are the steps to gate a new feature in Gitaly behind a feature flag. require.NoError(t, err) ``` -### Gitlab-Rails +### GitLab Rails -1. Add feature flag to `lib/gitlab/gitaly_client.rb` (in gitlab-rails): +1. Add feature flag to `lib/gitlab/gitaly_client.rb` (in GitLab Rails): ```ruby SERVER_FEATURE_FLAGS = %w[go-find-all-tags].freeze diff --git a/doc/development/kubernetes.md b/doc/development/kubernetes.md index 4b2d48903ac..f4528667814 100644 --- a/doc/development/kubernetes.md +++ b/doc/development/kubernetes.md @@ -107,7 +107,7 @@ Mitigation strategies include: ## Debugging Logs related to the Kubernetes integration can be found in -[kubernetes.log](../administration/logs.md#kuberneteslog). On a local +[`kubernetes.log`](../administration/logs.md#kuberneteslog). On a local GDK install, this will be present in `log/kubernetes.log`. Some services such as diff --git a/doc/development/logging.md b/doc/development/logging.md index 4f63c84fc0e..b43f1029cc6 100644 --- a/doc/development/logging.md +++ b/doc/development/logging.md @@ -133,7 +133,7 @@ importer progresses. Here's what to do: logs in `/var/log/gitlab/gitlab-rails/*.log` every hour and [keep at most 30 compressed files](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate). On GitLab.com, that setting is only 6 compressed files. These settings should suffice - for most users, but you may need to tweak them in [omnibus-gitlab](https://gitlab.com/gitlab-org/omnibus-gitlab). + for most users, but you may need to tweak them in [Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab). 1. If you add a new file, submit an issue to the [production tracker](https://gitlab.com/gitlab-com/gl-infra/production/issues) or diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 3181b3a88cc..4740cf4de7b 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -1,12 +1,12 @@ # Migration Style Guide When writing migrations for GitLab, you have to take into account that -these will be ran by hundreds of thousands of organizations of all sizes, some with +these will be run by hundreds of thousands of organizations of all sizes, some with many years of data in their database. In addition, having to take a server offline for an upgrade small or big is a -big burden for most organizations. For this reason it is important that your -migrations are written carefully, can be applied online and adhere to the style +big burden for most organizations. For this reason, it is important that your +migrations are written carefully, can be applied online, and adhere to the style guide below. Migrations are **not** allowed to require GitLab installations to be taken @@ -85,7 +85,38 @@ be possible to downgrade in case of a vulnerability or bugs. In your migration, add a comment describing how the reversibility of the migration was tested. -## Multi Threading +## Atomicity + +By default, migrations are single transaction. That is, a transaction is opened +at the beginning of the migration, and committed after all steps are processed. + +Running migrations in a single transaction makes sure that if one of the steps fails, +none of the steps will be executed, leaving the database in valid state. +Therefore, either: + +- Put all migrations in one single-transaction migration. +- If necessary, put most actions in one migration and create a separate migration + for the steps that cannot be done in a single transaction. + +For example, if you create an empty table and need to build an index for it, +it is recommended to use a regular single-transaction migration and the default +rails schema statement: [`add_index`](https://api.rubyonrails.org/v5.2/classes/ActiveRecord/ConnectionAdapters/SchemaStatements.html#method-i-add_index). +This is a blocking operation, but it won't cause problems because the table is not yet used, +and therefore it does not have any records yet. + +## Heavy operations in a single transaction + +When using a single-transaction migration, a transaction will hold on a database connection +for the duration of the migration, so you must make sure the actions in the migration +do not take too much time: In general, queries executed in a migration need to fit comfortably +within `15s` on GitLab.com. + +In case you need to insert, update, or delete a significant amount of data, you: + +- Must disable the single transaction with `disable_ddl_transaction!`. +- Should consider doing it in a [Background Migration](background_migrations.md). + +## Multi-Threading Sometimes a migration might need to use multiple Ruby threads to speed up a migration. For this to work your migration needs to include the module @@ -122,16 +153,16 @@ pool. This ensures each thread has its own connection object, and won't time out when trying to obtain one. **NOTE:** PostgreSQL has a maximum amount of connections that it allows. This -limit can vary from installation to installation. As a result it's recommended -you do not use more than 32 threads in a single migration. Usually 4-8 threads +limit can vary from installation to installation. As a result, it's recommended +you do not use more than 32 threads in a single migration. Usually, 4-8 threads should be more than enough. ## Removing indexes -When removing an index make sure to use the method `remove_concurrent_index` instead -of the regular `remove_index` method. The `remove_concurrent_index` method -automatically drops concurrent indexes when using PostgreSQL, removing the -need for downtime. To use this method you must disable single-transaction mode +If the table is not empty when removing an index, make sure to use the method +`remove_concurrent_index` instead of the regular `remove_index` method. +The `remove_concurrent_index` method drops indexes concurrently, so no locking is required, +and there is no need for downtime. To use this method, you must disable single-transaction mode by calling the method `disable_ddl_transaction!` in the body of your migration class like so: @@ -149,19 +180,25 @@ end Note that it is not necessary to check if the index exists prior to removing it. +For a small table (such as an empty one or one with less than `1,000` records), +it is recommended to use `remove_index` in a single-transaction migration, +combining it with other operations that don't require `disable_ddl_transaction!`. + ## Adding indexes -If you need to add a unique index please keep in mind there is the possibility +If you need to add a unique index, please keep in mind there is the possibility of existing duplicates being present in the database. This means that should always _first_ add a migration that removes any duplicates, before adding the unique index. -When adding an index make sure to use the method `add_concurrent_index` instead -of the regular `add_index` method. The `add_concurrent_index` method -automatically creates concurrent indexes when using PostgreSQL, removing the -need for downtime. To use this method you must disable transactions by calling -the method `disable_ddl_transaction!` in the body of your migration class like -so: +When adding an index to a non-empty table make sure to use the method +`add_concurrent_index` instead of the regular `add_index` method. +The `add_concurrent_index` method automatically creates concurrent indexes +when using PostgreSQL, removing the need for downtime. + +To use this method, you must disable single-transactions mode +by calling the method `disable_ddl_transaction!` in the body of your migration +class like so: ```ruby class MyMigration < ActiveRecord::Migration[4.2] @@ -179,16 +216,20 @@ class MyMigration < ActiveRecord::Migration[4.2] end ``` +For a small table (such as an empty one or one with less than `1,000` records), +it is recommended to use `add_index` in a single-transaction migration, combining it with other +operations that don't require `disable_ddl_transaction!`. + ## Adding foreign-key constraints -When adding a foreign-key constraint to either an existing or new -column remember to also add a index on the column. +When adding a foreign-key constraint to either an existing or a new column also +remember to add an index on the column. This is **required** for all foreign-keys, e.g., to support efficient cascading deleting: when a lot of rows in a table get deleted, the referenced records need to be deleted too. The database has to look for corresponding records in the referenced table. Without an index, this will result in a sequential scan on the -table which can take a long time. +table, which can take a long time. Here's an example where we add a new column with a foreign key constraint. Note it includes `index: true` to create an index for it. @@ -202,13 +243,17 @@ class Migration < ActiveRecord::Migration[4.2] end ``` -When adding a foreign-key constraint to an existing column, we -have to employ `add_concurrent_foreign_key` and `add_concurrent_index` +When adding a foreign-key constraint to an existing column in a non-empty table, +we have to employ `add_concurrent_foreign_key` and `add_concurrent_index` instead of `add_reference`. +For an empty table (such as a fresh one), it is recommended to use +`add_reference` in a single-transaction migration, combining it with other +operations that don't require `disable_ddl_transaction!`. + ## Adding Columns With Default Values -When adding columns with default values you must use the method +When adding columns with default values to non-empty tables, you must use `add_column_with_default`. This method ensures the table is updated without requiring downtime. This method is not reversible so you must manually define the `up` and `down` methods in your migration class. @@ -232,10 +277,14 @@ end ``` Keep in mind that this operation can easily take 10-15 minutes to complete on -larger installations (e.g. GitLab.com). As a result you should only add default -values if absolutely necessary. There is a RuboCop cop that will fail if this -method is used on some tables that are very large on GitLab.com, which would -cause other issues. +larger installations (e.g. GitLab.com). As a result, you should only add +default values if absolutely necessary. There is a RuboCop cop that will fail if +this method is used on some tables that are very large on GitLab.com, which +would cause other issues. + +For a small table (such as an empty one or one with less than `1,000` records), +use `add_column` and `change_column_default` in a single-transaction migration, +combining it with other operations that don't require `disable_ddl_transaction!`. ## Updating an existing column @@ -253,8 +302,10 @@ update_column_in_batches(:projects, :foo, 10) do |table, query| end ``` -To perform a computed update, the value can be wrapped in `Arel.sql`, so Arel -treats it as an SQL literal. The below example is the same as the one above, but +If a computed update is needed, the value can be wrapped in `Arel.sql`, so Arel +treats it as an SQL literal. It's also a required deprecation for [Rails 6](https://gitlab.com/gitlab-org/gitlab-ce/issues/61451). + +The below example is the same as the one above, but the value is set to the product of the `bar` and `baz` columns: ```ruby @@ -275,12 +326,12 @@ staging environment - or asking someone else to do so for you - beforehand. By default, an integer column can hold up to a 4-byte (32-bit) number. That is a max value of 2,147,483,647. Be aware of this when creating a column that will -hold file sizes in byte units. If you are tracking file size in bytes this +hold file sizes in byte units. If you are tracking file size in bytes, this restricts the maximum file size to just over 2GB. To allow an integer column to hold up to an 8-byte (64-bit) number, explicitly set the limit to 8-bytes. This will allow the column to hold a value up to -9,223,372,036,854,775,807. +`9,223,372,036,854,775,807`. Rails migration example: @@ -294,9 +345,11 @@ add_column(:projects, :foo, :integer, default: 10, limit: 8) ## Timestamp column type -By default, Rails uses the `timestamp` data type that stores timestamp data without timezone information. -The `timestamp` data type is used by calling either the `add_timestamps` or the `timestamps` method. -Also Rails converts the `:datetime` data type to the `timestamp` one. +By default, Rails uses the `timestamp` data type that stores timestamp data +without timezone information. The `timestamp` data type is used by calling +either the `add_timestamps` or the `timestamps` method. + +Also, Rails converts the `:datetime` data type to the `timestamp` one. Example: @@ -317,14 +370,16 @@ def up end ``` -Instead of using these methods one should use the following methods to store timestamps with timezones: +Instead of using these methods, one should use the following methods to store +timestamps with timezones: - `add_timestamps_with_timezone` - `timestamps_with_timezone` -This ensures all timestamps have a time zone specified. This in turn means existing timestamps won't -suddenly use a different timezone when the system's timezone changes. It also makes it very clear which -timezone was used in the first place. +This ensures all timestamps have a time zone specified. This, in turn, means +existing timestamps won't suddenly use a different timezone when the system's +timezone changes. It also makes it very clear which timezone was used in the +first place. ## Storing JSON in database @@ -359,7 +414,7 @@ Make sure your migration can be reversed. ## Data migration Please prefer Arel and plain SQL over usual ActiveRecord syntax. In case of -using plain SQL you need to quote all input manually with `quote_string` helper. +using plain SQL, you need to quote all input manually with `quote_string` helper. Example with Arel: @@ -384,7 +439,7 @@ select_all("SELECT name, COUNT(id) as cnt FROM tags GROUP BY name HAVING COUNT(i end ``` -If you need more complex logic you can define and use models local to a +If you need more complex logic, you can define and use models local to a migration. For example: ```ruby @@ -395,13 +450,13 @@ class MyMigration < ActiveRecord::Migration[4.2] end ``` -When doing so be sure to explicitly set the model's table name so it's not +When doing so be sure to explicitly set the model's table name, so it's not derived from the class name or namespace. ### Renaming reserved paths -When a new route for projects is introduced that could conflict with any -existing records. The path for this records should be renamed, and the +When a new route for projects is introduced, it could conflict with any +existing records. The path for these records should be renamed, and the related data should be moved on disk. Since we had to do this a few times already, there are now some helpers to help diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md index b3285de4705..2c7e5935435 100644 --- a/doc/development/namespaces_storage_statistics.md +++ b/doc/development/namespaces_storage_statistics.md @@ -20,8 +20,8 @@ every time the project is saved. The summary of those statistics per namespace is then retrieved by [`Namespaces#with_statistics`](https://gitlab.com/gitlab-org/gitlab-ce/blob/v12.2.0.pre/app/models/namespace.rb#L70) scope. Analyzing this query we noticed that: -* It takes up to `1.2` seconds for namespaces with over `15k` projects. -* It can't be analyzed with [ChatOps](chatops_on_gitlabcom.md), as it times out. +- It takes up to `1.2` seconds for namespaces with over `15k` projects. +- It can't be analyzed with [ChatOps](chatops_on_gitlabcom.md), as it times out. Additionally, the pattern that is currently used to update the project statistics (the callback) doesn't scale adequately. It is currently one of the largest @@ -62,8 +62,8 @@ REFRESH MATERIALIZED VIEW root_namespace_storage_statistics; While this implied a single query update (and probably a fast one), it has some downsides: -* Materialized views syntax varies from PostgreSQL and MySQL. While this feature was worked on, MySQL was still supported by GitLab. -* Rails does not have native support for materialized views. We'd need to use a specialized gem to take care of the management of the database views, which implies additional work. +- Materialized views syntax varies from PostgreSQL and MySQL. While this feature was worked on, MySQL was still supported by GitLab. +- Rails does not have native support for materialized views. We'd need to use a specialized gem to take care of the management of the database views, which implies additional work. ### Attempt B: An update through a CTE @@ -131,8 +131,8 @@ WHERE namespace_id IN ( Even though this approach would make aggregating much easier, it has some major downsides: -* We'd have to migrate **all namespaces** by adding and filling a new column. Because of the size of the table, dealing with time/cost will not be great. The background migration will take approximately `153h`, see <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29772>. -* Background migration has to be shipped one release before, delaying the functionality by another milestone. +- We'd have to migrate **all namespaces** by adding and filling a new column. Because of the size of the table, dealing with time/cost will not be great. The background migration will take approximately `153h`, see <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/29772>. +- Background migration has to be shipped one release before, delaying the functionality by another milestone. ### Attempt E (final): Update the namespace storage statistics in async way @@ -155,10 +155,10 @@ but we refresh them through Sidekiq jobs and in different transactions: This implementation has the following benefits: -* All the updates are done async, so we're not increasing the length of the transactions for `project_statistics`. -* We're doing the update in a single SQL query. -* It is compatible with PostgreSQL and MySQL. -* No background migration required. +- All the updates are done async, so we're not increasing the length of the transactions for `project_statistics`. +- We're doing the update in a single SQL query. +- It is compatible with PostgreSQL and MySQL. +- No background migration required. The only downside of this approach is that namespaces' statistics are updated up to `1.5` hours after the change is done, which means there's a time window in which the statistics are inaccurate. Because we're still not @@ -171,8 +171,8 @@ performant approach of aggregating the root namespaces. All the details regarding this use case can be found on: -* <https://gitlab.com/gitlab-org/gitlab-ce/issues/62214> -* Merge Request with the implementation: <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28996> +- <https://gitlab.com/gitlab-org/gitlab-ce/issues/62214> +- Merge Request with the implementation: <https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28996> Performance of the namespace storage statistics were measured in staging and production (GitLab.com). All results were posted on <https://gitlab.com/gitlab-org/gitlab-ce/issues/64092>: No problem has been reported so far. diff --git a/doc/development/omnibus.md b/doc/development/omnibus.md index 0ba354d28a2..ea5c18f1a8c 100644 --- a/doc/development/omnibus.md +++ b/doc/development/omnibus.md @@ -1,32 +1,32 @@ -# What you should know about omnibus packages +# What you should know about Omnibus packages -Most users install GitLab using our omnibus packages. As a developer it can be -good to know how the omnibus packages differ from what you have on your laptop +Most users install GitLab using our Omnibus packages. As a developer it can be +good to know how the Omnibus packages differ from what you have on your laptop when you are coding. ## Files are owned by root by default -All the files in the Rails tree (`app/`, `config/` etc.) are owned by 'root' in -omnibus installations. This makes the installation simpler and it provides -extra security. The omnibus reconfigure script contains commands that give -write access to the 'git' user only where needed. +All the files in the Rails tree (`app/`, `config/` etc.) are owned by `root` in +Omnibus installations. This makes the installation simpler and it provides +extra security. The Omnibus reconfigure script contains commands that give +write access to the `git` user only where needed. -For example, the 'git' user is allowed to write in the `log/` directory, in +For example, the `git` user is allowed to write in the `log/` directory, in `public/uploads`, and they are allowed to rewrite the `db/schema.rb` file. In other cases, the reconfigure script tricks GitLab into not trying to write a file. For instance, GitLab will generate a `.secret` file if it cannot find one -and write it to the Rails root. In the omnibus packages, reconfigure writes the +and write it to the Rails root. In the Omnibus packages, reconfigure writes the `.secret` file first, so that GitLab never tries to write it. ## Code, data and logs are in separate directories -The omnibus design separates code (read-only, under `/opt/gitlab`) from data +The Omnibus design separates code (read-only, under `/opt/gitlab`) from data (read/write, under `/var/opt/gitlab`) and logs (read/write, under `/var/log/gitlab`). To make this happen the reconfigure script sets custom paths where it can in GitLab config files, and where there are no path settings, it uses symlinks. For example, `config/gitlab.yml` is treated as data so that file is a symlink. -The same goes for `public/uploads`. The `log/` directory is replaced by omnibus +The same goes for `public/uploads`. The `log/` directory is replaced by Omnibus with a symlink to `/var/log/gitlab/gitlab-rails`. diff --git a/doc/development/session.md b/doc/development/session.md index 9edce3dbda0..971795d8816 100644 --- a/doc/development/session.md +++ b/doc/development/session.md @@ -17,7 +17,7 @@ When storing values in a session it is best to: - Use simple primitives and avoid storing objects to avoid marshaling complications. - Clean up after unneeded variables to keep memory usage in Redis down. -## Gitlab::Session +## GitLab::Session Sometimes you might want to persist data in the session instead of another store like the database. `Gitlab::Session` lets you access this without passing the session around extensively. For example, you could access it from within a policy without having to pass the session through to each place permissions are checked from. diff --git a/doc/development/shell_commands.md b/doc/development/shell_commands.md index 7bdf676be58..1300c99622e 100644 --- a/doc/development/shell_commands.md +++ b/doc/development/shell_commands.md @@ -35,7 +35,7 @@ Gitlab::Popen.popen(%W(find /some/path -not -path /some/path -mmin +120 -delete) This coding style could have prevented CVE-2013-4490. -## Always use the configurable git binary path for git commands +## Always use the configurable Git binary path for Git commands ```ruby # Wrong @@ -114,7 +114,7 @@ user = `whoami` user, exit_status = Gitlab::Popen.popen(%W(whoami)) ``` -In other repositories, such as gitlab-shell you can also use `IO.popen`. +In other repositories, such as GitLab Shell you can also use `IO.popen`. ```ruby # Safe IO.popen example diff --git a/doc/development/uploads.md b/doc/development/uploads.md index 234539bb673..681ce9d9fe8 100644 --- a/doc/development/uploads.md +++ b/doc/development/uploads.md @@ -174,14 +174,14 @@ sequenceDiagram c ->>+w: POST /some/url/upload w->>+s: save the incoming file on a temporary location - s-->>-w: + s-->>-w: w->>+r: POST /some/url/upload Note over w,r: file was replaced with its location<br>and other metadata opt requires async processing r->>+redis: schedule a job - redis-->>-r: + redis-->>-r: end r-->>-c: request result @@ -230,17 +230,17 @@ sequenceDiagram w->>+os: PUT file Note over w,os: file is stored on a temporary location. Rails select the destination - os-->>-w: + os-->>-w: w->>+r: POST /some/url/upload Note over w,r: file was replaced with its location<br>and other metadata r->>+os: move object to final destination - os-->>-r: + os-->>-r: opt requires async processing r->>+redis: schedule a job - redis-->>-r: + redis-->>-r: end r-->>-c: request result @@ -268,4 +268,3 @@ sequenceDiagram This option affect the response to the `/authorize` call. When not enabled, the API response will not contain presigned URLs and workhorse will write the file the shared disk, on the path is provided by rails, acting like object storage was disabled. Once the request reachs rails, it will schedule an object storage upload as a sidekiq job. - |