diff options
| author | Rémy Coutable <remy@rymai.me> | 2016-05-18 23:40:33 -0500 |
|---|---|---|
| committer | Rémy Coutable <remy@rymai.me> | 2016-05-18 23:40:33 -0500 |
| commit | f29d8b64e4bbea0dce64b4dfbae14422d64899c9 (patch) | |
| tree | eb2e0ebc962bfcc8bb04c21af274bd537e649959 /doc | |
| parent | 715959e58190eca661ea377b949af3515d8da913 (diff) | |
| parent | 4607323e130fe5b04e830f7a6de8083b070808f1 (diff) | |
| download | gitlab-ce-f29d8b64e4bbea0dce64b4dfbae14422d64899c9.tar.gz | |
Merge remote-tracking branch 'origin/master' into eReGeBe/gitlab-ce-feature/milestone-md
Signed-off-by: Rémy Coutable <remy@rymai.me>
Diffstat (limited to 'doc')
68 files changed, 1051 insertions, 185 deletions
diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 43ab153d76d..7f53915a4d7 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -58,4 +58,4 @@ to the naming scheme `GITLAB_#{name in 1_settings.rb in upper case}`. It's possible to preconfigure the GitLab docker image by adding the environment variable `GITLAB_OMNIBUS_CONFIG` to the `docker run` command. -For more information see the ['preconfigure-docker-container' section in the Omnibus documentation](http://doc.gitlab.com/omnibus/docker/#preconfigure-docker-container). +For more information see the ['preconfigure-docker-container' section in the Omnibus documentation](http://docs.gitlab.com/omnibus/docker/#preconfigure-docker-container). diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md index b1fe34ed9a1..136f570ac27 100644 --- a/doc/administration/high_availability/load_balancer.md +++ b/doc/administration/high_availability/load_balancer.md @@ -60,4 +60,4 @@ Read more on high-availability configuration: configure custom domains with custom SSL, which would not be possible if SSL was terminated at the load balancer. -[gitlab-pages]: http://doc.gitlab.com/ee/pages/administration.html +[gitlab-pages]: http://docs.gitlab.com/ee/pages/administration.html diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index e4e124e200a..49ff5d536a1 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -113,4 +113,4 @@ Read more on high-availability configuration: 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) -[udp-log-shipping]: http://doc.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only "UDP log shipping" +[udp-log-shipping]: http://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only "UDP log shipping" diff --git a/doc/api/README.md b/doc/api/README.md index ff039f1886f..27c5962decf 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -33,7 +33,7 @@ following locations: - [Build triggers](build_triggers.md) - [Build Variables](build_variables.md) - [Runners](runners.md) -- [Licenses](licenses.md) +- [Open source license templates](licenses.md) ## Authentication diff --git a/doc/api/build_triggers.md b/doc/api/build_triggers.md index 4a12e962b62..0881a7d7a90 100644 --- a/doc/api/build_triggers.md +++ b/doc/api/build_triggers.md @@ -101,8 +101,18 @@ DELETE /projects/:id/triggers/:token | Attribute | Type | required | Description | |-----------|---------|----------|--------------------------| | `id` | integer | yes | The ID of a project | -| `token` | string | yes | The `token` of a project | +| `token` | string | yes | The `token` of a trigger | ``` curl -X DELETE -H "PRIVATE_TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/1/triggers/7b9148c158980bbd9bcea92c17522d" ``` + +```json +{ + "created_at": "2015-12-23T16:25:56.760Z", + "deleted_at": "2015-12-24T12:32:20.100Z", + "last_used": null, + "token": "7b9148c158980bbd9bcea92c17522d", + "updated_at": "2015-12-24T12:32:20.100Z" +} +``` diff --git a/doc/api/commits.md b/doc/api/commits.md index 6341440c58b..57c2e1d9b87 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -12,6 +12,8 @@ GET /projects/:id/repository/commits | --------- | ---- | -------- | ----------- | | `id` | integer | yes | The ID of a project | | `ref_name` | string | no | The name of a repository branch or tag or if not given the default branch | +| `since` | string | no | Only commits after or in this date will be returned in ISO 8601 format YYYY-MM-DDTHH:MM:SSZ | +| `until` | string | no | Only commits before or in this date will be returned in ISO 8601 format YYYY-MM-DDTHH:MM:SSZ | ```bash curl -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/5/repository/commits" diff --git a/doc/api/issues.md b/doc/api/issues.md index 3e78149f442..fc7a7ae0c0c 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -77,7 +77,8 @@ Example response: "created_at" : "2016-01-04T15:31:51.081Z", "iid" : 6, "labels" : [], - "subscribed" : false + "subscribed" : false, + "user_notes_count": 1 } ] ``` @@ -154,7 +155,8 @@ Example response: "title" : "Ut commodi ullam eos dolores perferendis nihil sunt.", "updated_at" : "2016-01-04T15:31:46.176Z", "created_at" : "2016-01-04T15:31:46.176Z", - "subscribed" : false + "subscribed" : false, + "user_notes_count": 1 } ] ``` @@ -216,7 +218,8 @@ Example response: "title" : "Ut commodi ullam eos dolores perferendis nihil sunt.", "updated_at" : "2016-01-04T15:31:46.176Z", "created_at" : "2016-01-04T15:31:46.176Z", - "subscribed": false + "subscribed": false, + "user_notes_count": 1 } ``` @@ -271,7 +274,8 @@ Example response: "description" : null, "updated_at" : "2016-01-07T12:44:33.959Z", "milestone" : null, - "subscribed" : true + "subscribed" : true, + "user_notes_count": 0 } ``` @@ -329,7 +333,8 @@ Example response: "id" : 85, "assignee" : null, "milestone" : null, - "subscribed" : true + "subscribed" : true, + "user_notes_count": 0 } ``` diff --git a/doc/api/labels.md b/doc/api/labels.md index 3730c07c5a7..b857d81768e 100644 --- a/doc/api/labels.md +++ b/doc/api/labels.md @@ -165,3 +165,73 @@ Example response: "description": "Documentation" } ``` + +## Subscribe to a label + +Subscribes the authenticated user to a label to receive notifications. If the +operation is successful, status code `201` together with the updated label is +returned. If the user is already subscribed to the label, the status code `304` +is returned. If the project or label is not found, status code `404` is +returned. + +``` +POST /projects/:id/labels/:label_id/subscription +``` + +| Attribute | Type | Required | Description | +| ---------- | ----------------- | -------- | ------------------------------------ | +| `id` | integer | yes | The ID of a project | +| `label_id` | integer or string | yes | The ID or title of a project's label | + +```bash +curl -X POST -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/labels/1/subscription +``` + +Example response: + +```json +{ + "name": "Docs", + "color": "#cc0033", + "description": "", + "open_issues_count": 0, + "closed_issues_count": 0, + "open_merge_requests_count": 0, + "subscribed": true +} +``` + +## Unsubscribe from a label + +Unsubscribes the authenticated user from a label to not receive notifications +from it. If the operation is successful, status code `200` together with the +updated label is returned. If the user is not subscribed to the label, the +status code `304` is returned. If the project or label is not found, status code +`404` is returned. + +``` +DELETE /projects/:id/labels/:label_id/subscription +``` + +| Attribute | Type | Required | Description | +| ---------- | ----------------- | -------- | ------------------------------------ | +| `id` | integer | yes | The ID of a project | +| `label_id` | integer or string | yes | The ID or title of a project's label | + +```bash +curl -X DELETE -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/labels/1/subscription +``` + +Example response: + +```json +{ + "name": "Docs", + "color": "#cc0033", + "description": "", + "open_issues_count": 0, + "closed_issues_count": 0, + "open_merge_requests_count": 0, + "subscribed": false +} +``` diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 2057f9d77aa..8217e30fe25 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -67,7 +67,8 @@ Parameters: }, "merge_when_build_succeeds": true, "merge_status": "can_be_merged", - "subscribed" : false + "subscribed" : false, + "user_notes_count": 1 } ] ``` @@ -130,7 +131,8 @@ Parameters: }, "merge_when_build_succeeds": true, "merge_status": "can_be_merged", - "subscribed" : true + "subscribed" : true, + "user_notes_count": 1 } ``` @@ -230,6 +232,7 @@ Parameters: "merge_when_build_succeeds": true, "merge_status": "can_be_merged", "subscribed" : true, + "user_notes_count": 1, "changes": [ { "old_path": "VERSION", @@ -308,7 +311,8 @@ Parameters: }, "merge_when_build_succeeds": true, "merge_status": "can_be_merged", - "subscribed" : true + "subscribed" : true, + "user_notes_count": 0 } ``` @@ -378,7 +382,8 @@ Parameters: }, "merge_when_build_succeeds": true, "merge_status": "can_be_merged", - "subscribed" : true + "subscribed" : true, + "user_notes_count": 1 } ``` @@ -472,7 +477,8 @@ Parameters: }, "merge_when_build_succeeds": true, "merge_status": "can_be_merged", - "subscribed" : true + "subscribed" : true, + "user_notes_count": 1 } ``` @@ -537,7 +543,8 @@ Parameters: }, "merge_when_build_succeeds": true, "merge_status": "can_be_merged", - "subscribed" : true + "subscribed" : true, + "user_notes_count": 1 } ``` @@ -602,7 +609,8 @@ Example response: "title" : "Consequatur vero maxime deserunt laboriosam est voluptas dolorem.", "created_at" : "2016-01-04T15:31:51.081Z", "iid" : 6, - "labels" : [] + "labels" : [], + "user_notes_count": 1 }, ] ``` diff --git a/doc/api/projects.md b/doc/api/projects.md index de1faadebf5..f5f195b97df 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -424,6 +424,7 @@ Parameters: - `builds_enabled` (optional) - `wiki_enabled` (optional) - `snippets_enabled` (optional) +- `container_registry_enabled` (optional) - `public` (optional) - if `true` same as setting visibility_level = 20 - `visibility_level` (optional) - `import_url` (optional) @@ -447,6 +448,7 @@ Parameters: - `builds_enabled` (optional) - `wiki_enabled` (optional) - `snippets_enabled` (optional) +- `container_registry_enabled` (optional) - `public` (optional) - if `true` same as setting visibility_level = 20 - `visibility_level` (optional) - `import_url` (optional) @@ -472,6 +474,7 @@ Parameters: - `builds_enabled` (optional) - `wiki_enabled` (optional) - `snippets_enabled` (optional) +- `container_registry_enabled` (optional) - `public` (optional) - if `true` same as setting visibility_level = 20 - `visibility_level` (optional) - `public_builds` (optional) diff --git a/doc/api/runners.md b/doc/api/runners.md index cc6c6b7cb2f..ddfa298f79d 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -275,7 +275,7 @@ POST /projects/:id/runners | `runner_id` | integer | yes | The ID of a runner | ``` -curl -X POST -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/project/9/runners" -F "runner_id=9" +curl -X POST -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/9/runners" -F "runner_id=9" ``` Example response: @@ -306,7 +306,7 @@ DELETE /projects/:id/runners/:runner_id | `runner_id` | integer | yes | The ID of a runner | ``` -curl -X DELETE -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/project/9/runners/9" +curl -X DELETE -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/9/runners/9" ``` Example response: diff --git a/doc/api/services.md b/doc/api/services.md index 7d45b2cf463..83ac7845156 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -491,7 +491,7 @@ Jira issue tracker Set JIRA service for a project. -> Setting `project_url`, `issues_url` and `new_issue_url` will allow a user to easily navigate to the Jira issue tracker. See the [integration doc](http://doc.gitlab.com/ce/integration/external-issue-tracker.html) for details. Support for referencing commits and automatic closing of Jira issues directly from GitLab is [available in GitLab EE.](http://doc.gitlab.com/ee/integration/jira.html) +> Setting `project_url`, `issues_url` and `new_issue_url` will allow a user to easily navigate to the Jira issue tracker. See the [integration doc](http://docs.gitlab.com/ce/integration/external-issue-tracker.html) for details. Support for referencing commits and automatic closing of Jira issues directly from GitLab is [available in GitLab EE.](http://docs.gitlab.com/ee/integration/jira.html) ``` PUT /projects/:id/services/jira diff --git a/doc/api/users.md b/doc/api/users.md index 7d2b4897cff..7e848586dbd 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -20,6 +20,7 @@ GET /users "name": "John Smith", "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/cd8.jpeg", + "web_url": "http://localhost:3000/u/john_smith" }, { "id": 2, @@ -27,6 +28,7 @@ GET /users "name": "Jack Smith", "state": "blocked", "avatar_url": "http://gravatar.com/../e32131cd8.jpeg", + "web_url": "http://localhost:3000/u/jack_smith" } ] ``` @@ -45,21 +47,31 @@ GET /users "email": "john@example.com", "name": "John Smith", "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/index.jpg", + "web_url": "http://localhost:3000/u/john_smith", "created_at": "2012-05-23T08:00:58Z", + "is_admin": false, "bio": null, + "location": null, "skype": "", "linkedin": "", "twitter": "", "website_url": "", - "extern_uid": "john.smith", - "provider": "provider_name", + "last_sign_in_at": "2012-06-01T11:41:01Z", + "confirmed_at": "2012-05-23T09:05:22Z", "theme_id": 1, "color_scheme_id": 2, - "is_admin": false, - "avatar_url": "http://localhost:3000/uploads/user/avatar/1/cd8.jpeg", + "projects_limit": 100, + "current_sign_in_at": "2012-06-02T06:36:55Z", + "identities": [ + {"provider": "github", "extern_uid": "2435223452345"}, + {"provider": "bitbucket", "extern_uid": "john.smith"}, + {"provider": "google_oauth2", "extern_uid": "8776128412476123468721346"} + ], "can_create_group": true, - "current_sign_in_at": "2014-03-19T13:12:15Z", - "two_factor_enabled": true + "can_create_project": true, + "two_factor_enabled": true, + "external": false }, { "id": 2, @@ -67,24 +79,27 @@ GET /users "email": "jack@example.com", "name": "Jack Smith", "state": "blocked", + "avatar_url": "http://localhost:3000/uploads/user/avatar/2/index.jpg", + "web_url": "http://localhost:3000/u/jack_smith", "created_at": "2012-05-23T08:01:01Z", + "is_admin": false, "bio": null, "location": null, "skype": "", "linkedin": "", "twitter": "", "website_url": "", - "extern_uid": "jack.smith", - "provider": "provider_name", + "last_sign_in_at": null, + "confirmed_at": "2012-05-30T16:53:06.148Z", "theme_id": 1, "color_scheme_id": 3, - "is_admin": false, - "avatar_url": "http://localhost:3000/uploads/user/avatar/1/cd8.jpeg", - "can_create_group": true, - "can_create_project": true, "projects_limit": 100, "current_sign_in_at": "2014-03-19T17:54:13Z", - "two_factor_enabled": false + "identities": [], + "can_create_group": true, + "can_create_project": true, + "two_factor_enabled": true, + "external": false } ] ``` @@ -124,6 +139,7 @@ Parameters: "name": "John Smith", "state": "active", "avatar_url": "http://localhost:3000/uploads/user/avatar/1/cd8.jpeg", + "web_url": "http://localhost:3000/u/john_smith", "created_at": "2012-05-23T08:00:58Z", "is_admin": false, "bio": null, @@ -152,23 +168,31 @@ Parameters: "email": "john@example.com", "name": "John Smith", "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/index.jpg", + "web_url": "http://localhost:3000/u/john_smith", "created_at": "2012-05-23T08:00:58Z", - "confirmed_at": "2012-05-23T08:00:58Z", - "last_sign_in_at": "2015-03-23T08:00:58Z", + "is_admin": false, "bio": null, "location": null, "skype": "", "linkedin": "", "twitter": "", "website_url": "", - "extern_uid": "john.smith", - "provider": "provider_name", + "last_sign_in_at": "2012-06-01T11:41:01Z", + "confirmed_at": "2012-05-23T09:05:22Z", "theme_id": 1, "color_scheme_id": 2, - "is_admin": false, + "projects_limit": 100, + "current_sign_in_at": "2012-06-02T06:36:55Z", + "identities": [ + {"provider": "github", "extern_uid": "2435223452345"}, + {"provider": "bitbucket", "extern_uid": "john.smith"}, + {"provider": "google_oauth2", "extern_uid": "8776128412476123468721346"} + ], "can_create_group": true, "can_create_project": true, - "projects_limit": 100 + "two_factor_enabled": true, + "external": false } ``` @@ -261,21 +285,33 @@ GET /user "username": "john_smith", "email": "john@example.com", "name": "John Smith", - "private_token": "dd34asd13as", "state": "active", + "avatar_url": "http://localhost:3000/uploads/user/avatar/1/index.jpg", + "web_url": "http://localhost:3000/u/john_smith", "created_at": "2012-05-23T08:00:58Z", + "is_admin": false, "bio": null, "location": null, "skype": "", "linkedin": "", "twitter": "", "website_url": "", + "last_sign_in_at": "2012-06-01T11:41:01Z", + "confirmed_at": "2012-05-23T09:05:22Z", "theme_id": 1, "color_scheme_id": 2, - "is_admin": false, + "projects_limit": 100, + "current_sign_in_at": "2012-06-02T06:36:55Z", + "identities": [ + {"provider": "github", "extern_uid": "2435223452345"}, + {"provider": "bitbucket", "extern_uid": "john_smith"}, + {"provider": "google_oauth2", "extern_uid": "8776128412476123468721346"} + ], "can_create_group": true, "can_create_project": true, - "projects_limit": 100 + "two_factor_enabled": true, + "external": false, + "private_token": "dd34asd13as" } ``` diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 4b1788a9af0..ca52a483a59 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -8,7 +8,7 @@ This is one of new trends in Continuous Integration/Deployment to: 1. create application image, 1. run test against created image, -1. push image to remote registry, +1. push image to remote registry, 1. deploy server from pushed image It's also useful in case when your application already has the `Dockerfile` that can be used to create and test image: @@ -41,27 +41,27 @@ GitLab Runner then executes build scripts as `gitlab-runner` user. --description "My Runner" ``` -2. Install Docker on server. +2. Install Docker Engine on server. - For more information how to install Docker on different systems checkout the [Supported installations](https://docs.docker.com/installation/). + For more information how to install Docker Engine on different systems checkout the [Supported installations](https://docs.docker.com/engine/installation/). 3. Add `gitlab-runner` user to `docker` group: - + ```bash $ sudo usermod -aG docker gitlab-runner ``` 4. Verify that `gitlab-runner` has access to Docker: - + ```bash $ sudo -u gitlab-runner -H docker info ``` - + You can now verify that everything works by adding `docker info` to `.gitlab-ci.yml`: ```yaml before_script: - docker info - + build_image: script: - docker build -t my-docker-image . @@ -75,37 +75,80 @@ For more information please checkout [On Docker security: `docker` group conside ## 2. Use docker-in-docker executor -Second approach is to use special Docker image with all tools installed (`docker` and `docker-compose`) and run build script in context of that image in privileged mode. +The second approach is to use the special Docker image with all tools installed +(`docker` and `docker-compose`) and run the build script in context of that +image in privileged mode. + In order to do that follow the steps: 1. Install [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/#installation). -1. Register GitLab Runner from command line to use `docker` and `privileged` mode: +1. Register GitLab Runner from the command line to use `docker` and `privileged` + mode: ```bash - $ sudo gitlab-runner register -n \ + sudo gitlab-runner register -n \ --url https://gitlab.com/ci \ --token RUNNER_TOKEN \ --executor docker \ --description "My Docker Runner" \ - --docker-image "gitlab/dind:latest" \ + --docker-image "docker:latest" \ --docker-privileged ``` - - The above command will register new Runner to use special [gitlab/dind](https://registry.hub.docker.com/u/gitlab/dind/) image which is provided by GitLab Inc. - The image at the start runs Docker daemon in [docker-in-docker](https://blog.docker.com/2013/09/docker-can-now-run-within-docker/) mode. + + The above command will register a new Runner to use the special + `docker:latest` image which is provided by Docker. **Notice that it's using + the `privileged` mode to start the build and service containers.** If you + want to use [docker-in-docker] mode, you always have to use `privileged = true` + in your Docker containers. + + The above command will create a `config.toml` entry similar to this: + + ``` + [[runners]] + url = "https://gitlab.com/ci" + token = TOKEN + executor = "docker" + [runners.docker] + tls_verify = false + image = "docker:latest" + privileged = true + disable_cache = false + volumes = ["/cache"] + [runners.cache] + Insecure = false + ``` + + If you want to use the Shared Runners available on your GitLab CE/EE + installation in order to build Docker images, then make sure that your + Shared Runners configuration has the `privileged` mode set to `true`. 1. You can now use `docker` from build script: - + ```yaml + image: docker:latest + + services: + - docker:dind + before_script: - - docker info - - build_image: + - docker info + + build: + stage: build script: - - docker build -t my-docker-image . - - docker run my-docker-image /script/to/run/tests + - docker build -t my-docker-image . + - docker run my-docker-image /script/to/run/tests ``` -1. However, by enabling `--docker-privileged` you are effectively disables all security mechanisms of containers and exposing your host to privilege escalation which can lead to container breakout. -For more information, check out [Runtime privilege](https://docs.docker.com/reference/run/#runtime-privilege-linux-capabilities-and-lxc-configuration).
\ No newline at end of file +1. However, by enabling `--docker-privileged` you are effectively disabling all + the security mechanisms of containers and exposing your host to privilege + escalation which can lead to container breakout. + + For more information, check out the official Docker documentation on + [Runtime privilege and Linux capabilities][docker-cap]. + +An example project using this approach can be found here: https://gitlab.com/gitlab-examples/docker. + +[docker-in-docker]: https://blog.docker.com/2013/09/docker-can-now-run-within-docker/ +[docker-cap]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index bd748f1b986..56ac2195c49 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -64,7 +64,7 @@ You can see some widely used services examples in the relevant documentation of ### How is service linked to the build To better understand how the container linking works, read -[Linking containers together](https://docs.docker.com/userguide/dockerlinks/). +[Linking containers together][linking-containers]. To summarize, if you add `mysql` as service to your application, the image will then be used to create a container that is linked to the build container. @@ -239,8 +239,8 @@ is specific to your project. Then create some service containers: ``` -docker run -d -n service-mysql mysql:latest -docker run -d -n service-postgres postgres:latest +docker run -d --name service-mysql mysql:latest +docker run -d --name service-postgres postgres:latest ``` This will create two service containers, named `service-mysql` and @@ -273,7 +273,7 @@ creation. [Docker Fundamentals]: https://docs.docker.com/engine/understanding-docker/ [hub]: https://hub.docker.com/ [linking-containers]: https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/ -[tutum/wordpress]: https://registry.hub.docker.com/u/tutum/wordpress/ -[postgres-hub]: https://registry.hub.docker.com/u/library/postgres/ -[mysql-hub]: https://registry.hub.docker.com/u/library/mysql/ +[tutum/wordpress]: https://hub.docker.com/r/tutum/wordpress/ +[postgres-hub]: https://hub.docker.com/r/_/postgres/ +[mysql-hub]: https://hub.docker.com/r/_/mysql/ [runner-priv-reg]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/configuration/advanced-configuration.md#using-a-private-docker-registry diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index cc059dc4376..61294be599d 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -4,12 +4,12 @@ - [Test and deploy a Ruby application to Heroku](test-and-deploy-ruby-application-to-heroku.md) - [Test and deploy a Python application to Heroku](test-and-deploy-python-application-to-heroku.md) - [Test a Clojure application](test-clojure-application.md) -- [Using `dpl` as deployment tool](deployment/README.md) +- [Using `dpl` as deployment tool](../deployment/README.md) - Help your favorite programming language and GitLab by sending a merge request with a guide for that language. ## Outside the documentation -- [Blost post about using GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) +- [Blog post about using GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) - [Repo's with examples for various languages](https://gitlab.com/groups/gitlab-examples) - [The .gitlab-ci.yml file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index aeadd6a448e..26953014502 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -40,7 +40,7 @@ repository with the following content: #!/bin/bash # We need to install dependencies only for Docker -[[ ! -e /.dockerinit ]] && exit 0 +[[ ! -e /.dockerenv ]] && [[ ! -e /.dockerinit ]] && exit 0 set -xe @@ -60,7 +60,7 @@ docker-php-ext-install pdo_mysql You might wonder what `docker-php-ext-install` is. In short, it is a script provided by the official php docker image that you can use to easilly install extensions. For more information read the the documentation at -<https://hub.docker.com/_/php/>. +<https://hub.docker.com/r/_/php/>. Now that we created the script that contains all prerequisites for our build environment, let's add it in `.gitlab-ci.yml`: @@ -92,7 +92,7 @@ Finally, commit your files and push them to GitLab to see your build succeeding The final `.gitlab-ci.yml` should look similar to this: ```yaml -# Select image from https://hub.docker.com/_/php/ +# Select image from https://hub.docker.com/r/_/php/ image: php:5.6 before_script: @@ -278,7 +278,7 @@ that runs on [GitLab.com](https://gitlab.com) using our publicly available Want to hack on it? Simply fork it, commit and push your changes. Within a few moments the changes will be picked by a public runner and the build will begin. -[php-hub]: https://hub.docker.com/_/php/ +[php-hub]: https://hub.docker.com/r/_/php/ [phpenv]: https://github.com/phpenv/phpenv [phpenv-installation]: https://github.com/phpenv/phpenv#installation [php-example-repo]: https://gitlab.com/gitlab-examples/php diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md index a236da53fe9..e4d3970deac 100644 --- a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -8,7 +8,7 @@ This is what the `.gitlab-ci.yml` file looks like for this project: ```yaml test: script: - # this configures django application to use attached postgres database that is run on `postgres` host + # this configures Django application to use attached postgres database that is run on `postgres` host - export DATABASE_URL=postgres://postgres:@postgres:5432/python-test-app - apt-get update -qy - apt-get install -y python-dev python-pip @@ -37,7 +37,7 @@ production: ``` This project has three jobs: -1. `test` - used to test rails application, +1. `test` - used to test Django application, 2. `staging` - used to automatically deploy staging environment every push to `master` branch 3. `production` - used to automatically deploy production environmnet for every created tag @@ -61,12 +61,12 @@ gitlab-ci-multi-runner register \ --non-interactive \ --url "https://gitlab.com/ci/" \ --registration-token "PROJECT_REGISTRATION_TOKEN" \ - --description "python-3.2" \ + --description "python-3.5" \ --executor "docker" \ - --docker-image python:3.2 \ + --docker-image python:3.5 \ --docker-postgres latest ``` -With the command above, you create a runner that uses [python:3.2](https://registry.hub.docker.com/u/library/python/) image and uses [postgres](https://registry.hub.docker.com/u/library/postgres/) database. +With the command above, you create a runner that uses [python:3.5](https://hub.docker.com/r/_/python/) image and uses [postgres](https://hub.docker.com/r/_/postgres/) database. To access PostgreSQL database you need to connect to `host: postgres` as user `postgres` without password. diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md index f5645d586ae..08c10d391ea 100644 --- a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md @@ -1,5 +1,5 @@ ## Test and Deploy a ruby application -This example will guide you how to run tests in your Ruby application and deploy it automatically as Heroku application. +This example will guide you how to run tests in your Ruby on Rails application and deploy it automatically as Heroku application. You can checkout the example [source](https://gitlab.com/ayufan/ruby-getting-started) and check [CI status](https://gitlab.com/ayufan/ruby-getting-started/builds?scope=all). @@ -32,7 +32,7 @@ production: ``` This project has three jobs: -1. `test` - used to test rails application, +1. `test` - used to test Rails application, 2. `staging` - used to automatically deploy staging environment every push to `master` branch 3. `production` - used to automatically deploy production environmnet for every created tag @@ -62,6 +62,6 @@ gitlab-ci-multi-runner register \ --docker-postgres latest ``` -With the command above, you create a runner that uses [ruby:2.2](https://registry.hub.docker.com/u/library/ruby/) image and uses [postgres](https://registry.hub.docker.com/u/library/postgres/) database. +With the command above, you create a runner that uses [ruby:2.2](https://hub.docker.com/r/_/ruby/) image and uses [postgres](https://hub.docker.com/r/_/postgres/) database. To access PostgreSQL database you need to connect to `host: postgres` as user `postgres` without password. diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 6a42a935abd..386b8e29fcf 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -212,8 +212,8 @@ If you want to receive e-mail notifications about the result status of the builds, you should explicitly enable the **Builds Emails** service under your project's settings. -For more information read the [Builds emails service documentation] -(../../project_services/builds_emails.md). +For more information read the +[Builds emails service documentation](../../project_services/builds_emails.md). ## Builds badge diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index c66d77122b2..aaf3aa77837 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -16,7 +16,7 @@ services: - mysql:latest variables: - # Configure mysql environment variables (https://hub.docker.com/_/mysql/) + # Configure mysql environment variables (https://hub.docker.com/r/_/mysql/) MYSQL_DATABASE: el_duderino MYSQL_ROOT_PASSWORD: mysql_strong_password ``` @@ -114,5 +114,5 @@ available [shared runners](../runners/README.md). Want to hack on it? Simply fork it, commit and push your changes. Within a few moments the changes will be picked by a public runner and the build will begin. -[hub-mysql]: https://hub.docker.com/_/mysql/ +[hub-mysql]: https://hub.docker.com/r/_/mysql/ [mysql-example-repo]: https://gitlab.com/gitlab-examples/mysql diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index 17d21dbda1c..f787cc0a124 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -110,5 +110,5 @@ available [shared runners](../runners/README.md). Want to hack on it? Simply fork it, commit and push your changes. Within a few moments the changes will be picked by a public runner and the build will begin. -[hub-pg]: https://hub.docker.com/_/postgres/ +[hub-pg]: https://hub.docker.com/r/_/postgres/ [postgres-example-repo]: https://gitlab.com/gitlab-examples/postgres diff --git a/doc/ci/services/redis.md b/doc/ci/services/redis.md index b281e8f9f60..80705024d2f 100644 --- a/doc/ci/services/redis.md +++ b/doc/ci/services/redis.md @@ -65,5 +65,5 @@ that runs on [GitLab.com](https://gitlab.com) using our publicly available Want to hack on it? Simply fork it, commit and push your changes. Within a few moments the changes will be picked by a public runner and the build will begin. -[hub-redis]: https://hub.docker.com/_/redis/ +[hub-redis]: https://hub.docker.com/r/_/redis/ [redis-example-repo]: https://gitlab.com/gitlab-examples/redis diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index 7f825e6a065..7c0fb225dac 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -57,7 +57,7 @@ before_script: # WARNING: Use this only with the Docker executor, if you use it with shell # you will overwrite your user's SSH config. - mkdir -p ~/.ssh - - '[[ -f /.dockerinit ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config' + - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config' ``` As a final step, add the _public_ key from the one you created earlier to the diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 9f7c1bfe6a0..79ed512aabb 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -85,6 +85,12 @@ curl -X POST \ In this case, the project with ID `9` will get rebuilt on `master` branch. +Alternatively, you can pass the `token` and `ref` arguments in the query string: + +```bash +curl -X POST \ + "https://gitlab.example.com/api/v3/projects/9/trigger/builds?token=TOKEN&ref=master" +``` ### Triggering a build within `.gitlab-ci.yml` diff --git a/doc/customization/libravatar.md b/doc/customization/libravatar.md index bd2c242afc2..c46ce2ee203 100644 --- a/doc/customization/libravatar.md +++ b/doc/customization/libravatar.md @@ -67,3 +67,16 @@ Run `sudo gitlab-ctl reconfigure` for changes to take effect. In order to use a different set other than `identicon`, replace `&d=identicon` portion of the URL with another supported set. For example, you can use `retro` set in which case the URL would look like: `plain_url: "http://cdn.libravatar.org/avatar/%{hash}?s=%{size}&d=retro"` + + +## Usage examples + +#### For Microsoft Office 365 + +If your users are Office 365-users, the "GetPersonaPhoto" service can be used. Note that this service requires login, so this use case is +most useful in a corporate installation, where all users have access to Office 365. + +```ruby +gitlab_rails['gravatar_plain_url'] = 'http://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' +gitlab_rails['gravatar_ssl_url'] = 'https://outlook.office365.com/owa/service.svc/s/GetPersonaPhoto?email=%{email}&size=HR120x120' +``` diff --git a/doc/development/README.md b/doc/development/README.md index 3f3ef068f96..aa7d54c01d0 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -8,6 +8,7 @@ - [How to dump production data to staging](db_dump.md) - [Instrumentation](instrumentation.md) - [Migration Style Guide](migration_style_guide.md) for creating safe migrations +- [Performance guidelines](performance.md) - [Rake tasks](rake_tasks.md) for development - [Shell commands](shell_commands.md) in the GitLab codebase - [Sidekiq debugging](sidekiq_debugging.md) diff --git a/doc/development/doc_styleguide.md b/doc/development/doc_styleguide.md index 187ec9e7b75..8292b393757 100644 --- a/doc/development/doc_styleguide.md +++ b/doc/development/doc_styleguide.md @@ -127,7 +127,7 @@ Inside the document: ``` If the document you are editing resides in a place other than the GitLab CE/EE `doc/` directory, instead of the relative link, use the full path: - `http://doc.gitlab.com/ce/administration/restart_gitlab.html`. + `http://docs.gitlab.com/ce/administration/restart_gitlab.html`. Replace `reconfigure` with `restart` where appropriate. ## Installation guide @@ -266,5 +266,5 @@ curl -X PUT -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" -d "restricted_signup_domai [cURL]: http://curl.haxx.se/ "cURL website" [single spaces]: http://www.slate.com/articles/technology/technology/2011/01/space_invaders.html -[gfm]: http://doc.gitlab.com/ce/markdown/markdown.html#newlines "GitLab flavored markdown documentation" +[gfm]: http://docs.gitlab.com/ce/markdown/markdown.html#newlines "GitLab flavored markdown documentation" [doc-restart]: ../administration/restart_gitlab.md "GitLab restart documentation" diff --git a/doc/development/instrumentation.md b/doc/development/instrumentation.md index c1cf2e77c26..9168c70945a 100644 --- a/doc/development/instrumentation.md +++ b/doc/development/instrumentation.md @@ -1,12 +1,125 @@ # Instrumenting Ruby Code -GitLab Performance Monitoring allows instrumenting of custom blocks of Ruby -code. This can be used to measure the time spent in a specific part of a larger -chunk of code. The resulting data is stored as a field in the transaction that -executed the block. +GitLab Performance Monitoring allows instrumenting of both methods and custom +blocks of Ruby code. Method instrumentation is the primary form of +instrumentation with block-based instrumentation only being used when we want to +drill down to specific regions of code within a method. -To start measuring a block of Ruby code you should use `Gitlab::Metrics.measure` -and give it a name: +## Instrumenting Methods + +Instrumenting methods is done by using the `Gitlab::Metrics::Instrumentation` +module. This module offers a few different methods that can be used to +instrument code: + +* `instrument_method`: instruments a single class method. +* `instrument_instance_method`: instruments a single instance method. +* `instrument_class_hierarchy`: given a Class this method will recursively + instrument all sub-classes (both class and instance methods). +* `instrument_methods`: instruments all public class methods of a Module. +* `instrument_instance_methods`: instruments all public instance methods of a + Module. + +To remove the need for typing the full `Gitlab::Metrics::Instrumentation` +namespace you can use the `configure` class method. This method simply yields +the supplied block while passing `Gitlab::Metrics::Instrumentation` as its +argument. An example: + +``` +Gitlab::Metrics::Instrumentation.configure do |conf| + conf.instrument_method(Foo, :bar) + conf.instrument_method(Foo, :baz) +end +``` + +Using this method is in general preferred over directly calling the various +instrumentation methods. + +Method instrumentation should be added in the initializer +`config/initializers/metrics.rb`. + +### Examples + +Instrumenting a single method: + +``` +Gitlab::Metrics::Instrumentation.configure do |conf| + conf.instrument_method(User, :find_by) +end +``` + +Instrumenting an entire class hierarchy: + +``` +Gitlab::Metrics::Instrumentation.configure do |conf| + conf.instrument_class_hierarchy(ActiveRecord::Base) +end +``` + +Instrumenting all public class methods: + +``` +Gitlab::Metrics::Instrumentation.configure do |conf| + conf.instrument_methods(User) +end +``` + +### Checking Instrumented Methods + +The easiest way to check if a method has been instrumented is to check its +source location. For example: + +``` +method = Rugged::TagCollection.instance_method(:[]) + +method.source_location +``` + +If the source location points to `lib/gitlab/metrics/instrumentation.rb` you +know the method has been instrumented. + +If you're using Pry you can use the `$` command to display the source code of a +method (along with its source location), this is easier than running the above +Ruby code. In case of the above snippet you'd run the following: + +``` +$ Rugged::TagCollection#[] +``` + +This will print out something along the lines of: + +``` +From: /path/to/your/gitlab/lib/gitlab/metrics/instrumentation.rb @ line 148: +Owner: #<Module:0x0055f0865c6d50> +Visibility: public +Number of lines: 21 + +def #{name}(#{args_signature}) + trans = Gitlab::Metrics::Instrumentation.transaction + + if trans + start = Time.now + retval = super + duration = (Time.now - start) * 1000.0 + + if duration >= Gitlab::Metrics.method_call_threshold + trans.increment(:method_duration, duration) + + trans.add_metric(Gitlab::Metrics::Instrumentation::SERIES, + { duration: duration }, + method: #{label.inspect}) + end + + retval + else + super + end +end +``` + +## Instrumenting Ruby Blocks + +Measuring blocks of Ruby code is done by calling `Gitlab::Metrics.measure` and +passing it a block. For example: ```ruby Gitlab::Metrics.measure(:foo) do @@ -14,6 +127,10 @@ Gitlab::Metrics.measure(:foo) do end ``` +The block is executed and the execution time is stored as a set of fields in the +currently running transaction. If no transaction is present the block is yielded +without measuring anything. + 3 values are measured for a block: 1. The real time elapsed, stored in NAME_real_time. diff --git a/doc/development/performance.md b/doc/development/performance.md new file mode 100644 index 00000000000..fb37b3a889c --- /dev/null +++ b/doc/development/performance.md @@ -0,0 +1,258 @@ +# Performance Guidelines + +This document describes various guidelines to follow to ensure good and +consistent performance of GitLab. + +## Workflow + +The process of solving performance problems is roughly as follows: + +1. Make sure there's an issue open somewhere (e.g., on the GitLab CE issue + tracker), create one if there isn't. See [#15607][#15607] for an example. +2. Measure the performance of the code in a production environment such as + GitLab.com (see the [Tooling](#tooling) section below). Performance should be + measured over a period of _at least_ 24 hours. +3. Add your findings based on the measurement period (screenshots of graphs, + timings, etc) to the issue mentioned in step 1. +4. Solve the problem. +5. Create a merge request, assign the "performance" label and ping the right + people (e.g. [@yorickpeterse][yorickpeterse] and [@joshfng][joshfng]). +6. Once a change has been deployed make sure to _again_ measure for at least 24 + hours to see if your changes have any impact on the production environment. +7. Repeat until you're done. + +When providing timings make sure to provide: + +* The 95th percentile +* The 99th percentile +* The mean + +When providing screenshots of graphs, make sure that both the X and Y axes and +the legend are clearly visible. If you happen to have access to GitLab.com's own +monitoring tools you should also provide a link to any relevant +graphs/dashboards. + +## Tooling + +GitLab provides two built-in tools to aid the process of improving performance: + +* [Sherlock](doc/development/profiling.md#sherlock) +* [GitLab Performance Monitoring](doc/monitoring/performance/monitoring.md) + +GitLab employees can use GitLab.com's performance monitoring systems located at +<http://performance.gitlab.net>, this requires you to log in using your +`@gitlab.com` Email address. Non-GitLab employees are advised to set up their +own InfluxDB + Grafana stack. + +## Benchmarks + +Benchmarks are almost always useless. Benchmarks usually only test small bits of +code in isolation and often only measure the best case scenario. On top of that, +benchmarks for libraries (e.g., a Gem) tend to be biased in favour of the +library. After all there's little benefit to an author publishing a benchmark +that shows they perform worse than their competitors. + +Benchmarks are only really useful when you need a rough (emphasis on "rough") +understanding of the impact of your changes. For example, if a certain method is +slow a benchmark can be used to see if the changes you're making have any impact +on the method's performance. However, even when a benchmark shows your changes +improve performance there's no guarantee the performance also improves in a +production environment. + +When writing benchmarks you should almost always use +[benchmark-ips](https://github.com/evanphx/benchmark-ips). Ruby's `Benchmark` +module that comes with the standard library is rarely useful as it runs either a +single iteration (when using `Benchmark.bm`) or two iterations (when using +`Benchmark.bmbm`). Running this few iterations means external factors (e.g. a +video streaming in the background) can very easily skew the benchmark +statistics. + +Another problem with the `Benchmark` module is that it displays timings, not +iterations. This means that if a piece of code completes in a very short period +of time it can be very difficult to compare the timings before and after a +certain change. This in turn leads to patterns such as the following: + +```ruby +Benchmark.bmbm(10) do |bench| + bench.report 'do something' do + 100.times do + ... work here ... + end + end +end +``` + +This however leads to the question: how many iterations should we run to get +meaningful statistics? + +The benchmark-ips Gem basically takes care of all this and much more, and as a +result of this should be used instead of the `Benchmark` module. + +In short: + +1. Don't trust benchmarks you find on the internet. +2. Never make claims based on just benchmarks, always measure in production to + confirm your findings. +3. X being N times faster than Y is meaningless if you don't know what impact it + will actually have on your production environment. +4. A production environment is the _only_ benchmark that always tells the truth + (unless your performance monitoring systems are not set up correctly). +5. If you must write a benchmark use the benchmark-ips Gem instead of Ruby's + `Benchmark` module. + +## Importance of Changes + +When working on performance improvements, it's important to always ask yourself +the question "How important is it to improve the performance of this piece of +code?". Not every piece of code is equally important and it would be a waste to +spend a week trying to improve something that only impacts a tiny fraction of +our users. For example, spending a week trying to squeeze 10 milliseconds out of +a method is a waste of time when you could have spent a week squeezing out 10 +seconds elsewhere. + +There is no clear set of steps that you can follow to determine if a certain +piece of code is worth optimizing. The only two things you can do are: + +1. Think about what the code does, how it's used, how many times it's called and + how much time is spent in it relative to the total execution time (e.g., the + total time spent in a web request). +2. Ask others (preferably in the form of an issue). + +Some examples of changes that aren't really important/worth the effort: + +* Replacing double quotes with single quotes. +* Replacing usage of Array with Set when the list of values is very small. +* Replacing library A with library B when both only take up 0.1% of the total + execution time. +* Calling `freeze` on every string (see [String Freezing](#string-freezing)). + +## Slow Operations & Sidekiq + +Slow operations (e.g. merging branches) or operations that are prone to errors +(using external APIs) should be performed in a Sidekiq worker instead of +directly in a web request as much as possible. This has numerous benefits such +as: + +1. An error won't prevent the request from completing. +2. The process being slow won't affect the loading time of a page. +3. In case of a failure it's easy to re-try the process (Sidekiq takes care of + this automatically). +4. By isolating the code from a web request it will hopefully be easier to test + and maintain. + +It's especially important to use Sidekiq as much as possible when dealing with +Git operations as these operations can take quite some time to complete +depending on the performance of the underlying storage system. + +## Git Operations + +Care should be taken to not run unnecessary Git operations. For example, +retrieving the list of branch names using `Repository#branch_names` can be done +without an explicit check if a repository exists or not. In other words, instead +of this: + +```ruby +if repository.exists? + repository.branch_names.each do |name| + ... + end +end +``` + +You can just write: + +```ruby +repository.branch_names.each do |name| + ... +end +``` + +## Caching + +Operations that will often return the same result should be cached using Redis, +in particular Git operations. When caching data in Redis, make sure the cache is +flushed whenever needed. For example, a cache for the list of tags should be +flushed whenever a new tag is pushed or a tag is removed. + +When adding cache expiration code for repositories, this code should be placed +in one of the before/after hooks residing in the Repository class. For example, +if a cache should be flushed after importing a repository this code should be +added to `Repository#after_import`. This ensures the cache logic stays within +the Repository class instead of leaking into other classes. + +When caching data, make sure to also memoize the result in an instance variable. +While retrieving data from Redis is much faster than raw Git operations, it still +has overhead. By caching the result in an instance variable, repeated calls to +the same method won't end up retrieving data from Redis upon every call. When +memoizing cached data in an instance variable, make sure to also reset the +instance variable when flushing the cache. An example: + + +```ruby +def first_branch + @first_branch ||= cache.fetch(:first_branch) { branches.first } +end + +def expire_first_branch_cache + cache.expire(:first_branch) + @first_branch = nil +end +``` + +## Anti-Patterns + +This is a collection of [anti-patterns][anti-pattern] that should be avoided +unless these changes have a measurable, significant and positive impact on +production environments. + +### String Freezing + +In recent Ruby versions calling `freeze` on a String leads to it being allocated +only once and re-used. For example, on Ruby 2.3 this will only allocate the +"foo" String once: + +```ruby +10.times do + 'foo'.freeze +end +``` + +Blindly adding a `.freeze` call to every String is an anti-pattern that should +be avoided unless one can prove (using production data) the call actually has a +positive impact on performance. + +This feature of Ruby wasn't really meant to make things faster directly, instead +it was meant to reduce the number of allocations. Depending on the size of the +String and how frequently it would be allocated (before the `.freeze` call was +added), this _may_ make things faster, but there's no guarantee it will. + +Another common flavour of this is to not only freeze a String, but also assign +it to a constant, for example: + +```ruby +SOME_CONSTANT = 'foo'.freeze + +9000.times do + SOME_CONSTANT +end +``` + +The only reason you should be doing this is to prevent somebody from mutating +the global String. However, since you can just re-assign constants in Ruby +there's nothing stopping somebody from doing this elsewhere in the code: + +```ruby +SOME_CONSTANT = 'bar' +``` + +### Moving Allocations to Constants + +Storing an object as a constant so you only allocate it once _may_ improve +performance, but there's no guarantee this will. Looking up constants has an +impact on runtime performance, and as such, using a constant instead of +referencing an object directly may even slow code down. + +[#15607]: https://gitlab.com/gitlab-org/gitlab-ce/issues/15607 +[yorickpeterse]: https://gitlab.com/u/yorickpeterse +[joshfng]: https://gitlab.com/u/joshfng +[anti-pattern]: https://en.wikipedia.org/wiki/Anti-pattern diff --git a/doc/development/testing.md b/doc/development/testing.md index 672e3fb4649..33eed29ba5c 100644 --- a/doc/development/testing.md +++ b/doc/development/testing.md @@ -64,6 +64,7 @@ the command line via `bundle exec teaspoon`, or via a web browser at methods. - Use `context` to test branching logic. - Don't `describe` symbols (see [Gotchas](gotchas.md#dont-describe-symbols)). +- Don't supply the `:each` argument to hooks since it's the default. - Prefer `not_to` to `to_not`. - Try to match the ordering of tests to the ordering within the class. - Try to follow the [Four-Phase Test][four-phase-test] pattern, using newlines diff --git a/doc/gitlab-basics/create-issue.md b/doc/gitlab-basics/create-issue.md index 87f078def04..5221d85b661 100644 --- a/doc/gitlab-basics/create-issue.md +++ b/doc/gitlab-basics/create-issue.md @@ -24,4 +24,4 @@ You may assign the Issue to a user, add a milestone and add labels (they are all  -Your Issue will now be added to the Issue Tracker and will be ready to be reviewed. You can comment on it and mention the people involved. You can also link Issues to the Merge Requests where the Issues are solved. To do this, you can use an [Issue closing pattern](http://doc.gitlab.com/ce/customization/issue_closing.html). +Your Issue will now be added to the Issue Tracker and will be ready to be reviewed. You can comment on it and mention the people involved. You can also link Issues to the Merge Requests where the Issues are solved. To do this, you can use an [Issue closing pattern](http://docs.gitlab.com/ce/customization/issue_closing.html). diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index b545d62549d..f737dffc024 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -14,7 +14,7 @@ Fill out the required information: 1. Select a [visibility level](https://gitlab.com/help/public_access/public_access) -1. You can also [import your existing projects](http://doc.gitlab.com/ce/workflow/importing/README.html) +1. You can also [import your existing projects](http://docs.gitlab.com/ce/workflow/importing/README.html) 1. Click on "create project" diff --git a/doc/hooks/custom_hooks.md b/doc/hooks/custom_hooks.md index dcdf49d3379..820934f97f1 100644 --- a/doc/hooks/custom_hooks.md +++ b/doc/hooks/custom_hooks.md @@ -2,7 +2,7 @@ **Note: Custom git hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. -Please explore [webhooks](../web_hooks/web_hooks.md) as an option if you do not have filesystem access. For a user configurable Git Hooks interface, please see [GitLab Enterprise Edition Git Hooks](http://doc.gitlab.com/ee/git_hooks/git_hooks.html).** +Please explore [webhooks](../web_hooks/web_hooks.md) as an option if you do not have filesystem access. For a user configurable Git Hooks interface, please see [GitLab Enterprise Edition Git Hooks](http://docs.gitlab.com/ee/git_hooks/git_hooks.html).** Git natively supports hooks that are executed on different actions. Examples of server-side git hooks include pre-receive, post-receive, and update. diff --git a/doc/install/installation.md b/doc/install/installation.md index e721e70a596..fa11eb9ba6a 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -6,7 +6,7 @@ Since an installation from source is a lot of work and error prone we strongly r One reason the Omnibus package is more reliable is its use of Runit to restart any of the GitLab processes in case one crashes. On heavily used GitLab instances the memory usage of the Sidekiq background worker will grow over time. -Omnibus packages solve this by [letting the Sidekiq terminate gracefully](http://doc.gitlab.com/ce/operations/sidekiq_memory_killer.html) if it uses too much memory. +Omnibus packages solve this by [letting the Sidekiq terminate gracefully](http://docs.gitlab.com/ce/operations/sidekiq_memory_killer.html) if it uses too much memory. After this termination Runit will detect Sidekiq is not running and will start it. Since installations from source don't have Runit, Sidekiq can't be terminated and its memory usage will grow over time. @@ -157,22 +157,64 @@ Create a `git` user for GitLab: ## 5. Database -We recommend using a PostgreSQL database. For MySQL check [MySQL setup guide](database_mysql.md). *Note*: because we need to make use of extensions you need at least pgsql 9.1. +We recommend using a PostgreSQL database. For MySQL check the +[MySQL setup guide](database_mysql.md). - # Install the database packages - sudo apt-get install -y postgresql postgresql-client libpq-dev +> **Note**: because we need to make use of extensions you need at least pgsql 9.1. - # Create a user for GitLab +1. Install the database packages: + + ```bash + sudo apt-get install -y postgresql postgresql-client libpq-dev postgresql-contrib + ``` + +1. Create a database user for GitLab: + + ```bash sudo -u postgres psql -d template1 -c "CREATE USER git CREATEDB;" + ``` + +1. Create the GitLab production database and grant all privileges on database: - # Create the GitLab production database & grant all privileges on database + ```bash sudo -u postgres psql -d template1 -c "CREATE DATABASE gitlabhq_production OWNER git;" + ``` + +1. Create the `pg_trgm` extension (required for GitLab 8.6+): + + ```bash + sudo -u postgres psql -d template1 -c "CREATE EXTENSION IF NOT EXISTS pg_trgm;" + ``` + +1. Try connecting to the new database with the new user: - # Try connecting to the new database with the new user + ```bash sudo -u git -H psql -d gitlabhq_production + ``` + +1. Check if the `pg_trgm` extension is enabled: + + ```bash + SELECT true AS enabled + FROM pg_available_extensions + WHERE name = 'pg_trgm' + AND installed_version IS NOT NULL; + ``` + + If the extension is enabled this will produce the following output: - # Quit the database session + ``` + enabled + --------- + t + (1 row) + ``` + +1. Quit the database session: + + ```bash gitlabhq_production> \q + ``` ## 6. Redis @@ -227,7 +269,7 @@ sudo usermod -aG redis git ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 8-7-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 8-8-stable gitlab **Note:** You can change `8-7-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index 0245febfcd8..44d2a14f366 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -132,5 +132,5 @@ To disable the relative URL: 1. Follow the same as above starting from 2. and set up the GitLab URL to one that doesn't contain a relative path. -[omnibus-rel]: http://doc.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab "How to setup relative URL in Omnibus GitLab" +[omnibus-rel]: http://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab "How to setup relative URL in Omnibus GitLab" [restart gitlab]: ../administration/restart_gitlab.md#installations-from-source "How to restart GitLab" diff --git a/doc/install/requirements.md b/doc/install/requirements.md index eb9fe5e1b1b..df8e8bdc476 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -64,7 +64,10 @@ If you have enough RAM memory and a recent CPU the speed of GitLab is mainly lim ### Memory You need at least 2GB of addressable memory (RAM + swap) to install and use GitLab! -With less memory GitLab will give strange errors during the reconfigure run and 500 errors during usage. +The operating system and any other running applications will also be using memory +so keep in mind that you need at least 2GB available before running GitLab. With +less memory GitLab will give strange errors during the reconfigure run and 500 +errors during usage. - 512MB RAM + 1.5GB of swap is the absolute minimum but we strongly **advise against** this amount of memory. See the unicorn worker section below for more advice. - 1GB RAM + 1GB swap supports up to 100 users but it will be very slow @@ -77,6 +80,10 @@ With less memory GitLab will give strange errors during the reconfigure run and - 128GB RAM supports up to 32,000 users - More users? Run it on [multiple application servers](https://about.gitlab.com/high-availability/) +We recommend having at least 1GB of swap on your server, even if you currently have +enough available RAM. Having swap will help reduce the chance of errors occuring +if your available memory changes. + Notice: The 25 workers of Sidekiq will show up as separate processes in your process overview (such as top or htop) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about many you need of those. ## Gitlab Runner @@ -142,4 +149,4 @@ On a very active server (10,000 active users) the Sidekiq process can use 1GB+ o - Firefox (Latest released version and [latest ESR version](https://www.mozilla.org/en-US/firefox/organizations/)) - Safari 7+ (known problem: required fields in html5 do not work) - Opera (Latest released version) -- Internet Explorer (IE) 10+ but please make sure that you have the `Compatibility View` mode disabled. +- Internet Explorer (IE) 11+ but please make sure that you have the `Compatibility View` mode disabled. diff --git a/doc/integration/README.md b/doc/integration/README.md index 6fe04aa2a06..fd330dd7a7d 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -19,7 +19,7 @@ See the documentation below for details on how to configure these services. GitLab Enterprise Edition contains [advanced Jenkins support][jenkins]. -[jenkins]: http://doc.gitlab.com/ee/integration/jenkins.html +[jenkins]: http://docs.gitlab.com/ee/integration/jenkins.html ## Project services diff --git a/doc/integration/cas.md b/doc/integration/cas.md index e6b2071f193..e34e306f9ac 100644 --- a/doc/integration/cas.md +++ b/doc/integration/cas.md @@ -27,17 +27,18 @@ To enable the CAS OmniAuth provider you must register your application with your ```ruby gitlab_rails['omniauth_providers'] = [ { - name: "cas3", - label: "cas", - args: { - url: 'CAS_SERVER', - login_url: '/CAS_PATH/login', - service_validate_url: '/CAS_PATH/p3/serviceValidate', - logout_url: '/CAS_PATH/logout'} } - } + "name"=> "cas3", + "label"=> "cas", + "args"=> { + "url"=> 'CAS_SERVER', + "login_url"=> '/CAS_PATH/login', + "service_validate_url"=> '/CAS_PATH/p3/serviceValidate', + "logout_url"=> '/CAS_PATH/logout' + } } ] ``` + For installations from source: @@ -57,6 +58,8 @@ To enable the CAS OmniAuth provider you must register your application with your 1. Save the configuration file. +1. Run `gitlab-ctl reconfigure` for the omnibus package. + 1. Restart GitLab for the changes to take effect. On the sign in page there should now be a CAS tab in the sign in form. diff --git a/doc/integration/github.md b/doc/integration/github.md index 1890edd7a4c..e7497e475c9 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -9,7 +9,9 @@ GitHub will generate an application ID and secret key for you to use. 1. Navigate to your individual user settings or an organization's settings, depending on how you want the application registered. It does not matter if the application is registered as an individual or an organization - that is entirely up to you. -1. Select "Applications" in the left menu. +1. Select "OAuth applications" in the left menu. + +1. If you already have applications listed, switch to the "Developer applications" tab. 1. Select "Register new application". @@ -60,12 +62,26 @@ GitHub will generate an application ID and secret key for you to use. For installation from source: + For GitHub.com: + ``` - { name: 'github', app_id: 'YOUR_APP_ID', app_secret: 'YOUR_APP_SECRET', args: { scope: 'user:email' } } ``` + + For GitHub Enterprise: + + ``` + - { name: 'github', app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET', + url: "https://github.example.com/", + args: { scope: 'user:email' } } + ``` + + __Replace `https://github.example.com/` with your GitHub URL.__ + 1. Change 'YOUR_APP_ID' to the client ID from the GitHub application page from step 7. 1. Change 'YOUR_APP_SECRET' to the client secret from the GitHub application page from step 7. diff --git a/doc/integration/img/enabled-oauth-sign-in-sources.png b/doc/integration/img/enabled-oauth-sign-in-sources.png Binary files differnew file mode 100644 index 00000000000..95f8bbdcd24 --- /dev/null +++ b/doc/integration/img/enabled-oauth-sign-in-sources.png diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index cab329c0dec..820f40f81a9 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -11,6 +11,7 @@ of the configured mechanisms. - [Supported Providers](#supported-providers) - [Enable OmniAuth for an Existing User](#enable-omniauth-for-an-existing-user) - [OmniAuth configuration sample when using Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master#omniauth-google-twitter-github-login) +- [Enable or disable Sign In with an OmniAuth provider without disabling import sources](#enable-or-disable-sign-in-with-an-omniauth-provider-without-disabling-import-sources) ## Supported Providers @@ -191,3 +192,17 @@ experience [in the public Wiki](https://github.com/gitlabhq/gitlab-public-wiki/w While we can't officially support every possible authentication mechanism out there, we'd like to at least help those with specific needs. + +## Enable or disable Sign In with an OmniAuth provider without disabling import sources + +>**Note:** +This setting was introduced with version 8.8 of GitLab. + +Administrators are able to enable or disable Sign In via some OmniAuth providers. + +>**Note:** +By default Sign In is enabled via all the OAuth Providers that have been configured in `config/gitlab.yml`. + +In order to enable/disable an OmniAuth provider, go to Admin Area -> Settings -> Sign-in Restrictions section -> Enabled OAuth Sign-In sources and select the providers you want to enable or disable. + + diff --git a/doc/intro/README.md b/doc/intro/README.md index ab298d3808e..382d10aaf40 100644 --- a/doc/intro/README.md +++ b/doc/intro/README.md @@ -39,4 +39,4 @@ Install and update your GitLab installation. - [Install GitLab](https://about.gitlab.com/installation/) - [Update GitLab](https://about.gitlab.com/update/) -- [Explore Omnibus GitLab configuration options](http://doc.gitlab.com/omnibus/settings/configuration.html) +- [Explore Omnibus GitLab configuration options](http://docs.gitlab.com/omnibus/settings/configuration.html) diff --git a/doc/logs/logs.md b/doc/logs/logs.md index 27937e51764..ef5affa2ebd 100644 --- a/doc/logs/logs.md +++ b/doc/logs/logs.md @@ -1,6 +1,6 @@ ## Log system GitLab has advanced log system so everything is logging and you can analize your instance using various system log files. -In addition to system log files, GitLab Enterprise Edition comes with Audit Events. Find more about them [in Audit Events documentation](http://doc.gitlab.com/ee/administration/audit_events.html) +In addition to system log files, GitLab Enterprise Edition comes with Audit Events. Find more about them [in Audit Events documentation](http://docs.gitlab.com/ee/administration/audit_events.html) System log files are typically plain text in a standard log file format. This guide talks about how to read and use these system log files. diff --git a/doc/monitoring/performance/gitlab_configuration.md b/doc/monitoring/performance/gitlab_configuration.md index 90e99302210..771584268d9 100644 --- a/doc/monitoring/performance/gitlab_configuration.md +++ b/doc/monitoring/performance/gitlab_configuration.md @@ -37,4 +37,4 @@ Read more on: - [Introduction to GitLab Performance Monitoring](introduction.md) - [InfluxDB Configuration](influxdb_configuration.md) - [InfluxDB Schema](influxdb_schema.md) -- [Grafana Install/Configuration](grafana_configuration.md +- [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/monitoring/performance/grafana_configuration.md b/doc/monitoring/performance/grafana_configuration.md index a79c8d48d3b..168bd85c26a 100644 --- a/doc/monitoring/performance/grafana_configuration.md +++ b/doc/monitoring/performance/grafana_configuration.md @@ -59,34 +59,53 @@ This will drop you in to an InfluxDB interactive session. Copy the entire contents below and paste it in to the interactive session: ``` -CREATE RETENTION POLICY gitlab_30d ON gitlab DURATION 30d REPLICATION 1 DEFAULT -CREATE RETENTION POLICY seven_days ON gitlab DURATION 7d REPLICATION 1 -CREATE CONTINUOUS QUERY rails_transaction_counts_seven_days ON gitlab BEGIN SELECT count("duration") AS "count" INTO gitlab.seven_days.rails_transaction_counts FROM rails_transactions GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY sidekiq_transaction_counts_seven_days ON gitlab BEGIN SELECT count("duration") AS "count" INTO gitlab.seven_days.sidekiq_transaction_counts FROM sidekiq_transactions GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY rails_method_call_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean" INTO gitlab.seven_days.rails_method_call_timings FROM rails_method_calls GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY sidekiq_method_call_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean" INTO gitlab.seven_days.sidekiq_method_call_timings FROM sidekiq_method_calls GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY rails_method_call_timings_per_method_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS duration_95th, percentile("duration", 99.000) AS duration_99th, mean("duration") AS duration_mean INTO gitlab.seven_days.rails_method_call_timings_per_method FROM rails_method_calls GROUP BY time(1m), method END; -CREATE CONTINUOUS QUERY sidekiq_method_call_timings_per_method_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS duration_95th, percentile("duration", 99.000) AS duration_99th, mean("duration") AS duration_mean INTO gitlab.seven_days.sidekiq_method_call_timings_per_method FROM sidekiq_method_calls GROUP BY time(1m), method END; -CREATE CONTINUOUS QUERY rails_memory_usage_per_minute ON gitlab BEGIN SELECT percentile(value, 95.000) AS memory_95th, percentile(value, 99.000) AS memory_99th, mean(value) AS memory_mean INTO gitlab.seven_days.rails_memory_usage_per_minute FROM rails_memory_usage GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY sidekiq_memory_usage_per_minute ON gitlab BEGIN SELECT percentile(value, 95.000) AS memory_95th, percentile(value, 99.000) AS memory_99th, mean(value) AS memory_mean INTO gitlab.seven_days.sidekiq_memory_usage_per_minute FROM sidekiq_memory_usage GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY sidekiq_file_descriptors_per_minute ON gitlab BEGIN SELECT sum(value) AS value INTO gitlab.seven_days.sidekiq_file_descriptors_per_minute FROM sidekiq_file_descriptors GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY rails_file_descriptors_per_minute ON gitlab BEGIN SELECT sum(value) AS value INTO gitlab.seven_days.rails_file_descriptors_per_minute FROM rails_file_descriptors GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY rails_gc_counts_per_minute ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.seven_days.rails_gc_counts_per_minute FROM rails_gc_statistics GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY sidekiq_gc_counts_per_minute ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.seven_days.sidekiq_gc_counts_per_minute FROM sidekiq_gc_statistics GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY rails_gc_timings_per_minute ON gitlab BEGIN SELECT percentile(total_time, 95.000) AS duration_95th, percentile(total_time, 99.000) AS duration_99th, mean(total_time) AS duration_mean INTO gitlab.seven_days.rails_gc_timings_per_minute FROM rails_gc_statistics GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY sidekiq_gc_timings_per_minute ON gitlab BEGIN SELECT percentile(total_time, 95.000) AS duration_95th, percentile(total_time, 99.000) AS duration_99th, mean(total_time) AS duration_mean INTO gitlab.seven_days.sidekiq_gc_timings_per_minute FROM sidekiq_gc_statistics GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY rails_gc_major_minor_per_minute ON gitlab BEGIN SELECT sum(major_gc_count) AS major, sum(minor_gc_count) AS minor INTO gitlab.seven_days.rails_gc_major_minor_per_minute FROM rails_gc_statistics GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY sidekiq_gc_major_minor_per_minute ON gitlab BEGIN SELECT sum(major_gc_count) AS major, sum(minor_gc_count) AS minor INTO gitlab.seven_days.sidekiq_gc_major_minor_per_minute FROM sidekiq_gc_statistics GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY grape_internal_allowed_request_counts_per_minute ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.seven_days.grape_internal_allowed_request_counts_per_minute FROM rails_transactions WHERE request_uri = '/api/v3/internal/allowed' GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY grape_internal_allowed_request_timings_per_minute ON gitlab BEGIN SELECT percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean("duration") AS duration_mean INTO gitlab.seven_days.grape_internal_allowed_request_timings_per_minute FROM rails_transactions WHERE request_uri = '/api/v3/internal/allowed' GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY grape_internal_allowed_sql_timings_per_minute ON gitlab BEGIN SELECT percentile(sql_duration, 95) AS duration_95th, percentile(sql_duration, 99) AS duration_99th, mean(sql_duration) AS duration_mean INTO gitlab.seven_days.grape_internal_allowed_sql_timings_per_minute FROM rails_transactions WHERE request_uri = '/api/v3/internal/allowed' GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY grape_internal_authorized_keys_request_counts_per_minute ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.seven_days.grape_internal_authorized_keys_request_counts_per_minute FROM rails_transactions WHERE request_uri = '/api/v3/internal/authorized_keys' GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY grape_internal_authorized_keys_request_timings_per_minute ON gitlab BEGIN SELECT percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean("duration") AS duration_mean INTO gitlab.seven_days.grape_internal_authorized_keys_request_timings_per_minute FROM rails_transactions WHERE request_uri = '/api/v3/internal/authorized_keys' GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY grape_internal_authorized_keys_sql_timings_per_minute ON gitlab BEGIN SELECT percentile(sql_duration, 95) AS duration_95th, percentile(sql_duration, 99) AS duration_99th, mean(sql_duration) AS duration_mean INTO gitlab.seven_days.grape_internal_authorized_keys_sql_timings_per_minute FROM rails_transactions WHERE request_uri = '/api/v3/internal/authorized_keys' GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY rails_transaction_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS duration_95th, percentile("duration", 99.000) AS duration_99th, mean("duration") AS duration_mean, percentile(sql_duration, 95.000) AS sql_duration_95th, percentile(sql_duration, 99.000) AS sql_duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(view_duration, 95.000) AS view_duration_95th, percentile(view_duration, 99.000) AS view_duration_99th, mean(view_duration) AS view_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(cache_duration) AS cache_duration_mean INTO gitlab.seven_days.rails_transaction_timings FROM rails_transactions GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY sidekiq_transaction_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS duration_95th, percentile("duration", 99.000) AS duration_99th, mean("duration") AS duration_mean, percentile(sql_duration, 95.000) AS sql_duration_95th, percentile(sql_duration, 99.000) AS sql_duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(view_duration, 95.000) AS view_duration_95th, percentile(view_duration, 99.000) AS view_duration_99th, mean(view_duration) AS view_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(cache_duration) AS cache_duration_mean INTO gitlab.seven_days.sidekiq_transaction_timings FROM sidekiq_transactions GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY grape_transaction_counts_seven_days ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.seven_days.grape_transaction_counts FROM rails_transactions WHERE action !~ /.+/ GROUP BY time(1m) END; -CREATE CONTINUOUS QUERY grape_transaction_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS duration_95th, percentile("duration", 99.000) AS duration_99th, mean("duration") AS duration_mean, percentile(sql_duration, 95.000) AS sql_duration_95th, percentile(sql_duration, 99.000) AS sql_duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(cache_duration) AS cache_duration_mean INTO gitlab.seven_days.grape_transaction_timings FROM rails_transactions WHERE action !~ /.+/ GROUP BY time(1m) END; +CREATE RETENTION POLICY default ON gitlab DURATION 1h REPLICATION 1 DEFAULT +CREATE RETENTION POLICY downsampled ON gitlab DURATION 7d REPLICATION 1 +CREATE CONTINUOUS QUERY grape_git_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_git_timings_per_action FROM gitlab."default".rails_method_calls WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND method =~ /^(Rugged|Gitlab::Git)/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY grape_markdown_render_timings_overall ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.grape_markdown_render_timings_overall FROM gitlab."default".rails_transactions WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_markdown_render_timings_per_action ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.grape_markdown_render_timings_per_action FROM gitlab."default".rails_transactions WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY grape_markdown_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_markdown_timings_overall FROM gitlab."default".rails_method_calls WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND method =~ /^Banzai/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_method_call_timings_per_action_and_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_method_call_timings_per_action_and_method FROM gitlab."default".rails_method_calls WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), method, action END; +CREATE CONTINUOUS QUERY grape_method_call_timings_per_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_method_call_timings_per_method FROM gitlab."default".rails_method_calls WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), method END; +CREATE CONTINUOUS QUERY grape_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.grape_transaction_counts_overall FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_transaction_counts_per_action ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.grape_transaction_counts_per_action FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY grape_transaction_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.grape_transaction_timings_overall FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_transaction_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.grape_transaction_timings_per_action FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_file_descriptor_counts ON gitlab BEGIN SELECT sum(value) AS count INTO gitlab.downsampled.rails_file_descriptor_counts FROM gitlab."default".rails_file_descriptors GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_gc_counts ON gitlab BEGIN SELECT sum(count) AS total, sum(minor_gc_count) AS minor, sum(major_gc_count) AS major INTO gitlab.downsampled.rails_gc_counts FROM gitlab."default".rails_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_gc_timings ON gitlab BEGIN SELECT mean(total_time) AS duration_mean, percentile(total_time, 95) AS duration_95th, percentile(total_time, 99) AS duration_99th INTO gitlab.downsampled.rails_gc_timings FROM gitlab."default".rails_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_git_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_git_timings_per_action FROM gitlab."default".rails_method_calls WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND method =~ /^(Rugged|Gitlab::Git)/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_markdown_render_timings_overall ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.rails_markdown_render_timings_overall FROM gitlab."default".rails_transactions WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_markdown_render_timings_per_action ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.rails_markdown_render_timings_per_action FROM gitlab."default".rails_transactions WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_markdown_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_markdown_timings_overall FROM gitlab."default".rails_method_calls WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND method =~ /^Banzai/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_memory_usage_overall ON gitlab BEGIN SELECT mean(value) AS memory_mean, percentile(value, 95) AS memory_95th, percentile(value, 99) AS memory_99th INTO gitlab.downsampled.rails_memory_usage_overall FROM gitlab."default".rails_memory_usage GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_method_call_timings_per_action_and_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_method_call_timings_per_action_and_method FROM gitlab."default".rails_method_calls WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), method, action END; +CREATE CONTINUOUS QUERY rails_method_call_timings_per_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_method_call_timings_per_method FROM gitlab."default".rails_method_calls WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), method END; +CREATE CONTINUOUS QUERY rails_object_counts_overall ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.rails_object_counts_overall FROM gitlab."default".rails_object_counts GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_object_counts_per_type ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.rails_object_counts_per_type FROM gitlab."default".rails_object_counts GROUP BY time(1m), type END; +CREATE CONTINUOUS QUERY rails_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.rails_transaction_counts_overall FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_transaction_counts_per_action ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.rails_transaction_counts_per_action FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_transaction_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.rails_transaction_timings_overall FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_transaction_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.rails_transaction_timings_per_action FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_view_timings_per_action_and_view ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_view_timings_per_action_and_view FROM gitlab."default".rails_views WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), action, view END; +CREATE CONTINUOUS QUERY sidekiq_file_descriptor_counts ON gitlab BEGIN SELECT sum(value) AS count INTO gitlab.downsampled.sidekiq_file_descriptor_counts FROM gitlab."default".sidekiq_file_descriptors GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_gc_counts ON gitlab BEGIN SELECT sum(count) AS total, sum(minor_gc_count) AS minor, sum(major_gc_count) AS major INTO gitlab.downsampled.sidekiq_gc_counts FROM gitlab."default".sidekiq_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_gc_timings ON gitlab BEGIN SELECT mean(total_time) AS duration_mean, percentile(total_time, 95) AS duration_95th, percentile(total_time, 99) AS duration_99th INTO gitlab.downsampled.sidekiq_gc_timings FROM gitlab."default".sidekiq_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_git_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_git_timings_per_action FROM gitlab."default".sidekiq_method_calls WHERE method =~ /^(Rugged|Gitlab::Git)/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_markdown_render_timings_overall ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.sidekiq_markdown_render_timings_overall FROM gitlab."default".sidekiq_transactions WHERE (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_markdown_render_timings_per_action ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.sidekiq_markdown_render_timings_per_action FROM gitlab."default".sidekiq_transactions WHERE (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_markdown_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_markdown_timings_overall FROM gitlab."default".sidekiq_method_calls WHERE method =~ /^Banzai/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_memory_usage_overall ON gitlab BEGIN SELECT mean(value) AS memory_mean, percentile(value, 95) AS memory_95th, percentile(value, 99) AS memory_99th INTO gitlab.downsampled.sidekiq_memory_usage_overall FROM gitlab."default".sidekiq_memory_usage GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_method_call_timings_per_action_and_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_method_call_timings_per_action_and_method FROM gitlab."default".sidekiq_method_calls GROUP BY time(1m), method, action END; +CREATE CONTINUOUS QUERY sidekiq_method_call_timings_per_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_method_call_timings_per_method FROM gitlab."default".sidekiq_method_calls GROUP BY time(1m), method END; +CREATE CONTINUOUS QUERY sidekiq_object_counts_overall ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.sidekiq_object_counts_overall FROM gitlab."default".sidekiq_object_counts GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_object_counts_per_type ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.sidekiq_object_counts_per_type FROM gitlab."default".sidekiq_object_counts GROUP BY time(1m), type END; +CREATE CONTINUOUS QUERY sidekiq_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.sidekiq_transaction_counts_overall FROM gitlab."default".sidekiq_transactions GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_transaction_counts_per_action ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.sidekiq_transaction_counts_per_action FROM gitlab."default".sidekiq_transactions GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_transaction_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.sidekiq_transaction_timings_overall FROM gitlab."default".sidekiq_transactions GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_transaction_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.sidekiq_transaction_timings_per_action FROM gitlab."default".sidekiq_transactions GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_view_timings_per_action_and_view ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_view_timings_per_action_and_view FROM gitlab."default".sidekiq_views GROUP BY time(1m), action, view END; +CREATE CONTINUOUS QUERY web_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.web_transaction_counts_overall FROM gitlab."default".rails_transactions GROUP BY time(1m) END; ``` ## Import Dashboards diff --git a/doc/monitoring/performance/influxdb_configuration.md b/doc/monitoring/performance/influxdb_configuration.md index 63aa03985ef..c30cd2950d8 100644 --- a/doc/monitoring/performance/influxdb_configuration.md +++ b/doc/monitoring/performance/influxdb_configuration.md @@ -181,7 +181,7 @@ Read more on: - [Introduction to GitLab Performance Monitoring](introduction.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Schema](influxdb_schema.md) -- [Grafana Install/Configuration](grafana_configuration.md +- [Grafana Install/Configuration](grafana_configuration.md) [influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management [influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ diff --git a/doc/monitoring/performance/influxdb_schema.md b/doc/monitoring/performance/influxdb_schema.md index d31b3788f36..41861860b6d 100644 --- a/doc/monitoring/performance/influxdb_schema.md +++ b/doc/monitoring/performance/influxdb_schema.md @@ -85,4 +85,4 @@ Read more on: - [Introduction to GitLab Performance Monitoring](introduction.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Configuration](influxdb_configuration.md) -- [Grafana Install/Configuration](grafana_configuration.md +- [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/operations/sidekiq_memory_killer.md b/doc/operations/sidekiq_memory_killer.md index 811c2192a19..b5e78348989 100644 --- a/doc/operations/sidekiq_memory_killer.md +++ b/doc/operations/sidekiq_memory_killer.md @@ -36,5 +36,5 @@ The MemoryKiller is controlled using environment variables. Existing jobs get 30 seconds to finish. After that, the MemoryKiller tells Sidekiq to shut down, and an external supervision mechanism (e.g. Runit) must restart Sidekiq. -- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL`: defaults to 'SIGTERM'. The name of +- `SIDEKIQ_MEMORY_KILLER_SHUTDOWN_SIGNAL`: defaults to `SIGKILL`. The name of the final signal sent to the Sidekiq process when we want it to shut down. diff --git a/doc/permissions/permissions.md b/doc/permissions/permissions.md index 6219693b8a8..b76ce31cbad 100644 --- a/doc/permissions/permissions.md +++ b/doc/permissions/permissions.md @@ -27,6 +27,7 @@ documentation](../workflow/add-user/add-user.md). | Manage issue tracker | | ✓ | ✓ | ✓ | ✓ | | Manage labels | | ✓ | ✓ | ✓ | ✓ | | See a commit status | | ✓ | ✓ | ✓ | ✓ | +| See a container registry | | ✓ | ✓ | ✓ | ✓ | | Manage merge requests | | | ✓ | ✓ | ✓ | | Create new merge request | | | ✓ | ✓ | ✓ | | Create new branches | | | ✓ | ✓ | ✓ | @@ -37,6 +38,8 @@ documentation](../workflow/add-user/add-user.md). | Write a wiki | | | ✓ | ✓ | ✓ | | Cancel and retry builds | | | ✓ | ✓ | ✓ | | Create or update commit status | | | ✓ | ✓ | ✓ | +| Update a container registry | | | ✓ | ✓ | ✓ | +| Remove a container registry image | | | ✓ | ✓ | ✓ | | Create new milestones | | | | ✓ | ✓ | | Add new team members | | | | ✓ | ✓ | | Push to protected branches | | | | ✓ | ✓ | diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index 6be954ad68b..a49c43b8ef2 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -8,4 +8,4 @@ - [User management](user_management.md) - [Webhooks](web_hooks.md) - [Import](import.md) of git repositories in bulk -- [Rebuild authorized_keys file](http://doc.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators +- [Rebuild authorized_keys file](http://docs.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 4329ac30a1c..fa976134341 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -295,36 +295,49 @@ Deleting tmp directories...[DONE] ### Omnibus installations -We will assume that you have installed GitLab from an omnibus package and run -`sudo gitlab-ctl reconfigure` at least once. +This procedure assumes that: -First make sure your backup tar file is in `/var/opt/gitlab/backups` (or wherever `gitlab_rails['backup_path']` points to). +- You have installed the exact same version of GitLab Omnibus with which the + backup was created +- You have run `sudo gitlab-ctl reconfigure` at least once +- GitLab is running. If not, start it using `sudo gitlab-ctl start`. + +First make sure your backup tar file is in the backup directory described in the +`gitlab.rb` configuration `gitlab_rails['backup_path']`. The default is +`/var/opt/gitlab/backups`. ```shell sudo cp 1393513186_gitlab_backup.tar /var/opt/gitlab/backups/ ``` -Next, restore the backup by running the restore command. You need to specify the -timestamp of the backup you are restoring. +Stop the processes that are connected to the database. Leave the rest of GitLab +running: ```shell -# Stop processes that are connected to the database sudo gitlab-ctl stop unicorn sudo gitlab-ctl stop sidekiq +# Verify +sudo gitlab-ctl status +``` +Next, restore the backup, specifying the timestamp of the backup you wish to +restore: + +```shell # This command will overwrite the contents of your GitLab database! sudo gitlab-rake gitlab:backup:restore BACKUP=1393513186 +``` -# Start GitLab -sudo gitlab-ctl start +Restart and check GitLab: -# Check GitLab +```shell +sudo gitlab-ctl start sudo gitlab-rake gitlab:check SANITIZE=true ``` If there is a GitLab version mismatch between your backup tar file and the installed -version of GitLab, the restore command will abort with an error. Install a package for -the [required version](https://www.gitlab.com/downloads/archives/) and try again. +version of GitLab, the restore command will abort with an error. Install the +[correct GitLab version](https://www.gitlab.com/downloads/archives/) and try again. ## Configure cron to make daily backups diff --git a/doc/security/README.md b/doc/security/README.md index 4cd0fdd4094..38706e48ec5 100644 --- a/doc/security/README.md +++ b/doc/security/README.md @@ -8,3 +8,4 @@ - [User File Uploads](user_file_uploads.md) - [How we manage the CRIME vulnerability](crime_vulnerability.md) - [Enforce Two-factor authentication](two_factor_authentication.md) +- [Send email confirmation on sign-up](user_email_confirmation.md) diff --git a/doc/security/user_email_confirmation.md b/doc/security/user_email_confirmation.md new file mode 100644 index 00000000000..4293944ae8b --- /dev/null +++ b/doc/security/user_email_confirmation.md @@ -0,0 +1,7 @@ +# User email confirmation at sign-up + +Gitlab admin can enable email confirmation on sign-up, if you want to confirm all +user emails before they are able to sign-in. + +In the Admin area under **Settings** (`/admin/application_settings`), go to section +**Sign-in Restrictions** and look for **Send confirmation email on sign-up** option. diff --git a/doc/update/8.7-to-8.8.md b/doc/update/8.7-to-8.8.md new file mode 100644 index 00000000000..b4d9212289c --- /dev/null +++ b/doc/update/8.7-to-8.8.md @@ -0,0 +1,154 @@ +# From 8.7 to 8.8 + +Make sure you view this update guide from the tag (version) of GitLab you would +like to install. In most cases this should be the highest numbered production +tag (without rc in it). You can select the tag in the version dropdown at the +top left corner of GitLab (below the menu bar). + +If the highest number stable branch is unclear please check the +[GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation +guide links by version. + +### 1. Stop server + + sudo service gitlab stop + +### 2. Backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 3. Get latest code + +```bash +sudo -u git -H git fetch --all +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +``` + +For GitLab Community Edition: + +```bash +sudo -u git -H git checkout 8-8-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +sudo -u git -H git checkout 8-8-stable-ee +``` + +### 4. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v2.7.2 +``` + +### 5. Update gitlab-workhorse + +Install and compile gitlab-workhorse. This requires +[Go 1.5](https://golang.org/dl) which should already be on your system from +GitLab 8.1. + +```bash +cd /home/git/gitlab-workhorse +sudo -u git -H git fetch --all +sudo -u git -H git checkout v0.7.1 +sudo -u git -H make +``` + +### 6. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without postgres') +sudo -u git -H bundle install --without postgres development test --deployment + +# PostgreSQL installations (note: the line below states '--without mysql') +sudo -u git -H bundle install --without mysql development test --deployment + +# Optional: clean up old gems +sudo -u git -H bundle clean + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Clean up assets and cache +sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production + +``` + +### 7. Update configuration files + +#### Git configuration + +Disable `git gc --auto` because GitLab runs `git gc` for us already. + +```sh +sudo -u git -H git config --global gc.auto 0 +``` + +#### Nginx configuration + +Ensure you're still up-to-date with the latest NGINX configuration changes: + +```sh +# For HTTPS configurations +git diff origin/8-7-stable:lib/support/nginx/gitlab-ssl origin/8-8-stable:lib/support/nginx/gitlab-ssl + +# For HTTP configurations +git diff origin/8-7-stable:lib/support/nginx/gitlab origin/8-8-stable:lib/support/nginx/gitlab +``` + +If you are using Apache instead of NGINX please see the updated [Apache templates]. +Also note that because Apache does not support upstreams behind Unix sockets you +will need to let gitlab-workhorse listen on a TCP port. You can do this +via [/etc/default/gitlab]. + +[Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache +[/etc/default/gitlab]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-8-stable/lib/support/init.d/gitlab.default.example#L37 + +#### Init script + +Ensure you're still up-to-date with the latest init script changes: + + sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab + +### 8. Start application + + sudo service gitlab start + sudo service nginx restart + +### 9. Check application status + +Check if GitLab and its environment are configured correctly: + + sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production + +To make sure you didn't miss anything run a more thorough check: + + sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production + +If all items are green, then congratulations, the upgrade is complete! + +## Things went south? Revert to previous version (8.6) + +### 1. Revert the code to the previous version + +Follow the [upgrade guide from 8.6 to 8.7](8.6-to-8.7.md), except for the +database migration (the backup is already migrated to the previous version). + +### 2. Restore from the backup + +```bash +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above. diff --git a/doc/update/README.md b/doc/update/README.md index a770633c9b8..975d72164b4 100644 --- a/doc/update/README.md +++ b/doc/update/README.md @@ -29,7 +29,7 @@ Based on your installation, choose a section below that fits your needs. ## Omnibus Packages -- The [Omnibus update guide](http://doc.gitlab.com/omnibus/update/README.html) +- The [Omnibus update guide](http://docs.gitlab.com/omnibus/update/README.html) contains the steps needed to update an Omnibus GitLab package. ## Installation from source @@ -86,10 +86,10 @@ possible. information about configuring GitLab to work with a MySQL database. - [Restoring from backup after a failed upgrade](restore_after_failure.md) -[omnidocker]: http://doc.gitlab.com/omnibus/docker/README.html +[omnidocker]: http://docs.gitlab.com/omnibus/docker/README.html [source-ee]: https://gitlab.com/gitlab-org/gitlab-ee/tree/master/doc/update [source-ce]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update [ee-ce]: ../downgrade_ee_to_ce/README.md [ce]: https://about.gitlab.com/features/#community [ee]: https://about.gitlab.com/features/#enterprise -[omni-ce-ee]: http://doc.gitlab.com/omnibus/update/README.html#from-community-edition-to-enterprise-edition +[omni-ce-ee]: http://docs.gitlab.com/omnibus/update/README.html#from-community-edition-to-enterprise-edition diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index f446ed0a35b..60729316cde 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -47,7 +47,7 @@ sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_SHELL_VERSION` -b v`ca ```bash cd /home/git/gitlab-workhorse sudo -u git -H git fetch -sudo -u git -H git checkout `cat /home/git/gitlab/GITLAB_WORKHORSE_VERSION` -b `cat /home/git/gitlab/GITLAB_WORKHORSE_VERSION` +sudo -u git -H git checkout v`cat /home/git/gitlab/GITLAB_WORKHORSE_VERSION` -b v`cat /home/git/gitlab/GITLAB_WORKHORSE_VERSION` sudo -u git -H make ``` diff --git a/doc/web_hooks/web_hooks.md b/doc/web_hooks/web_hooks.md index c1c51302e79..45506ac1d7c 100644 --- a/doc/web_hooks/web_hooks.md +++ b/doc/web_hooks/web_hooks.md @@ -13,6 +13,19 @@ You can configure webhooks to listen for specific events like pushes, issues or Webhooks can be used to update an external issue tracker, trigger CI builds, update a backup mirror, or even deploy to your production server. +## Webhook endpoint tips + +If you are writing your own endpoint (web server) that will receive +GitLab webhooks keep in mind the following things: + +- Your endpoint should send its HTTP response as fast as possible. If + you wait too long, GitLab may decide the hook failed and retry it. +- Your endpoint should ALWAYS return a valid HTTP response. If you do + not do this then GitLab will think the hook failed and retry it. + Most HTTP libraries take care of this for you automatically but if + you are writing a low-level hook this is important to remember. +- GitLab ignores the HTTP status code returned by your endpoint. + ## SSL Verification By default, the SSL certificate of the webhook endpoint is verified based on diff --git a/doc/workflow/gitlab_flow.md b/doc/workflow/gitlab_flow.md index 1b354bcc0f1..2b2f140f8bf 100644 --- a/doc/workflow/gitlab_flow.md +++ b/doc/workflow/gitlab_flow.md @@ -131,7 +131,7 @@ When you feel comfortable with it to be merged you assign it to the person that There is room for more feedback and after the assigned person feels comfortable with the result the branch is merged. If the assigned person does not feel comfortable they can close the merge request without merging. -In GitLab it is common to protect the long-lived branches (e.g. the master branch) so that normal developers [can't modify these protected branches](http://doc.gitlab.com/ce/permissions/permissions.html). +In GitLab it is common to protect the long-lived branches (e.g. the master branch) so that normal developers [can't modify these protected branches](http://docs.gitlab.com/ce/permissions/permissions.html). So if you want to merge it into a protected branch you assign it to someone with master authorizations. ## Issues with GitLab flow @@ -187,7 +187,7 @@ If you have an issue that spans across multiple repositories, the best thing is  With git you can use an interactive rebase (`rebase -i`) to squash multiple commits into one and reorder them. -In GitLab EE and .com you can also [rebase before merge](http://doc.gitlab.com/ee/workflow/rebase_before_merge.html) from the web interface. +In GitLab EE and .com you can also [rebase before merge](http://docs.gitlab.com/ee/workflow/rebase_before_merge.html) from the web interface. This functionality is useful if you made a couple of commits for small changes during development and want to replace them with a single commit or if you want to make the order more logical. However you should never rebase commits you have pushed to a remote server. Somebody can have referred to the commits or cherry-picked them. diff --git a/doc/workflow/groups.md b/doc/workflow/groups.md index 52bf611dc5e..34ada1774d8 100644 --- a/doc/workflow/groups.md +++ b/doc/workflow/groups.md @@ -54,7 +54,7 @@ If necessary, you can increase the access level of an individual user for a spec ## Managing group memberships via LDAP In GitLab Enterprise Edition it is possible to manage GitLab group memberships using LDAP groups. -See [the GitLab Enterprise Edition documentation](http://doc.gitlab.com/ee/integration/ldap.html) for more information. +See [the GitLab Enterprise Edition documentation](http://docs.gitlab.com/ee/integration/ldap.html) for more information. ## Allowing only admins to create groups diff --git a/doc/workflow/importing/import_projects_from_github.md b/doc/workflow/importing/import_projects_from_github.md index e670e415c71..a7dfac2c120 100644 --- a/doc/workflow/importing/import_projects_from_github.md +++ b/doc/workflow/importing/import_projects_from_github.md @@ -44,5 +44,5 @@ case the namespace is taken, the project will be imported on the user's namespace.
[gh-import]: ../../integration/github.md "GitHub integration"
-[ee-gh]: http://doc.gitlab.com/ee/integration/github.html "GitHub integration for GitLab EE"
+[ee-gh]: http://docs.gitlab.com/ee/integration/github.html "GitHub integration for GitLab EE"
[new-project]: ../../gitlab-basics/create-project.md "How to create a new project in GitLab"
diff --git a/doc/workflow/importing/import_projects_from_gitlab_com.md b/doc/workflow/importing/import_projects_from_gitlab_com.md index 1117db98e7e..dcc00074b75 100644 --- a/doc/workflow/importing/import_projects_from_gitlab_com.md +++ b/doc/workflow/importing/import_projects_from_gitlab_com.md @@ -2,7 +2,7 @@ You can import your existing GitLab.com projects to your GitLab instance. But keep in mind that it is possible only if GitLab support is enabled on your GitLab instance. -You can read more about GitLab support [here](http://doc.gitlab.com/ce/integration/gitlab.html) +You can read more about GitLab support [here](http://docs.gitlab.com/ce/integration/gitlab.html) To get to the importer page you need to go to "New project" page.  diff --git a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md index 31620044b15..9fe065fa680 100644 --- a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md +++ b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md @@ -4,7 +4,7 @@ Managing large files such as audio, video and graphics files has always been one of the shortcomings of Git. The general recommendation is to not have Git repositories larger than 1GB to preserve performance. -GitLab already supports [managing large files with git annex](http://doc.gitlab.com/ee/workflow/git_annex.html) +GitLab already supports [managing large files with git annex](http://docs.gitlab.com/ee/workflow/git_annex.html) (EE only), however in certain environments it is not always convenient to use different commands to differentiate between the large files and regular ones. @@ -127,7 +127,7 @@ To prevent this from happening, set the lfs url in project Git config: ```bash -git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs/objects/batch" +git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs" ``` ### Credentials are always required when pushing an object @@ -152,4 +152,4 @@ If you are using OS X you can use `osxkeychain` to store and encrypt your creden For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases). More details about various methods of storing the user credentials can be found -on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
\ No newline at end of file +on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). diff --git a/doc/workflow/merge_requests.md b/doc/workflow/merge_requests.md index 6d57b5d98cd..1b5718c91c1 100644 --- a/doc/workflow/merge_requests.md +++ b/doc/workflow/merge_requests.md @@ -12,9 +12,9 @@ Locate the section for your GitLab remote in the `.git/config` file. It looks li fetch = +refs/heads/*:refs/remotes/origin/* ``` -Now add the line `fetch = +refs/merge-requests/*/head:refs/remotes/origin/merge-requests/*` to this section. +Now add the line `fetch = +refs/merge-requests/*/head:refs/remotes/origin/merge-requests/*` to this section. -It should looks like this: +It should look like this: ``` [remote "origin"] @@ -43,7 +43,7 @@ $ git checkout origin/merge-requests/1  -It you add `w=1` option to URL, you can see diff without whitespace changes. +If you click the "Hide whitespace changes" button, you can see the diff without whitespace changes.  diff --git a/doc/workflow/merge_requests/commit_compare.png b/doc/workflow/merge_requests/commit_compare.png Binary files differindex 46b3a56a59b..dfd7ee220f0 100644 --- a/doc/workflow/merge_requests/commit_compare.png +++ b/doc/workflow/merge_requests/commit_compare.png diff --git a/doc/workflow/merge_requests/merge_request_diff.png b/doc/workflow/merge_requests/merge_request_diff.png Binary files differindex ed08ae91bec..f368423c746 100644 --- a/doc/workflow/merge_requests/merge_request_diff.png +++ b/doc/workflow/merge_requests/merge_request_diff.png diff --git a/doc/workflow/merge_requests/merge_request_diff_without_whitespace.png b/doc/workflow/merge_requests/merge_request_diff_without_whitespace.png Binary files differindex 67d67a64d12..b2d03bb66f9 100644 --- a/doc/workflow/merge_requests/merge_request_diff_without_whitespace.png +++ b/doc/workflow/merge_requests/merge_request_diff_without_whitespace.png diff --git a/doc/workflow/shortcuts.png b/doc/workflow/shortcuts.png Binary files differindex 83e562d6929..beb6c53ec77 100644 --- a/doc/workflow/shortcuts.png +++ b/doc/workflow/shortcuts.png |
