diff options
Diffstat (limited to 'doc')
48 files changed, 920 insertions, 248 deletions
diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index b1be078d672..341a00009e5 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -19,7 +19,15 @@ and the [protocol documentation](https://github.com/git/git/blob/master/Document From the client side, `git` `v2.18.0` or newer must be installed. From the server side, if we want to configure SSH we need to set the `sshd` -server to accept the `GIT_PROTOCOL` environment, +server to accept the `GIT_PROTOCOL` environment. + +In installations using [GitLab Helm Charts](../install/kubernetes/gitlab_chart.md) +and [All-in-one docker image](https://docs.gitlab.com/omnibus/docker/), the SSH +service is already configured to accept the `GIT_PROTOCOL` environment and users +need not do anything more. + +For Omnibus GitLab and installations from source, you have to manually update +the SSH configuration of your server: ``` # /etc/ssh/sshd_config diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 7c1ef43499d..a9ba40c870c 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -684,7 +684,7 @@ cache, queues, and shared_state. To make this work with Sentinel: ``` 1. Note that for each persistence class, GitLab will default to using the configuration specified in `gitlab_rails['redis_sentinels']` unless - overriden by the settings above. + overridden by the settings above. 1. Be sure to include BOTH configuration options for each persistent classes. For example, if you choose to configure a cache instance, you must specify both `gitlab_rails['redis_cache_instance']` and `gitlab_rails['redis_cache_sentinels']` for GitLab to generate the proper configuration files. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index b1b670c3b42..2d9fdedcbeb 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -1,4 +1,4 @@ -# GitLab Prometheus +# Monitoring GitLab with Prometheus > **Notes:** > - Prometheus and the various exporters listed in this page are bundled in the @@ -24,10 +24,10 @@ dashboard tool like [Grafana]. ## Configuring Prometheus ->**Note:** +NOTE: **Note:** For installations from source you'll have to install and configure it yourself. -Prometheus and it's exporters are on by default, starting with GitLab 9.0. +Prometheus and its exporters are on by default, starting with GitLab 9.0. Prometheus will run as the `gitlab-prometheus` user and listen on `http://localhost:9090`. By default Prometheus is only accessible from the GitLab server itself. Each exporter will be automatically set up as a @@ -43,17 +43,17 @@ To disable Prometheus and all of its exporters, as well as any added in the futu ``` 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to - take effect + take effect. -## Changing the port and address Prometheus listens on +### Changing the port and address Prometheus listens on ->**Note:** +NOTE: **Note:** The following change was added in [GitLab Omnibus 8.17][1261]. Although possible, it's not recommended to change the port Prometheus listens on as this might affect or conflict with other services running on the GitLab server. Proceed at your own risk. -In order to access Prometheus from outside the GitLab server you will need to +In order to access Prometheus from outside the GitLab server you will need to set a FQDN or IP in `prometheus['listen_address']`. To change the address/port that Prometheus listens on: @@ -77,6 +77,60 @@ To change the address/port that Prometheus listens on: 1. Save the file and [reconfigure GitLab][reconfigure] for the changes to take effect +### Using an external Prometheus server + +NOTE: **Note:** +Prometheus and most exporters do not support authentication. We do not recommend exposing them outside the local network. + +A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for highly available deployments of GitLab with multiple nodes. + +To use an external Prometheus server: + +1. Edit `/etc/gitlab/gitlab.rb`. +1. Disable the bundled Prometheus: + + ```ruby + prometheus['enable'] = false + ``` + +1. Set each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, for example: + + ```ruby + gitlab_monitor['listen_address'] = '0.0.0.0' + gitlab_monitor['listen_port'] = '9168' + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + ``` + +1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/). +1. Add the Prometheus server IP address to the [monitoring IP whitelist](../ip_whitelist.html). For example: + + ```ruby + gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] + ``` + +1. [Reconfigure GitLab][reconfigure] to apply the changes +1. Edit the Prometheus server's configuration file. +1. Add each node's exporters to the Prometheus server's + [scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E). + For example, a sample snippet using `static_configs`: + + ```yaml + scrape_configs: + - job_name: 'gitlab_exporters' + static_configs: + - targets: ['1.1.1.1:9168', '1.1.1.1:9236', '1.1.1.1:9236', '1.1.1.1:9100', '1.1.1.1:9121', '1.1.1.1:9187'] + + - job_name: 'gitlab_metrics' + metrics_path: /-/metrics + static_configs: + - targets: ['1.1.1.1:443'] + ``` + +1. Restart the Prometheus server. + ## Viewing performance metrics You can visit `http://localhost:9090` for the dashboard that Prometheus offers by default. @@ -86,7 +140,7 @@ If SSL has been enabled on your GitLab instance, you may not be able to access Prometheus on the same browser as GitLab if using the same FQDN due to [HSTS][hsts]. We plan to [provide access via GitLab][multi-user-prometheus], but in the interim there are some workarounds: using a separate FQDN, using server IP, using a separate browser for Prometheus, resetting HSTS, or -having [Nginx proxy it][nginx-custom-config]. +having [NGINX proxy it][nginx-custom-config]. The performance data collected by Prometheus can be viewed directly in the Prometheus console or through a compatible dashboard tool. @@ -102,26 +156,7 @@ Sample Prometheus queries: - **Data transmitted:** `rate(node_network_transmit_bytes_total{device!="lo"}[5m])` - **Data received:** `rate(node_network_receive_bytes_total{device!="lo"}[5m])` -## Configuring Prometheus to monitor Kubernetes - -> Introduced in GitLab 9.0. -> Pod monitoring introduced in GitLab 9.4. - -If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][] to monitor them. - -To disable the monitoring of Kubernetes: - -1. Edit `/etc/gitlab/gitlab.rb` -1. Add or find and uncomment the following line and set it to `false`: - - ```ruby - prometheus['monitor_kubernetes'] = false - ``` - -1. Save the file and [reconfigure GitLab][reconfigure] for the changes to - take effect - -## GitLab Prometheus metrics +## GitLab metrics > Introduced in GitLab 9.3. @@ -129,17 +164,10 @@ GitLab monitors its own internal service metrics, and makes them available at th [➔ Read more about the GitLab Metrics.](gitlab_metrics.md) -## Prometheus exporters - -There are a number of libraries and servers which help in exporting existing -metrics from third-party systems as Prometheus metrics. This is useful for cases -where it is not feasible to instrument a given system with Prometheus metrics -directly (for example, HAProxy or Linux system stats). You can read more in the -[Prometheus exporters and integrations upstream documentation][prom-exporters]. +## Bundled software metrics -While you can use any exporter you like with your GitLab installation, the -following ones documented here are bundled in the Omnibus GitLab packages -making it easy to configure and use. +Many of the GitLab dependencies bundled in Omnibus GitLab are preconfigured to +export Prometheus metrics. ### Node exporter @@ -166,6 +194,25 @@ The GitLab monitor exporter allows you to measure various GitLab metrics, pulled [➔ Read more about the GitLab monitor exporter.](gitlab_monitor_exporter.md) +## Configuring Prometheus to monitor Kubernetes + +> Introduced in GitLab 9.0. +> Pod monitoring introduced in GitLab 9.4. + +If your GitLab server is running within Kubernetes, Prometheus will collect metrics from the Nodes and [annotated Pods](https://prometheus.io/docs/operating/configuration/#kubernetes_sd_config) in the cluster, including performance data on each container. This is particularly helpful if your CI/CD environments run in the same cluster, as you can use the [Prometheus project integration][prometheus integration] to monitor them. + +To disable the monitoring of Kubernetes: + +1. Edit `/etc/gitlab/gitlab.rb`. +1. Add or find and uncomment the following line and set it to `false`: + + ```ruby + prometheus['monitor_kubernetes'] = false + ``` + +1. Save the file and [reconfigure GitLab][reconfigure] for the changes to + take effect. + [grafana]: https://grafana.net [hsts]: https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security [multi-user-prometheus]: https://gitlab.com/gitlab-org/multi-user-prometheus diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index aec9a359ada..f85a1f791f9 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -48,11 +48,12 @@ _The uploads are stored by default in 1. Save the file and [restart GitLab][] for the changes to take effect. -### Using object storage +### Using object storage **[CORE ONLY]** > **Notes:** > -> - [Introduced][ee-3867] in [GitLab Enterprise Edition Premium][eep] 10.5. +> - [Introduced][ee-3867] in [GitLab Premium][eep] 10.5. +> - [Introduced][ce17358] in [GitLab Core][ce] 10.7. > - Since version 11.1, we support direct_upload to S3. If you don't want to use the local disk where GitLab is installed to store the @@ -197,4 +198,6 @@ _The uploads are stored by default in [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" [restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" [eep]: https://about.gitlab.com/gitlab-ee/ "GitLab Enterprise Edition Premium" +[ce]: https://about.gitlab.com/gitlab-ce/ "GitLab Community Edition" [ee-3867]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3867 +[ce-17358]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17358 diff --git a/doc/api/avatar.md b/doc/api/avatar.md index 7faed893066..aa6f7c185ae 100644 --- a/doc/api/avatar.md +++ b/doc/api/avatar.md @@ -4,7 +4,7 @@ ## Get a single avatar URL -Get a single avatar URL for a given email addres. If user with matching public +Get a single avatar URL for a given email address. If user with matching public email address is not found, results from external avatar services are returned. This endpoint can be accessed without authentication. In case public visibility is restricted, response will be `403 Forbidden` when unauthenticated. diff --git a/doc/api/commits.md b/doc/api/commits.md index 994eefa423f..7d9b52ec24f 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -290,7 +290,7 @@ Example response: ## Revert a commit -> [Introduced][ce-22919] in GitLab 11.6. +> [Introduced][ce-22919] in GitLab 11.5. Reverts a commit in a given branch. diff --git a/doc/api/events.md b/doc/api/events.md index ccac5b8bb60..e1c6b801a77 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -71,7 +71,7 @@ Parameters: Example request: ``` -curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/events&target_type=issue&action=created&after=2017-01-31&before=2017-03-01 +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/events?target_type=issue&action=created&after=2017-01-31&before=2017-03-01 ``` Example response: @@ -276,7 +276,7 @@ Parameters: Example request: ``` -curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/:project_id/events&target_type=issue&action=created&after=2017-01-31&before=2017-03-01 +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/:project_id/events?target_type=issue&action=created&after=2017-01-31&before=2017-03-01 ``` Example response: diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index 574be52801c..7b4c9a8fbb3 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -235,5 +235,22 @@ Response: } ``` +## Delete a pipeline + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22988) in GitLab 11.6. + +``` +DELETE /projects/:id/pipelines/:pipeline_id +``` + +| Attribute | Type | Required | Description | +|------------|---------|----------|---------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `pipeline_id` | integer | yes | The ID of a pipeline | + +``` +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" --request "DELETE" "https://gitlab.example.com/api/v4/projects/1/pipelines/46" +``` + [ce-5837]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5837 [ce-7209]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7209 diff --git a/doc/api/users.md b/doc/api/users.md index ee24aa09156..e3633c46041 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -1072,7 +1072,6 @@ Example response: [ { "active" : true, - "token" : "EsMo-vhKfXGwX9RKrwiy", "scopes" : [ "api" ], @@ -1089,7 +1088,6 @@ Example response: "read_user" ], "revoked" : true, - "token" : "ZcZRpLeEuQRprkRjYydY", "name" : "mytoken2", "created_at" : "2017-03-17T17:19:28.697Z", "id" : 3, @@ -1125,7 +1123,6 @@ Example response: ```json { "active" : true, - "token" : "EsMo-vhKfXGwX9RKrwiy", "scopes" : [ "api" ], @@ -1142,6 +1139,8 @@ Example response: > Requires admin permissions. +> Token values are returned once. Make sure you save it - you won't be able to access it again. + It creates a new impersonation token. Note that only administrators can do this. You are only able to create impersonation tokens to impersonate the user and perform both API calls and Git reads and writes. The user will not see these tokens in their profile diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/container_registry_checkbox.png b/doc/ci/examples/laravel_with_gitlab_and_envoy/img/container_registry_checkbox.png Binary files differdeleted file mode 100644 index a56c07a0da7..00000000000 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/container_registry_checkbox.png +++ /dev/null diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index b6989d229d1..b090ea014dc 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -444,9 +444,7 @@ On your GitLab project repository navigate to the **Registry** tab.  -You may need to [enable Container Registry](../../../user/project/container_registry.md#enable-the-container-registry-for-your-project) to your project to see this tab. You'll find it under your project's **Settings > General > Sharing and permissions**. - - +You may need to [enable Container Registry](../../../user/project/container_registry.md#enable-the-container-registry-for-your-project) to your project to see this tab. You'll find it under your project's **Settings > General > Permissions**. To start using Container Registry on our machine, we first need to login to the GitLab registry using our GitLab username and password: diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 371703a12c8..c628895ee1a 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -294,7 +294,7 @@ runners will not use regular runners, they must be tagged accordingly. [jobs]: #jobs [jobs-yaml]: yaml/README.md#jobs -[manual]: yaml/README.md#manual +[manual]: yaml/README.md#whenmanual [env-manual]: environments.md#manually-deploying-to-environments [stages]: yaml/README.md#stages [runners]: runners/README.html diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 4e8ce10c9cb..1d98e8426fe 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -17,7 +17,7 @@ There are two places defined variables can be used. On the: | Definition | Can be expanded? | Expansion place | Description | |--------------------------------------|-------------------|-----------------|--------------| -| `environment:url` | yes | GitLab | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<ul><li>Supported: all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules)</li><li>Not suported: variables defined in Runner's `config.toml` and variables created in job's `script`</li></ul> | +| `environment:url` | yes | GitLab | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<ul><li>Supported: all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules)</li><li>Not supported: variables defined in Runner's `config.toml` and variables created in job's `script`</li></ul> | | `environment:name` | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support: <ul><li>variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`)</li><li>[persisted variables](#persisted-variables)</li></ul> | | `variables` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `image` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index aab5f268ef9..ec709280085 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -476,6 +476,7 @@ docker build: - Dockerfile - docker/scripts/* - dockerfiles/**/* + - more_scripts/*.{rb,py,sh} ``` In the scenario above, if you are pushing multiple commits to GitLab to an @@ -485,6 +486,7 @@ one of the commits contains changes to either: - The `Dockerfile` file. - Any of the files inside `docker/scripts/` directory. - Any of the files and subfolders inside `dockerfiles` directory. +- Any of the files with `rb`, `py`, `sh` extensions inside `more_scripts` directory. CAUTION: **Warning:** There are some caveats when using this feature with new branches and tags. See @@ -1602,10 +1604,11 @@ test: ## `include` -> Introduced in [GitLab Edition Premium][ee] 10.5. -> Available for Starter, Premium and Ultimate [versions][gitlab-versions] since 10.6. +> Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 10.5. +> Available for Starter, Premium and Ultimate since 10.6. > Behaviour expanded in GitLab 10.8 to allow more flexible overriding. -> Available for Libre since [11.4](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21603) +> [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21603) +to GitLab Core in 11.4 Using the `include` keyword, you can allow the inclusion of external YAML files. @@ -1767,7 +1770,7 @@ stages: production: script: - - install_depedencies + - install_dependencies - deploy - notify_owner ``` diff --git a/doc/development/README.md b/doc/development/README.md index 14dfe8eb1f3..d2dd62ecac5 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -30,6 +30,7 @@ description: 'Learn how to contribute to GitLab.' ## Backend guides - [GitLab utilities](utilities.md) +- [Logging](logging.md) - [API styleguide](api_styleguide.md) Use this styleguide if you are contributing to the API. - [GraphQL API styleguide](api_graphql_styleguide.md) Use this @@ -51,6 +52,7 @@ description: 'Learn how to contribute to GitLab.' - [Prometheus metrics](prometheus_metrics.md) - [Guidelines for reusing abstractions](reusing_abstractions.md) - [DeclarativePolicy framework](policies.md) +- [Switching to Rails 5](switching_to_rails5.md) ## Performance guides diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 66d8a4f2f6e..01d99c46f89 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -10,39 +10,182 @@ For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/ Both EE and CE require some add-on components called gitlab-shell and Gitaly. These components are available from the [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. -## Physical office analogy +## GitLab Omnibus Component by Component -You can imagine GitLab as a physical office. +This document is designed to be consumed by systems adminstrators and GitLab Support Engineers who want to understand more about the internals of GitLab and how they work together. -**The repositories** are the goods GitLab handles. -They can be stored in a warehouse. -This can be either a hard disk, or something more complex, such as a NFS filesystem; +When deployed, GitLab should be considered the amalgamation of the below processes. When troubleshooting or debugging, be as specific as possible as to which component you are referencing. That should increase clarity and reduce confusion. -**Nginx** acts like the front-desk. -Users come to Nginx and request actions to be done by workers in the office; +### GitLab Process Descriptions -**The database** is a series of metal file cabinets with information on: - - The goods in the warehouse (metadata, issues, merge requests etc); - - The users coming to the front desk (permissions) +As of this writing, a fresh GitLab 11.3.0 install will show the following processes with `gitlab-ctl status`: -**Redis** is a communication board with “cubby holes” that can contain tasks for office workers; +``` +run: alertmanager: (pid 30829) 14207s; run: log: (pid 13906) 2432044s +run: gitaly: (pid 30771) 14210s; run: log: (pid 13843) 2432046s +run: gitlab-monitor: (pid 30788) 14209s; run: log: (pid 13868) 2432045s +run: gitlab-workhorse: (pid 30758) 14210s; run: log: (pid 13855) 2432046s +run: logrotate: (pid 30246) 3407s; run: log: (pid 13825) 2432047s +run: nginx: (pid 30849) 14207s; run: log: (pid 13856) 2432046s +run: node-exporter: (pid 30929) 14206s; run: log: (pid 13877) 2432045s +run: postgres-exporter: (pid 30935) 14206s; run: log: (pid 13931) 2432044s +run: postgresql: (pid 13133) 2432214s; run: log: (pid 13848) 2432046s +run: prometheus: (pid 30807) 14209s; run: log: (pid 13884) 2432045s +run: redis: (pid 30560) 14274s; run: log: (pid 13807) 2432047s +run: redis-exporter: (pid 30946) 14205s; run: log: (pid 13869) 2432045s +run: sidekiq: (pid 30953) 14205s; run: log: (pid 13810) 2432047s +run: unicorn: (pid 30960) 14204s; run: log: (pid 13809) 2432047s +``` + +### Layers + +GitLab can be considered to have two layers from a process perspective: + +- **Monitoring**: Anything from this layer is not required to deliver GitLab the application, but will allow administrators more insight into their infrastructure and what the service as a whole is doing. +- **Core**: Any process that is vital for the delivery of GitLab as as platform. If any of these processes halt there will be a GitLab outage. For the Core layer, you can further divide into: + - **Processors**: These processes are responsible for actually performing operations and presenting the service. + - **Data**: These services store/expose structured data for the GitLab service. + +### alertmanager + +- Omnibus configuration options +- Layer: Monitoring + +[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. + +### gitaly + +- [Omnibus confiugration options](https://gitlab.com/gitlab-org/gitaly/tree/master/doc/configuration) +- Layer: Core Service (Data) + +Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab. (Think GitLab.com or High Availablity Deployments) As of 11.3.0, This service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly). + +### gitlab-monitor + +- Omnibus configuration options +- Layer: Monitoring + +GitLab Monitor is a process disigned in house that allows us to export metrics about GitLab application internals to prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor) + +### gitlab-workhorse + +- Omnibus configuration options +- Layer: Core Service (Processor) + +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alieviate pressure from unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. + +### logrotate + +- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) +- Layer: Core Service + +GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common opensource offering. + +### nginx + +- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/nginx.html) +- Layer: Core Service (Processor) + +Nginx as as an ingress port for all HTTP requests and routes them to the approriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver. + +### node-exporter + +- [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/node_exporter.html) +- Layer: Monitoring + +[Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine. (Think CPU/Disk/Load) It's just a packaged version of the common open source offering from the Prometheus project. + +### postgres-exporter + +- [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/postgres_exporter.html) +- Layer: Monitoring + +[Postgres-exporter](https://github.com/wrouesnel/postgres_exporter) is the community provided Prometheus exporter that will deliver data about Postgres to prometheus for use in Grafana Dashboards. + +### postgresql + +- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/database.html) +- Layer: Core Service (Data) + +GitLab packages the popular Database to provide storage for Application meta data and user information. + +### prometheus + +- [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/) +- Layer: Monitoring + +Prometheus is a time-series tool that helps GitLab administrators expose metrics about the individual processes used to provide GitLab the service. + +### redis + +- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +- Layer: Core Service (Data) + +Redis is packaged to provide a place to store: + +- session data +- temporary cache information +- background job queues. + +### redis-exporter + +- [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/redis_exporter.html) +- Layer: Monitoring + +[Redis Exporter](https://github.com/oliver006/redis_exporter) is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Graphana. + +### sidekiq + +- Omnibus configuration options +- Layer: Core Service (Processor) + +Sidekiq is a Ruby background job processor that pulls jobs from the redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background. + +### unicorn + +- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/unicorn.html) +- Layer: Core Service (Processor) + +[Unicorn](https://bogomips.org/unicorn/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version. + +### Additional Processes + +### GitLab Pages + +TODO + +### Mattermost + +TODO + +## GitLab by Request Type + +GitLab provides two "interfaces" for end users to access the service: + +- Web HTTP Requests (Viewing the UI/API) +- Git HTTP/SSH Requests (Pushing/Pulling Git Data) + +It's important to understand the distinction as some processes are used in both and others are exclusive to a specific request type. + +### GitLab Web HTTP Request Cycle + +When making a request to an HTTP Endpoint (Think `/users/sign_in`) the request will take the following path through the GitLab Service: + +- nginx - Acts as our first line reverse proxy +- gitlab-workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on unicorn. +- unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn. +- Postgres/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retreive data. -**Sidekiq** is a worker that primarily handles sending out emails. -It takes tasks from the Redis communication board; -**A Unicorn worker** is a worker that handles quick/mundane tasks. -They work with the communication board (Redis). -Their job description: - - check permissions by checking the user session stored in a Redis “cubby hole”; - - make tasks for Sidekiq; - - fetch stuff from the warehouse or move things around in there; +### GitLab Git Request Cycle -**GitLab-shell** is a third kind of worker that takes orders from a fax machine (SSH) instead of the front desk (HTTP). -GitLab-shell communicates with Sidekiq via the “communication board” (Redis), and asks quick questions of the Unicorn workers either directly or via the front desk. +Below we describe the different pathing that HTTP vs. SSH Git requests will take. There is some overlap with the Web Request Cycle but also some differences. -**Gitaly** is a back desk that is specialized on reaching the disks to perform git operations efficiently and keep a copy of the result of costly operations. All git operations go through Gitaly. +### Web Request (80/443) +TODO -**GitLab Enterprise Edition (the application)** is the collection of processes and business practices that the office is run by. +### SSH Request (22) +TODO ## System Layout diff --git a/doc/development/changelog.md b/doc/development/changelog.md index f06d40d1dbb..cd0a1f46d27 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -133,15 +133,15 @@ If you're working on the GitLab EE repository, the entry will be added to ### Arguments -| Argument | Shorthand | Purpose | -| ----------------- | --------- | ---------------------------------------------------------------------------------------------------------- | -| [`--amend`] | | Amend the previous commit | -| [`--force`] | `-f` | Overwrite an existing entry | -| [`--merge-request`] | `-m` | Set merge request ID | -| [`--dry-run`] | `-n` | Don't actually write anything, just print | -| [`--git-username`] | `-u` | Use Git user.name configuration as the author | -| [`--type`] | `-t` | The category of the change, valid options are: added, fixed, changed, deprecated, removed, security, other | -| [`--help`] | `-h` | Print help message | +| Argument | Shorthand | Purpose | +| ----------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| [`--amend`] | | Amend the previous commit | +| [`--force`] | `-f` | Overwrite an existing entry | +| [`--merge-request`] | `-m` | Set merge request ID | +| [`--dry-run`] | `-n` | Don't actually write anything, just print | +| [`--git-username`] | `-u` | Use Git user.name configuration as the author | +| [`--type`] | `-t` | The category of the change, valid options are: `added`, `fixed`, `changed`, `deprecated`, `removed`, `security`, `performance`, `other` | +| [`--help`] | `-h` | Print help message | [`--amend`]: #-amend [`--force`]: #-force-or-f diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md index 350593cc813..1019a1fd0e2 100644 --- a/doc/development/feature_flags.md +++ b/doc/development/feature_flags.md @@ -33,7 +33,7 @@ You can follow the progress on that [in the issue on our issue tracker](https:// In general, it's better to have a group- or user-based gate, and you should prefer it over the use of percentage gates. This would make debugging easier, as you -filter for example logs and errors based on actors too. Futhermore, this allows +filter for example logs and errors based on actors too. Furthermore, this allows for enabling for the `gitlab-org` group first, while the rest of the users aren't impacted. diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index f58d79fccf1..c4ac53f45ac 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -41,6 +41,7 @@ are very appreciative of the work done by translators and proofreaders! - Nikita Grylov - [GitLab](https://gitlab.com/nixel2007), [Crowdin](https://crowdin.com/profile/nixel2007) - Alexy Lustin - [GitLab](https://gitlab.com/allustin), [Crowdin](https://crowdin.com/profile/lustin) - Spanish + - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) - Ukrainian - Volodymyr Sobotovych - [GitLab](https://gitlab.com/wheleph), [Crowdin](https://crowdin.com/profile/wheleph) - Andrew Vityuk - [GitLab](https://gitlab.com/3_1_3_u), [Crowdin](https://crowdin.com/profile/andruwa13) diff --git a/doc/development/logging.md b/doc/development/logging.md new file mode 100644 index 00000000000..abd08c420da --- /dev/null +++ b/doc/development/logging.md @@ -0,0 +1,144 @@ +# GitLab Developers Guide to Logging + +[GitLab Logs](../administration/logs.md) play a critical role for both +administrators and GitLab team members to diagnose problems in the field. + +## Don't use `Rails.logger` + +Currently `Rails.logger` calls all get saved into `production.log`, which contains +a mix of Rails' logs and other calls developers have inserted in the code base. +For example: + +``` +Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200 +Processing by Projects::TreeController#show as HTML + Parameters: {"project_id"=>"gitlabhq/yaml_db", "id"=>"master"} + + ... + + Namespaces"."created_at" DESC, "namespaces"."id" DESC LIMIT 1 [["id", 26]] + CACHE (0.0ms) SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."source_id" = $1 AND "members"."source_type" = $2 AND "members"."user_id" = 1 ORDER BY "members"."created_at" DESC, "members"."id" DESC LIMIT 1 [["source_id", 18], ["source_type", "Project"]] + CACHE (0.0ms) SELECT "members".* FROM "members" WHERE "members"."source_type" = 'Project' AND "members". + (1.4ms) SELECT COUNT(*) FROM "merge_requests" WHERE "merge_requests"."target_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened')) [["target_project_id", 18]] + Rendered layouts/nav/_project.html.haml (28.0ms) + Rendered layouts/_collapse_button.html.haml (0.2ms) + Rendered layouts/_flash.html.haml (0.1ms) + Rendered layouts/_page.html.haml (32.9ms) +Completed 200 OK in 166ms (Views: 117.4ms | ActiveRecord: 27.2ms) +``` + +These logs suffer from a number of problems: + +1. They often lack timestamps or other contextual information (e.g. project ID, user) +2. They may span multiple lines, which make them hard to find via Elasticsearch. +3. They lack a common structure, which make them hard to parse by log +forwarders, such as Logstash or Fluentd. This also makes them hard to +search. + +Note that currently on GitLab.com, any messages in `production.log` will +NOT get indexed by Elasticsearch due to the sheer volume and noise. They +do end up in Google Stackdriver, but it is still harder to search for +logs there. See the [GitLab.com logging +documentation](https://gitlab.com/gitlab-com/runbooks/blob/master/howto/logging.md) +for more details. + +## Use structured (JSON) logging + +Structured logging solves these problems. Consider the example from an API request: + +```json +{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30} +``` + +In a single line, we've included all the information that a user needs +to understand what happened: the timestamp, HTTP method and path, user +ID, etc. + +### How to use JSON logging + +Suppose you want to log the events that happen in a project +importer. You want to log issues created, merge requests, etc. as the +importer progresses. Here's what to do: + +1. Look at [the list of GitLab Logs](../administration/logs.md) to see +if your log message might belong with one of the existing log files. +1. If there isn't a good place, consider creating a new filename, but +check with a maintainer if it makes sense to do so. A log file should +make it easy for people to search pertinent logs in one place. For +example, `geo.log` contains all logs pertaining to GitLab Geo. +To create a new file: + 1. Choose a filename (e.g. `importer_json.log`). + 1. Create a new subclass of `Gitlab::JsonLogger`: + + ```ruby + module Gitlab + module Import + class Logger < ::Gitlab::JsonLogger + def self.file_name_noext + 'importer_json' + end + end + end + end + ``` + + 1. In your class where you want to log, you might initialize the logger as an instance variable: + + ```ruby + attr_accessor :logger + + def initialize + @logger = Gitlab::Import::Logger.build + end + ``` + + Note that it's useful to memoize this because creating a new logger + each time you log will open a file, adding unnecessary overhead. + +1. Now insert log messages into your code. When adding logs, + make sure to include all the context as key-value pairs: + + ```ruby + # BAD + logger.info("Unable to create project #{project.id}") + ``` + + ```ruby + # GOOD + logger.info("Unable to create project", project_id: project.id) + ``` + +1. Be sure to create a common base structure of your log messages. For example, + all messages might have `current_user_id` and `project_id` to make it easier + to search for activities by user for a given time. + +1. Do NOT mix and match types. Elasticsearch won't be able to index your + logs properly if you [mix integer and string + types](https://www.elastic.co/guide/en/elasticsearch/guide/current/mapping.html#_avoiding_type_gotchas): + + ```ruby + # BAD + logger.info("Import error", error: 1) + logger.info("Import error", error: "I/O failure") + ``` + + ```ruby + # GOOD + logger.info("Import error", error_code: 1, error: "I/O failure") + ``` + +## Additional steps with new log files + +1. Consider log retention settings. By default, Omnibus will rotate any +logs in `/var/log/gitlab/gitlab-rails/*.log` every hour and [keep at +most 30 compressed files](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate). +On GitLab.com, that setting is only 6 compressed files. These settings should suffice +for most users, but you may need to tweak them in [omnibus-gitlab](https://gitlab.com/gitlab-org/omnibus-gitlab). + +1. If you add a new file, submit an issue to the [production +tracker](https://gitlab.com/gitlab-com/gl-infra/production/issues) or +a merge request to the [gitlab_fluentd](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd) +project. See [this example](https://gitlab.com/gitlab-cookbooks/gitlab_fluentd/merge_requests/51/diffs). + +1. Be sure to update the [GitLab CE/EE documentation](../administration/logs.md) and the [GitLab.com +runbooks](https://gitlab.com/gitlab-com/runbooks/blob/master/howto/logging.md). diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index 6f31e5b82e5..e4e532bb4ed 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -187,12 +187,7 @@ end When adding a foreign-key constraint to either an existing or new column remember to also add a index on the column. -This is _required_ if the foreign-key constraint specifies -`ON DELETE CASCADE` or `ON DELETE SET NULL` behavior. On a cascading -delete, the [corresponding record needs to be retrieved using an -index](https://www.cybertec-postgresql.com/en/postgresql-indexes-and-foreign-keys/) -(otherwise, we'd need to scan the whole table) for subsequent update or -deletion. +This is _required_ for all foreign-keys. Here's an example where we add a new column with a foreign key constraint. Note it includes `index: true` to create an index for it. diff --git a/doc/development/new_fe_guide/development/testing.md b/doc/development/new_fe_guide/development/testing.md index 748478768de..082acbedcd2 100644 --- a/doc/development/new_fe_guide/development/testing.md +++ b/doc/development/new_fe_guide/development/testing.md @@ -231,7 +231,7 @@ Their abstraction level is comparable to how a user would interact with the UI. <details> <summary>Vuex stores</summary> When testing the frontend code of a page as a whole, the interaction between Vue components and Vuex stores is covered as well. -<details> +</details> ## Feature tests diff --git a/doc/development/switching_to_rails5.md b/doc/development/switching_to_rails5.md new file mode 100644 index 00000000000..c9a4ce1a1d1 --- /dev/null +++ b/doc/development/switching_to_rails5.md @@ -0,0 +1,27 @@ +# Switching to Rails 5 + +GitLab switched recently to Rails 5. This is a big change (especially for backend development) and it introduces couple of temporary inconveniences. + +## After the switch, I found a broken feature. What do I do? + +Many fixes and tweaks were done to make our codebase compatible with Rails 5, but it's possible that not all issues were found. If you find an bug, please create an issue and assign it the ~rails5 label. + +## It takes much longer to run CI pipelines that build GitLab. Why? + +We are temporarily running CI pipelines with Rails 4 and 5 so that we ensure we remain compatible with Rails 4 in case we must revert back to Rails 4 from Rails 5 (this can double the duration of CI pipelines). + +We might revert back to Rails 4 if we found a major issue we were unable to quickly fix. + +Once we are sure we can stay with Rails 5, we will stop running CI pipelines with Rails 4. + +## Can I skip running Rails 4 tests? + +If you are sure that your merge request doesn't introduce any incompatibility, you can just include `norails4` anywhere in your branch name and Rails 4 tests will be skipped. + +## CI is failing on my test with Rails 4. How can I debug it? + +You can run specs locally with Rails 4 using the following command: + +```sh +BUNDLE_GEMFILE=Gemfile.rails4 RAILS5=0 bundle exec rspec ... +``` diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 36d150c8a5b..1830641431e 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -24,7 +24,7 @@ Review Apps are automatically deployed by each pipeline, both in [`scripts/review_apps/review-apps.sh`][review-apps.sh] - These scripts are basically [our official Auto DevOps scripts][Auto-DevOps.gitlab-ci.yml] where the - default CNG images are overriden with the images built and stored in the + default CNG images are overridden with the images built and stored in the [`CNG-mirror` project's registry][cng-mirror-registry]. - Since we're using [the official GitLab Helm chart][helm-chart], this means you get a dedicated environment for your branch that's very close to what it @@ -33,7 +33,7 @@ Review Apps are automatically deployed by each pipeline, both in thanks to the direct link to it from the MR widget. The default username is `root` and its password can be found in the 1Password secure note named **gitlab-{ce,ee} Review App's root password** (note that there's currently - [a bug where the default password seems to be overriden][password-bug]). + [a bug where the default password seems to be overridden][password-bug]). **Additional notes:** diff --git a/doc/install/requirements.md b/doc/install/requirements.md index dcc6d75724e..1b7e0d1d0ab 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -104,7 +104,7 @@ features of GitLab work with MySQL/MariaDB: 1. MySQL support for subgroups was [dropped with GitLab 9.3][post]. See [issue #30472][30472] for more information. 1. Geo does [not support MySQL](https://docs.gitlab.com/ee/administration/geo/replication/database.html#mysql-replication). This means no supported Disaster Recovery solution if using MySQL. **[PREMIUM ONLY]** -1. [Zero downtime migrations][../update/README.md#upgrading-without-downtime] do not work with MySQL. +1. [Zero downtime migrations](../update/README.md#upgrading-without-downtime) do not work with MySQL. 1. GitLab [optimizes the loading of dashboard events](https://gitlab.com/gitlab-org/gitlab-ce/issues/31806) using [PostgreSQL LATERAL JOINs](https://blog.heapanalytics.com/postgresqls-powerful-new-join-type-lateral/). 1. In general, SQL optimized for PostgreSQL may run much slower in MySQL due to differences in query planners. For example, subqueries that work well in PostgreSQL @@ -197,7 +197,13 @@ use the CI features. ## Supported web browsers -We support the current and the previous major release of Firefox, Chrome/Chromium, Safari and Microsoft browsers (Microsoft Edge and Internet Explorer 11). +We support the current and the previous major release of: + +- Firefox +- Chrome/Chromium +- Safari +- Microsoft Edge +- Internet Explorer 11 Each time a new browser version is released, we begin supporting that version and stop supporting the third most recent version. diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 3647f600b21..6bb2e236dc1 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -657,6 +657,8 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | `REVIEW_DISABLED` | From GitLab 11.0, this variable can be used to disable the `review` and the manual `review:stop` job. If the variable is present, these jobs will not be created. | | `DAST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dast` job. If the variable is present, the job will not be created. | | `PERFORMANCE_DISABLED` | From GitLab 11.0, this variable can be used to disable the `performance` job. If the variable is present, the job will not be created. | +| `OLD_REPORTS_DISABLED` | From GitLab 11.5, this variable can be used to disable the `sast` job. If the variable is present, the job will not be created. | +| `NEW_REPORTS_DISABLED` | From GitLab 11.5, this variable can be used to disable the `sast_dashboard` job. If the variable is present, the job will not be created. | TIP: **Tip:** Set up the replica variables using a diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index 8555c5e91ea..d1729d70158 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -78,5 +78,20 @@ git push In case you're running an older version of Git (< 2.9), consider upgrading to >= 2.9 (see [Broken pipe when pushing to Git repository][Broken-Pipe]). +## Timeout during git push/pull + +If pulling/pushing from/to your repository ends up taking more than 50 seconds, +a timeout will be issued with a log of the number of operations performed +and their respective timings, like the example below: + +``` +remote: Running checks for branch: master +remote: Scanning for LFS objects... (153ms) +remote: Calculating new repository size... (cancelled after 729ms) +``` + +This could be used to further investigate what operation is performing poorly +and provide GitLab with more information on how to improve the service. + [SSH troubleshooting]: ../../ssh/README.md#troubleshooting "SSH Troubleshooting" [Broken-Pipe]: https://stackoverflow.com/questions/19120120/broken-pipe-when-pushing-to-git-repository/36971469#36971469 "StackOverflow: 'Broken pipe when pushing to Git repository'" diff --git a/doc/university/training/topics/tags.md b/doc/university/training/topics/tags.md index 9526bcbfb82..14c39457838 100644 --- a/doc/university/training/topics/tags.md +++ b/doc/university/training/topics/tags.md @@ -22,7 +22,7 @@ comments: false **Additional resources** -<http://git-scm.com/book/en/Git-Basics-Tagging> +<https://git-scm.com/book/en/Git-Basics-Tagging> ---------- diff --git a/doc/user/project/clusters/eks_and_gitlab/img/new_project.png b/doc/user/project/clusters/eks_and_gitlab/img/new_project.png Binary files differdeleted file mode 100644 index 02afc099f10..00000000000 --- a/doc/user/project/clusters/eks_and_gitlab/img/new_project.png +++ /dev/null diff --git a/doc/user/project/clusters/eks_and_gitlab/img/rbac.png b/doc/user/project/clusters/eks_and_gitlab/img/rbac.png Binary files differindex c8adaad13c2..517e4f7ca44 100644 --- a/doc/user/project/clusters/eks_and_gitlab/img/rbac.png +++ b/doc/user/project/clusters/eks_and_gitlab/img/rbac.png diff --git a/doc/user/project/clusters/eks_and_gitlab/index.md b/doc/user/project/clusters/eks_and_gitlab/index.md index 45d77e075f1..fa2ed21f980 100644 --- a/doc/user/project/clusters/eks_and_gitlab/index.md +++ b/doc/user/project/clusters/eks_and_gitlab/index.md @@ -1,123 +1,139 @@ # Connecting and deploying to an Amazon EKS cluster -## Introduction +In this tutorial, we will show how to integrate an +[Amazon EKS](https://aws.amazon.com/eks/) cluster with GitLab and begin +deploying applications. -In this tutorial, we will show how to integrate an [Amazon EKS](https://aws.amazon.com/eks/) cluster with GitLab, and begin deploying applications. +## Introduction For an end-to-end walkthrough we will: -1. Start with a new project based on the sample Ruby on Rails template -1. Integrate an EKS cluster -1. Utilize [Auto DevOps](../../../../topics/autodevops/) to build, test, and deploy our application +1. Start with a new project based on the sample Ruby on Rails template. +1. Integrate an EKS cluster. +1. Utilize [Auto DevOps](../../../../topics/autodevops/) to build, test, and deploy our application. You will need: -1. An account on GitLab, like [GitLab.com](https://gitlab.com) -1. An Amazon EKS cluster (with worker nodes properly configured) -1. `kubectl` [installed and configured for access to the EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#get-started-kubectl) +1. An account on GitLab, like [GitLab.com](https://gitlab.com). +1. An Amazon EKS cluster (with worker nodes properly configured). +1. `kubectl` [installed and configured for access to the EKS cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#get-started-kubectl). -If you don't have an Amazon EKS cluster, one can be created by following [the EKS getting started guide](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html). +If you don't have an Amazon EKS cluster, one can be created by following the +[EKS getting started guide](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html). ## Creating a new project -On GitLab, create a new project by clicking on the `+` icon in the top navigation bar, and selecting `New project`. - - +On GitLab, create a new project by clicking on the `+` icon in the top navigation +bar and selecting **New project**. -On the new project screen, click on the `Create from template` tab, and select `Use template` for the Ruby on Rails sample project. +On the new project screen, click on the **Create from template** tab, and select +"Use template" for the Ruby on Rails sample project. -Give the project a name, and then select `Create project`. +Give the project a name, and then select **Create project**.  ## Configuring and connecting the EKS cluster -From the left side bar, hover over `Operations` and select `Kubernetes`, then click on `Add Kubernetes cluster`, and finally `Add an existing Kubernetes cluster`. +From the left side bar, hover over **Operations > Kubernetes > Add Kubernetes cluster**, +then click **Add an existing Kubernetes cluster**. -A few details from the EKS cluster will be required to connect it to GitLab. +A few details from the EKS cluster will be required to connect it to GitLab: -1. **Retrieve the certificate**: A valid Kubernetes certificate is needed to authenticate to the EKS cluster. We will use the certificate created by default. Open a shell and use `kubectl` to retrieve it: - - List the secrets with `kubectl get secrets`, and one should named similar to `default-token-xxxxx`. Copy that token name for use below. - - Get the certificate with `kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 -D` +1. **Retrieve the certificate**: A valid Kubernetes certificate is needed to + authenticate to the EKS cluster. We will use the certificate created by default. + Open a shell and use `kubectl` to retrieve it: -1. **Create admin token**: A `cluster-admin` token is required to install and manage Helm Tiller. GitLab establishes mutual SSL auth with Helm Tiller and creates limited service accounts for each application. To create the token we will create an admin service account as follows: + - List the secrets with `kubectl get secrets`, and one should named similar to + `default-token-xxxxx`. Copy that token name for use below. + - Get the certificate with: - 1. Create a file called `eks-admin-service-account.yaml` with the text below: + ```sh + kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 -D + ``` - ```yaml - apiVersion: v1 - kind: ServiceAccount - metadata: - name: eks-admin - namespace: kube-system - ``` +1. **Create admin token**: A `cluster-admin` token is required to install and + manage Helm Tiller. GitLab establishes mutual SSL auth with Helm Tiller + and creates limited service accounts for each application. To create the + token we will create an admin service account as follows: - 2. Apply the service account to your cluster: + 2.1. Create a file called `eks-admin-service-account.yaml` with contents: - ```bash - kubectl apply -f eks-admin-service-account.yaml - ``` + ```yaml + apiVersion: v1 + kind: ServiceAccount + metadata: + name: eks-admin + namespace: kube-system + ``` - Output: + 2.2. Apply the service account to your cluster: - ```bash - serviceaccount "eks-admin" created - ``` + ```bash + kubectl apply -f eks-admin-service-account.yaml + ``` - 3. Create a file called `eks-admin-cluster-role-binding.yaml` with the text below: + Output: - ```yaml - apiVersion: rbac.authorization.k8s.io/v1beta1 - kind: ClusterRoleBinding - metadata: - name: eks-admin - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: eks-admin - namespace: kube-system - ``` + ```bash + serviceaccount "eks-admin" created + ``` - 4. Apply the cluster role binding to your cluster: + 2.3. Create a file called `eks-admin-cluster-role-binding.yaml` with contents: - ```bash - kubectl apply -f eks-admin-cluster-role-binding.yaml - ``` + ```yaml + apiVersion: rbac.authorization.k8s.io/v1beta1 + kind: ClusterRoleBinding + metadata: + name: eks-admin + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin + subjects: + - kind: ServiceAccount + name: eks-admin + namespace: kube-system + ``` - Output: + 2.4. Apply the cluster role binding to your cluster: - ```bash - clusterrolebinding "eks-admin" created - ``` + ```bash + kubectl apply -f eks-admin-cluster-role-binding.yaml + ``` - 5. Retrieve the token for the `eks-admin` service account. Copy the `<authentication_token>` value from the output. + Output: - ```bash - kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}') - ``` + ```bash + clusterrolebinding "eks-admin" created + ``` - Output: - - ```yaml - Name: eks-admin-token-b5zv4 - Namespace: kube-system - Labels: <none> - Annotations: kubernetes.io/service-account.name=eks-admin - kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 + 2.5. Retrieve the token for the `eks-admin` service account: - Type: kubernetes.io/service-account-token + ```bash + kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep eks-admin | awk '{print $1}') + ``` - Data - ==== - ca.crt: 1025 bytes - namespace: 11 bytes - token: <authentication_token> - ``` + Copy the `<authentication_token>` value from the output: + + ```yaml + Name: eks-admin-token-b5zv4 + Namespace: kube-system + Labels: <none> + Annotations: kubernetes.io/service-account.name=eks-admin + kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 -1. The API server endpoint is also required, so GitLab can connect to the cluster. This is displayed on the AWS EKS console, when viewing the EKS cluster details. + Type: kubernetes.io/service-account-token + + Data + ==== + ca.crt: 1025 bytes + namespace: 11 bytes + token: <authentication_token> + ``` + +1. The API server endpoint is also required, so GitLab can connect to the cluster. + This is displayed on the AWS EKS console, when viewing the EKS cluster details. You now have all the information needed to connect the EKS cluster: @@ -130,17 +146,26 @@ You now have all the information needed to connect the EKS cluster:  -Click on `Add Kubernetes cluster`, the cluster is now connected to GitLab. At this point, [Kubernetes deployment variables](../#deployment-variables) will automatically be available during CI jobs, making it easy to interact with the cluster. +Click on **Add Kubernetes cluster**, the cluster is now connected to GitLab. +At this point, [Kubernetes deployment variables](../#deployment-variables) will +automatically be available during CI/CD jobs, making it easy to interact with the cluster. If you would like to utilize your own CI/CD scripts to deploy to the cluster, you can stop here. -## Disable Role-Based Access Control (RBAC) - Optional +## Disable Role-Based Access Control (RBAC) (optional) -When connecting a cluster via GitLab integration, you may specify whether the cluster is RBAC-enabled or not. This will affect how GitLab interacts with the cluster for certain operations. If you **did not** check the "RBAC-enabled cluster" checkbox at creation time, GitLab will assume RBAC is disabled for your cluster when interacting with it. If so, you must disable RBAC on your cluster for the integration to work properly. +When connecting a cluster via GitLab integration, you may specify whether the +cluster is RBAC-enabled or not. This will affect how GitLab interacts with the +cluster for certain operations. If you **did not** check the "RBAC-enabled cluster" +checkbox at creation time, GitLab will assume RBAC is disabled for your cluster +when interacting with it. If so, you must disable RBAC on your cluster for the +integration to work properly.  -> **Note**: Disabling RBAC means that any application running in the cluster, or user who can authenticate to the cluster, has full API access. This is a [security concern](https://docs.gitlab.com/ee/user/project/clusters/#security-implications), and may not be desirable. +NOTE: **Note**: Disabling RBAC means that any application running in the cluster, +or user who can authenticate to the cluster, has full API access. This is a +[security concern](../index.md#security-implications), and may not be desirable. To effectively disable RBAC, global permissions can be applied granting full access: @@ -154,56 +179,100 @@ kubectl create clusterrolebinding permissive-binding \ ## Deploy services to the cluster -GitLab supports one-click deployment of helpful services to the cluster, many of which support Auto DevOps. Back on the Kubernetes cluster screen in GitLab, a list of applications is now available to deploy. +GitLab supports one-click deployment of helpful services to the cluster, many of +which support Auto DevOps. Back on the Kubernetes cluster screen in GitLab, a +list of applications is now available to deploy. -First install Helm Tiller, a package manager for Kubernetes. This enables deployment of the other applications. +First, install Helm Tiller, a package manager for Kubernetes. This enables +deployment of the other applications.  ### Deploying NGINX Ingress (optional) -Next, if you would like the deployed app to be reachable on the internet, deploy the Ingress. Note that this will also cause an [Elastic Load Balancer](https://aws.amazon.com/documentation/elastic-load-balancing/) to be created, which will incur additional AWS costs. +Next, if you would like the deployed app to be reachable on the internet, deploy +the Ingress. Note that this will also cause an +[Elastic Load Balancer](https://aws.amazon.com/documentation/elastic-load-balancing/) +to be created, which will incur additional AWS costs. + +Once installed, you may see a `?` for "Ingress IP Address". This is because the +created ELB is available at a DNS name, not an IP address. To get the DNS name, +run: -Once installed, you may see a `?` for `Ingress IP Address`. This is because the created ELB is available at a DNS name, not an IP address. To get the DNS name, run: `kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps -o jsonpath="{.status.loadBalancer.ingress[0].hostname}"`. Note, you may see a trailing `%` on some Kubernetes versions, do not include it. +```sh +kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps -o jsonpath="{.status.loadBalancer.ingress[0].hostname}" +``` + +Note that you may see a trailing `%` on some Kubernetes versions, **do not include it**. -The Ingress is now available at this address, and will route incoming requests to the proper service based on the DNS name in the request. To support this, a wildcard DNS CNAME record should be created for the desired domain name. For example `*.myekscluster.com` would point to the Ingress hostname obtained earlier. +The Ingress is now available at this address and will route incoming requests to +the proper service based on the DNS name in the request. To support this, a +wildcard DNS CNAME record should be created for the desired domain name. For example, +`*.myekscluster.com` would point to the Ingress hostname obtained earlier.  ### Deploying the GitLab Runner (optional) -If the project is on GitLab.com, free shared runners are available and you do not have to deploy one. If a project specific runner is desired, or there are no shared runners, it is easy to deploy one. +If the project is on GitLab.com, free shared Runners are available and you do +not have to deploy one. If a project specific Runner is desired, or there are no +shared Runners, it is easy to deploy one. -Simply click on the `Install` button for the GitLab Runner. It is important to note that the runner deployed is set as **privileged**, which means it essentially has root access to the underlying machine. This is required to build docker images, and so is on by default. +Simply click on the **Install** button for the GitLab Runner. It is important to +note that the Runner deployed is set as **privileged**, which means it essentially +has root access to the underlying machine. This is required to build docker images, +and so is on by default. ### Deploying Prometheus (optional) -GitLab is able to monitor applications automatically, utilizing [Prometheus](../../integrations/prometheus.html). Kubernetes container CPU and memory metrics are automatically collected, and response metrics are retrieved from NGINX Ingress as well. +GitLab is able to monitor applications automatically, utilizing +[Prometheus](../../integrations/prometheus.html). Kubernetes container CPU and +memory metrics are automatically collected, and response metrics are retrieved +from NGINX Ingress as well. -To enable monitoring, simply install Prometheus into the cluster with the `Install` button. +To enable monitoring, simply install Prometheus into the cluster with the +**Install** button. ## Create a default Storage Class -Amazon EKS does not have a default Storage Class out of the box, which means requests for persistent volumes will not be automatically fulfilled. As part of Auto DevOps, the deployed Postgres instance requests persistent storage, and without a default storage class it will fail to start. +Amazon EKS doesn't have a default Storage Class out of the box, which means +requests for persistent volumes will not be automatically fulfilled. As part +of Auto DevOps, the deployed Postgres instance requests persistent storage, +and without a default storage class it will fail to start. -If a default Storage Class does not already exist and is desired, follow Amazon's [short guide](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html) to create one. +If a default Storage Class doesn't already exist and is desired, follow Amazon's +[guide on storage classes](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html) +to create one. -Alternatively, disable Postgres by setting the project variable [`POSTGRES_ENABLED`](../../../../topics/autodevops/#environment-variables) to `false`. +Alternatively, disable Postgres by setting the project variable +[`POSTGRES_ENABLED`](../../../../topics/autodevops/#environment-variables) to `false`. ## Deploy the app to EKS -With RBAC disabled and services deployed, [Auto DevOps](https://docs.gitlab.com/ee/topics/autodevops/) can now be leveraged to build, test, and deploy the app. To enable, click on `Settings` in the left sidebar, then `CI/CD`. You will see a section for `Auto DevOps`, expand it. Click on the radio button to `Enable Auto DevOps`. +With RBAC disabled and services deployed, +[Auto DevOps](../../../../topics/autodevops/index.md) can now be leveraged +to build, test, and deploy the app. -If a wildcard DNS entry was created resolving to the Load Balancer, enter it in the `domain` field. Otherwise, the deployed app will not be externally available outside of the cluster. To save, click `Save changes`. +[Enable Auto DevOps](../../../../topics/autodevops/index.md##enablingdisabling-auto-devops-at-the-project-level) +if not already enabled. If a wildcard DNS entry was created resolving to the +Load Balancer, enter it in the `domain` field under the Auto DevOps settings. +Otherwise, the deployed app will not be externally available outside of the cluster.  -A new pipeline will automatically be created, which will begin to build, test, and deploy the app. +A new pipeline will automatically be created, which will begin to build, test, +and deploy the app. -After the pipeline has finished, your app will be running in EKS and available to users. Click on `CI/CD` tab in the left navigation bar, and choose `Environments`. +After the pipeline has finished, your app will be running in EKS and available +to users. Click on **CI/CD > Environments**.  -You will see a list of the environments and their deploy status, as well as options to browse to the app, view monitoring metrics, and even access a shell on the running pod. +You will see a list of the environments and their deploy status, as well as +options to browse to the app, view monitoring metrics, and even access a shell +on the running pod. + +## Learn more -To learn more about Auto DevOps, review our [documentation](../../../../topics/autodevops/). +To learn more on automatically deploying your applications, +read about [Auto DevOps](../../../../topics/autodevops/index.md). diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 3fbd4c21eab..ca262e4b76e 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -49,8 +49,8 @@ new Kubernetes cluster to your project: NOTE: **Note:** You need Maintainer [permissions] and above to access the Kubernetes page. -1. Click on **Add Kubernetes cluster**. -1. Click on **Create with Google Kubernetes Engine**. +1. Click **Add Kubernetes cluster**. +1. Click **Create with Google Kubernetes Engine**. 1. Connect your Google account if you haven't done already by clicking the **Sign in with Google** button. 1. From there on, choose your cluster's settings: @@ -78,8 +78,8 @@ To add an existing Kubernetes cluster to your project: NOTE: **Note:** You need Maintainer [permissions] and above to access the Kubernetes page. -1. Click on **Add Kubernetes cluster**. -1. Click on **Add an existing Kubernetes cluster** and fill in the details: +1. Click **Add Kubernetes cluster**. +1. Click **Add an existing Kubernetes cluster** and fill in the details: - **Kubernetes cluster name** (required) - The name you wish to give the cluster. - **Environment scope** (required)- The [associated environment](#setting-the-environment-scope) to this cluster. @@ -228,7 +228,11 @@ twice, which can lead to confusion during deployments. | [Prometheus](https://prometheus.io/docs/introduction/overview/) | 10.4+ | Prometheus is an open-source monitoring and alerting system useful to supervise your deployed applications. | [stable/prometheus](https://github.com/helm/charts/tree/master/stable/prometheus) | | [GitLab Runner](https://docs.gitlab.com/runner/) | 10.6+ | GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/), the open-source continuous integration service included with GitLab that coordinates the jobs. When installing the GitLab Runner via the applications, it will run in **privileged mode** by default. Make sure you read the [security implications](#security-implications) before doing so. | [runner/gitlab-runner](https://gitlab.com/charts/gitlab-runner) | | [JupyterHub](http://jupyter.org/) | 11.0+ | [JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) is a multi-user service for managing notebooks across a team. [Jupyter Notebooks](https://jupyter-notebook.readthedocs.io/en/latest/) provide a web-based interactive programming environment used for data analysis, visualization, and machine learning. We use [this](https://gitlab.com/gitlab-org/jupyterhub-user-image/blob/master/Dockerfile) custom Jupyter image that installs additional useful packages on top of the base Jupyter. You will also see ready-to-use DevOps Runbooks built with Nurtch's [Rubix library](https://github.com/amit1rrr/rubix). More information on creating executable runbooks can be found at [Nurtch Documentation](http://docs.nurtch.com/en/latest). **Note**: Authentication will be enabled for any user of the GitLab server via OAuth2. HTTPS will be supported in a future release. | [jupyter/jupyterhub](https://jupyterhub.github.io/helm-chart/) | -| [Knative](https://cloud.google.com/knative) | 0.1.2 | Knative provides a platform to create, deploy, and manage serverless workloads from a Kubernetes cluster. It is used in conjunction with, and includes [Istio](https://istio.io) to provide an external IP address for all programs hosted by Knative. You will be prompted to enter a wildcard domain where your applications will be exposed. Configure your DNS server to use the external IP address for that domain. For any application created and installed, they will be accessible as <program_name>.<kubernetes_namespace>.<domain_name>. **Note**: This will require your kubernetes cluster to have RBAC enabled. | [knative/knative](https://storage.googleapis.com/triggermesh-charts) +| [Knative](https://cloud.google.com/knative) | 11.5+ | Knative provides a platform to create, deploy, and manage serverless workloads from a Kubernetes cluster. It is used in conjunction with, and includes [Istio](https://istio.io) to provide an external IP address for all programs hosted by Knative. You will be prompted to enter a wildcard domain where your applications will be exposed. Configure your DNS server to use the external IP address for that domain. For any application created and installed, they will be accessible as `<program_name>.<kubernetes_namespace>.<domain_name>`. This will require your kubernetes cluster to have [RBAC enabled](#role-based-access-control-rbac). | [knative/knative](https://storage.googleapis.com/triggermesh-charts) + +NOTE: **Note:** +As of GitLab 11.6 Helm Tiller will be upgraded to the latest version supported +by GitLab before installing any of the above applications. ## Getting the external IP address @@ -255,14 +259,17 @@ your ingress application in which case you should manually determine it. ### Manually determining the IP address -If the cluster is on GKE, click on the **Google Kubernetes Engine** link in the +If the cluster is on GKE, click the **Google Kubernetes Engine** link in the **Advanced settings**, or go directly to the [Google Kubernetes Engine dashboard](https://console.cloud.google.com/kubernetes/) -and select the proper project and cluster. Then click on **Connect** and execute +and select the proper project and cluster. Then click **Connect** and execute the `gcloud` command in a local terminal or using the **Cloud Shell**. If the cluster is not on GKE, follow the specific instructions for your Kubernetes provider to configure `kubectl` with the right credentials. +The output of the following examples will show the external IP address of your +cluster. This information can then be used to set up DNS entries and forwarding +rules that allow external access to your deployed applications. If you installed the Ingress [via the **Applications**](#installing-applications), run the following command: @@ -271,26 +278,23 @@ run the following command: kubectl get svc --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' ``` -NOTE: **Note:** For Istio/Knative, the command will be different: + ```bash kubectl get svc --namespace=istio-system knative-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' ``` -Otherwise, you can list the IP addresses of all load balancers: +Some Kubernetes clusters return a hostname instead, like [Amazon EKS](https://aws.amazon.com/eks/). For these platforms, run: ```bash -kubectl get svc --all-namespaces -o jsonpath='{range.items[?(@.status.loadBalancer.ingress)]}{.status.loadBalancer.ingress[*].ip} ' +kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps -o jsonpath="{.status.loadBalancer.ingress[0].hostname}". ``` -> **Note**: Some Kubernetes clusters return a hostname instead, like [Amazon EKS](https://aws.amazon.com/eks/). For these platforms, run: -> ```bash -> kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps -o jsonpath="{.status.loadBalancer.ingress[0].hostname}". -> ``` +Otherwise, you can list the IP addresses of all load balancers: -The output is the external IP address of your cluster. This information can then -be used to set up DNS entries and forwarding rules that allow external access to -your deployed applications. +```bash +kubectl get svc --all-namespaces -o jsonpath='{range.items[?(@.status.loadBalancer.ingress)]}{.status.loadBalancer.ingress[*].ip} ' +``` ### Using a static IP @@ -300,7 +304,7 @@ your apps will not be able to be reached, and you'd have to change the DNS record again. In order to avoid that, you should change it into a static reserved IP. -[Read how to promote an ephemeral external IP address in GKE.](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip) +Read how to [promote an ephemeral external IP address in GKE](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip). ### Pointing your DNS at the cluster IP @@ -406,7 +410,7 @@ service account of the cluster integration. After you have successfully added your cluster information, you can enable the Kubernetes cluster integration: -1. Click the "Enabled/Disabled" switch +1. Click the **Enabled/Disabled** switch 1. Hit **Save** for the changes to take effect You can now start using your Kubernetes cluster for your deployments. @@ -423,7 +427,7 @@ When you remove a cluster, you only remove its relation to GitLab, not the cluster itself. To remove the cluster, you can do so by visiting the GKE dashboard or using `kubectl`. -To remove the Kubernetes cluster integration from your project, simply click on the +To remove the Kubernetes cluster integration from your project, simply click the **Remove integration** button. You will then be able to follow the procedure and add a Kubernetes cluster again. @@ -486,7 +490,13 @@ the deployment variables above, ensuring any pods you create are labelled with ## Read more -- [Connecting and deploying to an Amazon EKS cluster](eks_and_gitlab/index.md) +### Integrating Amazon EKS cluster with GitLab + +- Learn how to [connect and deploy to an Amazon EKS cluster](eks_and_gitlab/index.md). + +### Serverless + +- [Run serverless workloads on Kubernetes with Knative.](serverless/index.md) [permissions]: ../../permissions.md [ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/clusters/serverless/img/deploy-stage.png b/doc/user/project/clusters/serverless/img/deploy-stage.png Binary files differnew file mode 100644 index 00000000000..dc2f8af9c63 --- /dev/null +++ b/doc/user/project/clusters/serverless/img/deploy-stage.png diff --git a/doc/user/project/clusters/serverless/img/dns-entry.png b/doc/user/project/clusters/serverless/img/dns-entry.png Binary files differnew file mode 100644 index 00000000000..2e7655c6041 --- /dev/null +++ b/doc/user/project/clusters/serverless/img/dns-entry.png diff --git a/doc/user/project/clusters/serverless/img/install-knative.png b/doc/user/project/clusters/serverless/img/install-knative.png Binary files differnew file mode 100644 index 00000000000..dd576a9df35 --- /dev/null +++ b/doc/user/project/clusters/serverless/img/install-knative.png diff --git a/doc/user/project/clusters/serverless/img/knative-app.png b/doc/user/project/clusters/serverless/img/knative-app.png Binary files differnew file mode 100644 index 00000000000..54301e1786f --- /dev/null +++ b/doc/user/project/clusters/serverless/img/knative-app.png diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md new file mode 100644 index 00000000000..bdbc4f7f09d --- /dev/null +++ b/doc/user/project/clusters/serverless/index.md @@ -0,0 +1,137 @@ +# Serverless + +> Introduced in GitLab 11.5. + +Run serverless workloads on Kubernetes using [Knative](https://cloud.google.com/knative/). + +## Overview + +Knative extends Kubernetes to provide a set of middleware components that are useful to build modern, source-centric, container-based applications. Knative brings some significant benefits out of the box through its main components: + +- [Build:](https://github.com/knative/build) Source-to-container build orchestration +- [Eventing:](https://github.com/knative/eventing) Management and delivery of events +- [Serving:](https://github.com/knative/serving) Request-driven compute that can scale to zero + +For more information on Knative, visit the [Knative docs repo](https://github.com/knative/docs). + +## Requirements + +To run Knative on Gitlab, you will need: + +1. **Kubernetes:** An RBAC-enabled Kubernetes cluster is required to deploy Knative. + The simplest way to get started is to add a cluster using [GitLab's GKE integration](https://docs.gitlab.com/ee/user/project/clusters/#adding-and-creating-a-new-gke-cluster-via-gitlab). + GitLab recommends +1. **Helm Tiller:** Helm is a package manager for Kubernetes and is required to install + all the other applications. +1. **Domain Name:** Knative will provide its own load balancer using Istio. It will provide an + external IP address for all the applications served by Knative. You will be prompted to enter a + wildcard domain where your applications will be served. Configure your DNS server to use the + external IP address for that domain. +1. **Serverless `gitlab-ci.yml` Template:** GitLab uses [Kaniko](https://github.com/GoogleContainerTools/kaniko) + to build the application and the [TriggerMesh CLI](https://github.com/triggermesh/tm), to simplify the + deployment of knative services and functions. + + Add the following `.gitlab-ci.yml` to the root of your repository (you may skip this step if using the sample + [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) mentioned below). + + ```yaml + stages: + - build + - deploy + + build: + stage: build + image: + name: gcr.io/kaniko-project/executor:debug + entrypoint: [""] + only: + - master + script: + - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json + - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE + + deploy: + stage: deploy + image: gcr.io/triggermesh/tm@sha256:e3ee74db94d215bd297738d93577481f3e4db38013326c90d57f873df7ab41d5 + only: + - master + environment: production + script: + - echo "$CI_REGISTRY_IMAGE" + - tm -n "$KUBE_NAMESPACE" --config "$KUBECONFIG" deploy service "$CI_PROJECT_NAME" --from-image "$CI_REGISTRY_IMAGE" --wait + ``` + +1. **Dockerfile:** Knative requires a Dockerfile in order to build your application. It should be included + at the root of your project's repo and expose port 8080. + +## Installing Knative via GitLab's Kubernetes integration + +NOTE: **Note:** +Minimum recommended cluster size to run Knative is 3-nodes, 6 vCPUs, and 22.50 GB memory. RBAC must be enabled. + +You may download the sample [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) to get started. + +1. [Add a Kubernetes cluster](https://docs.gitlab.com/ce/user/project/clusters/) and install Helm. + +1. Once Helm has been successfully installed, on the Knative app section, enter the domain to be used with + your application and click "Install". + +  + +1. After the Knative installation has finished, retrieve the Istio Ingress IP address by running the following command: + + ```bash + kubectl get svc --namespace=istio-system knative-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' + ``` + + Output: + + ```bash + 35.161.143.124 my-machine-name:~ my-user$ + ``` + +1. The ingress is now available at this address and will route incoming requests to the proper service based on the DNS + name in the request. To support this, a wildcard DNS A record should be created for the desired domain name. For example, + if your Knative base domain is `knative.example.com` then you need to create an A record with domain `*.knative.example.com` + pointing the ip address of the ingress. + +  + +## Deploy the application with Knative + +With all the pieces in place, you can simply create a new CI pipeline to deploy the Knative application. Navigate to +**CI/CD >> Pipelines** and click the **Run Pipeline** button at the upper-right part of the screen. Then, on the +Pipelines page, click **Create pipeline**. + +## Obtain the URL for the Knative deployment + +Once all the stages of the pipeline finish, click the **deploy** stage. + + + +The output will look like this: + +```bash +Running with gitlab-runner 11.5.0~beta.844.g96d88322 (96d88322) + on docker-auto-scale 72989761 +Using Docker executor with image gcr.io/triggermesh/tm@sha256:e3ee74db94d215bd297738d93577481f3e4db38013326c90d57f873df7ab41d5 ... +Pulling docker image gcr.io/triggermesh/tm@sha256:e3ee74db94d215bd297738d93577481f3e4db38013326c90d57f873df7ab41d5 ... +Using docker image sha256:6b3f6590a9b30bd7aafb9573f047d930c70066e43955b4beb18a1eee175f6de1 for gcr.io/triggermesh/tm@sha256:e3ee74db94d215bd297738d93577481f3e4db38013326c90d57f873df7ab41d5 ... +Running on runner-72989761-project-4342902-concurrent-0 via runner-72989761-stg-srm-1541795796-27929c96... +Cloning repository... +Cloning into '/builds/danielgruesso/knative'... +Checking out 8671ad20 as master... +Skipping Git submodules setup +$ echo "$CI_REGISTRY_IMAGE" +registry.staging.gitlab.com/danielgruesso/knative +$ tm -n "$KUBE_NAMESPACE" --config "$KUBECONFIG" deploy service "$CI_PROJECT_NAME" --from-image "$CI_REGISTRY_IMAGE" --wait +Deployment started. Run "tm -n knative-4342902 describe service knative" to see the details +Waiting for ready state....... +Service domain: knative.knative-4342902.knative.info +Job succeeded +``` + +The second to last line, labeled **Service domain** contains the URL for the deployment. Copy and paste the domain into your +browser to see the app live. + +
\ No newline at end of file diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 3e4be043199..42da2210fab 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -17,6 +17,7 @@ the [GitHub rake task](../../../administration/raketasks/github_import.md) to im GitHub without the constraints of a Sidekiq worker. The following aspects of a project are imported: + * Repository description (GitLab.com & 7.7+) * Git repository data (GitLab.com & 7.7+) * Issues (GitLab.com & 7.7+) diff --git a/doc/user/project/merge_requests/img/comment-on-any-diff-line.png b/doc/user/project/merge_requests/img/comment-on-any-diff-line.png Binary files differnew file mode 100644 index 00000000000..856ede41527 --- /dev/null +++ b/doc/user/project/merge_requests/img/comment-on-any-diff-line.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 6de2ab07fc4..8a68185798b 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -141,6 +141,15 @@ you hide discussions that are no longer relevant. [Read more about resolving discussion comments in merge requests reviews.](../../discussions/index.md) +## Commenting on any file line in merge requests + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/13950) in GitLab 11.5. + +GitLab provides a way of leaving comments in any part of the file being changed +in a Merge Request. To do so, click the **...** button in the gutter of the Merge Request diff UI to expand the diff lines and leave a comment, just as you would for a changed line. + + + ## Resolve conflicts When a merge request has conflicts, GitLab may provide the option to resolve @@ -168,11 +177,11 @@ administrator to do so. ### Adding patches when creating a merge request via e-mail -> **Note**: This feature was [implemented in GitLab 11.5](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22723) +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22723) in GitLab 11.5. You can add commits to the merge request being created by adding -patches as attachments to the email, all attachments with a filename -ending in `.patch` will be considered patches. The patches will be processed +patches as attachments to the email. All attachments with a filename +ending in `.patch` will be considered patches and they will be processed ordered by name. The combined size of the patches can be 2MB. @@ -185,7 +194,7 @@ branch already exists, the patches will be applied on top of it. ## Find the merge request that introduced a change -> **Note**: this feature was [implemented in GitLab 10.5](https://gitlab.com/gitlab-org/gitlab-ce/issues/2383). +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2383) in GitLab 10.5. When viewing the commit details page, GitLab will link to the merge request (or merge requests, if it's in more than one) containing that commit. diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index 3cf46231a9d..7168fe63887 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -6,6 +6,26 @@ Milestones in GitLab are a way to track issues and merge requests created to ach Milestones allow you to organize issues and merge requests into a cohesive group, with an optional start date and an optional due date. +## Milestones as Agile sprints + +Milestones can be used as Agile sprints. +Set the milestone start date and due date to represent +the start and end of your Agile sprint. +Set the milestone title to the name of your Agile sprint, +such as `November 2018 sprint`. +Add an issue to your Agile sprint by associating +the milestone to the issue. + +## Milestones as releases + +Milestones can be used as releases. +Set the milestone due date to represent the release date of your release. +(And leave the milestone start date blank.) +Set the the milestone title to the version of your release, +such as `Version 9.4`. +Add an issue to your release by associating +the milestone to the issue. + ## Project milestones and group milestones - **Project milestones** can be assigned to issues or merge requests in that project only. diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 6d822d3f7f2..1710bba2fd0 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -53,17 +53,35 @@ To get started with the command line, please read through the Use GitLab's [file finder](../../../workflow/file_finder.md) to search for files in a repository. +### Supported markup languages and extensions + +GitLab supports a number of markup languages (sometimes called [lightweight +markup languages](https://en.wikipedia.org/wiki/Lightweight_markup_language)) +that you can use for the content of your files in a repository. They are mostly +used for documentation purposes. + +Just pick the right extension for your files and GitLab will render them +according to the markup language. + +| Markup language | Extensions | +| --------------- | ---------- | +| Plain text | `txt` | +| [Markdown](../../markdown.md) | `mdown`, `mkd`, `mkdn`, `md`, `markdown` | +| [reStructuredText](http://docutils.sourceforge.net/rst.html) | `rst` | +| [Asciidoc](https://asciidoctor.org/docs/what-is-asciidoc/) | `adoc`, `ad`, `asciidoc` | +| [Textile](https://txstyle.org/) | `textile` | +| [rdoc](http://rdoc.sourceforge.net/doc/index.html) | `rdoc` | +| [Orgmode](https://orgmode.org/) | `org` | +| [creole](http://www.wikicreole.org/) | `creole` | +| [Mediawiki](https://www.mediawiki.org/wiki/MediaWiki) | `wiki`, `mediawiki` | + ### Repository README and index files When a `README` or `index` file is present in a repository, its contents will be automatically pre-rendered by GitLab without opening it. -They can either be plain text or have an extension of a supported markup language: - -- Asciidoc: `README.adoc` or `index.adoc` -- Markdown: `README.md` or `index.md` -- reStructuredText: `README.rst` or `index.rst` -- Text: `README.txt` or `index.txt` +They can either be plain text or have an extension of a +[supported markup language](#supported-markup-languages-and-extensions): Some things to note about precedence: @@ -75,10 +93,6 @@ Some things to note about precedence: precedence over `README.md`, and `README.rst` will take precedence over `README`. -NOTE: **Note:** -`index` files without an extension will not automatically pre-render. You'll -have to explicitly open them to see their contents. - ### Jupyter Notebook files > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2508) in GitLab 9.1 diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 084d1161633..d6754372816 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -18,6 +18,8 @@ Adjust your project's name, description, avatar, [default branch](../repository/  +The project description also partially supports [standard markdown](../../markdown.md#standard-markdown). You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#links), and [line-breaks](../../markdown.md#line-breaks) to add more context to the project description. + ### Sharing and permissions Set up your project's access, [visibility](../../../public_access/public_access.md), and enable [Container Registry](../container_registry.md) for your projects: diff --git a/doc/user/search/img/dashboard_links.png b/doc/user/search/img/dashboard_links.png Binary files differnew file mode 100644 index 00000000000..2c472c7e464 --- /dev/null +++ b/doc/user/search/img/dashboard_links.png diff --git a/doc/user/search/img/issues_assigned_to_you.png b/doc/user/search/img/issues_assigned_to_you.png Binary files differindex 36c670eedd5..d2fff5e9a67 100644 --- a/doc/user/search/img/issues_assigned_to_you.png +++ b/doc/user/search/img/issues_assigned_to_you.png diff --git a/doc/user/search/img/left_menu_bar.png b/doc/user/search/img/left_menu_bar.png Binary files differdeleted file mode 100644 index d68a71cba8e..00000000000 --- a/doc/user/search/img/left_menu_bar.png +++ /dev/null diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 3f9d07dacaa..78c1294346b 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -2,27 +2,27 @@ ## Issues and merge requests -To search through issues and merge requests in multiple projects, you can use the left-sidebar. +To search through issues and merge requests in multiple projects, you can use the **Issues** or **Merge Requests** links +in the top-right part of your screen. -Click the menu bar, then **Issues** or **Merge Requests**, which work in the same way, -therefore, the following notes are valid for both. +Both of them work in the same way, therefore, the following notes are valid for both. The number displayed on their right represents the number of issues and merge requests assigned to you. - + When you click **Issues**, you'll see the opened issues assigned to you straight away:  -You can filter them by **Author**, **Assignee**, **Milestone**, and **Labels**, -searching through **Open**, **Closed**, and **All** issues. +You can search through **Open**, **Closed**, or **All** issues. -Of course, you can combine all filters together. +You can also filter the results using the search and filter field. This works in the same way as the ones found in the +per project pages described below. ### Issues and MRs assigned to you or created by you -You'll find a shortcut to issues and merge requests create by you or assigned to you +You'll also find shortcuts to issues and merge requests created by you or assigned to you on the search field on the top-right of your screen:  diff --git a/doc/workflow/shortcuts.md b/doc/workflow/shortcuts.md index b2f1cbec204..7863dd8c242 100644 --- a/doc/workflow/shortcuts.md +++ b/doc/workflow/shortcuts.md @@ -93,4 +93,4 @@ You can see GitLab's keyboard shortcuts by using 'shift + ?' | Keyboard Shortcut | Description | | ----------------- | ----------- | -| <kbd>⌘</kbd> + <kbd>p</kbd> | Go to file | +| <kbd>Cmd</kbd>/<kbd>Ctrl</kbd> + <kbd>p</kbd> | Go to file | |
