diff options
Diffstat (limited to 'doc')
57 files changed, 1194 insertions, 408 deletions
diff --git a/doc/README.md b/doc/README.md index ee69684b53b..909740211a6 100644 --- a/doc/README.md +++ b/doc/README.md @@ -1,8 +1,12 @@ # GitLab Community Edition documentation +## University + +[University](university/README.md) contain guides to learn Git and GitLab through courses and videos. + ## User documentation -- [Account Security](user/account/security.md) Securing your account via two-factor authentication, etc. +- [Account Security](user/profile/account/two_factor_authentication.md) Securing your account via two-factor authentication, etc. - [API](api/README.md) Automate GitLab via a simple and powerful API. - [CI/CD](ci/README.md) GitLab Continuous Integration (CI) and Continuous Delivery (CD) getting started, `.gitlab-ci.yml` options, and examples. - [GitLab as OAuth2 authentication service provider](integration/oauth_provider.md). It allows you to login to other applications from GitLab. @@ -19,7 +23,6 @@ - [SSH](ssh/README.md) Setup your ssh keys and deploy keys for secure access to your projects. - [Webhooks](web_hooks/web_hooks.md) Let GitLab notify you when new code has been pushed to your project. - [Workflow](workflow/README.md) Using GitLab functionality and importing projects from GitHub and SVN. -- [University](university/README.md) Learn Git and GitLab through videos and courses. - [Git Attributes](user/project/git_attributes.md) Managing Git attributes using a `.gitattributes` file. - [Git cheatsheet](https://gitlab.com/gitlab-com/marketing/raw/master/design/print/git-cheatsheet/print-pdf/git-cheatsheet.pdf) Download a PDF describing the most used Git operations. @@ -41,7 +44,7 @@ - [Operations](administration/operations.md) Keeping GitLab up and running. - [Raketasks](raketasks/README.md) Backups, maintenance, automatic webhook setup and the importing of projects. - [Repository checks](administration/repository_checks.md) Periodic Git repository checks. -- [Repository storages](administration/repository_storages.md) Manage the paths used to store repositories. +- [Repository storage paths](administration/repository_storage_paths.md) Manage the paths used to store repositories. - [Security](security/README.md) Learn what you can do to further secure your GitLab instance. - [System hooks](system_hooks/system_hooks.md) Notifications when users, projects and keys are changed. - [Update](update/README.md) Update guides to upgrade your installation. @@ -50,14 +53,14 @@ - [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. - [Git LFS configuration](workflow/lfs/lfs_administration.md) - [Housekeeping](administration/housekeeping.md) Keep your Git repository tidy and fast. -- [GitLab Performance Monitoring](administration/monitoring/performance/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics. +- [GitLab performance monitoring with InfluxDB](administration/monitoring/performance/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics. +- [GitLab performance monitoring with Prometheus](administration/monitoring/performance/prometheus.md) Configure GitLab and Prometheus for measuring performance metrics. - [Request Profiling](administration/monitoring/performance/request_profiling.md) Get a detailed profile on slow requests. - [Monitoring uptime](user/admin_area/monitoring/health_check.md) Check the server status using the health check endpoint. - [Debugging Tips](administration/troubleshooting/debug.md) Tips to debug problems when things go wrong - [Sidekiq Troubleshooting](administration/troubleshooting/sidekiq.md) Debug when Sidekiq appears hung and is not processing jobs. - [High Availability](administration/high_availability/README.md) Configure multiple servers for scaling or high availability. - [Container Registry](administration/container_registry.md) Configure Docker Registry with GitLab. -- [Multiple mountpoints for the repositories storage](administration/repository_storages.md) Define multiple repository storage paths to distribute the storage load. ## Contributor documentation diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 04723365277..f6027b2f99e 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -65,11 +65,15 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # # Example: 'Paris' or 'Acme, Ltd.' label: 'LDAP' - + + # Example: 'ldap.mydomain.com' host: '_your_ldap_server' + # This port is an example, it is sometimes different but it is always an integer and not a string port: 389 uid: 'sAMAccountName' method: 'plain' # "tls" or "ssl" or "plain" + + # Examples: 'america\\momo' or 'CN=Gitlab Git,CN=Users,DC=mydomain,DC=com' bind_dn: '_the_full_dn_of_the_user_you_will_bind_with' password: '_the_password_of_the_bind_user' @@ -101,7 +105,7 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # Base where we can search for users # - # Ex. ou=People,dc=gitlab,dc=example + # Ex. 'ou=People,dc=gitlab,dc=example' or 'DC=mydomain,DC=com' # base: '' @@ -112,6 +116,9 @@ main: # 'main' is the GitLab 'provider ID' of this LDAP server # # Note: GitLab does not support omniauth-ldap's custom filter syntax. # + # Below an example for get only specific users + # Example: '(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))' + # user_filter: '' # LDAP attributes that GitLab will use to create an account for the LDAP user. diff --git a/doc/administration/monitoring/performance/introduction.md b/doc/administration/monitoring/performance/introduction.md index 79904916b7e..8b106e89cc2 100644 --- a/doc/administration/monitoring/performance/introduction.md +++ b/doc/administration/monitoring/performance/introduction.md @@ -12,6 +12,11 @@ documents in order to understand and properly configure GitLab Performance Monit - [InfluxDB Schema](influxdb_schema.md) - [Grafana Install/Configuration](grafana_configuration.md) +>**Note:** +Omnibus GitLab 8.16 includes Prometheus as an additional tool to collect +metrics. It will eventually replace InfluxDB when their metrics collection is +on par. Read more in the [Prometheus documentation](prometheus.md). + ## Introduction to GitLab Performance Monitoring GitLab Performance Monitoring makes it possible to measure a wide variety of statistics diff --git a/doc/administration/monitoring/performance/prometheus.md b/doc/administration/monitoring/performance/prometheus.md new file mode 100644 index 00000000000..51c63325064 --- /dev/null +++ b/doc/administration/monitoring/performance/prometheus.md @@ -0,0 +1,102 @@ +# GitLab Prometheus + +>**Notes:** +- Prometheus and the node exporter are bundled in the Omnibus GitLab package + since GitLab 8.16. For installations from source you will have to install + them yourself. Over subsequent releases additional GitLab metrics will be + captured. +- Prometheus services are off by default but will be on starting with GitLab 9.0. + +[Prometheus] is a powerful time-series monitoring service, providing a flexible +platform for monitoring GitLab and other software products. +GitLab provides out of the box monitoring with Prometheus, providing easy +access to high quality time-series monitoring of GitLab services. + +## Overview + +Prometheus works by periodically connecting to data sources and collecting their +performance metrics. To view and work with the monitoring data, you can either +connect directly to Prometheus or utilize a dashboard tool like [Grafana]. + +## Configuring Prometheus + +>**Note:** +Available since Omnibus GitLab 8.16. For installations from source you'll +have to install and configure it yourself. + +To enable Prometheus: + +1. Edit `/etc/gitlab/gitlab.rb` +1. Find and uncomment the following line, making sure it's set to `true`: + + ```ruby + prometheus['enable'] = true + ``` + +1. Save the file and [reconfigure GitLab][reconfigure] for the changes to + take effect + +By default, Prometheus will run as the `gitlab-prometheus` user and listen on +TCP port `9090` under localhost. If the [node exporter](#node-exporter) service +has been enabled, it will automatically be set up as a monitoring target for +Prometheus. + +## Viewing Performance Metrics + +After you have [enabled Prometheus](#configuring-prometheus), you can visit +`<your_domain_name>:9090` for the dashboard that Prometheus offers by default. + +The performance data collected by Prometheus can be viewed directly in the +Prometheus console or through a compatible dashboard tool. +The Prometheus interface provides a [flexible query language][prom-query] to work +with the collected data where you can visualize their output. +For a more fully featured dashboard, Grafana can be used and has +[official support for Prometheus][prom-grafana]. + +## Prometheus exporters + +There are a number of libraries and servers which help in exporting existing +metrics from third-party systems as Prometheus metrics. This is useful for cases +where it is not feasible to instrument a given system with Prometheus metrics +directly (for example, HAProxy or Linux system stats). You can read more in the +[Prometheus exporters and integrations documentation][prom-exporters]. + +While you can use any exporter you like with your GitLab installation, the +following ones documented here are bundled in the Omnibus GitLab packages +making it easy to configure and use. + +### Node exporter + +>**Note:** +Available since Omnibus GitLab 8.16. For installations from source you'll +have to install and configure it yourself. + +The [node exporter] allows you to measure various machine resources such as +memory, disk and CPU utilization. + +To enable the node exporter: + +1. [Enable Prometheus](#configuring-prometheus) +1. Edit `/etc/gitlab/gitlab.rb` +1. Find and uncomment the following line, making sure it's set to `true`: + + ```ruby + node_exporter['enable'] = true + ``` + +1. Save the file and [reconfigure GitLab][reconfigure] for the changes to + take effect + +Prometheus it will now automatically begin collecting performance data from +the node exporter. You can visit `<your_domain_name>:9100/metrics` for a real +time representation of the metrics that are collected. Refresh the page and +you will see the data change. + +[grafana]: https://grafana.net +[node exporter]: https://github.com/prometheus/node_exporter +[prometheus]: https://prometheus.io +[prom-query]: https://prometheus.io/docs/querying/basics +[prom-grafana]: https://prometheus.io/docs/visualization/grafana/ +[scrape-config]: https://prometheus.io/docs/operating/configuration/#%3Cscrape_config%3E +[prom-exporters]: https://prometheus.io/docs/instrumenting/exporters/ +[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index f3c2e72341f..33b9b28433a 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -27,6 +27,7 @@ Ruby Version: 2.1.5p273 Gem Version: 2.4.3 Bundler Version: 1.7.6 Rake Version: 10.3.2 +Redis Version: 3.2.5 Sidekiq Version: 2.17.8 GitLab information diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index bc2b1f20ed3..ee37ea49874 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -13,12 +13,12 @@ checks failed you can see their output on the admin log page under ## Periodic checks -GitLab periodically runs a repository check on all project repositories and -wiki repositories in order to detect data corruption problems. A -project will be checked no more than once per week. If any projects +When enabled, GitLab periodically runs a repository check on all project +repositories and wiki repositories in order to detect data corruption problems. +A project will be checked no more than once per month. If any projects fail their repository checks all GitLab administrators will receive an email -notification of the situation. This notification is sent out no more -than once a day. +notification of the situation. This notification is sent out once a week on +Sunday, by default. ## Disabling periodic checks diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md new file mode 100644 index 00000000000..d6aa6101026 --- /dev/null +++ b/doc/administration/repository_storage_paths.md @@ -0,0 +1,102 @@ +# Repository storage paths + +> [Introduced][ce-4578] in GitLab 8.10. + +GitLab allows you to define multiple repository storage paths to distribute the +storage load between several mount points. + +>**Notes:** +> +- You must have at least one storage path called `default`. +- The paths are defined in key-value pairs. The key is an arbitrary name you + can pick to name the file path. +- The target directories and any of its subpaths must not be a symlink. + +## Configure GitLab + +>**Warning:** +In order for [backups] to work correctly, the storage path must **not** be a +mount point and the GitLab user should have correct permissions for the parent +directory of the path. In Omnibus GitLab this is taken care of automatically, +but for source installations you should be extra careful. +> +The thing is that for compatibility reasons `gitlab.yml` has a different +structure than Omnibus. In `gitlab.yml` you indicate the path for the +repositories, for example `/home/git/repositories`, while in Omnibus you +indicate `git_data_dirs`, which for the example above would be `/home/git`. +Then, Omnibus will create a `repositories` directory under that path to use with +`gitlab.yml`. +> +This little detail matters because while restoring a backup, the current +contents of `/home/git/repositories` [are moved to][raketask] `/home/git/repositories.old`, +so if `/home/git/repositories` is the mount point, then `mv` would be moving +things between mount points, and bad things could happen. Ideally, +`/home/git` would be the mount point, so then things would be moving within the +same mount point. This is guaranteed with Omnibus installations (because they +don't specify the full repository path but the parent path), but not for source +installations. + +--- + +Now that you've read that big fat warning above, let's edit the configuration +files and add the full paths of the alternative repository storage paths. In +the example below, we add two more mountpoints that are named `nfs` and `cephfs` +respectively. + +**For installations from source** + +1. Edit `gitlab.yml` and add the storage paths: + + ```yaml + repositories: + # Paths where repositories can be stored. Give the canonicalized absolute pathname. + # NOTE: REPOS PATHS MUST NOT CONTAIN ANY SYMLINK!!! + storages: # You must have at least a 'default' storage path. + default: /home/git/repositories + nfs: /mnt/nfs/repositories + cephfs: /mnt/cephfs/repositories + ``` + +1. [Restart GitLab] for the changes to take effect. + +>**Note:** +The [`gitlab_shell: repos_path` entry][repospath] in `gitlab.yml` will be +deprecated and replaced by `repositories: storages` in the future, so if you +are upgrading from a version prior to 8.10, make sure to add the configuration +as described in the step above. After you make the changes and confirm they are +working, you can remove the `repos_path` line. + +--- + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the + default one: + + ```ruby + git_data_dirs({ + "default" => "/var/opt/gitlab/git-data", + "nfs" => "/mnt/nfs/git-data", + "cephfs" => "/mnt/cephfs/git-data" + }) + ``` + + Note that Omnibus stores the repositories in a `repositories` subdirectory + of the `git-data` directory. + +## Choose where new project repositories will be stored + +Once you set the multiple storage paths, you can choose where new projects will +be stored via the **Application Settings** in the Admin area. + + + +Beginning with GitLab 8.13.4, multiple paths can be chosen. New projects will be +randomly placed on one of the selected paths. + +[ce-4578]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4578 +[restart gitlab]: restart_gitlab.md#installations-from-source +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure +[backups]: ../raketasks/backup_restore.md +[raketask]: https://gitlab.com/gitlab-org/gitlab-ce/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56 +[repospath]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-9-stable/config/gitlab.yml.example#L457 diff --git a/doc/administration/repository_storages.md b/doc/administration/repository_storages.md index ab70557b69a..9d41ba77f34 100644 --- a/doc/administration/repository_storages.md +++ b/doc/administration/repository_storages.md @@ -1,102 +1,3 @@ # Repository storages -> [Introduced][ce-4578] in GitLab 8.10. - -GitLab allows you to define multiple repository storage paths to distribute the -storage load between several mount points. - ->**Notes:** -> -- You must have at least one storage path called `default`. -- The paths are defined in key-value pairs. The key is an arbitrary name you - can pick to name the file path. -- The target directories and any of its subpaths must not be a symlink. - -## Configure GitLab - ->**Warning:** -In order for [backups] to work correctly, the storage path must **not** be a -mount point and the GitLab user should have correct permissions for the parent -directory of the path. In Omnibus GitLab this is taken care of automatically, -but for source installations you should be extra careful. -> -The thing is that for compatibility reasons `gitlab.yml` has a different -structure than Omnibus. In `gitlab.yml` you indicate the path for the -repositories, for example `/home/git/repositories`, while in Omnibus you -indicate `git_data_dirs`, which for the example above would be `/home/git`. -Then, Omnibus will create a `repositories` directory under that path to use with -`gitlab.yml`. -> -This little detail matters because while restoring a backup, the current -contents of `/home/git/repositories` [are moved to][raketask] `/home/git/repositories.old`, -so if `/home/git/repositories` is the mount point, then `mv` would be moving -things between mount points, and bad things could happen. Ideally, -`/home/git` would be the mount point, so then things would be moving within the -same mount point. This is guaranteed with Omnibus installations (because they -don't specify the full repository path but the parent path), but not for source -installations. - ---- - -Now that you've read that big fat warning above, let's edit the configuration -files and add the full paths of the alternative repository storage paths. In -the example below, we add two more mountpoints that are named `nfs` and `cephfs` -respectively. - -**For installations from source** - -1. Edit `gitlab.yml` and add the storage paths: - - ```yaml - repositories: - # Paths where repositories can be stored. Give the canonicalized absolute pathname. - # NOTE: REPOS PATHS MUST NOT CONTAIN ANY SYMLINK!!! - storages: # You must have at least a 'default' storage path. - default: /home/git/repositories - nfs: /mnt/nfs/repositories - cephfs: /mnt/cephfs/repositories - ``` - -1. [Restart GitLab] for the changes to take effect. - ->**Note:** -The [`gitlab_shell: repos_path` entry][repospath] in `gitlab.yml` will be -deprecated and replaced by `repositories: storages` in the future, so if you -are upgrading from a version prior to 8.10, make sure to add the configuration -as described in the step above. After you make the changes and confirm they are -working, you can remove the `repos_path` line. - ---- - -**For Omnibus installations** - -1. Edit `/etc/gitlab/gitlab.rb` by appending the rest of the paths to the - default one: - - ```ruby - git_data_dirs({ - "default" => "/var/opt/gitlab/git-data", - "nfs" => "/mnt/nfs/git-data", - "cephfs" => "/mnt/cephfs/git-data" - }) - ``` - - Note that Omnibus stores the repositories in a `repositories` subdirectory - of the `git-data` directory. - -## Choose where new project repositories will be stored - -Once you set the multiple storage paths, you can choose where new projects will -be stored via the **Application Settings** in the Admin area. - - - -Beginning with GitLab 8.13.4, multiple paths can be chosen. New projects will be -randomly placed on one of the selected paths. - -[ce-4578]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4578 -[restart gitlab]: restart_gitlab.md#installations-from-source -[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure -[backups]: ../raketasks/backup_restore.md -[raketask]: https://gitlab.com/gitlab-org/gitlab-ce/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56 -[repospath]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-9-stable/config/gitlab.yml.example#L457 +This document was moved to a [new location](repository_storage_paths.md). diff --git a/doc/api/README.md b/doc/api/README.md index f65b934b9db..20f28e8d30e 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -104,6 +104,13 @@ that needs access to the GitLab API. Once you have your token, pass it to the API using either the `private_token` parameter or the `PRIVATE-TOKEN` header. +> [Introduced][ce-5951] in GitLab 8.15. + +Personal Access Tokens can be created with one or more scopes that allow various actions +that a given token can perform. Although there are only two scopes available at the +moment – `read_user` and `api` – the groundwork has been laid to add more scopes easily. + +At any time you can revoke any personal access token by just clicking **Revoke**. ### Session Cookie @@ -380,3 +387,4 @@ programming languages. Visit the [GitLab website] for a complete list. [GitLab website]: https://about.gitlab.com/applications/#api-clients "Clients using the GitLab API" [lib-api-url]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/api/api.rb [ce-3749]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3749 +[ce-5951]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951 diff --git a/doc/api/enviroments.md b/doc/api/enviroments.md index 1299aca8c45..e0ee20d9610 100644 --- a/doc/api/enviroments.md +++ b/doc/api/enviroments.md @@ -78,7 +78,7 @@ PUT /projects/:id/environments/:environments_id | `external_url` | string | no | The new external_url | ```bash -curl --request PUT --data "name=staging&external_url=https://staging.example.gitlab.com" --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/1/environment/1" +curl --request PUT --data "name=staging&external_url=https://staging.example.gitlab.com" --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/1/environments/1" ``` Example response: @@ -106,7 +106,7 @@ DELETE /projects/:id/environments/:environment_id | `environment_id` | integer | yes | The ID of the environment | ```bash -curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/1/environment/1" +curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/1/environments/1" ``` Example response: diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index ffc310ec8c7..5377bf9ee80 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -14,6 +14,12 @@ Apart from those, here is an collection of tutorials and guides on setting up yo - [Test a Phoenix application](test-phoenix-application.md) - [Using `dpl` as deployment tool](deployment/README.md) - [Example project that shows how to use Review Apps](https://gitlab.com/gitlab-examples/review-apps-nginx/) +- [Run PHP Composer & NPM scripts then deploy them to a staging server](deployment/composer-npm-deploy.md) +- Help your favorite programming language and GitLab by sending a merge request + with a guide for that language. + +## Outside the documentation + - [Blog post about using GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) - [Repositories with examples for various languages](https://gitlab.com/groups/gitlab-examples) - [The .gitlab-ci.yml file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) diff --git a/doc/ci/examples/deployment/composer-npm-deploy.md b/doc/ci/examples/deployment/composer-npm-deploy.md new file mode 100644 index 00000000000..5334a73e1f5 --- /dev/null +++ b/doc/ci/examples/deployment/composer-npm-deploy.md @@ -0,0 +1,156 @@ +## Running Composer and NPM scripts with deployment via SCP + +This guide covers the building dependencies of a PHP project while compiling assets via an NPM script. + +While is possible to create your own image with custom PHP and Node JS versions, for brevity, we will use an existing [Docker image](https://hub.docker.com/r/tetraweb/php/) that contains both PHP and NodeJS installed. + + +```yaml +image: tetraweb/php +``` + +The next step is to install zip/unzip packages and make composer available. We will place these in the `before_script` section: + +```yaml +before_script: + - apt-get update + - apt-get install zip unzip + - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" + - php composer-setup.php + - php -r "unlink('composer-setup.php');" +``` + +This will make sure we have all requirements ready. Next, we want to run `composer update` to fetch all PHP dependencies and `npm install` to load node packages, then run the `npm` script. We need to append them into `before_script` section: + +```yaml +before_script: + # ... + - php composer.phar update + - npm install + - npm run deploy +``` + +In this particular case, the `npm deploy` script is a Gulp script that does the following: + +1. Compile CSS & JS +2. Create sprites +3. Copy various assets (images, fonts) around +4. Replace some strings + +All these operations will put all files into a `build` folder, which is ready to be deployed to a live server. + +### How to transfer files to a live server? + +You have multiple options: rsync, scp, sftp and so on. For now, we will use scp. + +To make this work, you need to add a GitLab Secret Variable (accessible on _gitlab.example/your-project-name/variables_). That variable will be called `STAGING_PRIVATE_KEY` and it's the **private** ssh key of your server. + +#### Security tip + +Create a user that has access **only** to the folder that needs to be updated! + +After you create that variable, you need to make sure that key will be added to the docker container on run: + +```yaml +before_script: + # - .... + - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + - mkdir -p ~/.ssh + - eval $(ssh-agent -s) + - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config' +``` + +In order, this means that: + +1. We check if the `ssh-agent` is available and we install it if it's not; +2. We create the `~/.ssh` folder; +3. We make sure we're running bash; +4. We disable host checking (we don't ask for user accept when we first connect to a server; and since every build will equal a first connect, we kind of need this) + +And this is basically all you need in the `before_script` section. + +## How to deploy things? + +As we stated above, we need to deploy the `build` folder from the docker image to our server. To do so, we create a new job: + +```yaml +stage_deploy: + artifacts: + paths: + - build/ + only: + - dev + script: + - ssh-add <(echo "$STAGING_PRIVATE_KEY") + - ssh -p22 server_user@server_host "mkdir htdocs/wp-content/themes/_tmp" + - scp -P22 -r build/* server_user@server_host:htdocs/wp-content/themes/_tmp + - ssh -p22 server_user@server_host "mv htdocs/wp-content/themes/live htdocs/wp-content/themes/_old && mv htdocs/wp-content/themes/_tmp htdocs/wp-content/themes/live" + - ssh -p22 server_user@server_host "rm -rf htdocs/wp-content/themes/_old" +``` + +### What's going on here? + +1. `only:dev` means that this build will run only when something is pushed to the `dev` branch. You can remove this block completely and have everything be ran on every push (but probably this is something you don't want) +2. `ssh-add ...` we will add that private key you added on the web UI to the docker container +3. We will connect via `ssh` and create a new `_tmp` folder +4. We will connect via `scp` and upload the `build` folder (which was generated by a `npm` script) to our previously created `_tmp` folder +5. We will connect again to `ssh` and move the `live` folder to an `_old` folder, then move `_tmp` to `live`. +6. We connect to ssh and remove the `_old` folder + +What's the deal with the artifacts? We just tell GitLab CI to keep the `build` directory (later on, you can download that as needed). + +#### Why we do it this way? + +If you're using this only for stage server, you could do this in two steps: + +```yaml +- ssh -p22 server_user@server_host "rm -rf htdocs/wp-content/themes/live/*" +- scp -P22 -r build/* server_user@server_host:htdocs/wp-content/themes/live +``` + +The problem is that there will be a small period of time when you won't have the app on your server. + +So we use so many steps because we want to make sure that at any given time we have a functional app in place. + +## Where to go next? + +Since this was a WordPress project, I gave real life code snippets. Some ideas you can pursuit: + +- Having a slightly different script for `master` branch will allow you to deploy to a production server from that branch and to a stage server from any other branches; +- Instead of pushing it live, you can push it to WordPress official repo (with creating a SVN commit & stuff); +- You could generate i18n text domains on the fly. + +--- + +Our final `.gitlab-ci.yml` will look like this: + +```yaml +image: tetraweb/php + +before_script: + - apt-get update + - apt-get install zip unzip + - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" + - php composer-setup.php + - php -r "unlink('composer-setup.php');" + - php composer.phar update + - npm install + - npm run deploy + - 'which ssh-agent || ( apt-get update -y && apt-get install openssh-client -y )' + - mkdir -p ~/.ssh + - eval $(ssh-agent -s) + - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config' + +stage_deploy: + artifacts: + paths: + - build/ + only: + - dev + script: + - ssh-add <(echo "$STAGING_PRIVATE_KEY") + - ssh -p22 server_user@server_host "mkdir htdocs/wp-content/themes/_tmp" + - scp -P22 -r build/* server_user@server_host:htdocs/wp-content/themes/_tmp + - ssh -p22 server_user@server_host "mv htdocs/wp-content/themes/live htdocs/wp-content/themes/_old && mv htdocs/wp-content/themes/_tmp htdocs/wp-content/themes/live" + - ssh -p22 server_user@server_host "rm -rf htdocs/wp-content/themes/_old" +```
\ No newline at end of file diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 1d782200cca..869743ce80a 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -61,7 +61,18 @@ correctly with your CI builds: 1. First, make sure you have used [relative URLs](#configuring-the-gitmodules-file) for the submodules located in the same GitLab server. -1. Then, use `git submodule sync/update` in `before_script`: +1. Next, if you are using `gitlab-ci-multi-runner` v1.10+, you can set the + `GIT_SUBMODULE_STRATEGY` variable to either `normal` or `recursive` to tell + the runner to fetch your submodules before the build: + ```yaml + variables: + GIT_SUBMODULE_STRATEGY: recursive + ``` + See the [`.gitlab-ci.yml` reference](yaml/README.md#git-submodule-strategy) + for more details about `GIT_SUBMODULE_STRATEGY`. + +1. If you are using an older version of `gitlab-ci-multi-runner`, then use + `git submodule sync/update` in `before_script`: ```yaml before_script: diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 7158b2e7895..f11257be5c3 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -86,7 +86,7 @@ used for time of the build. The configuration of this feature is covered in ### before_script `before_script` is used to define the command that should be run before all -builds, including deploy builds. This can be an array or a multi-line string. +builds, including deploy builds, but after the restoration of artifacts. This can be an array or a multi-line string. ### after_script @@ -1034,6 +1034,41 @@ variables: GIT_STRATEGY: none ``` +## Git Submodule Strategy + +> Requires GitLab Runner v1.10+. + +The `GIT_SUBMODULE_STRATEGY` variable is used to control if / how Git +submodules are included when fetching the code before a build. Like +`GIT_STRATEGY`, it can be set in either the global [`variables`](#variables) +section or the [`variables`](#job-variables) section for individual jobs. + +There are three posible values: `none`, `normal`, and `recursive`: + +- `none` means that submodules will not be included when fetching the project + code. This is the default, which matches the pre-v1.10 behavior. + +- `normal` means that only the top-level submodules will be included. It is + equivalent to: + ``` + $ git submodule sync + $ git submodule update --init + ``` + +- `recursive` means that all submodules (including submodules of submodules) + will be included. It is equivalent to: + ``` + $ git submodule sync --recursive + $ git submodule update --init --recursive + ``` + +Note that for this feature to work correctly, the submodules must be configured +(in `.gitmodules`) with either: +- the HTTP(S) URL of a publicly-accessible repository, or +- a relative path to another repository on the same GitLab server. See the + [Git submodules](../git_submodules.md) documentation. + + ## Build stages attempts > Introduced in GitLab, it requires GitLab Runner v1.9+. diff --git a/doc/development/README.md b/doc/development/README.md index 6f2ca7b8590..265df98fb87 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -18,6 +18,7 @@ - [Frontend guidelines](frontend.md) - [SQL guidelines](sql.md) for working with SQL queries - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers +- [`Gemfile` guidelines](gemfile.md) ## Process diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 1ef34c79971..e4a0e0b92bc 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -9,7 +9,7 @@ code is effective, understandable, and maintainable. Any developer can, and is encouraged to, perform code review on merge requests of colleagues and contributors. However, the final decision to accept a merge -request is up to one of our merge request "endbosses", denoted on the +request is up to one the project's maintainers, denoted on the [team page](https://about.gitlab.com/team). ## Everyone @@ -81,15 +81,15 @@ balance in how deep the reviewer can interfere with the code created by a reviewee. - Learning how to find the right balance takes time; that is why we have - minibosses that become merge request endbosses after some time spent on - reviewing merge requests. + reviewers that become maintainers after some time spent on reviewing merge + requests. - Finding bugs and improving code style is important, but thinking about good design is important as well. Building abstractions and good design is what makes it possible to hide complexity and makes future changes easier. - Asking the reviewee to change the design sometimes means the complete rewrite - of the contributed code. It's usually a good idea to ask another merge - request endboss before doing it, but have the courage to do it when you - believe it is important. + of the contributed code. It's usually a good idea to ask another maintainer or + reviewer before doing it, but have the courage to do it when you believe it is + important. - There is a difference in doing things right and doing things right now. Ideally, we should do the former, but in the real world we need the latter as well. A good example is a security fix which should be released as soon as diff --git a/doc/development/frontend.md b/doc/development/frontend.md index 9e782ab977f..f79bd23dc90 100644 --- a/doc/development/frontend.md +++ b/doc/development/frontend.md @@ -297,16 +297,74 @@ For our currently-supported browsers, see our [requirements][requirements]. ## Gotchas -### Phantom.JS (used by Teaspoon & Rspec) chokes, returning vague JavaScript errors - -If you see very generic JavaScript errors (e.g. `jQuery is undefined`) being thrown in tests, but -can't reproduce them manually, you may have included `ES6`-style JavaScript in files that don't -have the `.js.es6` file extension. Either use ES5-friendly JavaScript or rename the file you're -working in (`git mv <file.js> <file.js.es6>`). - -Similar errors will be thrown if you're using -any of the [array methods introduced in ES6](http://www.2ality.com/2014/05/es6-array-methods.html) -whether or not you've updated the file extension. +### Spec errors due to use of ES6 features in `.js` files + +If you see very generic JavaScript errors (e.g. `jQuery is undefined`) being +thrown in Teaspoon, Spinach, or Rspec tests but can't reproduce them manually, +you may have included `ES6`-style JavaScript in files that don't have the +`.js.es6` file extension. Either use ES5-friendly JavaScript or rename the file +you're working in (`git mv <file.js> <file.js.es6>`). + +### Spec errors due to use of unsupported JavaScript + +Similar errors will be thrown if you're using JavaScript features not yet +supported by our test runner's version of webkit, whether or not you've updated +the file extension. Examples of unsupported JavaScript features are: + +- Array.from +- Array.find +- Array.first +- Object.assign +- Async functions +- Generators +- Array destructuring +- For Of +- Symbol/Symbol.iterator +- Spread + +Until these are polyfilled or transpiled appropriately, they should not be used. +Please update this list with additional unsupported features or when any of +these are made usable. + +### Spec errors due to JavaScript not enabled + +If, as a result of a change you've made, a feature now depends on JavaScript to +run correctly, you need to make sure a JavaScript web driver is enabled when +specs are run. If you don't you'll see vague error messages from the spec +runner, and an explosion of vague console errors in the HTML snapshot. + +To enable a JavaScript driver in an `rspec` test, add `js: true` to the +individual spec or the context block containing multiple specs that need +JavaScript enabled: + +```ruby + +# For one spec +it 'presents information about abuse report', js: true do + # assertions... +end + +describe "Admin::AbuseReports", js: true do + it 'presents information about abuse report' do + # assertions... + end + it 'shows buttons for adding to abuse report' do + # assertions... + end +end +``` +In Spinach, the JavaScript driver is enabled differently. In the `*.feature` +file for the failing spec, add the `@javascript` flag above the Scenario: +``` +@javascript +Scenario: Developer can approve merge request + Given I am a "Shop" developer + And I visit project "Shop" merge requests page + And merge request 'Bug NS-04' must be approved + And I click link "Bug NS-04" + When I click link "Approve" + Then I should see approved merge request "Bug NS-04" +``` diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md new file mode 100644 index 00000000000..ec9718cea71 --- /dev/null +++ b/doc/development/gemfile.md @@ -0,0 +1,14 @@ +# `Gemfile` guidelines + +When adding a new entry to `Gemfile` or upgrading an existing dependency pay +attention to the following rules. + +## No gems fetched from git repositories + +We do not allow gems that are fetched from git repositories. All gems have +to be available in the RubyGems index. We want to minimize external build +dependencies and build times. + +## License compliance + +Refer to [licensing guidelines](licensing.md) for ensuring license compliance. diff --git a/doc/development/merge_request_performance_guidelines.md b/doc/development/merge_request_performance_guidelines.md index 0363bf8c1d5..8232a0a113c 100644 --- a/doc/development/merge_request_performance_guidelines.md +++ b/doc/development/merge_request_performance_guidelines.md @@ -3,7 +3,7 @@ To ensure a merge request does not negatively impact performance of GitLab _every_ merge request **must** adhere to the guidelines outlined in this document. There are no exceptions to this rule unless specifically discussed -with and agreed upon by merge request endbosses and performance specialists. +with and agreed upon by backend maintainers and performance specialists. To measure the impact of a merge request you can use [Sherlock](profiling.md#sherlock). It's also highly recommended that you read @@ -40,9 +40,9 @@ section below for more information. about the impact. Sometimes it's hard to assess the impact of a merge request. In this case you -should ask one of the merge request (mini) endbosses to review your changes. You -can find a list of these endbosses at <https://about.gitlab.com/team/>. An -endboss in turn can request a performance specialist to review the changes. +should ask one of the merge request reviewers to review your changes. You can +find a list of these reviewers at <https://about.gitlab.com/team/>. A reviewer +in turn can request a performance specialist to review the changes. ## Query Counts diff --git a/doc/gitlab-basics/add-image.md b/doc/gitlab-basics/add-image.md index 476b48a217c..1a44123aa81 100644 --- a/doc/gitlab-basics/add-image.md +++ b/doc/gitlab-basics/add-image.md @@ -1,62 +1,56 @@ # How to add an image -The following are the steps to add images to your repository in -GitLab: +Using your standard tool for copying files (e.g. Finder in Mac OS, or Explorer +in Windows, or...), put the image file into the GitLab project. You can find the +project as a regular folder in your files. -Find the image that you’d like to add. +Go to your [shell](command-line-commands.md), and move into the folder of your +Gitlab project. This usually means running the following command until you get +to the desired destination: -In your computer files, find the GitLab project to which you'd like to add the image -(you'll find it as a regular file). Click on every file until you find exactly where you'd -like to add the image. There, paste the image. - -Go to your [shell](command-line-commands.md), and add the following commands: - -Add this command for every directory that you'd like to open: ``` -cd NAME-OF-FILE-YOU'D-LIKE-TO-OPEN +cd NAME-OF-FOLDER-YOU'D-LIKE-TO-OPEN ``` -Create a new branch: -``` -git checkout -b NAME-OF-BRANCH -``` +Check if your image is actually present in the directory (if you are in Windows, +use `dir` instead): -Check if your image was correctly added to the directory: ``` ls ``` You should see the name of the image in the list shown. -Move up the hierarchy through directories: -``` -cd ../ -``` +Check the status: -Check the status and you should see your image’s name in red: ``` git status ``` -Add your changes: +Your image's name should appear in red, so `git` took notice of it! Now add it +to the repository: + ``` git add NAME-OF-YOUR-IMAGE ``` -Check the status and you should see your image’s name in green: +Check the status again, your image's name should have turned green: + ``` git status ``` -Add the commit: +Commit: + ``` -git commit -m “DESCRIBE COMMIT IN A FEW WORDS” +git commit -m "DESCRIBE COMMIT IN A FEW WORDS" ``` -Now you can push (send) your changes (in the branch NAME-OF-BRANCH) to GitLab (the git remote named 'origin'): +Now you can push (send) your changes (in the branch NAME-OF-BRANCH) to GitLab +(the git remote named 'origin'): + ``` git push origin NAME-OF-BRANCH ``` -Your image will be added to your branch in your repository in GitLab. Create a [Merge Request](add-merge-request.md) -to integrate your changes to your project. +Your image will be added to your branch in your repository in GitLab. diff --git a/doc/install/database_mysql.md b/doc/install/database_mysql.md index 322680f0cf4..65bfb0f7d6e 100644 --- a/doc/install/database_mysql.md +++ b/doc/install/database_mysql.md @@ -4,7 +4,7 @@ We do not recommend using MySQL due to various issues. For example, case [(in)sensitivity](https://dev.mysql.com/doc/refman/5.0/en/case-sensitivity.html) and [problems](https://bugs.mysql.com/bug.php?id=65830) that [suggested](https://bugs.mysql.com/bug.php?id=50909) [fixes](https://bugs.mysql.com/bug.php?id=65830) [have](https://bugs.mysql.com/bug.php?id=63164). -## MySQL +## Initial database setup # Install the database packages sudo apt-get install -y mysql-server mysql-client libmysqlclient-dev @@ -32,8 +32,11 @@ We do not recommend using MySQL due to various issues. For example, case [(in)se # If this fails, check your MySQL config files (e.g. `/etc/mysql/*.cnf`, `/etc/mysql/conf.d/*`) for the setting "innodb = off" mysql> SET storage_engine=INNODB; + # If you have MySQL < 5.7.7 and want to enable utf8mb4 character set support with your GitLab install, you must set the following NOW: + mysql> SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda, innodb_large_prefix=1; + # Create the GitLab production database - mysql> CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; + mysql> CREATE DATABASE IF NOT EXISTS `gitlabhq_production` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_general_ci`; # Grant the GitLab user necessary permissions on the database mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES, REFERENCES ON `gitlabhq_production`.* TO 'git'@'localhost'; @@ -51,7 +54,203 @@ We do not recommend using MySQL due to various issues. For example, case [(in)se # Quit the database session mysql> \q - # You are done installing the database and can go back to the rest of the installation. + # You are done installing the database for now and can go back to the rest of the installation. + +Please proceed to the rest of the installation before running through the utf8mb4 support section. + + +### MySQL utf8mb4 support + +After installation or upgrade, remember to [convert any new tables](#convert) to `utf8mb4`/`utf8mb4_general_ci`. + +--- + +GitLab 8.14 has introduced [a feature](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7420) requiring `utf8mb4` encoding to be supported in your GitLab MySQL Database, which is not the case if you have setup your database before GitLab 8.16. + +Follow the below instructions to ensure you use the most up to date requirements for your GitLab MySQL Database. + +**We are about to do the following:** +- Ensure you can enable `utf8mb4` encoding and `utf8mb4_general_ci` collation for your GitLab DB, tables and data. +- Convert your GitLab tables and data from `utf8`/`utf8_general_ci` to `utf8mb4`/`utf8mb4_general_ci` + +### Check for utf8mb4 support + +#### Check for InnoDB File-Per-Table Tablespaces + +We need to check, enable and maybe convert your existing GitLab DB tables to the [InnoDB File-Per-Table Tablespaces](http://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) as a prerequise for supporting **utfb8mb4 with long indexes** required by recent GitLab databases. + + # Login to MySQL + mysql -u root -p + + # Type the MySQL root password + mysql > use gitlabhq_production; + + # Check your MySQL version is >= 5.5.3 (GitLab requires 5.5.14+) + mysql > SHOW VARIABLES LIKE 'version'; + +---------------+-----------------+ + | Variable_name | Value | + +---------------+-----------------+ + | version | 5.5.53-0+deb8u1 | + +---------------+-----------------+ + + # Note where is your MySQL data dir for later: + mysql > select @@datadir; + +----------------+ + | @@datadir | + +----------------+ + | /var/lib/mysql | + +----------------+ + + # Note whether your MySQL server runs with innodb_file_per_table ON or OFF: + mysql> SELECT @@innodb_file_per_table; + +-------------------------+ + | @@innodb_file_per_table | + +-------------------------+ + | 1 | + +-------------------------+ + + # You can now quit the database session + mysql> \q + +> You need **MySQL 5.5.3 or later** to perform this update. + +Whatever the results of your checks above, we now need to check if your GitLab database has been created using [InnoDB File-Per-Table Tablespaces](http://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html) (i.e. `innodb_file_per_table` was set to **1** at initial setup time). + +> Note: This setting is [enabled by default](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_file_per_table) since MySQL 5.6.6. + + # Run this command with root privileges, replace the data dir if different: + sudo ls -lh /var/lib/mysql/gitlabhq_production/*.ibd | wc -l + + # Run this command with root privileges, replace the data dir if different: + sudo ls -lh /var/lib/mysql/gitlabhq_production/*.frm | wc -l + + +- **Case 1: a result > 0 for both commands** + +Congrats, your GitLab database uses the right InnoDB tablespace format. + +However, you must still ensure that any **future tables** created by GitLab will still use the right format: + +- If `SELECT @@innodb_file_per_table` returned **1** previously, your server is running correctly. +> It's however a requirement to check *now* that this setting is indeed persisted in your [my.cnf](https://dev.mysql.com/doc/refman/5.7/en/tablespace-enabling.html) file! + +- If `SELECT @@innodb_file_per_table` returned **0** previously, your server is not running correctly. +> [Enable innodb_file_per_table](https://dev.mysql.com/doc/refman/5.7/en/tablespace-enabling.html) by running in a MySQL session as root the command `SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda;` and persist the two settings in your [my.cnf](https://dev.mysql.com/doc/refman/5.7/en/tablespace-enabling.html) file + +Now, if you have a **different result** returned by the 2 commands above, it means you have a **mix of tables format** uses in your GitLab database. This can happen if your MySQL server had different values for `innodb_file_per_table` in its life and you updated GitLab at different moments with those inconsistent values. So keep reading. + +- **Case 2: a result equals to "0" OR not the same result for both commands** + +Unfortunately, none or only some of your GitLab database tables use the GitLab requirement of [InnoDB File-Per-Table Tablespaces](http://dev.mysql.com/doc/refman/5.7/en/innodb-multiple-tablespaces.html). + +Let's enable what we need on the running server: + + # Login to MySQL + mysql -u root -p + + # Type the MySQL root password + + # Enable innodb_file_per_table and set innodb_file_format on the running server: + mysql > SET GLOBAL innodb_file_per_table=1, innodb_file_format=Barracuda; + + # You can now quit the database session + mysql> \q + +> Now, **persist** [innodb_file_per_table](https://dev.mysql.com/doc/refman/5.6/en/tablespace-enabling.html) and [innodb_file_format](https://dev.mysql.com/doc/refman/5.6/en/innodb-file-format-enabling.html) in your `my.cnf` file. + +Ensure at this stage that your GitLab instance is indeed **stopped**. + +Now, let's convert all the GitLab database tables to the new tablespace format: + + # Login to MySQL + mysql -u root -p + + # Type the MySQL root password + mysql > use gitlabhq_production; + + # Safety check: you should still have those values set as follow: + mysql> SELECT @@innodb_file_per_table, @@innodb_file_format; + +-------------------------+----------------------+ + | @@innodb_file_per_table | @@innodb_file_format | + +-------------------------+----------------------+ + | 1 | Barracuda | + +-------------------------+----------------------+ + + mysql > SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` ENGINE=InnoDB;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_TYPE="BASE TABLE"; + + # If previous query returned results, copy & run all shown SQL statements + + # You can now quit the database session + mysql> \q + +--- + +#### Check for proper InnoDB File Format, Row Format, Large Prefix and tables conversion + +We need to check, enable and probably convert your existing GitLab DB tables to use the [Barracuda InnoDB file format](https://dev.mysql.com/doc/refman/5.6/en/innodb-file-format.html), the [DYNAMIC row format](https://dev.mysql.com/doc/refman/5.6/en/glossary.html#glos_dynamic_row_format) and [innodb_large_prefix](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix) as a second prerequisite for supporting **utfb8mb4 with long indexes** used by recent GitLab databases. + + # Login to MySQL + mysql -u root -p + + # Type the MySQL root password + mysql > use gitlabhq_production; + + # Set innodb_file_format and innodb_large_prefix on the running server: + # Note: These are the default settings only for MySQL 5.7.7 and later. + + mysql > SET GLOBAL innodb_file_format=Barracuda, innodb_large_prefix=1; + + # Your DB must be (still) using utf8/utf8_general_ci as default encoding and collation. + # We will NOT change the default encoding and collation on the DB in order to support future GitLab migrations creating tables + # that require "long indexes support" on installations using MySQL <= 5.7.9. + # However, when such migrations occur, you will have to follow this guide again to convert the newly created tables to the proper encoding/collation. + + # This should return the following: + mysql> SELECT @@character_set_database, @@collation_database; + +--------------------------+----------------------+ + | @@character_set_database | @@collation_database | + +--------------------------+----------------------+ + | utf8 | utf8_general_ci | + +--------------------------+----------------------+ + +> Now, ensure that [innodb_file_format](https://dev.mysql.com/doc/refman/5.6/en/tablespace-enabling.html) and [innodb_large_prefix](http://dev.mysql.com/doc/refman/5.7/en/innodb-parameters.html#sysvar_innodb_large_prefix) are **persisted** in your `my.cnf` file. + +#### Tables and data conversion to utf8mb4 +<a name="convert"></a> + +Now that you have a persistent MySQL setup, you can safely upgrade tables after setup or upgrade time: + + # Convert tables not using ROW_FORMAT DYNAMIC: + + mysql> SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` ROW_FORMAT=DYNAMIC;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_TYPE="BASE TABLE" AND ROW_FORMAT!="Dynamic"; + + # !! If previous query returned results, copy & run all shown SQL statements + + # Convert tables/columns not using utf8mb4/utf8mb4_general_ci as encoding/collation: + + mysql > SET foreign_key_checks = 0; + + mysql > SELECT CONCAT('ALTER TABLE `', TABLE_NAME,'` CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci;') AS 'Copy & run these SQL statements:' FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_SCHEMA="gitlabhq_production" AND TABLE_COLLATION != "utf8mb4_general_ci" AND TABLE_TYPE="BASE TABLE"; + + # !! If previous query returned results, copy & run all shown SQL statements + + # Turn foreign key checks back on + mysql > SET foreign_key_checks = 1; + + # You can now quit the database session + mysql> \q + +Ensure your GitLab database configuration file uses a proper connection encoding and collation: + +```sudo -u git -H editor config/database.yml``` + + production: + adapter: mysql2 + encoding: utf8mb4 + collation: utf8mb4_general_ci + +[Restart your GitLab instance](../administration/restart_gitlab.md). + ## MySQL strings limits diff --git a/doc/install/installation.md b/doc/install/installation.md index 9cebed34b7e..3e7674e13ab 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -124,7 +124,7 @@ Download Ruby and compile it: mkdir /tmp/ruby && cd /tmp/ruby curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.3.tar.gz - echo 'a8db9ce7f9110320f33b8325200e3ecfbd2b534b ruby-2.3.3.tar.gz' | shasum -c - && tar xzf ruby-2.3.3.tar.gz + echo '1014ee699071aa2ddd501907d18cbe15399c997d ruby-2.3.3.tar.gz' | shasum -c - && tar xzf ruby-2.3.3.tar.gz cd ruby-2.3.3 ./configure --disable-install-rdoc make diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md index 0c53584d201..af8a1c4e5ed 100644 --- a/doc/integration/oauth_provider.md +++ b/doc/integration/oauth_provider.md @@ -74,8 +74,10 @@ in the **Authorized applications** section under **Profile Settings > Applicatio --- -As you can see, the default scope `api` is used, which is the only scope that -GitLab supports so far. At any time you can revoke any access by just clicking -**Revoke**. +GitLab's OAuth applications support scopes, which allow various actions that any given +application can perform. Although there are only two scopes available at the +moment – `read_user` and `api` – the groundwork has been laid to add more scopes easily. + +At any time you can revoke any access by just clicking **Revoke**. [oauth]: http://oauth.net/2/ "OAuth website" diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 4c933cef9b7..98a680d0dbe 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -41,6 +41,9 @@ that are in common for all providers that we need to consider. - `allow_single_sign_on` allows you to specify the providers you want to allow to automatically create an account. It defaults to `false`. If `false` users must be created manually or they will not be able to sign in via OmniAuth. +- `auto_link_ldap_user` can be used if you have [LDAP / ActiveDirectory](ldap.md) + integration enabled. It defaults to false. When enabled, users automatically + created through OmniAuth will be linked to their LDAP entry as well. - `block_auto_created_users` defaults to `true`. If `true` auto created users will be blocked by default and will have to be unblocked by an administrator before they are able to sign in. @@ -52,6 +55,10 @@ SAML, Shibboleth, Crowd or Google, or set it to `false` otherwise any user on the Internet will be able to successfully sign in to your GitLab without administrative approval. +>**Note:** +`auto_link_ldap_user` requires the `uid` of the user to be the same in both LDAP +and the OmniAuth provider. + To change these settings: * **For omnibus package** @@ -72,6 +79,7 @@ To change these settings: # using an array, e.g. ["saml", "twitter"], or as true/false to allow all providers or none. # User accounts will be created automatically when authentication was successful. gitlab_rails['omniauth_allow_single_sign_on'] = ['saml', 'twitter'] + gitlab_rails['omniauth_auto_link_ldap_user'] = true gitlab_rails['omniauth_block_auto_created_users'] = true ``` @@ -99,6 +107,8 @@ To change these settings: # User accounts will be created automatically when authentication was successful. allow_single_sign_on: ["saml", "twitter"] + auto_link_ldap_user: true + # Locks down those users until they have been cleared by the admin (default: true). block_auto_created_users: true ``` diff --git a/doc/profile/2fa_u2f_authenticate.png b/doc/profile/2fa_u2f_authenticate.png Binary files differdeleted file mode 100644 index b224ab14195..00000000000 --- a/doc/profile/2fa_u2f_authenticate.png +++ /dev/null diff --git a/doc/profile/two_factor_authentication.md b/doc/profile/two_factor_authentication.md index 3f6dfe03d14..60918a0339c 100644 --- a/doc/profile/two_factor_authentication.md +++ b/doc/profile/two_factor_authentication.md @@ -1,143 +1 @@ -# Two-factor Authentication (2FA) - -Two-factor Authentication (2FA) provides an additional level of security to your -GitLab account. Once enabled, in addition to supplying your username and -password to login, you'll be prompted for a code generated by an application on -your phone. - -By enabling 2FA, the only way someone other than you can log into your account -is to know your username and password *and* have access to your phone. - -> **Note:** -When you enable 2FA, don't forget to back up your recovery codes. For your safety, if you -lose your codes for GitLab.com, we can't disable or recover them. - -In addition to a phone application, GitLab supports U2F (universal 2nd factor) devices as -the second factor of authentication. Once enabled, in addition to supplying your username and -password to login, you'll be prompted to activate your U2F device (usually by pressing -a button on it), and it will perform secure authentication on your behalf. - -> **Note:** Support for U2F devices was added in version 8.8 - -The U2F workflow is only supported by Google Chrome at this point, so we _strongly_ recommend -that you set up both methods of two-factor authentication, so you can still access your account -from other browsers. - -> **Note:** GitLab officially only supports [Yubikey] U2F devices. - -## Enabling 2FA - -### Enable 2FA via mobile application - -**In GitLab:** - -1. Log in to your GitLab account. -1. Go to your **Profile Settings**. -1. Go to **Account**. -1. Click **Enable Two-factor Authentication**. - - - -**On your phone:** - -1. Install a compatible application. We recommend [Google Authenticator] -\(proprietary\) or [FreeOTP] \(open source\). -1. In the application, add a new entry in one of two ways: - * Scan the code with your phone's camera to add the entry automatically. - * Enter the details provided to add the entry manually. - -**In GitLab:** - -1. Enter the six-digit pin number from the entry on your phone into the **Pin - code** field. -1. Click **Submit**. - -If the pin you entered was correct, you'll see a message indicating that -Two-Factor Authentication has been enabled, and you'll be presented with a list -of recovery codes. - -### Enable 2FA via U2F device - -**In GitLab:** - -1. Log in to your GitLab account. -1. Go to your **Profile Settings**. -1. Go to **Account**. -1. Click **Enable Two-Factor Authentication**. -1. Plug in your U2F device. -1. Click on **Setup New U2F Device**. -1. A light will start blinking on your device. Activate it by pressing its button. - -You will see a message indicating that your device was successfully set up. -Click on **Register U2F Device** to complete the process. - - - -## Recovery Codes - -Should you ever lose access to your phone, you can use one of the ten provided -backup codes to login to your account. We suggest copying or printing them for -storage in a safe place. **Each code can be used only once** to log in to your -account. - -If you lose the recovery codes or just want to generate new ones, you can do so -from the **Profile Settings** > **Account** page where you first enabled 2FA. - -> **Note:** Recovery codes are not generated for U2F devices. - -## Logging in with 2FA Enabled - -Logging in with 2FA enabled is only slightly different than a normal login. -Enter your username and password credentials as you normally would, and you'll -be presented with a second prompt, depending on which type of 2FA you've enabled. - -### Log in via mobile application - -Enter the pin from your phone's application or a recovery code to log in. - - - -### Log in via U2F device - -1. Click **Login via U2F Device** -1. A light will start blinking on your device. Activate it by pressing its button. - -You will see a message indicating that your device responded to the authentication request. -Click on **Authenticate via U2F Device** to complete the process. - - - -## Disabling 2FA - -1. Log in to your GitLab account. -1. Go to your **Profile Settings**. -1. Go to **Account**. -1. Click **Disable**, under **Two-Factor Authentication**. - -This will clear all your two-factor authentication registrations, including mobile -applications and U2F devices. - -## Personal access tokens - -When 2FA is enabled, you can no longer use your normal account password to -authenticate with Git over HTTPS on the command line, you must use a personal -access token instead. - -1. Log in to your GitLab account. -1. Go to your **Profile Settings**. -1. Go to **Access Tokens**. -1. Choose a name and expiry date for the token. -1. Click on **Create Personal Access Token**. -1. Save the personal access token somewhere safe. - -When using git over HTTPS on the command line, enter the personal access token -into the password field. - -## Note to GitLab administrators - -You need to take special care to that 2FA keeps working after -[restoring a GitLab backup](../raketasks/backup_restore.md). - -[Google Authenticator]: https://support.google.com/accounts/answer/1066447?hl=en -[FreeOTP]: https://fedorahosted.org/freeotp/ -[YubiKey]: https://www.yubico.com/products/yubikey-hardware/ +This document was moved to [user/profile/account](../user/profile/account/two_factor_authentication.md). diff --git a/doc/project_services/kubernetes.md b/doc/project_services/kubernetes.md index 59d5da702f8..99aa9e44bdb 100644 --- a/doc/project_services/kubernetes.md +++ b/doc/project_services/kubernetes.md @@ -8,7 +8,7 @@ the [configuration](#configuration) section. If you have a single cluster that you want to use for all your projects, you can pre-fill the settings page with a default template. To configure the -template, see the [Services Templates](services-templates.md) document. +template, see the [Services Templates](services_templates.md) document. ## Configuration diff --git a/doc/project_services/project_services.md b/doc/project_services/project_services.md index 0f398874b8f..547d855d777 100644 --- a/doc/project_services/project_services.md +++ b/doc/project_services/project_services.md @@ -46,6 +46,7 @@ further configuration instructions and details. Contributions are welcome. | [Mattermost slash commands](mattermost_slash_commands.md) | Mattermost chat and ChatOps slash commands | | [Mattermost Notifications](mattermost.md) | Receive event notifications in Mattermost | | [Slack Notifications](slack.md) | Receive event notifications in Slack | +| [Slack slash commands](slack_slash_commands.md) | Slack chat and ChatOps slash commands | | PivotalTracker | Project Management Software (Source Commits Endpoint) | | Pushover | Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop | | [Redmine](redmine.md) | Redmine issue tracker | diff --git a/doc/public_access/img/restrict_visibility_levels.png b/doc/public_access/img/restrict_visibility_levels.png Binary files differnew file mode 100644 index 00000000000..c7d4d87981f --- /dev/null +++ b/doc/public_access/img/restrict_visibility_levels.png diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index a3921f1b89f..e8f4c73120c 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -52,7 +52,7 @@ for anonymous users. The group page now has a visibility level icon. ## Visibility of users -The public page of a user, located at `/u/username`, is always visible whether +The public page of a user, located at `/username`, is always visible whether you are logged in or not. When visiting the public page of a user, you can only see the projects which @@ -60,10 +60,13 @@ you are privileged to. If the public level is restricted, user profiles are only visible to logged in users. - ## Restricting the use of public or internal projects In the Admin area under **Settings** (`/admin/application_settings`), you can restrict the use of visibility levels for users when they create a project or a -snippet. This is useful to prevent people exposing their repositories to public +snippet: + + + +This is useful to prevent people exposing their repositories to public by accident. The restricted visibility settings do not apply to admin users. diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index c44930a4ceb..ec13c2446ef 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -1,6 +1,6 @@ # System hooks -Your GitLab instance can perform HTTP POST requests on the following events: `project_create`, `project_destroy`, `project_rename`, `project_transfer`, `user_add_to_team`, `user_remove_from_team`, `user_create`, `user_destroy`, `key_create`, `key_destroy`, `group_create`, `group_destroy`, `user_add_to_group` and `user_remove_from_group`. +Your GitLab instance can perform HTTP POST requests on the following events: `project_create`, `project_destroy`, `project_rename`, `project_transfer`, `project_update`, `user_add_to_team`, `user_remove_from_team`, `user_create`, `user_destroy`, `key_create`, `key_destroy`, `group_create`, `group_destroy`, `user_add_to_group` and `user_remove_from_group`. System hooks can be used, e.g. for logging or changing information in a LDAP server. @@ -88,6 +88,23 @@ X-Gitlab-Event: System Hook } ``` +**Project updated:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_update", + "name": "StoreCloud", + "owner_email": "johnsmith@gmail.com", + "owner_name": "John Smith", + "path": "storecloud", + "path_with_namespace": "jsmith/storecloud", + "project_id": 74, + "project_visibility": "private", +} +``` + **New Team Member:** ```json diff --git a/doc/update/8.15-to-8.16.md b/doc/update/8.15-to-8.16.md index 3d68fe201a7..2695a16ac0b 100644 --- a/doc/update/8.15-to-8.16.md +++ b/doc/update/8.15-to-8.16.md @@ -36,7 +36,7 @@ Download and compile Ruby: ```bash mkdir /tmp/ruby && cd /tmp/ruby curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.3/ruby-2.3.3.tar.gz -echo 'a8db9ce7f9110320f33b8325200e3ecfbd2b534b ruby-2.3.3.tar.gz' | shasum -c - && tar xzf ruby-2.3.3.tar.gz +echo '1014ee699071aa2ddd501907d18cbe15399c997d ruby-2.3.3.tar.gz' | shasum -c - && tar xzf ruby-2.3.3.tar.gz cd ruby-2.3.3 ./configure --disable-install-rdoc make @@ -97,6 +97,8 @@ sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS_ENV=production ``` +**MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). + ### 6. Update gitlab-workhorse Install and compile gitlab-workhorse. This requires diff --git a/doc/user/account/security.md b/doc/user/account/security.md index 816094bf8d2..9336dee7451 100644 --- a/doc/user/account/security.md +++ b/doc/user/account/security.md @@ -1,3 +1 @@ -# Account Security - -- [Two-Factor Authentication](two_factor_authentication.md) +This document was moved to [profile](../profile/index.md#security). diff --git a/doc/user/account/two_factor_authentication.md b/doc/user/account/two_factor_authentication.md index 881358ed94d..ea2c8307860 100644 --- a/doc/user/account/two_factor_authentication.md +++ b/doc/user/account/two_factor_authentication.md @@ -1,68 +1 @@ -# Two-Factor Authentication - -## Recovery options - -If you lose your code generation device (such as your mobile phone) and you need -to disable two-factor authentication on your account, you have several options. - -### Use a saved recovery code - -When you enabled two-factor authentication for your account, a series of -recovery codes were generated. If you saved those codes somewhere safe, you -may use one to sign in. - -First, enter your username/email and password on the GitLab sign in page. When -prompted for a two-factor code, enter one of the recovery codes you saved -previously. - -> **Note:** Once a particular recovery code has been used, it cannot be used again. - You may still use the other saved recovery codes at a later time. - -### Generate new recovery codes using SSH - -It's not uncommon for users to forget to save the recovery codes when enabling -two-factor authentication. If you have an SSH key added to your GitLab account, -you can generate a new set of recovery codes using SSH. - -Run `ssh git@gitlab.example.com 2fa_recovery_codes`. You will be prompted to -confirm that you wish to generate new codes. If you choose to continue, any -previously saved codes will be invalidated. - -```bash -$ ssh git@gitlab.example.com 2fa_recovery_codes -Are you sure you want to generate new two-factor recovery codes? -Any existing recovery codes you saved will be invalidated. (yes/no) -yes - -Your two-factor authentication recovery codes are: - -119135e5a3ebce8e -11f6v2a498810dcd -3924c7ab2089c902 -e79a3398bfe4f224 -34bd7b74adbc8861 -f061691d5107df1a -169bf32a18e63e7f -b510e7422e81c947 -20dbed24c5e74663 -df9d3b9403b9c9f0 - -During sign in, use one of the codes above when prompted for -your two-factor code. Then, visit your Profile Settings and add -a new device so you do not lose access to your account again. -``` - -Next, go to the GitLab sign in page and enter your username/email and password. -When prompted for a two-factor code, enter one of the recovery codes obtained -from the command line output. - -> **Note:** After signing in, you should immediately visit your **Profile Settings - -> Account** to set up two-factor authentication with a new device. - -### Ask a GitLab administrator to disable two-factor on your account - -If the above two methods are not possible, you may ask a GitLab global -administrator to disable two-factor authentication for your account. Please -be aware that this will temporarily leave your account in a less secure state. -You should sign in and re-enable two-factor authentication as soon as possible -after the administrator disables it. +This document was moved to [profile/account/two_factor_authentication](../profile/account/two_factor_authentication.md). diff --git a/doc/user/markdown.md b/doc/user/markdown.md index f6484688721..008872b59a7 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -300,6 +300,20 @@ You can add task lists to issues, merge requests and comments. To create a task - [x] Sub-task 2 - [ ] Sub-task 3 +Tasks formatted as ordered lists are supported as well: + +```no-highlight +1. [x] Completed task +1. [ ] Incomplete task + 1. [ ] Sub-task 1 + 1. [x] Sub-task 2 +``` + +1. [x] Completed task +1. [ ] Incomplete task + 1. [ ] Sub-task 1 + 1. [x] Sub-task 2 + Task lists can only be created in descriptions, not in titles. Task item state can be managed by editing the description's Markdown or by toggling the rendered check boxes. ### Videos @@ -650,7 +664,7 @@ This line is separated from the one above by two newlines, so it will be a *sepa This line is also a separate paragraph, but... This line is only separated by a single newline, so it's a separate line in the *same paragraph*. -This line is also a separate paragraph, and... +This line is also a separate paragraph, and... This line is on its own line, because the previous line ends with two spaces. ``` @@ -662,7 +676,7 @@ This line is separated from the one above by two newlines, so it will be a *sepa This line is also begins a separate paragraph, but... This line is only separated by a single newline, so it's a separate line in the *same paragraph*. -This line is also a separate paragraph, and... +This line is also a separate paragraph, and... This line is on its own line, because the previous line ends with two spaces. @@ -800,4 +814,4 @@ A link starting with a `/` is relative to the wiki root. [redcarpet]: https://github.com/vmg/redcarpet "Redcarpet website" [katex]: https://github.com/Khan/KaTeX "KaTeX website" [katex-subset]: https://github.com/Khan/KaTeX/wiki/Function-Support-in-KaTeX "Macros supported by KaTeX" -[asciidoctor-manual]: http://asciidoctor.org/docs/user-manual/#activating-stem-support "Asciidoctor user manual"
\ No newline at end of file +[asciidoctor-manual]: http://asciidoctor.org/docs/user-manual/#activating-stem-support "Asciidoctor user manual" diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 5ada8748d85..678fc3ffd1f 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -19,10 +19,12 @@ The following table depicts the various user permission levels in a project. | Action | Guest | Reporter | Developer | Master | Owner | |---------------------------------------|---------|------------|-------------|----------|--------| | Create new issue | ✓ | ✓ | ✓ | ✓ | ✓ | +| Create confidential issue | ✓ | ✓ | ✓ | ✓ | ✓ | +| View confidential issues | (✓) [^1] | ✓ | ✓ | ✓ | ✓ | | Leave comments | ✓ | ✓ | ✓ | ✓ | ✓ | -| See a list of builds | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| See a build log | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| Download and browse build artifacts | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | +| See a list of builds | ✓ [^2] | ✓ | ✓ | ✓ | ✓ | +| See a build log | ✓ [^2] | ✓ | ✓ | ✓ | ✓ | +| Download and browse build artifacts | ✓ [^2] | ✓ | ✓ | ✓ | ✓ | | View wiki pages | ✓ | ✓ | ✓ | ✓ | ✓ | | Pull project code | | ✓ | ✓ | ✓ | ✓ | | Download project | | ✓ | ✓ | ✓ | ✓ | @@ -63,11 +65,8 @@ The following table depicts the various user permission levels in a project. | Switch visibility level | | | | | ✓ | | Transfer project to another namespace | | | | | ✓ | | Remove project | | | | | ✓ | -| Force push to protected branches [^2] | | | | | | -| Remove protected branches [^2] | | | | | | - -[^1]: If **Public pipelines** is enabled in **Project Settings > CI/CD Pipelines** -[^2]: Not allowed for Guest, Reporter, Developer, Master, or Owner +| Force push to protected branches [^3] | | | | | | +| Remove protected branches [^3] | | | | | | ## Group @@ -156,17 +155,20 @@ users: | Run CI build | | ✓ | ✓ | ✓ | | Clone source and LFS from current project | | ✓ | ✓ | ✓ | | Clone source and LFS from public projects | | ✓ | ✓ | ✓ | -| Clone source and LFS from internal projects | | ✓ [^3] | ✓ [^3] | ✓ | -| Clone source and LFS from private projects | | ✓ [^4] | ✓ [^4] | ✓ [^4] | +| Clone source and LFS from internal projects | | ✓ [^4] | ✓ [^4] | ✓ | +| Clone source and LFS from private projects | | ✓ [^5] | ✓ [^5] | ✓ [^5] | | Push source and LFS | | | | | | Pull container images from current project | | ✓ | ✓ | ✓ | | Pull container images from public projects | | ✓ | ✓ | ✓ | -| Pull container images from internal projects| | ✓ [^3] | ✓ [^3] | ✓ | -| Pull container images from private projects | | ✓ [^4] | ✓ [^4] | ✓ [^4] | +| Pull container images from internal projects| | ✓ [^4] | ✓ [^4] | ✓ | +| Pull container images from private projects | | ✓ [^5] | ✓ [^5] | ✓ [^5] | | Push container images to current project | | ✓ | ✓ | ✓ | | Push container images to other projects | | | | | -[^3]: Only if user is not external one. -[^4]: Only if user is a member of the project. +[^1]: Guest users can only view the confidential issues they created themselves +[^2]: If **Public pipelines** is enabled in **Project Settings > CI/CD Pipelines** +[^3]: Not allowed for Guest, Reporter, Developer, Master, or Owner +[^4]: Only if user is not external one. +[^5]: Only if user is a member of the project. [ce-18994]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18994 [new-mod]: project/new_ci_build_permissions_model.md diff --git a/doc/profile/2fa.png b/doc/user/profile/account/img/2fa.png Binary files differindex bb464efa685..bb464efa685 100644 --- a/doc/profile/2fa.png +++ b/doc/user/profile/account/img/2fa.png diff --git a/doc/profile/2fa_auth.png b/doc/user/profile/account/img/2fa_auth.png Binary files differindex 0caaed10805..0caaed10805 100644 --- a/doc/profile/2fa_auth.png +++ b/doc/user/profile/account/img/2fa_auth.png diff --git a/doc/user/profile/account/img/2fa_u2f_authenticate.png b/doc/user/profile/account/img/2fa_u2f_authenticate.png Binary files differnew file mode 100644 index 00000000000..ff2e936764d --- /dev/null +++ b/doc/user/profile/account/img/2fa_u2f_authenticate.png diff --git a/doc/profile/2fa_u2f_register.png b/doc/user/profile/account/img/2fa_u2f_register.png Binary files differindex 1cc142aa851..1cc142aa851 100644 --- a/doc/profile/2fa_u2f_register.png +++ b/doc/user/profile/account/img/2fa_u2f_register.png diff --git a/doc/user/profile/account/index.md b/doc/user/profile/account/index.md new file mode 100644 index 00000000000..764354e1e96 --- /dev/null +++ b/doc/user/profile/account/index.md @@ -0,0 +1,5 @@ +# Profile settings + +## Account + +Set up [two-factor authentication](two_factor_authentication.md). diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md new file mode 100644 index 00000000000..cc688a7f99c --- /dev/null +++ b/doc/user/profile/account/two_factor_authentication.md @@ -0,0 +1,217 @@ +# Two-Factor Authentication + +Two-factor Authentication (2FA) provides an additional level of security to your +GitLab account. Once enabled, in addition to supplying your username and +password to login, you'll be prompted for a code generated by an application on +your phone. + +By enabling 2FA, the only way someone other than you can log into your account +is to know your username and password *and* have access to your phone. + +## Overview + +> **Note:** +When you enable 2FA, don't forget to back up your recovery codes. + +In addition to a phone application, GitLab supports U2F (universal 2nd factor) devices as +the second factor of authentication. Once enabled, in addition to supplying your username and +password to login, you'll be prompted to activate your U2F device (usually by pressing +a button on it), and it will perform secure authentication on your behalf. + +The U2F workflow is only supported by Google Chrome at this point, so we _strongly_ recommend +that you set up both methods of two-factor authentication, so you can still access your account +from other browsers. + +## Enabling 2FA + +There are two ways to enable two-factor authentication: via a mobile application +or a U2F device. + +### Enable 2FA via mobile application + +**In GitLab:** + +1. Log in to your GitLab account. +1. Go to your **Profile Settings**. +1. Go to **Account**. +1. Click **Enable Two-factor Authentication**. + + + +**On your phone:** + +1. Install a compatible application. We recommend [Google Authenticator] +\(proprietary\) or [FreeOTP] \(open source\). +1. In the application, add a new entry in one of two ways: + * Scan the code with your phone's camera to add the entry automatically. + * Enter the details provided to add the entry manually. + +**In GitLab:** + +1. Enter the six-digit pin number from the entry on your phone into the **Pin + code** field. +1. Click **Submit**. + +If the pin you entered was correct, you'll see a message indicating that +Two-Factor Authentication has been enabled, and you'll be presented with a list +of recovery codes. + +### Enable 2FA via U2F device + +> **Notes:** +- GitLab officially only supports [Yubikey] U2F devices. +- Support for U2F devices was added in GitLab 8.8. + +**In GitLab:** + +1. Log in to your GitLab account. +1. Go to your **Profile Settings**. +1. Go to **Account**. +1. Click **Enable Two-Factor Authentication**. +1. Plug in your U2F device. +1. Click on **Setup New U2F Device**. +1. A light will start blinking on your device. Activate it by pressing its button. + +You will see a message indicating that your device was successfully set up. +Click on **Register U2F Device** to complete the process. + + + +## Recovery Codes + +> **Note:** +Recovery codes are not generated for U2F devices. + +Should you ever lose access to your phone, you can use one of the ten provided +backup codes to login to your account. We suggest copying or printing them for +storage in a safe place. **Each code can be used only once** to log in to your +account. + +If you lose the recovery codes or just want to generate new ones, you can do so +from the **Profile settings ➔ Account** page where you first enabled 2FA. + +## Logging in with 2FA Enabled + +Logging in with 2FA enabled is only slightly different than a normal login. +Enter your username and password credentials as you normally would, and you'll +be presented with a second prompt, depending on which type of 2FA you've enabled. + +### Log in via mobile application + +Enter the pin from your phone's application or a recovery code to log in. + + + +### Log in via U2F device + +1. Click **Login via U2F Device** +1. A light will start blinking on your device. Activate it by pressing its button. + +You will see a message indicating that your device responded to the authentication request. +Click on **Authenticate via U2F Device** to complete the process. + + + +## Disabling 2FA + +1. Log in to your GitLab account. +1. Go to your **Profile Settings**. +1. Go to **Account**. +1. Click **Disable**, under **Two-Factor Authentication**. + +This will clear all your two-factor authentication registrations, including mobile +applications and U2F devices. + +## Personal access tokens + +When 2FA is enabled, you can no longer use your normal account password to +authenticate with Git over HTTPS on the command line, you must use a personal +access token instead. + +1. Log in to your GitLab account. +1. Go to your **Profile Settings**. +1. Go to **Access Tokens**. +1. Choose a name and expiry date for the token. +1. Click on **Create Personal Access Token**. +1. Save the personal access token somewhere safe. + +When using Git over HTTPS on the command line, enter the personal access token +into the password field. + +## Recovery options + +To disable two-factor authentication on your account (for example, if you +have lost your code generation device) you can: +* [Use a saved recovery code](#use-a-saved-recovery-code) +* [Generate new recovery codes using SSH](#generate-new-recovery-codes-using-SSH) +* [Ask a GitLab administrator to disable two-factor authentication on your account](#ask-a-gitlab-administrator-to-disable-two-factor-authentication-on-your-account) + +### Use a saved recovery code + +Enabling two-factor authentication for your account generated several recovery +codes. If you saved these codes, you can use one of them to sign in. + +To use a recovery code, enter your username/email and password on the GitLab +sign-in page. When prompted for a two-factor code, enter the recovery code. + +> **Note:** Once you use a recovery code, you cannot re-use it. You can still + use the other recovery codes you saved. + +### Generate new recovery codes using SSH + +Users often forget to save their recovery codes when enabling two-factor +authentication. If an SSH key is added to your GitLab account, you can generate +a new set of recovery codes with SSH. + +1. Run `ssh git@gitlab.example.com 2fa_recovery_codes`. +2. You are prompted to confirm that you want to generate new codes. Continuing this process invalidates previously saved codes. + ``` + bash + $ ssh git@gitlab.example.com 2fa_recovery_codes + Are you sure you want to generate new two-factor recovery codes? + Any existing recovery codes you saved will be invalidated. (yes/no) + + yes + + Your two-factor authentication recovery codes are: + + 119135e5a3ebce8e + 11f6v2a498810dcd + 3924c7ab2089c902 + e79a3398bfe4f224 + 34bd7b74adbc8861 + f061691d5107df1a + 169bf32a18e63e7f + b510e7422e81c947 + 20dbed24c5e74663 + df9d3b9403b9c9f0 + + During sign in, use one of the codes above when prompted for your + two-factor code. Then, visit your Profile Settings and add a new device + so you do not lose access to your account again. + ``` +3. Go to the GitLab sign-in page and enter your username/email and password. When prompted for a two-factor code, enter one of the recovery codes obtained +from the command-line output. + +> **Note:** After signing in, visit your **Profile Settings -> Account** immediately to set up two-factor authentication with a new + device. + +### Ask a GitLab administrator to disable two-factor authentication on your account + +If you cannot use a saved recovery code or generate new recovery codes, ask a +GitLab global administrator to disable two-factor authentication for your +account. This will temporarily leave your account in a less secure state. +Sign in and re-enable two-factor authentication as soon as possible. + +## Note to GitLab administrators + +- You need to take special care to that 2FA keeps working after +[restoring a GitLab backup](../raketasks/backup_restore.md). + +- To ensure 2FA authorizes correctly with TOTP server, you may want to ensure +your GitLab server's time is synchronized via a service like NTP. Otherwise, +you may have cases where authorization always fails because of time differences. + +[Google Authenticator]: https://support.google.com/accounts/answer/1066447?hl=en +[FreeOTP]: https://fedorahosted.org/freeotp/ +[YubiKey]: https://www.yubico.com/products/yubikey-hardware/ diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md new file mode 100644 index 00000000000..1760b182114 --- /dev/null +++ b/doc/user/project/issues/confidential_issues.md @@ -0,0 +1,68 @@ +# Confidential issues + +> [Introduced][ce-3282] in GitLab 8.6. + +Confidential issues are issues visible only to members of a project with +[sufficient permissions](#permissions-and-access-to-confidential-issues). +Confidential issues can be used by open source projects and companies alike to +keep security vulnerabilities private or prevent surprises from leaking out. + +## Making an issue confidential + +You can make an issue confidential either by creating a new issue or editing +an existing one. + +When you create a new issue, a checkbox right below the text area is available +to mark the issue as confidential. Check that box and hit the **Submit issue** +button to create the issue. For existing issues, edit them, check the +confidential checkbox and hit **Save changes**. + + + +## Making an issue non-confidential + +To make an issue non-confidential, all you have to do is edit it and unmark +the confidential checkbox. Once you save the issue, it will gain the default +visibility level you have chosen for your project. + +Every change from regular to confidential and vice versa, is indicated by a +system note in the issue's comments. + + + +## Indications of a confidential issue + +>**Note:** If you don't have [enough permissions](#permissions-and-access-to-confidential-issues), +you won't be able to see the confidential issues at all. + +There are a few things that visually separate a confidential issue from a +regular one. In the issues index page view, you can see the eye-slash icon +next to the issues that are marked as confidential. + + + +--- + +Likewise, while inside the issue, you can see the eye-slash icon right next to +the issue number, but there is also an indicator in the comment area that the +issue you are commenting on is confidential. + + + +## Permissions and access to confidential issues + +There are two kinds of level access for confidential issues. The general rule +is that confidential issues are visible only to members of a project with at +least [Reporter access][permissions]. However, a guest user can also create +confidential issues, but can only view the ones that they created themselves. + +Confidential issues are also hidden in search results for unprivileged users. +For example, here's what a user with Master and Guest access sees in the +project's search results respectively. + +| Master access | Guest access | +| :-----------: | :----------: | +|  |  | + +[permissions]: ../../permissions.md#project +[ce-3282]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3282 diff --git a/doc/user/project/issues/due_dates.md b/doc/user/project/issues/due_dates.md new file mode 100644 index 00000000000..b516d47ffa3 --- /dev/null +++ b/doc/user/project/issues/due_dates.md @@ -0,0 +1,37 @@ +# Due dates + +> [Introduced][ce-3614] in GitLab 8.7. + +Due dates can be used in issues to keep track of deadlines and make sure +features are shipped on time. Due dates require at least [Reporter permissions][permissions] +to be able to edit them. On the contrary, they can be seen by everybody. + +## Setting a due date + +When creating or editing an issue, you can see the due date field from where +a calendar will appear to help you choose the date you want. To remove it, +select the date text and delete it. + + + +A quicker way to set a due date is via the issue sidebar. Simply expand the +sidebar and select **Edit** to pick a due date or remove the existing one. +Changes are saved immediately. + + + +## Making use of due dates + +Issues that have a due date can be distinctively seen in the issues index page +with a calendar icon next to them. Issues where the date is past due will have +the icon and the date colored red. You can sort issues by those that are +_Due soon_ or _Due later_ from the dropdown menu in the right. + + + +Due dates also appear in your [todos list](../../../workflow/todos.md). + + + +[ce-3614]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3614 +[permissions]: ../../permissions.md#project diff --git a/doc/user/project/issues/img/confidential_issues_create.png b/doc/user/project/issues/img/confidential_issues_create.png Binary files differnew file mode 100644 index 00000000000..d259255599d --- /dev/null +++ b/doc/user/project/issues/img/confidential_issues_create.png diff --git a/doc/user/project/issues/img/confidential_issues_index_page.png b/doc/user/project/issues/img/confidential_issues_index_page.png Binary files differnew file mode 100644 index 00000000000..042461e2451 --- /dev/null +++ b/doc/user/project/issues/img/confidential_issues_index_page.png diff --git a/doc/user/project/issues/img/confidential_issues_issue_page.png b/doc/user/project/issues/img/confidential_issues_issue_page.png Binary files differnew file mode 100644 index 00000000000..b3568e9303a --- /dev/null +++ b/doc/user/project/issues/img/confidential_issues_issue_page.png diff --git a/doc/user/project/issues/img/confidential_issues_search_guest.png b/doc/user/project/issues/img/confidential_issues_search_guest.png Binary files differnew file mode 100644 index 00000000000..b85de90b4d5 --- /dev/null +++ b/doc/user/project/issues/img/confidential_issues_search_guest.png diff --git a/doc/user/project/issues/img/confidential_issues_search_master.png b/doc/user/project/issues/img/confidential_issues_search_master.png Binary files differnew file mode 100644 index 00000000000..bf2b9428875 --- /dev/null +++ b/doc/user/project/issues/img/confidential_issues_search_master.png diff --git a/doc/user/project/issues/img/confidential_issues_system_notes.png b/doc/user/project/issues/img/confidential_issues_system_notes.png Binary files differnew file mode 100644 index 00000000000..4005f9350f7 --- /dev/null +++ b/doc/user/project/issues/img/confidential_issues_system_notes.png diff --git a/doc/user/project/issues/img/due_dates_create.png b/doc/user/project/issues/img/due_dates_create.png Binary files differnew file mode 100644 index 00000000000..d2fe1172bab --- /dev/null +++ b/doc/user/project/issues/img/due_dates_create.png diff --git a/doc/user/project/issues/img/due_dates_edit_sidebar.png b/doc/user/project/issues/img/due_dates_edit_sidebar.png Binary files differnew file mode 100644 index 00000000000..6b37150e7db --- /dev/null +++ b/doc/user/project/issues/img/due_dates_edit_sidebar.png diff --git a/doc/user/project/issues/img/due_dates_issues_index_page.png b/doc/user/project/issues/img/due_dates_issues_index_page.png Binary files differnew file mode 100644 index 00000000000..defcd5eca39 --- /dev/null +++ b/doc/user/project/issues/img/due_dates_issues_index_page.png diff --git a/doc/user/project/issues/img/due_dates_todos.png b/doc/user/project/issues/img/due_dates_todos.png Binary files differnew file mode 100644 index 00000000000..92c9fd4021b --- /dev/null +++ b/doc/user/project/issues/img/due_dates_todos.png diff --git a/doc/user/project/merge_requests/img/merge_conflict_editor.png b/doc/user/project/merge_requests/img/merge_conflict_editor.png Binary files differnew file mode 100644 index 00000000000..6660920c191 --- /dev/null +++ b/doc/user/project/merge_requests/img/merge_conflict_editor.png diff --git a/doc/user/project/merge_requests/resolve_conflicts.md b/doc/user/project/merge_requests/resolve_conflicts.md index 4d7225bd820..68c49054e47 100644 --- a/doc/user/project/merge_requests/resolve_conflicts.md +++ b/doc/user/project/merge_requests/resolve_conflicts.md @@ -21,6 +21,18 @@ request into the source branch, resolving the conflicts using the options chosen. If the source branch is `feature` and the target branch is `master`, this is similar to performing `git checkout feature; git merge master` locally. +## Merge conflict editor + +> Introduced in GitLab 8.13. + +The merge conflict resolution editor allows for more complex merge conflicts, +which require the user to manually modify a file in order to resolve a conflict, +to be solved right form the GitLab interface. Use the **Edit inline** button +to open the editor. Once you're sure about your changes, hit the +**Commit conflict resolution** button. + + + ## Conflicts available for resolution GitLab allows resolving conflicts in a file where all of the below are true: diff --git a/doc/workflow/README.md b/doc/workflow/README.md index b317bd79ded..0b6f00c6aa4 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -7,6 +7,10 @@ - [Feature branch workflow](workflow.md) - [GitLab Flow](gitlab_flow.md) - [Groups](groups.md) +- Issues - The GitLab Issue Tracker is an advanced and complete tool for + tracking the evolution of a new idea or the process of solving a problem. + - [Confidential issues](../user/project/issues/confidential_issues.md) + - [Due date for issues](../user/project/issues/due_dates.md) - [Issue Board](../user/project/issue_board.md) - [Keyboard shortcuts](shortcuts.md) - [File finder](file_finder.md) |
