diff options
Diffstat (limited to 'doc')
47 files changed, 1964 insertions, 93 deletions
diff --git a/doc/README.md b/doc/README.md index d2660930653..e358da1c424 100644 --- a/doc/README.md +++ b/doc/README.md @@ -31,6 +31,7 @@ - [Environment Variables](administration/environment_variables.md) to configure GitLab. - [Operations](operations/README.md) Keeping GitLab up and running - [Raketasks](raketasks/README.md) Backups, maintenance, automatic webhook setup and the importing of projects. +- [Repository checks](administration/repository_checks.md) Periodic Git repository checks - [Security](security/README.md) Learn what you can do to further secure your GitLab instance. - [System hooks](system_hooks/system_hooks.md) Notifications when users, projects and keys are changed. - [Update](update/README.md) Update guides to upgrade your installation. @@ -40,6 +41,8 @@ - [Git LFS configuration](workflow/lfs/lfs_administration.md) - [Housekeeping](administration/housekeeping.md) Keep your Git repository tidy and fast. - [GitLab Performance Monitoring](monitoring/performance/introduction.md) Configure GitLab and InfluxDB for measuring performance metrics +- [Sidekiq Troubleshooting](administration/troubleshooting/sidekiq.md) Debug when Sidekiq appears hung and is not processing jobs +- [High Availability](administration/high_availability/README.md) Configure multiple servers for scaling or high availability ## Contributor documentation diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md new file mode 100644 index 00000000000..43d85ffb775 --- /dev/null +++ b/doc/administration/high_availability/README.md @@ -0,0 +1,35 @@ +# High Availability + +GitLab supports several different types of clustering and high-availability. +The solution you choose will be based on the level of scalability and +availability you require. The easiest solutions are scalable, but not necessarily +highly available. + +## Architecture + +### Active/Passive + +For pure high-availability/failover with no scaling you can use an +active/passive configuration. This utilizes DRBD (Distributed Replicated +Block Device) to keep all data in sync. DRBD requires a low latency link to +remain in sync. It is not advisable to attempt to run DRBD between data centers +or in different cloud availability zones. + +Components/Servers Required: + +- 2 servers/virtual machines (one active/one passive) + +### Active/Active + +This architecture scales easily because all application servers handle +user requests simultaneously. The database, Redis, and GitLab application are +all deployed on separate servers. The configuration is **only** highly-available +if the database, Redis and storage are also configured as such. + +**Steps to configure active/active:** + +1. [Configure the database](database.md) +1. [Configure Redis](redis.md) +1. [Configure NFS](nfs.md) +1. [Configure the GitLab application servers](gitlab.md) +1. [Configure the load balancers](load_balancer.md) diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md new file mode 100644 index 00000000000..538dada1bae --- /dev/null +++ b/doc/administration/high_availability/database.md @@ -0,0 +1,116 @@ +# Configuring a Database for GitLab HA + +You can choose to install and manage a database server (PostgreSQL/MySQL) +yourself, or you can use GitLab Omnibus packages to help. GitLab recommends +PostgreSQL. This is the database that will be installed if you use the +Omnibus package to manage your database. + +## Configure your own database server + +If you're hosting GitLab on a cloud provider, you can optionally use a +managed service for PostgreSQL. For example, AWS offers a managed Relational +Database Service (RDS) that runs PostgreSQL. + +If you use a cloud-managed service, or provide your own PostgreSQL: + +1. Set up a `gitlab` username with a password of your choice. The `gitlab` user + needs privileges to create the `gitlabhq_production` database. +1. Configure the GitLab application servers with the appropriate details. + This step is covered in [Configuring GitLab for HA](gitlab.md) + +## Configure using Omnibus + +1. Download/install GitLab Omnibus using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other + steps on the download page. +1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. + Be sure to change the `external_url` to match your eventual GitLab front-end + URL. + + ```ruby + external_url 'https://gitlab.example.com' + + # Disable all components except PostgreSQL + postgresql['enable'] = true + bootstrap['enable'] = false + nginx['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + redis['enable'] = false + gitlab_workhorse['enable'] = false + mailroom['enable'] = false + + # PostgreSQL configuration + postgresql['sql_password'] = 'DB password' + postgresql['md5_auth_cidr_addresses'] = ['0.0.0.0/0'] + postgresql['listen_address'] = '0.0.0.0' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. + + > **Note**: This `reconfigure` step will result in some errors. + That's OK - don't be alarmed. + +1. Open a database prompt: + + ``` + su - gitlab-psql + /bin/bash + psql -h /var/opt/gitlab/postgresql -d template1 + + # Output: + + psql (9.2.15) + Type "help" for help. + + template1=# + ``` + +1. Run the following command at the database prompt and you will be asked to + enter the new password for the PostgreSQL superuser. + + ``` + \password + + # Output: + + Enter new password: + Enter it again: + ``` + +1. Similarly, set the password for the `gitlab` database user. Use the same + password that you specified in the `/etc/gitlab/gitlab.rb` file for + `postgresql['sql_password']`. + + ``` + \password gitlab + + # Output: + + Enter new password: + Enter it again: + ``` + +1. Enable the `pg_trgm` extension: + ``` + CREATE EXTENSION pg_trgm; + + # Output: + + CREATE EXTENSION + ``` +1. Exit the database prompt by typing `\q` and Enter. +1. Exit the `gitlab-psql` user by running `exit` twice. +1. Run `sudo gitlab-ctl reconfigure` a final time. +1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations + from running on upgrade. Only the primary GitLab application server should + handle migrations. + +--- + +Read more on high-availability configuration: + +1. [Configure Redis](redis.md) +1. [Configure NFS](nfs.md) +1. [Configure the GitLab application servers](gitlab.md) +1. [Configure the load balancers](load_balancer.md) diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md new file mode 100644 index 00000000000..8a881ce8863 --- /dev/null +++ b/doc/administration/high_availability/gitlab.md @@ -0,0 +1,131 @@ +# Configuring GitLab for HA + +Assuming you have already configured a database, Redis, and NFS, you can +configure the GitLab application server(s) now. Complete the steps below +for each GitLab application server in your environment. + +> **Note:** There is some additional configuration near the bottom for + secondary GitLab application servers. It's important to read and understand + these additional steps before proceeding with GitLab installation. + +1. If necessary, install the NFS client utility packages using the following + commands: + + ``` + # Ubuntu/Debian + apt-get install nfs-common + + # CentOS/Red Hat + yum install nfs-utils nfs-utils-lib + ``` + +1. Specify the necessary NFS shares. Mounts are specified in + `/etc/fstab`. The exact contents of `/etc/fstab` will depend on how you chose + to configure your NFS server. See [NFS documentation](nfs.md) for the various + options. Here is an example snippet to add to `/etc/fstab`: + + ``` + 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 + 10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 + 10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 + ``` + +1. Create the shared directories. These may be different depending on your NFS + mount locations. + + ``` + mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data + ``` + +1. Download/install GitLab Omnibus using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other + steps on the download page. +1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. + Be sure to change the `external_url` to match your eventual GitLab front-end + URL. Depending your the NFS configuration, you may need to change some GitLab + data locations. See [NFS documentation](nfs.md) for `/etc/gitlab/gitlab.rb` + configuration values for various scenarios. The example below assumes you've + added NFS mounts in the default data locations. + + ```ruby + external_url 'https://gitlab.example.com' + + # Prevent GitLab from starting if NFS data mounts are not available + high_availability['mountpoint'] = '/var/opt/gitlab/git-data' + + # Disable components that will not be on the GitLab application server + postgresql['enable'] = false + redis['enable'] = false + + # PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + + # Redis connection details + gitlab_rails['redis_port'] = '6379' + gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server + gitlab_rails['redis_password'] = 'Redis Password' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +## Primary GitLab application server + +As a final step, run the setup rake task on the first GitLab application server. +It is not necessary to run this on additional application servers. + +1. Initialize the database by running `sudo gitlab-rake gitlab:setup`. + +> **WARNING:** Only run this setup task on **NEW** GitLab instances because it + will wipe any existing data. + +> **Note:** When you specify `https` in the `external_url`, as in the example + above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If + certificates are not present, Nginx will fail to start. See + [Nginx documentation](http://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) + for more information. + +## Additional configuration for secondary GitLab application servers + +Secondary GitLab servers (servers configured **after** the first GitLab server) +need some additional configuration. + +1. Configure shared secrets. These values can be obtained from the primary + GitLab server in `/etc/gitlab/gitlab-secrets.json`. Add these to + `/etc/gitlab/gitlab.rb` **prior to** running the first `reconfigure` in + the steps above. + + ```ruby + gitlab_shell['secret_token'] = 'fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860' + gitlab_rails['secret_token'] = 'b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa' + gitlab_ci['secret_key_base'] = '6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d' + gitlab_ci['db_key_base'] = 'bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964' + ``` + +1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations + from running on upgrade. Only the primary GitLab application server should + handle migrations. + +## Troubleshooting + +- `mount: wrong fs type, bad option, bad superblock on` + +You have not installed the necessary NFS client utilities. See step 1 above. + +- `mount: mount point /var/opt/gitlab/... does not exist` + +This particular directory does not exist on the NFS server. Ensure +the share is exported and exists on the NFS server and try to remount. + +--- + +Read more on high-availability configuration: + +1. [Configure the database](database.md) +1. [Configure Redis](redis.md) +1. [Configure NFS](nfs.md) +1. [Configure the load balancers](load_balancer.md) diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md new file mode 100644 index 00000000000..b1fe34ed9a1 --- /dev/null +++ b/doc/administration/high_availability/load_balancer.md @@ -0,0 +1,63 @@ +# Load Balancer for GitLab HA + +In an active/active GitLab configuration, you will need a load balancer to route +traffic to the application servers. The specifics on which load balancer to use +or the exact configuration is beyond the scope of GitLab documentation. We hope +that if you're managing HA systems like GitLab you have a load balancer of +choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, +and Citrix Net Scaler. This documentation will outline what ports and protocols +you need to use with GitLab. + +## Basic ports + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | -------- | +| 80 | 80 | HTTP | +| 443 | 443 | HTTPS [^1] | +| 22 | 22 | TCP | + +## GitLab Pages Ports + +If you're using GitLab Pages you will need some additional port configurations. +GitLab Pages requires a separate VIP. Configure DNS to point the +`pages_external_url` from `/etc/gitlab/gitlab.rb` at the new VIP. See the +[GitLab Pages documentation][gitlab-pages] for more information. + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | -------- | +| 80 | Varies [^2] | HTTP | +| 443 | Varies [^2] | TCP [^3] | + +## Alternate SSH Port + +Some organizations have policies against opening SSH port 22. In this case, +it may be helpful to configure an alternate SSH hostname that allows users +to use SSH on port 443. An alternate SSH hostname will require a new VIP +compared to the other GitLab HTTP configuration above. + +Configure DNS for an alternate SSH hostname such as altssh.gitlab.example.com. + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | -------- | +| 443 | 22 | TCP | + +--- + +Read more on high-availability configuration: + +1. [Configure the database](database.md) +1. [Configure Redis](redis.md) +1. [Configure NFS](nfs.md) +1. [Configure the GitLab application servers](gitlab.md) + +[^1]: When using HTTPS protocol for port 443, you will need to add an SSL + certificate to the load balancers. If you wish to terminate SSL at the + GitLab application server instead, use TCP protocol. +[^2]: The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + setting. See [GitLab Pages documentation][gitlab-pages] for more details. +[^3]: Port 443 for GitLab Pages should always use the TCP protocol. Users can + configure custom domains with custom SSL, which would not be possible + if SSL was terminated at the load balancer. + +[gitlab-pages]: http://doc.gitlab.com/ee/pages/administration.html diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md new file mode 100644 index 00000000000..e4e124e200a --- /dev/null +++ b/doc/administration/high_availability/nfs.md @@ -0,0 +1,116 @@ +# NFS + +## Required NFS Server features + +**File locking**: GitLab **requires** file locking which is only supported +natively in NFS version 4. NFSv3 also supports locking as long as +Linux Kernel 2.6.5+ is used. We recommend using version 4 and do not +specifically test NFSv3. + +**no_root_squash**: NFS normally changes the `root` user to `nobody`. This is +a good security measure when NFS shares will be accessed by many different +users. However, in this case only GitLab will use the NFS share so it +is safe. GitLab requires the `no_root_squash` setting because we need to +manage file permissions automatically. Without the setting you will receive +errors when the Omnibus package tries to alter permissions. Note that GitLab +and other bundled components do **not** run as `root` but as non-privileged +users. The requirement for `no_root_squash` is to allow the Omnibus package to +set ownership and permissions on files, as needed. + +### Recommended options + +When you define your NFS exports, we recommend you also add the following +options: + +- `sync` - Force synchronous behavior. Default is asynchronous and under certain + circumstances it could lead to data loss if a failure occurs before data has + synced. + +## Client mount options + +Below is an example of an NFS mount point we use on GitLab.com: + +``` +10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nobootwait,lookupcache=positive 0 2 +``` + +Notice several options that you should consider using: + +| Setting | Description | +| ------- | ----------- | +| `nobootwait` | Don't halt boot process waiting for this mount to become available +| `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. + +## Mount locations + +When using default Omnibus configuration you will need to share 5 data locations +between all GitLab cluster nodes. No other locations should be shared. The +following are the 5 locations you need to mount: + +| Location | Description | +| -------- | ----------- | +| `/var/opt/gitlab/git-data` | Git repository data. This will account for a large portion of your data +| `/var/opt/gitlab/.ssh` | SSH `authorized_keys` file and keys used to import repositories from some other Git services +| `/var/opt/gitlab/gitlab-rails/uploads` | User uploaded attachments +| `/var/opt/gitlab/gitlab-rails/shared` | Build artifacts, GitLab Pages, LFS objects, temp files, etc. If you're using LFS this may also account for a large portion of your data +| `/var/opt/gitlab/gitlab-ci/builds` | GitLab CI build traces + +Other GitLab directories should not be shared between nodes. They contain +node-specific files and GitLab code that does not need to be shared. To ship +logs to a central location consider using remote syslog. GitLab Omnibus packages +provide configuration for [UDP log shipping][udp-log-shipping]. + +### Consolidating mount points + +If you don't want to configure 5-6 different NFS mount points, you have a few +alternative options. + +#### Change default file locations + +Omnibus allows you to configure the file locations. With custom configuration +you can specify just one main mountpoint and have all of these locations +as subdirectories. Mount `/gitlab-data` then use the following Omnibus +configuration to move each data location to a subdirectory: + +```ruby +user['home'] = '/gitlab-data/home' +git_data_dir '/gitlab-data/git-data' +gitlab_rails['shared_path'] = '/gitlab-data/shared' +gitlab_rails['uploads_directory'] = "/gitlab-data/uploads" +gitlab_ci['builds_directory'] = '/gitlab-data/builds' +``` + +To move the `git` home directory, all GitLab services must be stopped. Run +`gitlab-ctl stop && initctl stop gitlab-runsvdir`. Then continue with the +reconfigure. + +Run `sudo gitlab-ctl reconfigure` to start using the central location. Please +be aware that if you had existing data you will need to manually copy/rsync it +to these new locations and then restart GitLab. + +#### Bind mounts + +Bind mounts provide a way to specify just one NFS mount and then +bind the default GitLab data locations to the NFS mount. Start by defining your +single NFS mount point as you normally would in `/etc/fstab`. Let's assume your +NFS mount point is `/gitlab-data`. Then, add the following bind mounts in +`/etc/fstab`: + +```bash +/gitlab-data/git-data /var/opt/gitlab/git-data none bind 0 0 +/gitlab-data/.ssh /var/opt/gitlab/.ssh none bind 0 0 +/gitlab-data/uploads /var/opt/gitlab/gitlab-rails/uploads none bind 0 0 +/gitlab-data/shared /var/opt/gitlab/gitlab-rails/shared none bind 0 0 +/gitlab-data/builds /var/opt/gitlab/gitlab-ci/builds none bind 0 0 +``` + +--- + +Read more on high-availability configuration: + +1. [Configure the database](database.md) +1. [Configure Redis](redis.md) +1. [Configure the GitLab application servers](gitlab.md) +1. [Configure the load balancers](load_balancer.md) + +[udp-log-shipping]: http://doc.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only "UDP log shipping" diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md new file mode 100644 index 00000000000..d89a1e582ca --- /dev/null +++ b/doc/administration/high_availability/redis.md @@ -0,0 +1,62 @@ +# Configuring Redis for GitLab HA + +You can choose to install and manage Redis yourself, or you can use GitLab +Omnibus packages to help. + +## Configure your own Redis server + +If you're hosting GitLab on a cloud provider, you can optionally use a +managed service for Redis. For example, AWS offers a managed ElastiCache service +that runs Redis. + +> **Note:** Redis does not require authentication by default. See + [Redis Security](http://redis.io/topics/security) documentation for more + information. We recommend using a combination of a Redis password and tight + firewall rules to secure your Redis service. + +## Configure using Omnibus + +1. Download/install GitLab Omnibus using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other + steps on the download page. +1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. + Be sure to change the `external_url` to match your eventual GitLab front-end + URL. + + ```ruby + external_url 'https://gitlab.example.com' + + # Disable all components except PostgreSQL + redis['enable'] = true + bootstrap['enable'] = false + nginx['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + postgresql['enable'] = false + gitlab_workhorse['enable'] = false + mailroom['enable'] = false + + # Redis configuration + redis['port'] = 6379 + redis['bind'] = '0.0.0.0' + + # If you wish to use Redis authentication (recommended) + redis['password'] = 'Redis Password' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. + + > **Note**: This `reconfigure` step will result in some errors. + That's OK - don't be alarmed. +1. Run `touch /etc/gitlab/skip-auto-migrations` to prevent database migrations + from running on upgrade. Only the primary GitLab application server should + handle migrations. + +--- + +Read more on high-availability configuration: + +1. [Configure the database](database.md) +1. [Configure NFS](nfs.md) +1. [Configure the GitLab application servers](gitlab.md) +1. [Configure the load balancers](load_balancer.md) diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md new file mode 100644 index 00000000000..3411e4af6a7 --- /dev/null +++ b/doc/administration/repository_checks.md @@ -0,0 +1,44 @@ +# Repository checks + +>**Note:** +This feature was [introduced][ce-3232] in GitLab 8.7. It is OFF by +default because it still causes too many false alarms. + +Git has a built-in mechanism, [git fsck][git-fsck], to verify the +integrity of all data commited to a repository. GitLab administrators +can trigger such a check for a project via the project page under the +admin panel. The checks run asynchronously so it may take a few minutes +before the check result is visible on the project admin page. If the +checks failed you can see their output on the admin log page under +'repocheck.log'. + +## Periodic checks + +GitLab periodically runs a repository check on all project repositories and +wiki repositories in order to detect data corruption problems. A +project will be checked no more than once per week. If any projects +fail their repository checks all GitLab administrators will receive an email +notification of the situation. This notification is sent out no more +than once a day. + +## Disabling periodic checks + +You can disable the periodic checks on the 'Settings' page of the admin +panel. + +## What to do if a check failed + +If the repository check fails for some repository you should look up the error +in repocheck.log (in the admin panel or on disk; see +`/var/log/gitlab/gitlab-rails` for Omnibus installations or +`/home/git/gitlab/log` for installations from source). Once you have +resolved the issue use the admin panel to trigger a new repository check on +the project. This will clear the 'check failed' state. + +If for some reason the periodic repository check caused a lot of false +alarms you can choose to clear ALL repository check states from the +'Settings' page of the admin panel. + +--- +[ce-3232]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3232 "Auto git fsck" +[git-fsck]: https://www.kernel.org/pub/software/scm/git/docs/git-fsck.html "git fsck documentation"
\ No newline at end of file diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md new file mode 100644 index 00000000000..134a7583762 --- /dev/null +++ b/doc/administration/troubleshooting/sidekiq.md @@ -0,0 +1,162 @@ +# Troubleshooting Sidekiq + +Sidekiq is the background job processor GitLab uses to asynchronously run +tasks. When things go wrong it can be difficult to troubleshoot. These +situations also tend to be high-pressure because a production system job queue +may be filling up. Users will notice when this happens because new branches +may not show up and merge requests may not be updated. The following are some +troubleshooting steps that will help you diagnose the bottleneck. + +> **Note:** GitLab administrators/users should consider working through these +debug steps with GitLab Support so the backtraces can be analyzed by our team. +It may reveal a bug or necessary improvement in GitLab. + +> **Note:** In any of the backtraces, be weary of suspecting cases where every + thread appears to be waiting in the database, Redis, or waiting to acquire + a mutex. This **may** mean there's contention in the database, for example, + but look for one thread that is different than the rest. This other thread + may be using all available CPU, or have a Ruby Global Interpreter Lock, + preventing other threads from continuing. + +## Thread dump + +Send the Sidekiq process ID the `TTIN` signal and it will output thread +backtraces in the log file. + +``` +kill -TTIN <sidekiq_pid> +``` + +Check in `/var/log/gitlab/sidekiq/current` or `$GITLAB_HOME/log/sidekiq.log` for +the backtrace output. The backtraces will be lengthy and generally start with +several `WARN` level messages. Here's an example of a single thread's backtrace: + +``` +2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: ActiveRecord::RecordNotFound: Couldn't find Note with 'id'=3375386 +2016-04-13T06:21:20.022Z 31517 TID-orn4urby0 WARN: /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/activerecord-4.2.5.2/lib/active_record/core.rb:155:in `find' +/opt/gitlab/embedded/service/gitlab-rails/app/workers/new_note_worker.rb:7:in `perform' +/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:150:in `execute_job' +/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/processor.rb:132:in `block (2 levels) in process' +/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:127:in `block in invoke' +/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:17:in `call' +/opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/sidekiq-4.0.1/lib/sidekiq/middleware/chain.rb:129:in `block in invoke' +/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/arguments_logger.rb:6:in `call' +... +``` + +In some cases Sidekiq may be hung and unable to respond to the `TTIN` signal. +Move on to other troubleshooting methods if this happens. + +## Process profiling with `perf` + +Linux has a process profiling tool called `perf` that is helpful when a certain +process is eating up a lot of CPU. If you see high CPU usage and Sidekiq won't +respond to the `TTIN` signal, this is a good next step. + +If `perf` is not installed on your system, install it with `apt-get` or `yum`: + +``` +# Debian +sudo apt-get install linux-tools + +# Ubuntu (may require these additional Kernel packages) +sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname -r` + +# Red Hat/CentOS +sudo yum install perf +``` + +Run perf against the Sidekiq PID: + +``` +sudo perf record -p <sidekiq_pid> +``` + +Let this run for 30-60 seconds and then press Ctrl-C. Then view the perf report: + +``` +sudo perf report + +# Sample output +Samples: 348K of event 'cycles', Event count (approx.): 280908431073 + 97.69% ruby nokogiri.so [.] xmlXPathNodeSetMergeAndClear + 0.18% ruby libruby.so.2.1.0 [.] objspace_malloc_increase + 0.12% ruby libc-2.12.so [.] _int_malloc + 0.10% ruby libc-2.12.so [.] _int_free +``` + +Above you see sample output from a perf report. It shows that 97% of the CPU is +being spent inside Nokogiri and `xmlXPathNodeSetMergeAndClear`. For something +this obvious you should then go investigate what job in GitLab would use +Nokogiri and XPath. Combine with `TTIN` or `gdb` output to show the +corresponding Ruby code where this is happening. + +## The GNU Project Debugger (gdb) + +`gdb` can be another effective tool for debugging Sidekiq. It gives you a little +more interactive way to look at each thread and see what's causing problems. + +> **Note:** Attaching to a process with `gdb` will suspends the normal operation + of the process (Sidekiq will not process jobs while `gdb` is attached). + +Start by attaching to the Sidekiq PID: + +``` +gdb -p <sidekiq_pid> +``` + +Then gather information on all the threads: + +``` +info threads + +# Example output +30 Thread 0x7fe5fbd63700 (LWP 26060) 0x0000003f7cadf113 in poll () from /lib64/libc.so.6 +29 Thread 0x7fe5f2b3b700 (LWP 26533) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +28 Thread 0x7fe5f2a3a700 (LWP 26534) 0x0000003f7ce0ba5e in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +27 Thread 0x7fe5f2939700 (LWP 26535) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +26 Thread 0x7fe5f2838700 (LWP 26537) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +25 Thread 0x7fe5f2737700 (LWP 26538) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +24 Thread 0x7fe5f2535700 (LWP 26540) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +23 Thread 0x7fe5f2434700 (LWP 26541) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +22 Thread 0x7fe5f2232700 (LWP 26543) 0x0000003f7ce0b68c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0 +21 Thread 0x7fe5f2131700 (LWP 26544) 0x00007fe5f7b570f0 in xmlXPathNodeSetMergeAndClear () +from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +... +``` + +If you see a suspicious thread, like the Nokogiri one above, you may want +to get more information: + +``` +thread 21 +bt + +# Example output +#0 0x00007ff0d6afe111 in xmlXPathNodeSetMergeAndClear () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#1 0x00007ff0d6b0b836 in xmlXPathNodeCollectAndTest () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#2 0x00007ff0d6b09037 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#3 0x00007ff0d6b09017 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#4 0x00007ff0d6b092e0 in xmlXPathCompOpEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#5 0x00007ff0d6b0bc37 in xmlXPathRunEval () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#6 0x00007ff0d6b0be5f in xmlXPathEvalExpression () from /opt/gitlab/embedded/service/gem/ruby/2.1.0/gems/nokogiri-1.6.7.2/lib/nokogiri/nokogiri.so +#7 0x00007ff0d6a97dc3 in evaluate (argc=2, argv=0x1022d058, self=<value optimized out>) at xml_xpath_context.c:221 +#8 0x00007ff0daeab0ea in vm_call_cfunc_with_frame (th=0x1022a4f0, reg_cfp=0x1032b810, ci=<value optimized out>) at vm_insnhelper.c:1510 +``` + +To output a backtrace from all threads at once: + +``` +apply all thread bt +``` + +## Check for blocking queries + +Sometimes the speed at which Sidekiq processes jobs can be so fast that it can +cause database contention. Check for blocking queries when backtraces above +show that many threads are stuck in the database adapter. + +The PostgreSQL wiki has details on the query you can run to see blocking +queries. The query is different based on PostgreSQL version. See +[Lock Monitoring](https://wiki.postgresql.org/wiki/Lock_Monitoring) for +the query details. diff --git a/doc/api/README.md b/doc/api/README.md index 7629ef294ac..ff039f1886f 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -33,6 +33,7 @@ following locations: - [Build triggers](build_triggers.md) - [Build Variables](build_variables.md) - [Runners](runners.md) +- [Licenses](licenses.md) ## Authentication @@ -108,6 +109,7 @@ The following table shows the possible return codes for API requests. | ------------- | ----------- | | `200 OK` | The `GET`, `PUT` or `DELETE` request was successful, the resource(s) itself is returned as JSON. | | `201 Created` | The `POST` request was successful and the resource is returned as JSON. | +| `304 Not Modified` | Indicates that the resource has not been modified since the last request. | | `400 Bad Request` | A required attribute of the API request is missing, e.g., the title of an issue is not given. | | `401 Unauthorized` | The user is not authenticated, a valid [user token](#authentication) is necessary. | | `403 Forbidden` | The request is not allowed, e.g., the user is not allowed to delete a project. | diff --git a/doc/api/issues.md b/doc/api/issues.md index f09847aef95..3e78149f442 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -298,6 +298,7 @@ PUT /projects/:id/issues/:issue_id | `milestone_id` | integer | no | The ID of a milestone to assign the issue to | | `labels` | string | no | Comma-separated label names for an issue | | `state_event` | string | no | The state event of an issue. Set `close` to close the issue and `reopen` to reopen it | +| `updated_at` | string | no | Date time string, ISO 8601 formatted, e.g. `2016-03-11T03:45:40Z` | ```bash curl -X PUT -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/4/issues/85?state_event=close @@ -406,6 +407,115 @@ Example response: } ``` +## Subscribe to an issue + +Subscribes the authenticated user to an issue to receive notifications. If the +operation is successful, status code `201` together with the updated issue is +returned. If the user is already subscribed to the issue, the status code `304` +is returned. If the project or issue is not found, status code `404` is +returned. + +``` +POST /projects/:id/issues/:issue_id/subscription +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | +| `issue_id` | integer | yes | The ID of a project's issue | + +```bash +curl -X POST -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/issues/93/subscription +``` + +Example response: + +```json +{ + "id": 92, + "iid": 11, + "project_id": 5, + "title": "Sit voluptas tempora quisquam aut doloribus et.", + "description": "Repellat voluptas quibusdam voluptatem exercitationem.", + "state": "opened", + "created_at": "2016-04-05T21:41:45.652Z", + "updated_at": "2016-04-07T12:20:17.596Z", + "labels": [], + "milestone": null, + "assignee": { + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/u/axel.block" + }, + "author": { + "name": "Kris Steuber", + "username": "solon.cremin", + "id": 10, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/7a190fecbaa68212a4b68aeb6e3acd10?s=80&d=identicon", + "web_url": "https://gitlab.example.com/u/solon.cremin" + } +} +``` + +## Unsubscribe from an issue + +Unsubscribes the authenticated user from the issue to not receive notifications +from it. If the operation is successful, status code `200` together with the +updated issue is returned. If the user is not subscribed to the issue, the +status code `304` is returned. If the project or issue is not found, status code +`404` is returned. + +``` +DELETE /projects/:id/issues/:issue_id/subscription +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | +| `issue_id` | integer | yes | The ID of a project's issue | + +```bash +curl -X DELETE -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/issues/93/subscription +``` + +Example response: + +```json +{ + "id": 93, + "iid": 12, + "project_id": 5, + "title": "Incidunt et rerum ea expedita iure quibusdam.", + "description": "Et cumque architecto sed aut ipsam.", + "state": "opened", + "created_at": "2016-04-05T21:41:45.217Z", + "updated_at": "2016-04-07T13:02:37.905Z", + "labels": [], + "milestone": null, + "assignee": { + "name": "Edwardo Grady", + "username": "keyon", + "id": 21, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/3e6f06a86cf27fa8b56f3f74f7615987?s=80&d=identicon", + "web_url": "https://gitlab.example.com/u/keyon" + }, + "author": { + "name": "Vivian Hermann", + "username": "orville", + "id": 11, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/5224fd70153710e92fb8bcf79ac29d67?s=80&d=identicon", + "web_url": "http://lgitlab.example.com/u/orville" + }, + "subscribed": false +} +``` + ## Comments on issues Comments are done via the [notes](notes.md) resource. diff --git a/doc/api/licenses.md b/doc/api/licenses.md new file mode 100644 index 00000000000..855b0eab56f --- /dev/null +++ b/doc/api/licenses.md @@ -0,0 +1,147 @@ +# Licenses + +## List license templates + +Get all license templates. + +``` +GET /licenses +``` + +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | --------------------- | +| `popular` | boolean | no | If passed, returns only popular licenses | + +```bash +curl https://gitlab.example.com/api/v3/licenses?popular=1 +``` + +Example response: + +```json +[ + { + "key": "apache-2.0", + "name": "Apache License 2.0", + "nickname": null, + "featured": true, + "html_url": "http://choosealicense.com/licenses/apache-2.0/", + "source_url": "http://www.apache.org/licenses/LICENSE-2.0.html", + "description": "A permissive license that also provides an express grant of patent rights from contributors to users.", + "conditions": [ + "include-copyright", + "document-changes" + ], + "permissions": [ + "commercial-use", + "modifications", + "distribution", + "patent-use", + "private-use" + ], + "limitations": [ + "trademark-use", + "no-liability" + ], + "content": " Apache License\n Version 2.0, January 2004\n [...]" + }, + { + "key": "gpl-3.0", + "name": "GNU General Public License v3.0", + "nickname": "GNU GPLv3", + "featured": true, + "html_url": "http://choosealicense.com/licenses/gpl-3.0/", + "source_url": "http://www.gnu.org/licenses/gpl-3.0.txt", + "description": "The GNU GPL is the most widely used free software license and has a strong copyleft requirement. When distributing derived works, the source code of the work must be made available under the same license.", + "conditions": [ + "include-copyright", + "document-changes", + "disclose-source", + "same-license" + ], + "permissions": [ + "commercial-use", + "modifications", + "distribution", + "patent-use", + "private-use" + ], + "limitations": [ + "no-liability" + ], + "content": " GNU GENERAL PUBLIC LICENSE\n Version 3, 29 June 2007\n [...]" + }, + { + "key": "mit", + "name": "MIT License", + "nickname": null, + "featured": true, + "html_url": "http://choosealicense.com/licenses/mit/", + "source_url": "http://opensource.org/licenses/MIT", + "description": "A permissive license that is short and to the point. It lets people do anything with your code with proper attribution and without warranty.", + "conditions": [ + "include-copyright" + ], + "permissions": [ + "commercial-use", + "modifications", + "distribution", + "private-use" + ], + "limitations": [ + "no-liability" + ], + "content": "The MIT License (MIT)\n\nCopyright (c) [year] [fullname]\n [...]" + } +] +``` + +## Single license template + +Get a single license template. You can pass parameters to replace the license +placeholder. + +``` +GET /licenses/:key +``` + +| Attribute | Type | Required | Description | +| ---------- | ------ | -------- | ----------- | +| `key` | string | yes | The key of the license template | +| `project` | string | no | The copyrighted project name | +| `fullname` | string | no | The full-name of the copyright holder | + +>**Note:** +If you omit the `fullname` parameter but authenticate your request, the name of +the authenticated user will be used to replace the copyright holder placeholder. + +```bash +curl -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/licenses/mit?project=My+Cool+Project +``` + +Example response: + +```json +{ + "key": "mit", + "name": "MIT License", + "nickname": null, + "featured": true, + "html_url": "http://choosealicense.com/licenses/mit/", + "source_url": "http://opensource.org/licenses/MIT", + "description": "A permissive license that is short and to the point. It lets people do anything with your code with proper attribution and without warranty.", + "conditions": [ + "include-copyright" + ], + "permissions": [ + "commercial-use", + "modifications", + "distribution", + "private-use" + ], + "limitations": [ + "no-liability" + ], + "content": "The MIT License (MIT)\n\nCopyright (c) 2016 John Doe\n [...]" +} +``` diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index 20db73ea6c0..2057f9d77aa 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -606,3 +606,151 @@ Example response: }, ] ``` + +## Subscribe to a merge request + +Subscribes the authenticated user to a merge request to receive notification. If +the operation is successful, status code `201` together with the updated merge +request is returned. If the user is already subscribed to the merge request, the +status code `304` is returned. If the project or merge request is not found, +status code `404` is returned. + +``` +POST /projects/:id/merge_requests/:merge_request_id/subscription +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | +| `merge_request_id` | integer | yes | The ID of the merge request | + +```bash +curl -X POST -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/merge_requests/17/subscription +``` + +Example response: + +```json +{ + "id": 17, + "iid": 1, + "project_id": 5, + "title": "Et et sequi est impedit nulla ut rem et voluptatem.", + "description": "Consequatur velit eos rerum optio autem. Quia id officia quaerat dolorum optio. Illo laudantium aut ipsum dolorem.", + "state": "opened", + "created_at": "2016-04-05T21:42:23.233Z", + "updated_at": "2016-04-05T22:11:52.900Z", + "target_branch": "ui-dev-kit", + "source_branch": "version-1-9", + "upvotes": 0, + "downvotes": 0, + "author": { + "name": "Eileen Skiles", + "username": "leila", + "id": 19, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/39ce4a2822cc896933ffbd68c1470e55?s=80&d=identicon", + "web_url": "https://gitlab.example.com/u/leila" + }, + "assignee": { + "name": "Celine Wehner", + "username": "carli", + "id": 16, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/f4cd5605b769dd2ce405a27c6e6f2684?s=80&d=identicon", + "web_url": "https://gitlab.example.com/u/carli" + }, + "source_project_id": 5, + "target_project_id": 5, + "labels": [], + "work_in_progress": false, + "milestone": { + "id": 7, + "iid": 1, + "project_id": 5, + "title": "v2.0", + "description": "Corrupti eveniet et velit occaecati dolorem est rerum aut.", + "state": "closed", + "created_at": "2016-04-05T21:41:40.905Z", + "updated_at": "2016-04-05T21:41:40.905Z", + "due_date": null + }, + "merge_when_build_succeeds": false, + "merge_status": "cannot_be_merged", + "subscribed": true +} +``` + +## Unsubscribe from a merge request + +Unsubscribes the authenticated user from a merge request to not receive +notifications from that merge request. If the operation is successful, status +code `200` together with the updated merge request is returned. If the user is +not subscribed to the merge request, the status code `304` is returned. If the +project or merge request is not found, status code `404` is returned. + +``` +DELETE /projects/:id/merge_requests/:merge_request_id/subscription +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of a project | +| `merge_request_id` | integer | yes | The ID of the merge request | + +```bash +curl -X DELETE -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v3/projects/5/merge_requests/17/subscription +``` + +Example response: + +```json +{ + "id": 17, + "iid": 1, + "project_id": 5, + "title": "Et et sequi est impedit nulla ut rem et voluptatem.", + "description": "Consequatur velit eos rerum optio autem. Quia id officia quaerat dolorum optio. Illo laudantium aut ipsum dolorem.", + "state": "opened", + "created_at": "2016-04-05T21:42:23.233Z", + "updated_at": "2016-04-05T22:11:52.900Z", + "target_branch": "ui-dev-kit", + "source_branch": "version-1-9", + "upvotes": 0, + "downvotes": 0, + "author": { + "name": "Eileen Skiles", + "username": "leila", + "id": 19, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/39ce4a2822cc896933ffbd68c1470e55?s=80&d=identicon", + "web_url": "https://gitlab.example.com/u/leila" + }, + "assignee": { + "name": "Celine Wehner", + "username": "carli", + "id": 16, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/f4cd5605b769dd2ce405a27c6e6f2684?s=80&d=identicon", + "web_url": "https://gitlab.example.com/u/carli" + }, + "source_project_id": 5, + "target_project_id": 5, + "labels": [], + "work_in_progress": false, + "milestone": { + "id": 7, + "iid": 1, + "project_id": 5, + "title": "v2.0", + "description": "Corrupti eveniet et velit occaecati dolorem est rerum aut.", + "state": "closed", + "created_at": "2016-04-05T21:41:40.905Z", + "updated_at": "2016-04-05T21:41:40.905Z", + "due_date": null + }, + "merge_when_build_succeeds": false, + "merge_status": "cannot_be_merged", + "subscribed": false +} +``` diff --git a/doc/api/notes.md b/doc/api/notes.md index 2e0936f11b5..7aa1c2155bf 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -89,6 +89,7 @@ Parameters: - `id` (required) - The ID of a project - `issue_id` (required) - The ID of an issue - `body` (required) - The content of a note +- `created_at` (optional) - Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z ### Modify existing issue note diff --git a/doc/api/projects.md b/doc/api/projects.md index ab716c229dc..de1faadebf5 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -491,6 +491,132 @@ Parameters: - `id` (required) - The ID of the project to be forked +### Star a project + +Stars a given project. Returns status code `201` and the project on success and +`304` if the project is already starred. + +``` +POST /projects/:id/star +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of the project | + +```bash +curl -X POST -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/5/star" +``` + +Example response: + +```json +{ + "id": 3, + "description": null, + "default_branch": "master", + "public": false, + "visibility_level": 10, + "ssh_url_to_repo": "git@example.com:diaspora/diaspora-project-site.git", + "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", + "web_url": "http://example.com/diaspora/diaspora-project-site", + "tag_list": [ + "example", + "disapora project" + ], + "name": "Diaspora Project Site", + "name_with_namespace": "Diaspora / Diaspora Project Site", + "path": "diaspora-project-site", + "path_with_namespace": "diaspora/diaspora-project-site", + "issues_enabled": true, + "open_issues_count": 1, + "merge_requests_enabled": true, + "builds_enabled": true, + "wiki_enabled": true, + "snippets_enabled": false, + "created_at": "2013-09-30T13: 46: 02Z", + "last_activity_at": "2013-09-30T13: 46: 02Z", + "creator_id": 3, + "namespace": { + "created_at": "2013-09-30T13: 46: 02Z", + "description": "", + "id": 3, + "name": "Diaspora", + "owner_id": 1, + "path": "diaspora", + "updated_at": "2013-09-30T13: 46: 02Z" + }, + "archived": true, + "avatar_url": "http://example.com/uploads/project/avatar/3/uploads/avatar.png", + "shared_runners_enabled": true, + "forks_count": 0, + "star_count": 1 +} +``` + +### Unstar a project + +Unstars a given project. Returns status code `200` and the project on success +and `304` if the project is not starred. + +``` +DELETE /projects/:id/star +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | The ID of the project | + +```bash +curl -X DELETE -H "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v3/projects/5/star" +``` + +Example response: + +```json +{ + "id": 3, + "description": null, + "default_branch": "master", + "public": false, + "visibility_level": 10, + "ssh_url_to_repo": "git@example.com:diaspora/diaspora-project-site.git", + "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", + "web_url": "http://example.com/diaspora/diaspora-project-site", + "tag_list": [ + "example", + "disapora project" + ], + "name": "Diaspora Project Site", + "name_with_namespace": "Diaspora / Diaspora Project Site", + "path": "diaspora-project-site", + "path_with_namespace": "diaspora/diaspora-project-site", + "issues_enabled": true, + "open_issues_count": 1, + "merge_requests_enabled": true, + "builds_enabled": true, + "wiki_enabled": true, + "snippets_enabled": false, + "created_at": "2013-09-30T13: 46: 02Z", + "last_activity_at": "2013-09-30T13: 46: 02Z", + "creator_id": 3, + "namespace": { + "created_at": "2013-09-30T13: 46: 02Z", + "description": "", + "id": 3, + "name": "Diaspora", + "owner_id": 1, + "path": "diaspora", + "updated_at": "2013-09-30T13: 46: 02Z" + }, + "archived": true, + "avatar_url": "http://example.com/uploads/project/avatar/3/uploads/avatar.png", + "shared_runners_enabled": true, + "forks_count": 0, + "star_count": 0 +} +``` + ### Archive a project Archives the project if the user is either admin or the project owner of this project. This action is diff --git a/doc/ci/api/builds.md b/doc/ci/api/builds.md index d100e261178..79761a893da 100644 --- a/doc/ci/api/builds.md +++ b/doc/ci/api/builds.md @@ -26,48 +26,114 @@ This API uses two types of authentication: ### Runs oldest pending build by runner - POST /ci/api/v1/builds/register +``` +POST /ci/api/v1/builds/register +``` -Parameters: +| Attribute | Type | Required | Description | +|-----------|---------|----------|---------------------| +| `token` | string | yes | Unique runner token | - * `token` (required) - Unique runner token +``` +curl -X POST "https://gitlab.example.com/ci/api/v1/builds/register" -F "token=t0k3n" +``` ### Update details of an existing build - PUT /ci/api/v1/builds/:id +``` +PUT /ci/api/v1/builds/:id +``` + +| Attribute | Type | Required | Description | +|-----------|---------|----------|----------------------| +| `id` | integer | yes | The ID of a project | +| `token` | string | yes | Unique runner token | +| `state` | string | no | The state of a build | +| `trace` | string | no | The trace of a build | + +``` +curl -X PUT "https://gitlab.example.com/ci/api/v1/builds/1234" -F "token=t0k3n" -F "state=running" -F "trace=Running git clone...\n" +``` + +### Incremental build trace update + +Using this method you need to send trace content as a request body. You also need to provide the `Content-Range` header +with a range of sent trace part. Note that you need to send parts in the proper order, so the begining of the part +must start just after the end of the previous part. If you provide the wrong part, then GitLab CI API will return `416 +Range Not Satisfiable` response with a header `Range: 0-X`, where `X` is the current trace length. + +For example, if you receive `Range: 0-11` in the response, then your next part must contain a `Content-Range: 11-...` +header and a trace part covered by this range. + +For a valid update API will return `202` response with: +* `Build-Status: {status}` header containing current status of the build, +* `Range: 0-{length}` header with the current trace length. + +``` +PATCH /ci/api/v1/builds/:id/trace.txt +``` Parameters: - * `id` (required) - The ID of a project - * `token` (required) - Unique runner token - * `state` (optional) - The state of a build - * `trace` (optional) - The trace of a build +| Attribute | Type | Required | Description | +|-----------|---------|----------|----------------------| +| `id` | integer | yes | The ID of a build | + +Headers: + +| Attribute | Type | Required | Description | +|-----------------|---------|----------|-----------------------------------| +| `BUILD-TOKEN` | string | yes | The build authorization token | +| `Content-Range` | string | yes | Bytes range of trace that is sent | + +``` +curl -X PATCH "https://gitlab.example.com/ci/api/v1/builds/1234/trace.txt" -H "BUILD-TOKEN=build_t0k3n" -H "Content-Range=0-21" -d "Running git clone...\n" +``` + ### Upload artifacts to build - POST /ci/api/v1/builds/:id/artifacts +``` +POST /ci/api/v1/builds/:id/artifacts +``` -Parameters: +| Attribute | Type | Required | Description | +|-----------|---------|----------|-------------------------------| +| `id` | integer | yes | The ID of a build | +| `token` | string | yes | The build authorization token | +| `file` | mixed | yes | Artifacts file | - * `id` (required) - The ID of a build - * `token` (required) - The build authorization token - * `file` (required) - Artifacts file +``` +curl -X POST "https://gitlab.example.com/ci/api/v1/builds/1234/artifacts" -F "token=build_t0k3n" -F "file=@/path/to/file" +``` ### Download the artifacts file from build - GET /ci/api/v1/builds/:id/artifacts +``` +GET /ci/api/v1/builds/:id/artifacts +``` -Parameters: +| Attribute | Type | Required | Description | +|-----------|---------|----------|-------------------------------| +| `id` | integer | yes | The ID of a build | +| `token` | string | yes | The build authorization token | - * `id` (required) - The ID of a build - * `token` (required) - The build authorization token +``` +curl "https://gitlab.example.com/ci/api/v1/builds/1234/artifacts" -F "token=build_t0k3n" +``` ### Remove the artifacts file from build - DELETE /ci/api/v1/builds/:id/artifacts +``` +DELETE /ci/api/v1/builds/:id/artifacts +``` -Parameters: +| Attribute | Type | Required | Description | +|-----------|---------|----------|-------------------------------| +| ` id` | integer | yes | The ID of a build | +| `token` | string | yes | The build authorization token | - * ` id` (required) - The ID of a build - * `token` (required) - The build authorization token +``` +curl -X DELETE "https://gitlab.example.com/ci/api/v1/builds/1234/artifacts" -F "token=build_t0k3n" +``` diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index bd748f1b986..84212fb3c61 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -239,8 +239,8 @@ is specific to your project. Then create some service containers: ``` -docker run -d -n service-mysql mysql:latest -docker run -d -n service-postgres postgres:latest +docker run -d --name service-mysql mysql:latest +docker run -d --name service-postgres postgres:latest ``` This will create two service containers, named `service-mysql` and diff --git a/doc/ci/examples/php.md b/doc/ci/examples/php.md index aeadd6a448e..db077927126 100644 --- a/doc/ci/examples/php.md +++ b/doc/ci/examples/php.md @@ -40,7 +40,7 @@ repository with the following content: #!/bin/bash # We need to install dependencies only for Docker -[[ ! -e /.dockerinit ]] && exit 0 +[[ ! -e /.dockerenv ]] && [[ ! -e /.dockerinit ]] && exit 0 set -xe diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 9aba4326e11..6a42a935abd 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -13,7 +13,7 @@ GitLab offers a [continuous integration][ci] service. If you and configure your GitLab project to use a [Runner], then each merge request or push triggers a build. -The `.gitlab-ci.yml` file tells the GitLab runner what do to. By default it +The `.gitlab-ci.yml` file tells the GitLab runner what to do. By default it runs three [stages]: `build`, `test`, and `deploy`. If everything runs OK (no non-zero return values), you'll get a nice green diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index 295d953db11..a06650b3387 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -7,6 +7,10 @@ through the coordinator API of GitLab CI. A runner can be specific to a certain project or serve any project in GitLab CI. A runner that serves all projects is called a shared runner. +Ideally, GitLab Runner should not be installed on the same machine as GitLab. +Read the [requirements documentation](../../install/requirements.md#gitlab-runner) +for more information. + ## Shared vs. Specific Runners A runner that is specific only runs for the specified project. A shared runner @@ -19,7 +23,7 @@ many projects, you can have a single or a small number of runners that handle multiple projects. This makes it easier to maintain and update runners. **Specific runners** are useful for jobs that have special requirements or for -projects with a very demand. If a job has certain requirements, you can set +projects with a specific demand. If a job has certain requirements, you can set up the specific runner with this in mind, while not having to do this for all runners. For example, if you want to deploy a certain project, you can setup a specific runner to have the right credentials for this. @@ -140,7 +144,7 @@ to it. This means that if you have shared runners setup for a project and someone forks that project, the shared runners will also serve jobs of this project. -# Attack vectors in runners +## Attack vectors in Runners Mentioned briefly earlier, but the following things of runners can be exploited. We're always looking for contributions that can mitigate these [Security Considerations](https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/security/index.md). diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index b0e53cbc261..70fb81492d6 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -1,17 +1,20 @@ ## Variables + When receiving a build from GitLab CI, the runner prepares the build environment. It starts by setting a list of **predefined variables** (Environment Variables) and a list of **user-defined variables** The variables can be overwritten. They take precedence over each other in this order: +1. Trigger variables 1. Secure variables -1. YAML-defined variables +1. YAML-defined job-level variables +1. YAML-defined global variables 1. Predefined variables For example, if you define: -1. API_TOKEN=SECURE as Secure Variable -1. API_TOKEN=YAML as YAML-defined variable +1. `API_TOKEN=SECURE` as Secure Variable +1. `API_TOKEN=YAML` as YAML-defined variable -The API_TOKEN will take the Secure Variable value: `SECURE`. +The `API_TOKEN` will take the Secure Variable value: `SECURE`. ### Predefined variables (Environment Variables) @@ -70,15 +73,20 @@ These variables can be later used in all executed commands and scripts. The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. +Variables can be defined at a global level, but also at a job level. + More information about Docker integration can be found in [Using Docker Images](../docker/using_docker_images.md). ### User-defined variables (Secure Variables) **This feature requires GitLab Runner 0.4.0 or higher** -GitLab CI allows you to define per-project **Secure Variables** that are set in build environment. +GitLab CI allows you to define per-project **Secure Variables** that are set in +the build environment. The secure variables are stored out of the repository (the `.gitlab-ci.yml`). -The variables are securely passed to GitLab Runner and are available in build environment. -It's desired method to use them for storing passwords, secret keys or whatever you want. +The variables are securely passed to GitLab Runner and are available in the +build environment. +It's desired method to use them for storing passwords, secret keys or whatever +you want. **The value of the variable can be shown in build log if explicitly asked to do so.** If your project is public or internal you can make the builds private. diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 7da9b31e30d..7e9bced7616 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1,5 +1,50 @@ # Configuration of your builds with .gitlab-ci.yml +This document describes the usage of `.gitlab-ci.yml`, the file that is used by +GitLab Runner to manage your project's builds. + +If you want a quick introduction to GitLab CI, follow our +[quick start guide](../quick_start/README.md). + +--- + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [.gitlab-ci.yml](#gitlab-ci-yml) + - [image and services](#image-and-services) + - [before_script](#before_script) + - [after_script](#after_script) + - [stages](#stages) + - [types](#types) + - [variables](#variables) + - [cache](#cache) + - [cache:key](#cache-key) +- [Jobs](#jobs) + - [script](#script) + - [stage](#stage) + - [job variables](#job-variables) + - [only and except](#only-and-except) + - [tags](#tags) + - [when](#when) + - [artifacts](#artifacts) + - [artifacts:name](#artifacts-name) + - [dependencies](#dependencies) + - [before_script and after_script](#before_script-and-after_script) +- [Hidden jobs](#hidden-jobs) +- [Special YAML features](#special-yaml-features) + - [Anchors](#anchors) +- [Validate the .gitlab-ci.yml](#validate-the-gitlab-ci-yml) +- [Skipping builds](#skipping-builds) +- [Examples](#examples) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> + +--- + +## .gitlab-ci.yml + From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) file (`.gitlab-ci.yml`) for the project configuration. It is placed in the root of your repository and contains definitions of how your project should be built. @@ -23,12 +68,10 @@ Of course a command can execute code directly (`./configure;make;make install`) or run a script (`test.sh`) in the repository. Jobs are used to create builds, which are then picked up by -[runners](../runners/README.md) and executed within the environment of the -runner. What is important, is that each job is run independently from each +[Runners](../runners/README.md) and executed within the environment of the +Runner. What is important, is that each job is run independently from each other. -## .gitlab-ci.yml - The YAML syntax allows for using more complex job specifications than in the above example: @@ -40,6 +83,9 @@ services: before_script: - bundle install +after_script: + - rm secrets + stages: - build - test @@ -64,6 +110,7 @@ There are a few reserved `keywords` that **cannot** be used as job names: | stages | no | Define build stages | | types | no | Alias for `stages` | | before_script | no | Define commands that run before each job's script | +| after_script | no | Define commands that run after each job's script | | variables | no | Define build variables | | cache | no | Define list of files that should be cached between subsequent runs | @@ -71,13 +118,21 @@ There are a few reserved `keywords` that **cannot** be used as job names: This allows to specify a custom Docker image and a list of services that can be used for time of the build. The configuration of this feature is covered in -separate document: [Use Docker](../docker/README.md). +[a separate document](../docker/README.md). ### before_script `before_script` is used to define the command that should be run before all builds, including deploy builds. This can be an array or a multi-line string. +### after_script + +>**Note:** +Introduced in GitLab 8.7 and GitLab Runner v1.2. + +`after_script` is used to define the command that will be run after for all +builds. This has to be an array or a multi-line string. + ### stages `stages` is used to define build stages that can be used by jobs. @@ -86,7 +141,8 @@ The specification of `stages` allows for having flexible multi stage pipelines. The ordering of elements in `stages` defines the ordering of builds' execution: 1. Builds of the same stage are run in parallel. -1. Builds of next stage are run after success. +1. Builds of the next stage are run after the jobs from the previous stage + complete successfully. Let's consider the following example, which defines 3 stages: @@ -98,9 +154,9 @@ stages: ``` 1. First all jobs of `build` are executed in parallel. -1. If all jobs of `build` succeeds, the `test` jobs are executed in parallel. -1. If all jobs of `test` succeeds, the `deploy` jobs are executed in parallel. -1. If all jobs of `deploy` succeeds, the commit is marked as `success`. +1. If all jobs of `build` succeed, the `test` jobs are executed in parallel. +1. If all jobs of `test` succeed, the `deploy` jobs are executed in parallel. +1. If all jobs of `deploy` succeed, the commit is marked as `success`. 1. If any of the previous jobs fails, the commit is marked as `failed` and no jobs of further stage are executed. @@ -133,6 +189,8 @@ These variables can be later used in all executed commands and scripts. The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. +Variables can be also defined on [job level](#job-variables). + ### cache >**Note:** @@ -278,23 +336,26 @@ job_name: | Keyword | Required | Description | |---------------|----------|-------------| -| script | yes | Defines a shell script which is executed by runner | +| script | yes | Defines a shell script which is executed by Runner | | image | no | Use docker image, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | | services | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | | stage | no | Defines a build stage (default: `test`) | | type | no | Alias for `stage` | +| variables | no | Define build variables on a job level | | only | no | Defines a list of git refs for which build is created | | except | no | Defines a list of git refs for which build is not created | -| tags | no | Defines a list of tags which are used to select runner | +| tags | no | Defines a list of tags which are used to select Runner | | allow_failure | no | Allow build to fail. Failed build doesn't contribute to commit status | | when | no | Define when to run build. Can be `on_success`, `on_failure` or `always` | | dependencies | no | Define other builds that a build depends on so that you can pass artifacts between them| | artifacts | no | Define list build artifacts | | cache | no | Define list of files that should be cached between subsequent runs | +| before_script | no | Override a set of commands that are executed before build | +| after_script | no | Override a set of commands that are executed after build | ### script -`script` is a shell script which is executed by the runner. For example: +`script` is a shell script which is executed by the Runner. For example: ```yaml job: @@ -373,15 +434,27 @@ job: The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, except master. +### job variables + +It is possible to define build variables using a `variables` keyword on a job +level. It works basically the same way as its global-level equivalent but +allows you to define job-specific build variables. + +When the `variables` keyword is used on a job level, it overrides global YAML +build variables and predefined variables. + +Build variables priority is defined in +[variables documentation](../variables/README.md). + ### tags -`tags` is used to select specific runners from the list of all runners that are +`tags` is used to select specific Runners from the list of all Runners that are allowed to run this project. -During the registration of a runner, you can specify the runner's tags, for +During the registration of a Runner, you can specify the Runner's tags, for example `ruby`, `postgres`, `development`. -`tags` allow you to run builds with runners that have the specified tags +`tags` allow you to run builds with Runners that have the specified tags assigned to them: ```yaml @@ -391,7 +464,7 @@ job: - postgres ``` -The specification above, will make sure that `job` is built by a runner that +The specification above, will make sure that `job` is built by a Runner that has both `ruby` AND `postgres` tags defined. ### when @@ -635,6 +708,23 @@ deploy: script: make deploy ``` +### before_script and after_script + +It's possible to overwrite globally defined `before_script` and `after_script`: + +```yaml +before_script +- global before script + +job: + before_script: + - execute this instead of global before script + script: + - my command + after_script: + - execute this after my script +``` + ## Hidden jobs >**Note:** diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 9f3fd69fc4e..6d04b9590e6 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -9,7 +9,7 @@ bundle exec rake setup ``` The `setup` task is a alias for `gitlab:setup`. -This tasks calls `db:setup` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and finally it calls `db:seed_fu` to seed the database. +This tasks calls `db:reset` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and finally it calls `db:seed_fu` to seed the database. Note: `db:setup` calls `db:seed` but this does nothing. ## Run tests diff --git a/doc/downgrade_ee_to_ce/README.md b/doc/downgrade_ee_to_ce/README.md new file mode 100644 index 00000000000..3625c4191b8 --- /dev/null +++ b/doc/downgrade_ee_to_ce/README.md @@ -0,0 +1,82 @@ +# Downgrading from EE to CE + +If you ever decide to downgrade your Enterprise Edition back to the Community +Edition, there are a few steps you need take before installing the CE package +on top of the current EE package, or, if you are in an installation from source, +before you change remotes and fetch the latest CE code. + +## Disable Enterprise-only features + +First thing to do is to disable the following features. + +### Authentication mechanisms + +Kerberos and Atlassian Crowd are only available on the Enterprise Edition, so +you should disable these mechanisms before downgrading and you should provide +alternative authentication methods to your users. + +### Git Annex + +Git Annex is also only available on the Enterprise Edition. This means that if +you have repositories that use Git Annex to store large files, these files will +no longer be easily available via Git. You should consider migrating these +repositories to use Git LFS before downgrading to the Community Edition. + +### Remove Jenkins CI Service entries from the database + +The `JenkinsService` class is only available on the Enterprise Edition codebase, +so if you downgrade to the Community Edition, you'll come across the following +error: + +``` +Completed 500 Internal Server Error in 497ms (ActiveRecord: 32.2ms) + +ActionView::Template::Error (The single-table inheritance mechanism failed to locate the subclass: 'JenkinsService'. This +error is raised because the column 'type' is reserved for storing the class in case of inheritance. Please rename this +column if you didn't intend it to be used for storing the inheritance class or overwrite Service.inheritance_column to +use another column for that information.) +``` + +All services are created automatically for every project you have, so in order +to avoid getting this error, you need to remove all instances of the +`JenkinsService` from your database: + +**Omnibus Installation** + +``` +$ sudo gitlab-rails runner "Service.where(type: 'JenkinsService').delete_all" +``` + +**Source Installation** + +``` +$ bundle exec rails runner "Service.where(type: 'JenkinsService').delete_all" production +``` + +## Downgrade to CE + +After performing the above mentioned steps, you are now ready to downgrade your +GitLab installation to the Community Edition. + +**Omnibus Installation** + +To downgrade an Omnibus installation, it is sufficient to install the Community +Edition package on top of the currently installed one. You can do this manually, +by directly [downloading the package](https://packages.gitlab.com/gitlab/gitlab-ce) +you need, or by adding our CE package repository and following the +[CE installation instructions](https://about.gitlab.com/downloads/). + +**Source Installation** + +To downgrade a source installation, you need to replace the current remote of +your GitLab installation with the Community Edition's remote, fetch the latest +changes, and checkout the latest stable branch: + +``` +$ git remote set-url origin git@gitlab.com:gitlab-org/gitlab-ce.git +$ git fetch --all +$ git checkout 8-x-stable +``` + +Remember to follow the correct [update guides](../update/README.md) to make +sure all dependencies are up to date. diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 03cb08dd1f1..eb9fe5e1b1b 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -79,6 +79,26 @@ With less memory GitLab will give strange errors during the reconfigure run and Notice: The 25 workers of Sidekiq will show up as separate processes in your process overview (such as top or htop) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about many you need of those. +## Gitlab Runner + +We strongly advise against installing GitLab Runner on the same machine you plan +to install GitLab on. Depending on how you decide to configure GitLab Runner and +what tools you use to exercise your application in the CI environment, GitLab +Runner can consume significant amount of available memory. + +Memory consumption calculations, that are available above, will not be valid if +you decide to run GitLab Runner and the GitLab Rails application on the same +machine. + +It is also not safe to install everything on a single machine, because of the +[security reasons] - especially when you plan to use shell executor with GitLab +Runner. + +We recommend using a separate machine for each GitLab Runner, if you plan to +use the CI features. + +[security reasons]: https://gitlab.com/gitlab-org/gitlab-ci-multi-runner/blob/master/docs/security/index.md + ## Unicorn Workers It's possible to increase the amount of unicorn workers and this will usually help for to reduce the response time of the applications and increase the ability to handle parallel requests. diff --git a/doc/integration/shibboleth.md b/doc/integration/shibboleth.md index a0be3dd4e5c..b6b2d4e5e88 100644 --- a/doc/integration/shibboleth.md +++ b/doc/integration/shibboleth.md @@ -76,3 +76,50 @@ sudo gitlab-ctl reconfigure ``` On the sign in page there should now be a "Sign in with: Shibboleth" icon below the regular sign in form. Click the icon to begin the authentication process. You will be redirected to IdP server (Depends on your Shibboleth module configuration). If everything goes well the user will be returned to GitLab and will be signed in. + +## Apache 2.4 / GitLab 8.6 update +The order of the first 2 Location directives is important. If they are reversed, +you will not get a shibboleth session! + +``` + <Location /> + Require all granted + ProxyPassReverse http://127.0.0.1:8181 + ProxyPassReverse http://YOUR_SERVER_FQDN/ + </Location> + + <Location /users/auth/shibboleth/callback> + AuthType shibboleth + ShibRequestSetting requireSession 1 + ShibUseHeaders On + Require shib-session + </Location> + + Alias /shibboleth-sp /usr/share/shibboleth + + <Location /shibboleth-sp> + Require all granted + </Location> + + <Location /Shibboleth.sso> + SetHandler shib + </Location> + + RewriteEngine on + + #Don't escape encoded characters in api requests + RewriteCond %{REQUEST_URI} ^/api/v3/.* + RewriteCond %{REQUEST_URI} !/Shibboleth.sso + RewriteCond %{REQUEST_URI} !/shibboleth-sp + RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA,NE] + + #Forward all requests to gitlab-workhorse except existing files + RewriteCond %{DOCUMENT_ROOT}/%{REQUEST_FILENAME} !-f [OR] + RewriteCond %{REQUEST_URI} ^/uploads/.* + RewriteCond %{REQUEST_URI} !/Shibboleth.sso + RewriteCond %{REQUEST_URI} !/shibboleth-sp + RewriteRule .* http://127.0.0.1:8181%{REQUEST_URI} [P,QSA] + + RequestHeader set X_FORWARDED_PROTO 'https' + RequestHeader set X-Forwarded-Ssl on +```
\ No newline at end of file diff --git a/doc/intro/README.md b/doc/intro/README.md index fecbbe6317b..ab298d3808e 100644 --- a/doc/intro/README.md +++ b/doc/intro/README.md @@ -25,6 +25,7 @@ Create merge requests and review code. - [Automatically close issues from merge requests](../customization/issue_closing.md) - [Automatically merge when your builds succeed](../workflow/merge_when_build_succeeds.md) - [Revert any commit](../workflow/revert_changes.md) +- [Cherry-pick any commit](../workflow/cherry_pick_changes.md) ## Test and Deploy diff --git a/doc/monitoring/performance/gitlab_configuration.md b/doc/monitoring/performance/gitlab_configuration.md index b856e7935a3..90e99302210 100644 --- a/doc/monitoring/performance/gitlab_configuration.md +++ b/doc/monitoring/performance/gitlab_configuration.md @@ -37,3 +37,4 @@ Read more on: - [Introduction to GitLab Performance Monitoring](introduction.md) - [InfluxDB Configuration](influxdb_configuration.md) - [InfluxDB Schema](influxdb_schema.md) +- [Grafana Install/Configuration](grafana_configuration.md diff --git a/doc/monitoring/performance/grafana_configuration.md b/doc/monitoring/performance/grafana_configuration.md index 416c9870aa0..168bd85c26a 100644 --- a/doc/monitoring/performance/grafana_configuration.md +++ b/doc/monitoring/performance/grafana_configuration.md @@ -59,26 +59,53 @@ This will drop you in to an InfluxDB interactive session. Copy the entire contents below and paste it in to the interactive session: ``` -CREATE RETENTION POLICY gitlab_30d ON gitlab DURATION 30d REPLICATION 1 DEFAULT -CREATE RETENTION POLICY seven_days ON gitlab DURATION 7d REPLICATION 1 -CREATE CONTINUOUS QUERY rails_transaction_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean", percentile(sql_duration, 95.000) AS "sql_duration_95th", percentile(sql_duration, 99.000) AS "sql_duration_99th", mean(sql_duration) AS "sql_duration_mean", percentile(view_duration, 95.000) AS "view_duration_95th", percentile(view_duration, 99.000) AS "view_duration_99th", mean(view_duration) AS "view_duration_mean" INTO gitlab.seven_days.rails_transaction_timings FROM gitlab.gitlab_30d.rails_transactions GROUP BY time(1m) END -CREATE CONTINUOUS QUERY sidekiq_transaction_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean", percentile(sql_duration, 95.000) AS "sql_duration_95th", percentile(sql_duration, 99.000) AS "sql_duration_99th", mean(sql_duration) AS "sql_duration_mean", percentile(view_duration, 95.000) AS "view_duration_95th", percentile(view_duration, 99.000) AS "view_duration_99th", mean(view_duration) AS "view_duration_mean" INTO gitlab.seven_days.sidekiq_transaction_timings FROM gitlab.gitlab_30d.sidekiq_transactions GROUP BY time(1m) END -CREATE CONTINUOUS QUERY rails_transaction_counts_seven_days ON gitlab BEGIN SELECT count("duration") AS "count" INTO gitlab.seven_days.rails_transaction_counts FROM gitlab.gitlab_30d.rails_transactions GROUP BY time(1m) END -CREATE CONTINUOUS QUERY sidekiq_transaction_counts_seven_days ON gitlab BEGIN SELECT count("duration") AS "count" INTO gitlab.seven_days.sidekiq_transaction_counts FROM gitlab.gitlab_30d.sidekiq_transactions GROUP BY time(1m) END -CREATE CONTINUOUS QUERY rails_method_call_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean" INTO gitlab.seven_days.rails_method_call_timings FROM gitlab.gitlab_30d.rails_method_calls GROUP BY time(1m) END -CREATE CONTINUOUS QUERY sidekiq_method_call_timings_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS "duration_95th", percentile("duration", 99.000) AS "duration_99th", mean("duration") AS "duration_mean" INTO gitlab.seven_days.sidekiq_method_call_timings FROM gitlab.gitlab_30d.sidekiq_method_calls GROUP BY time(1m) END -CREATE CONTINUOUS QUERY rails_method_call_timings_per_method_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS duration_95th, percentile("duration", 99.000) AS duration_99th, mean("duration") AS duration_mean INTO gitlab.seven_days.rails_method_call_timings_per_method FROM gitlab.gitlab_30d.rails_method_calls GROUP BY time(1m), method END -CREATE CONTINUOUS QUERY sidekiq_method_call_timings_per_method_seven_days ON gitlab BEGIN SELECT percentile("duration", 95.000) AS duration_95th, percentile("duration", 99.000) AS duration_99th, mean("duration") AS duration_mean INTO gitlab.seven_days.sidekiq_method_call_timings_per_method FROM gitlab.gitlab_30d.sidekiq_method_calls GROUP BY time(1m), method END -CREATE CONTINUOUS QUERY rails_memory_usage_per_minute ON gitlab BEGIN SELECT percentile(value, 95.000) AS memory_95th, percentile(value, 99.000) AS memory_99th, mean(value) AS memory_mean INTO gitlab.seven_days.rails_memory_usage_per_minute FROM gitlab.gitlab_30d.rails_memory_usage GROUP BY time(1m) END -CREATE CONTINUOUS QUERY sidekiq_memory_usage_per_minute ON gitlab BEGIN SELECT percentile(value, 95.000) AS memory_95th, percentile(value, 99.000) AS memory_99th, mean(value) AS memory_mean INTO gitlab.seven_days.sidekiq_memory_usage_per_minute FROM gitlab.gitlab_30d.sidekiq_memory_usage GROUP BY time(1m) END -CREATE CONTINUOUS QUERY sidekiq_file_descriptors_per_minute ON gitlab BEGIN SELECT sum(value) AS value INTO gitlab.seven_days.sidekiq_file_descriptors_per_minute FROM gitlab.gitlab_30d.sidekiq_file_descriptors GROUP BY time(1m) END -CREATE CONTINUOUS QUERY rails_file_descriptors_per_minute ON gitlab BEGIN SELECT sum(value) AS value INTO gitlab.seven_days.rails_file_descriptors_per_minute FROM gitlab.gitlab_30d.rails_file_descriptors GROUP BY time(1m) END -CREATE CONTINUOUS QUERY rails_gc_counts_per_minute ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.seven_days.rails_gc_counts_per_minute FROM gitlab.gitlab_30d.rails_gc_statistics GROUP BY time(1m) END -CREATE CONTINUOUS QUERY sidekiq_gc_counts_per_minute ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.seven_days.sidekiq_gc_counts_per_minute FROM gitlab.gitlab_30d.sidekiq_gc_statistics GROUP BY time(1m) END -CREATE CONTINUOUS QUERY rails_gc_timings_per_minute ON gitlab BEGIN SELECT percentile(total_time, 95.000) AS duration_95th, percentile(total_time, 99.000) AS duration_99th, mean(total_time) AS duration_mean INTO gitlab.seven_days.rails_gc_timings_per_minute FROM gitlab.gitlab_30d.rails_gc_statistics GROUP BY time(1m) END -CREATE CONTINUOUS QUERY sidekiq_gc_timings_per_minute ON gitlab BEGIN SELECT percentile(total_time, 95.000) AS duration_95th, percentile(total_time, 99.000) AS duration_99th, mean(total_time) AS duration_mean INTO gitlab.seven_days.sidekiq_gc_timings_per_minute FROM gitlab.gitlab_30d.sidekiq_gc_statistics GROUP BY time(1m) END -CREATE CONTINUOUS QUERY rails_gc_major_minor_per_minute ON gitlab BEGIN SELECT sum(major_gc_count) AS major, sum(minor_gc_count) AS minor INTO gitlab.seven_days.rails_gc_major_minor_per_minute FROM gitlab.gitlab_30d.rails_gc_statistics GROUP BY time(1m) END -CREATE CONTINUOUS QUERY sidekiq_gc_major_minor_per_minute ON gitlab BEGIN SELECT sum(major_gc_count) AS major, sum(minor_gc_count) AS minor INTO gitlab.seven_days.sidekiq_gc_major_minor_per_minute FROM gitlab.gitlab_30d.sidekiq_gc_statistics GROUP BY time(1m) END +CREATE RETENTION POLICY default ON gitlab DURATION 1h REPLICATION 1 DEFAULT +CREATE RETENTION POLICY downsampled ON gitlab DURATION 7d REPLICATION 1 +CREATE CONTINUOUS QUERY grape_git_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_git_timings_per_action FROM gitlab."default".rails_method_calls WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND method =~ /^(Rugged|Gitlab::Git)/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY grape_markdown_render_timings_overall ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.grape_markdown_render_timings_overall FROM gitlab."default".rails_transactions WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_markdown_render_timings_per_action ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.grape_markdown_render_timings_per_action FROM gitlab."default".rails_transactions WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY grape_markdown_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_markdown_timings_overall FROM gitlab."default".rails_method_calls WHERE (action !~ /.+/ OR action =~ /^Grape#/) AND method =~ /^Banzai/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_method_call_timings_per_action_and_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_method_call_timings_per_action_and_method FROM gitlab."default".rails_method_calls WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), method, action END; +CREATE CONTINUOUS QUERY grape_method_call_timings_per_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.grape_method_call_timings_per_method FROM gitlab."default".rails_method_calls WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), method END; +CREATE CONTINUOUS QUERY grape_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.grape_transaction_counts_overall FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_transaction_counts_per_action ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.grape_transaction_counts_per_action FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY grape_transaction_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.grape_transaction_timings_overall FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY grape_transaction_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.grape_transaction_timings_per_action FROM gitlab."default".rails_transactions WHERE action !~ /.+/ OR action =~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_file_descriptor_counts ON gitlab BEGIN SELECT sum(value) AS count INTO gitlab.downsampled.rails_file_descriptor_counts FROM gitlab."default".rails_file_descriptors GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_gc_counts ON gitlab BEGIN SELECT sum(count) AS total, sum(minor_gc_count) AS minor, sum(major_gc_count) AS major INTO gitlab.downsampled.rails_gc_counts FROM gitlab."default".rails_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_gc_timings ON gitlab BEGIN SELECT mean(total_time) AS duration_mean, percentile(total_time, 95) AS duration_95th, percentile(total_time, 99) AS duration_99th INTO gitlab.downsampled.rails_gc_timings FROM gitlab."default".rails_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_git_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_git_timings_per_action FROM gitlab."default".rails_method_calls WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND method =~ /^(Rugged|Gitlab::Git)/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_markdown_render_timings_overall ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.rails_markdown_render_timings_overall FROM gitlab."default".rails_transactions WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_markdown_render_timings_per_action ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.rails_markdown_render_timings_per_action FROM gitlab."default".rails_transactions WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_markdown_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_markdown_timings_overall FROM gitlab."default".rails_method_calls WHERE (action =~ /.+/ AND action !~ /^Grape#/) AND method =~ /^Banzai/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_memory_usage_overall ON gitlab BEGIN SELECT mean(value) AS memory_mean, percentile(value, 95) AS memory_95th, percentile(value, 99) AS memory_99th INTO gitlab.downsampled.rails_memory_usage_overall FROM gitlab."default".rails_memory_usage GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_method_call_timings_per_action_and_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_method_call_timings_per_action_and_method FROM gitlab."default".rails_method_calls WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), method, action END; +CREATE CONTINUOUS QUERY rails_method_call_timings_per_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_method_call_timings_per_method FROM gitlab."default".rails_method_calls WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), method END; +CREATE CONTINUOUS QUERY rails_object_counts_overall ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.rails_object_counts_overall FROM gitlab."default".rails_object_counts GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_object_counts_per_type ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.rails_object_counts_per_type FROM gitlab."default".rails_object_counts GROUP BY time(1m), type END; +CREATE CONTINUOUS QUERY rails_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.rails_transaction_counts_overall FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_transaction_counts_per_action ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.rails_transaction_counts_per_action FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_transaction_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.rails_transaction_timings_overall FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY rails_transaction_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.rails_transaction_timings_per_action FROM gitlab."default".rails_transactions WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY rails_view_timings_per_action_and_view ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.rails_view_timings_per_action_and_view FROM gitlab."default".rails_views WHERE action =~ /.+/ AND action !~ /^Grape#/ GROUP BY time(1m), action, view END; +CREATE CONTINUOUS QUERY sidekiq_file_descriptor_counts ON gitlab BEGIN SELECT sum(value) AS count INTO gitlab.downsampled.sidekiq_file_descriptor_counts FROM gitlab."default".sidekiq_file_descriptors GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_gc_counts ON gitlab BEGIN SELECT sum(count) AS total, sum(minor_gc_count) AS minor, sum(major_gc_count) AS major INTO gitlab.downsampled.sidekiq_gc_counts FROM gitlab."default".sidekiq_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_gc_timings ON gitlab BEGIN SELECT mean(total_time) AS duration_mean, percentile(total_time, 95) AS duration_95th, percentile(total_time, 99) AS duration_99th INTO gitlab.downsampled.sidekiq_gc_timings FROM gitlab."default".sidekiq_gc_statistics GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_git_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_git_timings_per_action FROM gitlab."default".sidekiq_method_calls WHERE method =~ /^(Rugged|Gitlab::Git)/ GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_markdown_render_timings_overall ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.sidekiq_markdown_render_timings_overall FROM gitlab."default".sidekiq_transactions WHERE (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_markdown_render_timings_per_action ON gitlab BEGIN SELECT mean(banzai_cached_render_real_time) AS cached_real_mean, percentile(banzai_cached_render_real_time, 95) AS cached_real_95th, percentile(banzai_cached_render_real_time, 99) AS cached_real_99th, mean(banzai_cached_render_cpu_time) AS cached_cpu_mean, percentile(banzai_cached_render_cpu_time, 95) AS cached_cpu_95th, percentile(banzai_cached_render_cpu_time, 99) AS cached_cpu_99th, sum(banzai_cached_render_call_count) AS cached_call_count, mean(banzai_cacheless_render_real_time) AS cacheless_real_mean, percentile(banzai_cacheless_render_real_time, 95) AS cacheless_real_95th, percentile(banzai_cacheless_render_real_time, 99) AS cacheless_real_99th, mean(banzai_cacheless_render_cpu_time) AS cacheless_cpu_mean, percentile(banzai_cacheless_render_cpu_time, 95) AS cacheless_cpu_95th, percentile(banzai_cacheless_render_cpu_time, 99) AS cacheless_cpu_99th, sum(banzai_cacheless_render_call_count) AS cacheless_call_count INTO gitlab.downsampled.sidekiq_markdown_render_timings_per_action FROM gitlab."default".sidekiq_transactions WHERE (banzai_cached_render_call_count > 0 OR banzai_cacheless_render_call_count > 0) GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_markdown_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_markdown_timings_overall FROM gitlab."default".sidekiq_method_calls WHERE method =~ /^Banzai/ GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_memory_usage_overall ON gitlab BEGIN SELECT mean(value) AS memory_mean, percentile(value, 95) AS memory_95th, percentile(value, 99) AS memory_99th INTO gitlab.downsampled.sidekiq_memory_usage_overall FROM gitlab."default".sidekiq_memory_usage GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_method_call_timings_per_action_and_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_method_call_timings_per_action_and_method FROM gitlab."default".sidekiq_method_calls GROUP BY time(1m), method, action END; +CREATE CONTINUOUS QUERY sidekiq_method_call_timings_per_method ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_method_call_timings_per_method FROM gitlab."default".sidekiq_method_calls GROUP BY time(1m), method END; +CREATE CONTINUOUS QUERY sidekiq_object_counts_overall ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.sidekiq_object_counts_overall FROM gitlab."default".sidekiq_object_counts GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_object_counts_per_type ON gitlab BEGIN SELECT sum(count) AS count INTO gitlab.downsampled.sidekiq_object_counts_per_type FROM gitlab."default".sidekiq_object_counts GROUP BY time(1m), type END; +CREATE CONTINUOUS QUERY sidekiq_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.sidekiq_transaction_counts_overall FROM gitlab."default".sidekiq_transactions GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_transaction_counts_per_action ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.sidekiq_transaction_counts_per_action FROM gitlab."default".sidekiq_transactions GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_transaction_timings_overall ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.sidekiq_transaction_timings_overall FROM gitlab."default".sidekiq_transactions GROUP BY time(1m) END; +CREATE CONTINUOUS QUERY sidekiq_transaction_timings_per_action ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th, mean(sql_duration) AS sql_duration_mean, percentile(sql_duration, 95) AS sql_duration_95th, percentile(sql_duration, 99) AS sql_duration_99th, mean(view_duration) AS view_duration_mean, percentile(view_duration, 95) AS view_duration_95th, percentile(view_duration, 99) AS view_duration_99th, mean(cache_read_duration) AS cache_read_duration_mean, percentile(cache_read_duration, 99) AS cache_read_duration_99th, percentile(cache_read_duration, 95) AS cache_read_duration_95th, mean(cache_write_duration) AS cache_write_duration_mean, percentile(cache_write_duration, 99) AS cache_write_duration_99th, percentile(cache_write_duration, 95) AS cache_write_duration_95th, mean(cache_delete_duration) AS cache_delete_duration_mean, percentile(cache_delete_duration, 99) AS cache_delete_duration_99th, percentile(cache_delete_duration, 95) AS cache_delete_duration_95th, mean(cache_exists_duration) AS cache_exists_duration_mean, percentile(cache_exists_duration, 99) AS cache_exists_duration_99th, percentile(cache_exists_duration, 95) AS cache_exists_duration_95th, mean(cache_duration) AS cache_duration_mean, percentile(cache_duration, 99) AS cache_duration_99th, percentile(cache_duration, 95) AS cache_duration_95th, mean(method_duration) AS method_duration_mean, percentile(method_duration, 99) AS method_duration_99th, percentile(method_duration, 95) AS method_duration_95th INTO gitlab.downsampled.sidekiq_transaction_timings_per_action FROM gitlab."default".sidekiq_transactions GROUP BY time(1m), action END; +CREATE CONTINUOUS QUERY sidekiq_view_timings_per_action_and_view ON gitlab BEGIN SELECT mean("duration") AS duration_mean, percentile("duration", 95) AS duration_95th, percentile("duration", 99) AS duration_99th INTO gitlab.downsampled.sidekiq_view_timings_per_action_and_view FROM gitlab."default".sidekiq_views GROUP BY time(1m), action, view END; +CREATE CONTINUOUS QUERY web_transaction_counts_overall ON gitlab BEGIN SELECT count("duration") AS count INTO gitlab.downsampled.web_transaction_counts_overall FROM gitlab."default".rails_transactions GROUP BY time(1m) END; ``` ## Import Dashboards @@ -91,21 +118,25 @@ JSON file. Open the dashboard dropdown menu and click 'Import' - + Click 'Choose file' and browse to the location where you downloaded or cloned the dashboard repository. Pick one of the JSON files to import. - + Once the dashboard is imported, be sure to click save icon in the top bar. If you do not save the dashboard after importing it will be removed when you navigate away. - + Repeat this process for each dashboard you wish to import. +Alternatively you can automatically import all the dashboards into your Grafana +instance. See the README of the [Grafana dashboards][grafana-dashboards] +repository for more information on this process. + [grafana-dashboards]: https://gitlab.com/gitlab-org/grafana-dashboards --- diff --git a/doc/monitoring/performance/influxdb_configuration.md b/doc/monitoring/performance/influxdb_configuration.md index 3a2b598b78f..63aa03985ef 100644 --- a/doc/monitoring/performance/influxdb_configuration.md +++ b/doc/monitoring/performance/influxdb_configuration.md @@ -181,6 +181,7 @@ Read more on: - [Introduction to GitLab Performance Monitoring](introduction.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Schema](influxdb_schema.md) +- [Grafana Install/Configuration](grafana_configuration.md [influxdb-retention]: https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management [influxdb documentation]: https://docs.influxdata.com/influxdb/v0.9/ diff --git a/doc/monitoring/performance/influxdb_schema.md b/doc/monitoring/performance/influxdb_schema.md index a5a8aebd2d1..d31b3788f36 100644 --- a/doc/monitoring/performance/influxdb_schema.md +++ b/doc/monitoring/performance/influxdb_schema.md @@ -85,3 +85,4 @@ Read more on: - [Introduction to GitLab Performance Monitoring](introduction.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Configuration](influxdb_configuration.md) +- [Grafana Install/Configuration](grafana_configuration.md diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md index 20aa90f0d69..17bb75ececd 100644 --- a/doc/public_access/public_access.md +++ b/doc/public_access/public_access.md @@ -58,6 +58,9 @@ you are logged in or not. When visiting the public page of a user, you can only see the projects which you are privileged to. +If the public level is restricted, user profiles are only visible to logged in users. + + ## Restricting the use of public or internal projects In the Admin area under **Settings** (`/admin/application_settings`), you can diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index 612376e3a49..c44930a4ceb 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -4,6 +4,12 @@ Your GitLab instance can perform HTTP POST requests on the following events: `pr System hooks can be used, e.g. for logging or changing information in a LDAP server. +> **Note:** +> +> We follow the same structure from Webhooks for Push and Tag events, but we never display commits. +> +> Same deprecations from Webhooks are valid here. + ## Hooks request example **Request header**: @@ -240,3 +246,110 @@ X-Gitlab-Event: System Hook "user_id": 41 } ``` + +## Push events + +Triggered when you push to the repository except when pushing tags. + +**Request header**: + +``` +X-Gitlab-Event: System Hook +``` + +**Request body:** + +```json +{ + "event_name": "push", + "before": "95790bf891e76fee5e1747ab589903a6a1f80f22", + "after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "ref": "refs/heads/master", + "checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "user_id": 4, + "user_name": "John Smith", + "user_email": "john@example.com", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 15, + "project":{ + "name":"Diaspora", + "description":"", + "web_url":"http://example.com/mike/diaspora", + "avatar_url":null, + "git_ssh_url":"git@example.com:mike/diaspora.git", + "git_http_url":"http://example.com/mike/diaspora.git", + "namespace":"Mike", + "visibility_level":0, + "path_with_namespace":"mike/diaspora", + "default_branch":"master", + "homepage":"http://example.com/mike/diaspora", + "url":"git@example.com:mike/diaspora.git", + "ssh_url":"git@example.com:mike/diaspora.git", + "http_url":"http://example.com/mike/diaspora.git" + }, + "repository":{ + "name": "Diaspora", + "url": "git@example.com:mike/diaspora.git", + "description": "", + "homepage": "http://example.com/mike/diaspora", + "git_http_url":"http://example.com/mike/diaspora.git", + "git_ssh_url":"git@example.com:mike/diaspora.git", + "visibility_level":0 + }, + "commits": [], + "total_commits_count": 0 +} +``` + +## Tag events + +Triggered when you create (or delete) tags to the repository. + +**Request header**: + +``` +X-Gitlab-Event: System Hook +``` + +**Request body:** + +```json +{ + "event_name": "tag_push", + "before": "0000000000000000000000000000000000000000", + "after": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", + "ref": "refs/tags/v1.0.0", + "checkout_sha": "5937ac0a7beb003549fc5fd26fc247adbce4a52e", + "user_id": 1, + "user_name": "John Smith", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 1, + "project":{ + "name":"Example", + "description":"", + "web_url":"http://example.com/jsmith/example", + "avatar_url":null, + "git_ssh_url":"git@example.com:jsmith/example.git", + "git_http_url":"http://example.com/jsmith/example.git", + "namespace":"Jsmith", + "visibility_level":0, + "path_with_namespace":"jsmith/example", + "default_branch":"master", + "homepage":"http://example.com/jsmith/example", + "url":"git@example.com:jsmith/example.git", + "ssh_url":"git@example.com:jsmith/example.git", + "http_url":"http://example.com/jsmith/example.git" + }, + "repository":{ + "name": "Example", + "url": "ssh://git@example.com/jsmith/example.git", + "description": "", + "homepage": "http://example.com/jsmith/example", + "git_http_url":"http://example.com/jsmith/example.git", + "git_ssh_url":"git@example.com:jsmith/example.git", + "visibility_level":0 + }, + "commits": [], + "total_commits_count": 0 +} +``` diff --git a/doc/update/8.5-to-8.6.md b/doc/update/8.5-to-8.6.md index b9abcbd2c12..6267f14eba4 100644 --- a/doc/update/8.5-to-8.6.md +++ b/doc/update/8.5-to-8.6.md @@ -46,7 +46,7 @@ sudo -u git -H git checkout 8-6-stable-ee ```bash cd /home/git/gitlab-shell sudo -u git -H git fetch --all -sudo -u git -H git checkout v2.6.11 +sudo -u git -H git checkout v2.6.12 ``` ### 5. Update gitlab-workhorse diff --git a/doc/update/8.6-to-8.7.md b/doc/update/8.6-to-8.7.md index 8599133a726..4a2c6ea91d2 100644 --- a/doc/update/8.6-to-8.7.md +++ b/doc/update/8.6-to-8.7.md @@ -45,8 +45,8 @@ sudo -u git -H git checkout 8-7-stable-ee ```bash cd /home/git/gitlab-shell -sudo -u git -H git fetch --all -sudo -u git -H git checkout v2.7.0 +sudo -u git -H git fetch --all --tags +sudo -u git -H git checkout v2.7.2 ``` ### 5. Update gitlab-workhorse diff --git a/doc/update/README.md b/doc/update/README.md index 0241f036830..a770633c9b8 100644 --- a/doc/update/README.md +++ b/doc/update/README.md @@ -1,18 +1,95 @@ -Depending on the installation method and your GitLab version, there are multiple update guides. Choose one that fits your needs. +# Updating GitLab + +Depending on the installation method and your GitLab version, there are multiple +update guides. + +There are currently 3 official ways to install GitLab: + +- Omnibus packages +- Source installation +- Docker installation + +Based on your installation, choose a section below that fits your needs. + +--- + +<!-- START doctoc generated TOC please keep comment here to allow auto update --> +<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE --> +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Omnibus Packages](#omnibus-packages) +- [Installation from source](#installation-from-source) +- [Installation using Docker](#installation-using-docker) +- [Upgrading between editions](#upgrading-between-editions) + - [Community to Enterprise Edition](#community-to-enterprise-edition) + - [Enterprise to Community Edition](#enterprise-to-community-edition) +- [Miscellaneous](#miscellaneous) + +<!-- END doctoc generated TOC please keep comment here to allow auto update --> ## Omnibus Packages -- [Omnibus update guide](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/update.md) contains the steps needed to update a GitLab [package](https://about.gitlab.com/downloads/). +- The [Omnibus update guide](http://doc.gitlab.com/omnibus/update/README.html) + contains the steps needed to update an Omnibus GitLab package. ## Installation from source -- [The individual upgrade guides](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update) are for those who have installed GitLab from source. -- [The CE to EE update guides](https://gitlab.com/subscribers/gitlab-ee/tree/master/doc/update) are for subscribers of the Enterprise Edition only. The steps are very similar to a version upgrade: stop the server, get the code, update config files for the new functionality, install libs and do migrations, update the init script, start the application and check the application status. -- [Upgrader](upgrader.md) is an automatic ruby script that performs the update for installations from source. -- [Patch versions](patch_versions.md) guide includes the steps needed for a patch version, eg. 6.2.0 to 6.2.1. +- [Upgrading Community Edition from source][source-ce] - The individual + upgrade guides are for those who have installed GitLab CE from source. +- [Upgrading Enterprise Edition from source][source-ee] - The individual + upgrade guides are for those who have installed GitLab EE from source. +- [Patch versions](patch_versions.md) guide includes the steps needed for a + patch version, eg. 6.2.0 to 6.2.1, and apply to both Community and Enterprise + Editions. + +## Installation using Docker + +GitLab provides official Docker images for both Community and Enterprise +editions. They are based on the Omnibus package and instructions on how to +update them are in [a separate document][omnidocker]. + +## Upgrading between editions + +GitLab comes in two flavors: [Community Edition][ce] which is MIT licensed, +and [Enterprise Edition][ee] which builds on top of the Community Edition and +includes extra features mainly aimed at organizations with more than 100 users. + +Below you can find some guides to help you change editions easily. + +### Community to Enterprise Edition + +>**Note:** +The following guides are for subscribers of the Enterprise Edition only. + +If you wish to upgrade your GitLab installation from Community to Enterprise +Edition, follow the guides below based on the installation method: + +- [Source CE to EE update guides][source-ee] - Find your version, and follow the + `-ce-to-ee.md` guide. The steps are very similar to a version upgrade: stop + the server, get the code, update config files for the new functionality, + install libraries and do migrations, update the init script, start the + application and check its status. +- [Omnibus CE to EE][omni-ce-ee] - Follow this guide to update your Omnibus + GitLab Community Edition to the Enterprise Edition. + +### Enterprise to Community Edition + +If you need to downgrade your Enterprise Edition installation back to Community +Edition, you can follow [this guide][ee-ce] to make the process as smooth as +possible. ## Miscellaneous -- [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating your database from MySQL to PostgreSQL. -- [MySQL installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/database_mysql.md) contains additional information about configuring GitLab to work with a MySQL database. +- [MySQL to PostgreSQL](mysql_to_postgresql.md) guides you through migrating + your database from MySQL to PostgreSQL. +- [MySQL installation guide](../install/database_mysql.md) contains additional + information about configuring GitLab to work with a MySQL database. - [Restoring from backup after a failed upgrade](restore_after_failure.md) + +[omnidocker]: http://doc.gitlab.com/omnibus/docker/README.html +[source-ee]: https://gitlab.com/gitlab-org/gitlab-ee/tree/master/doc/update +[source-ce]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/update +[ee-ce]: ../downgrade_ee_to_ce/README.md +[ce]: https://about.gitlab.com/features/#community +[ee]: https://about.gitlab.com/features/#enterprise +[omni-ce-ee]: http://doc.gitlab.com/omnibus/update/README.html#from-community-edition-to-enterprise-edition diff --git a/doc/web_hooks/web_hooks.md b/doc/web_hooks/web_hooks.md index 22e207b6d32..c1c51302e79 100644 --- a/doc/web_hooks/web_hooks.md +++ b/doc/web_hooks/web_hooks.md @@ -41,6 +41,7 @@ X-Gitlab-Event: Push Hook "before": "95790bf891e76fee5e1747ab589903a6a1f80f22", "after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", "ref": "refs/heads/master", + "checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", "user_id": 4, "user_name": "John Smith", "user_email": "john@example.com", @@ -118,9 +119,10 @@ X-Gitlab-Event: Tag Push Hook ```json { "object_kind": "tag_push", - "ref": "refs/tags/v1.0.0", "before": "0000000000000000000000000000000000000000", "after": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", + "ref": "refs/tags/v1.0.0", + "checkout_sha": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", "user_id": 1, "user_name": "John Smith", "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", diff --git a/doc/workflow/README.md b/doc/workflow/README.md index 25893f948ea..9efe41308dc 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -20,6 +20,7 @@ - [Milestones](milestones.md) - [Merge Requests](merge_requests.md) - [Revert changes](revert_changes.md) +- [Cherry-pick changes](cherry_pick_changes.md) - ["Work In Progress" Merge Requests](wip_merge_requests.md) - [Merge When Build Succeeds](merge_when_build_succeeds.md) - [Manage large binaries with Git LFS](lfs/manage_large_binaries_with_git_lfs.md) diff --git a/doc/workflow/cherry_pick_changes.md b/doc/workflow/cherry_pick_changes.md new file mode 100644 index 00000000000..4a499009842 --- /dev/null +++ b/doc/workflow/cherry_pick_changes.md @@ -0,0 +1,53 @@ +# Cherry-pick changes + +>**Note:** +This feature was [introduced][ce-3514] in GitLab 8.7. + +--- + +GitLab implements Git's powerful feature to [cherry-pick any commit][git-cherry-pick] +with introducing a **Cherry-pick** button in Merge Requests and commit details. + +## Cherry-picking a Merge Request + +After the Merge Request has been merged, a **Cherry-pick** button will be available +to cherry-pick the changes introduced by that Merge Request: + + + +--- + +You can cherry-pick the changes directly into the selected branch or you can opt to +create a new Merge Request with the cherry-pick changes: + + + +## Cherry-picking a Commit + +You can cherry-pick a Commit from the Commit details page: + + + +--- + +Similar to cherry-picking a Merge Request, you can opt to cherry-pick the changes +directly into the target branch or create a new Merge Request to cherry-pick the +changes: + + + +--- + +Please note that when cherry-picking merge commits, the mainline will always be the +first parent. If you want to use a different mainline then you need to do that +from the command line. + +Here is a quick example to cherry-pick a merge commit using the second parent as the +mainline: + +```bash +git cherry-pick -m 2 7a39eb0 +``` + +[ce-3514]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3514 "Cherry-pick button Merge Request" +[git-cherry-pick]: https://git-scm.com/docs/git-cherry-pick "Git cherry-pick documentation" diff --git a/doc/workflow/img/cherry_pick_changes_commit.png b/doc/workflow/img/cherry_pick_changes_commit.png Binary files differnew file mode 100644 index 00000000000..ae91d2cae53 --- /dev/null +++ b/doc/workflow/img/cherry_pick_changes_commit.png diff --git a/doc/workflow/img/cherry_pick_changes_commit_modal.png b/doc/workflow/img/cherry_pick_changes_commit_modal.png Binary files differnew file mode 100644 index 00000000000..f502f87677a --- /dev/null +++ b/doc/workflow/img/cherry_pick_changes_commit_modal.png diff --git a/doc/workflow/img/cherry_pick_changes_mr.png b/doc/workflow/img/cherry_pick_changes_mr.png Binary files differnew file mode 100644 index 00000000000..59c610e620b --- /dev/null +++ b/doc/workflow/img/cherry_pick_changes_mr.png diff --git a/doc/workflow/img/cherry_pick_changes_mr_modal.png b/doc/workflow/img/cherry_pick_changes_mr_modal.png Binary files differnew file mode 100644 index 00000000000..96a80f4726d --- /dev/null +++ b/doc/workflow/img/cherry_pick_changes_mr_modal.png diff --git a/doc/workflow/importing/import_projects_from_github.md b/doc/workflow/importing/import_projects_from_github.md index f693f430a42..e670e415c71 100644 --- a/doc/workflow/importing/import_projects_from_github.md +++ b/doc/workflow/importing/import_projects_from_github.md @@ -1,7 +1,8 @@ # Import your project from GitHub to GitLab
-_**Note:** In order to enable the GitHub import setting, you should first
-enable the [GitHub integration][gh-import] in your GitLab instance._
+>**Note:**
+In order to enable the GitHub import setting, you should first
+enable the [GitHub integration][gh-import] in your GitLab instance.
At its current state, GitHub importer can import:
@@ -10,10 +11,13 @@ At its current state, GitHub importer can import: - the issues (introduced in GitLab 7.7)
- the pull requests (introduced in GitLab 8.4)
- the wiki pages (introduced in GitLab 8.4)
+- the milestones (introduced in GitLab 8.7)
+- the labels (introduced in GitLab 8.7)
-It is not yet possible to import your labels, milestones and cross-repository
-pull requests (those from forks). We are working on improving this in the near
-future.
+With GitLab 8.7+, references to pull requests and issues are preserved.
+
+It is not yet possible to import your cross-repository pull requests (those from
+forks). We are working on improving this in the near future.
The importer page is visible when you [create a new project][new-project].
Click on the **GitHub** link and you will be redirected to GitHub for
diff --git a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md index ba91685a20b..31620044b15 100644 --- a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md +++ b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md @@ -44,7 +44,7 @@ check it into your Git repository: ```bash git clone git@gitlab.example.com:group/project.git -git lfs init # initialize the Git LFS project project +git lfs install # initialize the Git LFS project project git lfs track "*.iso" # select the file extensions that you want to treat as large files ``` diff --git a/doc/workflow/shortcuts.png b/doc/workflow/shortcuts.png Binary files differindex 83e562d6929..beb6c53ec77 100644 --- a/doc/workflow/shortcuts.png +++ b/doc/workflow/shortcuts.png diff --git a/doc/workflow/web_editor.md b/doc/workflow/web_editor.md index 5685a9d89dd..1832567a34c 100644 --- a/doc/workflow/web_editor.md +++ b/doc/workflow/web_editor.md @@ -85,7 +85,7 @@ Once you click it, a new branch will be created that diverges from the default branch of your project, by default `master`. The branch name will be based on the title of the issue and as suffix it will have its ID. Thus, the example screenshot above will yield a branch named -`et-cum-et-sed-expedita-repellat-consequatur-ut-assumenda-numquam-rerum-2`. +`2-et-cum-et-sed-expedita-repellat-consequatur-ut-assumenda-numquam-rerum`. After the branch is created, you can edit files in the repository to fix the issue. When a merge request is created based on the newly created branch, |
